Multi-modal video abstract generation method based on time sequence alignment and semantic enhancement

Abstract

The invention belongs to the technical field of computer vision, relates to a time sequence alignment and semantic enhancement-based multi-modal video abstract generation method, and designs a time sequence alignment and semantic enhancement-based multi-modal video abstract generation model, and outputs a corresponding video abstract according to video data; the model comprises a feature extraction module, a dual-scale attention module and a video abstract generation module, the feature extraction module is used for extracting visual features and text features, and the dual-scale attention module comprises a conditional self-attention network and a cooperative interaction cross-modal attention network. The visual features and the text features are aligned and fused on the local scale and the global scale respectively; the model is trained by using a back propagation algorithm, and classification loss, dynamic time warping loss based on different scales and semantic enhancement hinge loss are introduced into a loss function. Compared with the prior art, the generation effect of the video abstract can be further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a multimodal video summary generation method based on temporal alignment and semantic enhancement. Background Art

[0002] The task of video summarization aims to condense the video content into a concise and information-rich video, capturing the essence of the content while reducing the time required to watch. The process involves using video feature information to select key frames or key shots that represent the core content of the video to retain important information in the video. Initially, single-modal methods were usually used, that is, only visual features were used to obtain summary videos, but with the emergence of multimodal technology, video summarization has become more comprehensive and the information used is richer. The goal of multimodal video summarization methods is to train a model that can effectively identify the most relevant content in the video by utilizing the interactive information between different modalities, thereby obtaining a summary that highly summarizes the original video content. Since the video itself contains multiple modalities such as visual modality, audio modality, and text modality, by comprehensively utilizing these modalities, the content and core of the video can be better understood, thereby improving the performance of video summarization.

[0003] In the context of multimodal video summarization, existing methods usually adopt an encoder-decoder model, which encodes text and image / video pairs and then generates a summary. With the further development of deep learning technology, some methods have introduced self-attention mechanisms in the model, allowing the model to dynamically adjust the importance of different parts. Self-attention calculates the relationship between elements in the input sequence and generates attention weights, so that the model pays more attention to the information most relevant to the current generated content when processing the input. This mechanism is particularly important when processing long texts or long video frames, because it allows the model to capture long-distance dependencies and improve the selection of semantic representation. Recently, some methods have chosen a Transformer-based framework, the core of which is also the self-attention mechanism, which captures complex relationships in the sequence through global context information instead of relying solely on local context. Such methods enable the model to more effectively fuse visual and text features, thereby obtaining richer visual-text representations and enhancing the effect of multimodal video summarization.

[0004] The end-to-end video summarization method based on the Transformer architecture can effectively obtain multimodal feature information, and the information obtained has good temporal and semantic properties. Therefore, this technology has achieved good experimental results in some multimodal video summarization tasks. However, this technology still has the following defects: First, existing methods usually use a single-scale (global or local) Transformer structure, which leads to insufficient utilization of temporal information in each modality, that is, the global structure may ignore local details, while the local structure may not be able to integrate the global context. This limitation not only affects the model's understanding of temporal relationships, but also may lead to fragmentation of information, making it difficult for the model to effectively extract important frames or fragments.

[0005] Second, current work focuses on handling the semantic alignment of multimodal features through simple strategies, such as sequence modeling, which aims to model cross-modal correlations. However, sequence modeling usually ignores the complex interactions between modalities, which may lead to information loss or misunderstanding, thereby affecting the overall performance of the model in multimodal semantic understanding.

[0006] Therefore, it is necessary to design a new video summary generation method to further improve the generation effect of video summaries. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multimodal video summary generation method based on temporal alignment and semantic enhancement, so as to further improve the relevance, completeness and accuracy of the generated video summary.

[0008] The purpose of the present invention can be achieved by the following technical solutions: The present invention provides a multimodal video summary generation method based on temporal alignment and semantic enhancement, comprising the following steps: Obtain video data, input the trained multimodal video summary generation model based on temporal alignment and semantic enhancement, and output the corresponding video summary; The multimodal video summary generation model based on temporal alignment and semantic enhancement includes a feature extraction module, a dual-scale attention module and a video summary generation module connected in sequence, wherein the feature extraction module is used to extract visual features from input data through a pre-trained model. and text features The dual-scale attention module includes a parallel conditional self-attention network and a collaborative interactive cross-modal attention network. The conditional self-attention network integrates text features As conditional information, visual features are and text features Align and fuse to generate conditional video representation The collaborative interactive cross-modal attention network uses the attention mechanism to analyze visual features at a global scale. and text features Perform alignment and fusion to generate global text representation and global video representation ; The video summary generation module is used to create a video summary based on the output of the dual-scale attention module; The multimodal video summary generation model based on temporal alignment and semantic enhancement is trained using a back propagation algorithm. The loss function in the back propagation algorithm includes classification loss, dynamic time warping loss based on different scales, and semantic enhancement hinge loss. The dynamic time warping loss based on different scales is constructed based on Soft-DTW, and videos of different scales are represented in time sequence, including the conditional video representation. and the global video representation For alignment, the semantically enhanced hinge loss is constructed based on a semantic factor calculated based on the semantic difference information between hard negative samples.

[0009] Furthermore, the conditional self-attention network includes a self-attention layer and a text condition layer. The self-attention layer uses a multi-head self-attention mechanism to Calculate the dependencies between different frames; the text condition layer is based on the text features The conditional information is integrated into the output of the self-attention layer to generate a conditional video representation .

[0010] Furthermore, the conditional information is integrated into the output of the self-attention layer to generate a conditional video representation The process is as follows: , in, is the Sigmoid activation function, is a fully connected layer, is the output of the self-attention layer.

[0011] Furthermore, the collaborative interactive cross-modal attention network generates a global text representation and global video representation The process is as follows: Visual features and text features Perform linear mapping, perform matrix multiplication on the mapping result, and calculate visual features and text features The similarity matrix between ; Based on the similarity matrix Perform multi-head self-attention calculations to generate global text representations and global video representation .

[0012] Furthermore, the similarity matrix The calculation formula is as follows: , in, is the activation function, and is a linear layer.

[0013] Furthermore, a global text representation is generated through multi-head self-attention calculation and global video representation The process is as follows: , , , , in, and are the attention matrices for video and text modalities respectively, is the Sigmoid activation function.

[0014] Furthermore, the video summary generation module creates a video summary according to the output of the dual-scale attention module through KTS and Knapsack algorithms.

[0015] Furthermore, the expressions of the dynamic time warping loss based on different scales are as follows: , in, is the dynamic time warping loss based on different scales, is the smoothing parameter, , is a set of binary alignment matrices, is the alignment matrix, Based on conditional video representation and global video representation Compute the pairwise distance matrix.

[0016] Furthermore, the expression of the semantic enhancement hinge loss is as follows: , in, Hinge loss for semantic enhancement, is the interval parameter, is the similarity score between the visual representation and its negative textual representation, is the similarity score between the textual representation and its negative visual representation, is the similarity score between positive visual-text representation pairs, and is the semantic factor, and the calculation formula is as follows: , in, is the temperature parameter, for and The similarity score of and is the reduced set of description vectors derived from the dimension-reduced matrix.

[0017] Furthermore, the reduced description vector set derived from the dimension reduction matrix and The generation process is as follows: Obtain video data and its text description, process the text description through truncated singular value decomposition, and obtain a reduced dimension matrix; Calculate the cosine similarity between the rows of the reduced dimension matrix and its transposed matrix to obtain the information carrying semantic differences and .

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention proposes a multimodal video summary generation method based on temporal alignment and semantic enhancement, and designs a multimodal video summary generation model based on temporal alignment and semantic enhancement, which can generate corresponding video summaries according to video data. Specifically, the multimodal video summary generation model based on temporal alignment and semantic enhancement includes a feature extraction module, a dual-scale attention module and a video summary generation module connected in sequence. The feature extraction module is used to extract visual features from input data through a pre-trained model. and text features The dual-scale attention module includes a parallel conditional self-attention network and a collaborative interactive cross-modal attention network, which can fully utilize and effectively integrate information from different modalities at multiple scales. Among them, the conditional self-attention network integrates text features As conditional information, visual features are and text features Align and fuse to generate conditional video representation , which can further enhance the modeling of representations in the video modality, so that the model can effectively extract important frames or fragments; the collaborative interactive cross-modal attention network uses the attention mechanism to analyze visual features at a global scale. and text features Align and fuse to generate global text representation and global video representation , which can capture long-range dependencies. The above model is trained using the back-propagation algorithm, and considers three types of losses: classification loss, dynamic time warping loss based on different scales, and semantic enhancement hinge loss. Among them, the dynamic time warping loss based on different scales is built based on Soft-DTW, and the visual representations of different scales are aligned in time. Considering the dynamic time warping loss based on different scales in model training can improve the model's sensitivity to features of different scales and effectively utilize the temporal information within the modality; the semantic enhancement hinge loss is built based on the semantic factor, which is calculated based on the semantic difference information between difficult negative samples. It can model the semantic dependency between difficult negative samples. Considering the semantic enhancement hinge loss in model training can capture the semantic differences conveyed by difficult negative samples. Through the above method, the relevance, completeness and accuracy of the generated video summary can be further improved.

[0019] 2. In the present invention, the conditional self-attention network specifically includes a self-attention layer and a text condition layer. The self-attention layer uses a multi-head self-attention mechanism to Calculating the dependencies between different frames helps to understand complex information such as continuous actions and scene transitions in the video; the text conditional layer uses text features The conditional information is integrated into the output of the self-attention layer to generate conditional video representation , which realizes the cross-modal fusion of visual and textual information and can avoid the loss or misunderstanding of information.

[0020] 3. The present invention creates a video summary based on the output of the dual-scale attention module through the KTS and Knapsack algorithms, wherein the KTS algorithm can calculate the video segment level score from the video frame level score, and the Knapsack algorithm can select the optimal combination under limited resources. By combining the two with the dual-scale attention module, a high-quality video summary can be quickly generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the structure of a multimodal video summary generation model based on temporal alignment and semantic enhancement. in, For visual features, is the text feature, is the conditional video representation, is the global text representation, is the global video representation; Figure 2 is a schematic diagram of the structure of the conditional self-attention network. in, , , , , is a linear layer, is the output of the self-attention layer; Figure 3 This is a diagram of the visualization results of the qualitative experiment of the 9th video in the TVSum dataset. The histogram represents the importance score of the true value, and the darkened part is the true value; Figure 4 This is a schematic diagram of the visualization results of the qualitative experiment of the 9th video in the TVSum dataset. The histogram represents the importance score of the true value, and the darkened part is the part selected by the model of the present invention; Figure 5 This is a diagram of the visualization results of the qualitative experiment for the 50th video in the TVSum dataset. The histogram represents the importance score of the true value, and the darkened part is the true value. Figure 6 This is a schematic diagram of the visualization results of the qualitative experiment of the 50th video in the TVSum dataset. The histogram represents the importance score of the true value, and the darkened part is the part selected by the model of the present invention. DETAILED DESCRIPTION

[0022] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0023] Example: This embodiment provides a multimodal video summary generation method based on temporal alignment and semantic enhancement, and the specific steps are as follows: Obtain video data, input the trained multimodal video summary generation model based on temporal alignment and semantic enhancement, and output the corresponding video summary.

[0024] like Figure 1 As shown in Figure 1, the multimodal video summary generation model based on temporal alignment and semantic enhancement includes a feature extraction module, a dual-scale attention module, and a video summary generation module connected in sequence. The feature extraction module uses GoogLeNet pre-trained on ImageNet to extract visual features from video frames. , as the input of the visual modality, the pre-trained model GPT-2 is used to generate text descriptions for each frame, and the pre-trained RoBERTa is used to extract text features for each sentence , as input to the text modal.

[0025] The dual-scale attention module includes a parallel conditional self-attention network and a collaborative interactive cross-modal attention network to align and fuse multimodal video features at both local and global scales. Specifically, the conditional self-attention network plays an important role at the local scale. The traditional self-attention mechanism calculates attention weights only based on the characteristics of the input sequence itself. This method enables the model to assign different importance to different parts of the input sequence, thereby promoting more effective representation learning; however, the disadvantage of the traditional self-attention mechanism is that it is limited to a single modality and cannot utilize additional information from other modalities. To solve this problem, this embodiment proposes a conditional self-attention network that combines text features into a single modality. As conditional information, visual features are and text features Align and fuse to generate conditional video representation .

[0026] like Figure 2 As shown, the conditional self-attention network includes a self-attention layer and a text conditional layer. The self-attention layer uses a multi-head self-attention mechanism to model the dependencies between different frames and calculate the dependencies between different frames. This embodiment directly calculates the self-attention using a standard method: , in, , , Generate , , The linear layer, is the attention weight, is the feature dimension, is a linear layer. Then the output of the self-attention layer is generated through residual connection and layer normalization .

[0027] The text conditional layer is based on text features The conditional information is integrated into the output of the self-attention layer to generate conditional video representation , to enhance the exploration of multimodal data interactions: , in, is the Sigmoid activation function, is a fully connected layer, is the output of the self-attention layer.

[0028] However, conditional self-attention networks alone are not enough to fully utilize the information in multimodal data. Therefore, this embodiment proposes a collaborative interactive cross-modal attention network to model multimodal data from another global scale to supplement the understanding of multimodal data. The network outputs a global text representation and global video representation , these multimodal outputs will serve as inputs to subsequent loss functions to achieve comprehensive training of the model and further align and fuse multimodal features.

[0029] Collaborative Interactive Cross-modal Attention Network for Generating Global Text Representation and global video representation The specific process is as follows: First, through the linear layer Visual Features Perform linear mapping through the linear layer Text features Perform linear mapping and perform matrix multiplication on the mapping results to calculate visual features and text features The similarity matrix between , to facilitate the identification of cross-modal interactions. The similarity matrix is ​​calculated as follows: , in, is the activation function, and is a linear layer.

[0030] Then, based on the similarity matrix Perform multi-head self-attention calculations to adjust the influence of one modality on another. Specifically, calculate the attention matrix to capture the correlation between the modalities, thereby achieving cross-modal information fusion. The corresponding attention matrix for each modality is calculated as follows: , , in, and are the attention matrices for video and text modalities respectively, is the Sigmoid activation function.

[0031] Next, the attention matrix is ​​used to generate a multimodal fusion representation, that is, a text representation of the video interaction. Video representations that interact with text , in which each modality is affected by the other modality. This approach integrates multimodal information based on their respective attention mechanisms, which can further improve the summary generation effect.

[0032] , .

[0033] Both the conditional self-attention network and the collaborative interactive cross-modal attention network are deployed with a feed-forward neural network FFN. The outputs of the two networks are processed by FNN to obtain the prediction of the current frame.

[0034] The video summary generation module uses KTS (for details, see the paper Category-specific videosummarization, Potapov, Danila, Matthijs Douze, Zaid Harchaoui, and CordeliaSchmid., In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pp. 540-555. Springer International Publishing, 2014) and Knapsack algorithm, and creates a video summary based on the output of the dual-scale attention module. Specifically, this embodiment uses the segmentation result based on the KTS algorithm, calculates the video segment level score based on the video frame level score, and selects the top 15% of the video segments through the Knapsack algorithm to generate the final video summary.

[0035] This embodiment uses a back-propagation algorithm to train a multimodal video summary generation model based on temporal alignment and semantic enhancement. Three different loss functions are considered in the back-propagation algorithm, including classification loss, dynamic time warping loss based on different scales, and semantic enhancement hinge loss. The latter two losses can enhance the video summary in terms of timing and semantics.

[0036] This embodiment selects the video data sets TVSum and SumMe, and forms training and verification samples in accordance with the five-fold cross-validation principle. The network is trained by forward propagation using the training samples, and the effect of the trained model is verified on the test set. Forward propagation includes the calculation of the dual-scale attention network and the feedforward neural network. In the process of network training, the multimodal features after network alignment and fusion and the prediction scores of each sample are obtained. The multimodal features and prediction scores are used to propagate forward and backward from the last layer of the model in sequence. The loss functions used for back propagation include classification loss, dynamic time warping loss based on different scales, and semantically enhanced hinge loss. At the same time, the adaptive moment estimation Adam is used as the optimizer to update the gradient and modify the weight matrix of each layer.

[0037] The formula for classification loss is: , in, is a hyperparameter used to adjust the ratio between positive and negative sample losses; It is also a hyperparameter used to control the ratio between the loss of difficult and easy samples; Indicates a specific mode, is the sample size, represents the prediction score for each frame / sentence, represents the true label, if , then it means Frame / No. A sentence is a key frame / key sentence, and vice versa. The final classification loss is the sum of the two unimodal losses.

[0038] In order to utilize the temporal information between the video representations output by two networks of different scales, inspired by the temporal alignment in video representation learning, this embodiment adopts Soft-DTW (see the paper Soft-dtw: a differentiable loss function for time-series, Cuturi, Marco, and Mathieu Blondel.,In International conference on machine learning , pp. 894-903. PMLR, 2017) loss models the temporal alignment information between visual features. Soft-DTW is a differentiable variant of classic DTW and has been widely used in time series data, including video data. This embodiment uses outputs from different scales to model the correlation of time series and proposes a dynamic time warping loss based on different scales, scale-based Soft-DTW. The specific formula is as follows: , in, is the smoothing parameter, when When , Soft-DTW degenerates into classic DTW; , is a set of binary alignment matrices, To align the matrix, count from the first element of the main diagonal to the last element in the direction {↓, ↘, →}; Based on conditional video representation and global video representation Compute the pairwise distance matrix.

[0039] In contrastive learning, the commonly used loss function for difficult negative samples is the maximum hinge loss (LMH). This loss function targets the irrelevant visual-text feature pairs that are closest to the relevant sample features. Although LMH focuses on learning the semantic information carried by the difficult negative samples themselves, it fails to model the semantic dependencies between these difficult negative samples. To solve this problem, this embodiment introduces two semantic factors and constructs a new loss function, namely the semantically enhanced hinge loss, to model the semantic dependencies between difficult negative samples. The formula of the semantically enhanced hinge loss is as follows: , in, is the interval parameter, is the similarity score between the visual representation and its negative textual representation, is the similarity score between the textual representation and its negative visual representation, is the similarity score between positive visual-text representation pairs, and is a semantic factor, which is used to model the semantic dependency between difficult negative samples. The calculation formula is as follows: , in, is the temperature parameter, for and The similarity score of and is a set of reduced descriptions vectors derived from the dimension reduction matrix, and the generation process is as follows: First, get the video data and its text description (include sentences, each sentence includes words), the text description is processed by truncated singular value decomposition (SVD) to obtain a vector rows and The dimension reduction matrix of The truncated SVD formula is as follows: , , in, is a left singular vector matrix, is a right singular vector matrix, is a diagonal matrix with diagonal elements The singular values ​​of .

[0040] Since the description of the video may be redundant, in the dimension reduction matrix Compare the similarities between sentences and select the sentence with the best semantic similarity. sentences, and The number of is set as the batch size This can effectively eliminate the noise caused by repeated sentences and calculate the dimension reduction matrix The cosine similarity between the rows of its transposed matrix can be used to obtain the semantic difference information. and .

[0041] In summary, compared with the prior art, the method proposed in this embodiment has the following three innovations: First, a dual attention module consisting of a conditional self-attention network and a collaborative interactive cross-modal attention network is proposed to align and fuse multimodal video features at both local and global scales, which can make full use of information from different modalities.

[0042] Second, based on the dual-scale attention architecture, Soft-DTW is improved and a dynamic time warping loss based on different scales is proposed. The video representations of different scales extracted from the two attention networks are temporally aligned, which can make full use of the temporal information within the modality.

[0043] Third, a semantically enhanced hinge loss (SEHLoss) is proposed, which can capture the semantic differences between different hard negative samples and effectively utilize the semantic information between different modal features.

[0044] To verify the superiority of the above method, this embodiment conducted quantitative experiments on two public datasets (SumMe and TVSum). The results are shown in Table 1.

[0045] Table 1 Quantitative experiments

[0046] From the comparison in Table 1, it can be seen that the proposed method has achieved significant performance improvement over existing methods on the SumMe and TVSum datasets. Specifically, the proposed method performs particularly well on the SumMe dataset. Except for the τ indicator, the proposed method outperforms VJMHT in the other two indicators (i.e., F1 score and ρ). On the TVSum dataset, the proposed method also performs well, achieving the best F1 score, surpassing most existing methods; compared with the best performing CA-SUM and CTVSUM, there is only a small performance gap in the τ and ρ indicators. In general, by constructing a dual-scale attention network, the inherent temporal information in the video and the semantic differences conveyed by difficult negative samples are fully utilized, which helps the model achieve better performance.

[0047] In order to evaluate the contribution of each component in the present invention, this embodiment conducted an ablation experiment, and the results are shown in Table 2.

[0048] Table 2 Ablation experiment

[0049] This embodiment first explores whether the introduction of text modality input can improve performance, that is, whether the performance is improved when multimodal input is used. From the overall data presented in Table 2, the importance of learning the temporal information between video representations of different scales and utilizing the additional semantic information provided by difficult negative samples is verified. After the introduction of the dual-scale attention network, the network uses both video and text modalities, and the F1 score is significantly improved by about 1%. In addition, the addition of dynamic time warping losses based on different scales increases the F1 score from 50.0% to 52.2%. Similarly, the integration of semantic enhancement hinge loss also brings a 1.3% performance improvement. Finally, when all the proposed components are combined, the method of the present invention shows a significant improvement in the final performance, with the F1 score increasing from 50.0% to 53.6%. Considering that the F1 score may be unreliable and prone to irregular fluctuations, this embodiment also uses τ and ρ to evaluate the model performance, which increases from 0.089 to 0.093 and from 0.092 to 0.112, respectively.

[0050] The 9th video in the TVSum dataset is a BBC news video discussing a train-car collision, and the 50th video discusses whether a cat will eat dog food. Figure 4 The visualization results of the 9th video in the TVSum dataset are shown. Figure 5 and Figure 6 The visualization results of the 50th video in the TVSum dataset are shown, which is a qualitative experiment. The histogram represents the importance score of the true value, and the darkened part corresponds to the true value or the part selected by the model of the present invention. The method of the present invention generates a high-quality summary and effectively captures the temporal information of the video. Although slight differences in the results are observed in some frames, the salient areas in the true annotation are almost all captured. Specifically, as shown in Figures 3 and Figure 4 As shown in the comparison, the proposed model mainly ignores some scattered frames at the beginning and middle of the 9th video, and hardly selects any content beyond the real labeled selected frames; Figure 5 and Figure 6 As shown in the comparison, although slight differences are observed in some frames of the results of the 50th video, the performance of the model of the present invention is highly consistent with the true value annotation.

[0051] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0052] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A multimodal video summary generation method based on temporal alignment and semantic enhancement, characterized in that: The following steps are involved: Obtain video data, input the trained multimodal video summary generation model based on temporal alignment and semantic enhancement, and output the corresponding video summary; The multimodal video summary generation model based on temporal alignment and semantic enhancement includes a feature extraction module, a dual-scale attention module and a video summary generation module connected in sequence, wherein the feature extraction module is used to extract visual features from input data through a pre-trained model. and text features The dual-scale attention module includes a parallel conditional self-attention network and a collaborative interactive cross-modal attention network. The conditional self-attention network integrates text features As conditional information, visual features are and text features Align and fuse to generate conditional video representation The collaborative interactive cross-modal attention network uses the attention mechanism to analyze visual features at a global scale. and text features Align and fuse to generate global text representation and global video representation ; The video summary generation module is used to create a video summary based on the output of the dual-scale attention module; The multimodal video summary generation model based on temporal alignment and semantic enhancement is trained using a back propagation algorithm. The loss function in the back propagation algorithm includes classification loss, dynamic time warping loss based on different scales, and semantic enhancement hinge loss. The dynamic time warping loss based on different scales is constructed based on Soft-DTW, and videos of different scales are represented in time sequence, including the conditional video representation. and the global video representation For alignment, the semantically enhanced hinge loss is constructed based on a semantic factor calculated based on the semantic difference information between hard negative samples.

2. According to claim 1, a multimodal video summary generation method based on temporal alignment and semantic enhancement is characterized in that: The conditional self-attention network includes a self-attention layer and a text condition layer. The self-attention layer uses a multi-head self-attention mechanism to Calculate the dependencies between different frames; the text condition layer is based on the text features The conditional information is integrated into the output of the self-attention layer to generate a conditional video representation .

3. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 2, characterized in that: Incorporate the conditional information into the output of the self-attention layer to generate a conditional video representation The process is as follows: , in, is the Sigmoid activation function, is a fully connected layer, is the output of the self-attention layer.

4. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 1, characterized in that: The collaborative interactive cross-modal attention network generates a global text representation and global video representation The process is as follows: Visual features and text features Perform linear mapping, perform matrix multiplication on the mapping result, and calculate visual features and text features The similarity matrix between ; Based on the similarity matrix Perform multi-head self-attention calculations to generate global text representations and global video representation .

5. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 4, characterized in that: Similarity Matrix The calculation formula is as follows: , in, is the activation function, and is a linear layer.

6. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 4, characterized in that: Generating global text representation via multi-head self-attention computation and global video representation The process is as follows: , , , , in, and are the attention matrices for video and text modalities respectively, is the Sigmoid activation function.

7. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 1, characterized in that: The video summary generation module creates a video summary based on the output of the dual-scale attention module through KTS and Knapsack algorithms.

8. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 1, characterized in that: The expressions of the dynamic time warping loss based on different scales are as follows: , in, is the dynamic time warping loss based on different scales, is the smoothing parameter, , is a set of binary alignment matrices, is the alignment matrix, Based on conditional video representation and global video representation Compute the pairwise distance matrix.

9. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 1, characterized in that: The expression of the semantic enhancement hinge loss is as follows: , in, Hinge loss for semantic enhancement, is the interval parameter, is the similarity score between the visual representation and its negative textual representation, is the similarity score between the textual representation and its negative visual representation, is the similarity score between positive visual-text representation pairs, and is the semantic factor, and the calculation formula is as follows: , in, is the temperature parameter, for and The similarity score of and is the reduced set of description vectors derived from the dimension-reduced matrix.

10. The method for generating multimodal video summarization based on temporal alignment and semantic enhancement according to claim 9, characterized in that: Reduced set of description vectors from the dimension-reduced matrix and The generation process is as follows: Obtain video data and its text description, process the text description through truncated singular value decomposition, and obtain a reduced dimension matrix; Calculate the cosine similarity between the rows of the reduced dimension matrix and its transposed matrix to obtain the information carrying semantic differences and .

Images