A dense video event description method based on multimodal representation

Abstract

The present invention discloses a dense video event description method based on multimodal representation, the method comprising: using a multimodal feature fusion encoder to obtain encoded visual features, audio features and modal information sharing features; obtaining the final generation prediction result through a proposal generator; obtaining the final description prediction result through a description generator; calculating the contrast loss and KL divergence loss function, and using the back propagation method to update the network parameters of the multimodal feature fusion encoder and the proposal generator. The present invention introduces modal information sharing features to map features of different modalities to the same feature space, interact and align feature information, and alleviate the modal gap that persists between heterogeneous modalities; by introducing contrast loss in separate modal features and modal information sharing features, the contextual semantic information in the features is enriched to the greatest extent, and the loss of semantic information when different modal features are mapped to the same feature space is avoided; the performance of event description generation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of dense video event description methods, and in particular to a dense video event description method based on multimodal representation. Background Art

[0002] With the rapid development of natural language processing and computer vision, video description has received more and more attention in recent years. The purpose of traditional video description tasks is to describe the content of a short video in one sentence. The video is usually manually cropped to ensure that only one main event occurs in the video. Great progress has been made in this field, but real videos are always uncut and complex. Limited by this condition, traditional video description tasks cannot be well applied to real-life video data, and dense video description tasks have been proposed.

[0003] The dense video event description task requires the model to locate all possible events in an untrimmed video and describe each event in one sentence. The dense video event description task brings two challenges: one is that the video event proposal requires the model to locate all possible events in the video, and the other is how to convert the video event proposal into a natural language description. For locating all possible events in the video, compared with the video description of a single event video clip, dense video event description requires the model to understand longer and more complex video feature sequences. Many well-designed methods currently rely only on visual information and ignore the audio information in the video. Some work only uses simple feature connections for the use of multimodal information, resulting in insufficient utilization of valuable multimodal information.

[0004] In order to better utilize the multimodal information in the video, it is urgent to design a dense video event description model that can capture the commonalities between different modal features, thereby bridging their heterogeneous differences and alleviating the problem of multimodal information fusion. Inspired by the research in the field of audio-visual speech recognition, it is shown that the multimodal model can benefit from individual modal information features and modal information sharing features. A new multimodal dense video event description method proposed in the present invention introduces modal information sharing features in the dense video event description task to achieve video event description generation and alleviate the persistent modal gap between heterogeneous modalities. At the same time, considering that mapping individual modal information features and modal information sharing features to the same feature space at the same time may cause contextual speech information loss, contrastive learning is introduced to define the frame indexes that express the same semantics in the individual modal information features and modal information sharing features as positive samples, so as to enrich the contextual semantic information represented by the final modal features to the greatest extent. Summary of the invention

[0005] The purpose of the present invention is to provide a dense video event description method based on multimodal representation in view of the deficiencies of the prior art.

[0006] The objective of the present invention is achieved through the following technical solution: A dense video event description method based on multimodal representation comprises the following steps:

[0007] (1) extracting initial visual features of a video dataset through a pre-trained visual feature extraction network 13D, extracting initial audio features of a video dataset through a pre-trained audio feature extraction network VGGish, and using a multimodal feature fusion encoder to obtain encoded visual features, audio features, and modal information sharing features; wherein the multimodal feature fusion encoder includes a linear layer, a self-attention mechanism, a cross-attention mechanism, and a fully connected layer;

[0008] (2) The encoded audio features and visual features are sent to the proposal generator, and sequentially pass through the first-layer parallel proposal head, the proposal feature interaction layer, the second-layer parallel proposal head and the proposal object pool of the proposal generator, and finally select the K proposals with the highest confidence from the proposal object pool as the final generated prediction results; wherein the first-layer parallel proposal head includes multiple parallel proposal heads, the proposal feature interaction layer includes an audio feature proposal pool and a visual feature proposal pool, and the second-layer parallel proposal head includes multiple parallel proposal heads;

[0009] (3) sending the encoded audio features, visual features and modal information sharing features to a description generator to obtain a final description prediction result; wherein the description generator includes an audio feature addition fusion block, a visual feature addition fusion block and a feedforward layer, the audio feature addition fusion block includes a self-attention mechanism and a cross-attention mechanism, the visual feature addition fusion block includes a self-attention mechanism and a cross-attention mechanism, and the feedforward layer includes a fully connected layer and a softmax function;

[0010] (4) Calculate the contrast loss based on the encoded visual features, audio features, and modal information sharing features, and calculate the KL divergence loss function based on the final generated prediction results and the final description prediction results. Minimize the contrast loss and KL divergence loss function as the optimization goal, and use the back propagation method to update the network parameters of the multimodal feature fusion encoder and proposal generator.

[0011] Furthermore, the step (1) includes the following sub-steps:

[0012] (1.1) Input the videos in the video dataset into the pre-trained visual feature extraction network 13D and the pre-trained audio feature extraction network VGGish respectively to extract the initial visual features V0 and initial audio features A0 of the videos;

[0013] (1.2) The initial visual feature V0 and the initial audio feature A0 are respectively passed through the linear layer of the multimodal feature fusion encoder to obtain the first visual feature V1 and the first audio feature A1 with the same feature dimension; the initial visual feature V0 and the initial audio feature A0 are feature concatenated to obtain the initial modal information sharing feature f av0 ;

[0014] (1.3) The first visual feature V1, the first audio feature A1 and the initial modal information shared feature f av0 The self-attention mechanism is used to capture the contextual semantic information of each modality and perform preliminary modality information alignment to obtain the visual feature V calculated by the self-attention mechanism. self , audio feature A self Shared features with modal information

[0015] (1.4) The visual feature V calculated by the self-attention mechanism obtained in step (1.3) self and audio feature A self Perform cross-attention mechanism calculation to obtain the second visual feature V′ and the second audio feature A′;

[0016] (1.5) The second visual feature V′, the second audio feature A′ and the modal information shared feature calculated by the self-attention mechanism Perform cross-attention calculations on each pair to obtain the third visual feature h V , the third audio feature h A and the third modality information shares features h AV ;

[0017] (1.6) The third visual feature h V , the third audio feature h A and the third modality information shares features h AV They are sent to the fully connected layer to obtain the encoded visual features f that fully interact with the modal information. V , audio features f A and modal information sharing feature f AV .

[0018] Furthermore, the step (2) includes the following sub-steps:

[0019] (2.1) The encoded audio feature f A , visual features V The data is sent to the proposal generator and first passes through the first layer of parallel proposal headers, and then generates initial proposal features corresponding to audio and vision through multiple parallel proposal headers respectively;

[0020] (2.2) The initial proposal features corresponding to the audio are placed in the audio feature proposal pool of the proposal feature interaction layer, and the initial proposal features corresponding to the vision are placed in the visual feature proposal pool of the proposal feature interaction layer. The initial proposal features generated by three adjacent proposal heads are selected from the audio feature proposal pool and the visual feature proposal pool for feature splicing to obtain the audio features and visual features after feature splicing.

[0021] (2.3) The concatenated audio features and visual features are sent to the second-layer parallel proposal head, and information is exchanged through multiple parallel proposal heads to generate the final proposal features;

[0022] (2.4) The final proposal features are sent to the proposal object pool, and the event segment center point, event segment length and event segment confidence of each final proposal feature are calculated according to the following formula, so as to select the top K final proposals with the highest confidence from the proposal object pool as the final generated prediction results:

[0023] center=O+σ(P C )

[0024] length = L exp(Pl)

[0025] confidence=σ(P o )

[0026] Among them, center represents the center point of the event segment, length represents the length of the event segment, confidence represents the confidence of the event segment, O and L are the initial anchor point center and initial anchor point length calculated by the data set clustering algorithm, σ() is the sigmoid function, exp() is the coefficient, P l represents the central feature of the event segment, P C represents the event segment length feature, P O Represents the confidence feature of the event segment.

[0027] Furthermore, each of the proposal heads is a full convolutional network with convolution kernels of different sizes, and the generated proposal features are made to have the same dimension by adjusting the padding parameter and the stride parameter;

[0028] The output value of each proposal header includes an event segment center feature, an event segment length feature, and an event segment confidence feature.

[0029] Furthermore, the step (3) includes the following sub-steps:

[0030] (3.1) The encoded audio feature f A , modal information sharing feature f AVand the current word embedding C are fed into the audio feature addition fusion block of the description generator, and the encoded visual feature f V , modal information sharing feature f Av The current word embedding C is fed into the visual feature addition fusion block of the description generator to obtain the final interaction result C of the text modality feature and the audio feature a The final interaction result C of text modality features and visual features v ;

[0031] (3.2) Final interaction result C of text modality features and audio features a The final interaction result C of text modality features and visual features v It is sent to the feedforward layer, passes through the fully connected layer and the softmax function in turn, and obtains the final description prediction result.

[0032] Furthermore, in the audio feature addition fusion block, the current word embedding C is calculated through the self-attention mechanism to obtain the word embedding feature C′; the encoded audio feature f A and modal information sharing feature f AV The cross attention mechanism is calculated with the word embedding feature C′ to obtain the initial interaction results of the text modality feature and the audio feature. Initial interaction results of text modality features and modality information sharing features Initial interaction results of word embedding feature C′, text modality feature and audio feature Initial interaction results of text modality features and modality information sharing features Perform feature concatenation to obtain the final interaction result C of text modality features and audio features a ;

[0033] In the visual feature addition fusion block, the current word embedding C is calculated through the self-attention mechanism to obtain the word embedding feature C′; the encoded visual feature f V and modal information sharing feature f AV The cross attention mechanism is calculated with the word embedding feature C′ to obtain the initial interaction results of the text modality feature and the visual feature. Initial interaction results of text modality features and modality information sharing features Initial interaction results of word embedding feature C′, text modality feature and visual feature Initial interaction results of text modality features and modality information sharing features Perform feature concatenation to obtain the final interaction result C of text modality features and visual features v .

[0034] Furthermore, the step (4) includes the following sub-steps:

[0035] (4.1) The KL divergence loss function is calculated based on the final generation prediction result output by the proposal generator and the final description prediction result output by the description generator. The calculation formula is:

[0036]

[0037] Where x is the description word, X is the sum of the description words in the current batch, P(x) is the predicted probability distribution of the description word, Q(x) is the true probability distribution of the description word, and D KL represents the KL divergence loss function;

[0038] (4.2) Calculate the encoded audio feature f A and modal information sharing feature f AV The contrast loss between the two and the encoded visual features f V and modal information sharing feature f Av The contrast loss between them is calculated as:

[0039]

[0040] Among them, L enc represents the contrast loss, T A and T V Respectively represent the length of audio features and visual features, i and j represent the indexes of audio features and visual features in the modal information sharing features, τ represents the temperature coefficient;

[0041] (4.3) Taking minimizing the contrast loss and KL divergence loss function as the optimization goal, the back-propagation method is used to update the network parameters of the multimodal feature fusion encoder and proposal generator.

[0042] The beneficial effect of the present invention is that the dense video event description method based on multimodal representation described in the present invention introduces modal information sharing features in the dense video event description task, maps the features of different modalities to the same feature space, performs feature alignment, and alleviates the modal gap that persists between heterogeneous modalities; at the same time, the contrast loss in contrastive learning is introduced in the individual modal features and the modal information sharing features, which enriches the contextual semantic information in the features to the greatest extent, and can effectively avoid the loss of semantic information when different modal features are mapped to the same feature space; and in this process, through the calculation of the multi-layer attention mechanism, the important information in the modal features will also be mined; the dense video event description method described in the present invention achieves performance improvement on three dense video dataset benchmarks, proving the effectiveness of the dense video event description method described in the present invention for generating dense video event descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the architecture of the multimodal feature fusion encoder in the present invention;

[0044] Figure 2 is a schematic diagram of the architecture of the proposal generator in the present invention;

[0045] Figure 3 It is a schematic diagram of the architecture of the description generator in the present invention. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. It should be understood that the above general description and the detailed description that follows are merely exemplary and explanatory and cannot limit the present application.

[0047] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0048] It should be understood that, although the term first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of..." or "when..." or "in response to determination". Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process and method including a series of elements include not only those elements, but also include other elements not clearly listed, or also include elements inherent to such process and method. In the absence of more restrictions, the elements limited by the statement "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0049] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0050] The dense video event description method based on multimodal representation of the present invention can make full use of the multimodal information in the video, locate all possible events in an untrimmed video, and generate an event description for each located event. Figure 1 As shown, the method specifically comprises the following steps:

[0051] (1) The initial visual features of the video dataset are extracted through the pre-trained visual feature extraction network 13D, the initial audio features of the video dataset are extracted through the pre-trained audio feature extraction network VGGish, and the encoded visual features, audio features and modal information sharing features are obtained using a multimodal feature fusion encoder, such as Figure 1 As shown in the figure, the multimodal feature fusion encoder includes a linear layer, a self-attention mechanism, a cross-attention mechanism, and a fully connected layer.

[0052] (1.1) The videos in the video dataset are respectively input into the pre-trained visual feature extraction network 13D and the pre-trained audio feature extraction network VGGish to extract the initial visual features V0 and initial audio features A0 of the videos.

[0053] Specifically, the videos in the video dataset are respectively input into the visual feature extraction network 13D pre-trained on the Kinetics 400 data and the audio feature extraction network VGGish pre-trained on the AudioSet dataset to extract the initial visual features V0 and initial audio features A0 of the videos, and finally obtain 1024-d initial visual features V0 and initial audio features A0.

[0054] It should be understood that other existing known feature extraction networks can also be used to extract target features of samples, for example, C3D is used as the network structure of the feature extraction network, the number of feature extraction step frames is 64, and the window size is 64. Of course, other known feature extraction networks can also be used, and the batch size of each small batch of sample data can be set according to actual needs.

[0055] (1.2) The initial visual feature V0 and the initial audio feature A0 are respectively passed through the linear layer of the multimodal feature fusion encoder to obtain the first visual feature Y1 and the first audio feature A1 with the same feature dimension; the initial visual feature V0 and the initial audio feature A0 are feature concatenated to obtain the initial modal information sharing feature f av0 .

[0056] (1.3) The first visual feature V1, the first audio feature A1 and the initial modal information shared feature fav0 The self-attention mechanism is used to capture the contextual semantic information of each modality and perform preliminary modality information alignment to obtain the visual feature V calculated by the self-attention mechanism. self , audio feature A self Shared features with modal information The calculation formulas are:

[0057] V self =SelfAttention(Q v ,K v ,V v )

[0058] A self =SelfAttention(Q a ,K a ,V a )

[0059]

[0060] Among them, SelfAttention() is the self-attention mechanism, Q v , K v and V v are the query, key and value of the first video feature sequence, Q a , K a and V a are the query, key and value of the first audio feature sequence, Q av , K av and V av are the query, key, and value of the feature sequence of the initial modal information sharing feature, respectively.

[0061] (1.4) The visual feature V calculated by the self-attention mechanism obtained in step (1.3) self and audio feature A self The cross attention mechanism is calculated to obtain the second visual feature V′ and the second audio feature A′, and the calculation formulas are:

[0062] V'=CrossAttention(V self ,A self ,A self )

[0063] A'=CrossAttention(A self ,V self ,V self )

[0064] Among them, CrossAttention() is the cross attention mechanism.

[0065] (1.5) The second visual feature V′, the second audio feature A′ and the modal information shared feature calculated by the self-attention mechanism Perform cross-attention calculations on each pair to obtain the third visual feature h V , the third audio feature h A and the third modality information shares features h AV , and the calculation formulas are:

[0066]

[0067] Among them, LN() is layer normalization, and m has no practical meaning and only refers to the second visual feature V′ and the second audio feature A′.

[0068] (1.6) The third visual feature h V , the third audio feature h A and the third modality information shares features h AV They are sent to the fully connected layer to obtain the encoded visual features f that fully interact with the modal information. V , audio features f A and modal information sharing feature f AV , and the calculation formulas are:

[0069] f V =FullyConnected(hV)

[0070] f A =FullyConnected(h A )

[0071] f AV =FullyConnected(h AV )

[0072] Among them, FullyConnected() is the fully connected layer.

[0073] It should be understood that by constructing the modal information sharing feature f av , and performs a series of attention mechanism calculations to alleviate the persistent modality gap between heterogeneous modalities. Through the self-attention mechanism, the feature information of a single modality can filter its own important information, and in the cross-attention mechanism calculation process, the important information between different modalities can be better aligned, laying a good foundation for subsequent event location and event description tasks.

[0074] (2) The encoded audio features and visual features are sent to the proposal generator, which passes through the first-layer parallel proposal head, proposal feature interaction layer, second-layer parallel proposal head and proposal object pool of the proposal generator in sequence. Finally, the K proposals with the highest confidence are selected from the proposal object pool as the final generated prediction results, such as Figure 2 As shown, the first layer of parallel proposal headers includes multiple parallel proposal headers, the proposal feature interaction layer includes an audio feature proposal pool and a visual feature proposal pool, and the second layer of parallel proposal headers includes multiple parallel proposal headers.

[0075] Furthermore, each proposal head is a fully convolutional network with convolution kernels of different sizes. The padding and stride parameters are adjusted to make the generated proposal features have the same dimension. The output value of each proposal head includes the event segment center feature P l , event segment length feature P C and event segment confidence feature P O .

[0076] It should be understood that a series of parallel proposal heads with convolution kernels of different sizes can be better used to locate events of varying lengths in the video.

[0077] (2.1) The encoded audio feature f A , visual features V It is sent to the proposal generator and first passes through the first layer of parallel proposal heads, and then generates the initial proposal features corresponding to the audio and the initial proposal features corresponding to the vision through multiple parallel proposal heads.

[0078] (2.2) The initial proposal features corresponding to the audio are placed in the audio feature proposal pool of the proposal feature interaction layer, and the initial proposal features corresponding to the vision are placed in the visual feature proposal pool of the proposal feature interaction layer. The initial proposal features generated by three adjacent proposal heads are selected from the audio feature proposal pool and the visual feature proposal pool respectively for feature splicing to obtain the audio features and visual features after feature splicing.

[0079] It should be understood that the selection of adjacent proposal headers for information exchange is to prevent the situation where positioning events are completely covered. Too few or too many proposal header interactions will result in insufficient proposal information interaction and excessive duplication of positioning events, respectively.

[0080] (2.3) The concatenated audio features and visual features are sent to the second-layer parallel proposal head, and information is interacted through multiple parallel proposal heads to generate the final proposal features.

[0081] (2.4) The final proposal features are sent to the proposal object pool, and the event segment center point, event segment length and event segment confidence of each final proposal feature are calculated according to the following formula, so as to select the top K final proposals with the highest confidence from the proposal object pool as the final generated prediction results. In this embodiment, the top 100 proposals with the highest confidence are selected.

[0082] center=O+σ(P C )

[0083] length = L exp(P l )

[0084] confidence=σ(PO)

[0085] Among them, center represents the center point of the event segment, length represents the length of the event segment, confidence represents the confidence of the event segment, O and L are the initial anchor point center and initial anchor point length calculated by the data set clustering algorithm, σ() is the sigmoid function, and exp() is the coefficient.

[0086] (3) The encoded audio features, visual features, and modal information sharing features are fed into the description generator to obtain the final description prediction result, such as Figure 3 As shown in the figure, the description generator includes an audio feature addition fusion block, a visual feature addition fusion block and a feedforward layer. The audio feature addition fusion block includes a self-attention mechanism and a cross-attention mechanism. The visual feature addition fusion block includes a self-attention mechanism and a cross-attention mechanism. The feedforward layer includes a fully connected layer and a softmax function.

[0087] (3.1) The encoded audio feature f A , modal information sharing feature f AV and the current word embedding C are fed into the audio feature addition fusion block of the description generator, and the encoded visual feature f V , modal information sharing feature f AV The current word embedding C is fed into the visual feature addition fusion block of the description generator to obtain the final interaction result C of the text modality feature and the audio feature a The final interaction result C of text modality features and visual features v .

[0088] Furthermore, in the audio feature addition fusion block, the current word embedding C is calculated through the self-attention mechanism to obtain the word embedding feature C′; the encoded audio feature f A and modal information sharing feature f AV The cross attention mechanism is calculated with the word embedding feature C′ to obtain the initial interaction result C of the text modality feature and the audio feature.fA The initial interaction result C of the text modality feature and the modality information sharing feature fAV ; The initial interaction result C of word embedding feature C′, text modality feature and audio feature fA The initial interaction result C of the text modality feature and the modality information sharing feature fAV Perform feature concatenation to obtain the final interaction result C of text modality features and audio features a , and the calculation formulas are:

[0089] C fA = CrossAttention(C',f A ,f A )

[0090] C fAV = CrossAttention(C',f AV ,f AV )

[0091] C a =C fA +C fAV +C'

[0092] Furthermore, in the visual feature addition fusion block, the current word embedding C is calculated through the self-attention mechanism to obtain the word embedding feature C′; the encoded visual feature f V and modal information sharing feature f AV The cross attention mechanism is calculated with the word embedding feature C′ to obtain the initial interaction result C of the text modality feature and the visual feature. fV The initial interaction result C of the text modality feature and the modality information sharing feature fAV ; The initial interaction result C of word embedding feature C′, text modality feature and visual feature fV The initial interaction result C of the text modality feature and the modality information sharing feature fAV Perform feature splicing to obtain the final interaction result C of text modality features and visual features v , and the calculation formulas are:

[0093] C fV = CrossAttention(C',f V ,f V )

[0094] C fAV =CrossAttention(C',fAV,f AV )

[0095] C v =C fV +CfAV +C

[0096] It should be understood that the audio feature addition fusion block and the visual feature addition fusion block can better fuse features of different forms, so that individual modal features, modal information sharing features and word embeddings can jointly guide the generation of descriptions.

[0097] (3.2) Final interaction result C of text modality features and audio features a The final interaction result C of text modality features and visual features v It is sent to the feedforward layer, passes through the fully connected layer and the softmax function in turn, and the final description prediction result is obtained, which is expressed as:

[0098] C * =softmax(FullyConnected(C a ,C v ))

[0099] Among them, C * It represents the final description of the prediction result, and softmax() is the softmax function.

[0100] (4) The contrast loss is calculated based on the encoded visual features, audio features and modal information sharing features, and the KL divergence loss function is calculated based on the final generated prediction results and the final description prediction results. The network parameters of the multimodal feature fusion encoder and proposal generator are updated using the back-propagation method with the minimization of the contrast loss and the KL divergence loss function as the optimization goal.

[0101] (4.1) The KL divergence loss function is calculated based on the final generation prediction result output by the proposal generator and the final description prediction result output by the description generator. The calculation formula is:

[0102]

[0103] Where x is the description word, X is the sum of the description words in the current batch, P(x) is the predicted probability distribution of the description word, Q(x) is the true probability distribution of the description word, and D KL represents the KL divergence loss function.

[0104] (4.2) Calculate the encoded audio feature f A and modal information sharing feature f AV The contrast loss between the two and the encoded visual features f V and modal information sharing feature f AV The contrast loss between them is calculated as:

[0105]

[0106] Among them, L enc represents the contrast loss, T A and T V They represent the length of audio features and the length of visual features respectively, i and j represent the indexes of audio features and visual features in the modal information sharing features respectively, and τ represents the temperature coefficient.

[0107] It should be understood that the two different representations of the individual modality features and the modality information sharing features in a certain index frame express similar semantics, and they are defined as positive samples to enhance consistency. Conversely, the mismatched frames are defined as negative samples.

[0108] (4.3) Taking minimizing the contrast loss and KL divergence loss function as the optimization goal, the back-propagation method is used to update the network parameters of the multimodal feature fusion encoder and proposal generator.

[0109] Exemplarily, when working: three different datasets are used, including the ActivityNet Captions dataset, a subset of the ActivityNet Captions dataset (retaining videos with visual features, audio features, and text features in the ActivityNet Captions dataset), and the YouCook2 dataset, which are used for training, verification, and testing.

[0110] The ActivityNet dataset consists of more than 20K Youtube videos with different categories and annotations, of which 10K videos are used as training sets, 5K videos as validation sets, and 5K videos as test sets. Each video in the dataset has an average duration of 2 minutes and contains 3.65 subtitles with timestamps (start time, end time). Each subtitle consists of about 13.65 words. Since the test set does not provide manual descriptions, a subset of the model is trained on the training set, and the results of the test evaluation study are shown on the validation set.

[0111] The ActivityNet Captions dataset subset retains videos that have visual features, audio features, and text features in the ActivityNet Captions dataset. The final ActivityNet Captions dataset subset consists of 4K videos, of which 2.6K videos are used as training sets and 1.4K videos are used as test sets.

[0112] YouCook2 dataset contains 2K instructional videos of 89 recipes in daily life. The average duration of each video is 5.26 minutes, and it contains about 7.7 event clips with subtitles. Referring to the official division method, the dataset is divided into 1.3K video training set, 0.4K video validation set, and 0.3K video test set. Similar to ActivityNetCaptions, since the test set does not provide manual descriptions, a subset of the model is trained on the training set, and the results of the test evaluation study are obtained on the validation set.

[0113] The experimental evaluation indicators are: standard METEOR, BLEU{3,4} machine translation evaluation indicators are used to evaluate subtitle performance, and other indicators such as Recall, Precision, F1 are also used to evaluate the model event localization performance.

[0114] The experimental parameter settings are as follows: the training is based on pytorch1.8 and cuda12.2, and the experiment uses 4 Nvidia GeForceRTX 4090. During the training process, SGD is used as the optimizer, the learning rate is set to 0.00005, and the batch size is 16. In the proposal generation stage, in order to make the proposal head work properly, the audio feature sequence and the visual feature sequence are padded to 800 and 300 respectively. In the subtitle generation stage, choose to fill the feature sequence with the length of the longest feature sequence in the current batch. For each manual description, only the first 30 words are retained (if the number of words exceeds, we truncate it). For each word that appears in the manual description, it is recorded in the dictionary. We set the number of encoder and decoder layers to 2. Select 3 adjacent proposal features in the proposal feature interaction architecture to interact.

[0115] The proposed model is compared with the most advanced methods on three data sets, and the performance comparison results are shown in Tables 1, 2, 3 and 4. The With GT proposal column and the With learned proposal column represent the real event proposals in the data set and the event proposals predicted in the model localization stage, respectively. The upper part shows the results of the most advanced dense event description method, and the last row shows the results of the dense video event description method based on multimodal representation described in the present invention.

[0116] Table 1: Event localization capability - results of ActivityNet Captions dataset

[0117] method Pre Rec F1 PDVC 55.40 58.10 56.72 UEDVC 59.00 60.30 59.64 MPP-net 55.58 58.37 56.86 CM 49.10 88.32 63.11 SBS 57.89 59.25 58.56 Vid2seq 52.70 53.90 52.40 The present invention 53.29 77.40 63.15

[0118] Table 2: Description generation capability - results on ActivityNet Captions dataset

[0119]

[0120] Table 3: Description generation capability - results on a subset of the ActivityNet Captions dataset

[0121]

[0122]

[0123] Table 4: Description generation and event localization capabilities - results on YouCook2 dataset

[0124]

[0125] Specifically, the model proposed in the present invention outperforms other methods in most indicators on the datasets ActivityNet Caption, ActivityNetCaption subset and YouCook2, achieves the best results in event localization capability evaluation, and finally obtains excellent results. As shown in Table 1, compared with the recall accuracy imbalance problem of other methods, the model proposed in the present invention finally obtains better results in F1 score in balancing the results of the two. As shown in Table 2, the model proposed in the present invention has obvious advantages under the condition of With learned proposal, which further proves the superiority of the model event localization method proposed in the present invention. In addition, the model proposed in the present invention obtains results similar to SOTA under the condition of With GT proposal. In addition, as shown in Tables 3 and 4, the model proposed in the present invention can outperform the recent methods in other datasets (ActivityNet Captions dataset subset and YouCook2 dataset), which shows the versatility of the method proposed in the present invention.

[0126] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dense video event description method based on multimodal representation, characterized in that: The following steps are involved: (1) extracting initial visual features of a video dataset through a pre-trained visual feature extraction network I3D, extracting initial audio features of a video dataset through a pre-trained audio feature extraction network VGGish, and using a multimodal feature fusion encoder to obtain encoded visual features, audio features, and modal information sharing features; wherein the multimodal feature fusion encoder includes a linear layer, a self-attention mechanism, a cross-attention mechanism, and a fully connected layer; (2) The encoded audio features and visual features are sent to the proposal generator, and sequentially pass through the first-layer parallel proposal head, the proposal feature interaction layer, the second-layer parallel proposal head and the proposal object pool of the proposal generator, and finally select the K proposals with the highest confidence from the proposal object pool as the final generated prediction results; wherein the first-layer parallel proposal head includes multiple parallel proposal heads, the proposal feature interaction layer includes an audio feature proposal pool and a visual feature proposal pool, and the second-layer parallel proposal head includes multiple parallel proposal heads; (3) sending the encoded audio features, visual features and modal information sharing features to a description generator to obtain a final description prediction result; wherein the description generator includes an audio feature addition fusion block, a visual feature addition fusion block and a feedforward layer, the audio feature addition fusion block includes a self-attention mechanism and a cross-attention mechanism, the visual feature addition fusion block includes a self-attention mechanism and a cross-attention mechanism, and the feedforward layer includes a fully connected layer and a softmax function; (4) The contrast loss is calculated based on the encoded visual features, audio features and modal information sharing features, and the KL divergence loss function is calculated based on the final generated prediction results and the final description prediction results. The network parameters of the multimodal feature fusion encoder and proposal generator are updated using the back-propagation method with the minimization of the contrast loss and the KL divergence loss function as the optimization goal.

2. The dense video event description method based on multimodal representation according to claim 1 is characterized in that: The step (1) comprises the following sub-steps: (1.1) Input the videos in the video dataset into the pre-trained visual feature extraction network I3D and the pre-trained audio feature extraction network VGGish respectively to extract the initial visual features V0 and initial audio features A0 of the videos; (1.2) The initial visual feature V0 and the initial audio feature A0 are respectively passed through the linear layer of the multimodal feature fusion encoder to obtain the first visual feature V1 and the first audio feature A1 with the same feature dimension; the initial visual feature V0 and the initial audio feature A0 are feature concatenated to obtain the initial modal information sharing feature f av0 ; (1.3) The first visual feature V1, the first audio feature A1 and the initial modal information shared feature f av0 The self-attention mechanism is used to capture the contextual semantic information of each modality and perform preliminary modality information alignment to obtain the visual feature V calculated by the self-attention mechanism. self , audio feature A self Shared features with modal information (1.4) The visual feature V calculated by the self-attention mechanism obtained in step (1.3) self and audio feature A self Perform cross-attention mechanism calculation to obtain the second visual feature V′ and the second audio feature A′; (1.5) The second visual feature V′, the second audio feature A′ and the modal information shared feature calculated by the self-attention mechanism Perform cross-attention calculations on each pair to obtain the third visual feature h V , the third audio feature h A and the third modality information shares features h AV ; (1.6) The third visual feature h V , the third audio feature h A and the third modality information shares features h AV They are sent to the fully connected layer to obtain the encoded visual features f that fully interact with the modal information. V , audio features f A and modal information sharing feature f AV .

3. The dense video event description method based on multimodal representation according to claim 1 is characterized in that: The step (2) comprises the following sub-steps: (2.1) The encoded audio feature f A , visual features V The data is sent to the proposal generator and first passes through the first layer of parallel proposal headers, and then generates initial proposal features corresponding to audio and vision through multiple parallel proposal headers respectively; (2.2) The initial proposal features corresponding to the audio are placed in the audio feature proposal pool of the proposal feature interaction layer, and the initial proposal features corresponding to the vision are placed in the visual feature proposal pool of the proposal feature interaction layer. The initial proposal features generated by three adjacent proposal heads are selected from the audio feature proposal pool and the visual feature proposal pool for feature splicing to obtain the audio features and visual features after feature splicing. (2.3) The concatenated audio features and visual features are sent to the second-layer parallel proposal head, and information is exchanged through multiple parallel proposal heads to generate the final proposal features; (2.4) The final proposal features are sent to the proposal object pool, and the event segment center point, event segment length and event segment confidence of each final proposal feature are calculated according to the following formula, so as to select the top K final proposals with the highest confidence from the proposal object pool as the final generated prediction results: center=O+σ(P C ) length=L·exp(P l ) confidence=σ(P O ) Among them, center represents the center point of the event segment, length represents the length of the event segment, confidence represents the confidence of the event segment, O and L are the initial anchor point center and initial anchor point length calculated by the data set clustering algorithm, σ() is the sigmoid function, exp() is the coefficient, P l represents the central feature of the event segment, P C represents the event segment length feature, P O Represents the confidence feature of the event segment.

4. The dense video event description method based on multimodal representation according to claim 1 or 3, characterized in that: Each of the proposal heads is a fully convolutional network with convolution kernels of different sizes, and the generated proposal features are made to have the same dimension by adjusting the padding parameters and stride parameters; The output value of each proposal header includes an event segment center feature, an event segment length feature, and an event segment confidence feature.

5. The dense video event description method based on multimodal representation according to claim 1 is characterized in that: The step (3) comprises the following sub-steps: (3.1) The encoded audio feature f A , modal information sharing feature f AV and the current word embedding C are fed into the audio feature addition fusion block of the description generator, and the encoded visual feature f V , modal information sharing feature f AV The current word embedding C is fed into the visual feature addition fusion block of the description generator to obtain the final interaction result C of the text modality feature and the audio feature a The final interaction result C of text modality features and visual features v ; (3.2) Final interaction result C of text modality features and audio features a The final interaction result C of text modality features and visual features v It is sent to the feedforward layer, passes through the fully connected layer and the softmax function in turn, and obtains the final description prediction result.

6. The dense video event description method based on multimodal representation according to claim 1 or 5, characterized in that: In the audio feature addition fusion block, the current word embedding C is calculated through the self-attention mechanism to obtain the word embedding feature C′; the encoded audio feature f A and modal information sharing feature f AV The cross attention mechanism is calculated with the word embedding feature C′ to obtain the initial interaction results of the text modality feature and the audio feature. Initial interaction results of text modality features and modality information sharing features Initial interaction results of word embedding feature C′, text modality feature and audio feature Initial interaction results of text modality features and modality information sharing features Perform feature concatenation to obtain the final interaction result C of text modality features and audio features a ; In the visual feature addition fusion block, the current word embedding C is calculated through the self-attention mechanism to obtain the word embedding feature C′; The encoded visual feature f V and modal information sharing feature f AV The cross attention mechanism is calculated with the word embedding feature C′ to obtain the initial interaction results of the text modality feature and the visual feature. Initial interaction results of text modality features and modality information sharing features Initial interaction results of word embedding feature C′, text modality feature and visual feature Initial interaction results of text modality features and modality information sharing features Perform feature concatenation to obtain the final interaction result C of text modality features and visual features v .

7. The dense video event description method based on multimodal representation according to claim 1 is characterized in that: The step (4) comprises the following sub-steps: (4.1) The KL divergence loss function is calculated based on the final generation prediction result output by the proposal generator and the final description prediction result output by the description generator. The calculation formula is: Among them, x is the description word, X is the sum of the description words in the current batch, P(x) is the predicted probability distribution of the description word, Q(x) is the true probability distribution of the description word, and D KL represents the KL divergence loss function; (4.2) Calculate the encoded audio feature f A and modal information sharing feature f AV The contrast loss between the two and the encoded visual features f V and modal information sharing feature f AV The contrast loss between them is calculated as: Among them, L enc represents the contrast loss, T A and T V Respectively represent the length of audio features and visual features, i and j represent the indexes of audio features and visual features in the modal information sharing features, τ represents the temperature coefficient; (4.3) Taking minimizing the contrast loss and KL divergence loss function as the optimization goal, the back-propagation method is used to update the network parameters of the multimodal feature fusion encoder and proposal generator.

Images