Audio and video event positioning method based on fine tuning pre-training large model

Abstract

The invention discloses an audio and video event positioning method based on a fine-tuning pre-training large model, and the method comprises the steps: 1, obtaining video data, and separating video image frame data from audio data; 2, respectively carrying out feature extraction on the frame data and the audio data through a coding layer of a pre-trained large model; 3, features in a single mode are aggregated by querying cue words, and event features are extracted; 4, providing context information of a downstream task through a global cue word, and fusing the audio features and the video frame features; and 5, constructing an event positioning module, and converting the obtained features into video event classification and positioning results. According to the method, the learnable cue word is introduced, and the capability of pre-training the large model is migrated to the audio and video event positioning task, so that the audio and video event positioning task is completed under the condition of extremely low trainable parameter quantity.

Description

Technical Field

[0001] The present invention relates to the field of audio and video event positioning, and in particular to an audio and video event positioning method based on a fine-tuned pre-trained large model. Background Art

[0002] Hearing and vision are important ways for humans to perceive the real world. Video content contains important event information in the auditory and visual modalities. The audio and video event localization task needs to process information from both the audio and visual modalities at the same time, with the goal of classifying and locating events in the video. In recent years, the audio and video event localization task has received increasing attention as a basic task in the direction of video understanding.

[0003] The existing mainstream audio and video event localization methods adopt basically the same strategy: first use two models in the image dataset and audio dataset as the video frame image feature encoder and audio feature encoder, and then process the encoded video frame features and audio features to obtain the audio and video event localization results. This method encodes the video frame and audio independently into features. Although this processing strategy has achieved good results, it also has inevitable limitations. First, based on the encoder pre-trained on a single modality dataset, the method of fusing the visual and audio features extracted from the original input can be regarded as a late fusion. These methods lack the ability to make early interactions between the two modal information in the feature extraction stage. Secondly, the encoder pre-trained on a single modality dataset does not participate in the model training process, resulting in the extracted features may contain information irrelevant to audio and video event localization, resulting in suboptimal performance.

[0004] Transformer-based large-scale vision pre-trained models, such as Vision Transformer, can learn effective image feature representations by treating the image as a sequence of small image blocks, converting each small image block into a one-dimensional feature, and then processing it using a standard transformer encoder. In recent years, the method of training transformer models on audio and video datasets and applying them to downstream tasks in the field of audio and video processing has achieved good performance. However, due to the large number of parameters of the transformer model, training a transformer model from scratch requires a lot of computing resources. In order to utilize the capabilities of the pre-trained transformer model while avoiding the large computational overhead required to train the full model, predecessors proposed the method of prompt fine-tuning. The prompt fine-tuning method freezes the parameters of the pre-trained model during the model training process, that is, the parameters of the pre-trained model are not updated, and only the continuous prompt vector is trained. Specifically, prompt fine-tuning introduces a learnable continuous vector at the input of the transformer model, called a prompt vector, and only the parameters of the prompt vector are updated during the training process. Existing methods mainly focus on enhancing the few-shot learning ability of the original model, which is difficult to apply to audio and video event localization tasks. Summary of the invention

[0005] In order to overcome the shortcomings of the existing methods, the present invention provides an audio and video event localization method based on fine-tuning a pre-trained large model, so that the original capabilities of the pre-trained large model can be transferred to the audio and video event localization task through prompt fine-tuning, thereby improving the accuracy of audio and video event localization.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The audio and video event localization method based on fine-tuning the pre-trained large model of the present invention is characterized in that it is carried out according to the following steps:

[0008] Step 1: Get a real video data V and divide it into T non-overlapping audio and video clips, denoted as {V' t ,A' t |t=1,2,…,T}, where V' t represents the tth video clip, A' t V' t The corresponding audio clip;

[0009] Sample the tth video segment V' t A frame of image in the t-th frame image V t ;

[0010] A' tTransformed into the t-th two-dimensional audio spectrum A t ;

[0011] Get the event category label of V and positioning tags , among which, among which, A label indicating whether V belongs to the kth audio or video event category, and ; C represents the number of event categories of audio and video clips; V' t and A' t Whether it belongs to the location tag of the event segment;

[0012] Step 2: Build a pre-trained large model, including: a feature encoding layer and N transformer layers;

[0013] Step 2.1: The feature encoding layer converts V t Divide it into n non-overlapping image blocks, flatten each image block and then splice it to get V t Visual modality feature representation ;

[0014] The feature encoding layer is implemented by the convolutional layer on A t Process and get A t Audio modal feature representation ;

[0015] Step 2.2, N transformer layers for audio and video modality feature set { } to obtain the visual modality feature representation sequence of V and audio modality feature representation sequence ;

[0016] Step 3: Construct an event localization module, including a video event category classifier and an event correlation predictor, and and Fusion features after splicing on the feature dimension Convert and get the event category prediction label of V accordingly And the location prediction label of the event segment :

[0017] Step 4: Based on and , and Construct the total loss function L for back propagation:

[0018] Step 5: Based on the real video event labels, the query prompt words, global prompt words and event location module to be learned are trained, and the total loss function L is calculated. At the same time, the adaptive moment estimation optimization method is used with the learning rate To update the parameters until L converges, the optimal audio and video event localization model is obtained, which is used to process the input audio and video to obtain the event localization result in the video.

[0019] The audio and video event localization method based on fine-tuning the pre-trained large model described in the present invention is also characterized in that step 2.2 is performed as follows:

[0020] Step 2.2.1, initialize i=1;

[0021] Step 2.2.2: Randomly initialize the query prompt words to be learned for the visual modality in the i-th transformer layer , and thus use formula (1) to obtain the t-th intermediate visual modality feature representation output by the i-th transformer layer and V t Event clues related to audio and video events ;

[0022] (1)

[0023] In formula (1), represents the i-th transformer layer, V represents the output of the i-1th transformer layer t The visual modality feature representation of is, when i=1, let = , Represents the concatenation operation on the dimension of feature quantity;

[0024] Step 2.2.3: Randomly initialize the query prompt words to be learned for the audio modality in the i-th transformer layer , and thus use formula (2) to obtain the t-th intermediate audio modal feature representation output by the i-th transformer layer and A t Event clues related to audio and video events ;

[0025] (2)

[0026] In formula (2), A represents the output of the i-1th transformer layer t The audio modal feature representation of is, when i=1, let = ;

[0027] Step 2.2.4: Randomly initialize the global cue words to be learned for the visual modality in the i-th transformer layer , and then use formula (3) to get the output V of the i-th transformer layer t Visual modality feature representation , Intermediate query prompt words in audio mode , the intermediate global prompt word of the visual modality ;

[0028] (3)

[0029] Step 2.2.5: Randomly initialize the global cue words to be learned for the audio modality in the i-th transformer layer , and then use formula (4) to get the output A of the i-th transformer layer t Audio modal feature representation , intermediate query hint words in visual mode , the middle global prompt word of the audio mode ;

[0030] (4)

[0031] Step 2.2.6: After assigning i+1 to i, return to step 2.2.2 and execute sequentially until i>N, thus obtaining V t The Nth visual modality feature representation and A t The Nth audio modality feature representation , and then obtain the visual modality feature representation sequence of V and audio modality feature representation sequence , where N represents the total number of transformer layers.

[0032] Further, the step 3 is performed as follows:

[0033] Step 3.1: The video event category classifier uses formula (5) to obtain the overall feature representation of V , and then use formula (6) to get the event category prediction label of V :

[0034] (5)

[0035] (6)

[0036] In formula (5), Maxpool represents the maximum pooling layer in the video event category classifier; Representing parameters of a fully connected layer in the video event category classifier;

[0037] Step 3.2: The event correlation predictor uses formula (7) to obtain the location prediction label of the event segment of V :

[0038] (7)

[0039] In formula (7), Represents the parameters of the fully connected layer in the event correlation predictor.

[0040] Further, step 4 is performed as follows:

[0041] Step 4.1: Use formula (8) to obtain the cross entropy loss function value of the video event category classifier :

[0042] (8)

[0043] In formula (8), express The probability of belonging to the kth audio and video event category;

[0044] Step 4.2: Use formula (9) to obtain the binary cross entropy loss function value of the event correlation predictor :

[0045] (9)

[0046] In formula (9), express The tth audio and video clip V' t and A' t The predicted probability of belonging to the event segment;

[0047] Step 4.3: Use formula (10) to get the total loss function L:

[0048] (10)

[0049] In formula (10), is a balance parameter.

[0050] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the audio and video event locating method, and the processor is configured to execute the program stored in the memory.

[0051] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the audio and video event locating method when the computer program is run by a processor.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention uses the prompt fine-tuning technology to transfer the capabilities of the pre-trained model to the audio and video event localization task. With a small number of trainable parameters, it can achieve a good end-to-end audio and video event localization effect. Compared with the existing audio and video event localization methods, it reduces the number of trainable parameters and achieves excellent performance on the public AVE dataset. The experimental results show that the positioning accuracy of the method proposed in the present invention is basically the same as that of the most advanced methods in the audio and video event localization task, and the number of trainable parameters is much lower than that of the most advanced methods, which greatly reduces the difficulty of training and resource consumption.

[0054] 2. The present invention aggregates event-related information within the modality through query hint vectors. In order to achieve information interaction between the audio modality and the visual modality and reduce the amount of calculation at the same time, the query hint vector is used to aggregate event-related information within the modality. In the attention calculation process of the transformer layer, the required amount of calculation is proportional to the square of the number of input vectors, so directly calculating the attention of the feature vectors of the two modalities will incur a large amount of computational overhead. At the same time, the feature vector contains a large amount of redundant information, which will interfere with subsequent event location. The use of query hint vectors not only aggregates event-related information within the modality through the attention mechanism, but also greatly reduces the computational overhead during subsequent inter-modal information fusion, thereby improving the efficiency of inter-modal fusion.

[0055] 3. The present invention uses a global cue vector to provide contextual information related to the audio and video event location task, guiding the information fusion between the audio modality and the visual modality. The learnable global cue vector updates parameters during the training process. By calculating the attention weight between the feature vector of the modality and the global cue vector, the information fusion between the two modalities is guided, and information filtering is achieved at the same time, avoiding noise unrelated to the event from affecting the final event location result, thereby improving the accuracy of audio and video event location. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the inventive method;

[0057] Figure 2 This is a structural diagram of the transformer layer in the pre-trained model used in the present invention;

[0058] Figure 3It is a structural diagram of the method of the present invention. DETAILED DESCRIPTION

[0059] In this embodiment, a method for locating audio and video events based on fine-tuning a pre-trained large model is provided. For the specific process, see Figure 1 , is to use the pre-trained model to extract audio and video features, and aggregate the information within the modality through the query prompt vector, and guide the information fusion between modalities through the global prompt vector to locate audio and video events. The algorithm structure diagram of the whole method can be found in Figure 2 Specifically, the method is carried out according to the following steps:

[0060] Step 1: Get a real video data V and divide it into T non-overlapping audio and video segments, denoted as {V't, A't|t=1,2,…,T}, where V't represents the tth video segment and A't represents the audio segment corresponding to V't;

[0061] Sample a frame image in the t-th video segment V't, recorded as the t-th frame image Vt;

[0062] Convert A't into the t-th two-dimensional audio spectrogram At; , where M and F represent the time length and frequency range respectively; in this embodiment, the values ​​of M and F of the audio spectrum are both 224.

[0063] Get the event category label of V and positioning tags , among which, among which, A label indicating whether V belongs to the kth audio or video event category, and ; C represents the number of event categories of audio and video clips; V' t and A' t Whether it belongs to the location tag of the event segment;

[0064] In this embodiment, the AVE dataset is used to train and evaluate the model, including 4143 video sequences of 28 different event categories, 3339 of which are selected for training the model, and the rest are used for evaluating the model. The video duration in the AVE dataset is 10 seconds, and the video is segmented by seconds, that is, T is 10, and 10 video segments are obtained. One frame is sampled for each video segment, and the audio is separated to obtain 10 audio segments.

[0065] Step 2: Build a pre-trained large model, including: a feature encoding layer and N transformer layers; each transformer layer consists of a layer normalization layer, a multi-head attention layer, a random dropout layer, and a multi-layer perceptron arrangement;

[0066] In this embodiment, the basic version and enlarged version of the Vision Transformer model and the basic version and enlarged version of the Swin-Transformer model are used as pre-training models. The layer normalization, multi-head attention, random inactivation and multi-layer perceptron connection in the Transformer layer are as follows: Figure 3 shown.

[0067] Step 2.1: The feature encoding layer converts V t Divide into n non-overlapping image blocks, flatten each image block and splice it to get V t The feature representation , where d is the number of feature representation dimensions of the pre-trained large model.

[0068] The feature encoding layer passes through the convolution layer to A t Process and get A t The feature representation .

[0069] In this embodiment, the number n of divided image blocks is 196, and the number of feature representation dimensions of the pre-trained large model is the number d of convolution kernels in the model, which is 768. Similarly, the value of m is 196.

[0070] Step 2.2: N transformer layers for audio and video modality feature sets Processing is performed to obtain fused audio and video features;

[0071] Step 2.2.1, initialize i=1;

[0072] Step 2.2.2: Randomly initialize the query prompt words to be learned for the visual modality in the i-th transformer layer , and thus use formula (1) to obtain the t-th intermediate visual modality feature representation output by the i-th transformer layer and V t Event clues related to audio and video events ;

[0073] (1)

[0074] In formula (1), represents the i-th transformer layer, V represents the output of the i-1th transformer layer t The visual modality feature representation of is, when i=1, let = , Represents the concatenation operation on the dimension of feature quantity;

[0075] Step 2.2.3: Randomly initialize the query prompt words to be learned for the audio modality in the i-th transformer layer , and thus use formula (2) to obtain the t-th intermediate audio modal feature representation output by the i-th transformer layer and A t Event clues related to audio and video events ;

[0076] (2)

[0077] In formula (2), A represents the output of the i-1th transformer layer t The audio modal feature representation of is, when i=1, let = ;

[0078] Step 2.2.4: Randomly initialize the global cue words to be learned for the visual modality in the i-th transformer layer , and then use formula (3) to get the output V of the i-th transformer layer t Visual modality feature representation , Intermediate query prompt words in audio mode , the intermediate global prompt word of the visual modality ;

[0079] (3)

[0080] Step 2.2.5: Randomly initialize the global cue words to be learned for the audio modality in the i-th transformer layer , and then use formula (4) to get the output A of the i-th transformer layer t Audio modal feature representation , intermediate query hint words in visual mode , the middle global prompt word of the audio mode ;

[0081] (4)

[0082] Step 2.2.6: After assigning i+1 to i, return to step 2.2.2 and execute sequentially until i>N, thus obtaining V t The Nth visual modality feature representation and A t The Nth audio modality feature representation , and then obtain the visual modality feature representation sequence of V and audio modality feature representation sequence , where N represents the total number of transformer layers.

[0083] Step 3: Construct an event localization module, including a video event category classifier and an event correlation predictor, and and Fusion features after splicing on the feature dimension After conversion, the video event classification and positioning results are obtained accordingly:

[0084] Step 3.1: The video event category classifier uses formula (5) to obtain the overall feature representation of V , and then use formula (6) to get the event category prediction label of V :

[0085] (5)

[0086] (6)

[0087] In formula (5), Maxpool represents the maximum pooling layer in the video event category classifier; represents the parameters of the fully connected layer in the video event category classifier; in this embodiment, d is the number of feature dimensions of the model, which is 768, and C is the number of event categories in the data set, which is 28.

[0088] Step 3.2: The event correlation predictor uses formula (7) to obtain the event segment prediction probability of V :

[0089] (7)

[0090] In formula (7), Represents the parameters of the fully connected layer in the event correlation predictor.

[0091] Step 4: Construct the total loss function L for back propagation:

[0092] Step 4.1: Use formula (8) to obtain the cross entropy loss function value of the video event category classifier :

[0093] (8)

[0094] In formula (8), express The probability of belonging to the kth audio and video event category;

[0095] Step 4.2: Use formula (9) to obtain the binary cross entropy loss function value of the event correlation predictor :

[0096] (9)

[0097] In formula (9), express The tth audio and video clip V' t and A' t The predicted probability of belonging to the event segment;

[0098] Step 4.3: Use formula (10) to get the total loss function L:

[0099] (10)

[0100] In formula (10), is a balance parameter; in this embodiment, the balance parameter The value is 0.3.

[0101] Step 5 Based on the real video event labels, the query prompt words and global prompt words to be learned and the event location module are trained, and the total loss function L is calculated. At the same time, the adaptive moment estimation optimization method is used to update the parameters with the learning rate lr until L converges, thereby obtaining the optimal audio and video event location model, which is used to process the input audio and video and obtain the event location result in the video. In this embodiment, different learning rates are set for the parameters of the prompt vector and the event location module. The learning rate of the prompt vector parameter is 0.02, and the learning rate of the event location module parameter is 0.00005.

[0102] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0103] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0104] Example

[0105] In order to verify the effectiveness of the method of the present invention, the commonly used AVE data set was selected for training and testing in this embodiment.

[0106] The method is trained based on the AVE dataset, which has 4143 videos and 28 different event categories. The videos in the AVE dataset have segment-level annotation information, that is, the annotation information of each video contains the category of the audio and video events in the video, as well as the start and end time points of the event. Each video is 10 seconds long, and the audio and video events in it last at least 2 seconds.

[0107] In the present invention, the event category prediction accuracy of all video segments is used as the evaluation index.

[0108] In this embodiment, five methods are selected to compare the effects with the method of the present invention. The selected methods are AVEL (Audio-Visual Event Localization network), DAM (Dual Attention Match), MM-Pyramid (Multimodal Pyramid Attentional Network), PSP (Positive Sample Propagation), and CACE-Net (Co-guidance Attention and Contrastive Enhancement).

[0109] According to the experimental results, the results are shown in Table 1 and Table 2:

[0110] Table 1 Experimental results of deblurring of the proposed method and five selected comparative methods on the AVE dataset

[0111]

[0112] The experimental results show that the proposed method has better effects than the other five methods on the AVE dataset, thus proving the feasibility of the proposed method. The experiment shows that the proposed method can effectively utilize the original capabilities of the pre-trained model, achieve capability transfer through prompt fine-tuning, and complete audio and video tasks.

Claims

1. A method for locating audio and video events based on fine-tuning a pre-trained large model, characterized in that: The steps are as follows: Step 1: Get a real video data V and divide it into T non-overlapping audio and video clips, denoted as {V' t ,A' t |t=1,2,…,T}, where V' t represents the tth video clip, A' t V' t The corresponding audio clip; Sample the tth video segment V' t A frame of image in the t-th frame image V t ; A' t Transformed into the t-th two-dimensional audio spectrum A t ; Get the event category label of V and positioning tags , among which, among which, A label indicating whether V belongs to the kth audio or video event category, and ; C represents the number of event categories of audio and video clips; V' t and A' t Whether it belongs to the location tag of the event segment; Step 2: Build a pre-trained large model, including: a feature encoding layer and N transformer layers; Step 2.1: The feature encoding layer converts V t Divide it into n non-overlapping image blocks, flatten each image block and then splice it to get V t Visual modality feature representation ; The feature encoding layer is implemented by the convolutional layer on A t Process and get A t Audio modal feature representation ; Step 2.2: N transformer layers perform feature extraction on the audio and video modality { } to obtain the visual modality feature representation sequence of V and audio modality feature representation sequence ; Step 3: Construct an event localization module, including a video event category classifier and an event correlation predictor, and and Fusion features after splicing on the feature dimension Convert and get the event category prediction label of V accordingly And the location prediction label of the event segment : Step 4: Based on and , and Construct the total loss function L for back propagation: Step 5: Based on the real video event labels, the query prompt words, global prompt words and event location module to be learned are trained, and the total loss function L is calculated. At the same time, the adaptive moment estimation optimization method is used with the learning rate To update the parameters until L converges, the optimal audio and video event localization model is obtained, which is used to process the input audio and video to obtain the event localization result in the video.

2. According to claim 1, a method for locating audio and video events based on fine-tuning pre-trained large models is characterized in that: The step 2.2 is carried out as follows: Step 2.2.1, initialize i=1; Step 2.2.2: Randomly initialize the query prompt words to be learned for the visual modality in the i-th transformer layer , and thus use formula (1) to obtain the t-th intermediate visual modality feature representation output by the i-th transformer layer and V t Event clues related to audio and video events ; (1) In formula (1), represents the i-th transformer layer, V represents the output of the i-1th transformer layer t The visual modality feature representation of is, when i=1, let = , Represents the concatenation operation on the dimension of feature quantity; Step 2.2.3: Randomly initialize the query prompt words to be learned for the audio modality in the i-th transformer layer , and thus use formula (2) to obtain the t-th intermediate audio modal feature representation output by the i-th transformer layer and A t Event clues related to audio and video events ; (2) In formula (2), A represents the output of the i-1th transformer layer t The audio modal feature representation of is, when i=1, let = ; Step 2.2.4: Randomly initialize the global cue words to be learned for the visual modality in the i-th transformer layer , and then use formula (3) to get the output V of the i-th transformer layer t Visual modality feature representation , Intermediate query prompt words in audio mode , the intermediate global prompt word of the visual modality ; (3) Step 2.2.5: Randomly initialize the global cue words to be learned for the audio modality in the i-th transformer layer , and then use formula (4) to get the output A of the i-th transformer layer t Audio modal feature representation , intermediate query hint words in visual mode , the middle global prompt word of the audio mode ; (4) Step 2.2.6: After assigning i+1 to i, return to step 2.2.2 and execute sequentially until i>N, thus obtaining V t The Nth visual modality feature representation and A t The Nth audio modality feature representation , and then obtain the visual modality feature representation sequence of V and audio modality feature representation sequence , where N represents the total number of transformer layers.

3. The method for locating audio and video events based on fine-tuning pre-trained large models according to claim 2, characterized in that: The step 3 is carried out as follows: Step 3.1: The video event category classifier uses formula (5) to obtain the overall feature representation of V , and then use formula (6) to get the event category prediction label of V : (5) (6) In formula (5), Maxpool represents the maximum pooling layer in the video event category classifier; Representing parameters of a fully connected layer in the video event category classifier; Step 3.2: The event correlation predictor uses formula (7) to obtain the location prediction label of the event segment of V : (7) In formula (7), Represents the parameters of the fully connected layer in the event correlation predictor.

4. The method for locating audio and video events based on fine-tuning a pre-trained large model according to claim 3, characterized in that: The step 4 is carried out as follows: Step 4.1: Use formula (8) to obtain the cross entropy loss function value of the video event category classifier : (8) In formula (8), express The probability of belonging to the kth audio and video event category; Step 4.2: Use formula (9) to obtain the binary cross entropy loss function value of the event correlation predictor : (9) In formula (9), express The tth audio and video clip V' t and A' t The predicted probability of belonging to the event segment; Step 4.3: Use formula (10) to get the total loss function L: (10) In formula (10), is a balance parameter.

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the audio and video event positioning method described in any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the audio and video event locating method according to any one of claims 1 to 4 are executed.

Images