An audio and video event localization method based on fine-tuning pre-trained large model

Abstract

The present invention discloses a method for localizing audio and video events based on a fine-tuned pre-trained large model, comprising the following steps: 1. acquiring video data and separating video frame data from audio data; 2. performing feature extraction on the frame data and audio data respectively through the encoding layer of the pre-trained large model; 3. aggregating features within a single modality by querying prompt words to extract event features; 4. providing contextual information for downstream tasks through global prompt words to fuse audio features and video frame features; and 5. constructing an event localization module to convert the obtained features into video event classification and localization results. By introducing learnable prompt words, the present invention transfers the capabilities of the pre-trained large model to the audio and video event localization task, thereby completing the audio and video event localization task with an extremely low number of trainable parameters.

Description

Technical Field

[0001] The present invention relates to the field of audio and video event positioning, and in particular to an audio and video event positioning method based on a fine-tuned pre-trained large model. Background Art

[0002] Hearing and vision are important pathways for humans to perceive the real world. Video content contains important event information from both the auditory and visual modalities. The task of audio and video event localization requires processing information from both the audio and visual modalities to classify and locate events in the video. In recent years, audio and video event localization, as a fundamental task in video understanding, has garnered increasing attention.

[0003] Existing mainstream audio and video event localization methods adopt a basically similar strategy: first, two models, one based on an image dataset and the other on an audio dataset, are used as a video frame image feature encoder and an audio feature encoder, and then the encoded video frame features and audio features are processed to obtain audio and video event localization results. This method encodes video frames and audio independently into features. Although this processing strategy has achieved good results, it also has inevitable limitations. First, the method of fusing visual and audio features extracted from the original input based on an encoder pre-trained on a single modality dataset can be considered a late fusion. These methods lack the ability to perform early interaction between the two modal information during the feature extraction stage. Second, the encoder pre-trained on a single modality dataset does not participate in the model training process, resulting in the extracted features potentially containing information irrelevant to audio and video event localization, resulting in suboptimal performance.

[0004] Large-scale transformer-based pre-trained vision models, such as the Vision Transformer, can learn effective image feature representations. They do this by viewing an image as a sequence of small patches, converting each patch into a one-dimensional feature, and then processing it using a standard transformer encoder. In recent years, methods that train transformer models on audio and video datasets and apply them to downstream tasks in audio and video processing have achieved promising results. However, due to the large number of parameters in transformer models, training a transformer model from scratch requires significant computational resources. To leverage the capabilities of pre-trained transformer models while avoiding the significant computational overhead of training a full model, researchers have proposed the method of prompt fine-tuning. During model training, prompt fine-tuning freezes the parameters of the pre-trained model, essentially training only a continuous prompt vector without updating the pre-trained model parameters. Specifically, prompt fine-tuning introduces a learnable continuous vector, called a prompt vector, at the input of the transformer model, and only updates the parameters of the prompt vector during training. Existing methods primarily focus on enhancing the few-shot learning capabilities of the original model, making them difficult to apply to audio and video event localization tasks. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing methods, the present invention provides an audio and video event localization method based on fine-tuning a pre-trained large model, so as to transfer the original capabilities of the pre-trained large model to the audio and video event localization task through prompt fine-tuning, thereby improving the accuracy of audio and video event localization.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The audio and video event localization method based on fine-tuning the pre-trained large model of the present invention is characterized in that it is carried out according to the following steps:

[0008] Step 1: Get a real video data V and divide it into T non-overlapping audio and video clips, denoted as {V' t ,A' t |t=1,2,…,T}, where V' t represents the tth video clip, A' t Indicates V' t The corresponding audio clip;

[0009] Sample the tth video segment V' t A frame of image in the t-th frame image V t ;

[0010] A' tConverted into the t-th two-dimensional audio spectrum A t ;

[0011] Get the event category label of V and positioning tags , among which, among which, Indicates whether V belongs to the kth audio and video event category, and ; C represents the number of event categories of audio and video clips; Indicates V' t and A' t Whether it belongs to the location tag of the event segment;

[0012] Step 2: Build a pre-trained large model, including: a feature encoding layer and N transformer layers;

[0013] Step 2.1, the feature encoding layer converts V t Divide into n non-overlapping image blocks, flatten each image block and then splice it to get V t Visual modality feature representation ;

[0014] The feature encoding layer is processed by the convolution layer t Process and get A t Audio modal feature representation of ;

[0015] Step 2.2, N transformer layers for audio and video modality feature set { } is processed to obtain the visual modality feature representation sequence of V and audio modality feature representation sequence ;

[0016] Step 3: Construct the event location module, including the video event category classifier and event correlation predictor, and and Fusion features after splicing on the feature dimension Perform the conversion and obtain the event category prediction label of V accordingly And the location prediction label of the event segment :

[0017] Step 4: Based on and , and Construct the total loss function L for back propagation:

[0018] Step 5: Based on the real video event labels, the query prompt words and global prompt words to be learned and the event positioning module are trained, and the total loss function L is calculated. At the same time, the adaptive moment estimation optimization method is used with the learning rate To update the parameters until L converges, the optimal audio and video event localization model is obtained, which is used to process the input audio and video to obtain the event localization result in the video.

[0019] The audio and video event localization method based on fine-tuning the pre-trained large model described in the present invention is also characterized in that step 2.2 is performed as follows:

[0020] Step 2.2.1, initialize i=1;

[0021] Step 2.2.2: Randomly initialize the query hint words to be learned for the visual modality in the i-th transformer layer , and thus use formula (1) to obtain the t-th intermediate visual modality feature representation output by the i-th transformer layer and V t Event clues related to audio and video events ;

[0022] (1)

[0023] In formula (1), represents the i-th transformer layer, V represents the output of the i-1th transformer layer t The visual modality feature representation of , when i=1, let = , Represents the concatenation operation on the dimension of feature quantity;

[0024] Step 2.2.3: Randomly initialize the query hint words to be learned for the audio modality in the i-th transformer layer , and thus use formula (2) to obtain the t-th intermediate audio modal feature representation output by the i-th transformer layer and A t Event clues related to audio and video events ;

[0025] (2)

[0026] In formula (2), A represents the output of the i-1th transformer layer t The audio modal feature representation of , when i=1, let = ;

[0027] Step 2.2.4: Randomly initialize the global cue words to be learned for the visual modality in the i-th transformer layer , and then use formula (3) to get the V output of the i-th transformer layer t Visual modality feature representation , intermediate query prompt words in audio mode , the intermediate global prompt word of the visual modality ;

[0028] (3)

[0029] Step 2.2.5: Randomly initialize the global cue words to be learned for the audio modality in the i-th transformer layer , and thus use formula (4) to get the output A of the i-th transformer layer t Audio modal feature representation , intermediate query prompt words in visual modality , the middle global prompt word of the audio mode ;

[0030] (4)

[0031] Step 2.2.6: After assigning i+1 to i, return to step 2.2.2 and execute sequentially until i>N, thus obtaining V t The Nth visual modality feature representation and A t The Nth audio modality feature representation , and then obtain the visual modality feature representation sequence of V and audio modality feature representation sequence , where N represents the total number of transformer layers.

[0032] Furthermore, step 3 is performed as follows:

[0033] Step 3.1: The video event category classifier uses formula (5) to obtain the overall feature representation of V , and then use formula (6) to get the event category prediction label of V :

[0034] (5)

[0035] (6)

[0036] In formula (5), Maxpool represents the maximum pooling layer in the video event category classifier; Representing parameters of the fully connected layer in the video event category classifier;

[0037] Step 3.2: The event correlation predictor uses formula (7) to obtain the location prediction label of the event segment of V :

[0038] (7)

[0039] In formula (7), Represents the parameters of the fully connected layer in the event correlation predictor.

[0040] Furthermore, step 4 is performed as follows:

[0041] Step 4.1: Use formula (8) to obtain the cross entropy loss function value of the video event category classifier :

[0042] (8)

[0043] In formula (8), express The probability of belonging to the kth audio and video event category;

[0044] Step 4.2: Use formula (9) to obtain the binary cross entropy loss function value of the event correlation predictor :

[0045] (9)

[0046] In formula (9), express The tth audio and video clip V' t and A' t The predicted probability of belonging to the event segment;

[0047] Step 4.3: Use formula (10) to get the total loss function L:

[0048] (10)

[0049] In formula (10), is the balance parameter.

[0050] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the audio and video event positioning method, and the processor is configured to execute the program stored in the memory.

[0051] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium. The characteristic of the computer program is that the steps of the audio and video event locating method are executed when the computer program is run by a processor.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. This paper utilizes prompt fine-tuning technology to transfer the capabilities of pre-trained models to the task of audio and video event localization. This method achieves excellent end-to-end audio and video event localization with a small number of trainable parameters. Compared to existing audio and video event localization methods, it reduces the number of trainable parameters and achieves excellent performance on the public AVE dataset. Experimental results show that the positioning accuracy of the proposed method for audio and video event localization is basically on par with the performance of the most advanced methods, while requiring far fewer trainable parameters than the most advanced methods, significantly reducing the difficulty of training and resource consumption.

[0054] 2. The present invention aggregates event-related information within the modality through query hint vectors. In order to achieve information interaction between the audio modality and the visual modality and reduce the amount of computation, the query hint vector is used to aggregate event-related information within the modality. In the attention calculation process of the transformer layer, the required amount of computation is proportional to the square of the number of input vectors, so directly calculating the attention of the feature vectors of the two modalities will generate a large amount of computational overhead. At the same time, the feature vector contains a large amount of redundant information, which will interfere with subsequent event positioning. The use of query hint vectors not only aggregates event-related information within the modality through the attention mechanism, but also greatly reduces the computational overhead during subsequent inter-modal information fusion, thereby improving the efficiency of inter-modal fusion.

[0055] 3. This invention uses a global cue vector to provide contextual information relevant to the audio and video event localization task, guiding information fusion between the audio and visual modalities. The learnable global cue vector updates its parameters during the training process. By calculating the attention weight between the modality's feature vector and the global cue vector, information fusion between the two modalities is guided, while simultaneously filtering information to prevent event-unrelated noise from affecting the final event localization results, thereby improving the accuracy of audio and video event localization. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flow chart of the invention method;

[0057] Figure 2 This is a structural diagram of the transformer layer in the pre-trained model used in the present invention;

[0058] Figure 3Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0059] In this embodiment, a method for locating audio and video events based on fine-tuning a pre-trained large model is provided. For the specific process, see Figure 1 , is to use the pre-trained model to extract audio and video features, and aggregate the information within the modality through query hint vectors, and guide the information fusion between modalities through the global hint vector to locate audio and video events. The algorithm structure diagram of the whole method can be found in Figure 2 Specifically, the method is carried out in the following steps:

[0060] Step 1: Get a real video data V and divide it into T non-overlapping audio and video segments, denoted as {V't, A't|t=1,2,…,T}, where V't represents the t-th video segment and A't represents the audio segment corresponding to V't;

[0061] Sample a frame image from the t-th video segment V't, recorded as the t-th frame image Vt;

[0062] Convert A't into the t-th two-dimensional audio spectrogram At; , where M and F represent the time length and frequency range respectively; in this embodiment, the values ​​of M and F of the audio spectrogram are both 224.

[0063] Get the event category label of V and positioning tags , among which, among which, Indicates whether V belongs to the kth audio and video event category, and ; C represents the number of event categories of audio and video clips; Indicates V' t and A' t Whether it belongs to the location tag of the event segment;

[0064] In this example, the AVE dataset was used to train and evaluate the model. This dataset contains 4,143 video sequences from 28 different event categories. 3,339 of these videos were selected for model training, and the remaining were used for model evaluation. Each video in the AVE dataset is 10 seconds long. The videos are segmented by second, with a T value of 10, resulting in 10 video segments. One frame of each video segment was sampled, and the audio was separated to produce 10 audio segments.

[0065] Step 2: Build a large pre-trained model, including a feature encoding layer and N transformer layers. Each transformer layer consists of a layer normalization layer, a multi-head attention layer, a random dropout layer, and a multi-layer perceptron arrangement.

[0066] In this embodiment, the basic version and enlarged version of the Vision Transformer model, and the basic version and enlarged version of the Swin-Transformer model are used as pre-training models. The layer normalization, multi-head attention, random inactivation and multi-layer perceptron connection in the Transformer layer are as follows. Figure 3 shown.

[0067] Step 2.1, the feature encoding layer converts V t Divide into n non-overlapping image blocks, flatten each image block and splice it to get V t Feature representation , where d is the number of feature representation dimensions of the pre-trained large model.

[0068] The feature encoding layer is transformed into A through the convolution layer. t Process and get A t Feature representation .

[0069] In this embodiment, the number n of divided image blocks is 196, and the feature representation dimension of the pre-trained large model is the number d of convolution kernels in the model, which is 768. Similarly, the value of m is 196.

[0070] Step 2.2: N transformer layers for audio and video modality feature sets Processing is performed to obtain fused audio and video features;

[0071] Step 2.2.1, initialize i=1;

[0072] Step 2.2.2: Randomly initialize the query hint words to be learned for the visual modality in the i-th transformer layer , and thus use formula (1) to obtain the t-th intermediate visual modality feature representation output by the i-th transformer layer and V t Event clues related to audio and video events ;

[0073] (1)

[0074] In formula (1), represents the i-th transformer layer, V represents the output of the i-1th transformer layer t The visual modality feature representation of , when i=1, let = , Represents the concatenation operation on the dimension of feature quantity;

[0075] Step 2.2.3: Randomly initialize the query hint words to be learned for the audio modality in the i-th transformer layer , and thus use formula (2) to obtain the t-th intermediate audio modal feature representation output by the i-th transformer layer and A t Event clues related to audio and video events ;

[0076] (2)

[0077] In formula (2), A represents the output of the i-1th transformer layer t The audio modal feature representation of , when i=1, let = ;

[0078] Step 2.2.4: Randomly initialize the global cue words to be learned for the visual modality in the i-th transformer layer , and then use formula (3) to get the V output of the i-th transformer layer t Visual modality feature representation , intermediate query prompt words in audio mode , the intermediate global prompt word of the visual modality ;

[0079] (3)

[0080] Step 2.2.5: Randomly initialize the global cue words to be learned for the audio modality in the i-th transformer layer , and thus use formula (4) to get the output A of the i-th transformer layer t Audio modal feature representation , intermediate query prompt words in visual modality , the middle global prompt word of the audio mode ;

[0081] (4)

[0082] Step 2.2.6: After assigning i+1 to i, return to step 2.2.2 and execute sequentially until i>N, thus obtaining V t The Nth visual modality feature representation and A t The Nth audio modality feature representation , and then obtain the visual modality feature representation sequence of V and audio modality feature representation sequence , where N represents the total number of transformer layers.

[0083] Step 3: Construct the event location module, including the video event category classifier and event correlation predictor, and and Fusion features after splicing on the feature dimension Perform the conversion and obtain the corresponding video event classification and positioning results:

[0084] Step 3.1: The video event category classifier uses formula (5) to obtain the overall feature representation of V , and then use formula (6) to get the event category prediction label of V :

[0085] (5)

[0086] (6)

[0087] In formula (5), Maxpool represents the maximum pooling layer in the video event category classifier; represents the parameters of the fully connected layer in the video event category classifier; in this embodiment, d is the number of feature dimensions of the model, which is 768, and C is the number of event categories in the dataset, which is 28.

[0088] Step 3.2: The event correlation predictor uses formula (7) to obtain the event segment prediction probability of V :

[0089] (7)

[0090] In formula (7), Represents the parameters of the fully connected layer in the event correlation predictor.

[0091] Step 4. Construct the total loss function L for back propagation:

[0092] Step 4.1: Use formula (8) to obtain the cross entropy loss function value of the video event category classifier :

[0093] (8)

[0094] In formula (8), express The probability of belonging to the kth audio and video event category;

[0095] Step 4.2: Use formula (9) to obtain the binary cross entropy loss function value of the event correlation predictor :

[0096] (9)

[0097] In formula (9), express The tth audio and video clip V' t and A' t The predicted probability of belonging to the event segment;

[0098] Step 4.3: Use formula (10) to get the total loss function L:

[0099] (10)

[0100] In formula (10), is the balance parameter; in this embodiment, the balance parameter The value is 0.3.

[0101] Step 5: Train the query prompts and global prompts to be learned, as well as the event localization module, based on real video event labels. The total loss function L is calculated. The adaptive moment estimation optimization method is used to update the parameters at a learning rate of lr until L converges. This results in an optimal audio and video event localization model, which is used to process the input audio and video and obtain event localization results in the video. In this embodiment, different learning rates are set for the prompt vector and event localization module parameters: the learning rate for the prompt vector parameters is 0.02, and the learning rate for the event localization module parameters is 0.00005.

[0102] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0103] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0104] Example

[0105] In order to verify the effectiveness of the method of the present invention, the commonly used AVE dataset was selected for training and testing in this embodiment.

[0106] This method is trained on the AVE dataset, which contains 4,143 videos covering 28 different event categories. The videos in the AVE dataset are annotated at the segment level. Each video is annotated with the category of the audio and video event, as well as the start and end time of the event. Each video is 10 seconds long, with audio and video events lasting at least 2 seconds.

[0107] In the present invention, the event category prediction accuracy of all video segments is used as the evaluation indicator.

[0108] In this embodiment, five methods are selected to compare the effects with the method of the present invention. The selected methods are AVEL (Audio-Visual Event Localization network), DAM (Dual Attention Match), MM-Pyramid (Multimodal Pyramid Attentional Network), PSP (Positive Sample Propagation), and CACE-Net (Co-guidance Attention and Contrastive Enhancement).

[0109] According to the experimental results, the results are shown in Table 1 and Table 2:

[0110] Table 1 Experimental results of deblurring using the proposed method and five selected comparative methods on the AVE dataset

[0111]

[0112] Experimental results show that our method performs better than the other five methods on the AVE dataset, demonstrating the feasibility of our proposed method. The experiments demonstrate that our proposed method can effectively leverage the existing capabilities of the pre-trained model, transferring these capabilities through prompt fine-tuning to complete audio and video tasks.

Claims

1. A method for localizing audio and video events based on fine-tuning a pre-trained large model, characterized in that: The steps are as follows: Step 1: Get a real video data V and divide it into T non-overlapping audio and video clips, denoted as {V' t ,A' t |t=1,2,…,T}, where V' t represents the tth video clip, A' t Indicates V' t The corresponding audio clip; Sample the tth video segment V' t A frame of image in the t-th frame image V t ; A' t Converted into the t-th two-dimensional audio spectrum A t ; Get the event category label of V and positioning tags , among which, among which, Indicates whether V belongs to the kth audio and video event category, and ; C represents the number of event categories of audio and video clips; Indicates V' t and A' t Whether it belongs to the location tag of the event segment; Step 2: Build a pre-trained large model, including: a feature encoding layer and N transformer layers; Step 2.1, the feature encoding layer converts V t Divide into n non-overlapping image blocks, flatten each image block and then splice it to get V t Visual modality feature representation ; The feature encoding layer is processed by the convolution layer t Process and get A t Audio modal feature representation ; Step 2.2, N transformer layers for audio and video modality feature set { } is processed to obtain the visual modality feature representation sequence of V and audio modality feature representation sequence ; Step 2.2.1, initialize i=1; Step 2.2.2: Randomly initialize the query hint words to be learned for the visual modality in the i-th transformer layer , and thus use formula (1) to obtain the t-th intermediate visual modality feature representation output by the i-th transformer layer and V t Event clues related to audio and video events ; (1) In formula (1), represents the i-th transformer layer, V represents the output of the i-1th transformer layer t The visual modality feature representation of , when i=1, let = , Represents the concatenation operation on the dimension of feature quantity; Step 2.2.3: Randomly initialize the query hint words to be learned for the audio modality in the i-th transformer layer , and thus use formula (2) to obtain the t-th intermediate audio modal feature representation output by the i-th transformer layer and A t Event clues related to audio and video events ; (2) In formula (2), A represents the output of the i-1th transformer layer t The audio modal feature representation of , when i=1, let = ; Step 2.2.4: Randomly initialize the global cue words to be learned for the visual modality in the i-th transformer layer , and then use formula (3) to get the V output of the i-th transformer layer t Visual modality feature representation , intermediate query prompt words in audio mode , the intermediate global prompt word of the visual modality ; (3) Step 2.2.5: Randomly initialize the global cue words to be learned for the audio modality in the i-th transformer layer , and thus use formula (4) to get the output A of the i-th transformer layer t Audio modal feature representation of , intermediate query prompt words in visual modality , the middle global prompt word of the audio mode ; (4) Step 2.2.6: After assigning i+1 to i, return to step 2.2.2 and execute sequentially until i>N, thus obtaining V t The Nth visual modality feature representation and A t The Nth audio modality feature representation , and then obtain the visual modality feature representation sequence of V and audio modality feature representation sequence , where N represents the total number of transformer layers; Step 3: Construct the event location module, including the video event category classifier and event correlation predictor, and and Fusion features after splicing on the feature dimension Perform the conversion and obtain the event category prediction label of V accordingly And the location prediction label of the event segment : Step 4: Based on and , and Construct the total loss function L for back propagation: Step 4.1: Use formula (8) to obtain the cross entropy loss function value of the video event category classifier : (8) In formula (8), express The probability of belonging to the kth audio and video event category; Step 4.2: Use formula (9) to obtain the binary cross entropy loss function value of the event correlation predictor : (9) In formula (9), express The tth audio and video clip V' t and A' t The predicted probability of belonging to the event segment; Step 4.3: Use formula (10) to get the total loss function L: (10) In formula (10), is the balance parameter; Step 5: Based on the real video event labels, the query prompt words and global prompt words to be learned and the event positioning module are trained, and the total loss function L is calculated. At the same time, the adaptive moment estimation optimization method is used with the learning rate To update the parameters until L converges, the optimal audio and video event localization model is obtained, which is used to process the input audio and video to obtain the event localization result in the video.

2. The method for locating audio and video events based on fine-tuning a pre-trained large model according to claim 1, characterized in that: Described step 3 is carried out as follows: Step 3.1: The video event category classifier uses formula (5) to obtain the overall feature representation of V , and then use formula (6) to get the event category prediction label of V : (5) (6) In formula (5), Maxpool represents the maximum pooling layer in the video event category classifier; Representing parameters of the fully connected layer in the video event category classifier; Step 3.2: The event correlation predictor uses formula (7) to obtain the location prediction label of the event segment of V : (7) In formula (7), Represents the parameters of the fully connected layer in the event correlation predictor.

3. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the audio and video event positioning method according to any one of claims 1-2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the audio and video event locating method according to any one of claims 1 to 2 are executed.

Images