Audiovisual event localization system and method based on cross-modal consistency and temporal multi-granularity collaboration

Abstract

The present invention discloses an audio-visual event positioning system and method based on cross-modal consistency and temporal multi-granularity collaboration, and relates to the technical field of computer vision. A unimodal feature encoding module obtains unimodal features of video and audio; a cross-modal consistency collaboration module is composed of a plurality of multimodal collaboration layers stacked, each of which includes a visual guidance module and an audio guidance module, and the visual guidance module and the audio guidance module are both composed of a cross-modal interaction branch and a temporal consistency branch, and the video features and audio features output by each multimodal collaboration layer are spliced ​​to obtain multi-granularity temporal features; a temporal multi-granularity collaboration module performs temporal collaboration from coarse-grained to fine-grained and from fine-grained to coarse-grained multi-granularity temporal features; a decoding module predicts event categories and event times through features enhanced by temporal bidirectional collaboration. The present invention can be used for processing, analyzing and positioning long sequences of audio-visual videos.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an audiovisual event localization system and method based on cross-modal consistency and temporal multi-granularity collaboration. Background Art

[0002] In recent years, there has been a surge of interest in the joint exploration and understanding of audio and visual signals, resulting in many audio-visual learning tasks and applications. However, most of these studies focus on clipped videos of short duration, typically 5 or 10 seconds. In practice, there is often a more complex application scenario, namely dense audio-visual event localization, which focuses on scene understanding of uncut videos with audio tracks of varying duration.

[0003] Dense audiovisual event localization can be considered an extension of the existing audiovisual event localization task. Both tasks aim to identify and localize audiovisual events, i.e., events that occur simultaneously in audio and visual tracks. However, existing audiovisual event detection systems have the following drawbacks:

[0004] First, it focuses on short videos (fixed at 10 seconds), where each video contains only one specific event category. However, dense audio-visual event localization systems need to process untrimmed videos, which have variable lengths and most of them exceed 10 seconds.

[0005] For example, in 2018, Yapeng Tian and other authors published an article titled "Audio-Visual Event Localization in Unconstrained Videos" at the top international conference European Conference on Computer Vision. The article first defined the problem of audio-visual event localization in unconstrained videos, that is, how to use audio and visual information simultaneously to locate and identify events in videos. This paper first proposed the task of audio and video localization in videos, and proposed a dataset for audio and video events. Traditional audio-visual event localization methods are based on the dataset proposed in this article. However, this dataset focuses on short videos with a fixed length of 10 seconds, and a video contains only one event. Traditional methods are limited by the limitations of this dataset and cannot handle more complex and dense audio and video event localization tasks.

[0006] Second, in real scenarios, each video usually contains dense events, and events from multiple categories may have different time spans, which indicates the diversity of events. Traditional audio-visual event detection methods and systems do not pay attention to the temporal diversity of events and cannot handle audio-visual events with different time spans.

[0007] For example, in 2023, Tiantian Geng et al. published an article titled "Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline" at the top international conference Conference on Computer Vision and Pattern Recognition. The article first proposed the task of locating dense audio-visual events and introduced the first untrimmed audio-visual dataset, with an average of 2.8 audio-visual events per video. These events are usually interrelated and may occur simultaneously, simulating real-life scenarios. The article also proposed a method for locating dense audio-visual events, but the method cannot handle multi-scale audio-visual events and is not very effective in processing audio-visual events of shorter duration. Summary of the invention

[0008] In order to overcome the above-mentioned defects in the prior art, the present invention provides an audio-visual event localization system based on cross-modal consistency and temporal multi-granularity collaboration, which can be used for processing, analyzing and localizing long sequences of audio-visual videos.

[0009] To achieve the above object, the present invention adopts the following technical solutions, including:

[0010] An audiovisual event positioning system based on cross-modal consistency and temporal multi-granularity collaboration includes a positioning model for positioning audiovisual events; the positioning model includes: a single-modal feature encoding module, a cross-modal consistency collaboration module, a temporal multi-granularity collaboration module and a decoding module;

[0011] The unimodal feature encoding module is used to extract features from the video and audio, and perform feature encoding to obtain unimodal features of the video and audio, respectively;

[0012] Inputting the unimodal features of video and audio into a cross-modal consistency collaboration module; the cross-modal consistency collaboration module is composed of a plurality of stacked multimodal collaboration layers, each of which includes a visual guidance module and an audio guidance module, the visual guidance module and the audio guidance module have the same network structure, both of which are composed of a cross-modal interaction branch and a temporal consistency branch, and are used to perform cross-modal interaction on the unimodal features of video and audio and maintain temporal consistency, and the visual guidance module and the audio guidance module output video features and audio features respectively;

[0013] The video features and audio features output by each multimodal collaboration layer are spliced ​​to obtain multi-granularity temporal features, and are input into a temporal multi-granularity collaboration module; the temporal multi-granularity collaboration module includes a coarse-to-fine-granularity collaboration module and a fine-to-coarse-granularity collaboration module, which are used to perform temporal collaboration from coarse-grained to fine-grained and from fine-grained to coarse-grained on the multi-granularity temporal features, and obtain features enhanced by temporal bidirectional collaboration;

[0014] The features enhanced by the temporal bidirectional collaboration are input into a decoding module; the decoding module is used to predict the category and time of each audiovisual event in the audio and video.

[0015] Preferably, the cross-modal consistency collaboration module is composed of L c The multimodal collaboration layers are stacked together; between two adjacent multimodal collaboration layers, the output of the previous multimodal collaboration layer is used as the input of the next multimodal collaboration layer; the input of the first multimodal collaboration layer is the output of the unimodal feature encoding module, that is, the unimodal feature F of the video V and the unimodal features of the audio F A ;

[0016] In the nth cross-modal collaboration layer, the enhanced audio features and video features are obtained through the cross-modal interaction branches of the audio guidance module and the visual guidance module, respectively, which are recorded as and The specific processing method is as follows:

[0017] ;

[0018] ;

[0019] Where n = 1,2,...,L c ; , ;

[0020] MHA is a multi-head attention module, which is calculated as:

[0021] ;

[0022] Among them, W Q , W K , W V are all learnable parameters, Softmax( ) represents the Softmax activation function, and the superscript T Represents a matrix transpose operation;

[0023] In the nth cross-modal collaboration layer, the masked audio features and video features are obtained through the temporal consistency branches of the audio guidance module and the visual guidance module, respectively, and are recorded as and The specific processing method is as follows:

[0024] ;

[0025] ;

[0026] Among them, σ is the Sigmoid function, W v and W a is a learnable parameter, g V and g A are the timing weights of video and audio respectively;

[0027] The masked audio features and video features output by each multimodal collaboration layer are concatenated to obtain multi-granularity temporal features, denoted as .

[0028] Preferably, F V ∈R T×D , F A ∈R T×D ; , ; Where T is the time series length of the unimodal feature; T n is the time series length of the output feature of the nth cross-modal collaboration layer, T n = T / 2 n-1 ; D is the feature dimension; R represents the data shape.

[0029] Preferably, the multi-granularity temporal features of each multimodal collaboration layer are input into the temporal multi-granularity collaboration module, and the specific processing method of the temporal multi-granularity collaboration module is as follows:

[0030] First, the coarse-to-fine-grained collaboration module is used to perform time series collaboration from coarse-grained to fine-grained multi-grained time series features to obtain the updated fine-grained feature G k , k = 1,2,...,L c -1, the specific processing method is as follows:

[0031] ;

[0032] ;

[0033] Among them, U k is the updated coarse-grained feature, W u is a learnable parameter, ReLU is the activation function, G Lc = Z Lc ;

[0034] Then, the fine-to-coarse-grained collaboration module is used to perform temporal collaboration from fine-grained to coarse-grained to obtain the enhanced temporal bidirectional collaboration feature O. m , m = 2,...,L c The specific processing method is as follows:

[0035] ;

[0036] ;

[0037] in, is the feature after pooling operation, MaxPooling is the pooling operation, O 1 = G 1 .

[0038] Preferably, in the unimodal feature encoding module, the video and audio are firstly extracted to obtain the video feature V and the audio feature A; then the video feature V and the audio feature A are respectively input into their respective unimodal Transformer networks for unimodal feature encoding to obtain the unimodal features F of the video and F of the audio. V and the unimodal features of the audio F A .

[0039] Preferably, a pre-trained two-stream I3D model is used to process the video and extract the video feature V; and a pre-trained VGGish model is used to process the audio and extract the audio feature A.

[0040] Preferably, the decoding module comprises a classification head and a regression head; the classification head is used to predict the category of each audiovisual event; the regression head is used to predict the time of each audiovisual event, including the start time and end time of the event.

[0041] Preferably, the classification head consists of three one-dimensional convolutional layers and a Sigmoid activation function; the regression head consists of three one-dimensional convolutional layers and a ReLU activation function.

[0042] The audiovisual event localization method based on cross-modal consistency and temporal multi-granularity collaboration is applicable to the above-mentioned audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration. The training method of the localization model specifically includes the following steps:

[0043] S1, obtain samples, build data sets, and perform training initialization settings;

[0044] The sample is audio and video, and the label of the sample is the category and time of each audiovisual event contained in the audio and video; define the current number of training times as d, and let the model parameter of the dth training be ε d; Set the total number of training times to epoch; divide the data set into a training set and a validation set, and set the initial best accuracy of the validation set to 0;

[0045] S2, using the unimodal feature encoding module to process the samples in the training set, namely the audio and video, to obtain the unimodal features of the video and the unimodal features of the audio;

[0046] S3, using the cross-modal consistency collaboration module, performs cross-modal interaction on the unimodal features of video and audio and maintains temporal consistency, splicing the video features and audio features output by each multimodal collaboration layer to obtain multi-granularity temporal features;

[0047] S4, input the multi-granularity time series features of each multimodal collaboration layer into the time series multi-granularity collaboration module, use the coarse-to-fine-grained collaboration module to perform time series collaboration from coarse-grained to fine-grained on the multi-granularity time series features to obtain updated fine-grained features; use the fine-to-coarse-grained collaboration module to perform time series collaboration from fine-grained to coarse-grained on the updated fine-grained features to obtain features enhanced by time series bidirectional collaboration;

[0048] S5, use the decoding module to decode the features enhanced by temporal bidirectional collaboration and predict the category Y of the audiovisual event P and time Q P , where Y P ∈R T× C , T represents the time series length of the feature, C represents the total number of categories; the time of the audiovisual event includes the start time and end time of the event, Q P ∈R T× C ×2 ;

[0049] S6, calculate the category prediction loss value L reg And the time prediction loss value L cls :

[0050] ;

[0051] ;

[0052] Among them, Y y is the true category label of the event, Q y is the real time label of the event; FocalLoss is the cross entropy loss function, IOULoss is the intersection-over-union loss function;

[0053] Calculate the total loss L(ε) during model training d ):

[0054] ;

[0055] Where s(t) is an indicator function, indicating whether there is an audiovisual event at time t. If yes, s(t)=1, otherwise s(t)=0; t=1,2,...,T;

[0056] According to the total loss L(ε d ) performs back propagation and updates the model parameters to obtain the positioning model for this training;

[0057] S7, running the positioning model trained this time on the validation set. If the average accuracy on the validation set is greater than the existing best accuracy, the best model parameters are updated to the model parameters obtained from this training, and the best accuracy is updated to the average accuracy on the validation set; otherwise, the best model parameters and best accuracy are not updated.

[0058] S8, determine whether the number of training times d reaches the total number of training epochs. If so, end the training process and output the optimal model parameters; otherwise, perform the next training in the same manner as steps S2-S7, and update the number of training times d=d+1.

[0059] A computer program product includes a computer program / instruction, which implements the above-mentioned audiovisual event localization method based on cross-modal consistency and temporal multi-granularity collaboration when executed by a processor.

[0060] The advantages of the present invention are:

[0061] (1) This paper comprehensively analyzes the three key features in the development of dense audio-visual event localization systems, and proposes a new network based on Transformer, cross-modal consistency and temporal multi-granularity collaboration mechanism, which can be used for processing, analysis and localization of long sequences of audio-visual videos.

[0062] (2) The present invention designs a cross-modal collaboration module, which includes a multimodal interaction branch and a temporal consistency branch. It not only considers the semantic consistency of events in cross-modal interaction, but also considers the temporal consistency, and can encode more effective audio-visual feature representations.

[0063] (3) The present invention designs a temporal multi-granularity collaboration module, which adopts a temporal collaboration mechanism from coarse-grained to fine-grained and from fine-grained to coarse-grained, realizing bidirectional collaboration of temporal information from coarse-grained to fine-grained and from fine-grained to coarse-grained, bidirectionally utilizing audio-visual features across various granularities for event positioning, and can more accurately analyze and locate audio-visual events of various granularities. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a system architecture diagram of the positioning model of the present invention.

[0065] Figure 2This is a flow chart of the training method of the positioning model of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The present invention aims to improve the performance of the dense audiovisual event location system and enhance its ability to locate and classify audiovisual events of different time spans. To this end, the present invention first analyzes three key features in the development of the dense audiovisual event location system:

[0068] 1. Cross-modal event consistency: The present invention is mainly used for dense audio-visual event detection, where audio-visual events are the intersection of audio events and video events. Therefore, when developing a system or model, attention should be paid to capturing the shared consistent event semantics of audio and video modalities.

[0069] 2. Cross-modal temporal consistency: Not all temporal segments contain audio or video events, that is, some segments in the video may be background or noise. In a temporal segment, even if one modality contains an event, if the event does not appear in the other modality, it cannot be located as an audiovisual event, thus requiring temporal consistency between the audio and video modalities.

[0070] 3. Inconsistency in event duration: In practical applications, a video usually contains multiple audiovisual events with different durations, which requires system development or model design to pay attention to the problem of inconsistent event duration.

[0071] To solve the above problems, the present invention proposes an audiovisual event localization system and method based on cross-modal consistency and temporal multi-granularity collaboration.

[0072] Example 1

[0073] The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration of the present invention can effectively perceive audiovisual events of different lengths, types, and times, thereby realizing accurate localization and analysis of these events, such as Figure 1 As shown, the system includes a positioning model for positioning audiovisual events; the positioning model includes: a single-modal feature encoding module, a cross-modal consistency collaboration module, a temporal multi-granularity collaboration module and a decoding module.

[0074] The unimodal feature encoding module extracts features from the video and audio respectively, and performs feature encoding to obtain unimodal features of the video and unimodal features of the audio.

[0075] The unimodal features of video and audio are input into a cross-modal consistency collaboration module; the cross-modal consistency collaboration module is composed of a plurality of stacked multimodal collaboration layers, each of which includes a visual guidance module and an audio guidance module; the network structure of the visual guidance module and the audio guidance module is the same, both of which are composed of a cross-modal interaction branch and a temporal consistency branch, and are used to perform cross-modal interaction on the unimodal features of video and audio and maintain temporal consistency, and the visual guidance module and the audio guidance module output video features and audio features respectively.

[0076] The video features and audio features output by each multimodal collaboration layer are spliced ​​to obtain multi-granularity temporal features, and are input into a temporal multi-granularity collaboration module; the temporal multi-granularity collaboration module includes a coarse-to-fine-granularity collaboration module and a fine-to-coarse-granularity collaboration module, which are used to perform temporal collaboration from coarse-grained to fine-grained and from fine-grained to coarse-grained on the multi-granularity temporal features to obtain features enhanced by temporal bidirectional collaboration.

[0077] The features enhanced by the temporal bidirectional collaboration are input into a decoding module; the decoding module is used to predict event categories and event times.

[0078] Specifically, in the unimodal feature encoding module, the pre-trained two-stream I3D model is used to process the video and extract the video feature V, V∈R T×dv ; Use the pre-trained VGGish model to process the audio and extract the audio features A, A∈R T×da ; Where T is the time series length, T=224, dv, da represent feature dimensions, and R represents data shape. In this process, the length of the video may change, so the feature vector obtained above is cropped or padded to the time series length T. The video feature V and audio feature A are input into their own independent unimodal Transformer (self-attention) networks, and the self-attention mechanism is used to achieve temporal interaction within each modality to obtain the unimodal feature F of the processed video. V and the unimodal features of the audio F A ; F V ∈R T×D , F A ∈R T×D , D represents the feature dimension.

[0079] The obtained unimodal feature F of the video V and the unimodal features of the audio F A Input into the cross-modal consistency collaboration module.

[0080] Specifically, in the cross-modal consistency collaboration module, L c The multimodal collaboration layers are stacked, L c= 6; between two adjacent multimodal collaboration layers, the output of the previous multimodal collaboration layer is used as the input of the next multimodal collaboration layer; the input of the first multimodal collaboration layer is the output of the unimodal feature encoding module, that is, the unimodal feature F of the video V and the unimodal features of the audio F A .

[0081] In the nth cross-modal collaboration layer, the cross-modal interaction branches of the audio guidance module and the visual guidance module are used to perform cross-modal interaction on the unimodal features of video and audio, respectively, to obtain enhanced audio features and video features, which are recorded as and The specific processing method is as follows:

[0082] ;

[0083] ;

[0084] Where n = 1,2,...,L c ; , ; , ; T n = T / 2 n-1 , T n The length of the time series of the output features of the nth cross-modal collaboration layer;

[0085] MHA is a multi-head attention module, which is calculated as:

[0086] ;

[0087] Among them, W Q ∈R D×D , W K ∈R D×D , W V ∈R D×D are all learnable parameters, Softmax( ) represents the Softmax activation function, and the superscript T Represents a matrix transpose operation.

[0088] In the nth cross-modal collaboration layer, the audio features and video features are masked through the temporal consistency branches of the audio guidance module and the visual guidance module respectively, that is, the temporal consistency is maintained, and the masked audio features and video features are obtained respectively, which are recorded as and The specific processing method is as follows:

[0089] ;

[0090] ;

[0091] Among them, σ is the Sigmoid function, W v ∈R D×1 and W a ∈R D×1 are all learnable parameters, g V and g A are the timing weights of video and audio respectively; , .

[0092] The audio features and video features output by each multimodal collaboration layer are concatenated to obtain multi-granularity temporal features, denoted as , Z n ∈R Tn×2D .

[0093] Generally speaking, coarse-grained time series features can represent longer fragments of continuous events, while fine-grained time series features can accurately represent the start and end time of events. n Send to the multi-granularity timing collaboration module.

[0094] The multi-granularity timing collaboration module includes a coarse-to-fine-granularity collaboration module and a fine-to-coarse-granularity collaboration module.

[0095] Specifically, in the coarse-to-fine-grained collaboration module, the k-th layer temporal feature Z k Considered as fine-grained features, the temporal features Z of the k+1th layer k+1 Considered as coarse-grained features, k = 1,2,...,L c -1. The coarse-grained feature Z k+1 Send it to a linear layer, and then pass it through the ReLU activation function to get the updated coarse-grained feature U k Then the updated coarse-grained feature U k and fine-grained features Z k Send it to a multi-head attention module to achieve temporal collaboration from coarse-grained to fine-grained, and obtain the updated fine-grained feature G k The specific calculation process is as follows:

[0096] ;

[0097] ;

[0098] Among them, W u ∈R Tk×Tk+1 is a learnable parameter, ReLU is the activation function, G Lc = Z Lc .

[0099] Specifically, in the coarse-grained collaborative module, the processed m-1th layer feature G m-1 Considered as fine-grained features, the m-th layer feature G m Considered as coarse-grained features, m = 2,...,L c . The fine-grained feature G m-1 After the maximum pooling layer with a pooling step size of 2, the features after the pooling operation are obtained , and then with the coarse-grained feature G m It is sent to the multi-head attention module to achieve temporal collaboration from fine-grained to coarse-grained, and the feature O after temporal bidirectional collaboration enhancement is obtained. m , the specific calculation process is as follows:

[0100] ;

[0101] ;

[0102] Among them, MaxPooling is a pooling operation, O 1 = G 1 , .

[0103] Specifically, the decoding module includes a classification head and a regression head; the classification head is used to predict the category of each audio-visual event; the regression head is used to predict the time of each audio-visual event, including the start time and end time of the event; the classification head is composed of three one-dimensional convolutional layers and a Sigmoid activation function; the regression head is composed of three one-dimensional convolutional layers and a ReLU activation function.

[0104] Example 2

[0105] like Figure 2 As shown, based on the audio-visual event localization system of the above-mentioned embodiment 1, the training method of the localization model specifically includes the following steps:

[0106] S1, obtain samples, build data sets, and perform training initialization settings.

[0107] The samples are audio and video, and the labels of the samples are the categories and times of the audiovisual events contained in the audio and video. The data set is divided into a training set and a validation set, and the initial best accuracy of the validation set is 0. Define the current number of training times as d, and initialize d=1; let the model parameter of the dth training be ε d ; Set the total number of training epochs to 40; Use the Adam optimizer for training and set the learning rate to 0.0001.

[0108] S2, using the unimodal feature encoding module to process the samples in the training set, namely the audio and video, to obtain the unimodal features of the video and the unimodal features of the audio.

[0109] Step S2 is specifically as follows:

[0110] S21, use the pre-trained two-stream I3D model to process the video and extract video features V, V∈R T×dv ; Use the pre-trained VGGish model to process the audio and extract the audio features A, A∈R T×da ; Where T is the maximum time series length, T=224, dv, da represent the feature dimensions at each timestamp. In this process, the length of the video may change, so the obtained feature vector is cropped or padded to the maximum time series length.

[0111] S22, the video feature V and the audio feature A are input into their own independent unimodal Transformer networks, and the self-attention mechanism is used to realize the temporal interaction within each modality to obtain the unimodal feature F of the processed video V and the unimodal features of the audio F A ; F V ∈R T×D , F A ∈R T×D , D represents the feature dimension at each timestamp.

[0112] S3, using the cross-modal consistency collaboration module, performs cross-modal interaction on the unimodal features of video and audio and maintains temporal consistency, splicing the video features and audio features output by each multimodal collaboration layer to obtain multi-granularity temporal features.

[0113] In step S3, the specific process of the nth cross-modal collaboration layer is as follows:

[0114] S31, the cross-modal interaction branch of the audio guidance module performs cross-modal interaction on the unimodal features of video and audio to obtain enhanced audio features :

[0115] ;

[0116] The cross-modal interaction branch of the visual guidance module performs cross-modal interaction on the unimodal features of video and audio to obtain enhanced video features. :

[0117] ;

[0118] Where n = 1,2,...,L c ; , ; , ; T n = T / 2 n-1 , Tn The length of the time series of the output features of the nth cross-modal collaboration layer;

[0119] MHA is a multi-head attention module, which is calculated as:

[0120] ;

[0121] Among them, W Q ∈R D×D , W K ∈R D×D , W V ∈R D×D are all learnable parameters, Softmax( ) represents the Softmax activation function, and the superscript T Represents a matrix transpose operation.

[0122] S32, the temporal consistency branch of the audio guidance module, adds the enhanced video features of another modality, namely the video modality After the multi-head self-attention mechanism, a linear layer and Sigmoid activation function are used to generate a temporal weight g V , and then use the weight g V Multiply the enhanced audio features , thus obtaining the masked audio features , the specific calculation method is as follows:

[0123] ;

[0124] ;

[0125] The temporal consistency branch of the video guidance module takes the enhanced audio features of the other modality, the audio modality After the multi-head self-attention mechanism, a linear layer and Sigmoid activation function are used to generate a temporal weight g A , and then use the weight g A Multiply by the enhanced video features , thus obtaining the masked video features , the specific calculation method is as follows:

[0126] ;

[0127] ;

[0128] Among them, σ is the Sigmoid function, W v ∈R D×1 and W a ∈R D×1 are all learnable parameters, g V and gA are the timing weights of video and audio respectively.

[0129] S33, concatenate the masked audio features and video features to obtain multi-granularity temporal features, denoted as .

[0130] S4, input the multi-granularity temporal features of each multimodal collaboration layer into the temporal multi-granularity collaboration module, use the coarse-to-fine-grained collaboration module to perform temporal collaboration from coarse-grained to fine-grained on the multi-granularity temporal features to obtain updated fine-grained features; use the fine-to-coarse-grained collaboration module to perform temporal collaboration from fine-grained to coarse-grained on the updated fine-grained features to obtain features enhanced by temporal bidirectional collaboration.

[0131] The specific process of step S4 is as follows:

[0132] S41, in the coarse-to-fine granularity collaboration module, the k-th layer temporal feature Z k Considered as fine-grained features, the temporal features Z of the k+1th layer k+1 Considered as coarse-grained features, k = 1,2,...,L c -1. The coarse-grained feature Z k+1 Send it to a linear layer, and then pass it through the ReLU activation function to get the updated coarse-grained feature U k Then the updated coarse-grained feature U k and fine-grained features Z k Send it to a multi-head attention module to achieve temporal collaboration from coarse-grained to fine-grained, and obtain the updated fine-grained feature G k The specific calculation process is as follows:

[0133] ;

[0134] ;

[0135] Among them, W u ∈R Tk×Tk+1 is a learnable parameter, ReLU is the activation function, G Lc = Z Lc .

[0136] S42, in the coarse-grained collaborative module, the processed m-1th layer feature G m-1 Considered as fine-grained features, the m-th layer feature G m Considered as coarse-grained features, m = 2,...,L c . The fine-grained feature G m-1 After the maximum pooling layer with a pooling step size of 2, the features after the pooling operation are obtained , and then with the coarse-grained feature Gm It is sent to the multi-head attention module to achieve temporal collaboration from fine-grained to coarse-grained, and the feature O after temporal bidirectional collaboration enhancement is obtained. m , the specific calculation process is as follows:

[0137] ;

[0138] ;

[0139] Among them, O 1 = G 1 .

[0140] S5, use the decoding module to decode the features enhanced by temporal bidirectional collaboration and predict the category Y of the audiovisual event P and time Q P , where Y P ∈R T× C , T represents the time series length of the feature, C represents the total number of categories, and C = 100 during training; the time of the audiovisual event includes the start time and end time of the event, Q P ∈R T× C ×2 ;

[0141] S6, calculate the category prediction loss value L reg And the time prediction loss value L cls :

[0142] ;

[0143] ;

[0144] Among them, Y y is the true category label of the event, Q y is the real time label of the event; FocalLoss is the cross entropy loss function, IOULoss is the intersection-over-union loss function;

[0145] Calculate the total loss L(ε) during model training d ):

[0146] ;

[0147] Where s(t) is an indicator function, indicating whether there is an audiovisual event at time t. If yes, s(t)=1, otherwise s(t)=1; t=1,2,...,T;

[0148] According to the total loss L(ε d ) is used for back propagation, and the Adam optimizer is used to update the model parameters to obtain the positioning model for this training.

[0149] S7, run the positioning model trained this time on the validation set. If the average accuracy on the validation set is greater than the existing best accuracy, update the best model parameters to the model parameters obtained from this training, and update the best accuracy to the average accuracy on the validation set; otherwise, do not update the best model parameters and best accuracy.

[0150] S8, determine whether the number of training times d reaches the total number of training epochs. If so, end the training process and output the optimal model parameters; otherwise, perform the next training in the same manner as steps S2-S7, and update the number of training times d=d+1.

[0151] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration, characterized by: It includes a positioning model for positioning audiovisual events; the positioning model includes: a single-modal feature encoding module, a cross-modal consistency collaboration module, a temporal multi-granularity collaboration module and a decoding module; The unimodal feature encoding module is used to extract features from the video and audio, and perform feature encoding to obtain unimodal features of the video and audio, respectively; Inputting the unimodal features of video and audio into a cross-modal consistency collaboration module; the cross-modal consistency collaboration module is composed of a plurality of stacked multimodal collaboration layers, each of which includes a visual guidance module and an audio guidance module, the visual guidance module and the audio guidance module have the same network structure, both of which are composed of a cross-modal interaction branch and a temporal consistency branch, and are used to perform cross-modal interaction on the unimodal features of video and audio and maintain temporal consistency, and the visual guidance module and the audio guidance module output video features and audio features respectively; The video features and audio features output by each multimodal collaboration layer are spliced ​​to obtain multi-granularity temporal features, and are input into a temporal multi-granularity collaboration module; the temporal multi-granularity collaboration module includes a coarse-to-fine-granularity collaboration module and a fine-to-coarse-granularity collaboration module, which are used to perform temporal collaboration from coarse-grained to fine-grained and from fine-grained to coarse-grained on the multi-granularity temporal features, and obtain features enhanced by temporal bidirectional collaboration; The features enhanced by the temporal bidirectional collaboration are input into a decoding module; the decoding module is used to predict the category and time of each audiovisual event in the audio and video.

2. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 1 is characterized in that: The cross-modal consistency collaboration module is composed of L c The multimodal collaboration layers are stacked together; between two adjacent multimodal collaboration layers, the output of the previous multimodal collaboration layer is used as the input of the next multimodal collaboration layer; the input of the first multimodal collaboration layer is the output of the unimodal feature encoding module, that is, the unimodal feature F of the video V and the unimodal features of the audio F A ; In the nth cross-modal collaboration layer, the enhanced audio features and video features are obtained through the cross-modal interaction branches of the audio guidance module and the visual guidance module, respectively, which are recorded as and The specific processing method is as follows: ； ； Where n = 1,2,...,L c ; , ; MHA is a multi-head attention module, which is calculated as: ； Among them, W Q , W K , W V are all learnable parameters, Softmax( ) represents the Softmax activation function, and the superscript T Represents a matrix transpose operation; In the nth cross-modal collaboration layer, the masked audio features and video features are obtained through the temporal consistency branches of the audio guidance module and the visual guidance module, respectively, and are recorded as and The specific processing method is as follows: ； ； Among them, σ is the Sigmoid function, W v and W a is a learnable parameter, g V and g A are the timing weights of video and audio respectively; The masked audio features and video features output by each multimodal collaboration layer are concatenated to obtain multi-granularity temporal features, denoted as .

3. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 2 is characterized in that: F V ∈R T×D , F A ∈R T×D ; , ; Where T is the time series length of the unimodal feature; T n is the time series length of the output feature of the nth cross-modal collaboration layer, T n = T / 2 n-1 ; D is the feature dimension; R represents the data shape.

4. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 2 is characterized in that: The multi-granularity temporal features of each multimodal collaboration layer are input into the temporal multi-granularity collaboration module. The specific processing method of the temporal multi-granularity collaboration module is as follows: First, the coarse-to-fine-grained collaboration module is used to perform time series collaboration from coarse-grained to fine-grained multi-grained time series features to obtain the updated fine-grained feature G k , k = 1,2,...,L c -1, the specific processing method is as follows: ； ； Among them, U k is the updated coarse-grained feature, W u is a learnable parameter, ReLU is the activation function, G Lc = Z Lc ; Then, the fine-to-coarse-grained collaboration module is used to perform temporal collaboration from fine-grained to coarse-grained to obtain the enhanced temporal bidirectional collaboration feature O. m , m = 2,...,L c The specific processing method is as follows: ； ； in, is the feature after pooling operation, MaxPooling is the pooling operation, O 1 = G 1 .

5. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 1, characterized in that: In the unimodal feature encoding module, the video and audio features are first extracted to obtain the video feature V and the audio feature A. Then, the video feature V and the audio feature A are input into their respective unimodal Transformer networks for unimodal feature encoding to obtain the video unimodal feature F and the audio feature F. V and the unimodal features of the audio F A .

6. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 5, characterized in that: Use the pre-trained two-stream I3D model to process the video and extract the video feature V; use the pre-trained VGGish model to process the audio and extract the audio feature A.

7. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 1 is characterized in that: The decoding module includes a classification head and a regression head; the classification head is used to predict the category of each audio-visual event; the regression head is used to predict the time of each audio-visual event, including the start time and end time of the event.

8. The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to claim 7, characterized in that: The classification head consists of three one-dimensional convolutional layers and a Sigmoid activation function; the regression head consists of three one-dimensional convolutional layers and a ReLU activation function.

9. An audiovisual event localization method based on cross-modal consistency and temporal multi-granularity collaboration, characterized in that: The audiovisual event localization system based on cross-modal consistency and temporal multi-granularity collaboration according to any one of claims 1 to 8 above, wherein the training method of the localization model specifically comprises the following steps: S1, obtain samples, build data sets, and perform training initialization settings; The sample is audio and video, and the label of the sample is the category and time of each audiovisual event contained in the audio and video; define the current number of training times as d, and let the model parameter of the dth training be ε d ; Set the total number of training times to epoch; divide the data set into a training set and a validation set, and set the initial best accuracy of the validation set to 0; S2, using the unimodal feature encoding module to process the samples in the training set, namely the audio and video, to obtain the unimodal features of the video and the unimodal features of the audio; S3, using the cross-modal consistency collaboration module, performs cross-modal interaction on the unimodal features of video and audio and maintains temporal consistency, splicing the video features and audio features output by each multimodal collaboration layer to obtain multi-granularity temporal features; S4, input the multi-granularity time series features of each multimodal collaboration layer into the time series multi-granularity collaboration module, use the coarse-to-fine-grained collaboration module to perform time series collaboration from coarse-grained to fine-grained on the multi-granularity time series features to obtain updated fine-grained features; use the fine-to-coarse-grained collaboration module to perform time series collaboration from fine-grained to coarse-grained on the updated fine-grained features to obtain features enhanced by time series bidirectional collaboration; S5, use the decoding module to decode the features enhanced by temporal bidirectional collaboration and predict the category Y of the audiovisual event P and time Q P , where Y P ∈R T× C , T represents the time series length of the feature, C represents the total number of categories; the time of the audiovisual event includes the start time and end time of the event, Q P ∈R T× C ×2 ; S6, calculate the category prediction loss value L reg And the time prediction loss value L cls : ； ； Among them, Y y is the true category label of the event, Q y is the real time label of the event; FocalLoss is the cross entropy loss function, IOULoss is the intersection-over-union loss function; Calculate the total loss L(ε) during model training d ): ； Where s(t) is an indicator function, indicating whether there is an audiovisual event at time t. If yes, s(t)=1, otherwise s(t)=0; t=1,2,...,T; According to the total loss L(ε d ) performs back propagation and updates the model parameters to obtain the positioning model for this training; S7, running the positioning model trained this time on the validation set. If the average accuracy on the validation set is greater than the existing best accuracy, the best model parameters are updated to the model parameters obtained from this training, and the best accuracy is updated to the average accuracy on the validation set; otherwise, the best model parameters and best accuracy are not updated. S8, determine whether the number of training times d reaches the total number of training epochs. If so, end the training process and output the optimal model parameters; otherwise, perform the next training in the same manner as steps S2-S7, and update the number of training times d=d+1.

10. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the audiovisual event localization method based on cross-modal consistency and temporal multi-granularity collaboration as described in claim 9.

Images