An audio-video event detection method based on audio-visual multi-modal fusion
By employing a multimodal fusion approach in audio and video event detection, common and specific semantic information from different modalities is learned, solving the problem of temporal misalignment between modalities and achieving effective fusion of cross-modal features and improved accuracy of event detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI INST OF MICROSYSTEM & INFORMATION TECH CHINESE ACAD OF SCI
- Filing Date
- 2023-01-11
- Publication Date
- 2026-06-02
AI Technical Summary
Existing audio and video event detection methods struggle to effectively fuse across modalities when temporal modes are not aligned, especially when it is difficult to extract complementary information between heterogeneous modalities and when redundant information is not filtered out.
A method based on audiovisual multimodal fusion is adopted. Common semantic and specific semantic information of 2D visual features, 3D visual features and audio features are learned through the first modality common-private subspace and the second modality common-private subspace, respectively. Adaptive fusion is performed using a hybrid attention module, and finally event prediction is performed through attention pooling layer.
It enhances the neural network's ability to learn complementary information, reduces feature redundancy, aggregates unique event cues from each modality, and improves the accuracy and efficiency of event detection.
Smart Images
Figure CN116310945B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to an audio and video event detection method based on audio-visual multimodal fusion. Background Technology
[0002] The goal of video event detection is to identify information about events within a video and present it in a structured form. Specifically, in unedited video, it involves identifying the category of each event instance and locating the start and end times of each event. Event detection is a fundamental core technology in the field of artificial intelligence, widely applied to tasks such as event graph construction and video-related text summarization. For example, traditional event graph construction relies on textual information, while video-based event detection methods can enrich the data sources of the knowledge base.
[0003] Existing video event detection methods can be categorized into unimodal and multimodal methods based on the input modality. Unimodal detection typically uses only video or audio information as input, while multimodal detection uses both audio and video, leveraging audiovisual synesthesia for multimodal learning. The multi-source information provides richer supervision, thus improving detection performance. Furthermore, based on training methods, they can be divided into fully supervised and weakly supervised approaches. Fully supervised methods require labeling each frame of the training video with event categories or background (without any events occurring), which is time-consuming and labor-intensive. Weakly supervised methods only require identifying which events occurred in a video during training, without needing to label each frame. This means that the start and end times of each event cannot be obtained during training, nor can it be determined which modality the event occurred in—whether it's an audible, visible, or audiovisual event (audiovisual events exist in both the video and audio tracks, and may not be aligned temporally). Therefore, weakly supervised methods, while saving on dataset labeling costs, increase the difficulty of event detection.
[0004] In existing technologies, the HAN basic framework can detect audio and video events in audio-visual multimodal fusion, such as... Figure 1As shown, its inputs are 2D and 3D visual features extracted from video input and auditory features extracted from audio input. Its outputs are classification results for audible events, visible events, and audiovisual events at both the segment and video levels. The start and end times of each event can be derived from the classification result of each segment. This framework adaptively finds segments with event discrimination information through an attention module and fuses cross-modal features to obtain single-frame prediction results. Then, an attention pooling layer aggregates these single-frame prediction results to obtain video-level prediction results. For the visual branch input, this scheme only uses fully connected layers and stacking operations to simply fuse 2D and 3D features projected to the same subspace, but it does not consider the redundant information between these two types of features. Furthermore, although the attention mechanism can adaptively select segments that provide more event cues in the time and modality dimensions to predict event categories and achieve cross-modal feature fusion, the attention mechanism implements weight allocation, which cannot bridge the inherent differences between heterogeneous modalities. Especially when modalities are misaligned and contain different event semantics, complementary information between modalities is even more difficult to extract. Summary of the Invention
[0005] This invention provides an audio and video event detection method based on audio-visual multimodal fusion to solve the cross-modal fusion problem in audio and video event detection tasks when modal temporal sequences are not aligned.
[0006] The technical solution adopted by this invention to solve its technical problem is: to provide an audio and video event detection method based on audio-visual multimodal fusion, comprising the following steps:
[0007] Extract 2D visual features, 3D visual features, and audio features from the audio and video files to be detected;
[0008] The 2D visual features and 3D visual features are input into the first modality public-private subspace, and the first modality public-private subspace aggregates the 2D visual features and 3D visual features into a single visual feature;
[0009] The visual and audio features are input into the second modality public-private subspace to obtain visual and auditory representations;
[0010] The visual and auditory representations are input into the attention pooling layer to obtain event prediction results at the fragment level and video level.
[0011] The first modality shared-private subspace and the second modality shared-private subspace have the same structure, both including two parallel parameter-shared encoders and parameter-non-shared encoders, and a hybrid attention module connected to the output of the parameter-shared encoder and the parameter-non-shared encoder respectively.
[0012] The 2D visual features were extracted using a ResNet152 network.
[0013] The 3D visual features are extracted using a ResNet 3D network.
[0014] The audio features were extracted using the VGGish network.
[0015] The parameter-shared encoder in the first modal shared-private subspace projects the 2D visual features and 3D visual features into a subspace with the same semantics to obtain common semantic information between the 2D and 3D visual features. The parameter-non-shared encoder in the first modal shared-private subspace projects the 2D and 3D visual features into two independent subspaces to obtain specific semantic information between the 2D and 3D visual features. The hybrid attention module in the first modal shared-private subspace combines self-attention network and cross-attention network to adaptively fuse the common semantic information between the 2D and 3D visual features and the specific semantic information between the 2D and 3D visual features, aggregating the 2D and 3D visual features into a single visual feature.
[0016] The parameter-shared encoder in the second modality shared-private subspace projects visual and audio features into a subspace with the same semantics to obtain common semantic information between the visual and audio features. The non-parametric shared encoder in the second modality shared-private subspace projects the visual and audio features into two independent subspaces to obtain specific semantic information between the visual and audio features. The hybrid attention module in the second modality shared-private subspace combines self-attention networks and cross-attention networks to adaptively fuse the common semantic information between the visual and audio features and the specific semantic information between the visual and audio features, and obtains visual and auditory representations through a fully connected layer.
[0017] Beneficial effects
[0018] By employing the above-mentioned technical solution, this invention has the following advantages and positive effects compared with existing technologies: This invention treats 2D and 3D features as two independent feature paths, optimizing them at the feature level to achieve complementarity between the two modalities, filtering out redundant information, and providing aggregated visual features for the next stage of visual-auditory fusion. This invention projects different modalities to the same subspace using a parameter-sharing encoder to learn common semantic information of different modalities, and projects different modalities to different subspaces using a parameter-non-sharing encoder to learn modality-specific semantic information. This design strengthens the neural network's learning of complementary information, reduces feature redundancy, and aggregates unique event cues of each modality. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the basic framework of HAN in the prior art;
[0020] Figure 2 This is a general framework diagram of an embodiment of the present invention;
[0021] Figure 3 This is a flowchart of the audio and video event detection method based on audio-visual multimodal fusion according to an embodiment of the present invention;
[0022] Figure 4 This is a test set video – a diagram illustrating three different categories of events. Detailed Implementation
[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
[0024] The embodiments of the present invention relate to an audio and video event detection method based on audio-visual multimodal fusion, the method being based on... Figure 2The framework shown extracts 2D visual features from video input using a ResNet 152 network and 3D visual features from video input using a ResNet 3D network, resulting in 2D and 3D visual features. A VGGish network extracts audio features from audio input. The 2D and 3D visual features are input into a first-stage modality public-private subspace learning network to obtain aggregated visual features. These aggregated visual features, along with the extracted audio features, are then input into a second-stage modality public-private subspace learning network to update and obtain the final visual and auditory representations. These representations are then passed through attention pooling layers and fully connected layers to output segment-level and video-level event prediction results. During training, the framework can be optimized using objective functions such as orthogonal loss, adversarial loss, and metric learning. Figure 3 As shown, the specific steps include:
[0025] Step 1: Extract 2D visual features, 3D visual features, and audio features from the audio and video files to be detected. In this step, 2D visual features are extracted using a ResNet152 network, 3D visual features are extracted using a ResNet 3D network, and audio features are extracted using a VGGish network.
[0026] Step 2: Input the 2D visual features and 3D visual features into the first modality public-private subspace, where the first modality public-private subspace aggregates the 2D visual features and 3D visual features into a single visual feature.
[0027] The first modal public-private subspace in this step includes two parallel parameter-shared encoders and parameter-non-shared encoders, as well as a hybrid attention module connected to the outputs of the parameter-shared encoder and the parameter-non-shared encoder, respectively.
[0028] The parameter-shared encoder projects the 2D and 3D visual features into a subspace with the same semantics to obtain common semantic information between them. The parameter-non-shared encoder projects the 2D and 3D visual features into two independent subspaces to obtain specific semantic information, which is used to provide event cues in misaligned situations. The processing steps of the parameter-shared and parameter-non-shared encoders in this process can be represented by the following two expressions:
[0029] C 2d =E c (θ c ,F 2d ),C 3d =E c (θ c,F 3d )
[0030] S 2d =E s (θ 2d ,F 2d ),S 3d =E s (θ 3d ,F 3d )
[0031] Among them, E c and E s F represents the parameter-shared encoder and the parameter-non-shared encoder in this step, respectively. 2d and F 3d C represents the extracted 2D visual features and 3D visual features, respectively. 2d and C 3d S represents 2D and 3D features with the same semantics, respectively. 2d and S 3d These represent the specific semantic information of 2D visual features and the specific semantic information of 3D visual features, respectively.
[0032] The hybrid attention module in this step is the same as the hybrid attention module in the basic HAN framework. It adaptively fuses the common semantic information between the 2D and 3D visual features and the specific semantic information between the 2D and 3D visual features by combining self-attention networks and cross-attention networks, aggregating the 2D and 3D visual features into a single visual feature. This is represented by the following two expressions:
[0033]
[0034]
[0035] Where C and S represent the common semantic information between 2D visual features and 3D visual features, and the specific semantic information between 2D visual features and 3D visual features, respectively. t and S t φ represents the common semantic information between 2D visual features and 3D visual features at time t, and the specific semantic information between 2D visual features and 3D visual features, respectively. s and φ c These represent self-attention networks and cross-attention networks, respectively.
[0036] The common semantic information between the 2D and 3D visual features and the specific semantic information of the 2D and 3D visual features are input in parallel into the hybrid attention module, thereby aggregating the 2D and 3D visual features into a single visual feature, which can solve the long-term dependency between information in multiple frames.
[0037] Step 3: Input the fused visual and audio features into the second modality public-private subspace to obtain visual and auditory representations.
[0038] The second modal shared-private subspace in this step has the same structure as the first modal shared-private subspace, also including two parallel parameter-shared encoders and parameter-non-shared encoders, as well as a hybrid attention module connected to the outputs of the parameter-shared encoder and parameter-non-shared encoder respectively.
[0039] The parameter-shared encoder projects visual and audio features into a subspace with the same semantic meaning to obtain common semantic information between the visual and audio features. The non-parameter-shared encoder projects the visual and audio features into two independent subspaces to obtain specific semantic information of the visual and audio features, which is used to provide event cues in the case of misalignment. The processing procedures of the parameter-shared encoder and the parameter-non-parameter-shared encoder in this step can be represented by the following two expressions:
[0040] C a =E c ′(θ c ,F a ),C v =E c ′(θ c ,F v )
[0041] S a =E s ′(θ a ,F a ),S v =E s ′(θ v ,F v )
[0042] Among them, E c ′ and E s ′ represent the parameter-shared encoder and parameter-non-shared encoder in this step, respectively, F a and F v C represents audio features and visual features respectively. a and C v S represents audio features and visual features that have the same semantics, respectively. a and S v These represent the specific semantic information of audio features and the specific semantic information of visual features, respectively.
[0043] The hybrid attention module in this step adaptively combines self-attention and cross-attention networks to fuse the common semantic information between visual and audio features and the specific semantic information of visual and audio features. This fusion is then passed through a fully connected layer to obtain visual and auditory representations. The hybrid attention module in this step is the same as the one in step 2, except that its input consists of the common semantic information between visual and audio features and the specific semantic information of visual and audio features. After fusion, the final visual and auditory representations are obtained through a fully connected layer.
[0044] Step 4: Input the visual and auditory representations into the attention pooling layer to obtain event prediction results at the fragment level and video level.
[0045] The attention pooling layer in this step is the same as the attention pooling layer in the basic HAN framework. This attention pooling layer treats the multi-label video detection task under weak supervision as a multi-label-multi-example learning task. It uses visual and auditory representations, calculates the attention weights of the audio, video and audio-visual branches through a multi-layer neural network and the Softmax function, and aggregates the visual and auditory representations based on the weights to obtain the event prediction results at the segment level and video level.
[0046] For audio and video event detection tasks, Tian et al. proposed an LLP dataset containing 11,849 videos, each 10 seconds long, covering 25 categories. The dataset was divided into a training set of 10,000 videos, a validation set of 649 videos, and a test set of 1,200 videos. Following a weakly supervised approach, the training set videos were annotated only at the video level, while the test and validation sets were fully annotated down to the event category for each second. Figure 4 The annotation of a video in the test set is shown. It can be seen that the video contains three types of events: the dog and specch events appear simultaneously in both the audio and video tracks, thus belonging to audiovisual events; while clapping only appears in the audio track, therefore it is an audible event. It is worth noting that cross-modal events can be asynchronous in the audio and video; for example, a specch event may appear in the first 2 seconds of the video but last for 10 seconds in the audio. The method used in this implementation can accurately detect these events.
[0047] It is easy to see that the invention treats 2D and 3D features as two independent feature paths, optimizing them at the feature level to achieve complementarity between the two modalities, filtering out redundant information, and providing aggregated visual features for the next stage of visual-auditory fusion. This invention projects different modalities into the same subspace using a parameter-sharing encoder to learn common semantic information across modalities, and projects different modalities into different subspaces using a parameter-non-sharing encoder to learn modality-specific semantic information. This design enhances the neural network's ability to learn complementary information, reduces feature redundancy, and aggregates unique event cues from each modality.
Claims
1. A method for detecting audio and video events based on audiovisual multimodal fusion, characterized in that, Includes the following steps: Extract 2D visual features, 3D visual features, and audio features from the audio and video files to be detected; The 2D and 3D visual features are input into a first modal shared-private subspace, which aggregates the 2D and 3D visual features into a single visual feature. A parameter-shared encoder in the first modal shared-private subspace projects the 2D and 3D visual features into a subspace with the same semantics to obtain common semantic information between the 2D and 3D visual features. A parameter-non-shared encoder in the first modal shared-private subspace projects the 2D and 3D visual features into two independent subspaces to obtain specific semantic information of the 2D and 3D visual features. A hybrid attention module in the first modal shared-private subspace combines self-attention and cross-attention networks to adaptively fuse the common semantic information between the 2D and 3D visual features and the specific semantic information of the 2D and 3D visual features, aggregating the 2D and 3D visual features into a single visual feature. The visual and audio features are input into a second modality shared-private subspace to obtain visual and auditory representations. A parameter-shared encoder in the second modality shared-private subspace projects the visual and audio features into a subspace with the same semantic meaning to obtain common semantic information between the visual and audio features. A non-parametric shared encoder in the second modality shared-private subspace projects the visual and audio features into two independent subspaces to obtain specific semantic information of the visual and audio features. A hybrid attention module in the second modality shared-private subspace adaptively fuses the common semantic information between the visual and audio features and the specific semantic information of the visual and audio features by combining a self-attention network and a cross-attention network, and obtains visual and auditory representations through a fully connected layer. The visual and auditory representations are input into the attention pooling layer to obtain event prediction results at the fragment level and video level. The first modality shared-private subspace and the second modality shared-private subspace have the same structure, both including two parallel parameter-shared encoders and parameter-non-shared encoders, and a hybrid attention module connected to the output of the parameter-shared encoder and the parameter-non-shared encoder respectively.
2. The audio-visual event detection method based on audio-visual multimodal fusion according to claim 1, characterized in that, The 2D visual features were extracted using a ResNet152 network.
3. The audio-visual event detection method based on audio-visual multimodal fusion according to claim 1, characterized in that, The 3D visual features are extracted using a ResNet 3D network.
4. The audio-visual event detection method based on audio-visual multimodal fusion according to claim 1, characterized in that, The audio features were extracted using the VGGish network.