A method and system for audio-visual event localization based on spatial channel feature fusion

Abstract

The present invention discloses an audio-visual event positioning method and system based on spatial channel feature fusion, comprising: extracting visual features and auditory features from target audio and video respectively; fusing the visual features and auditory features in space and channels to obtain spatial channel features; wherein the visual features and auditory features are respectively subjected to linear transformation, and the visual features and auditory features after the linear transformation are subjected to matrix multiplication operation to obtain initial fusion features, and the initial fusion features are subjected to feature extraction in channel dimension and spatial dimension respectively to obtain spatial attention map and channel attention map, and spatial channel features are obtained according to the spatial attention map and channel attention map; audio-visual features are obtained according to the auditory features and spatial channel features, and the event and background recognition and event category classification are performed according to the audio-visual features. The audio-visual space and channel information are effectively extracted, and the accuracy of the audio-visual event recognition task is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio-visual event positioning, and in particular to an audio-visual event positioning method and system based on spatial channel feature fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Visual and audio signals usually coexist in natural environments. Scenes where audio and visual streams coexist in video clips are called audio-visual events (AVEs). For example, when you hear the roar of a train, you can imagine the image of a train speeding by; when you see a picture of a train running on the railway, you can imagine the roar of the train. Research on audio-visual bimodal deep learning in recent years has shown that the fusion of auditory and visual information will improve the system's performance in processing audio-visual tasks.

[0004] The task of localizing audiovisual events requires determining the visible and audible parts of a video clip. However, unconstrained videos usually contain various noises, such as wind, sounds outside the visual scene, or interfering objects in the vision that are not related to the sound, which increases the difficulty of localizing audiovisual activities. In addition, there is a wealth of complementary information in visual and audio signals. How to effectively mine and fuse complementary information in audio and video is also a problem that needs to be solved. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an audio-visual event localization method and system based on spatial channel feature fusion, and proposes a spatial channel feature fusion module that fuses audio-visual spatial features and channel features, so as to effectively extract audio-visual spatial and channel information and effectively improve the accuracy of audio-visual event recognition tasks.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an audiovisual event localization method based on spatial channel feature fusion, comprising:

[0008] Extract visual features and auditory features from the target audio and video respectively;

[0009] The visual features and the auditory features are subjected to spatial and channel feature fusion to obtain spatial channel features; wherein the visual features and the auditory features are subjected to linear transformations of different dimensions respectively, and the visual features and the auditory features after the linear transformation are subjected to matrix multiplication operations to obtain initial fusion features, and the initial fusion features are subjected to channel dimension and spatial dimension feature extraction respectively to obtain spatial attention maps and channel attention maps, and the spatial channel features are obtained according to the spatial attention maps and the channel attention maps;

[0010] The audio-visual features are obtained based on the auditory features and spatial channel features, and the events and backgrounds are identified and the event categories are classified based on the audio-visual features to achieve audio-visual event positioning.

[0011] As an optional implementation, the target audio and video is segmented into a number of non-overlapping segments, the visual content and the auditory content in each segment are extracted, and the visual features and the auditory features are extracted from the visual content and the auditory content respectively.

[0012] As an optional implementation, average pooling and softmax operations are performed on the initial fusion features in the channel dimension to obtain a spatial attention map.

[0013] As an optional implementation, average pooling and sigmoid operations are performed on the initial fusion features in the spatial dimension to obtain a channel attention map.

[0014] As an optional implementation, the process of obtaining spatial channel features according to the spatial attention map and the channel attention map includes:

[0015] Perform matrix multiplication of the spatial attention map and the visual feature to obtain the spatial attention feature;

[0016] Perform matrix multiplication of the channel attention map and the visual feature to obtain the channel attention feature;

[0017] The visual features are added to the channel attention features to obtain the channel features, the channel features are linearly transformed, and the linearly transformed channel features and the spatial attention features are added to obtain the spatial channel features.

[0018] As an optional implementation, the process of obtaining audiovisual features based on auditory features and spatial channel features includes: using a bidirectional LSTM network to extract features of the auditory features and spatial channel features respectively, and using a feature splicing method to obtain audiovisual features from the output features of the bidirectional LSTM network.

[0019] As an optional implementation, the process of identifying events and backgrounds and classifying event categories based on audio-visual features includes: determining that audio and video clips with prediction scores above a threshold contain audio-visual events, determining that audio and video clips with prediction scores below the threshold are background clips, and marking event categories in the audio and video clips of audio-visual events.

[0020] In a second aspect, the present invention provides an audiovisual event localization system based on spatial channel feature fusion, comprising:

[0021] A feature extraction module is configured to extract visual features and auditory features of the target audio and video respectively;

[0022] The spatial channel feature fusion module is configured to perform spatial and channel feature fusion on the visual features and the auditory features to obtain spatial channel features; wherein the visual features and the auditory features are respectively subjected to linear transformation, and the visual features and the auditory features after the linear transformation are subjected to matrix multiplication operation to obtain initial fusion features, and the initial fusion features are respectively subjected to channel dimension and spatial dimension feature extraction to obtain a spatial attention map and a channel attention map, and the spatial channel features are obtained according to the spatial attention map and the channel attention map;

[0023] The positioning module is configured to obtain audio-visual features based on auditory features and spatial channel features, identify events and backgrounds and classify event categories based on the audio-visual features, thereby realizing audio-visual event positioning.

[0024] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention proposes an audio-visual event localization method and system based on spatial channel feature fusion, and proposes a spatial channel feature fusion module that integrates audio-visual spatial features and channel features to effectively extract audio-visual spatial and channel information. Abandoning the complex fusion module, a simple feature splicing method is used for audio-visual event recognition, which effectively improves the accuracy of audio-visual event recognition tasks and can also well complete the sound source localization task.

[0028] The present invention proposes a method and system for audiovisual event localization based on spatial channel feature fusion, which mainly studies supervised and weakly supervised audiovisual event localization in unconstrained video clips and sound source localization in audiovisual events, and also marks the sound source position of audiovisual events in public datasets.

[0029] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1A flow chart of an audiovisual event localization method based on spatial channel feature fusion provided in Example 1 of the present invention;

[0032] Figure 2 This is a structural diagram of an audio-visual event localization network based on spatial channel feature fusion provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0037] Example 1

[0038] At present, the relevant research on audio-visual event localization includes: Tian Yapeng et al. first proposed audio-visual event localization, collected a new audio-visual event data set, and proposed a dual multimodal residual module to extract visual information and auditory information that passed through two LSTMs respectively, thereby realizing audio-visual event localization. In addition, in order to improve the accuracy of audio-visual event localization, Ramaswamy proposed a fusion module and segmented attention mechanism based on a bilinear model.

[0039] The sound source localization problem requires identifying which pixel or area in a video emits a sound. Early studies found that important sound location information is encoded in the synchronization of audio and video signals. Recent studies have shown that a simple cross-modal attention model can also be used to accurately locate the sound source in audio-visual classification task learning. Therefore, the audio-visual localization of this embodiment locates the sound source in the middle layer of the spatial channel feature fusion module through class activation mapping.

[0040] like Figure 1-2As shown, this embodiment proposes an audiovisual event localization method based on spatial channel feature fusion, including:

[0041] Extract visual features and auditory features from the target audio and video respectively;

[0042] The visual features and the auditory features are subjected to spatial and channel feature fusion to obtain spatial channel features; wherein the visual features and the auditory features are subjected to linear transformations of different dimensions respectively, and the visual features and the auditory features after the linear transformation are subjected to matrix multiplication operations to obtain initial fusion features, and the initial fusion features are subjected to channel dimension and spatial dimension feature extraction respectively to obtain spatial attention maps and channel attention maps, and the spatial channel features are obtained according to the spatial attention maps and the channel attention maps;

[0043] The audio-visual features are obtained based on the auditory features and spatial channel features, and the events and backgrounds are identified and the event categories are classified based on the audio-visual features.

[0044] In this embodiment, the target audio and video Where C represents the total number of event categories and the background; first, the target audio and video are divided into T non-overlapping segments Among them, V t and A t Represents visual content and auditory content respectively;

[0045] Use the trained convolutional neural network to extract visual features from visual content and auditory content respectively and auditory features Among them, d v represents the number of channels after the visual content is processed by the convolutional neural network, S represents the spatial dimension, and d a Indicates the number of channels after the auditory content is processed by the convolutional neural network.

[0046] In this embodiment, a spatial channel feature fusion module is designed to extract spatial information and channel information that fuses visual features and auditory features.

[0047] Existing attention models are generally space-based, that is, the spatial probability of re-weighting the feature map of the last transformation layer of the convolutional neural network that encodes the input image; however, since the convolutional neural network extracts information features by fusing spatial and channel information within the local receptive field, the spatial channel feature fusion module proposed in this embodiment introduces a channel attention mechanism based on the spatial attention mechanism, which helps to model the importance of features extracted by different convolution kernels.

[0048] Specifically, in the Spatial Attention section:

[0049] First, the visual features and auditory features Perform the corresponding linear transformation to obtain and

[0050] Then, the linearly transformed visual features are and auditory characteristics Perform matrix multiplication to obtain the initial fusion features

[0051]

[0052] Finally, Perform average pooling and softmax operations on the channel dimension to obtain the spatial attention map α, and combine the spatial attention map α with the visual features Perform matrix multiplication to obtain spatial attention features

[0053] In the channel attention part, it is similar to the spatial attention part:

[0054] First, the visual features and auditory features Perform the corresponding linear transformation to obtain and

[0055] Then, and Perform matrix multiplication to obtain the initial fusion features

[0056] The difference is that at this time Perform average pooling and sigmoid operations on the spatial dimension to obtain the channel attention map β, and combine the channel attention map β with the visual features Perform matrix multiplication to obtain channel attention features

[0057] At the same time, in order to reduce information loss, this embodiment uses the idea of ​​residual network to convert visual features into A direct channel is constructed between the channel attention feature and the spatial attention feature to obtain the required channel feature; the channel feature is linearly transformed to make it consistent with the dimension of the spatial attention feature, and then the linearly transformed channel feature and the spatial attention feature are added and averaged to obtain the spatial channel feature.

[0058] In this embodiment, the auditory feature and spatial channel characteristics They are sent into the bidirectional LSTM network respectively. Different from the existing various complex structures, this embodiment adopts a simple feature splicing method, and directly splices the features output by the bidirectional LSTM network to obtain audio-visual features, and obtains the time information of the audio-visual feature sequence.

[0059] In this embodiment, the audio-visual features are used to perform event background recognition and event classification. Specifically:

[0060] First, the audio-visual features are used to identify events and backgrounds through a fully connected layer. Clips with prediction scores higher than the threshold are considered to contain audio-visual events, while clips with scores lower than the threshold are considered to be background clips. The loss function for event background recognition is set to the binary cross entropy loss function Loss binary ;

[0061] Then, the audio-visual features are classified into event categories using the multi-classification cross entropy loss function Loss multi , annotating the predicted event categories in the segments predicted as audiovisual events;

[0062] The total loss function is: Loss = Loss binary +Loss multi .

[0063] This example uses the public dataset The Audio-Visual Event dataset to evaluate the above method. The dataset contains 4143 video clips and 28 categories, covering various audio-visual activities such as human speeches, car driving, airplane roars, animal sounds, etc. Each video clip lasts for 10 seconds, and each video clip is annotated at the second level. For weak supervision tasks, 178 unannotated noise samples are also introduced. The event classification of each video clip is predicted, and the overall accuracy of the two AVE tasks is used as the performance evaluation indicator.

[0064] Audiovisual event localization: Visual features of each 10-second video extracted using VGG-19 and VGGish and auditory features where d v The value is 512, S represents the spatial dimension H×W, H and W are both 7, d a The value is 128. It is worth noting that for the supervised audio-visual event localization task, event-related segment labels and event category labels are provided during training. In the weakly supervised approach, only video-level event category labels can be accessed, but the segment labels and event category labels of the video must also be predicted.

[0065] Table 1 shows the performance comparison between the method of this embodiment and various methods in supervised and weakly supervised event localization tasks. It can be seen that the prediction accuracy of the method of this embodiment reaches 76.02%, and reaches 70.5% in the weakly supervised audiovisual event localization task, both exceeding the various compared methods, which shows the effectiveness of the model.

[0066] Table 1 Performance comparison

[0067]

[0068] To evaluate the effectiveness of the spatial channel feature fusion module we proposed, two sets of ablation experiments were conducted. In the first set of experiments, the spatial channel feature fusion module was directly deleted and the audio-visual features were input into the bidirectional LSTM. The results are shown in Table 2. It can be seen that the network without the spatial channel feature fusion module has a significant decrease in the indicators of supervised and weakly supervised audio-visual event localization.

[0069] In the second set of experiments, the channel attention module was partially shielded so that the network only used the spatial attention module, but the network indicators using only the spatial attention module also dropped significantly. This verifies that the spatial channel feature fusion module can effectively model audio-visual features, thereby improving the accuracy of the AVE task.

[0070] Table 2 Ablation experiment results

[0071]

[0072]

[0073] Sound source localization in the middle frame of audio-visual events: Because the original audio-visual event dataset does not have the sound source position labeled, this embodiment uses labelImg to label the sound source position of the middle frame of the audio-visual event clip in the test set video, where the frame size is 224x224 pixels. Specifically, the pixels marked as the sound source position box are labeled as 1, and the rest are labeled as 0; the heat map α of the sound source location is obtained in the spatial channel feature fusion module.

[0074] This embodiment introduces a quantitative sound localization performance evaluation index, namely, consistent intersection over union (cIoU), and defines cIoU as:

[0075]

[0076] Where i represents the pixel index of the middle frame, τ represents the threshold value set to 0.01, A(τ) represents the set of all pixels on the heat map α whose value is greater than τ, and g i Represents a set of pixels whose labeled data is 1.

[0077] The numerator in this formula represents the relationship between all pixels on the heat map α with values ​​greater than τ and g i The sum of the products of the corresponding points of the pixel points with a value of 1, and the denominator is g i It represents the sum of pixels with labeled data of 1 and all pixels with values ​​greater than τ on the heat map α and g i The sum of the products of the corresponding points of the pixel points with a value of 0 (the pixels outside the annotation box are marked as 1 at this time). The cIoU value of the sound source localization performance is 0.343.

[0078] The scene where the audiovisual streams appear simultaneously in the video clip is called an audiovisual event. This embodiment mainly studies the supervised and weakly supervised audiovisual event localization in unconstrained video clips and the sound source localization in the audiovisual event. A spatial channel feature fusion module that fuses audiovisual spatial features and channel features is proposed to effectively extract and fuse the spatial and channel information of audiovisual features, abandon the complex fusion module, and use a simple feature splicing method to identify audiovisual events.

[0079] After improving the audio-visual event recognition module, this embodiment conducted experiments on the public dataset AVE, and also marked the sound source positions of the intermediate frames where the audio-visual events exist in the AVE dataset. It has been verified that the method of this embodiment can effectively improve the audio-visual event recognition task, and can also complete the sound source localization task well.

[0080] Example 2

[0081] This embodiment provides an audiovisual event localization system based on spatial channel feature fusion, including:

[0082] A feature extraction module is configured to extract visual features and auditory features of the target audio and video respectively;

[0083] The spatial channel feature fusion module is configured to perform spatial and channel feature fusion on the visual features and the auditory features to obtain spatial channel features; wherein the visual features and the auditory features are respectively subjected to linear transformations of different dimensions, and the visual features and the auditory features after the linear transformations are subjected to matrix multiplication operations to obtain initial fusion features, and the initial fusion features are respectively subjected to feature extraction of channel dimensions and spatial dimensions to obtain spatial attention maps and channel attention maps, and the spatial channel features are obtained according to the spatial attention maps and the channel attention maps;

[0084] The positioning module is configured to obtain audio-visual features based on auditory features and spatial channel features, and to identify events and backgrounds and classify event categories based on the audio-visual features.

[0085] It should be noted that the above modules correspond to the steps described in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0086] In further embodiments, there is also provided:

[0087] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0088] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0089] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0090] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in Example 1 is completed.

[0091] The method in Example 1 can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here.

[0092] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for audiovisual event localization based on spatial channel feature fusion, characterized in that: include: Extract visual features and auditory features from the target audio and video respectively; The visual features and the auditory features are subjected to spatial and channel feature fusion to obtain spatial channel features; wherein the visual features and the auditory features are subjected to linear transformations of different dimensions respectively, and the visual features and the auditory features after the linear transformation are subjected to matrix multiplication operations to obtain initial fusion features, and the initial fusion features are subjected to channel dimension and spatial dimension feature extraction respectively to obtain spatial attention maps and channel attention maps, and the spatial channel features are obtained according to the spatial attention maps and the channel attention maps; The audio-visual features are obtained based on the auditory features and spatial channel features, and the events and backgrounds are identified and the event categories are classified based on the audio-visual features.

2. The method for locating audiovisual events based on spatial channel feature fusion according to claim 1, characterized in that: The target audio and video are divided into several non-overlapping segments, the visual content and auditory content in each segment are extracted, and the visual features and auditory features of the visual content and auditory content are extracted respectively.

3. The audiovisual event localization method based on spatial channel feature fusion according to claim 1, characterized in that: The initial fusion features are averaged and softmaxed in the channel dimension to obtain a spatial attention map.

4. The method for locating audiovisual events based on spatial channel feature fusion according to claim 1, characterized in that: The initial fusion features are averaged and sigmoid operated in the spatial dimension to obtain the channel attention map.

5. The method for locating audiovisual events based on spatial channel feature fusion according to claim 1, characterized in that: The process of obtaining spatial channel features based on the spatial attention map and the channel attention map includes: Perform matrix multiplication of the spatial attention map and the visual feature to obtain the spatial attention feature; Perform matrix multiplication of the channel attention map and the visual feature to obtain the channel attention feature; The visual features are added to the channel attention features to obtain the channel features, the channel features are linearly transformed, and the linearly transformed channel features and the spatial attention features are added to obtain the spatial channel features.

6. The method for locating audiovisual events based on spatial channel feature fusion according to claim 1, characterized in that: The process of obtaining audio-visual features according to auditory features and spatial channel features includes: using a bidirectional LSTM network to extract features of the auditory features and spatial channel features respectively, and using a feature splicing method to obtain audio-visual features from the output features of the bidirectional LSTM network.

7. The method for locating audiovisual events based on spatial channel feature fusion according to claim 1, characterized in that: The process of identifying events and backgrounds and classifying event categories based on audio-visual features includes: judging that audio and video clips with prediction scores higher than a threshold value contain audio-visual events, judging that audio and video clips with prediction scores lower than the threshold value are background clips, and marking event categories in audio and video clips of audio-visual events.

8. An audio-visual event localization system based on spatial channel feature fusion, characterized in that: include: A feature extraction module is configured to extract visual features and auditory features of the target audio and video respectively; The spatial channel feature fusion module is configured to perform spatial and channel feature fusion on the visual features and the auditory features to obtain spatial channel features; wherein the visual features and the auditory features are respectively subjected to linear transformations of different dimensions, and the visual features and the auditory features after the linear transformations are subjected to matrix multiplication operations to obtain initial fusion features, and the initial fusion features are respectively subjected to feature extraction of channel dimensions and spatial dimensions to obtain spatial attention maps and channel attention maps, and the spatial channel features are obtained according to the spatial attention maps and the channel attention maps; The positioning module is configured to obtain audio-visual features based on auditory features and spatial channel features, and to identify events and backgrounds and classify event categories based on the audio-visual features.

9. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 7 is completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 7.

Images