Audio-visual source positioning method based on visual enhancement and audio-visual bidirectional semantic guidance

Abstract

The invention belongs to the technical field of audio-visual fusion, particularly relates to an audio-visual source positioning method based on visual enhancement and audio-visual two-way semantic guidance, and solves the problems that only one spatial position is extracted to represent audio-visual similarity to cause neglect of rich visual signals, and in addition, when the audio-visual connection is enhanced, the visual effect is not influenced. The method comprises the following steps: extracting features of an input image and an audio which are naturally matched in time; enhancing the visual key area of the input image to obtain a new input image; constraining the key area by utilizing comparison; extracting features again to obtain new image features; enhanced image features and enhanced audio features are obtained through the audio-visual bidirectional semantic guidance module; performing audio-visual constraint on the two types of enhanced features; and calculating audiovisual attention to realize visual localization of the sound source. According to the method, the visual signal with richer information is enhanced by enhancing the visual key area, and the mutual perception of the image mode and the audio mode is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of audio-visual fusion, and in particular to an audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance. Background Art

[0002] The goal of audiovisual localization is to highlight visual areas or locations of interest by referencing the input audio.

[0003] In most cases, sound can be visually associated with its source, because audio and vision often appear at the same time. Visual and auditory modalities are the two main ways for humans to obtain external information. Multimodal fusion integrates these two types of information to provide more comprehensive and rich data for positioning, improving the accuracy, robustness and reliability of positioning. When it comes to sound source localization, traditional methods usually rely on a single perception modality, such as using a microphone array to obtain sound information and determine the location of the sound source by the time difference of sound wave propagation. These methods mainly focus on the processing and analysis of sound information, but due to the complexity of the real-world environment, there are factors such as mixing, environmental noise, and echo. At this time, a single modality alone cannot accurately locate and identify. The sound source localization based on audio-visual fusion combines a variety of data information, uses different sensory modalities for cross-validation and supplementation, has a stronger ability to adapt to the environment, and can provide more reliable sound source localization information in complex environments.

[0004] Early work to address this problem mainly relied on statistical modeling of cross-modal relationships, such as mutual information and canonical correlation analysis. However, as shallow models, these methods only show advantages in simple audio-visual scenes. In recent years, deep neural networks have brought this task to a new stage. Most methods adopt the audio-visual correspondence learning framework, which learns to maximize the similarity between sound features and the visual embedding of the same sound-producing object, that is, measuring audio-visual similarity to reflect the associated area of ​​sound on the image, and using audio-visual correspondence or time synchronization as a supervisory signal to learn feature representation. In the simple method of audio-visual localization, learning at least one position association between audio and image to align audio and visual space is simple and effective, but vision often carries richer information. The method of selecting only one pixel position in the spatial dimension of audio-visual association to represent the audio-visual similarity value is prone to lose more effective information. In addition, when strengthening the audio-visual connection, most studies often focus on using audio to guide images, and the processing of audio features is often only simple feature extraction, and some studies also ignore the importance of visual semantics and auditory semantics for scene understanding. Summary of the invention

[0005] In order to overcome the technical defects of only extracting one spatial position to represent audio-visual similarity, which will cause the neglect of rich visual signals, and ignoring the importance of visual semantics and auditory semantics to scene understanding when strengthening the audio-visual connection, the present invention provides an audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance.

[0006] The present invention provides an audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance, the steps are:

[0007] S1, extracting features of input images and audio that naturally match in time through the backbone model;

[0008] S2. Enhance the visual key area of ​​the input image to obtain a new input image. The operation process is:

[0009] First calculate the audio-visual similarity Where B refers to the batch size taken by the backbone model for each training, H represents the height of the image feature, W represents the width of the image feature, and the audiovisual similarity sim is:

[0010]

[0011] Where <·> represents Einstein summation and ||·||2 is the L2 norm; the attention map of audio-visual matching is then obtained by extracting the diagonal values ​​of the first two dimensions of sim The attention map is upsampled to the same size as the input image and expanded to three dimensions, and then the attention map is normalized to -1 to 1;

[0012]

[0013] in max(·) is the maximum value, min(·) is the minimum value, and the attention map is the same size as the input image; then The positions greater than 0 in the image are set to 1, and the rest are set to 0, which is multiplied by the input image and finally added to the input image to obtain a new input image with enhanced visual key areas.

[0014] S3, using contrast to constrain the visual key areas of the new input image;

[0015] S4, the new input image reuses the new pre-trained model to extract features to obtain new image features;

[0016] S5, after the audio-visual bidirectional semantic guidance module, the enhanced new image features and enhanced audio features are obtained;

[0017] First, audio semantics is used to guide new image features. The operation process is as follows:

[0018]

[0019] Where δ is the ReLU function, AP(·) is the two-dimensional average pooling of H and W dimensions, and It is a linear layer that transforms the image semantics V s As the query Q i , audio semantics A s As key K i With value V i , using Transformer encoder to learn cross-modal semantic perception:

[0020] Q i =V s W i Q ,K i =A s W i K ,V i =A s W i V

[0021] out=Concat(head1,...,head n )W o ,

[0022] O F =LN[V s +D[out]],U(V,S)=LN[O F +φ(O F W 1 )W 2 ]

[0023] Where d is the query Q i , key K i , value V i The same dimension of , softmax is normalization, LN[·] is layer normalization, D[·] is Dropout, φ is ReLU and Dropout, W i Q , W i K , W i V , W 0 , W 1 With W 2 It is a linear layer. The final image semantics U(V,S) is multiplied by the input new image features to obtain enhanced new image features.

[0024]

[0025] in is element-wise multiplication;

[0026] At the same time, the input new image features are deeply refined to obtain more detailed semantic features, and then fused with the input audio features. The operation process is as follows:

[0027]

[0028] in σ is the Sigmoid function, BN[·] is batch normalization, and f 1×1 (·) is a 1×1 convolution, and The Sobel operator is used to perform 3×3 convolution on the horizontal space and vertical space respectively, f 3×3 (·) is a 3×3 convolution, ψ is ReLU and batch normalization, and ⊙ is concatenation; finally, fold the spatial dimensions to the same dimension. The image semantics is extracted using linear transformation and then reduced to a semantic dimension consistent with the audio, and the enhanced audio features are obtained using skip connections. The operation process is:

[0029]

[0030] in is a linear layer, sq[·] is a dimensionality reduction operation;

[0031] S6. Performing audio-visual constraints on the enhanced new image features and the enhanced audio features;

[0032] S7. Calculate audio-visual attention to achieve visual localization of the sound source.

[0033] Preferably, the backbone model in step S1 is ResNet18, and the image features are extracted from the input RGB image of size 224×224 using ResNet18 pre-trained on ImageNet. Perform short-time Fourier transform on the audio to obtain a spectrogram, and use a non-pre-trained ResNet18 to extract audio features from the input spectrogram The parameters of the non-pretrained ResNet18 are random, B refers to the batch size taken for each training, H and W represent the height and width of the image features respectively, and d represents the unified channel dimension of the image features and audio features.

[0034] Preferably, step S3 includes: obtaining an audiovisual similarity matrix S through the audiovisual similarity sim va Similarity matrix with audio and video The operation process is:

[0035]

[0036] S av =S va T ,

[0037] Where τ is a hyperparameter, max H,W It takes the maximum value of H dimension and W dimension, and uses the cross entropy loss in the same batch to constrain the similarity. i For example, i∈[1,B], corresponding to image V i is a positive sample, and is regarded as the rest of the images V in the same batch j (j≠i) is a negative sample, and the operation process is:

[0038]

[0039] Where L CE (·) is the cross entropy loss function, Represents the similarity value between an image i and all audios in the same batch, represents the similarity value between an audio i and all images in the same batch, are sample labels [1,2,...,B].

[0040] Preferably, in step S4, the new input image V′ uses the new pre-trained ResNet18 to extract new image features

[0041] Preferably, step S6 includes:

[0042] The audio-visual constraints include the false negative sample suppression strategy and the intra-batch comparison constraint. The intra-batch comparison constraint is the same as the comparison constraint operation in step S3, except that the audio-visual similarity sim is replaced by sim se , get L s :

[0043]

[0044] In the false negative sample suppression strategy, the enhanced new image features are first average pooled and then the self-similarity matrix is ​​calculated. The self-similarity matrix of the enhanced audio features is calculated, and then the two self-similarity matrices are constrained by the distance loss with the audio-visual similarity matrix. The operation process is as follows:

[0045]

[0046] Preferably, in step S7, calculating the audiovisual attention to realize the visual positioning of the sound source includes: first folding the enhanced new image features into B×C×HW, increasing the dimension of the enhanced audio features to B×1×C, performing matrix multiplication on the enhanced new image features and the enhanced audio features to obtain an attention map of B×1×HW, and then re-expanding the last dimension of the attention map to obtain the final audiovisual positioning map

[0047] Compared with the prior art, the technical solution provided by the present invention has the following technical effects: by using the method of visual key area enhancement, by mining the distribution of early audiovisual attention on the image, it plays a role in providing a priori knowledge of the location of a pseudo sound source, and as the training deepens, it provides a more accurate candidate region of interest for the image; this will obviously make the information of each picture richer, which is beneficial to the final image description; in view of the neglect of semantic information and enhanced audio in current research, an audio-visual bidirectional semantic guidance module is proposed, in which the Transformer encoder structure is used to assist the image feature fusion audio semantics, extract the deep spatial information of the image to condense the semantic guidance audio features, and promote the multimodal fusion and the semantic information consistent between the learning modalities. The present invention strengthens the visual signal with richer information by enhancing the visual key area, plays the role of emphasizing the vocal area to suppress the non-vocal background, promotes the multimodal fusion and the semantic information consistent between the learning modalities through the audio-visual bidirectional semantic guidance module, and enhances the mutual perception of the image modality and the audio modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is a structural schematic diagram of an audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance described in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0053] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] In one embodiment, Figure 1 As shown in FIG. 1 , the audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance includes the following steps:

[0055] S1, extracting features of input images and audio that naturally match in time through the backbone model;

[0056] First, we obtain audio-visual pairs, extract image frames and 3 seconds of audio around the frame from the video. There are 10,000 pairs of image data and audio data in the training set and 250 pairs in the test set. Then, we preprocess the image and audio, set the RGB image resolution to 224×224, perform short-time Fourier transform on the 3-second audio, and calculate the logarithmic spectrum with a size of 257×276. The backbone model is ResNet18, and ResNet18 pre-trained on ImageNet is used to extract image features from the input RGB image of size 224×224. Perform short-time Fourier transform on the audio to obtain a spectrogram, and use a non-pre-trained ResNet18 to extract audio features from the input spectrogram The parameters of the non-pretrained ResNet18 are random, B refers to the batch size taken for each training, H and W represent the height and width of the image features respectively, and d represents the unified channel dimension of the image features and audio features;

[0057] S2. Enhance the visual key area of ​​the input image to obtain a new input image. The operation process is:

[0058] First calculate the audio-visual similarity The operation process is:

[0059]

[0060] Where <·> represents Einstein summation and ||·||2 is the L2 norm; the attention map of audio-visual matching is then obtained by extracting the diagonal values ​​of the first two dimensions of sim The attention map is upsampled to the same size as the input image and expanded to three dimensions, and then the attention map is normalized to -1 to 1;

[0061]

[0062] in max(·) is the maximum value, min(·) is the minimum value, and the attention map is the same size as the input image; then The positions greater than 0 in the image are set to 1, and the rest are set to 0, which is multiplied by the input image and finally added to the input image to obtain a new input image with enhanced visual key areas.

[0063] S3, using contrast to constrain the visual key area of ​​the new input image; obtaining the audio-visual similarity matrix S through the audio-visual similarity sim va Similarity matrix with audio and video The operation process is:

[0064]

[0065] S av =S va T ,

[0066] Where τ is a hyperparameter, max H,W It takes the maximum value of H dimension and W dimension, and uses the cross entropy loss in the same batch to constrain the similarity. i For example, i∈[1,B], corresponding to image V i is a positive sample, and is regarded as the rest of the images V in the same batch j (j≠i) is a negative sample, and the operation process is:

[0067]

[0068] Where n is the number of samples, Represents the similarity value between an image i and all audios in the same batch, represents the similarity value between an audio i and all images in the same batch, is the sample label [1,2,...,B];

[0069] S4, the new input image V′ uses the new pre-trained ResNet18 to extract new image features

[0070] S5, after the audio-visual bidirectional semantic guidance module, the enhanced new image features and enhanced audio features are obtained;

[0071] First, audio semantics is used to guide new image features. The operation process is as follows:

[0072]

[0073] Where δ is the ReLU function, AP(·) is the two-dimensional average pooling of H and W dimensions, and It is a linear layer that transforms the image semantics Vs As the query Q i , audio semantics A s As key K i With value V i , using Transformer encoder to learn cross-modal semantic perception:

[0074]

[0075] Where d is the query Q i , key K i , value V i The same dimension of , softmax is normalization, LN[·] is layer normalization, D[·] is Dropout, φ is ReLU and Dropout, W i Q , W i K , W i V , W 0 , W 1 With W 2 It is a linear layer. The final image semantics U(V,S) is multiplied by the input new image features to obtain enhanced new image features.

[0076]

[0077] in is element-wise multiplication;

[0078] At the same time, the input new image features are deeply refined to obtain more detailed semantic features, and then fused with the input audio features. The operation process is as follows:

[0079]

[0080] in is the Sigmoid function, BN[·] is batch normalization, and f 1×1 (·) is a 1×1 convolution, and The Sobel operator is used to perform 3×3 convolution on the horizontal space and vertical space respectively, f 3 × 3 (·) is a 3×3 convolution, ψ is ReLU and batch normalization, and ⊙ is concatenation; finally, fold the spatial dimensions to the same dimension. The image semantics is extracted using linear transformation and then reduced to a semantic dimension consistent with the audio, and the enhanced audio features are obtained using skip connections. The operation process is:

[0081]

[0082] in is a linear layer, sq[·] is a dimensionality reduction operation;

[0083] S6. Performing audio-visual constraints on the enhanced new image features and the enhanced audio features;

[0084] First, use contrast to constrain the key areas:

[0085] The audio-visual similarity matrix S is obtained by the audio-visual similarity sim va Similarity matrix with audio and video The operation process is:

[0086]

[0087] S av =S va T ,

[0088] Where τ is a hyperparameter, max H , w is the maximum value of H dimension and W dimension, and the similarity is constrained by cross entropy loss within the same batch, with audio A i For example, i∈[1, B], corresponding to image V i is a positive sample, and is regarded as the rest of the images V in the same batch j (j≠i) is a negative sample, and the operation process is:

[0089]

[0090] Where L CE (·) is the cross entropy loss function, Represents the similarity value between an image i and all audios in the same batch, represents the similarity value between an audio i and all images in the same batch, is the sample label [1,2,...,B];

[0091] Secondly, the audio-visual constraints include the false negative sample suppression strategy and the intra-batch comparison constraint. The intra-batch comparison constraint is the same as the comparison constraint operation in step S3, except that the audio-visual similarity sim is replaced by sim se , get L s :

[0092]

[0093] In the false negative sample suppression strategy, the enhanced new image features are first average pooled and then the self-similarity matrix is ​​calculated. The self-similarity matrix of the enhanced audio features is calculated, and then the two self-similarity matrices are constrained by the distance loss with the audio-visual similarity matrix. The operation process is as follows:

[0094]

[0095] The final loss is: L total =L key +L s +L f ;

[0096] S7. Calculating audio-visual attention to realize visual positioning of the sound source includes: first folding the enhanced new image features into B×C×HW, increasing the dimension of the enhanced audio features to B×1×C, performing matrix multiplication on the enhanced new image features and the enhanced audio features to obtain an attention map of B×1×HW, and then re-expanding the last dimension of the attention map to obtain the final audio-visual positioning map

[0097] The audio-visual source localization fusion network based on visual enhancement and audio-visual bidirectional semantic guidance described in the present invention is trained: the network parameters are updated by the Adam optimizer, and the learning rate is 0.0001. The present invention builds and trains the network based on the Pytorch deep learning framework, and the hardware environment for model training is NVIDIA TITAN XP 12GB. The audio-visual source localization fusion network based on visual enhancement and audio-visual bidirectional semantic guidance described in the present invention is tested again: 250 pairs of images and audio are input into the audio-visual source localization fusion network, and finally 250 positioning images are obtained.

[0098] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. An audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance, characterized in that: The steps are: S1, extracting features of input images and audio that naturally match in time through the backbone model; S2. Enhance the visual key area of ​​the input image to obtain a new input image. The operation process is: First calculate the audio-visual similarity Where B refers to the batch size taken by the backbone model for each training, H represents the height of the image feature, W represents the width of the image feature, and the audiovisual similarity sim is: Where <·> represents Einstein summation and ||·||2 is the L2 norm; the attention map of audio-visual matching is then obtained by extracting the diagonal values ​​of the first two dimensions of sim The attention map is upsampled to the same size as the input image and expanded to three dimensions, and then the attention map is normalized to -1 to 1; in max(·) is the maximum value, min(·) is the minimum value, and the attention map is the same size as the input image; then The positions greater than 0 in the image are set to 1, and the rest are set to 0, which is multiplied by the input image and finally added to the input image to obtain a new input image with enhanced visual key areas. S3, using contrast to constrain the visual key areas of the new input image; S4, the new input image reuses the new pre-trained model to extract features to obtain new image features; S5, after the audio-visual bidirectional semantic guidance module, the enhanced new image features and enhanced audio features are obtained; First, audio semantics is used to guide new image features. The operation process is as follows: Where δ is the ReLU function, AP(·) is the two-dimensional average pooling of H and W dimensions, and It is a linear layer that transforms the image semantics V s As the query Q i , audio semantics A s As the key Ki and the value Vi, use the Transformer encoder to learn cross-modal semantic perception: Where d is the query Q i , key K i , value V i The same dimension of , softmax is normalization, LN[·] is layer normalization, D[·] is Dropout, φ is ReLU and Dropout, W i Q , W i K , W i V , W 0 , W 1 With W 2 It is a linear layer. The final image semantics U(V, S) is multiplied by the input new image features to obtain enhanced new image features. in is element-wise multiplication; At the same time, the input new image features are refined in deep spatial features and then fused with the input audio features. The operation process is as follows: in σ is the Sigmoid function, BN[·] is batch normalization, and f 1×1 (·) is a 1×1 convolution, and The Sobel operator is used to perform 3×3 convolution on the horizontal space and vertical space respectively, f 3×3 (·) is a 3×3 convolution, ψ is ReLU and batch normalization, and ⊙ is concatenation; finally, fold the spatial dimensions to the same dimension. The image semantics is extracted using linear transformation and then reduced to a semantic dimension consistent with the audio, and the enhanced audio features are obtained using skip connections. The operation process is: in is a linear layer, sq[·] is a dimensionality reduction operation; S6. Performing audio-visual constraints on the enhanced new image features and the enhanced audio features; S7. Calculate audio-visual attention to achieve visual localization of the sound source.

2. The audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance according to claim 1 is characterized in that: The backbone model in step S1 is ResNet18, which uses ImageNet pre-trained ResNet18 to extract image features from an input RGB image of size 224×224. Perform short-time Fourier transform on the audio to obtain a spectrogram, and use a non-pre-trained ResNet18 to extract audio features from the input spectrogram The parameters of the non-pretrained ResNet18 are random, B refers to the batch size taken for each training, H and W represent the height and width of the image features respectively, and d represents the unified channel dimension of the image features and audio features.

3. The audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance according to claim 1 is characterized in that: Step S3 includes: The audio-visual similarity matrix S is obtained by the audio-visual similarity sim va Similarity matrix with audio and video The operation process is: S av =S va T , Where τ is a hyperparameter, max H,W It takes the maximum value of H dimension and W dimension, and uses the cross entropy loss in the same batch to constrain the similarity. i For example, i∈[1,B], corresponding to image V i is a positive sample, and is regarded as the rest of the images V in the same batch j (j≠i) is a negative sample, and the operation process is: Where L CE (·) is the cross entropy loss function, Represents the similarity value between an image i and all audios in the same batch, represents the similarity value between an audio i and all images in the same batch, are sample labels [1,2,...,B].

4. The audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance according to claim 1, characterized in that: In step S4, the new input image V′ uses the new pre-trained ResNet18 to extract new image features 5. The audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance according to claim 1, characterized in that: Step S6 includes: The audio-visual constraints include the false negative sample suppression strategy and the intra-batch comparison constraint. The intra-batch comparison constraint is the same as the comparison constraint operation in step S3, except that the audio-visual similarity sim is replaced by sim se , get L s : In the false negative sample suppression strategy, the enhanced new image features are first average-pooled and then the self-similarity matrix is ​​calculated. The self-similarity matrix of the enhanced audio features is calculated, and then the two self-similarity matrices are constrained by the distance loss with the audio-visual similarity matrix. The operation process is as follows:

6. The audio-visual source localization method based on visual enhancement and audio-visual bidirectional semantic guidance according to claim 5, characterized in that: In step S7, the audiovisual attention is calculated to realize the visual positioning of the sound source, including: firstly, the enhanced new image features are folded into B×C×HW, the enhanced audio features are increased in dimension to B×1×C, the enhanced new image features and the enhanced audio features are subjected to matrix multiplication to obtain the attention map of B×1×HW, and then the last dimension of the attention map is re-expanded to obtain the final audiovisual positioning map

Images