Sound source localization method based on transmembrane attention mechanism

Abstract

The invention discloses a sound source localization method based on a transmembrane attention mechanism, and the method comprises the steps: S1, collecting an audio signal, calculating the logarithmic Mel spectrum of the audio signal, and generating an MFCC audio feature; s2, receiving a video signal, and extracting frame-level video features of the video signal by using a convolutional neural network; s3, encoding the audio features and the video features, aligning modal features of the audio and the video through a contrast learning layer, optimizing modal sharing information and generating modal alignment features; s4, calculating a cross-modal attention weight: dynamically capturing the relevance between the audio features and the video features by using a cross-modal attention mechanism; and S5, screening candidate proposal regions with high confidence through the relevance between the audio features and the video features, weighting the candidate proposal regions, calculating the similarity between each candidate proposal region and the audio features, generating weighted region features, and outputting the sound source position and range. According to the invention, audio and video features can be fused in a transmembrane state, and sound source localization is accurately realized.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and multimodal data processing, and in particular to a sound source localization method based on a trans-membrane state attention mechanism. Background Art

[0002] Existing sound source localization technology faces many challenges in complex natural environments. First, background noise interference significantly affects the accuracy of sound source localization. In applications such as bird dynamic monitoring, environmental noise (such as wind, leaf friction, other animal sounds and human activity sounds) often overlaps with the target sound source (such as bird singing), increasing the difficulty of audio signal extraction and recognition. Traditional sound source localization methods mainly rely on audio signals and lack effective noise suppression mechanisms, which makes it difficult to accurately distinguish between target sound sources and noise, thus affecting the positioning effect.

[0003] Video information can provide important positioning clues in many applications, but traditional video processing methods are often interfered by information redundancy in complex scenes. Background and irrelevant objects in video images may take up too much computing resources and hinder the accurate recognition of targets. Although it is feasible to reduce redundant information by extracting target regions, traditional region selection methods often rely on manual settings or simple algorithms and lack the necessary accuracy and adaptability.

[0004] In addition, inaccurate alignment of audio and video modalities is also a key issue. Due to the difference in propagation speed of audio and video signals, as well as the different positions and angles of sensors, audio and video data are often not accurately synchronized, which brings difficulties to multimodal data fusion. Traditional modal fusion methods such as simple feature splicing are difficult to fully explore the deep correlation between audio and video data, thus affecting the information fusion effect, and further affecting the accuracy and robustness of sound source localization.

[0005] Therefore, in order to meet these challenges, a new method is urgently needed to more effectively integrate audio and video data, utilize their complementary advantages, and improve the accuracy of sound source localization and anti-interference ability. Summary of the invention

[0006] In order to solve the problems of low sound source localization accuracy and difficulty in fusing audio and video data in the prior art, the present invention proposes a sound source localization method based on a trans-membrane state attention mechanism.

[0007] The specific technical solution is as follows: A sound source localization method based on a transmembrane state attention mechanism, comprising the following steps:

[0008] S1: Collect audio signals, sample, frame and filter the audio signals for noise reduction, calculate the logarithmic Mel spectrum of the audio signals, and generate MFCC audio features;

[0009] S2: Receive video signals, unify the frame rate of video signals, crop and normalize the video signals,

[0010] Extract frame-level video features of video signals using convolutional neural networks;

[0011] S3: Encode audio features and video features, construct positive and negative sample pairs, align the modal features of audio and video through the contrast learning layer, optimize the modal sharing information and generate modal alignment features;

[0012] S4: Calculate the cross-membrane attention weight: use the audio feature as the query vector and the video feature as the key vector.

[0013] Use cross-modal attention mechanism to dynamically capture the correlation between audio features and video features;

[0014] S5: Filter high-confidence candidate proposal regions through the correlation between audio features and video features, weight the candidate proposal regions, calculate the similarity between each candidate proposal region and the audio features, generate weighted regional features, and output the sound source location and range.

[0015] Furthermore, the generation of the audio features includes: multiplying each frame of the framed audio signal by a Hanning window, performing a short-time Fourier transform on each frame signal to generate an amplitude spectrum, calculating a Mel power spectrum through a Mel filter group, taking the logarithm of the Mel power spectrum and performing a discrete cosine transform.

[0016] Furthermore, the extraction of video features includes: generating a number of candidate proposal regions through selective search in a convolutional neural network to capture potential target regions.

[0017] Furthermore, the positive and negative sample pairs include positive sample pairs and negative sample pairs, the positive sample pairs come from time-step aligned audio features and video features, and the negative sample pairs come from random non-aligned audio features and video features.

[0018] Furthermore, the modal alignment feature is generated using the formula:

[0019]

[0020] in, represents the cosine similarity of audio features and video features at time steps i and j, τ represents the temperature parameter, and n is the number of batch samples.

[0021] Furthermore, the cross-membrane state attention weight is calculated using the formula:

[0022]

[0023] Among them, Q is the projection of audio features, K is the projection of video features, and d k is the feature dimension of the key vector.

[0024] Furthermore, the weighting of the candidate proposal area adopts a global region representation GRM, which is generated by weighting local regions in video features and is used to highlight spatial regions related to sound sources. The global region representation GRM is updated in real time based on the temporal continuity of audio features to perform dynamic scene tracking and adapt to dynamic changes in the sound source position.

[0025] Furthermore, the generation of the global region representation GRM includes:

[0026] The generated attention weights are reshaped into the spatial dimension of each video feature map, and the reshaped attention weights are mapped to the global image area through the interpolation function to generate the position map m n ,

[0027] Where W×H represents the spatial size of the image;

[0028] Position map m n Normalize and generate the global region representation GRM using the formula:

[0029] g n =(m n -min(m n )) / max(m n ).

[0030] Furthermore, the similarity between each candidate proposal region and the audio feature is calculated using the formula:

[0031]

[0032] in, Represents the similarity value of each proposal, a n Represents audio features, represents the video features of each proposed region, e k is a binary mask indicating whether the proposed region exists.

[0033] Furthermore, the generation of the weighted regional features includes:

[0034] The similarity value of each proposal Normalize and generate weights

[0035] Generate weighted regional features

[0036] in, Represents the similarity value for each proposal The weights generated by normalization are Video features representing each proposed region.

[0037] The above technical solution has the following advantages or technical effects:

[0038] 1. The present invention adopts the global region representation GRM, which can effectively focus on the sound source related area and significantly improve the sound source localization accuracy.

[0039] 2. The present invention can automatically filter out complex background (such as leaves, grass) and environmental noise interference, and enhance the robustness of the model in complex natural scenes.

[0040] 3. The present invention realizes deep alignment of audio features and video features through a cross-modal attention mechanism, thereby optimizing the inter-modal synergy effect.

[0041] 4. The present invention also supports dynamic sound source tracking and is suitable for scenes such as bird flight and migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the method of the present invention;

[0043] Figure 2 It is a schematic diagram of the positioning model structure of the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] like Figure 1 As shown, a sound source localization method based on a trans-membrane state attention mechanism comprises the following steps:

[0046] S1: Collect audio signals, sample, frame and filter the audio signals for noise reduction, calculate the logarithmic Mel spectrum of the audio signals, and generate MFCC audio features.

[0047] S11: Collect the audio signal x(t) of the bird's singing and sample the audio signal. Since the frequency range of the bird's singing signal is relatively single, a sampling frequency of 16 kHz is selected to sample the audio signal x(t) into a discrete signal x[n];

[0048] S12: Perform framing and filtering noise reduction on the audio signal, and divide the sampling result into short-time frames with a frame length L of 20ms-40ms and a 50% overlap between frames. This setting can ensure feature continuity and reduce the probability of distortion. Multiply each frame signal by the Hanning window w(n), and the formula is:

[0049]

[0050] S13: Perform short-time Fourier transform (STFT) on each frame signal:

[0051]

[0052] Where X(f,t) represents the amplitude spectrum of frequency f at a given time t;

[0053] S14: Calculate the logarithmic Mel spectrum of the audio signal: using the formula m(f) = 2595·log 10 (1+f / 700) maps the linear frequency to the Mel scale m(f), calculates the Mel power spectrum M(m) through the Mel filter, takes the logarithm of the Mel power spectrum and performs discrete cosine transform to extract the MFCC audio features:

[0054]

[0055] S2: Receive video signals, unify the frame rate of the video signals, crop and normalize the video signals, and use a convolutional neural network to extract frame-level video features of the video signals.

[0056] S21: receiving a video signal, unifying the frame rate of the video signal, and decomposing the video into image frames at a frequency of F frames per second; cropping the video signal, cropping the video frame to a size of 224*224, and normalizing the pixels so that the pixel values ​​are between [0, 1];

[0057] S22: Extracting frame-level video features of video signals: Extracting video frame sequences using ResNet convolutional neural network The frame-level features V t :

[0058] V t =CNN(I t ).

[0059] S3: Encode audio features and video features, construct positive and negative sample pairs, align the modal features of audio and video through the contrast learning layer, optimize the modal sharing information and generate modal alignment features.

[0060] S31: Encode audio features and video features:

[0061] The audio features MFCC K Mapped to a fixed embedding dimension via a linear layer:

[0062] MFCC′ K =Linear(MFCC K );

[0063] Embed the time step position encoding to generate feature Inputt , and pass it into the Transformer encoder:

[0064] E a = TransformerEncoder(Input t );

[0065] The Transformer encoder models the global dependencies in the audio sequence through a multi-head self-attention mechanism and a feed-forward network, and outputs a high-dimensional embedding vector E a ;

[0066] The video frame level feature V t Project to a unified embedding dimension, embed the time step position code, and input it into the Transformer encoder to generate a high-dimensional embedding vector E v ;

[0067] Generate proposed regions: Use selective search to generate several proposed regions as candidate target regions.

[0068] S32: Construct positive and negative sample pairs and contrastive learning layer:

[0069] In the same video clip of the input, select the audio features and video features aligned with the time step as the positive sample pair, and randomly select the non-aligned audio features and video features from the same batch as the negative sample pair;

[0070] The InfoNCE loss function is used as the contrastive learning layer to learn modal sharing information. Its goal is to maximize the similarity between positive sample pairs and minimize the similarity between positive sample pairs and negative sample pairs. The modal features of audio and video are aligned through the contrastive learning layer, the modal sharing information is optimized and the modal alignment features are generated. The formula is as follows:

[0071]

[0072] in, Represents the audio feature E a and video feature E v Cosine similarity between time steps i and j, τ is the temperature parameter, and n is the number of batch samples.

[0073] S4: Calculate the transmembrane attention weight: Use the audio feature as the query vector Query and the video feature as the key vector Key. Use the projection matrix to map each modality to the same feature space, and then calculate the transmembrane attention weight. The formula for scaling the balanced attention distribution is as follows:

[0074]

[0075] Among them, Q is the projection of audio features, K is the projection of video features, and d k is the feature dimension of the key vector.

[0076] The cross-modal attention mechanism is a technology that can dynamically focus on information of different modalities. It can adaptively select the features most relevant to the current task and suppress redundant information, thereby enhancing the multimodal fusion effect and providing a more accurate and robust solution for sound source localization.

[0077] The cross-modal attention mechanism is used to dynamically capture the correlation between audio features and video features. When generating dynamic attention weights, the cross-modal attention mechanism can deeply explore the semantic and spatial correlation between audio features and video features. This process adjusts the weights of different regions in the video features to focus on the target sound source and suppress background information that is not related to the sound source. The final generated dynamic attention weight matrix w n It not only reflects the correlation between audio features and video features, but also provides guidance for further processing of video features, thereby focusing on video features and improving the accuracy and robustness of sound source localization.

[0078] S5: Filter high-confidence candidate proposal regions through the correlation between audio features and video features, weight the candidate proposal regions, calculate the similarity between each candidate proposal region and the audio features, generate weighted regional features, and output the sound source location and range.

[0079] S51: Generate global region representation GRM: generate the generated attention weight w n Reshape it into the spatial dimension c of each video feature map, and map the reshaped attention weights to the global image area through the interpolation function to generate The spatial size of the image W×H is much larger than the dimension c of the audio feature, so when interpolating, the low-dimensional vector w n Expanded to a high-dimensional position graph m n ;

[0080] This process introduces a certain degree of uncertainty, resulting in fuzzy sound source positioning. Although the interpolation operation helps generate the position map, it may also lead to overly smooth areas, making it difficult to accurately display the boundaries of the sound source. However, the position map m n It can still provide reliable spatial prior information for the location and shape of the sound source, so we n for further processing.

[0081] Position map m n After normalization, we get the global region representation (GRM) g n , the formula is as follows:

[0082] g n =(m n -min(m n )) / max(m n );

[0083] The global region representation GRM is the probability distribution of the sound source for each pixel in the image, which can effectively indicate the location and range of the sound source. It provides an unsupervised spatial constraint by weighting all proposed regions, so that the proposed regions that highly overlap with the global region representation GRM are given higher weights.

[0084] The global region indicates that the area with higher values ​​in the GRM contains some part of the sound source, and the area with lower values ​​is the background area that is not related to the sound source. Optionally, a fixed threshold of 0.6-0.7 is used.

[0085] By accumulating the pixels in the global region representation GRM, the confidence of each proposed region can be calculated, thereby screening out the K proposals that are most likely to contain the sound source region.

[0086] S52: Build a sound source localization network to achieve high-precision sound source localization;

[0087] Since some video frames have less than K proposed regions, the present invention proposes a binary mask e=[e 1, e 2 ,…,e K ], where e k Indicates whether the corresponding proposed region exists, thereby dealing with the problem of inconsistent number of proposed regions in different frames.

[0088] Calculate audio feature a n With each proposed video feature The cosine similarity between is used to measure the correlation between the two. The formula is as follows:

[0089]

[0090] Among them, the similarity value of each proposal Indicates the matching degree between the proposed region and the audio signal.

[0091] Normalize the similarity and generate weights Used to weight the video features of each proposal and generate weighted regional features This regional feature combines audio and video information and can accurately represent the location of the sound source.

[0092] The weighted mechanism adopted by the sound source localization network enables the sound source localization network to focus on areas that are highly relevant to the audio, filter out irrelevant background, achieve high-precision sound source localization, and have dynamic adaptability, which can update features in real time and adapt to changes in sound sources in dynamic scenes.

[0093] Since then, the positioning model has been successfully constructed. Figure 2 shown.

[0094] S53: In dynamic scenes, the global region representation GRM is updated in real time according to the temporal continuity of the audio features, and the positioning model dynamically tracks the position and range of the sound source to achieve sound source localization.

[0095] The method of the present invention supports real-time applications and processes continuous dynamic scenes through a sliding window mechanism.

[0096] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A sound source localization method based on a transmembrane state attention mechanism, characterized in that the steps include: S1: Collect audio signals, sample, frame and filter the audio signals for noise reduction, calculate the logarithmic Mel spectrum of the audio signals, and generate MFCC audio features; S2: receiving video signals, unifying the frame rates of the video signals, cropping and normalizing the video signals, and extracting frame-level video features of the video signals using a convolutional neural network; S3: Encode audio features and video features, construct positive and negative sample pairs, align the modal features of audio and video through the contrast learning layer, optimize the modal sharing information and generate modal alignment features; S4: Calculate cross-modal attention weights: Use audio features as query vectors and video features as key vectors, and use the cross-modal attention mechanism to dynamically capture the correlation between audio features and video features; S5: Filter high-confidence candidate proposal regions through the correlation between audio features and video features, weight the candidate proposal regions, calculate the similarity between each candidate proposal region and the audio features, generate weighted regional features, and output the sound source location and range.

2. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The generation of the audio features includes: multiplying each frame of the divided audio signal by a Hanning window, performing short-time Fourier transform on each frame signal to generate an amplitude spectrum, calculating a Mel power spectrum through a Mel filter bank, taking the logarithm of the Mel power spectrum and performing discrete cosine transform.

3. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The extraction of the video features includes: generating a number of candidate proposal regions through selective search in a convolutional neural network to capture potential target regions.

4. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The positive and negative sample pairs include positive sample pairs and negative sample pairs, the positive sample pairs come from audio features and video features aligned in time steps, and the negative sample pairs come from random non-aligned audio features and video features.

5. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The modal alignment feature is generated using the formula: in, represents the cosine similarity of audio features and video features at time steps i and j, τ represents the temperature parameter, and n is the number of batch samples.

6. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The transmembrane state attention weight is calculated using the formula: Among them, Q is the projection of audio features, K is the projection of video features, and d k is the feature dimension of the key vector.

7. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The weighting of the candidate proposal area adopts a global region representation GRM, which is generated by weighting the local regions in the video features and is used to highlight the spatial region related to the sound source. The global region representation GRM is updated in real time based on the temporal continuity of the audio features to perform dynamic scene tracking and adapt to the dynamic changes of the sound source position.

8. A sound source localization method based on a transmembrane state attention mechanism according to claim 7, characterized in that: The generation of the global region representation GRM includes: The generated attention weights are reshaped into the spatial dimension of each video feature map, and the reshaped attention weights are mapped to the global image area through the interpolation function to generate the position map m n , Where W×H represents the spatial size of the image; Position map n Normalize and generate the global region representation GRM using the formula: g n =(m n -min(m n )) / max(m n )。 9. A sound source localization method based on a transmembrane state attention mechanism according to claim 1, characterized in that: The similarity between each candidate proposed region and the audio feature is calculated using the formula: in, Represents the similarity value of each proposal, a n Represents audio features, represents the video features of each proposed region, e k is a binary mask indicating whether the proposed region exists.

10. A sound source localization method based on a transmembrane state attention mechanism according to claim 9, characterized in that: The generation of the weighted regional features includes: The similarity value of each proposal Normalize and generate weights Generate weighted regional features in, Represents the similarity value for each proposal The weights generated by normalization are Video features representing each proposed region.

Images