Method, apparatus, medium and product for selectively locating a sound source based on visual cues

Abstract

The present invention provides a method, device, medium and product for selectively localizing a sound source based on visual cues. The method includes the following steps: obtaining a mixed audio signal and a cue image; the mixed audio signal includes audio signals corresponding to different sound events triggered by at least two different sound sources; the cue image is associated with a target sound source, and the target sound source is the sound source that needs to be selectively localized from different sound sources; the sound event triggered by the object prompted by the cue image and the sound event triggered by the target sound source belong to the same sound event; processing the mixed audio signal and the cue image through a preset cross-instance audiovisual localization model, including estimating a target mask; and outputting a target arrival direction corresponding to the target sound source based on the target mask. The present invention realizes the selective localization of a specific sound source corresponding to a specific sound event from multiple different sound events triggered by multiple sound sources, and has a wide application range.

Description

Technical Field

[0001] The present invention relates to the technical field of sound processing, and particularly to a method, device, medium and product for selectively localizing a sound source based on visual cues. Background Art

[0002] Sound source localization (SSL) focuses on estimating the direction of arrival (DoA) of sound signals in a microphone array. Sound source localization is crucial for human-computer interaction (HRI) applications such as speech enhancement, speech separation, and speaker recognition. Traditionally, sound source localization has been regarded as a signal processing problem, and common methods include phase transform and generalized cross-correlation phase transform (GCC-PHAT), multiple signal classification (MUSIC), and steering response power and phase transform (SRP-PHAT). However, these traditional methods perform poorly when dealing with multiple sound sources.

[0003] More and more research has explored methods based on deep neural networks (DNNs) to improve the effect of multi-source sound source localization. Some frameworks combine deep neural networks with classical techniques to optimize peak detection or subspace estimation using the network, while other frameworks use an end-to-end architecture to directly learn spatial representations from raw audio. However, audio-only systems face inherent limitations: they cannot localize temporarily silent sound sources and exhibit performance degradation in noisy or reverberant environments.

[0004] To overcome these limitations, existing research has introduced the visual modality as a supplementary cue to provide continuous spatial context during acoustic silence. However, vision-only sound source localization methods are vulnerable to factors such as lighting changes, occlusion, and limited field of view (FoV), while audio cues can naturally compensate for these. In this context, audio and vision exhibit a natural synergy: audio ensures the continuous localization of occluded sources, while vision enhances robustness under adverse acoustic conditions.

[0005] Based on the complementarity of audio and visual modalities, a multi-modal audio-visual sound source localization (AV-SSL) method is proposed to overcome the limitations of single modalities. This method first extracts spatial features from the two modalities and then further combines them through a fusion model. The fused features are then input into a downstream network to predict the direction of arrival (DoA) of the sound source. Although this method can estimate the DoA of all sound sources through audio-visual collaboration, it cannot determine which sound source each DoA belongs to and cannot selectively extract the target sound source. For example, in a multi-source scenario with two different sound sources, a cat and a dog, when the microphone array collects the audio signal mixed with the meows of the cat and the barks of the dog, processing the mixed audio signal through the existing audio-visual localization model for sound source localization will obtain two DoAs, namely DoA1 and DoA2, and it is impossible to determine whether DoA1 corresponds to the cat and DoA2 corresponds to the dog, or vice versa. In addition, this method highly depends on paired audio-visual datasets where the visual data clearly corresponds to the sound source, and its practicality is poor. Summary of the Invention

[0006] The first object of the present invention is to provide a method for selectively localizing a sound source based on visual cues, which can perform selective localization according to a cue image and solve the problem of being unable to determine which sound source each direction of arrival belongs to.

[0007] The second object of the present invention is to provide a computer device for implementing the above method for selectively localizing a sound source based on visual cues.

[0008] The third object of the present invention is to provide a computer-readable storage medium for implementing the above method for selectively localizing a sound source based on visual cues.

[0009] The fourth object of the present invention is to provide a computer program product for implementing the above method for selectively localizing a sound source based on visual cues.

[0010] To achieve the above first object, the present invention provides a method for selectively localizing a sound source based on visual cues, which includes the following steps: obtaining a mixed audio signal and a cue image; the mixed audio signal includes audio signals corresponding to different sound events triggered by at least two different sound sources; the cue image is associated with the target sound source, and the target sound source is the sound source that needs to be selectively localized from different sound sources; the sound event triggered by the object prompted by the cue image and the sound event triggered by the target sound source belong to the same sound event; processing the mixed audio signal and the cue image through a preset cross-instance audio-visual localization model, including estimating a target mask; and outputting the target direction of arrival corresponding to the target sound source based on the target mask.

[0011] As can be seen from the above solution, the present invention uses the cue images of different instances within the same sound event category to locate the target sound source, realizing the selective localization of the specific sound source corresponding to a specific sound event among multiple different sound events triggered by multiple sound sources, and solving the problem that existing audio-visual sound source localization cannot determine which sound source each arrival direction belongs to. In addition, different from existing audio-visual sound source localization models that rely heavily on paired data sets, the present invention takes cue images as input and uses category-level semantic visual information to guide the recognition of the target sound source, significantly reducing the dependence on manually curated data sets, enhancing the generalization ability of the model, and having a wide range of applications.

[0012] A further solution is that the process of the cross-instance audio-visual localization model for processing the mixed audio signal and the cue image includes: extracting the mixed audio signal through an audio encoder to obtain audio semantic features; extracting the cue image through a visual encoder and expanding it along the time dimension to obtain expanded visual semantic features; connecting the audio semantic features and the expanded visual semantic features along the feature dimension to obtain a unified multi-modal input sequence; processing the unified multi-modal input sequence through a Conformer network to output semantic-aware multi-modal cue features; obtaining the real part of the spectrogram feature after the mixed audio signal undergoes a short-time Fourier transform, and extracting the real part of the spectrogram feature to obtain audio spatial features; after aligning the semantic-aware multi-modal cue features with the audio spatial features, obtaining aligned features, and estimating a target mask based on the aligned features; predicting the target arrival direction according to the target mask, and the target arrival direction corresponds to the target sound source.

[0013] Thus, it can be seen that the present invention first performs modal fusion at the semantic level to identify the semantic identity of the target sound source, and then generates a target mask therefrom to separate the spatial cues of the required target sound source. The semantics of this mixed audio signal act as a bridge to align the visual semantic features and the audio spatial features. This method not only aligns audio and vision at the spatial level but also ensures consistency at the semantic level, ensuring the consistency of the sound source and position mapping, and also being more in line with the hierarchy followed by human auditory perception: that is, first identifying the sound source from the semantics and then localizing the sound source spatially, so that the arrival direction of the target sound source can be accurately extracted even in a multi-source environment.

[0014] A further solution is to align the semantic-aware multi-modal cue features with the audio spatial features through a semantic-spatial matching module; the semantic-spatial matching module includes a cross-attention Conformer block and a self-attention Conformer block, and the semantic-aware multi-modal cue features and the audio spatial features are sequentially processed through the cross-attention Conformer block and the self-attention Conformer block to obtain aligned features.

[0015] Thus, it can be seen that heterogeneous semantic and spatial features can be effectively aligned, and the spatial cues of the target sound source can be accurately extracted.

[0016] A further solution is that when aligning the semantic-aware multimodal prompt features with the audio spatial features to obtain the aligned features, when estimating the target mask based on the aligned features, it includes: inputting the real part of the spectrogram features into the frequency ConMamba block to output the audio spatial features; aligning the semantic-aware multimodal prompt features and the audio spatial features through the semantic-spatial matching module to obtain the aligned features; inputting the aligned features into the temporal ConMamba block to obtain the target mask.

[0017] Thus, by integrating the narrowband and cross-band modeling strategies through the frequency ConMamba block and the temporal ConMamba block, the spatial features of local and global consistency can be effectively captured.

[0018] A further solution is that when aligning the semantic-aware multimodal prompt features with the audio spatial features to obtain the aligned features, when estimating the target mask based on the aligned features, it includes: inputting the real part of the spectrogram features into the first Conformer block to output the audio spatial features; aligning the semantic-aware multimodal prompt features and the audio spatial features through the semantic-spatial matching module to obtain the aligned features; inputting the aligned features into the second Conformer block to obtain the target mask.

[0019] A further solution is that the cross-instance audiovisual localization model is trained by deep learning. During the training process, it includes setting the dataset and training and evaluating on the dataset. The production of the dataset includes the spatial audio generation process and the prompt image generation process. The spatial audio generation process includes extracting the monaural audio segments from the video, resampling them, and convolving them with the simulated room impulse response to generate the binaural spatial mixed audio. The prompt image generation process includes extracting the video frames from the video, and selecting the frame with the highest score as the prompt image corresponding to the video by evaluating the semantic consistency between the video frames and the corresponding sound events in the video.

[0020] A further solution is that the estimation of the target mask and the prediction of the target arrival direction are respectively set with loss functions for adjustment.

[0021] Thus, the generalization ability can be improved and the training efficiency can be optimized.

[0022] To achieve the above second object, a computer device provided by the present invention includes a processor and a memory, wherein: a computer program is stored on the memory, and when the computer program is executed by the processor, it implements the above method for selectively localizing a sound source based on visual cues.

[0023] To achieve the above third object, a computer-readable storage medium provided by the present invention, wherein: when the computer program is executed by the processor, it implements the above method for selectively localizing a sound source based on visual cues.

[0024] To achieve the above-mentioned fourth object, a computer program product provided by the present invention includes computer instructions, wherein: when the computer instructions are executed by a processor, the method for selectively localizing a sound source based on visual cues as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram showing the selection and localization of a target sound source by the method for selectively localizing a sound source based on visual cues of the present invention.

[0026] Figure 2 It is a flowchart of the first embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0027] Figure 3 It is a schematic diagram showing the process of a cross-instance audiovisual localization model processing a mixed audio signal and a cue image in the first embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0028] Figure 4 It is a schematic diagram showing the specific structure of a semantic space module in the first embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0029] Figure 5 It is a schematic diagram showing the process of making a data set in the first embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0030] Figure 6 It is a schematic diagram showing the data set specification in the process of making a data set in the first embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0031] Figure 7 It is a schematic diagram showing the results of an ablation experiment in the first embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0032] Figure 8 It is a schematic diagram showing the results of comparison with a baseline model in the second embodiment of the method for selectively localizing a sound source based on visual cues of the present invention.

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] The method for selectively localizing a sound source based on visual cues of the present invention uses cue images corresponding to different sound events within the same sound event category to achieve the selection and localization of the target sound source corresponding to the cue image in a multi-sound-source scenario. The present invention also provides a computer device, a computer-readable storage medium, and a computer program product for implementing the above method for selectively localizing a sound source based on visual cues.

[0035] The multi-source sound of the present invention specifically refers to including at least two or more sound sources, and the sound events triggered by the sound sources belong to different sound events, and the number of sound sources varies according to the actual situation.

[0036] Taking the multi-source sound scenario including two sound sources as an example for illustration, refer to Figure 1 , in this multi-source sound scenario, there are two different sound sources, a cat and a dog. The sound events triggered by the cat and the dog both belong to two different sound events in the category of animal calls. Currently, the sound source to be selected for localization is the dog, that is, to determine the arrival direction of the dog. Specifically, the microphone array collects a mixed audio signal after mixing the interference audio signal from the cat and the target audio signal from the dog. Since the prompt image needs to be associated with the target sound source to be selected for localization, and the sound source to be selected for localization is the dog, the prompt image is an image of the dog. After receiving the mixed audio signal and the prompt image, the cross-instance audio-visual localization model outputs the target arrival direction through DOA estimation. The target arrival direction is the arrival direction corresponding to the dog, thereby realizing the selection and localization of the sound source.

[0037] The first embodiment of the method for selectively localizing a sound source based on visual cues:

[0038] Refer to Figure 2 , the method for selectively localizing a sound source based on visual cues in this embodiment is implemented by executing a computer program, and specifically includes the following steps:

[0039] S1: Obtain the mixed audio signal and the prompt image.

[0040] The mixed audio signal includes audio signals corresponding to sound events triggered by at least two or more different sound sources. The target sound source is the sound source to be selected and localized from different sound sources, and the sound events triggered by different sound sources are different.

[0041] The prompt image is associated with the target sound source, that is, the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event.

[0042] The mixed audio signal in this embodiment includes the audio signal of the cat and the audio signal of the dog. The sound source to be selected for localization is the cat, and the prompt image is an image of the cat. The cat in this image can be the same cat as the cat in the sound source or a different cat.

[0043] S2: Process the mixed audio signal and the prompt image through a preset cross-instance audio-visual localization model.

[0044] S3: Output the target arrival direction corresponding to the target sound source.

[0045] Thus, the target sound source can be localized according to the selection. In this embodiment, the target arrival direction corresponding to the cat is specified in the multi-source sound scenario of the cat and the dog.

[0046] The processes of the above steps S1 to S3 will be specifically described below.

[0047] For the symbols in the formulas that appear below, unless otherwise specified, matrices are represented by bold capital letters (e.g., X, Y), vectors are represented by bold lowercase letters (e.g., x, y), and scalar variables are represented by ordinary lowercase letters (e.g., x, y).

[0048] The above steps S1 to S3 can be described as a regression task. The DoA is discretized into 180 categories: , using the Gaussian vector to model the posterior probability distribution:

[0049] , where centered on the true value with a standard deviation of , and the normalization factor is omitted as it does not affect the model prediction. The mixed audio signal is the audio signal of two channels captured by the microphone, denoted as x. The user-specified cue image is denoted as I. The multimodal input is mapped to the DoA distribution through the cross-instance audiovisual localization model: , where are learnable parameters, is the cross-instance audiovisual localization model, and the target DoA is the DoA with the maximum probability: , is the target DoA.

[0050] It should be noted that the range of the predicted DoA above is only an exemplary illustration, and in different scenarios or different reference systems, the range of the predicted DoA may be different.

[0051] Refer to Figure 3 , the process by which the cross-instance audiovisual localization model receives the mixed audio signal and the cue image and processes to output the direction of arrival of the target specifically includes a semantic-level modality fusion stage, a target mask generation stage, and a target direction of arrival prediction stage. Among them, the semantic-level modality fusion stage corresponds to the left yellow module in Figure 3 , the target mask generation stage corresponds to the middle green module in Figure 3 , and the target DoA prediction stage corresponds to the right blue module in Figure 3 .

[0052] In the semantic-level modality fusion stage, the integration of visual and audio modalities is achieved at the semantic level. The mixed audio signal is extracted through the pre-trained audio encoder VGGish to obtain the audio semantic features. The audio semantic features are denoted as , where Indicates the audio duration. Indicates the feature dimension. The audio semantic features include multiple Audio Semantic Tokens. The prompt image is extracted by a pre-trained vision encoder CLIP to obtain visual semantic features. , Indicates the feature dimension. To be compatible with the sequence of audio semantic features, the visual semantic features are extended along the time dimension (the prompt image is a single-frame picture, and when unfolded along the time dimension, actually each frame is the same), resulting in extended visual semantic features. . The extended visual semantic features include multiple Visual Semantic Tokens. The audio semantic features and the extended visual semantic features are concatenated along the feature dimension to form a unified multi-modal input sequence. Then, this unified multi-modal input sequence is processed by a multi-modal Conformer network to output semantic-aware multi-modal prompt features , defined as: , where represents the concatenation along the feature dimension. The semantic-aware multi-modal prompt features include multiple Semantic Prompt Tokens.

[0053] In the target mask generation stage, a target mask is generated, which is used to selectively separate the spatial information of the target sound event corresponding to the target sound source, while suppressing the interference from non-target sound sources. After performing short-time Fourier transform (STFT) on the mixed audio signal, the real part of the spectrogram features can be obtained , where B represents the batch size and 2 represents the number of microphones. The real part of the spectrogram features is input into the frequency ConMamba block, which effectively captures the dependencies between frequencies while retaining the spatial features.

[0054] Continue to refer to Figure 3 , in the frequency ConMamba block, first, through two parallel linear projections, and are obtained: , , As the main representation for sequence modeling, as the modulation factor for sequence modeling. Then, goes through one-dimensional convolution (Conv1D) and uses the SiLU function for local feature extraction. The extracted local features are then processed by the SSM (Selective State Space Model) layer to capture the long-range dependencies between frequencies, and finally, the output is normalized using the first LN (LayerNorm) layer: Finally, the modulation factor of sequence modeling and are transformed through element-wise multiplication to generate the final output audio spatial features , .

[0055] Continuing to refer to Figure 3 , the semantic-spatial matching module will perform semantic-aware multimodal prompt feature and audio spatial features and alignment to extract spatial features related to the target sound source. Refer to Figure 4 , the semantic-spatial matching module includes a cross-attention Conformer block and a self-attention Conformer block for semantic-guided spatial feature optimization.

[0056] The cross-attention Conformer block selectively emphasizes relevant spatial details in the audio spatial features using the semantic-aware multimodal prompt feature . The cross-attention mechanism is adopted, where serves as the query, serves as the key and value (query): , , , respectively represent learnable projection matrices. The multi-head cross-attention mechanism (Multi-head Cross Attention) then calculates: , where represents the feature dimension. The refined output is obtained through the first convolutional (Conv) layer, the first feed-forward (FFN) layer, and the second LN (LayerNorm) layer .

[0057] To further refine the spatial-semantic alignment, the self-attention Conformer block is applied to to capture the dependencies within the sequence. The input is projected into query, key, and value representations as: , where , , respectively represent learnable projection matrices. The multi-head self-attention mechanism (Multi-head Self Attention) then calculates: , represents the feature dimension, and finally the aligned features are obtained through the second convolutional (Conv) layer, the second feed-forward (FFN) layer, and the third LN (LayerNorm) layer The alignment features include multiple alignment semantic tags.

[0058] The temporal ConMamba block has the same structure as the frequency ConMamba block and is used to independently process temporal frames. The alignment features are reshaped to obtain and then enter the temporal ConMamba block for processing. Subsequently, a target mask is output through an MLP layer . The target mask is applied to the real part through element-wise multiplication to obtain the masked spectrogram , , where is element-wise multiplication. Finally, the masked spectrogram is concatenated with the phase spectrogram ( is the imaginary part of the spectrogram feature) to obtain , which is used as the input for the target DoA prediction stage.

[0059] In the target DoA prediction stage, is used as the input to estimate the DoA of the target sound event, i.e., the target DoA. Specifically, after passing through the feature extractor, the posterior probability of the DoA is predicted through a Multi-Layer Perceptron (MLP) and a Softmax function: . Finally, according to , the target DoA is determined as the estimated value with the highest probability. In other embodiments, the target DoA can also be predicted according to other existing deep learning methods.

[0060] For the design of the loss function, in order to optimize the and the ground truth in the time-frequency bin-level reconstruction, the mean squared error (MSE) loss is adopted: .

[0061] In addition, the mean squared error (MSE) loss is used in the DoA estimation based on the posterior probability to optimize the predicted DoA distribution: . The final combined loss function is expressed as: .

[0062] In terms of the dataset setting, in this embodiment, the self-designed VGGSound-SSL dataset is used for training and evaluation.

[0063] The VGGSound-SSL dataset enriches 296 sound event categories with spatial audio and cue images. See Figure 5, for spatial audio generation, first extract the monophonic audio segments from each video and resample them to 16 kHz. To create realistic spatial audio, these monophonic audio segments are convolved with simulated room impulse responses (RIRs) to generate stereo spatial mixed audio. To ensure diverse spatial conditions, 10,000 RIRs are generated by varying key parameters, including microphone position, sound source position, room size, and reverberation time (T60). For prompt image generation, using CLIP (Contrastive Language-Image Pretraining), extract frame embeddings from the videos and text embeddings for the corresponding sound events, evaluate the semantic consistency between the video frames and the sound events by calculating the CLIP scores, and select the frame with the highest score as the prompt image determined from the video. See Figure 6 , Figure 6 shows the specifications of the specific VGGSound-SSL dataset.

[0064] Each training sample is constructed from randomly selected audio segments of two different sound events: one as the target source and the other as the interference source, and these two segments are mixed at a signal-to-noise ratio (SNR) of 0 dB. To visually prompt the target sound, images corresponding to different instances of the same sound event are randomly selected. The hyperparameter settings of all models are consistent, with a step size of 140k, a batch size of 16, using AdamW as the optimizer, a learning rate of 5e-3, and for the short-time Fourier transform, the frame size is set to 64 milliseconds and the hop size is set to 32 milliseconds.

[0065] Use the mean absolute error (MAE) and accuracy (ACC) as evaluation metrics. The lower the MAE and the higher the ACC, the better the performance. MAE evaluates the absolute angular difference between the true direction of arrival and the predicted direction of arrival : , where n is the sample index and N is the total number of samples. ACC evaluates the percentage of the direction of arrival that falls within a predefined tolerance threshold . In this embodiment, , indicating that predictions where the true direction of arrival differs from the predicted direction of arrival by ±5° are considered correct.

[0066] To evaluate the effectiveness of each mechanism in the cross-instance audiovisual localization model (VP-SelDoA) of this embodiment, ablation experiments are also conducted on the VGGSound-SSL dataset. Specifically, in the semantic-spatial matching module, the semantic-aware multimodal prompt features and the audio spatial features In terms of alignment, by comparing with two existing multimodal alignment methods, namely the Concatenation method and the Feature-wise Linear Modulation (FiLM) method, the effectiveness of the speech space matching mechanism of the cross-instance audiovisual localization model in this embodiment is evaluated. In the Concatenation method, the results of the ablation experiment are as Figure 7 shown. In the column corresponding to "V", "√" indicates the use of visual semantic features, and "×" indicates the non-use of visual semantic features; in the column corresponding to "A", "√" indicates the use of audio semantic features, and "×" indicates the non-use of audio semantic features.

[0067] From the results of the ablation experiment, the semantic space matching mechanism has been significantly improved compared with these baseline methods. Among them, the Mean Absolute Error (MAE) has decreased by 8.17°, and the Accuracy (ACC) has increased by 11.54%. This is due to the effective integration of the cross-attention and self-attention modules, achieving better alignment performance.

[0068] In addition, the ablation experiment further evaluates the impact of visual semantic features and audio semantic features on the semantic space matching mechanism. Continuing to refer to Figure 6 , removing these two features leads to a significant performance decline. Compared with the case without removal, the ACC decreases by 27.34%, and the MAE increases by 26.63°. This highlights the key role of external cues in guiding the separation of target sound events. When only relying on audio semantic features without visual semantic features, the ACC drops by 10.28%, and the MAE increases by 4.98°. This degradation is due to the lack of visual cues, while visual semantic features provide target sound information and achieve selective sound isolation. In addition, introducing audio semantic features as a bridge between visual semantic features and alignment features brings significant improvements. The ACC increases by 13.32%, and the MAE decreases by 9.07°. This indicates that audio semantic features effectively promote better alignment.

[0069] Second Embodiment of the Method for Selectively Localizing Sound Sources Based on Visual Cues:

[0070] The difference between this embodiment and the first embodiment of the method for selectively localizing sound sources based on visual cues above is that in the cross-instance audiovisual localization model, two existing Conformer blocks are used. Specifically, the first Conformer block replaces the frequency ConMamba block in the first embodiment, and the second Conformer block replaces the time ConMamba block in the first embodiment. The Conformer block can be referred to in the paper "Conformer: Convolution-augmented Transformer for Speech Recognition".

[0071] For the settings of the baseline models, including the performance of only the audio model and the audiovisual model on the output target DoA. Among them, the only audio models include SELDNet and GCC-MLP, and the audiovisual models include MLP-AVC, AVMLP, DGB, AVST, AVSELD, and CMAF (the cue images are encoded through the visual encoders proposed by each model). Although AVMLP, DGB, and CMAF were originally designed to track speakers over time, they are modified here to predict a single DoA instead of continuous time tracking, so as to serve as the baseline models for the models suitable for evaluating this embodiment. All baseline models are trained and evaluated through the VGGSound-SSL dataset under the same conditions as the cross-instance audiovisual localization models of the first and second embodiments above.

[0072] Subsequently, the cross-instance audiovisual localization model of the first embodiment of the method for selectively localizing sound sources based on visual cues is referred to as the VP-SelDoA model, and the cross-instance audiovisual localization model of the second embodiment of the method for selectively localizing sound sources based on visual cues is referred to as the VP-Conformer model.

[0073] See Figure 8 , Figure 8 shows the results of experiments on the VGGSound-SSL dataset by the VP-SelDoA model, the VP-Conformer model, and all baseline models at different signal-to-noise ratios. When the signal-to-noise ratio is 0 dB, the VP-SelDoA model performs the best, with the lowest MAE of only 12.40° and the highest ACC of 78.23%. The ACC of the only audio model is close to 50%, and this performance is as expected because without visual cues, the only audio model will randomly guess between two speakers, resulting in an accuracy of approximately 50%. These results prove the effectiveness of incorporating visual cues to selectively extract sound sources.

[0074] Among the audiovisual models, the AVSELD model and the CMAF model perform better than other audiovisual baseline models, indicating that the attention-based method more effectively integrates visual and audio features than the direct splicing technique. However, the VP-SelDoA model is the best, demonstrating the superior ability of the semantic space matching mechanism in alignment and fusion. Although the VP-Conformer model has a 5.92% decrease in ACC and a 3.73° increase in MAE compared to the VP-SelDoA model, it still has good results compared to other baseline models.

[0075] In addition, to evaluate the robustness of the model under challenging noise conditions, noise generalization tests were also conducted. All models were trained on data with a signal-to-noise ratio of 0 dB and evaluated within the specified signal-to-noise ratio range from -5 dB to -10 dB. The experiment examined the generalization ability of each model when facing gradually increasing interference. Continue to refer to Figure 8 , the VP-SelDoA model outperformed all baseline models in terms of performance, demonstrating its ability to effectively suppress interfering sounds while capturing target sound events.

[0076] In summary, the method for selectively localizing a sound source based on visual cues according to the present invention realizes the selection and localization of a specific sound source corresponding to a specific sound event among multiple different sound events triggered by multiple sound sources, solves the problem that existing audio-visual sound source localization cannot determine which sound source each arrival direction belongs to, significantly reduces the dependence on manually organized data sets, enhances the generalization ability of the model, and has a wide range of applications.

[0077] Embodiment of a computer device:

[0078] The computer device of this embodiment includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the method embodiment for selectively localizing a sound source based on visual cues described above.

[0079] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the computer device may include more or fewer components, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0080] For example, the processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microcontroller or the processor may also be any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0081] The memory can be used to store computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory and invoking the data stored in the memory, the controller realizes various functions of the computer device. For example, the memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the voice reception function, the function of converting voice into text, etc.); the data storage area can store the data created according to the use of the mobile phone (such as audio data, text data, etc.). In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, memory, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one magnetic disk storage device, flash device, or other volatile solid-state storage devices.

[0082] Embodiment of computer-readable storage medium:

[0083] If the modules integrated in the computer device in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the process of implementing the method embodiment of selectively positioning the sound source based on visual cues can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the controller, the steps of the method embodiment of selectively positioning the sound source based on visual cues can be realized. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The storage medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0084] Embodiment of computer program product:

[0085] The computer program product of this embodiment includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes each step of the above method embodiment for selectively positioning a sound source based on visual cues.

[0086] Finally, it should be emphasized that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for selectively localizing a sound source based on visual cues, characterized in that, The method includes the following steps: Obtain a mixed audio signal and a prompt image; the mixed audio signal includes audio signals corresponding to different sound events triggered by at least two different sound sources; the prompt image is associated with a target sound source, and the target sound source is the sound source that needs to be selectively located and identified from different sound sources; the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event; Process the mixed audio signal and the prompt image through a preset cross-instance audiovisual localization model, including estimating a target mask; Output a target arrival direction corresponding to the target sound source based on the target mask; The process of the cross-instance audiovisual localization model processing the mixed audio signal and the prompt image includes: Extract the mixed audio signal through an audio encoder to obtain audio semantic features; Extract the prompt image through a visual encoder and expand it along the time dimension to obtain expanded visual semantic features; Connect the audio semantic features and the expanded visual semantic features along the feature dimension to obtain a unified multimodal input sequence; The unified multimodal input sequence is processed through a Conformer network to output semantic-aware multimodal prompt features; After the mixed audio signal undergoes a short-time Fourier transform, the real part of the spectrogram feature is obtained, and the real part of the spectrogram feature is extracted to obtain audio spatial features; After aligning the semantic-aware multimodal prompt features with the audio spatial features, alignment features are obtained, and the target mask is estimated based on the alignment features; Predict the target arrival direction according to the target mask, and the target arrival direction corresponds to the target sound source; The cross-instance audiovisual localization model is trained through a deep learning method. During the training process, it includes setting a dataset and training and evaluating on the dataset; The production of the dataset includes a spatial audio generation process and a prompt image generation process. The spatial audio generation process includes extracting a monophonic audio segment from a video, resampling it, and convolving it with a simulated room impulse response to generate a binaural spatial mixed audio; the prompt image generation process includes extracting video frames from the video, and selecting the frame with the highest score as the prompt image corresponding to the video by evaluating the semantic consistency between the video frames and the corresponding sound events in the video.

2. The method for selectively locating a sound source based on visual cues according to claim 1, wherein: Align the semantic-aware multimodal prompt features with the audio spatial features through a semantic-spatial matching module; The semantic-spatial matching module includes a cross-attention Conformer block and a self-attention Conformer block. The semantic-aware multimodal prompt features and the audio spatial features are sequentially processed through the cross-attention Conformer block and the self-attention Conformer block to obtain the alignment features.

3. The method for selectively locating a sound source based on visual cues according to claim 2, wherein: When aligning the semantic-aware multimodal prompt feature with the audio spatial feature to obtain the alignment feature, when estimating the target mask based on the alignment feature, it includes: Input the real part of the spectrogram feature into the frequency ConMamba block to output the audio spatial feature; Align the semantic-aware multimodal prompt feature and the audio spatial feature through the semantic-spatial matching module to obtain the alignment feature; Input the alignment feature into the temporal ConMamba block to obtain the target mask.

4. The method for selectively localizing a sound source based on visual cues according to claim 2, wherein: When aligning the semantic-aware multimodal prompt feature with the audio spatial feature to obtain the alignment feature, when estimating the target mask based on the alignment feature, it includes: Input the real part of the spectrogram feature into the first Conformer block to output the audio spatial feature; Align the semantic-aware multimodal prompt feature and the audio spatial feature through the semantic-spatial matching module to obtain the alignment feature; Input the alignment feature into the second Conformer block to obtain the target mask.

5. The method for selectively localizing a sound source based on visual cues according to claim 1, wherein: The estimation of the target mask and the prediction of the target direction of arrival are respectively set with loss functions for adjustment.

6. A computer device, comprising a processor and a memory, wherein: A computer program is stored on the memory, and when the computer program is executed by the processor, it implements the method for selectively localizing a sound source based on visual cues according to any one of claims 1 to 5 above.

7. A computer-readable storage medium, on which a computer program is stored, wherein: When the computer program is executed by a processor, it implements the method for selectively localizing a sound source based on visual cues according to any one of claims 1 to 5 above.

8. A computer program product, comprising computer instructions, wherein: When the computer instructions are executed by a processor, it implements the method for selectively localizing a sound source based on visual cues according to any one of claims 1 to 5 above.

Images