Method and device for selectively positioning sound source based on visual prompt, medium and product

Abstract

The invention provides a method and a device for selectively positioning a sound source based on visual prompt, a medium and a product. The method comprises the following steps: acquiring a mixed audio signal and a prompt image; the mixed audio signal comprises audio signals corresponding to different sound events triggered by at least two different sound sources; the prompt image is associated with a target sound source, and the target sound source is a required sound source selected and positioned from different sound sources; the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event; processing the mixed audio signal and the prompt image through a preset cross-instance audio-visual positioning model, including estimating a target mask; and outputting a target arrival direction corresponding to the target sound source based on the target mask. According to the invention, the specific sound source corresponding to the specific sound event is selected and positioned from a plurality of different sound events triggered by a plurality of sound sources, and the application range is wide.

Description

Technical Field

[0001] The present invention relates to the technical field of sound processing, and in particular, to a method, device, medium, and product for selectively localizing a sound source based on visual cues. Background Art

[0002] Sound source localization (SSL) focuses on estimating the direction of arrival (DoA) of sound signals in a microphone array. Sound source localization is crucial for human-computer interaction (HRI) applications such as speech enhancement, speech separation, and speaker recognition. Traditionally, sound source localization has been regarded as a signal processing problem, and common methods include phase transform and generalized cross-correlation phase transform (GCC-PHAT), multiple signal classification (MUSIC), and steering response power and phase transform (SRP-PHAT). However, these traditional methods perform poorly when dealing with multiple sound sources.

[0003] More and more research has explored methods based on deep neural networks (DNNs) to improve the effect of multi-sound source localization. Some frameworks combine deep neural networks with classical techniques, using the network to optimize peak detection or subspace estimation, while other frameworks use an end-to-end architecture to directly learn spatial representations from raw audio. However, audio-only systems face inherent limitations: they cannot localize temporarily silent sound sources and exhibit performance degradation in noisy or reverberant environments.

[0004] To overcome these limitations, existing research has introduced the visual modality as a supplementary cue to provide continuous spatial context during acoustic silence. However, visual-only sound source localization methods are vulnerable to factors such as illumination changes, occlusion, and limited field of view (FoV), while audio cues can naturally compensate for these. In this context, audio and vision exhibit a natural synergy: audio ensures the continuous localization of occluded sources, while vision enhances robustness under adverse acoustic conditions.

[0005] Based on the complementarity of audio and visual modalities, a multi-modal audio-visual sound source localization (AV-SSL) method is proposed to overcome the limitations of single-modal. This method first extracts spatial features from the two modalities and then further combines them through a fusion model. The fused features are then input into a downstream network to predict the direction of arrival (DoA) of the sound source. Although this method can estimate the DoA of all sound sources through audio-visual collaboration, it cannot determine which sound source each DoA belongs to and cannot selectively extract the target sound source. For example, in a multi-source scenario with two different sound sources, a cat and a dog, when the microphone array collects the mixed audio signal of the meow and bark, processing the mixed audio signal through the existing audio-visual localization model for sound source localization will obtain two DoAs, namely DoA1 and DoA2, and it is impossible to determine whether DoA1 corresponds to the cat and DoA2 corresponds to the dog, or vice versa. In addition, this method highly depends on paired audio-visual datasets where the visual data clearly corresponds to the sound source, and its practicality is poor. Summary of the Invention

[0006] The first object of the present invention is to provide a method for selectively localizing a sound source based on visual cues, which can perform selective localization according to a cue image and solve the problem of being unable to determine which sound source each direction of arrival belongs to.

[0007] The second object of the present invention is to provide a computer device for implementing the above method for selectively localizing a sound source based on visual cues.

[0008] The third object of the present invention is to provide a computer-readable storage medium for implementing the above method for selectively localizing a sound source based on visual cues.

[0009] The fourth object of the present invention is to provide a computer program product for implementing the above method for selectively localizing a sound source based on visual cues.

[0010] To achieve the above first object, the present invention provides a method for selectively localizing a sound source based on visual cues, which includes the following steps: obtaining a mixed audio signal and a cue image; the mixed audio signal includes audio signals corresponding to different sound events triggered by at least two different sound sources; the cue image is associated with a target sound source, and the target sound source is the sound source that needs to be selectively localized from different sound sources; the sound event triggered by the object prompted by the cue image and the sound event triggered by the target sound source belong to the same sound event; processing the mixed audio signal and the cue image through a preset cross-instance audio-visual localization model, including estimating a target mask; and outputting a target direction of arrival corresponding to the target sound source based on the target mask.

[0011] As can be seen from the above solution, the present invention uses the cue images of different instances within the same sound event category to locate the target sound source, achieving the selective localization of the specific sound source corresponding to a specific sound event among multiple different sound events triggered by multiple sound sources, and solving the problem that existing audio-visual sound source localization cannot determine which sound source each arrival direction belongs to. In addition, different from existing audio-visual sound source localization models that rely heavily on paired data sets, the present invention uses the cue image as the input, utilizes the semantic visual information at the category level to guide the recognition of the target sound source, significantly reduces the dependence on manually sorted data sets, enhances the generalization ability of the model, and has a wide range of applications.

[0012] A further solution is that the process of the cross-instance audio-visual localization model for processing the mixed audio signal and the cue image includes: extracting the mixed audio signal through an audio encoder to obtain audio semantic features; extracting the cue image through a visual encoder and expanding it along the time dimension to obtain expanded visual semantic features; connecting the audio semantic features and the expanded visual semantic features along the feature dimension to obtain a unified multi-modal input sequence; processing the unified multi-modal input sequence through a Conformer network to output semantic-aware multi-modal cue features; obtaining the real part of the spectrogram feature after the mixed audio signal undergoes a short-time Fourier transform, and extracting the real part of the spectrogram feature to obtain audio spatial features; after aligning the semantic-aware multi-modal cue features with the audio spatial features, obtaining aligned features, and estimating a target mask based on the aligned features; predicting the target arrival direction according to the target mask, and the target arrival direction corresponds to the target sound source.

[0013] Thus, it can be seen that the present invention first performs modal fusion at the semantic level to identify the semantic identity of the target sound source, and then generates a target mask based on this to separate the spatial cues of the required target sound source. The semantics of this mixed audio signal serves as a bridge to align the visual semantic features and the audio spatial features, not only aligning audio and vision at the spatial level but also ensuring consistency at the semantic level, ensuring the consistency of the sound source and position mapping, and also more conforming to the hierarchy followed by human auditory perception: that is, first identifying the sound source from the semantics and then locating the sound source spatially, so that the arrival direction of the target sound source can be accurately extracted even in a multi-source environment.

[0014] A further solution is to align the semantic-aware multi-modal cue features with the audio spatial features through a semantic-spatial matching module; the semantic-spatial matching module includes a cross-attention Conformer block and a self-attention Conformer block, and the semantic-aware multi-modal cue features and the audio spatial features are sequentially processed through the cross-attention Conformer block and the self-attention Conformer block to obtain aligned features.

[0015] Thus, it can be seen that heterogeneous semantic and spatial features can be effectively aligned, and the spatial cues of the target sound source can be accurately extracted.

[0016] A further solution is that when aligning the semantic-aware multimodal prompt features with the audio spatial features to obtain the aligned features, when estimating the target mask based on the aligned features, it includes: inputting the real part of the spectrogram features into the frequency ConMamba block to output the audio spatial features; aligning the semantic-aware multimodal prompt features and the audio spatial features through the semantic-spatial matching module to obtain the aligned features; inputting the aligned features into the temporal ConMamba block to obtain the target mask.

[0017] It can be seen that by integrating the narrowband and cross-band modeling strategies through the frequency ConMamba block and the temporal ConMamba block, the spatial features of local and global consistency are effectively captured.

[0018] A further solution is that when aligning the semantic-aware multimodal prompt features with the audio spatial features to obtain the aligned features, when estimating the target mask based on the aligned features, it includes: inputting the real part of the spectrogram features into the first Conformer block to output the audio spatial features; aligning the semantic-aware multimodal prompt features and the audio spatial features through the semantic-spatial matching module to obtain the aligned features; inputting the aligned features into the second Conformer block to obtain the target mask.

[0019] A further solution is that the cross-instance audiovisual localization model is trained by deep learning. During the training process, it includes setting a dataset and training and evaluating on the dataset; the production of the dataset includes a spatial audio generation process and a prompt image generation process. The spatial audio generation process includes extracting a monophonic audio segment from the video, resampling it, and convolving it with a simulated room impulse response to generate a binaural spatial mixed audio; the prompt image generation process includes extracting video frames from the video, and selecting the frame with the highest score as the prompt image corresponding to the video by evaluating the semantic consistency between the video frames and the corresponding sound events in the video.

[0020] A further solution is that the estimation of the target mask and the prediction of the target arrival direction are respectively set with loss functions for adjustment.

[0021] It can be seen that the generalization ability can be improved and the training efficiency can be optimized.

[0022] To achieve the above second object, a computer device provided by the present invention includes a processor and a memory, wherein: a computer program is stored on the memory, and when the computer program is executed by the processor, the method for selectively localizing a sound source based on visual cues described above is implemented.

[0023] To achieve the above third object, a computer-readable storage medium provided by the present invention, wherein: when the computer program is executed by the processor, the method for selectively localizing a sound source based on visual cues described above is implemented.

[0024] To achieve the above fourth object, a computer program product provided by the present invention includes computer instructions, where: when the computer instructions are executed by a processor, the method for selectively locating a sound source based on visual cues as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 FIG. is a schematic diagram of the method for selectively locating a sound source based on visual cues of the present invention to achieve the selection and location of a target sound source.

[0026] Figure 2 FIG. is a flowchart of the first embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0027] Figure 3 FIG. is a schematic diagram of the process of the cross-instance audio-visual localization model processing the mixed audio signal and the cue image in the first embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0028] Figure 4 FIG. is a specific structural schematic diagram of the semantic space module in the first embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0029] Figure 5 FIG. is a schematic diagram of the process of making a data set in the first embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0030] Figure 6 FIG. is a schematic diagram of the data set specification in the process of making a data set in the first embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0031] Figure 7 FIG. is a schematic diagram of the results of the ablation experiment in the first embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0032] Figure 8 FIG. is a schematic diagram of the results of comparison with a baseline model in the second embodiment of the method for selectively locating a sound source based on visual cues of the present invention.

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] The method for selectively locating a sound source based on visual cues of the present invention uses cue images corresponding to different sound events within the same sound event category to achieve the selection and location of the target sound source corresponding to the cue image in a multi-sound source scenario. The present invention also provides a computer device, a computer-readable storage medium, and a computer program product for implementing the above method for selectively locating a sound source based on visual cues.

[0035] The multi - sound sources of the present invention specifically refer to including at least two or more sound sources, and the sound events triggered by the sound sources belong to different sound events. The number of sound sources varies according to the actual situation.

[0036] Taking the multi - sound source scenario including two sound sources as an example for illustration, refer to Figure 1 , in this multi - sound source scenario, there are two different sound sources, a cat and a dog. The sound events triggered by the cat and the dog both belong to two different sound events in the category of animal calls. Currently, the sound source to be selected for localization is the dog, that is, to determine the arrival direction of the dog. Specifically, the microphone array collects a mixed audio signal which is a mixture of the interference audio signal from the cat and the target audio signal from the dog. Since the prompt image needs to be associated with the target sound source to be selected for localization, and the sound source to be selected for localization is the dog, the prompt image is an image of the dog. After receiving the mixed audio signal and the prompt image, the cross - instance audio - visual localization model outputs the target arrival direction through DOA estimation. The target arrival direction is the arrival direction corresponding to the dog, thus realizing the selection and localization of the sound source.

[0037] The first embodiment of the method for selectively localizing a sound source based on visual cues: Refer to Figure 2 , the method for selectively localizing a sound source based on visual cues in this embodiment is implemented by executing a computer program, and specifically includes the following steps: S1: Obtain the mixed audio signal and the prompt image.

[0038] The mixed audio signal includes audio signals corresponding to sound events triggered by at least two or more different sound sources. The target sound source is the sound source to be selected and localized from different sound sources, and the sound events triggered by different sound sources are different.

[0039] The prompt image is associated with the target sound source, that is, the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event.

[0040] The mixed audio signal in this embodiment includes the audio signal of the cat and the audio signal of the dog. The sound source to be selected for localization is the cat, and the prompt image is an image of the cat. The cat in this image can be the same cat as the cat in the sound source or a different cat.

[0041] S2: Process the mixed audio signal and the prompt image through a preset cross - instance audio - visual localization model.

[0042] S3: Output the target arrival direction corresponding to the target sound source.

[0043] Thus, the target sound source can be localized according to the selection. In this embodiment, the target arrival direction corresponding to the cat is specified in the multi - sound source scenario of the cat and the dog.

[0044] The processes of the above steps S1 to S3 will be specifically described below.

[0045] For the symbols in the formulas that appear below, unless otherwise specified, matrices are represented by bold capital letters (e.g., X, Y), vectors are represented by bold lowercase letters (e.g., x, y), and scalar variables are represented by ordinary lowercase letters (e.g., x, y).

[0046] The above steps S1 to S3 can be described as a regression task. The DoA is discretized into 180 categories: , using the Gaussian vector to model the posterior probability distribution of: , where centered on the true value with a standard deviation of , and the normalization factor is omitted as it does not affect the model prediction. The mixed audio signal is the audio signals of two channels captured by the microphone, denoted as x. The cue image specified by the user is denoted as I. The multi-modal input is mapped to the DoA distribution through the cross-instance audiovisual localization model: , where are learnable parameters, is the cross-instance audiovisual localization model, and the target DoA is the DoA with the maximum probability: , is the target DoA.

[0047] It should be noted that the range of the predicted DoA above is only an exemplary description, and in different scenarios or different reference systems, the range of the predicted DoA may be different.

[0048] See Figure 3 , the cross-instance audiovisual localization model receives the mixed audio signal and the cue image, and the process of processing and outputting the direction of arrival of the target specifically includes a semantic-level modality fusion stage, a target mask generation stage, and a target direction of arrival prediction stage. Among them, the semantic-level modality fusion stage corresponds to Figure 3 the left yellow module in Figure 3 , the target mask generation stage corresponds to Figure 3 the middle green module in

[0049] In the semantic-level modality fusion stage, the integration of visual and audio modalities is achieved at the semantic level. The mixed audio signal is extracted through the pre-trained audio encoder VGGish to obtain the audio semantic features. The audio semantic features are denoted as , where represents the audio duration, Denote the feature dimension. The audio semantic features include multiple Audio Semantic Tokens. The prompt image is extracted by the pre-trained vision encoder CLIP to obtain visual semantic features , Denote the feature dimension. To be compatible with the sequence of audio semantic features, the visual semantic features are extended along the time dimension (the prompt image is a single-frame picture, and actually each frame is the same when unfolded along the time dimension), obtaining the extended visual semantic features . The extended visual semantic features include multiple Visual Semantic Tokens. The audio semantic features and the extended visual semantic features are concatenated along the feature dimension to form a unified multi-modal input sequence. Then, this unified multi-modal input sequence is processed by the multi-modal Conformer network to output the semantic-aware multi-modal prompt features , defined as: , where denotes the concatenation along the feature dimension. The semantic-aware multi-modal prompt features include multiple Semantic Prompt Tokens.

[0050] In the target mask generation stage, a target mask is generated, which is used to selectively separate the spatial information of the target sound event corresponding to the target sound source while suppressing the interference from non-target sound sources. After performing the short-time Fourier transform (STFT) on the mixed audio signal, the real part of the spectrogram feature can be obtained, where B represents the batch size and 2 represents the number of microphones. The real part of the spectrogram feature is input to the frequency ConMamba block, and the frequency ConMamba block effectively captures the dependencies between frequencies while retaining the spatial features.

[0051] Continue to refer to Figure 3 , in the frequency ConMamba block, first, through two parallel linear projections, obtain and : , , as the main representation for sequence modeling,[[]]ID=37]] as the modulation factor for sequence modeling. Then,[[]]ID=39]] goes through one-dimensional convolution (Conv1D) and uses the SiLU function for local feature extraction. The extracted local features are then processed by the SSM (Selective State Space Model) layer to capture the long-range dependencies between frequencies, and finally, the output is normalized using the first LN (LayerNorm) layer: ​​, finally, the modulation factor of sequence modeling and are transformed through element-wise multiplication to generate the final output audio spatial features , .

[0052] Continuing to refer to Figure 3 , the semantic-spatial matching module aligns the semantic-aware multimodal prompt features and the audio spatial features to extract the spatial features related to the target sound source. Refer to Figure 4 , the semantic-spatial matching module includes a cross-attention Conformer block and a self-attention Conformer block for semantic-guided spatial feature optimization.

[0053] The cross-attention Conformer block selectively emphasizes the relevant spatial details in the audio spatial features using the semantic-aware multimodal prompt features. The cross-attention mechanism is adopted, where serves as the query, serves as the key and value: , , , , represent learnable projection matrices respectively. The multi-head cross-attention mechanism then calculates: , where represents the feature dimension. The refined output is obtained through the first convolutional (Conv) layer, the first feed-forward (FFN) layer, and the second LN (LayerNorm) layer .

[0054] To further refine the spatial-semantic alignment, the self-attention Conformer block is applied to to capture the dependencies within the sequence. The input is projected into query, key, and value representations as: , where , , represent learnable projection matrices respectively. The multi-head self-attention mechanism then calculates: , represents the feature dimension, and finally the aligned features 。The alignment features include multiple alignment semantic tags.

[0055] The temporal ConMamba block has the same structure as the frequency ConMamba block, and the temporal ConMamba block is used to independently process temporal frames. The alignment features are reshaped to obtain and then enter the temporal ConMamba block for processing, and then output the target mask through an MLP layer 。The target mask is applied to the real part through element-wise multiplication to obtain the masked spectrogram , where is element-wise multiplication. Finally, the masked spectrogram is concatenated with the phase spectrogram ( is the imaginary part of the spectrogram feature) to obtain which is used as the input for the target DoA prediction stage.

[0056] In the target DoA prediction stage, is used as the input to estimate the DoA of the target sound event, that is, the target DoA. Specifically, after passing through the feature extractor, the posterior probability of the DoA is predicted through a Multi-Layer Perceptron (MLP) and a Softmax function: 。Finally, according to the target DoA is determined as the estimated value with the highest probability. In other embodiments, the target DoA can also be predicted according to other existing deep learning methods.

[0057] For the design of the loss function, in order to optimize the and the ground truth in the time-frequency bin-level reconstruction, the mean squared error (MSE) loss is adopted: 。

[0058] In addition, the mean squared error (MSE) loss is used in the DoA estimation based on the posterior probability to optimize the predicted DoA distribution: 。The final combined loss function is expressed as: 。

[0059] In terms of the dataset setting, in this embodiment, the self-designed VGGSound-SSL dataset is used for training and evaluation.

[0060] The VGGSound-SSL dataset enriches 296 sound event categories with spatial audio and cue images. See Figure 5, for spatial audio generation, first, mono audio segments are extracted from each video and resampled to 16 kHz. To create realistic spatial audio, these mono audio segments are convolved with simulated room impulse responses (RIRs) to generate stereo spatial mixed audio. To ensure diverse spatial conditions, 10,000 RIRs are generated by varying key parameters, including microphone position, sound source position, room size, and reverberation time (T60). For prompt image generation, using CLIP (Contrastive Language-Image Pretraining), frame embeddings are extracted from the video, and text embeddings are extracted for the corresponding sound events. The semantic consistency between the video frames and the sound events is evaluated by calculating the CLIP score, and the frame with the highest score is selected as the prompt image determined from the video. See Figure 6 , Figure 6 shows the specifications of the specific VGGSound-SSL dataset.

[0061] Each training sample is constructed from randomly selected audio segments of two different sound events: one as the target source and the other as the interference source, and these two segments are mixed at a signal-to-noise ratio (SNR) of 0 dB. To visually prompt the target sound, images corresponding to different instances of the same sound event are randomly selected. The hyperparameter settings of all models are consistent, with a step size of 140k, a batch size of 16, using AdamW as the optimizer, a learning rate of 5e-3, and for the short-time Fourier transform, the frame size is set to 64 milliseconds and the hop size is set to 32 milliseconds.

[0062] The mean absolute error (MAE) and accuracy (ACC) are used as evaluation metrics. The lower the MAE and the higher the ACC, the better the performance. MAE evaluates the absolute angular difference between the true direction of arrival and the predicted direction of arrival : , where n is the sample index and N is the total number of samples. ACC evaluates the percentage of the direction of arrival falling within a predefined tolerance threshold . In this embodiment, , indicating that predictions where the true direction of arrival and the predicted direction of arrival differ by ±5° are considered correct.

[0063] To evaluate the effectiveness of each mechanism in the cross-instance audiovisual localization model (VP-SelDoA) of this embodiment, ablation experiments are also conducted on the VGGSound-SSL dataset. Specifically, in the semantic-spatial matching module, the semantic-aware multimodal prompt feature and the audio spatial feature In terms of alignment, by comparing with two existing multimodal alignment methods, namely the Concatenation method and the Feature-wise Linear Modulation (FiLM) method, the effectiveness of the speech space matching mechanism of the cross-instance audiovisual localization model in this embodiment is evaluated. In the Concatenation method, the results of the ablation experiment are as shown in Figure 7 which, in the column corresponding to "V", "√" indicates the use of visual semantic features, and "×" indicates the non-use of visual semantic features; in the column corresponding to "A", "√" indicates the use of audio semantic features, and "×" indicates the non-use of audio semantic features.

[0064] From the results of the ablation experiment, the semantic space matching mechanism has been significantly improved compared with these baseline methods, where the Mean Absolute Error (MAE) has decreased by 8.17°, and the Accuracy (ACC) has increased by 11.54%. This is due to the effective integration of the cross-attention and self-attention modules, achieving better alignment performance.

[0065] In addition, the ablation experiment further evaluates the impact of visual semantic features and audio semantic features on the semantic space matching mechanism. Continuing to refer to Figure 6 , removing these two features leads to a significant performance degradation. Compared with the case without removal, the ACC decreases by 27.34%, and the MAE increases by 26.63°. This highlights the key role of external cues in guiding the separation of target sound events. When relying only on audio semantic features without visual semantic features, the ACC decreases by 10.28%, and the MAE increases by 4.98°. This degradation is due to the lack of visual cues, while visual semantic features provide target sound information and achieve selective sound isolation. In addition, introducing audio semantic features as a bridge between visual semantic features and alignment features brings significant improvements, with the ACC increasing by 13.32% and the MAE decreasing by 9.07°. This indicates that audio semantic features effectively promote better alignment.

[0066] Second Embodiment of the Method for Selectively Localizing Sound Sources Based on Visual Cues: The difference between this embodiment and the first embodiment of the method for selectively localizing sound sources based on visual cues described above is that in the cross-instance audiovisual localization model, two existing Conformer blocks are used, namely the first Conformer block replaces the frequency ConMamba block in the first embodiment, and the second Conformer block replaces the time ConMamba block in the first embodiment. The Conformer block can be referred to the paper "Conformer: Convolution-augmented Transformer for Speech Recognition".

[0067] For the settings of the baseline models, including the performance of the audio-only model and the audiovisual model on the output target DoA. Among them, the audio-only models include SELDNet and GCC-MLP, and the audiovisual models include MLP-AVC, AVMLP, DGB, AVST, AVSELD, and CMAF (the cue images are encoded through the visual encoders proposed by each model). Although AVMLP, DGB, and CMAF were originally designed to track speakers over time, they are modified here to predict a single DoA instead of continuous time tracking, thus serving as the baseline models for evaluating the models suitable for this embodiment. All baseline models are trained and evaluated on the VGGSound-SSL dataset under the same conditions as the cross-instance audiovisual localization models in the first and second embodiments above.

[0068] Subsequently, the cross-instance audiovisual localization model of the first embodiment of the method for selectively localizing sound sources based on visual cues is referred to as the VP-SelDoA model, and the cross-instance audiovisual localization model of the second embodiment of the method for selectively localizing sound sources based on visual cues is referred to as the VP-Conformer model. See Figure 8 , Figure 8 shows the results of experiments on the VGGSound-SSL dataset by the VP-SelDoA model, the VP-Conformer model, and all baseline models at different signal-to-noise ratios. At a signal-to-noise ratio of 0 dB, the VP-SelDoA model performs best, with the lowest MAE of only 12.40° and the highest ACC of 78.23%. The ACC of the audio-only models is close to 50%, and this performance is as expected because without visual cues, the audio-only models will randomly guess between two speakers, resulting in an accuracy of approximately 50%. These results demonstrate the effectiveness of incorporating visual cues to selectively extract sound sources.

[0069] Among the audiovisual models, the AVSELD model and the CMAF model perform better than other audiovisual baseline models, indicating that the attention-based method more effectively integrates visual and audio features than the direct splicing technique. However, the VP-SelDoA model is the best, demonstrating the superior ability of the semantic space matching mechanism in alignment and fusion. Although the VP-Conformer model has a 5.92% decrease in ACC and a 3.73° increase in MAE compared to the VP-SelDoA model, it still has good performance relative to other baseline models.

[0070] In addition, to evaluate the robustness of the models under challenging noise conditions, noise generalization experiments were also conducted. All models were trained on data with a signal-to-noise ratio of 0 dB and evaluated in the range of the specified signal-to-noise ratios, from -5 dB to -10 dB. The experiment examined the generalization ability of each model when facing gradually increasing interference. Continue to seeFigure 8 , the performance of the VP-SelDoA model is superior to that of all baseline models, demonstrating its ability to effectively suppress interfering sounds while capturing target sound events.

[0071] In summary, the method for selectively localizing sound sources based on visual cues of the present invention realizes the selection and localization of specific sound sources corresponding to specific sound events among multiple different sound events triggered by multiple sound sources, solves the problem that existing audio-visual sound source localization cannot determine which sound source each arrival direction belongs to, significantly reduces the dependence on manually sorted data sets, enhances the generalization ability of the model, and has a wide range of applications.

[0072] Embodiment of computer device: The computer device of this embodiment includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it realizes the above-mentioned method embodiment for selectively localizing sound sources based on visual cues.

[0073] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the computer device may include more or fewer components, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0074] For example, the processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microcontroller or the processor may also be any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0075] The memory can be used to store computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory and invoking the data stored in the memory, the controller realizes various functions of the computer device. For example, the memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the voice reception function, the function of converting voice into text, etc.); the data storage area can store the data created according to the use of the mobile phone (such as audio data, text data, etc.). In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, memory, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

[0076] Embodiment of computer-readable storage medium: If the modules integrated in the computer device in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the process of implementing the method embodiment of selectively positioning a sound source based on visual cues can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the controller, the steps of the method embodiment of selectively positioning a sound source based on visual cues can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The storage medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0077] Embodiment of computer program product: The computer program product of this embodiment includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes each step of the above method embodiment for selectively locating a sound source based on visual cues.

[0078] Finally, it should be emphasized that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for selectively locating a sound source based on visual cues, characterized in that: The following steps are involved: Acquire a mixed audio signal and a prompt image; the mixed audio signal includes audio signals corresponding to different sound events triggered by at least two different sound sources; the prompt image is associated with a target sound source, and the target sound source is a sound source selected and located from different sound sources; the sound event triggered by the object prompted by the prompt image and the sound event triggered by the target sound source belong to the same sound event; Processing mixed audio signals and cue images through a preset cross-instance audio-visual localization model, including estimating target masks; The target arrival direction corresponding to the target sound source is output based on the target mask.

2. The method for selectively locating a sound source based on visual cues according to claim 1, characterized in that: The following steps are involved: The process of the cross-instance audio-visual localization model processing the mixed audio signal and the prompt image includes: Extracting mixed audio signals through an audio encoder to obtain audio semantic features; The cue image is extracted through the visual encoder and expanded along the time dimension to obtain the extended visual semantic features; The audio semantic features and the extended visual semantic features are connected along the feature dimension to obtain a unified multimodal input sequence; The unified multimodal input sequence is processed by a Conformer network to output semantically-aware multimodal prompt features; The mixed audio signal is subjected to short-time Fourier transform to obtain a real part of a spectrum feature, and the real part of the spectrum feature is extracted to obtain an audio space feature; After aligning the semantic-aware multimodal cue feature with the audio spatial feature, an alignment feature is obtained, and a target mask is estimated based on the alignment feature; A target arrival direction is predicted according to the target mask, where the target arrival direction corresponds to the target sound source.

3. The method for selectively locating a sound source based on visual cues as claimed in claim 2, characterized in that: aligning the semantically-aware multimodal cue feature with the audio spatial feature through a semantic-spatial matching module; The semantic-spatial matching module includes a cross-attention Conformer block and a self-attention Conformer block. The semantic-aware multimodal cue feature and the audio spatial feature are processed by the cross-attention Conformer block and the self-attention Conformer block in sequence to obtain the alignment feature.

4. The method for selectively locating a sound source based on visual cues as claimed in claim 3, characterized in that: When the semantically-aware multimodal cue feature is aligned with the audio spatial feature, the alignment feature is obtained, and when the target mask is estimated based on the alignment feature, the method includes: Input the real part of the spectrogram feature into the frequency ConMamba block, and output the audio spatial feature; Aligning the semantic-aware multimodal cue feature and the audio spatial feature through the semantic-spatial matching module to obtain the aligned feature; The aligned features are input into the temporal ConMamba block to obtain the target mask.

5. The method for selectively locating a sound source based on visual cues as claimed in claim 3, characterized in that: When the semantically-aware multimodal cue feature is aligned with the audio spatial feature, the alignment feature is obtained, and when the target mask is estimated based on the alignment feature, the method includes: Input the real part of the spectrogram feature into a first Conformer block, and output the audio spatial feature; Aligning the semantic-aware multimodal cue feature with the audio spatial feature through a semantic-spatial matching module to obtain the aligned feature; The aligned features are input into the second Conformer block to obtain the target mask.

6. The method for selectively locating a sound source based on visual cues as claimed in claim 1, characterized in that: The cross-instance audio-visual localization model is trained by deep learning. The training process includes setting a data set, and performing training and evaluation on the data set. The production of the dataset includes a spatial audio generation process and a prompt image generation process. The spatial audio generation process includes extracting a mono audio segment from a video, and resampling it and convolving it with a simulated room impulse response to generate a dual-channel spatial mixed audio. The prompt image generation process includes extracting a video frame from the video, and selecting the frame with the highest score as the prompt image corresponding to the video by evaluating the semantic consistency between the video frame and the sound event corresponding to the video.

7. The method for selectively locating a sound source based on visual cues as claimed in claim 1, characterized in that: The estimation of the target mask and the prediction of the target arrival direction are adjusted by setting loss functions respectively.

8. A computer device comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the method for selectively locating a sound source based on visual cues as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for selectively locating a sound source based on visual cues as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method for selectively locating a sound source based on visual cues as described in any one of claims 1 to 7 is implemented.

Images