An event-driven ANN-SNN hybrid architecture sound source positioning method

Abstract

The application belongs to the technical field of computer vision and multi-modal data processing. The application provides an ANN-SNN hybrid architecture sound source positioning method based on event driving. The ANN branch is used to process spatial semantic information, realize high-fidelity semantic understanding of acoustic scene spatial distribution, and capture sound event timing dynamic characteristics by the SNN branch, so as to play the advantages of event driving, low power consumption and noise resistance. The DBSI mechanism is designed to extract semantic and time clues from the same input pulse stream in parallel, realize cross-modal information alignment and integration through a multi-scale feature fusion strategy, and improve multi-modal perception collaboration efficiency. The SGAM module introduces a hard attention mechanism of acoustic pulse gating management, accurately identifies the significant starting time of the sound event, activates visual positioning update only at the key time point, suppresses the positioning tailing phenomenon, and improves the timing resolution and response accuracy.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and multimodal data processing technology, and in particular to a sound source localization method based on an event-driven ANN-SNN hybrid architecture. Background Technology

[0002] The human cognitive system can integrate visual and auditory cues, possessing robust sound source localization capabilities. Inspired by this, sound source localization (SSL) technology simulates cross-modal perception, linking auditory and visual sources. It is a fundamental capability of multimodal systems, driving the development of fields such as autonomous driving.

[0003] Existing SSL methods mainly rely on symmetric two-stream artificial neural networks (ANNs). Although they can perform coarse-grained semantic matching, the frame-based synchronous processing method limits the spatiotemporal accuracy. They suffer from three major defects: lack of discriminative processing mechanism, lack of dynamic gating mechanism, and erosion of temporal transient features, which restrict the evolution of SSL methods.

[0004] The event-driven and asynchronous computational characteristics of spiking neural networks (SNNs) offer new ideas for overcoming limitations. SNNs can capture precise "temporal" information but lack the ability to capture "semantics," while ANNs do the opposite. Bridging the "semantic-temporal" gap is a key challenge in advancing robust SSL.

[0005] This application designs a dual-branch pulse interpretation (DBSI) mechanism to convert the original pulse sequence into two complementary signals, which are then fed into a pulse-guided alignment module (SGAM) to achieve accurate sound source localization.

[0006] Due to the scarcity of large-scale labeled data, self-supervised learning has become the dominant paradigm for SSL. Starting with the Attention-10k framework, subsequent works have continuously improved upon it, such as DMC and AV-Object. To improve discriminative ability, LVS employs a hard negative sample mining strategy, but this may suppress semantically relevant samples. Subsequent research has focused on robust sampling and noise suppression, such as EZ-VSL and SLVAC. Recent advances have focused on achieving fine-grained, object-centered alignment to resolve semantic ambiguity.

[10] Employing a coarse-to-fine alignment strategy, DSOL uses discriminative audiovisual matching to isolate vocal objects. To improve the separation of complex mixtures, training with artificially mixed audio is proposed, and NoPrior extends this idea through dynamic iterative recognition. Furthermore, ensuring semantic consistency is crucial: CMA prioritizes alignment quality, and CLIP-SSL incorporates knowledge from large-scale pre-trained visual language models. The latest trend is towards explicit object-centered modeling; AVGN uses grouped networks, and JSA employs joint slot attention, both aiming to decompose scenes into semantically distinct entities.

[0007] Despite the progress made by the above methods, limitations exist. Existing methods are based on traditional, frame-based ANNs and rely on dense synchronous processing modes. When integrating cross-temporal information, acoustic events become mixed with background noise and interference, resulting in loss of audiovisual feature discrimination, blurred localization maps, and attention "leakage," revealing the shortcomings of symmetric, frame-based processing paradigms in dynamic auditory environments.

[0008] Hybrid Artificial-Spiking Neural Networks (HSNs) are an emerging paradigm that combines the advantages of ANNs and SNNs. They integrate the high-level semantic representation capabilities of ANNs with the event-driven, sparse, and energy-efficient processing methods of SNNs, and have been used for efficient cross-modal processing. For example, in systems based on the Tianji chip, an ANN front-end processes audio and video input to trigger an SNN motion control module for real-time robot control; in the vision field, HSNs fuse static image features extracted by ANNs with dynamic event camera data processed by SNNs for high-speed tracking; and hybrid inference networks convert scene attributes derived from ANNs into pulse sequences to drive SNNs for logical reasoning.

[0009] However, applying existing hybrid designs to fine-grained SSL faces challenges. Existing methods rely on loose modular coupling, with modalities interacting only through high-level abstractions. This fails to meet the precise time alignment requirements of SSL, and the core issue is the unresolved mismatch between ANN features and SNN impulses. The lack of feature-level transformation and alignment mechanisms compromises the temporal accuracy of fine-grained SSL, leading to spatiotemporal misalignment of localization results.

[0010] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0011] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0012] The purpose of this disclosure is to provide an event-driven ANN-SNN hybrid architecture sound source localization method, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0013] According to embodiments of this disclosure, an event-driven ANN-SNN hybrid architecture-based sound source localization method is provided, comprising: Construct the SpikeSync model; the SpikeSync model includes an ANN encoder, an SNN encoder, a DBSI module, and an SGAM module; Video frames are input into an ANN encoder to extract spatial features, and the spatial features are mapped into multiple discrete slot embeddings through a slot attention mechanism, which serve as visual anchors. The Mel spectrogram of the audio data is input into the SNN encoder, which uses spiking neurons to process the Mel spectrogram and generate a binary pulse tensor. The DBSI module decouples the binary pulse tensor to generate continuous auditory embeddings and binary gated sequences; The SGAM module uses binary gating sequences as a dynamic control mechanism to perform event-driven attention masking on the semantic affinity between visual anchors and continuous auditory embeddings, resulting in attention weights that are only effective during periods of active acoustic events. Based on the spatial attention map corresponding to the attention weights and visual anchors, an event-driven sound source localization map is generated.

[0014] Furthermore, the DBSI module decouples the binary impulse tensor to generate continuous auditory embeddings and binary gated sequences, including the following steps: Rate coding branch through an analog time window Accumulate and normalize pulses internally to generate frame-level features. Then, for frame-level features... Perform global averaging to generate continuous auditory embeddings for cross-modal semantic alignment. ; The time event branch generates a binary gating sequence to indicate the active period of an acoustic event based on the instantaneous pulse activity of the binary pulse tensor at the time step.

[0015] Furthermore, frame-level features and continuous auditory embedding The expression is:

[0016] in, To simulate a time window, Let be the pulse characteristics of the t-th frame at the j-th simulation time step. The total number of frames in the time dimension of the input audio features; The expression for a binary gated sequence is:

[0017] in, For indicator functions, The number of channels for auditory features. Let be the pulse feature of the t-th frame at the j-th simulation time step and located on the c-th channel.

[0018] Furthermore, the time event branch uses an indicator function to determine whether there is pulse activity in all channels and within the simulation time window at each time step. If it exists, the value of the binary gating sequence at that time step is 1; otherwise, it is 0.

[0019] Furthermore, the SGAM module uses binary gating sequences as a dynamic control mechanism to perform event-driven attention masking on the semantic affinity between visual anchors and continuous auditory embeddings. The step of obtaining attention weights that are effective only during periods of active acoustic events includes: Visual anchors and continuous auditory embeddings are mapped to a common latent space, and the semantic affinity between them is calculated. The semantic affinity is hard masked by using a binary gating sequence, so that the semantic affinity is used to calculate the attention weight only at time steps where the binary gating sequence indicates an acoustic event; where the attention weight is set to zero at time steps where there is no acoustic event.

[0020] Furthermore, the expression for attention weights is:

[0021] in, For temperature parameters, For semantic affinity, Let be the semantic affinity of the n'th visual slot in frame t.

[0022] Furthermore, the step of generating an event-driven sound source localization map based on the spatial attention map corresponding to the attention weights and visual anchors includes: Generate a spatial attention map for each visual anchor point; By using attention weights, the spatial attention maps of all visual anchors are weighted and summed to generate a temporally resolved localization map; Event-driven aggregation is performed on the time-resolved localization maps for all time steps, i.e., weighted averaging is performed only on the localization maps at the time steps where the binary gating sequence indicates an acoustic event, to generate the final sound source localization map.

[0023] Furthermore, the expression for the time-resolved localization map is:

[0024] in, This is a spatial attention map. The total number of visual slots; The expression for the sound source localization map is:

[0025] in, To prevent extremely small constants with a denominator of zero.

[0026] Furthermore, the composite objective function of the SpikeSync model is:

[0027] in, As the first hyperparameter, This is the second hyperparameter. This is the third hyperparameter. For audiovisual contrast loss, For the regularization loss of the distribution rate, For slot orthogonality loss, For time smoothing loss; Audiovisual contrast loss for:

[0028] in, For batch size, This represents the optimal visual representation of the i-th sample. For audio embedding; Release rate regularization loss for:

[0029] in, This represents the total number of layers in the spiking neural network. The overall average distribution rate. Target distribution rate; orthogonality loss of the slot for:

[0030] in, For the nth visual slot feature in the i-th sample, The m-th visual slot feature in the i-th sample; Time smoothing loss for:

[0031] in, For continuously active pairs.

[0032] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the SpikeSync model is proposed using the aforementioned event-driven ANN-SNN hybrid architecture sound source localization method. This is the first innovative architecture combining artificial neural networks (ANN encoders) and spiking neural networks (SNN encoders) in a sound source localization task. This method utilizes ANN branches to specifically process spatial semantic information, achieving high-fidelity semantic understanding of spatial distributions in acoustic scenes. Simultaneously, it leverages SNN branches to efficiently capture the temporal dynamics of sound events, fully utilizing their advantages of event-driven operation, low power consumption, and noise resistance. This heterogeneous fusion architecture achieves an organic unity between high-level semantic modeling and fine-grained temporal dynamic analysis for the first time. Addressing the modal differences and representation incompatibility between pulse-based discrete auditory signals and continuous visual features, a dual-branch pulse interpretation (DBSI) mechanism is further designed. This mechanism can extract semantic and temporal cues in parallel from the same input pulse stream and achieves effective alignment and integration of cross-modal information through a multi-scale feature fusion strategy, significantly improving the collaborative efficiency of multimodal perception. To optimize the timing synchronization between audiovisual signals, a pulse-guided alignment module (SGAM module) was developed, innovatively introducing a hard attention mechanism managed by acoustic pulse gating. This mechanism can accurately identify significant start moments in sound events and activate visual localization updates only at key time points, thereby fundamentally suppressing the localization tailing phenomenon caused by temporal redundancy in traditional methods and improving the system's timing resolution and response accuracy. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0034] Figure 1 A comparison diagram is shown between the conventional SSL processing paradigm and our processing paradigm in an exemplary embodiment of this disclosure; Figure 2 The diagram illustrates the steps of an event-driven ANN-SNN hybrid architecture sound source localization method according to an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates an overview of the architecture of the SpikeSync model in an exemplary embodiment of this disclosure. Figure 4 Visual comparison diagrams of localization maps of different methods in exemplary embodiments of this disclosure on Flickr-SoundNet-Test and VGG-SS are shown. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0037] This example implementation provides a sound source localization method based on an event-driven ANN-SNN hybrid architecture. (See reference...) Figure 2 As shown, the event-driven ANN-SNN hybrid architecture-based sound source localization method may include: Step S1: Construct the SpikeSync model; wherein, the SpikeSync model includes an ANN encoder, an SNN encoder, a DBSI module, and an SGAM module; Step S2: Input the video frame into the ANN encoder to extract spatial features, and map the spatial features into multiple discrete slot embeddings through the slot attention mechanism as visual anchor points; Step S3: Input the Mel spectrogram of the audio data into the SNN encoder. The SNN encoder uses spiking neurons to process the Mel spectrogram and generate a binary pulse tensor. Step S4: The DBSI module decouples the binary impulse tensor to generate continuous auditory embeddings and binary gated sequences; Step S5: The SGAM module uses the binary gating sequence as a dynamic control mechanism to perform event-driven attention masking on the semantic affinity between the visual anchor and the continuous auditory embedding, and obtains attention weights that are only effective during the active period of the acoustic event. Step S6: Generate an event-driven sound source localization map based on the spatial attention map corresponding to the attention weights and visual anchors.

[0038] Based on the aforementioned event-driven ANN-SNN hybrid architecture for sound source localization, the SpikeSync model is proposed. This is the first innovative architecture to combine artificial neural networks (ANN encoders) and spiking neural networks (SNN encoders) in a sound source localization task. This method utilizes ANN branches to specifically process spatial semantic information, achieving high-fidelity semantic understanding of spatial distributions in acoustic scenes. Simultaneously, it leverages SNN branches to efficiently capture the temporal dynamics of sound events, fully utilizing their event-driven, low-power, and noise-resistant advantages. This heterogeneous fusion architecture achieves, for the first time, an organic unity between high-level semantic modeling and fine-grained temporal dynamic analysis. To address the modal differences and representation incompatibility between pulse-based discrete auditory signals and continuous visual features, a dual-branch pulse interpretation (DBSI) mechanism is further designed. This mechanism can extract semantic and temporal cues in parallel from the same input pulse stream and achieves effective alignment and integration of cross-modal information through a multi-scale feature fusion strategy, significantly improving the collaborative efficiency of multimodal perception. To optimize the timing synchronization between audiovisual signals, a pulse-guided alignment module (SGAM module) was developed, innovatively introducing a hard attention mechanism managed by acoustic pulse gating. This mechanism can accurately identify significant start moments in sound events and activate visual localization updates only at key time points, thereby fundamentally suppressing the localization tailing phenomenon caused by temporal redundancy in traditional methods and improving the system's timing resolution and response accuracy.

[0039] Below, we will refer to Figures 2 to 4 The steps of the event-driven ANN-SNN hybrid architecture sound source localization method described in this example implementation will be explained in more detail.

[0040] In one embodiment, a fundamental challenge in audiovisual learning lies in bridging the inherent representational differences between spatially dense visual signals and temporally sparse acoustic events. Traditional symmetric architectures often struggle to effectively capture this modal heterogeneity, leading to misaligned feature aggregation and high susceptibility to noise interference. To address this issue, this application proposes a method for constructing a heterogeneous backbone network that integrates unique inductive biases tailored to the computational needs of both visual and auditory modalities. Figure 2 The diagram shows the steps of a sound source localization method based on an event-driven ANN-SNN hybrid architecture.

[0041] like Figure 3 As shown, this application employs an asymmetric design, strategically integrating a deep ANN for visual semantic interpretation and a neuromorphic SNN for auditory temporal processing: 1) Visual flow, in order to obtain information from input video frames... Extracting object-centric representations, the visual flow utilizes an ANN encoder. First, dense spatial features are extracted. Then, using a slot attention mechanism, these features are mapped to a set of discrete slot embeddings. These "slots" act as spatial semantic anchors: one slot is specifically designed to capture background noise, while the others represent potential sound-producing entities. 2) Auditory flow, as a complement to visual context, focuses on capturing the transient characteristics of acoustic events. The SNN encoder utilizes LIF neurons to process Mel spectrograms. The pulse dynamics of LIF neurons enable them to respond rapidly to significant sound initiation while essentially suppressing subthreshold environmental noise. This process generates a binary pulse tensor containing spectral content and precise temporal information. Subsequently, this neuromorphic representation is processed via the DBSI mechanism and linked to static visual anchors in SGAM. The data is integrated to ultimately generate a sound source localization map.

[0042] However, directly using the raw binary impulse tensor A for cross-modal learning presents challenges because the discrete nature of impulses is incompatible with the continuous visual domain. Effective cross-modal alignment requires a semantically compatible representation to share the feature space with visual embeddings. However, simply integrating the impulses as continuous values ​​destroys the fine-grained temporal resolution needed to separate transient events from background noise. To balance this trade-off, this application proposes the DBSI strategy, which explicitly decouples the impulse tensor into two complementary signal streams: a rate-encoding branch for extracting semantic context for alignment, and a temporal event branch for deriving binary gating for noise filtering.

[0043] To construct a shared potential space, this application aggregates impulse activities in the time dimension, i.e., within a simulated time window. Accumulate and normalize pulses internally to generate frame-level features. Then a global average is performed to generate : (1) By converting discrete pulse sequences into normalized continuous embeddings This transformation bridges the modal gap, ensuring feature compatibility with the visual slot\Phi, thereby promoting robust contrast optimization.

[0044] To preserve the ability of SNNs to suppress background noise, this application extracts a parallel branch that strictly focuses on the event interval, utilizes the inherent dynamics of LIF neurons in naturally suppressing subthreshold noise, and derives a binary gating sequence based on instantaneous impulse activity. : (2) in For indicator functions. Unlike continuous embedding, binary gating... Hard time constraints were applied to isolate significant transients and ensure that the positioning process is strictly driven by valid impulse events.

[0045] By coordinating these two complementary views, the DBSI strategy effectively addresses the mismatch between pulse and analog signals, providing both rich semantics for cross-modal alignment and precise time gating for noise reduction.

[0046] To achieve accurate localization, audiovisual alignment must be strictly synchronized with active acoustic events. Traditional methods often indiscriminately enforce continuous alignment at all time steps, resulting in the allocation of non-zero attention weights during silent or noisy intervals, thus accumulating false positives. To address this problem, the SGAM proposed in this application utilizes pulse-derived binary gating. As a dynamic control mechanism, visual anchoring is strictly limited to the effective acoustic range.

[0047] First, the static visual slot Auditory features encoded by rate This is mapped to a common latent space. The projection is achieved through a learnable transformation matrix. and This is implemented to evaluate semantic compatibility between modalities. The original semantic affinity is calculated as follows: (3) However, relying solely on raw semantic affinity is insufficient because standard attention mechanisms still force cross-modal interactions during periods of audio inactivity, causing background noise to propagate into the localization results. To address this, this application utilizes temporal gating. Defined discrete event boundaries, apply impulse-triggered hard masking to the attention distribution: (4) in Temperature parameter. Binary gating. The introduction of this feature achieves efficient sparsity reduction of time dependence, ensuring that attention weights are applied only when the presence of acoustic activity is physically confirmed. Only then can the semantic correspondence be reflected.

[0048] After establishing the noise-filtered attention distribution, the system performs spatial decoding. The visual flow derives the spatial attention map for each slot. Subsequently by Weighted construction of time-resolved localization map: (5) To mitigate the signal attenuation inherent in standard time averaging caused by silence, the final output is synthesized using an event-driven aggregation strategy: (6) in Ensuring numerical stability. This selective aggregation enables SGAM to eliminate noise-induced dilution, generating high-contrast localization maps anchored only to significant acoustic events.

[0049] SpikeSync's optimization framework employs a composite objective function that integrates audiovisual contrastive learning, steady-state firing rate regularization, slot orthogonality constraints, and temporal smoothing penalties to ensure robust cross-modal anchoring.

[0050] Establishing cross-modal correspondences requires aligning the global auditory context with the most relevant local visual entities. This application proposes a dynamic matching mechanism to maximize the mutual information between audio embeddings and their semantically most relevant visual counterparts. Let... For having temperature The exponential cosine similarity. Best visual representation. Defined as a slot set with audio embedding The features with the highest similarity: (7) Based on this, audiovisual contrast loss is defined as: (8) in Indicates the batch size.

[0051] Meanwhile, to maintain the optimal information capacity of the spiking backbone and combat the instability of the SNN (such as neuronal inactivity or saturation), the global average firing rate of the l-th layer is constrained through steady-state constraints. Adjust the distribution rate to the target : (9) Furthermore, to avoid representation collapse and ensure that multiple slots do not redundantly encode the same visual region, an orthogonality constraint is imposed on the slot embedding. This objective penalizes the squared cosine similarity between slot pairs within each sample, forcing potential representations into mutually exclusive subspaces. (10) Finally, to address the high-frequency fluctuations caused by the discreteness of pulse dynamics, a time-smoothing constraint for active acoustic events is introduced. This regularization constraint applies to consecutive active pairs. To ensure that silence does not impair the characteristic trajectory: (11) in It is an auditory feature of rate coding.

[0052] The final composite objective function is a weighted combination: (12) in , ,and It is a hyperparameter that balances the contributions of different loss terms.

[0053] In one specific embodiment, the present application will be further described below with reference to the accompanying drawings and embodiments.

[0054] This application aims to address the representational discrepancies and noise interference issues arising from the spatial density of visual signals and the temporal sparsity of auditory signals in existing self-supervised sound source localization techniques. To this end, this application designs an event-driven localization method based on an asymmetric hybrid backbone network.

[0055] To ensure fairness and comprehensiveness in the comparison, this application trained and evaluated the SpikeSync model on widely adopted benchmark datasets. During training, subsets of the Flickr-SoundNet and VGG-Sound datasets were used, with sample sizes of 10k and 144k, respectively. For evaluation, two independent test sets were used: Flickr-SoundNet-Test (containing 250 pairs of labeled samples) and VGG-SS (containing 5k pairs of samples covering 220 categories). Notably, since VGG-SS provides category labels, it also supports evaluation for cross-modal retrieval tasks. During data preprocessing, this application extracted the center frame of the video as visual input and converted the 5-second audio segment in the middle into a 257-band log-Mel spectrogram.

[0056] The asymmetric backbone network of this application comprises two parts: a visual stream and an auditory stream. The visual stream uses a standard ResNet-18 pre-trained on ImageNet as the visual encoder; the auditory stream uses SpikingResNet-18 as the auditory encoder. By replacing the standard activation layers with Conv-BN-LIF modules, an event-driven processing mechanism is implemented. Furthermore, both the visual and auditory streams are equipped with projection heads to map features to a common latent space for alignment.

[0057] This application uses the AdamW optimizer to optimize SpikeSync (weight decay). Batch size SNN and visual encoder respectively use and The initial learning rate is set, and cosine annealing is applied. For SNNs, the LIF neuron parameters are set to... and Simulated step size Number of slots Its dimensions are set as follows: and Number of iterations The temperature in formula (9) And in formula (4) Set to 0.07. Target sparsity. Set to 0.2. Furthermore, the loss weight in formula (12) is set to... , , and .

[0058] This application uses two main metrics to evaluate the performance of SpikeSync: consensus intersection-over-union ratio (cIoU), which is used to calculate the proportion of samples whose intersection-over-union ratio (IoU) between the predicted localization map and the ground truth bounding box exceeds a certain threshold; and area under the curve (AUC), which integrates the accuracy curves under different IoU thresholds as a comprehensive metric to measure the robustness of the system.

[0059] Table 1. Comparison of sound source localization performance of different methods trained on the Flickr-SoundNet subset.

[0060] Table 2. Comparison of sound source localization performance of different methods trained on the VGG-Sound subset dataset.

[0061] First, the SpikeSync method proposed in this application is compared with the current leading self-supervised SSL methods. To ensure fairness, all methods are tested under the same training configuration as FNAC, excluding interference from external detectors and prior knowledge, focusing on evaluating the model's cross-modal alignment capability. As shown in Table 2, this application sets new state-of-the-art performance records on both datasets and at different training scales. In particular, on the VGGSound-144k dataset, although methods such as JSA and FNAC perform reasonably well, their reliance on continuous soft attention mechanisms leads to an inherent "attention leakage" problem, where the system tends to assign non-zero weights during quiet or noisy periods, resulting in blurred localization maps. In contrast, this application utilizes the inherent hard temporal gating mechanism of the SNN auditory pathway to achieve a highest cIoU of 41.87%. This mechanism acts as a strict filter, thoroughly suppressing subthreshold noise and ensuring that visual updates are triggered only by valid acoustic events, thus achieving clearer and more accurate localization.

[0062] Table 3. Evaluation results of zero-shot generalization ability on the Heard 110 and Unheard 110 datasets.

[0063] Furthermore, a robust SSL system should possess the ability to generalize to unseen sound categories. This application evaluates this ability through zero-shot transfer experiments on the Heard / Unheard 110 benchmark, where the model is trained solely on the "Heard 110" dataset. The results in Table 3 show that this application outperforms the comparison methods on both the "known" (Heard 110) and the more challenging "unknown" (Unheard 110) categories. This strong generalization ability confirms that this application has learned a category-agnostic temporal alignment principle, meaning the model does not rely on memory of specific category associations but rather on the physical synchronicity between auditory impulse events and visual changes for localization. By capturing the dynamic characteristics of these events, this application maintains robustness even with novel sound sources, a key capability for meeting real-world deployment requirements.

[0064] Figure 4 Visual comparison of localization maps on complex samples from VGG-SS and Flickr-SoundNet is shown. Figure 4 (a) in the image is the original input image. Figure 4 (b) in the figure is the actual boundary labeling map. Figure 4 (c) in the image is the location heatmap generated in this application. Figure 4 (d) in the image is the location heatmap generated by JSA. Figure 4 (e) in the image is the location heatmap generated by Alignment. Figure 4 (f) in the figure represents the localization heatmap generated by FNAC. Compared with recent methods such as JSA, Alignment, and FNAC, the activation localization map generated in this application is clearer and more compact. Other methods, due to their continuous and noise-sensitive alignment mechanisms, often exhibit attentional divergence, frequently "leaking" into irrelevant background areas. In contrast, the pulse-driven hard-gating mechanism of this application strictly restricts updates to salient acoustic events, generating accurate and object-centric heatmaps that greatly reduce background interference. This significant improvement in visual effects fully validates the effectiveness of the temporal anchoring strategy in aligning auditory events with visual semantics.

[0065] Table 4 Ablation experiment results on the impact of time gating strategy and encoder architecture on positioning performance

[0066] The core innovation of this application lies in the alignment of event-driven and hard gating. Table 3 verifies the effectiveness of this design through ablation experiments on the gating strategy. Replacing the SNN-based hard gating of this application with a standard ANN auditory encoder coupled with soft attention significantly degrades performance (cIoU drops from 41.87 to 37.22). The model that completely removes time gating performs the worst (32.65 cIoU), highlighting the necessity of filtering silent intervals. This experiment clearly demonstrates that impulse-based hard gating is superior to soft alternatives. While soft attention is beneficial, it cannot perfectly attenuate noise, and residual interference dilutes features. In contrast, the threshold-based firing mechanism of this application provides a binary, physical-mechanism-based gating that can completely shield insignificant time periods, thereby generating a cleaner correlated signal and achieving optimal performance.

[0067] Table 5 Ablation experiment results of the effectiveness of each component in the composite loss function

[0068] Finally, this application further analyzes the contribution of each component in the composite loss function. As shown in Table 5, the complete loss function combination produces the best results, and removing any component leads to performance degradation: 1) Pulse firing regularization ( ): Removing this item resulted in the largest performance degradation, confirming the criticality of a stable SNN firing rate for maintaining model training dynamics; 2) Temporal consistency loss ( Omitting this item reduces accuracy, highlighting the necessity of smoothing pulse-driven features between active frames; 3) Slot diversity loss ( Without this feature, slots would be redundant, highlighting its role in ensuring the diversity of visual representations. These results confirm that the overall design of this application has undergone global optimization, enabling robust training of hybrid networks.

[0069] Based on the aforementioned event-driven ANN-SNN hybrid architecture for sound source localization, the SpikeSync model is proposed. This is the first innovative architecture to combine artificial neural networks (ANN encoders) and spiking neural networks (SNN encoders) in a sound source localization task. This method utilizes ANN branches to specifically process spatial semantic information, achieving high-fidelity semantic understanding of spatial distributions in acoustic scenes. Simultaneously, it leverages SNN branches to efficiently capture the temporal dynamics of sound events, fully utilizing their event-driven, low-power, and noise-resistant advantages. This heterogeneous fusion architecture achieves, for the first time, an organic unity between high-level semantic modeling and fine-grained temporal dynamic analysis. To address the modal differences and representation incompatibility between pulse-based discrete auditory signals and continuous visual features, a dual-branch pulse interpretation (DBSI) mechanism is further designed. This mechanism can extract semantic and temporal cues in parallel from the same input pulse stream and achieves effective alignment and integration of cross-modal information through a multi-scale feature fusion strategy, significantly improving the collaborative efficiency of multimodal perception. To optimize the temporal synchronization between audiovisual signals, a pulse-guided alignment module (SGAM module) was developed, innovatively introducing a hard attention mechanism managed by acoustic pulse gating. This mechanism can accurately identify significant start moments in sound events and activate visual localization updates only at key time points, thereby fundamentally suppressing the localization tailing phenomenon caused by temporal redundancy in traditional methods and improving the system's temporal resolution and response accuracy. To verify the effectiveness of the proposed method, this application conducted a systematic and comprehensive experimental evaluation on two widely used public datasets. Experimental results show that SpikeSync achieves state-of-the-art performance in multiple aspects, demonstrating not only excellent sound source localization accuracy but also superior anti-interference robustness and strong cross-scene generalization ability, fully proving the application potential of this architecture in real-world complex environments.

[0070] It should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise" in the above description indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this disclosure.

[0071] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0072] In the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0073] In embodiments of this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0074] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0075] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A sound source localization method based on an event-driven ANN-SNN hybrid architecture, characterized in that, include: Construct the SpikeSync model; the SpikeSync model includes an ANN encoder, an SNN encoder, a DBSI module, and an SGAM module; Video frames are input into an ANN encoder to extract spatial features, and the spatial features are mapped into multiple discrete slot embeddings through a slot attention mechanism, which serve as visual anchors. The Mel spectrogram of the audio data is input into the SNN encoder, which uses spiking neurons to process the Mel spectrogram and generate a binary pulse tensor. The DBSI module decouples the binary pulse tensor to generate continuous auditory embeddings and binary gated sequences; The SGAM module uses binary gating sequences as a dynamic control mechanism to perform event-driven attention masking on the semantic affinity between visual anchors and continuous auditory embeddings, resulting in attention weights that are only effective during periods of active acoustic events. Based on the spatial attention map corresponding to the attention weights and visual anchors, an event-driven sound source localization map is generated.

2. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 1, characterized in that, The DBSI module's steps for decoupling the binary impulse tensor to generate continuous auditory embeddings and binary gated sequences include: Rate coding branch through an analog time window Accumulate and normalize pulses internally to generate frame-level features. Then, for frame-level features... Perform global averaging to generate continuous auditory embeddings for cross-modal semantic alignment. ; The time event branch generates a binary gating sequence to indicate the active period of an acoustic event based on the instantaneous pulse activity of the binary pulse tensor at the time step.

3. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 2, characterized in that, Frame-level features and continuous auditory embedding The expression is: in, To simulate a time window, Let be the pulse characteristics of the t-th frame at the j-th simulation time step. The total number of frames in the time dimension of the input audio features; The expression for a binary gated sequence is: in, For indicator functions, The number of channels for auditory features. Let be the pulse feature of the t-th frame at the j-th simulation time step and located on the c-th channel.

4. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 3, characterized in that, The time event branch uses an indicator function to determine whether there is pulse activity in all channels and within the simulation time window at each time step. If there is, the value of the binary gating sequence at that time step is 1; otherwise, it is 0.

5. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 4, characterized in that, The SGAM module uses binary gating sequences as a dynamic control mechanism to perform event-driven attention masking on the semantic affinity between visual anchors and continuous auditory embeddings. The step of obtaining attention weights that are effective only during periods of active acoustic events includes: Visual anchors and continuous auditory embeddings are mapped to a common latent space, and the semantic affinity between them is calculated. The semantic affinity is hard masked by using a binary gating sequence, so that the semantic affinity is used to calculate the attention weight only at time steps where the binary gating sequence indicates an acoustic event; where the attention weight is set to zero at time steps where there is no acoustic event.

6. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 5, characterized in that, The expression for attention weights is: in, For temperature parameters, For semantic affinity, Let be the semantic affinity of the n'th visual slot in frame t.

7. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 6, characterized in that, The step of generating an event-driven sound source localization map based on the spatial attention map corresponding to attention weights and visual anchors includes: Generate a spatial attention map for each visual anchor point; By using attention weights, the spatial attention maps of all visual anchors are weighted and summed to generate a temporally resolved localization map; Event-driven aggregation is performed on the time-resolved localization maps for all time steps, i.e., weighted averaging is performed only on the localization maps at the time steps where the binary gating sequence indicates an acoustic event, to generate the final sound source localization map.

8. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 7, characterized in that, The expression for the time-resolved localization map is: in, This is a spatial attention map. The total number of visual slots; The expression for the sound source localization map is: in, To prevent extremely small constants with a denominator of zero.

9. The event-driven ANN-SNN hybrid architecture sound source localization method according to claim 8, characterized in that, The composite objective function of the SpikeSync model is: in, As the first hyperparameter, This is the second hyperparameter. This is the third hyperparameter. For audiovisual contrast loss, For the regularization loss of the distribution rate, For slot orthogonality loss, For time smoothing loss; Audiovisual contrast loss for: in, For batch size, This represents the optimal visual representation of the i-th sample. For audio embedding; Release rate regularization loss for: in, This represents the total number of layers in the spiking neural network. The overall average distribution rate. Target distribution rate; orthogonality loss of the slot for: in, For the nth visual slot feature in the i-th sample, The m-th visual slot feature in the i-th sample; Time smoothing loss for: in, For continuously active pairs.

Images