A sound source positioning method and system based on an impulse neural network

Abstract

The present application relates to a kind of sound source positioning method and system based on pulse neural network, including extracting first acoustic feature set according to first audio signal set and performing pulse coding operation, obtain first pulse signal, use pulse neural network to execute inference operation as input signal using first pulse signal, obtain azimuth prediction result corresponding to sound source, utilize multi-channel audio perception object current position, using binary pulse coding, audio stream is coded as pulse information as pulse neural network input and pulse neural network inference is deployed, utilize the time sequence correlation of audio information, the mode of processing information of class biological is realized, solve the problem that traditional algorithm is long in computation time, hardware system is large in power consumption.

Description

Technical Field

[0001] This invention relates to the fields of sound source localization and artificial neural network application technology, and in particular to a sound source localization method and system based on a spiking neural network. Background Technology

[0002] Most existing sound source localization systems employ traditional sound source localization algorithms or artificial neural networks. Traditional sound source localization methods are based on signal / channel models and signal processing techniques, including the GCC-PHAT algorithm based on relative time delay estimation, the SRP-PHAT algorithm based on beamforming, and the MUSIC algorithm based on signal subspace. Although they have made significant progress in this field over the years, their performance is generally poor in the presence of noise and reverberation.

[0003] In recent years, an increasing number of sound source localization systems based on artificial neural networks (ANNs) have been proposed. Most reported work demonstrates that ANN-based sound source localization methods outperform traditional methods. For example, under low signal-to-noise ratio conditions, using CNNs improves overall sound source localization accuracy by a factor of two compared to the traditional SRP-PHAT method; and using CRNNs can reduce the mean angular error under reverberation conditions by 50% compared to the traditional MUSIC algorithm. Spiking neural networks (SNNs), as important models in neuroscience, possess characteristics such as biologically interpretable neuronal behavior, the use of spiking signals to transmit information, and operation in an event-driven computational model compared to traditional artificial neural networks.

[0004] Therefore, it can be seen that traditional sound source localization algorithms have poor robustness and cannot perceive the location of moving target sound sources in environments with high noise and strong reverberation, thus limiting their application scenarios. On the other hand, sound source localization systems based on artificial neural networks using existing technologies suffer from the drawback of high hardware system power consumption. The edge deployment of traditional algorithms and artificial neural network models consumes a lot of computing resources and consumes a lot of power.

[0005] Spiking neural networks (SNNs) are composed of interconnected spiking neurons, and the spiking neuron model is abstracted from biological neurons. Spiking neural networks have achieved significant success in many fields, such as image classification, object detection, speech recognition, natural language processing, and robot control, as they are more energy-efficient and biologically plausible models for these tasks. However, few studies have demonstrated the effectiveness of SNNs in the task of sound source localization. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a sound source localization method and system based on a spiking neural network. It utilizes multi-channel audio to perceive the current position of an object, employs binary pulse coding to encode the audio stream into pulse information, uses this information as input to the spiking neural network, and deploys the spiking neural network for inference. By leveraging the temporal correlation of audio information, it achieves a biological-like information processing approach, solving the problems of long computation time and high power consumption of traditional algorithms.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0008] A sound source localization method based on a spiking neural network, characterized by comprising:

[0009] S1. Obtain a first audio signal set, the first audio signal set including two or more independent audio signals to be identified recorded by two or more microphones respectively, and determine whether the first audio signal set is greater than a preset sound source length threshold.

[0010] S2. When it is determined that the first audio signal set is greater than the preset sound source length threshold, it is further determined whether there is active sound in the first audio signal set;

[0011] S3. When it is determined that there is active sound in the first audio signal set, the first acoustic feature set is extracted based on the first audio signal set. The first acoustic feature set includes the generalized cross-correlation-phase transform features corresponding to each audio signal to be identified.

[0012] S4. Perform a pulse coding operation on the first acoustic feature set to obtain the first pulse signal;

[0013] S5. Using the first pulse signal as the input signal, perform inference operations using a preset spiking neural network to generate the second pulse signal;

[0014] S6. Perform a decoding operation on the second pulse signal to generate the azimuth prediction result.

[0015] Furthermore, the method also includes:

[0016] A visual sound source localization result is generated based on the azimuth prediction result.

[0017] Furthermore, performing a pulse coding operation on the first acoustic feature set includes performing a pulse coding operation using a spiking neuron.

[0018] Furthermore, the spiking neuron includes a LIF neuron, which is calculated using numerical simulation according to Equation 1:

[0019]

[0020] in, This represents the input value of the i-th neuron in the n-th layer at time step t. This represents the membrane potential value after neurodynamic treatment. τ represents the membrane potential value of the i-th neuron in the n-th layer after it generates a pulse at time step t. m It is the membrane time constant that controls the decay, u rest It is the resting potential after discharge. It is the output of the neuron at time step t, V th θ is the voltage threshold, and θ is the Heaviside step function.

[0021] Furthermore, the generalized cross-correlation-phase transform feature is calculated using Equation 2:

[0022]

[0023] Where R represents the real part of the complex number, S * Let S denote the conjugate complex number, k denote the value at the k-th point of the short-time Fourier transform, and S p1 and S p2 N represents the short-time Fourier transform of the microphone for signals P1 and P2, respectively. s τ represents the length of the short-time Fourier transform, and τ represents the time delay.

[0024] Furthermore, the spiking neural network includes formula 3 to represent the probability of sound being present at an azimuth angle:

[0025]

[0026] Where θ represents the azimuth angle, and p(θ) represents a Gaussian-like vector. The actual label is represented by σ, which is a constant related to the width of the Gaussian function.

[0027] The present invention also relates to a sound source localization system based on a spiking neural network, characterized in that it comprises:

[0028] The audio acquisition module is used to acquire the first set of audio signals;

[0029] An audio preprocessing module is used to determine whether the first audio signal set is greater than a preset sound source length threshold, and to determine whether there is active sound in the first audio signal set;

[0030] The preprocessing module is used to extract a first acoustic feature set based on a first audio signal set, and perform a pulse coding operation on the first acoustic feature set to obtain a first pulse signal;

[0031] The inference calculation module is used to take the first pulse signal as an input signal, perform inference operations using a preset pulse neural network, and generate a second pulse signal.

[0032] The post-processing module is used to perform decoding operations on the second pulse signal and generate azimuth prediction results.

[0033] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0034] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0035] The memory is used for storage;

[0036] The processor is used to execute the above-described methods by invocation.

[0037] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0038] The beneficial effects of this invention are as follows:

[0039] The sound source localization method and system based on a spiking neural network described in this invention utilizes multi-channel audio to perceive the current position of an object. Binary pulse coding is used to encode the audio stream into pulse information, which serves as input to the spiking neural network for inference. By leveraging the temporal correlation of audio information, a biological-like information processing approach is achieved. Compared to the hundreds of milliseconds of computational latency of traditional algorithms, the sound source localization method and system of this invention, utilizing the characteristics of audio information processing, construct a spiking neural network that uses the temporal and spatial correlation of pulse sequences for inference, accelerating the perception speed and achieving a reaction speed at the 10ms level, thus solving the problem of lengthy computation times in traditional algorithms. Simultaneously, the sound source localization system operates with low hardware power consumption, solving the problem of high hardware system power consumption. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the sound source localization method based on a spiking neural network according to the present invention.

[0041] Figure 2 This is a schematic diagram of the sound source localization system based on a spiking neural network according to the present invention. Detailed Implementation

[0042] To better understand the content of this invention, a detailed description will be provided in conjunction with the accompanying drawings and embodiments.

[0043] The first aspect of this invention relates to a process flow as follows: Figure 1 The sound source localization method based on spiking neural networks shown includes:

[0044] S1. Obtain a first audio signal set, which includes two or more independent audio signals to be identified recorded by two or more microphones respectively, and determine whether the first audio signal set is greater than a preset sound source length threshold.

[0045] In practice, a microphone array can be used to record the audio signals emitted by an object. The microphone array consists of multiple microphone receivers arranged evenly, and each microphone produces a single-channel audio signal. For a sensor with four microphones, the audio signals of each channel will differ due to the different positions of the microphones. Sound source localization uses the data differences between the various audio channels to determine the spatial location of the sound source.

[0046] S2. When it is determined that the first audio signal set is greater than the preset sound source length threshold, it is further determined whether there is active sound in the first audio signal set.

[0047] S3. When it is determined that there is active sound in the first audio signal set, the first acoustic feature set is extracted based on the first audio signal set. The first acoustic feature set includes the generalized cross-correlation-phase transform features corresponding to each audio signal to be identified.

[0048] Preferably, the generalized cross-correlation-phase transform feature is obtained by formula 2:

[0049]

[0050] Where R represents the real part of the complex number, S * Let k denote the conjugate complex number, and k represent the value at the k-th point of the short-time Fourier transform. and N represents the short-time Fourier transform of the microphone for signals P1 and P2, respectively. s τ represents the length of the short-time Fourier transform, and τ represents the time delay.

[0051] Specifically, in sound source localization, time-delay-based methods have achieved significant success due to their computational simplicity and effectiveness, especially GCC-PHAT, which is widely used to estimate the TDoA between any two microphones. A direct encoding method is employed based on LIF neurons, using a pulse encoder composed of LIF neurons to encode the aforementioned acoustic features into a pulse sequence. The pulse encoder simultaneously serves as a primary feature extractor and a pulse generator. Specifically, the acoustic features are copied multiple times, with each copy serving as the input frame for each time step.

[0052] S4. Perform a pulse coding operation on the first acoustic feature set to obtain the first pulse signal.

[0053] Preferably, performing a pulse coding operation on the first acoustic feature set includes performing a pulse coding operation using a spiking neuron, wherein the spiking neuron includes a LIF neuron. Specifically, the LIF neuron performs numerical simulation calculations using Equation 1:

[0054]

[0055] in, This represents the input value of the i-th neuron in the n-th layer at time step t. This represents the membrane potential value after neurodynamic treatment. τ represents the membrane potential value of the i-th neuron in the n-th layer after it generates a pulse at time step t. m It is the membrane time constant that controls the decay, u rest It is the resting potential after discharge. It is the output of the neuron at time step t, V th θ is the voltage threshold, and θ is the Heaviside step function.

[0056] Specifically, the LIF neuron (accumulation-leakage-firing) model, whose discrete computational behavior is divided into four processes—charging, discharging, resetting, and leaking—is a widely used spiking neuron in deep learning.

[0057] S5. Using the first pulse signal as the input signal, perform inference operations using a preset spiking neural network to generate the second pulse signal.

[0058] Preferably, the training layer of the spiking neural network includes using Equation 3 to represent the probability of sound being present at the azimuth angle:

[0059]

[0060] Where θ represents the azimuth angle, and p(θ) represents a Gaussian-like vector. Let σ represent the actual label and σ be a constant related to the width of the Gaussian function. Preferably, the MSE loss function can be used to measure the similarity between p(θ) and the predicted value. Preferably, θ∈[1,360°], and σ can be set to 5.

[0061] In practical implementation, at the data level, the SLoClas dataset can be used to train and evaluate the model. This dataset is designed for real-world sound localization and classification tasks, containing 23.27 hours of data recorded by ReSpeaker Mic Array v2.0, with a total of 10 sounds, azimuth angles ranging from 1° to 360°, and intervals of 5°. At the model level, to fully support the PAICORE2.0 chip and avoid operators not supported by the chip, the preferred model consists only of fully connected layers, pooling layers, and LIF neurons. Preferably, the GCC-PHAT feature dimension extracted from the original multi-channel audio signal is 27×6×51, where 27 is the number of frames in each audio segment, 6 is the total number of pairs of microphones, and 51 is the GCC-PHAT feature dimension of each pair of microphones. The GCC-PHAT features are flattened to 27×306 dimensions and then fed into the subsequent neural network for processing. The specific network structure is Input(27, 306)-FC1024-FC1024-MP2d(27, 1)-FC720-FC360. In specific implementations, it is preferable to set the reset value of the LIF neuron to 0, the membrane time constant to 2.0, the voltage threshold to 1.0, and the time step to 8.

[0062] S6. Perform a decoding operation on the second pulse signal to generate the azimuth prediction result.

[0063] Preferably, it also includes generating a visualized sound source localization result based on the azimuth prediction result.

[0064] Preferably, for the sake of versatility, the spiking neural network can also be deployed on existing common platforms, such as Jetson Xavier NX. Taking a typical indoor sound source localization scenario as an example, the microphone array of the sensor outputs multi-channel digital audio signals.

[0065] In implementing the above method, a PC can encode the multi-channel audio signals captured by the microphone array into pulse signals. The FPGA, acting as a converter between pulse information and frames required by the spiking neural network chip (e.g., PAICORE2.0), sends the frames to PAICORE2.0 via the PCIe interface (2.5GT / s). PAICORE2.0 parses the frame information, feeds the pulse information into its internally built spiking neural network for inference, and obtains the azimuth information of the sound-emitting object. This information is then sent to the FPGA and then to the PC via frames, where the corresponding azimuth index is obtained. This achieves the perception of the horizontal direction of the sound source location and is simultaneously displayed on the monitor as a radar chart. The core of the sound source localization system is the encoding and model algorithm after obtaining the original audio signal, as well as the hardware system composed of the PC, FPGA, and PAICORE2.0.

[0066] Taking the PAICORE2.0 chip application as an example, before deployment to the chip, the configuration frame of the chip needs to be obtained using the chip toolchain PAIFLOW. For example, the original spiking neuron node is replaced with a spiking neuron node adapted to the toolchain using `replace_single_node`, 8-bit quantization is performed using the `quantize` method of the transformer, the `init_dag` method of the transformer is used to parse the SNN, and parameters are added to the DAG of the SNN using `set_params_for_dag`, finally obtaining a chip-compatible SNN model. The new SNN model is then converted into a computation graph, and further converted into an on-chip network, thus obtaining the specific mapping and configuration frame of the network on PAICORE2.0.

[0067] The hardware deployment generally involves seven steps: powering on the PAICORE2.0 board; programming the FPGA; installing the PCIe driver; resetting the device; configuring the serial port; configuring the configuration frame on PAICORE2.0 to complete the network configuration; and finally running the inference program. After completing the network configuration, input data needs to be sent to PAICORE2.0. PAICORE2.0 accepts input in the form of working frames. The result after each inference needs to be converted into a frame and sent to the FPGA via the PCIe interface to complete one input. PAICORE2.0 parses the frame, performs inference, and then outputs a working frame, which is sent back to the FPGA. This cycle repeats to realize the operation of the sound source localization system. In actual use, attention should be paid to the calibration of the microphone array reference frame to determine the 0° position and positive direction.

[0068] Another aspect of the present invention relates to a sound source localization system based on a spiking neural network, the structure of which is as follows: Figure 2 As shown, it includes:

[0069] The audio acquisition module is used to acquire the first set of audio signals;

[0070] An audio preprocessing module is used to determine whether the first audio signal set is greater than a preset sound source length threshold, and to determine whether there is active sound in the first audio signal set;

[0071] The preprocessing module is used to extract a first acoustic feature set based on a first audio signal set, and perform a pulse coding operation on the first acoustic feature set to obtain a first pulse signal;

[0072] The inference calculation module is used to take the first pulse signal as an input signal, perform inference operations using a preset pulse neural network, and generate a second pulse signal.

[0073] The post-processing module is used to perform decoding operations on the second pulse signal and generate azimuth prediction results.

[0074] Preferably, it also includes a visualization module, which is used to generate and visualize the sound source localization results based on the azimuth prediction results.

[0075] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0076] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0077] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device includes at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as [examples omitted]. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0078] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0079] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0080] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0084] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A sound source localization method based on a spiking neural network, characterized in that, include: S1. Obtain a first audio signal set, the first audio signal set including two or more independent audio signals to be identified recorded by two or more microphones respectively, and determine whether the first audio signal set is greater than a preset sound source length threshold. S2. When it is determined that the first audio signal set is greater than the preset sound source length threshold, it is further determined whether there is active sound in the first audio signal set; S3. When it is determined that there is active sound in the first audio signal set, the first acoustic feature set is extracted based on the first audio signal set. The first acoustic feature set includes the generalized cross-correlation-phase transform features corresponding to each audio signal to be identified. S4. Perform a pulse coding operation on the first acoustic feature set to obtain the first pulse signal; S5. Using the first pulse signal as the input signal, perform inference operations using a preset spiking neural network to generate the second pulse signal; S6. Perform a decoding operation on the second pulse signal to generate the azimuth prediction result.

2. The method as described in claim 1, characterized in that, The method further includes: A visual sound source localization result is generated based on the azimuth prediction result.

3. The method as described in claim 1, characterized in that, The pulse coding operation on the first acoustic feature set includes performing pulse coding operation using spiking neurons.

4. The method as described in claim 3, characterized in that, The spiking neuron includes a LIF neuron, which is calculated using numerical simulation according to Equation 1: in, This represents the input value of the i-th neuron in the n-th layer at time step t. This represents the membrane potential value after neurodynamic treatment. τ represents the membrane potential value of the i-th neuron in the n-th layer after it generates a pulse at time step t. m It is the membrane time constant that controls the decay, u rest It is the resting potential after discharge. It is the output of the neuron at time step t, V th θ is the voltage threshold, and θ is the Heaviside step function.

5. The method as described in claim 1, characterized in that, The generalized cross-correlation-phase transform feature is obtained by formula 2: Where R represents the real part of the complex number, S * Let S denote the conjugate complex number, k denote the value at the k-th point of the short-time Fourier transform, and S p1 and S p2 N represents the short-time Fourier transform of the microphone for signals P1 and P2, respectively. s τ represents the length of the short-time Fourier transform, and τ represents the time delay.

6. The method as described in claim 1, characterized in that, The pulse neural network includes a formula 3 that represents the probability of sound being present at an azimuth angle: Where θ represents the azimuth angle, and p(θ) represents a Gaussian-like vector. The actual label is represented by σ, which is a constant related to the width of the Gaussian function.

7. A sound source localization system based on a spiking neural network, characterized in that, include: The audio acquisition module is used to acquire the first set of audio signals; An audio preprocessing module is used to determine whether the first audio signal set is greater than a preset sound source length threshold, and to determine whether there is active sound in the first audio signal set; The preprocessing module is used to extract a first acoustic feature set based on a first audio signal set, and perform a pulse coding operation on the first acoustic feature set to obtain a first pulse signal; The inference calculation module is used to take the first pulse signal as an input signal, perform inference operations using a preset pulse neural network, and generate a second pulse signal. The post-processing module is used to perform decoding operations on the second pulse signal and generate azimuth prediction results.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store a first audio signal set, a first acoustic feature set, and a first pulse signal; The processor is configured to execute the method of any one of claims 1 to 6 by invoking a first set of audio signals, a first set of acoustic features, and a first pulse signal.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images