Beam forming method based on pulse neural network, sound source tracking method, chip and electronic device

Abstract

The application discloses a kind of pulse neural network, sound source tracking method and chip, to solve the problems of existing direction of arrival identification calculation complexity, precision is not high, real-time and poor noise immunity, the application is through a kind of pulse neural network, pulse neural network includes beam forming layer, the beam forming layer carries out beam forming;By maximizing the power of signal after beam forming, the beam forming vector corresponding to different angle space is calculated under different frequency channels;Based on all beam forming vectors, the weight matrix of the beam forming layer is obtained.The application utilizes the weight matrix established in advance in the beam forming layer of pulse neural network to form beam forming, improves the efficiency of beam forming, can reduce power consumption while ensuring the accuracy of direction of arrival estimation, has better robustness, faster processing speed.The application is suitable for brain-like computing field.

Description

Technical Field

[0001] This invention relates to spiking neural networks, sound source tracking methods, chips, and electronic devices, specifically to spiking neural networks (SNNs) for low-power, low-cost, and high-precision direction-of-arrival identification, sound source tracking methods, chips, and electronic devices. Background Technology

[0002] Sound localization is an instinct evolved from biological processes, enabling the rapid and effective identification of the direction of a sound source of interest in complex environments such as noisy conditions. With the development of artificial intelligence technology, biomimetic machine vision and machine hearing are finding wide application in cutting-edge fields such as video conferencing, intelligent robots, smart homes, quality video surveillance systems, and the Internet of Things.

[0003] Because the distance and direction between the sound source and each microphone in the microphone array are different, but each microphone in the array may receive the speech signal from that sound source, and because the sound source moves, room reverberation, interference from other sound sources, and noise (including but not limited to environmental noise and / or internal noise of electronic devices) inevitably degrade the quality of the speech signal, speech intelligibility, and the accuracy of sound source localization, current sound source localization technology is not biomimetic and lacks high sensitivity and robustness. These factors increase the difficulty and real-time nature of sound source localization, affect audiovisual effects, and degrade the performance of electronic devices that use voice as an interaction method. Therefore, after determining the location of the sound source, it is usually necessary to perform speech signal noise reduction, sound source separation, and other processing.

[0004] Most existing sound source localization / direction methods rely on algorithms such as singular value decomposition (SVD), subspace, beamforming, and generalized cross-correlation phase transform to improve the accuracy of sound source methods. Although these techniques have high precision, they require high computing performance from the equipment. The complex calculations not only consume a lot of storage resources and power, but are also difficult to implement in low-power hardware.

[0005] Moreover, existing sound source localization (SSL) based on deep learning artificial neural networks (ANN or RNN, etc.) lacks the internal dynamics of the neural network, is not biomimetic / intelligent enough, and its real-time performance needs improvement. In addition, it has high energy consumption and storage space requirements, and is mainly used for high-computing-power terminals connected to the network, which is not suitable for edge computing and Internet of Things scenarios.

[0006] If sound localization can be achieved on silicon in a biological or biomimetic manner, while achieving the sound source identification accuracy of conventional solutions, and improving the real-time performance, sensitivity, and noise resistance of sound source localization technology, and if it can be implemented and applied in low-cost and easy-to-use low-power hardware, it will be a major advancement in the commercial application of machine hearing in the field of edge intelligent computing. Summary of the Invention

[0007] To solve or alleviate some or all of the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A spiking neural network includes a beamforming layer for beamforming; wherein the beamforming layer is a layer in the spiking neural network; by maximizing the power of the signal after beamforming, beamforming vectors corresponding to different frequency channels and different angular spaces are calculated; and based on all beamforming vectors, a weight matrix of the beamforming layer is obtained.

[0009] In one embodiment, beamforming is performed by extracting the non-DC component from the input pulse of the beamforming layer or the signal after low-pass filtering of the input pulse of the beamforming layer based on the weight matrix, and the resulting beamforming signal is a zero DC signal.

[0010] In one embodiment, the calculated weight matrix is ​​used as the initial weights of the beamforming layer in the spiking neural network; based on the initial weights, the network configuration parameters are continuously adjusted during training to obtain the optimal configuration parameters that meet the accuracy requirements.

[0011] In one embodiment, the input pulses of the beamforming layer are low-pass filtered using synapses and / or neuronal membranes to obtain a filtered signal.

[0012] In one type of embodiment, the power is instantaneous power or average power.

[0013] In one embodiment, activity detection is performed on the beamforming signal or before acquiring the input pulse of the pulse neural network: when activity detection is performed on the beamforming signal, the beamforming signal component in the active frequency channel is identified; the pulse neural network determines the direction of arrival based on the beamforming signal component in the active frequency channel; when activity detection is performed before acquiring the input pulse of the pulse neural network, the signal components of each sensor in the active frequency channel are identified; pulse coding is performed on the signal components of each sensor in the active frequency channel to obtain the input of the pulse neural network in the active frequency channel, thereby obtaining the input pulse of the beamforming layer in the active frequency channel.

[0014] In one embodiment, the pulse neural network identifies the highest power of the beamforming signal, and the angular space corresponding to the highest power of the beamforming signal is the direction of arrival.

[0015] In one embodiment, the pulse neural network identifies the highest power of the beamforming signal component in the active frequency channel, and the angular space corresponding to the highest power is the direction of arrival.

[0016] In one embodiment, based on the pulse firing rate of the output layer neurons of the spiking neural network, the angular space corresponding to the output layer neuron with the maximum pulse firing rate is the direction of arrival.

[0017] In one type of embodiment, the spiking neural network is used to identify the direction of arrival of at least one of the following signals:

[0018] Electromagnetic waves, seismic waves, radar, physiological signals, or voice signals.

[0019] A sound source tracking method includes determining the target sound source direction of sound source data using a spiking neural network as described above; and performing sound source tracking based on the target sound source direction of the sound source data.

[0020] A first type of chip, the chip comprising a spiking neural network as described above.

[0021] In one embodiment, the chip is a neuromorphic chip.

[0022] An electronic device of the first type, comprising a spiking neural network as described above, or comprising a chip as described previously.

[0023] A beamforming method, the method comprising:

[0024] The beamforming method is applied to a pulse neural network;

[0025] The spiking neural network includes a beamforming layer, which is one layer in the spiking neural network;

[0026] By utilizing the weights of the beamforming layer, the input pulses of the beamforming layer or the input pulses of the beamforming layer after low-pass filtering are used to perform beamforming (i.e., weighting) to obtain the beamforming signal.

[0027] In one embodiment, the input pulses of the beamforming layer are low-pass filtered using synapses and / or neuronal membranes to obtain a filtered signal.

[0028] In one embodiment, beamforming vectors corresponding to different angular spaces under different frequency channels are calculated based on sample data; the frequency channel is one of multiple frequency channels obtained by channel decomposition of the signal received by any sensor.

[0029] Based on all the beamforming vectors, the weight matrix of the beamforming layer is obtained.

[0030] In one embodiment, based on the weight matrix, the non-DC component in the input pulse of the beamforming layer or the signal after low-pass filtering of the input pulse of the beamforming layer is extracted for beamforming, and the resulting beamforming signal is a zero DC signal.

[0031] In one embodiment, the beamforming signals of the sample data are linearly or nonlinearly clustered to calculate the beamforming vectors corresponding to different frequency channels and different angular spaces.

[0032] In one type of implementation, linear clustering is performed based on the power of the beamforming signal that maximizes the sample data.

[0033] In one type of embodiment, an autocorrelation matrix is ​​constructed based on the power of the beamforming signal from the sample data;

[0034] The autocorrelation matrix is ​​subjected to eigenvalue decomposition to obtain the eigenspace of the autocorrelation matrix;

[0035] The maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix is ​​obtained by maximizing the power of the beamforming signal of the sample data.

[0036] Based on the autocorrelation matrix and the maximum eigenvalue, the optimal beamforming vector is obtained.

[0037] In one embodiment, singular value decomposition is performed on the autocorrelation matrix to obtain the singular values ​​of the autocorrelation matrix and the eigenvectors corresponding to the singular values.

[0038] The eigenspace of the autocorrelation matrix is ​​obtained based on the singular values ​​of the autocorrelation matrix and the eigenvectors corresponding to the singular values.

[0039] In one embodiment, the mapping relationship between singular values ​​and eigenvalues ​​is obtained based on the eigenspace of the autocorrelation matrix and the singular values ​​of the autocorrelation matrix.

[0040] Based on the beamforming vector constraint, the mapping relationship between the singular values ​​and eigenvalues ​​is solved to obtain the maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix.

[0041] In one type of embodiment, the power is instantaneous power or average power.

[0042] In one type of implementation, during nonlinear clustering, sample data is mapped to a feature space;

[0043] In the feature space, linear clustering is performed on the mapped sample data.

[0044] In one embodiment, for a single frequency channel, the optimal beamforming vector of the sensor array in all angular spaces is calculated to obtain the beamforming submatrix of the sensor array in a single frequency channel.

[0045] The single frequency channel is one of multiple frequency channels obtained by channel decomposition of the signal received by any sensor;

[0046] The weight matrix is ​​obtained based on the beamforming submatrix of the sensor array across all frequency channels.

[0047] In one embodiment, the beamforming layer is the first layer of a spiking neural network.

[0048] A direction-of-arrival (DOA) identification method, the method comprising the beamforming method described above, and obtaining a beamforming signal;

[0049] The pulse neural network obtains the direction of arrival based on the beamforming signal.

[0050] In one type of embodiment, the spiking neural network is deployed with optimal configuration parameters obtained through training, or with optimal configuration parameters obtained through training after quantification.

[0051] During training, the calculated weight matrix of the beamforming layer is used as the initial weight of the beamforming layer (in the training network); based on the initialization, the network configuration parameters are continuously adjusted during training to obtain the optimal configuration parameters that meet the accuracy requirements.

[0052] In one embodiment, the pulse neural network identifies the highest power of the beamforming signal, and the angular space corresponding to the highest power of the beamforming signal is the direction of arrival.

[0053] In one embodiment, the angular space corresponding to the output layer neuron of the spiking neural network with the maximum pulse firing rate is the direction of arrival (ROA).

[0054] In one type of embodiment, activity detection is used to identify beamforming signals in active frequency channels;

[0055] The pulse neural network obtains the direction of arrival based on the beamforming signal components in the active frequency channel.

[0056] In one type of embodiment, the method of identifying beamforming signals in an active frequency channel using activity detection includes one of the following:

[0057] i) Perform activity detection on the beamforming signal to obtain the beamforming signal in the active frequency channel;

[0058] ii) Before acquiring the input pulse of the spiking neural network, perform activity detection to identify the signal components of each sensor in the active frequency channel;

[0059] The signal components of each sensor in the active frequency channel are pulse encoded to obtain the input of the pulse neural network in the active frequency channel, and then the input pulse of the beamforming layer in the active frequency channel is obtained.

[0060] Based on the weight matrix of the beamforming layer, beamforming is performed on the input pulse of the beamforming layer in the active frequency channel or the signal after low-pass filtering of the input pulse of the beamforming layer, to obtain the beamforming signal in the active frequency channel.

[0061] In one type of embodiment, the preset condition is that the energy or / and the average energy value are maximized.

[0062] In some embodiments, training is performed using online learning rules or offline learning rules.

[0063] In one type of implementation, training is performed using floating-point numbers or quantized data;

[0064] The quantified data refers to the quantified membrane potential and / or quantified synaptic current.

[0065] A direction of arrival (DOA) identification device, the DOA identification device comprising a spiking neural network,

[0066] The pulse neural network includes a beamforming layer, and beamforming is performed in the pulse domain based on the beamforming layer. The beamforming layer is a layer in the pulse neural network.

[0067] Using the weights of the beamforming layer, the input pulses of the beamforming layer or the processed input pulses of the beamforming layer are used to form beams to obtain a beamforming signal;

[0068] The pulse neural network obtains the direction of arrival based on the beamforming signal.

[0069] In one embodiment, beamforming vectors corresponding to different directions of arrival in different frequency channels are calculated based on sample data; the frequency channel is one of multiple frequency channels obtained after channel decomposition of the signal received by any sensor; and the weight matrix of the beamforming layer is obtained based on all the beamforming vectors.

[0070] In one embodiment, the beamforming signals of the sample data are clustered linearly or nonlinearly to obtain beamforming vectors of the training network at different frequency channels and corresponding to different angular spaces.

[0071] In one embodiment, the linear clustering is achieved by maximizing the power of the beamforming signal of the sample data.

[0072] In one embodiment, based on the weight matrix, the non-DC component in the input pulse of the beamforming layer or the signal after low-pass filtering of the input pulse of the beamforming layer is extracted for beamforming, and the resulting beamforming signal is a zero DC signal.

[0073] In one type of embodiment, an autocorrelation matrix is ​​constructed based on the power of the beamforming signal from the sample data;

[0074] The autocorrelation matrix is ​​subjected to eigenvalue decomposition to obtain the eigenspace of the autocorrelation matrix;

[0075] The maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix is ​​obtained by maximizing the power of the beamforming signal of the sample data.

[0076] Based on the autocorrelation matrix and the maximum eigenvalue, the optimal beamforming vector is obtained.

[0077] In one embodiment, singular value decomposition is performed on the autocorrelation matrix to obtain the singular values ​​of the autocorrelation matrix and the eigenvectors corresponding to the singular values.

[0078] The eigenspace of the autocorrelation matrix is ​​obtained based on the singular values ​​of the autocorrelation matrix and the eigenvectors corresponding to the singular values.

[0079] In one embodiment, the mapping relationship between singular values ​​and eigenvalues ​​is obtained based on the eigenspace of the autocorrelation matrix and the singular values ​​of the autocorrelation matrix.

[0080] Based on the beamforming vector constraint, the mapping relationship between the singular values ​​and eigenvalues ​​is solved to obtain the maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix.

[0081] In one embodiment, beamforming is performed by extracting the non-DC component from the input pulse of the beamforming layer or the signal after low-pass filtering of the input pulse of the beamforming layer based on the weight matrix obtained from the optimal beamforming vector, and the resulting beamforming signal is a zero DC signal.

[0082] In one type of implementation, during nonlinear clustering, sample data is mapped to a feature space;

[0083] In the feature space, linear clustering is performed on the mapped sample data.

[0084] In one embodiment, for a single frequency channel, the optimal beamforming vector of the sensor array in all angular spaces is calculated to obtain the beamforming submatrix of the sensor array in a single frequency channel.

[0085] The single frequency channel is one of multiple frequency channels obtained by channel decomposition of the signal received by any sensor;

[0086] The weight matrix is ​​obtained based on the beamforming submatrix of the sensor array across all frequency channels.

[0087] In some embodiments, the spiking neural network is deployed with optimal configuration parameters obtained through training, or with quantified optimal configuration parameters obtained through training.

[0088] During training, the calculated weight matrix of the beamforming layer is used as the initial weight of the beamforming layer in the training network. Based on the initialization, the network configuration parameters are continuously adjusted during training to obtain the optimal configuration parameters that meet the accuracy requirements.

[0089] In one embodiment, the pulse neural network identifies the highest power of the beamforming signal, and the angular space corresponding to the highest power of the beamforming signal is the direction of arrival.

[0090] In one embodiment, the angular space corresponding to the output layer neuron of the spiking neural network with the maximum pulse firing rate is the direction of arrival (ROA).

[0091] In one type of embodiment, activity detection is used to identify beamforming signals in active frequency channels;

[0092] The pulse neural network obtains the direction of arrival based on the beamforming signal components in the active frequency channel.

[0093] In one type of embodiment, the method of identifying beamforming signals in an active frequency channel using activity detection includes one of the following:

[0094] i) Perform activity detection on the beamforming signal to obtain the beamforming signal in the active frequency channel;

[0095] ii) Before acquiring the input pulse of the spiking neural network, perform activity detection to identify the signal components of each sensor in the active frequency channel;

[0096] The signal components of each sensor in the active frequency channel are pulse encoded to obtain the input of the pulse neural network in the active frequency channel, and then the input pulse of the beamforming layer in the active frequency channel is obtained.

[0097] Based on the weight matrix of the beamforming layer, beamforming is performed on the input pulse of the beamforming layer in the active frequency channel or the signal after low-pass filtering of the input pulse of the beamforming layer, to obtain the beamforming signal in the active frequency channel.

[0098] In one embodiment, the linear beamforming layer is the first layer of the spiking neural network.

[0099] In one type of embodiment, the power is instantaneous power or average power.

[0100] In one type of implementation, the network is trained using floating-point or quantized data;

[0101] The quantified data refers to the quantified membrane potential and / or quantified synaptic current.

[0102] In one embodiment, the direction of arrival is the direction of the sound source.

[0103] In one embodiment, the input pulses of the beamforming layer are low-pass filtered using synapses and / or neuronal membranes to obtain a filtered signal.

[0104] In one embodiment, the direction of arrival is the direction of the sound source.

[0105] A second type of chip, the chip including the aforementioned direction of arrival identification device.

[0106] A second type of electronic device, which includes the aforementioned direction of arrival identification device, or includes the aforementioned chip.

[0107] Some or all of the embodiments of the present invention have the following beneficial technical effects:

[0108] 1) This invention realizes an efficient beamforming technology that can use signal processing in spiking neural networks (SNNs) to extract the speaker's DoA, improve the efficiency of sound source localization, and has better real-time performance, robustness and angular resolution, providing a low-power solution for conventional subspace localization.

[0109] 2) This invention combines subspace positioning technology with pulse neural networks to generate beamforming vectors. When performing all processing in the pulse domain, it reaches the theoretical limit of conventional subspace methods, and the calculation is simple, saving computational costs.

[0110] 3) The subspace technique implemented in this invention extracts the vector signal v. nBeamforming is performed using the optimal non-DC eigenvector of [t], resulting in a beamforming signal with zero DC. Linear and / or nonlinear clustering methods are then applied to identify the DoA direction.

[0111] 4) In beamforming design, we can use a simple linear neuron model and use power maximization as the metric for beamforming design. Furthermore, even considering factors such as pulse generation and membrane potential resetting that cause neuronal nonlinearity, we can use the peak value generated by the pulse generation rate as a metric, which is proportional to the square of the average power.

[0112] 5) The beamforming method proposed in this invention is used to find the initialization of the first layer weight matrix in the SNN. The SNN is trained based on the initial weights to adjust the orientation results, which has better accuracy and sensitivity.

[0113] 6) High-frequency positioning is more accurate than low-frequency positioning. By performing activity detection on the beamforming signal in each angular space, the target beamforming signal with the maximum average power is obtained. The target angular space corresponding to the target beamforming signal is identified to determine the direction of the target sound source. This transforms the direction of arrival identification problem into an angular space classification identification problem, further improving processing speed and accuracy.

[0114] 7) Based on the algorithm-hardware co-design concept, this invention realizes the hardware implementation of the algorithm. In the test after adding a simple smoothing filter, the actual chip test results are basically consistent with the theoretical simulation results. It is more effective in positioning than the existing methods, with less jitter and better accuracy.

[0115] Further beneficial effects will be described in some embodiments.

[0116] The technical solutions / features disclosed above are intended to summarize the technical solutions and features described in the Detailed Embodiments section, and therefore the scope of the description may not be entirely the same. However, these new technical solutions disclosed in this section are also part of the numerous technical solutions disclosed in this invention document. The technical features disclosed in this section, together with the technical features disclosed in the subsequent Detailed Embodiments section and some contents in the drawings not explicitly described in the specification, disclose more technical solutions in a reasonable combination.

[0117] The technical solution formed by combining all the technical features disclosed at any position in this invention is used to support the summary of the technical solution, the modification of the patent document, and the disclosure of the technical solution. Attached Figure Description

[0118] Figure 1 This is a schematic diagram of the direction of arrival in a circular array provided in an embodiment of the present invention;

[0119] Figure 2 This is a schematic flowchart of the direction of arrival identification method provided in an embodiment of the present invention;

[0120] Figure 3 This is a schematic diagram of frequency channel decomposition provided in an embodiment of the present invention;

[0121] Figure 4 This is a schematic diagram of the structure of the spiking neural network provided in an embodiment of the present invention;

[0122] Figure 5 This is a schematic diagram of the beamforming layer provided in an embodiment of the present invention;

[0123] Figure 6 This is a schematic diagram of q(λ) monotonically decreasing provided in the embodiments of the present invention;

[0124] Figure 7 This is a schematic diagram of beamforming provided in an embodiment of the present invention;

[0125] Figure 8 This is a schematic diagram of beamforming signals for different frequency channels provided in the embodiments of the present invention;

[0126] Figure 9 This is a schematic diagram of the beamforming processing flow provided in an embodiment of the present invention;

[0127] Figure 10 This is a schematic diagram of the direction-of-arrival identification processing flow provided in an embodiment of the present invention;

[0128] Figure 11 This is a schematic diagram illustrating the mapping relationship between the ratio of the beamforming signal amplitude to the discharge threshold and the pulse rate provided in an embodiment of the present invention.

[0129] Figure 12 These are the test results for wave arrival mode identification in low-frequency channels provided in this embodiment of the invention;

[0130] Figure 13 These are the test results for wave arrival mode identification under high-frequency channels provided in this embodiment of the invention;

[0131] Figure 14 This is a schematic diagram comparing the performance of the sound source localization method provided in the embodiments of the present invention with that in the prior art;

[0132] Figure 15 This is a schematic flowchart of the sound source signal separation method provided in an embodiment of the present invention;

[0133] Figure 16 This is a flowchart illustrating the sound source tracking method provided in an embodiment of the present invention;

[0134] Figure 17The results are test findings of a low-power hardware-implemented neuromorphic chip provided in this embodiment of the invention performing sound source tracking in a low-frequency channel.

[0135] Figure 18 The results show the test results of a low-power hardware-implemented neuromorphic chip for sound source tracking in a high-frequency channel, as provided in this embodiment of the invention. Detailed Implementation

[0136] Since it is impossible to exhaustively describe all alternative solutions, the key points of the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Other technical solutions and details not disclosed in detail below generally belong to technical objectives or features that can be achieved by conventional means in the art, and due to space limitations, they will not be described in detail here.

[0137] Unless it refers to division, the " / " in any position in this invention represents logical "OR". The serial numbers "first", "second", etc., in any position in this invention are merely descriptive distinguishing marks and do not imply an absolute temporal or spatial order, nor do they imply that terms prefixed with such serial numbers necessarily refer to different things than the same terms prefixed with other modifiers.

[0138] This invention describes various key points used to combine into various specific embodiments, which will be incorporated into various methods and products. In this invention, even if a key point is described only when introducing a method / product solution, it means that the corresponding product / method solution also explicitly includes that technical feature.

[0139] The description of the existence or inclusion of a step, module, or feature at any location in this invention does not imply that such existence is exclusive or unique. Those skilled in the art can obtain other embodiments by supplementing the technical solutions disclosed in this invention with other technical means. The embodiments disclosed in this invention are generally for the purpose of disclosing certain embodiments, but this does not imply that opposite embodiments are excluded by this invention. As long as such opposite embodiments at least solve one of the technical problems of this invention, they are intended to be covered by this invention. Based on the key points described in the specific embodiments of this invention, those skilled in the art can substitute, delete, add, combine, or change the order of certain technical features to obtain a technical solution that still follows the concept of this invention. These solutions that do not depart from the technical concept of this invention are also within the protection scope of this invention.

[0140] Explanation of some important terms and symbols:

[0141] Neuromorphic chips, also known as brain-like chips, are event-driven, meaning they only perform calculations or processing when an event occurs. They achieve ultra-high real-time performance and ultra-low power consumption in their hardware circuitry. Based on type, neuromorphic chips are divided into those based on analog, digital, or mixed-signal circuits.

[0142] Spiking neural networks (SNNs) are a type of neuromorphic chip and represent the third generation of artificial neural networks. They possess rich spatiotemporal dynamics, diverse encoding mechanisms, and event-driven characteristics, resulting in low computational cost and low power consumption. Compared to artificial neural networks (ANNs), SNNs are more biomimetic and advanced. Brain-inspired computing or neuromorphic computing based on SNNs exhibits superior performance and computational overhead compared to traditional AI chips. It should be noted that this invention does not specifically limit the type of spiking neural network. Any neural network driven by pulse signals or events can be applied to the direction-of-arrival (DOA) recognition method provided in this invention. SNNs can be constructed according to actual application scenarios, such as combining spiking convolutional neural networks (SCNNs), spiking recurrent neural networks (SRNNs), and long short-term neural networks (LSTMs).

[0143] Direction of Arrival (DoA): The directional angle at which the audio signal from a sound source arrives at the microphone array. For sound sources with different DoAs, the delay in the audio signal reaching the microphone array varies. For example, a circular microphone array will be used as an example. Figure 1 As shown, Figure 1 This is a schematic diagram of the direction of arrival provided in an embodiment of the present invention. For a sound source with a defined direction of arrival, the relative delay time of its output audio signal reaching microphones at different positions in the microphone array is different.

[0144] Beamforming: Define {x(t-τ)} i ): i∈[M]} is the signal received at the microphone, where τ i =τ i(n) represents the relative delay time at microphone i∈[M], which is a function of the DoA of the audio signal n. In beamforming, different DoAs (called spatial matched filtering) can be obtained by weighting and delaying the accumulated received signals at different microphones. The DoA with the maximum power can then be found, which is the direction of the target sound source. For example, the direction of the target sound source can be determined by algorithms such as Delay and Sum, Minimum Variance Distortionless Response (MVDR), and SRP-PHAT.

[0145] In broadband applications, conventional methods typically involve applying beamforming in the frequency domain, decomposing the input spectrum into several narrowband channels, applying the narrowband beamforming method described above to each channel, and combining the results to obtain broadband beamforming and DoA estimation. Alternatively, beamforming can be applied in the time domain, delaying the input signals on different channels to achieve time synchronization, and then combining these results to estimate the DoA of the input signal.

[0146] This invention designs a highly efficient beamforming technique that performs beamforming in the pulse domain and utilizes signal processing in a spiking neural network (SNN) to extract the direction of arrival (DoA) of the sound source. Furthermore, to further improve positioning accuracy, this invention combines a subspace method with a data / event-driven SNN method. This achieves the positioning accuracy of conventional subspace methods while improving efficiency, real-time performance, and robustness, and reducing jitter, cost, and power consumption. To facilitate understanding of the technical solution of this invention, the beamforming method, direction of arrival identification method, device, chip, and electronic equipment provided by this invention will be described below in conjunction with practical application scenarios.

[0147] The direction of arrival recognition technology of the present invention is applicable to one-dimensional signals such as audio, electromagnetic waves, seismic waves, and physiological signals. The following text takes sound signals as an example. Accordingly, the microphone that receives the audio is replaced by a sensor or sensor array corresponding to the one-dimensional signal.

[0148] like Figure 2 As shown, Figure 2 This is a schematic flowchart of the direction of arrival (DOA) identification method provided in an embodiment of the present invention. The DOA identification method can be applied to electronic devices with audio information processing capabilities, or to chips with audio information processing functions; it can also be applied to electronic devices equipped with chips having audio information processing functions. Specifically, the DOA identification method includes at least steps S200 to S300:

[0149] Step S200: The beamforming layer based on the pulse neural network performs beamforming on the input pulse to obtain the beamforming signal.

[0150] This invention applies beamforming methods to SNNs (Short-Side Neural Networks). The beamforming layer of the SNN performs beamforming on the input pulses to obtain a beamforming signal. In the beamforming process of this invention, the input pulses of the beamforming layer, or the low-pass filtered signal of the input pulses, are weighted using the weight matrix of the beamforming layer to obtain the beamforming signal.

[0151] To utilize the signal processing power of SNN, the speech signal collected by the microphone needs to be converted into a combination of pulse features, for example, by converting audio into pulses through pulse coding, which are then used as input to the SNN for processing.

[0152] Considering that the sound source data collected in practical applications is a broadband signal, directly using the sound source received by the microphone to generate a pulse signal may reduce the accuracy of the direction-of-arrival (DOA) identification result. Therefore, to improve the accuracy of DOA identification, in some embodiments of this invention, after each microphone receives the sound source data, the sound source data is decomposed into multiple parallel frequency channels. These parallel channels are filtered according to frequency bands and the signal activity changing over time in different frequency bands is detected. Pulse coding is then performed on the decomposed sound source data to obtain the pulse signal of the sound source data received by any microphone. Optionally, channel decomposition of the sound source data can be performed using a filter bank, such as a bandpass filter bank or a narrowband filter bank.

[0153] Figure 3 This is a schematic diagram of frequency channel decomposition provided in an embodiment of the present invention. The received sound source data to be processed is decomposed into multiple frequency channels by a bandpass filter. Each frequency channel can only process or pass the sound source component corresponding to the center frequency of the bandpass filter in that frequency channel.

[0154] Figure 4 This is a schematic diagram of a spiking neural network provided in a preferred embodiment of the present invention, comprising multiple layers, each containing multiple neurons. In this preferred embodiment, the beamforming layer is the first layer of the spiking neural network. Of course, the beamforming layer can also be the second layer or other intermediate layers in an SNN network, and the present invention is not limited thereto. For simplicity, the following description uses the beamforming layer as the first layer of the SNN as an example. In this case, the input pulse can be the pulse signal of the sound source data to be processed, or it can be the pulse signal of the sound source data to be processed after low-pass filtering.

[0155] Let the input pulse of the SNN be S{c, m, t} or S c[m, t], where c represents the frequency channel, c∈N, N is the number of multiple frequency channels obtained after channel decomposition of the signal received by the sensor; m represents the microphone index, m∈M, M is the number of microphones in the microphone array; t represents the time step or timestamp of the pulse signal.

[0156] In SNN, the pulse signal S of the sound source data to be processed received by each microphone in the microphone array. c [m, t] is low-pass filtered through the synapse and / or neuronal cell membrane to obtain a low-pass filtered pulse signal. The synaptic response is... The impulse response of the cell membrane is Where τ s and τ m Let c represent the time constants of the synapse and cell membrane, respectively; c∈N represent the number of frequency channels; m∈M represent the microphone index; and t∈[T] represent the time step in the peak pulse signal S. A low-pass filter for the pulse signal is established based on the synaptic and / or membrane response. The pulse signal of the sound source data to be processed is then subjected to low-pass filtering according to the low-pass filter to obtain the filtered signal R. c [m, t], where R c [m, t] is related to S c The filtered pulse signal corresponding to [m, t].

[0157] In some embodiments, the synaptic response and membrane response are considered as a normalized joint impulse response h. sm [t], where, Based on the normalized joint impulse response h sm [t], at this time, the filtered signal R c [m, t] = h sm [t]*S c [m, t], the filtered signal R c [m, t] includes the DC component, the first, second, and higher harmonics.

[0158] Unlike existing non-pulse domain beamforming methods, the beamforming method provided in this invention is applied to a spiking neural network (SNN). It performs beamforming by weighting the pulse signals of the sound source data input to the SNN based on the existing weight matrix in the beamforming layer. Furthermore, this invention uses a novel subspace technique in the SNN. Utilizing the weight matrix obtained during training, it extracts the non-DC components or feature vectors from the input pulses of the beamforming layer or the low-pass filtered (synaptic and / or membrane potential) signals of the input pulses of the beamforming layer for beamforming. The resulting beamforming signal is a zero-DC signal.

[0159] Because when the pulse code is long enough to contain a large number of cycles, R cThe DC component of [m, t] is identical for all frequency channels c∈N and all microphones m∈M. This DC component does not transmit any S values ​​at different frequency channels c∈N. c The relative timing information [m, t] (i.e., DoA information), therefore the DoA information is located in R. c In higher harmonics of [m, t]. Especially since linear filtering preserves time information, i.e., h sm [t]*S c [m, t-δ]=(h sm *S c [m, t-δ], the higher harmonics in the filtered signal retain the relative time information of the signals received at different microphones.

[0160] Synaptic time constant τ s and cell membrane time constant τ m Adjustments can be made as needed. In a preferred embodiment, to preserve the relative intensity of different harmonics (including very strong DC components) across all channels c∈N, different low-pass filters h are designed for different frequency channels. sm [t], such that τ ∈ N for all frequency channels c∈N s *f channelc They are all the same, and τ makes all frequency channels c∈N the same. m *f channelc They are all the same.

[0161] In SNN beamforming, the input pulses of the beamforming layer or the low-pass filtered signal of the input pulses of the beamforming layer are weighted based on the weight matrix obtained during training to obtain the beamforming signal. At this time, the pulse signal of the sound source signal to be processed is guided to the preset angular space to obtain the beamforming signal in the preset space.

[0162] The weight matrix parameters of the beamforming layer are obtained through training, which includes: obtaining beamforming vectors of the SNN network (training network) at different frequencies and corresponding to different angular spaces based on sample data; and obtaining the weight matrix of the beamforming layer based on all beamforming vectors.

[0163] The weight matrix of the beamforming layer includes beamforming vectors at different frequencies and corresponding to different directions of arrival. In a preferred embodiment, for ease of differentiation, the weight matrix is ​​divided into multiple beamforming sub-matrices at a single frequency channel based on a specific frequency. The single frequency channel is one of multiple frequency channels obtained after channel decomposition of the signal received by any sensor. Simultaneously, the beamforming sub-matrix at any single frequency channel includes beamforming vectors at that frequency and corresponding to different directions of arrival. The above classification of the weight matrix is ​​merely an example, and the invention is not limited thereto. For example, it can be classified based on the direction of arrival or angular space direction. Based on a specific angular space, the weight matrix is ​​divided into multiple beamforming sub-matrices at a single angular space, and any beamforming sub-matrix includes beamforming vectors at multiple frequencies corresponding to that angular space. In other parts of the invention, the example of a weight matrix including multiple beamforming sub-matrices at a single frequency channel, and each beamforming sub-matrix including beamforming vectors at that single frequency and corresponding to different directions of arrival, is used for description. The rest of this paper details how the weight matrix of the beamforming layer is obtained during training to apply the subspace method to DoA estimation.

[0164] The pulse code corresponding to the sound source data received from a angular space (DoA n) is represented as follows: right After low-pass filtering, the pulse-code filtered signal is obtained.

[0165] As mentioned earlier, for simplicity, under a specific frequency channel or active frequency channel c, where c∈N, the filtered signal of the input pulse from any microphone m∈M in the microphone array is... It can be viewed as a vector signal v n [t] = R n [., t], where "." represents all possible cases. For M microphones, R n [.,t]=[R n [1, t], ..., R n [m, t], ...R n [M, t]], therefore, v n [t] represents an M-dimensional signal over time. M represents the number of microphones in the microphone array, and n represents the nth angular space.

[0166] In a preferred embodiment, the vector signal v n The DC component and its higher harmonics of [t] are the same across all its components. This invention is not limited thereto and can be adjusted by adjusting the synaptic or / or membrane time constant.

[0167] Optionally, the time constants of the synapse and the cell membrane can be the same. For example, the center frequency of each frequency channel adjusts the time constant of the synapse and / or membrane. For instance, the time constants of the synapse and cell membrane can be determined by 1 / center frequency based on the center frequency of each frequency channel. For example, the time constants of the synapse and cell membrane can be determined by 1 / center frequency based on the center frequency of the active frequency channel where the input pulse is located. Understandably, when the active frequency channel where the input pulse is located changes, the corresponding time constants of the synapse and cell membrane change.

[0168] Optionally, the time constants of the synapse and the cell membrane can be different. For example, the time constants of the synapse and the cell membrane can be adjusted separately based on the center frequency of each frequency channel. For instance, the synapse time constant can be determined by 1 / center frequency, and the cell membrane time constant can be determined by 1 / (2*center frequency). Alternatively, the synapse time constant can be obtained based on the center frequency of each frequency channel through a preset first mapping relationship, and the cell membrane time constant can be obtained based on the center frequency of each frequency channel through a preset second mapping relationship. The first mapping relationship determines the synapse time constant at the center frequency of different frequency channels, and the second mapping relationship determines the cell membrane time constant at the center frequency of different frequency channels. Both the first and second mapping relationships can be mapping data or mapping functions. For example, the time constants of the synapse and the cell membrane can also be determined through network training.

[0169] Due to the vector signal v n The DC component of [t] is the same for all frequency channels c∈N and all microphones m∈M. Therefore, as the sound source moves, DoAn changes, and the relative time information is reflected in the vector signal corresponding to different microphones. The relative timing information of higher harmonics. Therefore, the present invention is based on Beamforming is performed on the higher harmonic components of the signal v. Simultaneously, the DoA estimation problem is treated as a problem in the signal v. n The clustering problem on [t] involves finding a set of direction vectors or a set of unit norm vectors that cluster the signal v. n [t] directs to DoA n (where n is the angular space label).

[0170] In some embodiments of the present invention, the clustering problem can be linear clustering, for example, by using the linear weight matrix of an SNN to cluster the signal v. n [t] performs beamforming, transferring the signal v n [t] is directed to different DoA n, using signal v n[t] The beamforming vector is determined by linearly clustering the maximum average power of the beamforming signals on different DoA n.

[0171] In some embodiments of the present invention, the clustering problem can be nonlinear clustering. For example, the input pulse can be mapped to a certain feature space through the nonlinear mapping of SNN, and the signal is beamformed in the feature space. The signal in the feature space is guided to different DoA n, and linear clustering is performed by the maximum average power of the beamformed signal in the feature space on different DoA n to determine the beamforming vector.

[0172] Optionally, the input pulse v can be decomposed using signal decomposition methods. n The higher harmonic components are extracted from [t], and beamforming is performed based on the higher harmonic components and the weight matrix of the beamforming layer to obtain beamforming signals in different angular spaces.

[0173] Signal decomposition methods include, but are not limited to, pyramid decomposition, wavelet decomposition, contourlet decomposition, non-subsampled contourlet decomposition, and dual-tree complex wavelet decomposition. For example, a filtered signal can be decomposed into high-frequency and low-frequency components using signal decomposition methods, with the high-frequency components identified as higher-order harmonic components.

[0174] Considering that beamforming vectors generated for specific frequency channels cannot achieve good beamforming on other frequency channels, in some embodiments of the present invention, when beamforming input pulses by a beamforming layer based on a pulse neural network, the beamformed signal obtained by beamforming the beamforming submatrix corresponding to the active frequency channel can be determined as the beamforming signal.

[0175] Optionally, the active frequency channel of the pulse signal of the sound source data to be processed can be determined, and the input pulse obtained from the pulse signal of the sound source data to be processed can be beamformed using the linear layer of the pulse neural network and the beamforming sub-matrix corresponding to the active frequency channel to obtain the beamforming signal.

[0176] Optionally, the beamforming sub-matrix corresponding to all frequency channels in the beamforming layer of the pulse neural network can be used to beamform the input pulse to obtain the initial beamforming signal under different frequency channels. Activity detection is performed on the initial beamforming signal under different frequency channels to determine the active frequency channel that meets the preset requirements of the initial beamforming signal, and the initial beam information signal under the active frequency channel is determined as the beamforming signal.

[0177] Specifically, based on the power, energy, or amplitude of the initial beamforming signal in different frequency channels, the activity detection method for the channel components described above can be used to detect the activity of the initial beamforming signal in different frequency channels, thereby determining the active frequency channel that meets the preset requirements. These preset requirements can be maximum power, highest energy, or maximum amplitude.

[0178] In some embodiments of the present invention, the sound source data to be processed may be a speech signal in the current environment during the acquisition process; alternatively, the sound source data to be processed may also be a speech signal in the current environment acquired over a past period of time. The past period of time may be the past 1 second or the past 1 minute, and the embodiments of the present invention do not specifically limit this.

[0179] Optionally, speech signals from the environment can be collected using a microphone array, wherein the microphone array can be, for example, Figure 1 The circular microphone array shown can also be a linear microphone array, a distributed microphone array, or a cross-shaped microphone array. Each microphone array includes at least one microphone.

[0180] In some embodiments of the present invention, the sound source data to be processed can also be obtained by processing the speech signal in the current environment collected by the microphone array. Such processing includes, but is not limited to, filtering, noise reduction, and time-frequency analysis.

[0181] In some embodiments of the present invention, to improve the accuracy of direction-of-arrival (DOA) identification while reducing its power consumption, and to ensure that the DOA identification method can be applied to low-power chips, the embodiments of the present invention use a spiking neural network (SNN) for direction estimation to determine the direction of the sound source. Considering the spiking communication method and rich neuronal dynamics of the SNN, an effective encoding method is needed to encode the sound source data to be processed into pulse signals, ensuring that the SNN can perform ultra-low-power computation and handle time-series tasks.

[0182] Optionally, pulse coding can be performed on the sound source data to be processed using time coding to obtain the pulse signal of the sound source data to be processed. For example, the continuous changes in the sound source data to be processed are compared with a threshold value, and pulses are emitted at the moments when the sound source data changes rapidly to obtain the pulse signal of the sound source data to be processed. Optionally, pulse coding can be performed on the sound source data to be processed using phase coding to obtain the pulse signal of the sound source data to be processed. Optionally, pulse coding can be performed on the sound source data to be processed using stimulus estimation coding to obtain the pulse signal of the sound source data to be processed. In some embodiments, zero-crossing coding is used to pulse code the sound source data to be processed to obtain the pulse signal of the sound source data to be processed. The advantage of this coding method is that it is sensitive to the relative delay of the signals from different microphones, easily captures time difference information, and its DoA estimation performance is not significantly affected in reflection propagation environments or noisy environments.

[0183] In certain preferred embodiments of the present invention, considering that pulse coding is performed on each narrowband signal obtained after channel decomposition, the number of pulse signals will increase, the amount of data that the spiking neural network needs to process will increase, resulting in an increase in the processing time of direction-of-arrival (DOA) recognition and a decrease in the real-time performance of DOA recognition. Furthermore, each of the multiple narrowband signals obtained after channel decomposition has a different frequency range, and processing all channels increases the amount of data and resource consumption. Since the sound signal corresponding to the sound source direction has high energy and a specific frequency, it can be assumed that the energy in the sound source data received by each microphone is mainly concentrated in one or more frequency ranges. In the applicant's earlier application (Chinese patent application No. 202310109065.9), activity detection is performed based on the channel components of the sound source data received by each microphone after channel decomposition to obtain the target frequency (one or more frequencies). The activity detection module performs independent activity detection, joint activity detection, or local joint activity detection. The entire contents of that patent are incorporated herein by reference.

[0184] Step S300: Determine the direction of the target sound source corresponding to the sound source data to be processed based on the beamforming signal.

[0185] In some embodiments of the present invention, the pulse neural network obtains the direction of arrival (DOA) based on the beamforming signal output by the beamforming layer, and determines the DOA as the target sound source direction corresponding to the sound source data to be processed.

[0186] The direction-of-arrival (DoA) identification method provided in this invention combines subspace localization technology with spiking neural networks, treating the DoA estimation problem as a function of vector signal v. n A clustering problem on [t] will identify the signal v. n[t] is considered as a vector signal of DoA n (where n∈N), which is a novel formulation of the localization / orientation problem, and linear or nonlinear clustering methods are applied to identify DoA n.

[0187] In some embodiments of the present invention, a target beamforming signal that meets the preset signal requirements can be determined from the beamforming signals output by the beamforming layer, and the direction of arrival can be determined by determining the target angular space corresponding to the target beamforming signal.

[0188] Optionally, the preset signal requirements could be the maximum signal amplitude, the highest average signal power, or the highest neuron emission power. Understandably, based on the signal amplitude, average signal power, or neuron emission power of multiple beamforming signals, the beamforming signal with the maximum signal amplitude, maximum average signal power, or maximum emission power can be determined as the target beamforming signal.

[0189] In some embodiments of the present invention, considering that the sound signal received in the direction of the target sound source carries more energy than the sound signal in other directions, the beamforming signal with the maximum power can be determined based on the power of the beamforming signal in different angular spaces. The angular space corresponding to the highest power generated by beamforming is the direction of the target sound source, and the beamforming signal with the maximum power is the target beamforming signal. Optionally, the power can be the average power or the instantaneous power.

[0190] In a preferred embodiment, linear weights in the SNN are used to perform linear processing on the vector signal vn[t], and linear clustering is achieved by maximizing the average power of the signal after beamforming.

[0191] Optionally, taking average power as an example, the average power can be the average power of the beamforming signal over the time length corresponding to the time window. For instance, after obtaining beamforming signals in different angular spaces under the active frequency channel, for each angular space beamforming signal, the instantaneous power of the beamforming signal at each time point is determined based on the signal value at each time point in the beamforming signal. The instantaneous power of the beamforming signal at each time point is then averaged to obtain the average power of the beamforming signal.

[0192] Taking each microphone in the microphone array as an example, for each frequency channel, the beamforming submatrix of that frequency channel is obtained. in, This represents the beamforming vector based on the active frequency channel in different angular spaces, according to the input pulse vn[t] and the beamforming sub-matrix of the active frequency channel. The beamforming signal of the input pulse in each angular space under a defined active frequency channel. and through The average power of the beamforming signal in each angular space under this frequency channel is obtained.

[0193] Optionally, the explanation can be based on the average power, where the average power can be the average power of the beamforming signal over the time length corresponding to the time window. For example, the autocorrelation matrix of the pulse signal of the sound source data to be processed is obtained based on the pulse signal of the sound source data to be processed, and the average power of the beamforming signal in each angular space is obtained based on the beamforming vector in each angular space and the autocorrelation matrix of the pulse signal of the sound source data to be processed.

[0194] Optionally, since the average power of the beamforming signal in each angular space under this frequency channel can be determined by... The beamforming signal of the input pulse in each angular space under the active frequency channel is obtained from... Therefore, the autocorrelation matrix of the pulse signal of the sound source data to be processed can be obtained from the pulse signal of the sound source data to be processed. Based on the beamforming vector in each angular space and the autocorrelation matrix of the pulse signal of the sound source data to be processed, the average power of the beamforming signal in each angular space can be obtained.

[0195] In some embodiments of the present invention, the direction of arrival of the sound source corresponding to the sound source data to be processed can be obtained from the target angle space corresponding to the target beamforming vector corresponding to the target beamforming signal, and the direction of arrival of the target sound source corresponding to the sound source data to be processed can be determined from the direction of arrival of the sound source data to be processed.

[0196] Optionally, beamforming vector identification can be performed based on the target beamforming signal to determine the target beamforming vector corresponding to the target beamforming signal. The target angular space where the target beamforming vector is located can be determined as the direction of arrival of the sound source corresponding to the sound source data to be processed. The direction of arrival of the sound source corresponding to the sound source data to be processed can be determined as the direction of the target sound source corresponding to the sound source data to be processed.

[0197] Without compromising positioning accuracy, this solution provides a low-power solution for conventional subspace positioning. Furthermore, it utilizes pre-established weight matrices in the linear layers of a spiking neural network to form beamforming, thereby improving beamforming efficiency. This approach can reduce power consumption while ensuring the accuracy of sound source direction estimation, resulting in better robustness and faster processing speed.

[0198] To improve the reliability of pulse coding and provide higher-precision pulse information, thereby improving the accuracy of the target sound source location, local maxima in the sound source data to be processed can be determined based on the spectrum diagram of the sound source data. The pulse signal of the sound source data to be processed is then obtained based on the frequency and time information corresponding to the local maxima in the spectrum diagram. Specifically, the pulse coding method based on local maxima includes steps A01 to A04:

[0199] Step A01: Obtain the spectrogram of the sound source data to be processed.

[0200] In some implementations, a fast Fourier transform can be performed on the sound source data to be processed to obtain the spectrum of the sound source data.

[0201] Step A02: Based on the spectral values ​​of each signal point in the spectrum diagram of the sound source data to be processed, select the local maximum value at each time point in the sound source data to be processed.

[0202] Local maxima refer to the peak points with high energy in the sound source data to be processed.

[0203] In some implementations, a sequential filter can be used to filter the spectrogram of the sound source data to be processed, extracting the local maximum value at each time point in the spectrogram. The sequential filter can be a filter bank composed of multiple intermediate filters.

[0204] Step A03: Perform sparse processing on the local maximum values ​​at each time point in the selected sound source data to be processed to obtain the key points of the sound source data to be processed.

[0205] Considering that background noise may exist in the local maxima at each time point in the selected sound source data to be processed, in order to remove this background noise and improve the accuracy of the pulse signal, in some implementations, based on the spectral difference between the spectral value of the local maximum and the mean spectral value of the signal points adjacent to the local maximum in the spectrum diagram, the local maximum with a spectral difference greater than or equal to a preset spectral difference threshold is selected from multiple local maxima, and the selected local maximum with a spectral difference greater than or equal to the preset spectral difference threshold is set as the key point of the sound source data to be processed.

[0206] Optionally, the spectral values ​​of signal points adjacent to local maxima in the spectrum can be signal points within a preset size region centered on the local maximum. For example, the spectral value of the local maximum can be s. i =s(f i , t i Taking a preset size of p*p as an example, t i f iThe time information and frequency value representing local maxima can be obtained through [s] i (t+p, f), ..., s i [t,f+p)] defines a region of size p*p centered on the local maximum. Signal points in the spectrum within this region are defined as signal points adjacent to the local maximum in the spectrum.

[0207] In some implementations, for each local maximum value, if the spectral difference of the local maximum value is less than a preset spectral difference threshold, it indicates that the local maximum value is background noise, and the local maximum value is discarded; if the spectral difference of the local maximum value is greater than or equal to the preset spectral difference threshold, the local maximum value is set as a key point.

[0208] Step A04: Perform pulse encoding based on the time information and frequency values ​​of the key points to obtain the pulse signal of the sound source data to be processed.

[0209] In some implementations, pulse signals of the sound source data to be processed can be generated by time encoding based on the time information and frequency values ​​of key points.

[0210] In some implementations, for each key point, the frequency value of the key point is compared with a preset frequency threshold. If the frequency value of the key point is greater than the preset frequency threshold, the pulse mode of the key point is set to 1. If the frequency value of the key point is less than or equal to the preset frequency threshold, the pulse mode of the key point is set to 0. The pulse time of the pulse mode of the key point is then used to obtain the pulse signal of the sound source data to be processed.

[0211] This invention improves the reliability of pulse coding and the accuracy of classification by applying a sequential filter to the spectrogram, selecting local maxima, and generating pulse signals based on the selected local maxima, thereby improving the accuracy and robustness of sound source direction estimation.

[0212] Considering that beamforming vectors are frequency-dependent—meaning a beamforming vector designed for an input pulse at a specific frequency channel may not produce good beamforming on other frequency channels—to ensure good beamforming, a weight matrix needs to be obtained through training. During training, a finite number of frequency channels must be used, and the beam submatrix for each frequency channel must be calculated to obtain the weight matrix. Specifically, based on linear or nonlinear clustered sample data, beamforming vectors corresponding to different frequency channels and angular spaces are obtained for the training network; based on all beamforming vectors, the weight matrix of the beamforming layer is obtained.

[0213] The beamforming layer is equipped with neurons and a weight matrix, for example, such as... Figure 5As shown, Figure 5 This is a schematic diagram of the beamforming layer provided in an embodiment of the present invention. The synapses and cell membranes of neurons perform normalized joint impulse responses on the input pulse signals. The input pulse signals are then low-pass filtered using the normalized joint impulse responses to obtain a filtered signal. This filtered signal is then transmitted to the input neurons. The filtered signal is beamformed using the weight matrix in the neurons to obtain a beamforming signal. The target beamforming signal with the maximum average power is determined based on the average power of the beamforming signal. Pulse excitation is performed based on the target beamforming signal to obtain a pulse sequence, which is then transmitted to the intermediate layer. The detection layer is used to perform pulse excitation based on the number of pulses in the received pulse sequence to obtain a new pulse sequence, which is then transmitted to the output layer. The output layer is used to make direction decisions based on the new pulse sequence transmitted from the intermediate layer to determine the direction of the target sound source.

[0214] Considering that spiking neural networks on hardware or processors cannot directly make accurate directional decisions based on input pulse signals, and that the neurons and synaptic modules / units are merely hardware implementations, further processing is required. This includes defining the sets of neurons and synaptic modules / units, defining their connections, and defining the weights and corresponding time constants stored in the synaptic circuits. Therefore, pre-training is necessary to obtain the appropriate parameters, such as supervised or unsupervised training, and on-chip or off-chip training. This invention uses supervised training as an example, but is not limited to it. The configuration parameters obtained through training are mapped to hardware, such as a chip. After receiving signals collected from the environment, the chip runs its internal spiking neural network to automatically complete the inference process based on the received signals.

[0215] In some embodiments of the present invention, a beamforming matrix can be constructed based on sample sound source data in different angular spaces. This beamforming matrix is ​​used as the initial weights for the beamforming layer of the training network to initialize the training network. The initialized training network is then trained based on sample sound source data in different angular spaces. During training, the network's configuration parameters are continuously adjusted to obtain the optimal configuration parameters that meet the accuracy requirements. These optimal configuration parameters include weights, synaptic / membrane time constants, etc. For example, pulse signals from each frequency channel in different angular spaces can be used as the input to a spiking neural network, with the DoA direction as the target, to train the initialized training network.

[0216] In some embodiments, the beamforming layer in the training network can be trained using online or offline learning rules. The configuration parameters (such as weights, synaptic / membrane time constants, etc.) are continuously adjusted to obtain optimal or ideal configuration parameters that optimize or improve the predictive or inference accuracy of the training network. Based on these trained configuration parameters, the SNN performs inference, outputting or predicting results that closely match the input samples. Note that the samples (dataset, training set, or test set) are obtained from the acquisition of real-world environmental signals. During online learning, the training network and the inference network are the same network, both executed in hardware (such as a chip). During offline learning, the training network is the SNN on a training device (such as a CPU, GPU, or NPU), and the inference network is the SNN in hardware (such as a chip). Both have the same network structure, and the configuration parameters obtained during training are mapped or deployed to the SNN in the chip, enabling the chip to perform inference or prediction.

[0217] Optionally, for the sample sound source data in each angular space, the audio components of each frequency channel of the sample sound source data are pulse-coded to obtain the sample pulse signal of each frequency channel. The sample pulse signal of each angular space in a frequency channel is input into the synaptic and cell membrane normalized joint impulse response in the beamforming layer of the spiking neural network. The input sample pulse signal is low-pass filtered to obtain the filtered sample signal. Based on the filtered sample signal and the preset optimization model, the beamforming vector corresponding to the frequency channel in different angular spaces is obtained. The beamforming vectors corresponding to each frequency channel in different angular spaces are combined to obtain the beamforming matrix. The beamforming matrix is ​​used as the initial weight matrix of the beamforming layer of the training network. The filtered sample signal is beamformed based on the initial weight matrix of the beamforming layer of the training network to obtain the sample beamforming signal. After the sample beamforming signal is processed based on the intermediate layer of the training network, the sound source direction is estimated by the output layer of the training network, and the predicted angular space corresponding to the sample sound source data is output.

[0218] Then, based on the angular space labels corresponding to the sample sound source data and the preset angular space, the localization loss of the training network is determined. The configuration parameters of the training network are adjusted according to the localization loss. This process is repeated iteratively. When the training network meets the preset convergence condition, the configuration parameters of the training network when the preset convergence condition is met are synchronized to the chip.

[0219] The configuration parameters of the spiking neural network include, but are not limited to, the synaptic weights, firing times, thresholds, decay time constants, synaptic time constants, cell membrane time constants, and beamforming vectors of neurons in each network layer of the SNN. This invention is not limited thereto. The preset convergence condition can be that the localization loss is less than or equal to a preset localization loss threshold, or that the number of iterations is less than or equal to a preset number of iterations threshold.

[0220] Understandably, upon completion of each training round, the spiking neural network obtains the beamforming matrix for the current round of training based on the configuration parameters and sample sound source data obtained in the previous round of training. The beamforming matrix for the current round of training is used as the weight matrix for the current round of the beamforming layer. Beamforming is then performed based on the weight matrix for the current round of the beamforming layer to obtain the sample beamforming signal for the current round.

[0221] In some embodiments of the present invention, taking a circular microphone array as an example, the direction of arrival can be quantized into M angular spaces based on the number M microphones in the microphone array, wherein the angular resolution of each angular space is approximately 1.

[0222] Considering that the array resolution used to characterize the distance at which two targets can be distinguished in the presence of noise is related to the array geometry and spatial size, and that a larger array resolution results in better angular resolution, but when the number of array elements and the overall spatial span of the array are limited, it may lead to grating lobe problems and aliasing effects in the array response vector. That is, for two different DoA vectors n1 and n2, they may have the same array response vector, i.e., a(n1) = a(n2), making it impossible to determine the angle of the sound source and thus impossible to distinguish and find the correct DoA. Therefore, for microphone arrays with a limited number of microphones, this is not ideal. Based on this, to improve the accuracy of the target sound source direction, obtain better array resolution, and reduce the influence of grating lobes, in some embodiments of the present invention, when determining the angular resolution, the angular resolution can be determined according to the type of microphone array, the number of microphones in the microphone array, and a preset oversampling factor, and the angular space can be quantized according to the angular resolution. For example, a circular microphone array is used as an example. When the number of microphones in the microphone array is M and the preset oversampling factor is , by... The direction of arrival is quantized into M angular spaces.

[0223] In an optional implementation, after quantizing the angular space, the audio signal of the sound source in each angular space is acquired to obtain sample sound source data. Based on the angular space label of the sample sound source data and the predicted angular space, the localization loss of the spiking neural network is obtained.

[0224] Optionally, the localization loss of the spiking neural network can be obtained based on the angular space labels of the sample sound source data and the predicted angular space, using a preset loss function. The preset loss function can be a mean square loss function, a cross-entropy loss function, or an MSE surround loss function.

[0225] In one embodiment of the present invention, considering that it is difficult to train the SNN directly, an ANN can be built based on the parameters of the spiking neural network. By training the ANN, the target parameters of the spiking neural network are obtained. Based on the target parameters of the spiking neural network, the parameters of the SNN in the electronic device are adjusted to achieve the training of the spiking neural network.

[0226] Optionally, during ANN training, to ensure the consistency of parameters between the SNN and ANN in the electronic device, the parameters of the ANN can be synchronized to the SNN in the electronic device after each iteration of training. The sample pulse signal is then input into the spiking neural network of the electronic device. After processing by the spiking neural network, the predicted sector corresponding to the sample sound source data is output. Based on the predicted sector and the sector label corresponding to the sample sound source data, the localization loss of the spiking neural network is determined. The parameters of the ANN are adjusted based on the localization loss, and the adjusted parameters of the ANN are synchronized to the SNN in the electronic device.

[0227] Optionally, the training method described in the applicant's prior application (application number: CN202210789977.0) can be used directly for training. The contents of this prior application are incorporated herein by reference in their entirety. The direction-of-arrival (DOA) identification method provided in this embodiment of the invention estimates the direction of the pulse signal of the sound source data to be processed based on a spiking neural network, thereby obtaining the target direction of the sound source data. This method can reduce power consumption while ensuring the accuracy of the sound source direction estimation, exhibiting better robustness and faster processing speed. Furthermore, it combines subspace localization technology with a spiking neural network, utilizing synaptic and membrane potentials in the spiking neural network for low-pass filtering, and extracting the optimal non-DC feature vector for beamforming, further improving the estimation accuracy of the sound source target direction while reducing power consumption.

[0228] Considering that the vector signal obtained after normalizing the joint impulse response includes DC components and higher harmonic components, and that the DC component of vn(t) does not carry time information for sound source location estimation, and that the DC component is the same for the vector signal of each microphone at each time t, it is necessary to eliminate the influence of the DC component on the beamforming vector in the beamforming vector determination. Based on this, in some embodiments of the present invention, the DC component can be eliminated by applying a clustering method, and the beamforming vector of different angular spaces under each frequency channel can be determined to obtain the weight matrix of the linear layer of the SNN.

[0229] Specifically, linear or nonlinear clustering is performed on the sample pulse signals of the sample data to obtain beamforming vectors corresponding to different frequency channels and different angular spaces. The beamforming vectors are divided based on each frequency channel to obtain the beamforming sub-matrix of the sensor array in a single frequency channel. By connecting the beamforming sub-matrixes of each sensor in all frequency channels, the weight matrix is ​​obtained.

[0230] Linear clustering involves finding a set of direction vectors (such as beamforming vectors in this invention) or weight vectors (such as weight matrices) that form a linear line or plane, using this linear line or plane to separate the sample pulse signal into two or more segments. Nonlinear clustering uses a nonlinear shape to separate the sample pulse signal into two or more segments. In essence, linear clustering is linear beamforming based on the sample pulse signal from the sample data, while nonlinear clustering is nonlinear beamforming based on the sample pulse signal from the sample data.

[0231] Optionally, taking linear clustering as an example, linear clustering of the sample pulse signals of the sample data can yield beamforming vectors for the training network at different frequency channels and corresponding to different angular spaces in various ways. Examples include:

[0232] (1) Taking the sample pulse signal of a sample sound source located in an angular space as an example, the wavelength of the audio component of the sample sound source data in the angular space in the frequency channel, the incident angle of the angular space, and the number of microphones in the microphone array are used to obtain the steering vector a(θ) of each microphone in the microphone array in the angular space. Based on the steering vector in the angular space, through... The weight vectors of each microphone in the angular space are obtained, and the weight vectors of each microphone in the angular space are determined as the beamforming vectors of the angular space.

[0233] (2) The steering vector a(θ) of each microphone in the microphone array in the angular space can be obtained based on the wavelength of the audio component of the sample sound source data in the frequency channel, the incident angle of the angular space, and the number of microphones in the microphone array. Based on the autocorrelation matrix R of the sample vector signal in the angular space and the initial weight vector, the average output power of the noise component in the initial sample beamforming signal guided by the sample vector signal in the angular space can be obtained. The objective function is to minimize the output power of the noise component. T Given the constraint a(θ) = 1, the weight vector W of each microphone in this angular space is obtained as follows: The weight vector W of each microphone in the angular space is determined as the beamforming vector of the angular space.

[0234] (3) For each angular space, the average output power of the initial sample beamforming signal guided by the sample pulse signal to the angular space can be obtained based on the autocorrelation matrix R of the sample pulse signal in the angular space and the initial weight vector. With maximizing the output power as the objective function, the weight vector W of each microphone in each angular space is determined by the subspace decomposition method. The weight vector W of each microphone in the angular space is then determined as the beamforming vector of the angular space.

[0235] It should be noted that the above method for obtaining beamforming vectors based on linear clustering is only an illustrative example and does not constitute a limitation on the direction-of-arrival identification method provided in the embodiments of the present invention. The beamforming method can be determined according to the actual application scenario. For example, an objective function can be established based on the covariance matrix of the oversample pulse signal, and the weight vector of each microphone in each angular space can be obtained by solving the objective function. Alternatively, the filtered sample signal can be guided to each angular space through a data-independent beamforming method, an optimal beamforming method, or an adaptive beamforming method. Based on the sample beamforming signal in each angular space, the output power of the sample beamforming signal in each angular space can be obtained, and the beamforming vector of each microphone in each angular space in each frequency channel can be determined, thereby obtaining the beamforming matrix.

[0236] In some embodiments of the present invention, linear clustering can also be achieved by maximizing the power of the beamforming signal of the sample data to obtain the optimal beamforming vector.

[0237] Specifically, the method for determining the weight matrix includes at least steps S201 to S204:

[0238] Step S201: The sample pulse signal input to the SNN is low-pass filtered to obtain the sample input pulse signal.

[0239] In some embodiments of the present invention, for sample sound source data received in different angular spaces, channel decomposition is performed on the sample sound source data received in different angular spaces to obtain the audio components of the sample sound source data received in different angular spaces in each frequency channel, and pulse coding is performed on the audio components of the sample sound source data received in different angular spaces in each frequency channel to obtain the sample pulse signal of each angular space in different frequency channels.

[0240] In some embodiments of the present invention, for sample pulse signals in each angular space under different frequency channels, the sample pulse signals are subjected to low-pass filtering by normalizing the joint impulse response through synapses and cell membranes in the linear layer of the SNN to obtain the sample input pulse signal.

[0241] For example, for sample pulse signals in each angular space under different frequency channels A low-pass filter h is established by normalizing the combined impulse response of neurons in a spiking neural network through synapses and cell membranes. sm [t], and through The sample pulse signal is low-pass filtered to obtain the sample input pulse signal R. c [m,t].

[0242] Step S202: For each corner space under each frequency channel, perform clustering processing on the sample input pulse signal under that frequency channel to obtain the optimal beamforming vector for each corner space under that frequency channel.

[0243] In some embodiments of the present invention, for each frequency channel, the sample input pulse signal R of each microphone in the microphone array at each angular space under that frequency channel is taken. c [m, t] is defined as a sample vector signal v n [t] = R n [.,t], where,

[0244] In some embodiments of the present invention, for each corner space under each frequency channel, beamforming can be performed by clustering based on the sample vector signal of each corner space under that frequency channel to obtain the optimal beamforming vector for each corner space under that frequency channel.

[0245] Optionally, nonlinear beamforming can be performed based on the sample vector signal of each corner space under the frequency channel through nonlinear processing to obtain the optimal beamforming vector of each corner space under the frequency channel.

[0246] For example, taking a corner space under this frequency channel as an example, the sample vector signal of the corner space can be nonlinearly mapped to a certain feature space to obtain the sample mapped vector signal of the corner space in the feature space. In the feature space, the least squares support vector regression algorithm is applied to perform nonlinear beamforming to obtain the optimal beamforming vector of each corner space under this frequency channel.

[0247] Optionally, linear beamforming can be performed based on the sample vector signal of each corner space under the frequency channel through linear processing to obtain the optimal beamforming vector of each corner space under the frequency channel.

[0248] For example, taking the method for determining the beamforming vector based on subspace as an example, for this frequency channel, linear clustering can be performed based on the average power of the sample vector signals in different angular spaces under this frequency channel after maximizing beamforming. Specifically, the method for determining the beamforming vector based on subspace includes at least a1 to a4:

[0249] Step a1: Construct an autocorrelation matrix based on the power of the beamforming signal from the sample data.

[0250] In some embodiments of the present invention, ω[t] can be defined as ω T v n (t) represents the waveform of the sample beamforming signal obtained by beamforming the sample vector signal of the angular space DoAn. The average power of the sample beamforming signal can be obtained by... We obtain that, since ω[t] := ω T v n (t), therefore it can be obtained through The average power of the sample beamforming signal is obtained, where, Let be the autocorrelation matrix of the sample vector signal.

[0251] Step a2: Perform eigenvalue decomposition on the autocorrelation matrix to obtain the eigenspace of the autocorrelation matrix.

[0252] Since the DC component of the sample vector signal does not carry information about DoAn, and the DC component is the same for the vector signal of each microphone at each time t, it is necessary to eliminate the influence of the DC component on beamforming during beamforming vector determination. Therefore, to eliminate the influence of the DC component on the beamforming signal, in some embodiments of the present invention, the influence of the DC component on beamforming can be eliminated by maximizing the average power of the sample beamforming signal after beamforming.

[0253] Specifically, it can be based on P ω =ω T C n ω and the beamforming vector constraint ω are established by maximizing the power of the beamforming signal in the sample data. * =arg maxω T C n ω, stω T ω=1, ω T 1 = 0, where ω * The beamforming vector is the beamforming vector that has the maximum average power when the sample beamforming signal after beamforming has been formed, i.e., the optimal beamforming vector.

[0254] In some embodiments of the present invention, to solve for the optimal beamforming vector, the autocorrelation matrix can be eigenvalued by performing eigenvalue decomposition to obtain the eigenvalues ​​and corresponding eigenvectors of the autocorrelation matrix. The eigenspace of the autocorrelation matrix is ​​then obtained based on the eigenvalues ​​and their corresponding eigenvectors. The eigenspace includes the eigenvector matrix corresponding to the eigenvalues, and comprises a signal subspace and a noise subspace. The eigenspace and noise subspace are orthogonal, wherein the signal subspace is the eigenvector matrix corresponding to the larger eigenvalues, and the noise subspace is the eigenvector matrix corresponding to the smaller eigenvalues.

[0255] Optionally, the singular values ​​of the autocorrelation matrix and the corresponding eigenvectors can be obtained by performing singular value decomposition on the autocorrelation matrix. The eigenspace of the autocorrelation matrix can then be obtained based on the singular values ​​of the autocorrelation matrix and the corresponding eigenvectors.

[0256] Specifically, the beamforming vector constraint is transformed based on two constraints of Lagrange multipliers and the KKT method, and the average power is maximized to make C n If ω-λω-β1=0, then the beamforming vector can be expressed as ω=β(C n -λI) -1 1. Using ω T 1 = o constraint, for ω = β(C n -λI) -1 1. Perform another transformation to obtain λ that satisfies 1. T (C n -λI) -1 1 = 0. Singular value decomposition (SVD) is performed on the autocorrelation matrix to obtain the singular values ​​and corresponding eigenvectors. Based on the singular values, a diagonal matrix is ​​constructed, along with the eigenvector matrix, to obtain C. n =UΛU T , where U is the eigenvector matrix corresponding to the singular values, which is the eigensubspace, and Λ is the diagonal matrix constructed from the singular values.

[0257] Step a3: By maximizing the power of the beamforming signal of the sample data, the maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix is ​​obtained.

[0258] In some embodiments of the present invention, the maximum eigenvalue can be found from the singular values ​​corresponding to the eigenspace of the autocorrelation matrix by maximizing the power of the beamforming signal of the sample data.

[0259] Based on the eigenspace of the autocorrelation matrix, z = U is defined. T 1. Let z = U T Substitute 1 T (C n -λI)-1 1 = 0, thus obtaining the mapping relationship between singular values ​​and eigenvalues. From the mapping relationship between singular values ​​and eigenvalues, we can see that all singular values ​​λi of the autocorrelation matrix are non-negative. Therefore, for all λ≤0, q(λ)≥0 is satisfied, and as λ→-∞, q(λ) approaches 0. q(λ) monotonically decreases from -∞ to +∞ within its interval. For example, as shown in... Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the monotonically decreasing trend of q(λ) provided in an embodiment of the present invention. Therefore, q(λ) = 0 in the interval (λ) M , λ M-1 There are M-1 roots on (λ1, ..., (λ2, λ1)). Due to the monotonicity of q(λ), these roots can be found by a simple algorithm, such as alignment / binary search. This is achieved by maximizing the power of the sample beamforming signal through the KKT equations, such that ω... T C n ω=λω T ω-βω T Since 1 = λ, the maximum average power can be obtained by solving for the maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix.

[0260] Therefore, the maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix can be obtained by solving the KKT equation through the mapping relationship between singular values ​​and eigenvalues. Specifically, the solution for the maximum eigenvalue includes:

[0261] (1) Based on the eigenspace of the autocorrelation matrix and the singular values ​​of the autocorrelation matrix, the mapping relationship between the singular values ​​and the eigenvalues ​​is obtained.

[0262] (2) Based on the beamforming vector constraint, the mapping relationship between the singular values ​​and eigenvalues ​​is solved by the KKT equation to obtain the maximum eigenvalue corresponding to the eigenspace of the autocorrelation matrix.

[0263] In some embodiments of the present invention, the KKT equation ω is used. T C n ω=λω T ω-βω T 1 = λ pairs Solve for ω, where ω T ω=1, ω T 1 = 0, β is a preset control variable. Since q(λ) is monotonically decreasing for all λ values ​​from -∞ to +∞ in its interval, the largest root λ of q(λ) can be found in the interval (λ2, λ1). * So that the average power ω of the sample beamforming signal is... T C n Maximize ω.

[0264] Step a4: Obtain the optimal beamforming vector based on the autocorrelation matrix and the maximum eigenvalue.

[0265] In some embodiments of the present invention, it can be based on ω T C n ω=λω T ω-βω T 1 = λ, thus obtaining the mapping relationship between the beamforming vector and the eigenvalue ω = β(C n -λI) -1 Based on the autocorrelation matrix and the largest eigenvalue, through β(C n -λ * I) -1 The λ that yields q(λ) is the largest root λ. * The target norm vector ω is obtained by... By performing unit processing on the norm vector, the optimal beamforming vector ω in that angular space is obtained. * .

[0266] For example, such as Figure 7 As shown, Figure 7 The beamforming schematic provided in the embodiments of the present invention is as follows: Figure 7 It can be seen that although the sample vector signal v n [t] The optimal beamforming vector is obtained by solving the beamforming vector constraint based on maximizing the average power. The sample beamforming signal obtained by beamforming based on the optimal beamforming vector is a zero DC near-harmonic signal.

[0267] Step S203: For each corner space under each frequency channel, obtain the beamforming sub-matrix under that frequency channel based on the optimal beamforming vector in each corner space of that frequency channel.

[0268] In some embodiments of the present invention, for each frequency channel, taking a circular microphone array as an example, the microphone data volume of the microphone array as M, and the preset oversampling factor as η, after obtaining the beamforming vector of each frequency channel in different angular spaces, the beamforming vectors of the frequency channel in different angular spaces are connected to obtain an M*ηM beamforming submatrix.

[0269] Step S204: Based on the beamforming sub-matrix under all frequency channels, obtain the weight matrix.

[0270] In some embodiments of the present invention, the beamforming submatrix for each frequency channel is obtained. Then, the beamforming sub-matrices corresponding to each frequency channel are connected together to obtain a weight matrix W = [W1; W2; ...; W] of size M × ηMN.N The beamforming layer.

[0271] Considering that the resolution of the angular space is an important factor affecting the accuracy of direction of arrival recognition in the construction of the weight matrix, and the resolution of the angular space is affected by the array resolution and grating lobes, although a higher array resolution will obtain a more refined angular space resolution and more accurate direction of arrival recognition, it will also cause grating lobe effect, resulting in aliasing of the sound source direction.

[0272] For example, taking a circular geometric array with 16 microphones and a microphone radius of 10cm as an example, when the oversampling factor η is 4, and when the angular space corresponding to the sample sound source data is the 32nd angular space (out of 64 angular spaces), beamforming is performed on the sample sound source data in that angular space using the weight matrix in the linear layer to obtain beamforming signals for different frequency channels, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of beamforming signals for different frequency channels provided in embodiments of the present invention. Figure 8 As can be seen, the higher the center frequency of the frequency channel, the higher the array resolution. Moreover, the beamforming signals of all frequency channels generate the highest output power in the angular space corresponding to the sample sound source. At the same time, at a higher center frequency, a larger grating lobe effect will be caused, and angular space confusion will occur.

[0273] To achieve high array resolution while reducing grating lobe effects, the operating frequency range of the microphones needs to be determined based on the array geometry. For example, the operating frequency of the microphone array can be determined by using frequencies with wavelengths less than twice the maximum projected distance between consecutive microphones in the array.

[0274] In some embodiments of the present invention, after obtaining the beamforming matrix W = [W1; W2; ...; W...], ... N After that, the beamforming matrix W = [W1; W2; ...; W...] needs to be formed. N The weight matrix of the linear layer is synchronized to the chip. For example, in a chip using this invention, 8 bits are used to represent the weights, and 16 bits are used to store the synaptic current and membrane potential of each neuron in the spiking neural network. Therefore, a beamforming matrix W = [W1; W2; ...; W...] is required. N The beamforming vector of each frequency channel in different angular spaces is quantized, and the quantized beamforming matrix is ​​used as the weight matrix of the linear layer and synchronized to the chip.

[0275] Optionally, this can be achieved by adjusting v n [t] is quantized, based on the quantized v n[t]The beamforming vector of each frequency channel in different angular spaces is determined according to the above method, and then the quantized beamforming matrix is ​​obtained.

[0276] Optionally, the beamforming matrix W = [W1; W2; ...; W...] can be configured. N The beamforming submatrix of each frequency channel in the matrix is ​​normalized to obtain the quantized beamforming matrix. For example, the beamforming vector in the beamforming submatrix of each frequency channel can be used to... Where B represents the number of bits used in the quantization.

[0277] In some optional implementations, the sample pulse signals based on the sample sound source data are clustered to obtain a beamforming matrix. The beamforming matrix is ​​used as the weight matrix of the beamforming layer of the pulse neural network. The input pulse is beamformed based on the weight matrix in the beamforming layer to obtain a beamforming signal.

[0278] This invention employs a special type of beamforming on different frequency channels to improve resolution. Since beamforming vectors are frequency-dependent, a beamforming vector designed for pulse coding of a specific frequency channel may not produce good beamforming on other frequency channels. In a preferred embodiment, beamforming is performed only on the pulse signal corresponding to the active frequency channel. Furthermore, when determining the target DoA, only the firing rate of the neuron corresponding to the active frequency channel is used.

[0279] For example, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the beamforming processing flow provided in an embodiment of the present invention. In the processing flow shown, the weight matrix set in the beamforming layer of the pulse neural network includes a beamforming sub-matrix for each frequency channel. The input pulse passes through the beamforming layer of the SNN for beamforming, obtaining beamforming signals of the microphone array in different angular spaces. A second activity detection is performed on the output beamforming signals to select the beamforming signals of the active frequency channels. The beamforming signals of the active frequency channels are then pulse-excited or pulse-coded to generate a pulse sequence. Subsequent layers of the SNN determine the direction of arrival (DOA) based on the pulse sequence, and determine the direction of the target sound source based on the DOA. The subsequent layers of the SNN include intermediate layers, pulse layers, and output layers.

[0280] For example, taking the input pulse signal as the signal after low-pass filtering of the pulse signal of the sound source data to be processed, the pulse signal of the sound source data to be processed received by each microphone is normalized after joint impulse response to obtain the filtered signal of each microphone. The filtered signal of each microphone is determined as the input pulse. The input pulse is beamformed through the weight matrix of the beamforming layer in the SNN to obtain the beamforming signal of the microphone array in different angular spaces.

[0281] In some embodiments of the present invention, after obtaining the beamforming signal, the pulse neural network determines the direction of arrival based on the beamforming information, and determines the direction of the target sound source based on the direction of arrival.

[0282] like Figure 10 As shown, Figure 10 This is a schematic diagram of the direction-of-arrival (DOA) recognition processing flow provided in this embodiment of the invention. The microphone array receives the sound source data to be processed, which is then processed by the audio front-end to obtain pulse signals. These pulse signals are input into the SNN (Sound-Neural Network). The SNN processor inputs the pulse signals from each microphone into a weight matrix W = [W1; W2; ...; W...]. N The beamforming layer performs beamforming, and the output of the beamforming layer is input to the intermediate layer of the SNN. Finally, DoA estimation is performed in the output layer of the SNN to obtain the direction of arrival of the sound source data to be processed, and the direction of the target sound source is determined based on the direction of arrival of the sound source data to be processed.

[0283] Optional, such as Figure 10 As shown, the audio front end includes a coupled filter bank, a first activity detection module, and a pulse coding module. The filter bank is used to decompose the sound source data received by the microphone into multiple frequency channels to obtain audio components. The first activity detection module is used to identify the active frequency channel based on the signal frequency and signal energy of the audio components of each frequency channel. The pulse coding module is used to perform pulse coding on the audio components of the active frequency channel to obtain a pulse signal.

[0284] Optionally, the SNN processor utilizes the impulse response of synapses and / or neuronal membranes to perform low-pass filtering on the impulse signal of the sound source data to be processed, obtaining a low-pass filtered signal. This low-pass filtered signal is then used as the input impulse to the weight matrix W = [W1; W2; ...; W...]. N The beamforming layer of the SNN performs beamforming, and the output of the beamforming layer of the SNN is input to the intermediate layer of the SNN. Finally, DoA estimation is performed in the output layer of the SNN to determine the direction of the target sound source.

[0285] Optionally, the SNN processor utilizes the impulse response of synapses and / or neuronal membranes to perform low-pass filtering on the impulse signal of the sound source data to be processed, obtaining a low-pass filtered signal. This low-pass filtered signal is then fed into a linear layer for processing to obtain the input impulse, which is then input into a weight matrix W = [W1; W2; ...; W...]. N The beamforming layer of the SNN performs beamforming, and the output of the beamforming layer of the SNN is input to the intermediate layer of the SNN. Finally, DoA estimation is performed in the output layer of the SNN to determine the direction of the target sound source.

[0286] Considering that beamforming vectors are frequency-dependent, meaning that a beamforming vector designed for a pulse signal of a specific frequency channel may not produce good beamforming on other frequency channels, in DoA estimation, only those beamforming signals corresponding to the active frequency channel are used for DoA estimation. In some embodiments of the present invention, activity detection is used to identify the beamforming signal under the active frequency channel, and the pulse neural network obtains the direction of arrival based on the beamforming signal under the active frequency channel.

[0287] Optionally, the beamforming signal can be subjected to activity detection to obtain the beamforming signal of the active frequency channel. The beamforming signal can be the initial beamforming signal of each frequency channel in different angular spaces obtained by guiding the input pulse to each angular space of different frequency channels.

[0288] Specifically, the input pulse can be guided to each angular space of different frequency channels through the weight matrix of the beamforming layer, so as to obtain the initial beamforming signal of each frequency channel in different angular spaces. Activity detection is performed based on the initial beamforming signal of each frequency channel in different angular spaces, and the initial beamforming signal that is consistent with the active frequency channel is determined as the beamforming signal.

[0289] Optionally, activity detection can be performed based on the power or energy of the initial beamforming signal of each frequency channel in different angular spaces, and the frequency channel with the highest power or energy can be identified as the active frequency channel.

[0290] Among these, the detection methods include, but are not limited to, independent activity detection, joint activity detection, and local activity detection.

[0291] Considering that directing the input pulse to each angular space under different frequency channels will increase the amount of data processing for direction of arrival identification, in some embodiments of the present invention, activity detection can be performed before acquiring the input pulse of the spiking neural network to determine the active frequency channel. The beamforming layer of the spiking neural network then performs beamforming on the input pulse based on the beamforming submatrix corresponding to the active frequency channel in the weight matrix to obtain the beamforming signal under the active frequency channel.

[0292] Specifically, before acquiring the input pulse of the pulse neural network, activity detection is performed to identify the signal components of each sensor in the active frequency channel; pulse encoding is performed on the signal components of each sensor in the active frequency channel to obtain the input of the pulse neural network in the active frequency channel, thereby obtaining the input pulse of the beamforming layer in the active frequency channel; based on the weight matrix of the beamforming layer, beamforming is performed on the input pulse of the beamforming layer in the active frequency channel or the low-pass filtered signal of the input pulse of the beamforming layer in the low-pass filtered signal to obtain the beamforming signal in the active frequency channel.

[0293] For example, taking a microphone as an example, when performing activity detection on the sound source data to be processed for each microphone and determining the active frequency channel, the channel information of the active frequency channel is used as prior information. This prior information and the pulse signal of each microphone are used as input to a pulse neural network. The pulse neural network normalizes the pulse signal of each microphone and performs joint impulse response on it to obtain the filtered signal of each microphone. The filtered signal is used as the input pulse of the beamforming layer. Based on the prior information, the beamforming sub-matrix of the active frequency channel matching the prior information is determined from the beamforming matrix. Beamforming is then performed on the input pulse based on the beamforming sub-matrix of the active frequency channel to obtain the beamforming signal in different angular spaces under the active frequency channel. Here, the prior information can be the center frequency or channel identifier of the active frequency channel.

[0294] In some embodiments of the present invention, a target beamforming signal with the maximum average power can be determined based on the beamforming signal, and the direction of arrival can be determined based on the angular space corresponding to the beamforming vector of the target beamforming signal.

[0295] Although the beamforming vector in the weight matrix of the beamforming layer is calculated by maximizing the average power as a metric, neglecting the nonlinear effect of potential reset due to neuronal firing, the beamforming vector designed based on the linear assumption cannot be a good beamforming vector in the presence of neuronal-generated pulses. However, when the neuron has a small firing threshold, the pulse firing rate of the neuron for the beamforming signal of each microphone in different angular spaces will approximately be proportional to the average power of the beamforming signal of each microphone in different angular spaces. Therefore, the beamforming layer weight matrix obtained based on linear analysis during the training phase is still effective. The neuron can generate effective pulse sequences for the beamforming signal obtained based on the beamforming layer weight matrix, and direction-of-arrival estimation can be performed based on these pulse sequences.

[0296] In a spiking neural network, the membrane potential of a non-firing neuron can be obtained from a continuous and smooth function x(t). When the first pulse firing time occurs at time f1, i.e., x(f1) = γ, the membrane potential at time t ∈ (f1, f2) (i.e., before the next firing time f2) is obtained through... We obtain, where γ is the firing threshold of the neuron. This represents the attenuation factor of the membrane potential. Similarly, when another emission time occurs at f2, i.e. Then the membrane potential at t∈(f2, f3) will be determined by This is given. Therefore, it can be seen that during the emission time, the membrane potential after the neuron's potential reset will be... Given, where F is the set of launch times.

[0297] When the attenuation factor of the membrane potential is large, the duration of each pulse signal is And the peak value needs to be... This reduces the membrane potential to a lower level, indicating that the neuronal firing rate is at a lower level. In the vicinity, the average impulse firing rate of neurons can be expressed as: When the discharge threshold γ is much lower than the beamforming signal x ω When (t) is at its full amplitude, it can be Interpreted as max{x ω An approximation of the Lebesque integral of {(t), 0} is obtained. Therefore, the average firing rate of a neuron can be expressed as:

[0298] Because the beamforming signal is a near-harmonic signal with zero DC, i.e., x ωThe DC value (time average) of (t) is zero, therefore the average of its positive and negative values ​​is the same, and equal to half the average of its absolute values. This shows that when the attenuation factor of the membrane potential is large, and the discharge threshold is much smaller than the beamforming signal x... ω When the amplitude of (t) is full, the average firing rate of the neuron is proportional to the average absolute value of the beamforming signal. Furthermore, in practical applications, the average power and the absolute value are closely related. For example, for a sine curve with amplitude A, the square of its average power is equal to... The average of absolute values ​​equals They are very close to each other. Therefore, in this case, the average firing rate of the neurons can be approximated as the arithmetic mean square power of the beamforming signal.

[0299] When the decay factor of the membrane potential is small, each generated pulse will reduce the value of x(t) by γ at all times t. After at least a few seconds of transmission, the persistent effects caused by these pulses will decrease, and new pulses can occur again, which can be achieved through... Determine the pulse firing rate of the beamforming signal of the microphone array in different angular spaces. For a reduced γ, this can be achieved through... To approximate the pulse firing rate of the beamforming signal of the microphone array in different angular spaces, for a zero DC beamforming signal, it can be obtained by... This is used to approximate the pulse firing rate of the beamforming signal of the microphone array in different angular spaces.

[0300] Therefore, it can be seen that the average pulse firing rate of the neuron on the beamforming signal of the microphone array in different angular spaces is proportional to the average absolute value of the beamforming signal power of the microphone array in different angular spaces. For a sine curve with amplitude A, the average absolute value of its signal power can be approximated as the arithmetic mean square power of the signal. Thus, the average absolute value of the beamforming signal power is proportional to the square of the average pulse firing rate of the neuron on the beamforming signal of the microphone array in different angular spaces. Based on this, it is feasible to use the beamforming signal obtained by beamforming based on the weight matrix of the beamforming layer during the training phase (established through linear clustering of the maximum average power of the beamforming signal) for direction-of-arrival estimation.

[0301] Considering that the resetting of the membrane potential generated by the neuronal pulses leads to nonlinear effects in the neurons, the reliability of the direction of arrival estimation results can be guaranteed. By setting the membrane potential attenuation factor and the neuron firing threshold, the average pulse firing rate can be approximated as the square of the average power of the beamforming signal. Thus, the direction of arrival can be determined based on the target beamforming signal with the maximum pulse firing rate, using the pulse firing rate of the neuron to the beamforming signal.

[0302] like Figure 7 As shown, the beamforming signal obtained by beamforming is a zero DC signal, similar to a sine wave signal. The frequency of this beamforming signal is related to the frequency channel corresponding to the pulse signal. In some embodiments of the present invention, the pulse firing rate function can be obtained by changing the amplitude A of the beamforming signal. For example, such as Figure 11 As shown, Figure 11 This is a schematic diagram illustrating the mapping relationship between the ratio of beamforming signal amplitude to discharge threshold and pulse rate provided in this embodiment of the invention. The pulse firing rate and... Proportional growth (pulse firing rate close to) Especially for At that time, the pulse firing rate and (Proportional growth), meaning that when beamforming is complete, the sinusoidal amplitude of the beamforming signal is significantly greater than the discharge threshold Y. The angular space of the beamforming signal with the maximum average power is the true direction of arrival of the signal, i.e., the location of the target sound source. Figure 11 This indicates that the pulse firing rate of the neuron provides a good approximation of the amplitude of the zero DC signal obtained after beamforming.

[0303] To better illustrate the direction-of-arrival (DOA) estimation method provided in this embodiment of the invention, based on a practical application scenario, using a circular microphone array with 16 microphones and a filter bank consisting of 10 filters in the frequency range [1370, 2740] Hz as an example, the DOA identification results of the method applied to a practical application scenario are provided. For example, as shown below... Figure 12 , Figure 13 As shown, where, Figure 12 This is a schematic diagram of the direction of arrival (DOA) identification results when the activity frequency in the sound source data to be processed is low frequency. Figure 13 This is a schematic diagram of the direction of arrival identification results when the activity frequency in the sound source data to be processed is high. Figure 12 , Figure 13The results show that when the sound source direction changes from 60 degrees to -60 degrees, the algorithm should be able to identify the sound source direction. Computer simulation results and the test results of the neuromorphic chip all demonstrate that the direction-of-arrival (DOA) recognition method provided in this embodiment can respond promptly and determine the target sound source direction, and achieves better localization performance in the high-frequency channel. In particular, when a smoothing filter is added after channel decomposition (e.g., bandpass filtering), and the smoothed signal is provided to the SNN for sound source direction recognition via pulse coding, the jitter in the computer simulation results and the actual neuromorphic chip test results is greatly reduced, reaching an ideal state.

[0304] The direction-of-arrival estimation method provided in this embodiment of the invention and the direction-of-arrival identification method provided in Chinese patent application number 202310109065.9 are applied to the same real-world scenario. Figure 14 The diagram illustrates the performance comparison of two sound source localization methods. It can be seen that, in the same application scenario, the direction-of-arrival estimation method provided by this embodiment of the invention has a smaller computational load and better angular accuracy.

[0305] To better implement the direction-of-arrival (DOA) identification method provided in the embodiments of the present invention, a sound source signal separation method is provided based on the DOA identification method. Specifically, as follows: Figure 15 As shown, Figure 15 This is a schematic flowchart of a sound source signal separation method provided in an embodiment of the present invention. The sound source signal separation method shown includes at least steps S400 to S500:

[0306] Step S400: Perform sound source direction estimation on the sound source data to be separated to determine the sound signal of the candidate sound source corresponding to the sound source data to be separated.

[0307] Step S500: Based on the sound signals of each candidate sound source, determine the target sound source from multiple candidate sound sources.

[0308] In some embodiments of the present invention, the direction of arrival identification method in any of the above embodiments can be used to identify the direction of arrival of the sound source data to be separated, determine the candidate sound source corresponding to the sound source data to be separated and the target sound source direction of each candidate sound source, and perform signal separation on the sound source data to be separated according to each target sound source direction to determine the sound signal of the candidate sound source corresponding to the sound source data to be separated.

[0309] Optionally, for the sound source data to be separated, the channel decomposition method, activity detection method, and pulse coding method described above can be used to sequentially perform channel decomposition, activity detection, and pulse coding on the sound source data to be separated, thereby obtaining the pulse signal of the sound source data to be separated. The beamforming layer of the pulse neural network is used to beamform the pulse signal of the sound source data to be separated, guiding the pulse signal of the sound source data to be separated into different angular spaces, thereby obtaining the beamforming signal of the pulse signal of the sound source data to be separated in different angular spaces. Power activity detection is performed based on the beamforming signal in different angular spaces to determine the candidate sound sources corresponding to the sound source data to be separated and the target sound source direction of each candidate sound source. Based on the target sound source direction of each candidate sound source, the signal of the sound source data to be separated is separated to determine the sound signal of the candidate sound source corresponding to the sound source data to be separated.

[0310] For example, beamforming can be performed on the sound source data to be separated according to the target sound source direction of each candidate sound source, and the sound source data to be separated can be guided to each target sound source direction to obtain the beamforming signal in each target sound source direction. The beamforming signal in each target sound source direction can be determined as the sound signal of the candidate sound source in that target sound source direction.

[0311] For example, a separation method based on independent vector analysis with auxiliary function optimization can be used to separate the sound source data to be separated and determine the sound signal of the candidate sound source corresponding to the sound source data to be separated; alternatively, a separation method based on independent subspace analysis can be used to separate the sound source data to be separated and determine the sound signal of the candidate sound source corresponding to the sound source data to be separated.

[0312] Optionally, the pulse signal of the sound source data to be separated can be a beamforming signal in different angular spaces. Power activity detection can be performed based on the beamforming signal in different angular spaces to determine at least one target beamforming signal. The target angular space corresponding to each target beamforming signal can be determined. The target angular space corresponding to each target beamforming signal can be determined as the target sound source direction of each candidate sound source. Pulse decoding can be performed on each target beamforming signal to obtain the sound signal of the candidate sound source in the direction of each target sound source.

[0313] In some embodiments of the present invention, a target sound source can be determined from a plurality of candidate sound sources by evaluating the sound quality scores of the sound signals of each candidate sound source. For example, the target sound source with the highest sound quality score can be selected from a plurality of candidate sound sources based on the sound quality scores.

[0314] Optionally, there are multiple ways to evaluate the quality of the sound signal from each candidate sound source, including, for example:

[0315] (1) The sound quality score of each candidate sound source can be determined by calculating the signal interference ratio of the sound signal of each candidate sound source.

[0316] (2) The quality of the sound signal of each candidate sound source can be evaluated by calculating the signal distortion ratio of the sound signal of each candidate sound source, and the sound quality score of the sound signal of each candidate sound source can be determined.

[0317] (3) The sound quality score of each candidate sound source can be determined by calculating the maximum likelihood ratio of the sound signal of each candidate sound source.

[0318] (4) The quality of the sound signal of each candidate sound source can be evaluated by calculating the cepstral clustering of the sound signal of each candidate sound source, and the sound quality score of the sound signal of each candidate sound source can be determined.

[0319] (5) The sound quality score of each candidate sound source can be determined by calculating the frequency-weighted segmented signal-to-noise ratio of the sound signal of each candidate sound source.

[0320] (6) The sound quality score of each candidate sound source can be determined by calculating the speech quality perception evaluation score of the sound signal of each candidate sound source.

[0321] (7) The sound quality score of each candidate sound source can be determined by calculating the kurtosis value of the sound signal of each candidate sound source.

[0322] (8) The sound quality score of each candidate sound source can be determined by calculating the probability score corresponding to the speech feature vector of each candidate sound source's sound signal. The probability score is used to characterize the probability that the sound signal of each candidate sound source is the speech signal of the target sound source.

[0323] It should be noted that the method for evaluating the sound signal quality of each candidate sound source described above is merely illustrative and does not constitute a limitation on the sound signal processing method provided in the embodiments of the present invention. In practical applications, the corresponding sound quality score determination method can be selected based on the computational power of the electronic device in the actual application scenario.

[0324] In some embodiments of the present invention, after determining the sound signals of candidate sound sources corresponding to the sound source data to be separated, sound signal category identification can be performed based on the sound signals of each candidate sound source to determine the sound signal type of each candidate sound source. Based on the sound signal type of each candidate sound source, the candidate sound source whose sound signal type matches the target sound signal type is determined as the target sound source. The sound signal type includes, but is not limited to, environmental noise, human voice, keyboard sounds, animal sounds, etc.

[0325] For example, in a biometrics scenario, a target sound source can be identified as an animal sound by determining the sound signal category from the sound signals of candidate sound sources, and then radar or sound wave signals can be emitted in the direction of the target sound source. The signal returned by the target sound source can be received, and the distance between the target sound source and the microphone array can be determined based on the signal return time.

[0326] Optionally, the sound signal category of each candidate sound source can be identified based on the sound characteristics of the sound signal of each candidate sound source. These sound characteristics include, but are not limited to, the Mel-spectral density coefficients of the sound signal.

[0327] Optionally, Fourier transform can be performed on the sound signals of each candidate sound source to obtain the spectral energy distribution data of the sound signals of each candidate sound source. After dividing the spectral energy distribution data of the sound signals of each candidate sound source into critical bands using a triangular filter bank, amplitude weighting calculation and discrete cosine transform can be performed to obtain the Mel-Cepstral Coefficients of the sound signals. The Mel-Cepstral Coefficients of the sound signals are then submitted to a restricted Boltzmann machine for sound signal type identification to obtain the probability score of the sound signals of each candidate sound source belonging to each sound signal category. The sound signal type of each candidate sound source is determined based on the probability score.

[0328] The sound source signal separation method provided in this embodiment of the invention improves the accuracy of the direction of candidate sound sources by estimating the direction of the sound source data to be separated, and determines the final target sound source by evaluating the values ​​of each candidate sound source, thereby further improving the accuracy of the separated sound source signal and improving the problem of low stability of signal separation.

[0329] To better implement the direction-of-arrival (DOA) identification method provided in this embodiment of the invention, based on the DOA identification method and the application scenario of audio conferencing in a closed room, a sound source tracking method is provided, specifically, as follows: Figure 16 As shown, Figure 16 This is a schematic flowchart of a sound source tracking method provided in an embodiment of the present invention. The sound source tracking method shown includes at least steps S600 to S800:

[0330] In step S600, sound source data in the conference scene is received through the microphone array.

[0331] Step S700: Estimate the direction of the sound source data to determine the direction of the target sound source corresponding to the sound source data.

[0332] In a certain embodiment of the present invention, the direction of sound source data can be estimated by following the direction of arrival identification method in any of the above embodiments to determine the direction of the target sound source corresponding to the sound source data.

[0333] Step S900: Adjust the direction parameters of the sound source tracking device according to the direction of the target sound source.

[0334] Among them, the sound source tracking device is used to track target sound sources in an audio conference in a room, including but not limited to cameras, microphones, etc.

[0335] Understandably, the sound source tracking device and the microphone array receiving the sound source are on the same plane.

[0336] In one embodiment of the present invention, the current direction parameters of the sound source tracking device can be obtained, the target direction parameters of the sound source tracking device can be obtained according to the target sound source direction corresponding to the sound source data, and the direction parameters of the sound source tracking device can be adjusted according to the target direction parameters and the current direction parameters. The direction parameters include, but are not limited to, azimuth and pitch angles. For example,... Figure 17 and Figure 18 As shown, where, Figure 17 This is a schematic diagram of the sound source tracking results when the activity frequency is low. Figure 17 The first and second images from top to bottom show the sound source tracking effect before and after smoothing, respectively. Figure 18 This is a schematic diagram of the sound source tracking results when the activity frequency is high. Figure 18 The first and second images from top to bottom show the sound source tracking effect before and after smoothing, respectively. Figure 17 and Figure 18 As can be seen, the sound source tracking method provided in this embodiment of the invention can respond quickly according to the direction of the sound source, realize the rapid tracking of the sound source, and the tracking effect of the high-frequency channel is better.

[0337] The sound source tracking method provided in this embodiment of the invention improves the accuracy of the sound source direction by estimating the sound source direction from the sound source data, and quickly adjusts the direction parameters of the sound source tracking device based on the target sound source direction to achieve rapid sound source tracking.

[0338] This invention also provides a direction-of-arrival (DOA) identification device, which includes a pulse neural network. The pulse neural network includes a beamforming layer, and beamforming is performed in the pulse domain based on the beamforming layer, which is a layer in the pulse neural network. Using the weights of the beamforming layer, beamforming is performed on the input pulses of the beamforming layer or on processed input pulses to obtain a beamforming signal. The pulse neural network obtains the DOA based on the beamforming signal.

[0339] This invention also provides a first type of chip that utilizes any of the beamforming methods, direction-of-arrival (DOA) identification methods, sound source signal separation methods, sound source tracking methods, or includes any of the DOA identification devices described above.

[0340] This invention also provides a first type of electronic device, which stores the direction-of-arrival identification method, beamforming method, or sound source signal separation method described above, or the aforementioned chip. Although the invention has been described with reference to specific features and embodiments, various modifications, combinations, and substitutions can be made therein without departing from the invention. The scope of protection of this invention is not limited to the specific embodiments of the processes, machines, manufacturing processes, material compositions, apparatuses, methods, and steps described in the specification, and these methods and modules may also be implemented in one or more related, interdependent, cooperative, or front-end / back-end products or methods.

[0341] This invention also provides a spiking neural network, which includes a beamforming layer for beamforming; wherein the beamforming layer is a layer in the spiking neural network; by maximizing the power of the signal after beamforming, beamforming vectors corresponding to different frequency channels and different angular spaces are calculated; based on all beamforming vectors, the weight matrix of the beamforming layer is obtained.

[0342] In one embodiment, beamforming is performed by extracting the non-DC component from the input pulse of the beamforming layer or the signal after low-pass filtering of the input pulse of the beamforming layer based on the weight matrix, and the resulting beamforming signal is a zero DC signal.

[0343] In one embodiment, the calculated weight matrix is ​​used as the initial weight of the beamforming layer in the spiking neural network; based on the initial weight, the network configuration parameters are continuously adjusted during training to obtain the optimal configuration parameters that meet the accuracy requirements.

[0344] A second type of chip includes a spiking neural network as described above.

[0345] A second type of electronic device, which includes a spiking neural network as described above, or includes a second type of chip.

[0346] The first or second type of chip is a neuromorphic chip or a neuromorphic chip, meaning that the chip can be developed by simulating the morphological working mode of biological neurons. It is typically event-triggered and features low power consumption, low latency response, and no privacy leakage. Existing neuromorphic chips include Intel's Loihi, IBM's TrueNorth, and Synsense's Dynap-CNN, but this invention is not limited to these.

[0347] Therefore, the specification and drawings should be simply regarded as a description of some embodiments of the technical solutions defined by the appended claims, and thus the appended claims should be interpreted in accordance with the principle of the greatest reasonable interpretation, and are intended to cover as much as possible all modifications, variations, combinations or equivalents within the scope of the invention, while avoiding unreasonable interpretations.

[0348] To achieve better technical effects or for the needs of certain applications, those skilled in the art may make further improvements to the technical solution based on this invention. However, even if such improvements / designs are inventive and / or progressive, as long as they rely on the technical concept of this invention and cover the technical features defined in the claims, the technical solution should also fall within the protection scope of this invention.

[0349] The technical features mentioned in the appended claims may have alternative technical features, or the order of certain technical processes or material organization may be rearranged. Those skilled in the art, upon learning of this invention, will readily conceive of these alternative means, or alter the order of the technical processes or material organization, and then employ substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the claims explicitly define the aforementioned means and / or order, these modifications, alterations, and substitutions should all fall within the scope of protection of the claims based on the principle of equivalents.

[0350] The method steps or modules described in the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application or design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered outside the scope of protection claimed by this invention.

Claims

1. A beamforming method based on a pulse neural network, characterized in that: The spiking neural network includes a beamforming layer that performs beamforming; wherein, the beamforming layer is one layer in the spiking neural network; By maximizing the power of the signal after beamforming, the beamforming vectors corresponding to different frequency channels and different angular spaces are calculated. Based on all the beamforming vectors, the weight matrix of the beamforming layer is obtained; Based on the weight matrix, the non-DC component in the input pulse of the beamforming layer or the signal after low-pass filtering of the input pulse of the beamforming layer is extracted for beamforming, and the resulting beamforming signal is a zero DC signal.

2. The beamforming method based on a pulse neural network according to claim 1, characterized in that: The calculated weight matrix is ​​used as the initial weight of the beamforming layer in the pulse neural network. Based on the initial weights, the configuration parameters of the spiking neural network are continuously adjusted during training to obtain the optimal configuration parameters that meet the accuracy requirements.

3. The beamforming method based on a pulse neural network according to claim 2, characterized in that: The input pulses of the beamforming layer are low-pass filtered using synapses and / or neuronal membranes to obtain the filtered signal.

4. The beamforming method based on a pulse neural network according to claim 2, characterized in that: The power mentioned is instantaneous power.

5. The beamforming method based on a pulse neural network according to claim 2, characterized in that: The power mentioned is the average power.

6. The beamforming method based on a pulse neural network according to claim 2, characterized in that... : Activity detection is performed on the beamforming signal or before acquiring the input pulses of the spiking neural network: When the beamforming signal is subjected to activity detection, the beamforming signal component under the active frequency channel is identified; the pulse neural network determines the direction of arrival based on the beamforming signal component under the active frequency channel. When activity detection is performed before acquiring the input pulse of the pulse neural network, the signal components of each sensor in the active frequency channel are identified; pulse encoding is performed on the signal components of each sensor in the active frequency channel to obtain the input of the pulse neural network in the active frequency channel, and then the input pulse of the beamforming layer in the active frequency channel is obtained.

7. The beamforming method based on a pulse neural network according to claim 2, characterized in that: The pulse neural network identifies the highest power of the beamforming signal, and the angular space corresponding to the highest power of the beamforming signal is the direction of arrival.

8. The beamforming method based on a pulse neural network according to claim 6, characterized in that: The pulse neural network identifies the highest power of the beamforming signal component in the active frequency channel, and the angular space corresponding to the highest power is the direction of arrival.

9. The beamforming method based on a pulse neural network according to claim 8, characterized in that: Based on the pulse firing rate of the output layer neurons of the spiking neural network, the angular space corresponding to the output layer neuron with the maximum pulse firing rate is the direction of arrival.

10. The beamforming method based on a pulse neural network according to claim 7, characterized in that: The pulse neural network is used to identify the direction of arrival of at least one of the following signals: Electromagnetic waves, seismic waves, radar, physiological signals, or voice signals.

11. A sound source tracking method, characterized in that, The sound source tracking method includes: The beamforming method based on a pulse neural network as described in any one of claims 1 to 9 determines the direction of the sound source data. Sound source tracking is performed based on the target sound source direction of the sound source data.

12. A chip, characterized in that, The chip application uses the beamforming method based on a pulse neural network as described in any one of claims 1 to 10.

13. A chip as described in claim 12, characterized in that: The chip is a neuromorphic chip.

14. An electronic device, characterized in that, The electronic device applies the beamforming method based on a pulse neural network as described in any one of claims 1 to 10, or includes the chip as described in claim 12 or 13.

Images