Sound event detection and positioning method, device, equipment and readable storage medium

Abstract

The present application provides a method, device, equipment and readable storage medium for detecting and locating sound events, the method comprising: obtaining multiple audio signals collected by a microphone array; determining a cross-correlation function matrix between every two audio signals in the multiple audio signals; inputting each cross-correlation function matrix into a preset sound event classification model to obtain a category matrix of the sound event, the preset sound event classification model is a pre-trained neural network model; performing Fourier transform on each audio signal to obtain a spectrogram of each audio signal, and determining a sound source orientation matrix of the sound event based on the spectrum data of each audio signal; inputting the category matrix and the sound source orientation matrix into a preset matching network for matching, and obtaining the category of each sound event in the audio signal and the sound source orientation matching the category. The present application accurately and quickly realizes the detection and positioning of sound events.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a method, device, equipment and readable storage medium for detecting and locating sound events. Background Art

[0002] The existing multiple sound event detection, classification and positioning system can detect sound signals within a period of time, identify the classification of different sound events contained therein, and determine the direction of each sound event. The system has a wide range of applications in scenarios such as robot interaction, autonomous driving, and field search and rescue. However, the current sound event localization and detection system (SELD) is usually composed of two independent modules, namely sound event detection (SED) and direction detection (DOAE). At present, deep learning has been widely used in the SED module, while the DOAE module mostly uses traditional signal processing algorithms based on microphone arrays. Since direction detection and event detection are performed separately, the system cannot combine the content learned by SED and DOAE, especially in the task of event detection and positioning for multiple sound sources, and cannot further improve the accuracy of the system. Summary of the invention

[0003] The main purpose of the present application is to provide a sound event detection and positioning method, device, equipment and readable storage medium, aiming to accurately realize the detection and positioning of sound events.

[0004] In a first aspect, the present application provides a sound event detection and positioning method, which is applied to an electronic device, wherein the electronic device includes a microphone array, and the microphone array includes a plurality of microphone units. The method includes:

[0005] Acquire multiple audio signals collected by the microphone array;

[0006] determining a cross-correlation function matrix between every two of the audio signals in the plurality of audio signals;

[0007] Inputting each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of sound events, wherein the preset sound event classification model is a pre-trained neural network model;

[0008] Performing Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determining a sound source orientation matrix of the sound event according to the frequency spectrum data of each of the audio signals;

[0009] The category matrix and the sound source orientation matrix are input into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

[0010] In a second aspect, the present application further provides a sound event detection and positioning device, which is applied to an electronic device, wherein the electronic device includes a microphone array, the microphone array includes a plurality of microphone units, and the sound event detection and positioning device includes an acquisition module, a determination module, a generation module, and a matching module, wherein:

[0011] The acquisition module is used to acquire multiple audio signals collected by the microphone array;

[0012] The determining module is used to determine a cross-correlation function matrix between every two of the multiple audio signals;

[0013] The generating module is used to input each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of the sound events;

[0014] The determination module is further used to perform Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determine a sound source orientation matrix of the sound event according to the frequency spectrum data of each of the audio signals;

[0015] The matching module is used to input the category matrix and the sound source orientation matrix into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

[0016] In a third aspect, the present application also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the sound event detection and positioning method as described above are implemented.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the sound event detection and positioning method as described above are implemented.

[0018] The present application provides a method, device, equipment and readable storage medium for detecting and locating a sound event. The present application obtains multiple audio signals collected by a microphone array and determines the cross-correlation function matrix between each two audio signals in the multiple audio signals; then each cross-correlation function matrix is ​​input into a preset sound event classification model to obtain a category matrix of the sound event; then each audio signal is Fourier transformed to obtain a spectrum diagram of each audio signal, and the sound source orientation matrix of the sound event is determined based on the spectrum data of each audio signal; finally, the category matrix and the sound source orientation matrix are input into a preset matching network for matching to obtain the category of each sound event in the audio signal and the sound source orientation matching the category. By matching the sound event category matrix and the sound source orientation matrix of the audio signal, the matching sound source orientation of each sound event can be accurately determined, which greatly improves the accuracy of sound event detection and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of the steps of a sound event detection and positioning method provided in an embodiment of the present application;

[0021] Figure 2 for Figure 1 A schematic diagram of the sub-step flow chart of the sound event detection and positioning method in FIG.

[0022] Figure 3 A schematic diagram of a smoothing process provided in an embodiment of the present application;

[0023] Figure 4 A schematic block diagram of a sound event detection and positioning device provided in an embodiment of the present application;

[0024] Figure 5 for Figure 4 A schematic block diagram of a submodule of a sound event detection and positioning device in;

[0025] Figure 6 A schematic block diagram of the structure of an electronic device provided in an embodiment of the present application.

[0026] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] The embodiments of the present application provide a method, apparatus, device and readable storage medium for detecting and locating a sound event. The method for detecting and locating a sound event can be applied to an electronic device, which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, a smart speaker, a microcomputer and a wearable device.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] Please refer to Figure 1 , Figure 1 A schematic diagram of the steps of a sound event detection and positioning method provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the sound event detection and positioning method includes steps S101 to S105.

[0033] Step S101: Acquire multiple audio signals collected by the microphone array.

[0034] The audio signal is the frequency and amplitude change information of all the sound waves in the environment where the electronic device is located, including speech, singing, car horn sound, animal sound, etc. The microphone array includes a plurality of microphone units, adjacent microphone units are separated by a preset distance, and the plurality of audio signals include the audio signals collected by each microphone unit in the microphone array.

[0035] It can be understood that the number, arrangement and preset distance of microphone units included in the microphone array can be determined according to actual conditions. For example, the microphone array includes 5 microphone units, 5 microphone units are arranged in a row or column, or 5 microphone units are arranged in a rectangle, or adjacent microphone units are 5 cm apart.

[0036] Step S102: Determine a cross-correlation function matrix between every two of the multiple audio signals.

[0037] In one embodiment, each audio signal is subjected to frame processing to obtain a plurality of audio frames corresponding to each audio signal; a cross-correlation function between each audio frame of an audio signal and a corresponding audio frame of another audio signal is determined; and a cross-correlation function matrix between every two audio signals is determined based on the cross-correlation function between each audio frame of an audio signal and a corresponding audio frame of another audio signal, wherein the cross-correlation function includes a generalized cross-correlation function.

[0038] In one embodiment, each audio signal is framed to obtain a plurality of audio frames corresponding to each audio signal by: framing the audio signal through a preset time window to obtain a plurality of audio frames corresponding to each audio signal. The preset time window can be set according to actual conditions, and the embodiment of the present application does not specifically limit this. For example, the time window can be set to 15 seconds. The audio signal is framed through a preset time window to facilitate subsequent processing of the audio signal and improve the accuracy of the operation.

[0039] In one embodiment, a method for determining a cross-correlation function between each audio frame of an audio signal and a corresponding audio frame of another audio signal may be: performing Fourier transform on a plurality of audio frames corresponding to each audio signal to obtain Fourier coefficients of each audio frame corresponding to each audio signal. A cross-correlation function calculation formula is obtained, wherein the cross-correlation function calculation formula is:

[0040]

[0041] Among them, G ij (t,τ) is the cross-correlation function, F -1 is the inverse Fourier transform, X i (f, t) is the Fourier transform coefficient of the i-th microphone, and * is a sign of taking conjugate. Based on the cross-correlation function calculation formula and according to the audio frame transform coefficient of each microphone, the cross-correlation function between each audio frame of the audio signal and the corresponding audio frame of another audio signal is determined.

[0042] In one embodiment, the method for determining the cross-correlation function matrix between two audio signals may be: the cross-correlation function between each audio frame of an audio signal and the corresponding audio frame of another audio signal is used to form a corresponding cross-correlation function matrix. For example, the multiple audio frames of audio signal A are audio frame a1, audio frame a2, audio frame a3, audio frame a4 and audio frame a5, and the multiple audio frames of audio signal B are audio frame b1, audio frame b2, audio frame b3, audio frame b4 and audio frame b5, and the cross-correlation function between audio frame a1 and audio frame b1 is C1, the cross-correlation function between audio frame a2 and audio frame b2 is C2, the cross-correlation function between audio frame a3 and audio frame b3 is C3, the cross-correlation function between audio frame a4 and audio frame b4 is C4, and the cross-correlation function between audio frame a5 and audio frame b5 is C5, then the cross-correlation function matrix between audio signal A and audio signal B may be [C1, C2, C3, C4, C5].

[0043] Step S103: input each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of the sound events, wherein the preset sound event classification model is a pre-trained neural network model.

[0044] Among them, the preset sound event classification model is a pre-trained neural network model. The choice of the neural network can be selected according to actual conditions. For example, the neural network can be a convolutional neural network model, a recurrent neural network model, and a recurrent convolutional neural network model. Of course, it can also be other network training models. This application does not make specific limitations on this.

[0045] It should be noted that the training method of the preset sound event classification model can be: obtaining a cross-correlation function matrix, labeling the cross-correlation function matrix according to the category corresponding to the cross-correlation function matrix to form sample data, and iteratively training the neural network based on the sample data until the neural network model converges, thereby obtaining a sound event classification model.

[0046] In one embodiment, each cross-correlation function matrix is ​​input into a preset sound event classification model to obtain a category matrix of the sound event. The category of the sound event can be determined according to actual conditions, and the present application does not specifically limit this. For example, the category of the sound event is speech or car horn. By classifying the cross-correlation function matrix through the preset sound event classification model, the category matrix of the sound event can be accurately and quickly obtained.

[0047] Step S104: Perform Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determine a sound source orientation matrix of the sound event according to the frequency spectrum data of each of the audio signals.

[0048] Perform Fourier transform on each audio signal to obtain a spectrum diagram of each audio signal, and determine the sound source orientation matrix of the sound event based on the spectrum data of each audio signal.

[0049] In one embodiment, if Figure 2 As shown, step S104 includes sub-steps S1041 to S1044.

[0050] Sub-step S1041: determining a first spectrum covariance matrix according to the spectrum data of each of the audio signals.

[0051] Exemplarily, the frequency spectrum data of each audio signal is subjected to noise frequency point removal and echo frequency point removal to obtain target frequency spectrum data of each audio signal; and the first frequency spectrum covariance matrix is ​​determined according to the target frequency spectrum data of each audio signal.

[0052] In one embodiment, the method of removing noise frequency points and echo points from the spectrum data of each audio signal to obtain the target spectrum data of each audio signal includes: removing noise frequency points whose spectrum energy is less than a preset spectrum energy in the spectrum data of each audio signal to obtain the first spectrum data of each audio signal; determining the spectrum energy threshold of the echo points according to the second spectrum data of the audio signal collected by the microphone array at the previous moment; removing the echo points whose spectrum energy is less than the spectrum energy threshold in the first spectrum data of each audio signal to obtain the target spectrum data of each audio signal. By removing noise frequency points and echo points from the spectrum data of each audio signal, the subsequent sound source orientation matrix is ​​more accurate.

[0053] In one embodiment, the preset spectrum energy can be determined by obtaining a preset spectrum energy formula, which is A(t)=α×A(t-1), where A is the preset spectrum energy, t is the time window, and α is the update parameter, wherein the update parameter α is determined according to actual conditions, for example, α<1 in the noise time period, and α>1 in the signal time period. Based on the preset spectrum energy formula, and according to the previous time window and the update parameter, the preset spectrum energy is obtained.

[0054] Sub-step S1042: obtaining a second spectrum covariance matrix, wherein the second spectrum covariance matrix is ​​determined based on spectrum data of the audio signal collected by the microphone array at a previous moment.

[0055] The second spectral covariance matrix is ​​determined based on the spectral data of the audio signal collected by the microphone array at the previous moment, and the second spectral covariance matrix can also be determined based on the target spectral data of the audio signal collected by the microphone array at the previous moment.

[0056] Exemplarily, a second spectrum covariance matrix formula is obtained, and the second spectrum covariance matrix formula is:

[0057]

[0058] Should is the second spectrum covariance matrix, C is the number of time windows, X(t,f) is the Fourier transform coefficient, X H (t, f) is the transpose of Fourier transform coefficients. Based on the second spectrum covariance matrix formula, the second spectrum covariance matrix is ​​obtained according to the number of time windows, Fourier transform coefficients and Fourier transform coefficient matrix.

[0059] Sub-step S1043: determining a steering vector matrix according to the first spectrum covariance matrix and the second spectrum covariance matrix.

[0060] A steering vector of each microphone unit is obtained according to the first spectral covariance matrix and the second spectral covariance matrix; and a steering vector matrix of each sound event is generated based on the steering vector of each microphone unit, wherein the steering vectors of the microphone units in different sound events are different.

[0061] Specifically, the first spectrum covariance matrix formula and the second spectrum covariance matrix formula are combined, and based on each first spectrum covariance matrix and second spectrum covariance matrix, the steering vector of each microphone unit is obtained, and the steering vectors of the same event are integrated to generate a steering vector matrix for each sound event.

[0062] Among them, the first spectrum covariance matrix formula is:

[0063]

[0064] The R(t,f) is the first spectrum covariance matrix, is the energy value of the frequency point, h(f,θ i ) steering vector, h(f,θ i ) H is the transpose of the steering vector, f is the current frequency, t is the time window, and i is the i-th microphone.

[0065] It should be noted that the above first spectrum covariance matrix formula can be simplified to the following formula:

[0066]

[0067] By simplifying the first spectrum covariance matrix formula, the amount of calculation can be reduced and the error can be reduced.

[0068] Among them, the second spectrum covariance matrix formula, the second spectrum covariance matrix formula is:

[0069]

[0070] Should is the second spectrum covariance matrix, C is the number of time windows, X(t,f) is the Fourier transform coefficient, X H (t, f) is the transpose of Fourier transform coefficients. Based on the second spectrum covariance matrix formula, the second spectrum covariance matrix is ​​obtained according to the number of time windows, Fourier transform coefficients and Fourier transform coefficient matrix.

[0071] Sub-step S1044, determining the sound source orientation of each sound event according to each of the generated steering vector matrices, and determining the sound source orientation matrix according to the sound source orientation of each sound event.

[0072] Specifically, the characteristic vector value of the steering vector of each microphone unit is calculated. A first angle calculation formula and a second angle calculation formula are obtained, wherein the first angle calculation formula is:

[0073]

[0074] Among them, θ is the azimuth angle, f is the current frequency, F is the adopted frequency, d is the distance between the two microphones, c is the speed of light, t is the time window, and i is the element position number. is the eigenvector, < is the phase of the eigenvector; the second angle calculation formula is:

[0075]

[0076] Among them, θ is the azimuth angle, f is the current frequency, F is the adopted frequency, d is the distance between the two microphones, c is the speed of light, t is the time window, and i is the element position number. is the eigenvector, < is the phase of the eigenvector, based on the first angle calculation formula and the second angle calculation formula, and according to the phase of the eigenvector, the steering vector of the microphone is determined, and based on the steering vector based on each microphone unit, the sound source orientation matrix of each sound event is generated.

[0077] Step S105: input the category matrix and the sound source orientation matrix into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

[0078] Among them, the preset matching network is a pre-trained neural network model. The choice of the neural network can be selected according to actual conditions. For example, the neural network can be a bidirectional long short-term memory network, a convolutional neural network model, a recurrent neural network model and a recurrent convolutional neural network model. Of course, it can also be other network training models, and this application does not make specific limitations on this.

[0079] It should be noted that the training method of the preset matching network can be: obtaining a category matrix and a sound source orientation matrix, annotating the category matrix and the sound source orientation matrix to construct sample data, and iteratively training the neural network based on the sample data until the neural network model converges, thereby obtaining the preset matching network.

[0080] In one embodiment, the category matrix and the sound source orientation matrix are smoothed; and the smoothed category matrix and the sound source orientation matrix are input into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category. The preset matching network is a pre-trained bidirectional long short-term memory network. The smoothed category matrix and the sound source orientation matrix are input into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category, so as to accurately realize the detection and positioning of the sound event.

[0081] In one embodiment, the smoothing process for the category matrix and the sound source orientation matrix may be: removing the burrs in the time dimension of the category matrix and the sound source orientation matrix to obtain the smoothed category matrix and the sound source orientation matrix. Figure 3 As shown, the amplitude curve A is the category matrix or sound source orientation matrix before smoothing. The amplitude curve A includes burrs B and burrs C. The burrs B and burrs C whose amplitude of dimension change in the time dimension is less than the preset time length are removed to obtain the smoothed amplitude curve B.

[0082] The sound event detection and positioning method provided by the above embodiment obtains multiple audio signals collected by the microphone array, and determines the cross-correlation function matrix between each two audio signals in the multiple audio signals; then each cross-correlation function matrix is ​​input into the preset sound event classification model to obtain the category matrix of the sound event; then each audio signal is Fourier transformed to obtain the spectrum diagram of each audio signal, and the sound source orientation matrix of the sound event is determined according to the spectrum data of each audio signal; finally, the category matrix and the sound source orientation matrix are input into the preset matching network for matching, and the category of each sound event in the audio signal and the sound source orientation matching the category are obtained. By matching the sound event category matrix and the sound source orientation matrix of the audio signal, the matching sound source orientation of each sound event can be accurately determined, which greatly improves the accuracy of sound event detection and positioning.

[0083] See also Figure 4 ,like Figure 4 A schematic block diagram of a sound event detection and positioning device provided in an embodiment of the present application.

[0084] like Figure 4 As shown, the sound event detection and positioning device 200 includes an acquisition module 210, a determination module 220, a generation module 230, and a matching module 240, wherein:

[0085] The acquisition module 210 is used to acquire multiple audio signals collected by the microphone array;

[0086] The determination module 220 is configured to determine a cross-correlation function matrix between every two of the multiple audio signals;

[0087] The generating module 230 is used to input each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of the sound events;

[0088] The determination module 220 is further configured to perform Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determine a sound source orientation matrix of the sound event according to the frequency spectrum data of each of the audio signals;

[0089] The matching module 240 is used to input the category matrix and the sound source orientation matrix into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

[0090] In one embodiment, the determination module 220 is further configured to:

[0091] Performing frame processing on each of the audio signals to obtain a plurality of audio frames corresponding to each of the audio signals;

[0092] determining a cross-correlation function between each of the audio frames of one of the audio signals and a corresponding audio frame of another of the audio signals;

[0093] A cross-correlation function matrix between each two audio signals is determined according to the cross-correlation function between each audio frame of one audio signal and a corresponding audio frame of another audio signal.

[0094] In one embodiment, if Figure 5 As shown, the determination module 220 also includes:

[0095] A first determining submodule 221, configured to determine a first spectrum covariance matrix according to spectrum data of each of the audio signals;

[0096] A first acquisition submodule 222, configured to acquire a second spectrum covariance matrix, wherein the second spectrum covariance matrix is ​​determined based on spectrum data of the audio signal collected by the microphone array at a previous moment;

[0097] A second determining submodule 223, configured to determine a steering vector matrix according to the first spectrum covariance matrix and the second spectrum covariance matrix;

[0098] The third determination submodule 224 is used to determine the sound source orientation of each sound event according to each of the generated steering vector matrices, and determine the sound source orientation matrix according to the sound source orientation of each sound event.

[0099] In one embodiment, the first determining submodule 221 is further configured to:

[0100] Performing noise frequency point removal and echo frequency point removal on the frequency spectrum data of each of the audio signals to obtain target frequency spectrum data of each of the audio signals;

[0101] A first spectrum covariance matrix is ​​determined according to the target spectrum data of each of the audio signals.

[0102] In one embodiment, the first determining submodule 221 is further configured to:

[0103] Remove the noise frequency points whose spectrum energy is less than the preset spectrum energy in the spectrum data of each of the audio signals, so as to obtain the first spectrum data of each of the audio signals;

[0104] Determine a spectrum energy threshold of the echo audio point according to second spectrum data of the audio signal collected by the microphone array at the previous moment;

[0105] The echo points whose spectrum energy in the first spectrum data of each of the audio signals is less than the spectrum energy threshold are removed to obtain the target spectrum data of each of the audio signals.

[0106] In one embodiment, the second determining submodule 223 is further configured to:

[0107] a steering vector of each of the microphone units according to the first spectrum covariance matrix and the second spectrum covariance matrix;

[0108] A steering vector matrix for each sound event is generated based on the steering vector of each of the microphone units, wherein the steering vectors of the microphone units in different sound events are different.

[0109] In one embodiment, the matching module 223 is further configured to:

[0110] Performing smoothing processing on the category matrix and the sound source orientation matrix;

[0111] The smoothed category matrix and the sound source orientation matrix are input into a preset matching network for matching, wherein the preset matching network is a pre-trained bidirectional long short-term memory network.

[0112] See also Figure 6 , Figure 6 A schematic block diagram of the structure of an electronic device provided in an embodiment of the present application.

[0113] like Figure 6 As shown, the electronic device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0114] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the sound event detection and positioning methods.

[0115] The processor is used to provide computing and control capabilities to support the operation of the entire electronic device.

[0116] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any sound event detection and positioning method.

[0117] The network interface is used for communication. Those skilled in the art will appreciate that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] It should be understood that the bus is, for example, an I2C (Inter-integrated Circuit) bus, the memory may be a Flash chip, a read-only memory (ROM) disk, a CD, a USB flash drive or a mobile hard disk, etc., the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0119] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0120] Acquire multiple audio signals collected by the microphone array;

[0121] determining a cross-correlation function matrix between every two of the audio signals in the plurality of audio signals;

[0122] Inputting each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of sound events, wherein the preset sound event classification model is a pre-trained neural network model;

[0123] Performing Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determining a sound source orientation matrix of the sound event according to the frequency spectrum data of each of the audio signals;

[0124] The category matrix and the sound source orientation matrix are input into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

[0125] In one embodiment, when the processor implements the step of determining the cross-correlation function matrix between every two of the multiple audio signals, the processor is configured to implement:

[0126] Performing frame processing on each of the audio signals to obtain a plurality of audio frames corresponding to each of the audio signals;

[0127] determining a cross-correlation function between each of the audio frames of one of the audio signals and a corresponding audio frame of another of the audio signals;

[0128] A cross-correlation function matrix between each two audio signals is determined according to the cross-correlation function between each audio frame of one audio signal and a corresponding audio frame of another audio signal.

[0129] In one embodiment, when the processor determines the sound source orientation matrix of the sound event according to the spectrum data of each of the audio signals, it is configured to implement:

[0130] Determine a first spectrum covariance matrix according to the spectrum data of each of the audio signals;

[0131] Acquire a second spectrum covariance matrix, wherein the second spectrum covariance matrix is ​​determined based on spectrum data of the audio signal collected by the microphone array at a previous moment;

[0132] Determining a steering vector matrix according to the first spectrum covariance matrix and the second spectrum covariance matrix;

[0133] The sound source orientation of each sound event is determined according to each of the generated steering vector matrices, and the sound source orientation matrix is ​​determined according to the sound source orientation of each sound event.

[0134] In one embodiment, when the processor determines the first spectrum covariance matrix according to the spectrum data of each of the audio signals, it is configured to implement:

[0135] Performing noise frequency point removal and echo frequency point removal on the frequency spectrum data of each of the audio signals to obtain target frequency spectrum data of each of the audio signals;

[0136] A first spectrum covariance matrix is ​​determined according to the target spectrum data of each of the audio signals.

[0137] In one embodiment, when the processor removes noise frequency points and echo frequency points from the spectrum data of each of the audio signals to obtain the target spectrum data of each of the audio signals, the processor is used to implement:

[0138] Remove the noise frequency points whose spectrum energy is less than the preset spectrum energy in the spectrum data of each of the audio signals, so as to obtain the first spectrum data of each of the audio signals;

[0139] Determine a spectrum energy threshold of the echo audio point according to second spectrum data of the audio signal collected by the microphone array at the previous moment;

[0140] The echo points whose spectrum energy in the first spectrum data of each of the audio signals is less than the spectrum energy threshold are removed to obtain the target spectrum data of each of the audio signals.

[0141] In one embodiment, when the processor implements determining the steering vector matrix according to the first spectrum covariance matrix and the second spectrum covariance matrix, the processor is used to implement:

[0142] a steering vector of each of the microphone units according to the first spectrum covariance matrix and the second spectrum covariance matrix;

[0143] A steering vector matrix for each sound event is generated based on the steering vector of each of the microphone units, wherein the steering vectors of the microphone units in different sound events are different.

[0144] In one embodiment, the processor implements the preset matching network as a pre-trained bidirectional long short-term memory network, and when the category matrix and the sound source orientation matrix are input into the preset matching network for matching, it is used to implement:

[0145] Performing smoothing processing on the category matrix and the sound source orientation matrix;

[0146] The smoothed category matrix and the sound source orientation matrix are input into a preset matching network for matching, wherein the preset matching network is a pre-trained bidirectional long short-term memory network.

[0147] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned sound event detection and positioning control method embodiment, and will not be repeated here.

[0148] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the sound event detection and positioning method of the present application.

[0149] The computer-readable storage medium may be an internal storage unit of the electronic device described in the foregoing embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0150] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0151] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0152] The serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting and locating a sound event, characterized in that: Applied to an electronic device, the electronic device includes a microphone array, the microphone array includes a plurality of microphone units, and the method includes: Acquire multiple audio signals collected by the microphone array; Performing frame processing on each of the audio signals to obtain a plurality of audio frames corresponding to each of the audio signals; determining a cross-correlation function between each of the audio frames of one of the audio signals and a corresponding audio frame of another of the audio signals; Determine a cross-correlation function matrix between every two of the audio signals according to a cross-correlation function between each of the audio frames of one of the audio signals and a corresponding audio frame of another of the audio signals; Inputting each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of sound events, wherein the preset sound event classification model is a pre-trained neural network model; Performing Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determining a first frequency spectrum covariance matrix according to the frequency spectrum data of each of the audio signals; Acquire a second spectrum covariance matrix, wherein the second spectrum covariance matrix is ​​determined based on spectrum data of the audio signal collected by the microphone array at a previous moment; Determining a steering vector matrix according to the first spectrum covariance matrix and the second spectrum covariance matrix; Determine the sound source orientation of each sound event according to each of the generated steering vector matrices, and determine the sound source orientation matrix according to the sound source orientation of each sound event; The category matrix and the sound source orientation matrix are input into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

2. The sound event detection and positioning method according to claim 1, characterized in that: Determining a first spectrum covariance matrix according to the spectrum data of each of the audio signals comprises: Performing noise frequency point removal and echo frequency point removal on the frequency spectrum data of each of the audio signals to obtain target frequency spectrum data of each of the audio signals; A first spectrum covariance matrix is ​​determined according to the target spectrum data of each of the audio signals.

3. The sound event detection and positioning method according to claim 2, characterized in that: The noise frequency points and the echo frequency points are removed from the spectrum data of each of the audio signals to obtain the target spectrum data of each of the audio signals, including: Remove the noise frequency points whose spectrum energy is less than the preset spectrum energy in the spectrum data of each of the audio signals, so as to obtain the first spectrum data of each of the audio signals; Determine a spectrum energy threshold of the echo audio point according to second spectrum data of the audio signal collected by the microphone array at the previous moment; The echo points whose spectrum energy in the first spectrum data of each of the audio signals is less than the spectrum energy threshold are removed to obtain the target spectrum data of each of the audio signals.

4. The sound event detection and positioning method according to claim 1, characterized in that: The step of determining a steering vector matrix according to the first spectrum covariance matrix and the second spectrum covariance matrix comprises: a steering vector of each of the microphone units according to the first spectrum covariance matrix and the second spectrum covariance matrix; A steering vector matrix for each sound event is generated based on the steering vector of each of the microphone units, wherein the steering vectors of the microphone units in different sound events are different.

5. The sound event detection and positioning method according to any one of claims 1 to 4, characterized in that: The preset matching network is a pre-trained bidirectional long short-term memory network, and the inputting the category matrix and the sound source orientation matrix into the preset matching network for matching includes: Performing smoothing processing on the category matrix and the sound source orientation matrix; The smoothed category matrix and the sound source orientation matrix are input into a preset matching network for matching, wherein the preset matching network is a pre-trained bidirectional long short-term memory network.

6. A sound event detection and positioning device, the sound event detection and positioning device adopts the sound event detection and positioning method according to any one of claims 1 to 5, characterized in that: Applied to an electronic device, the electronic device includes a microphone array, the microphone array includes a plurality of microphone units, the sound event detection and positioning device includes an acquisition module, a determination module, a generation module, and a matching module, wherein: The acquisition module is used to acquire multiple audio signals collected by the microphone array; The determining module is used to determine a cross-correlation function matrix between every two of the multiple audio signals; The generating module is used to input each of the cross-correlation function matrices into a preset sound event classification model to obtain a category matrix of the sound events; The determination module is further used to perform Fourier transform on each of the audio signals to obtain a frequency spectrum of each of the audio signals, and determine a sound source orientation matrix of the sound event according to the frequency spectrum data of each of the audio signals; The matching module is used to input the category matrix and the sound source orientation matrix into a preset matching network for matching, so as to obtain the category of each sound event in the audio signal and the sound source orientation matching the category.

7. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the sound event detection and positioning method as claimed in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the sound event detection and positioning method according to any one of claims 1 to 5 are implemented.

Images