Acoustic event localization method, acoustic event localization device, and program

Abstract

When d1, ..., dN are defined as directions which are vectors used to represent the occurrence direction of an event corresponding to an acoustic signal, an acoustic event localization method according to the present invention comprises: a beamforming step for generating acoustic signals xd_i (i=1, ..., N) by beamforming an input acoustic signal x in directions di; a first type estimation step for calculating, from the acoustic signals xd_i (i=1, ..., N), probabilities p(y|xd_i) (i=1, ..., N) that the type of an event corresponding to each of the acoustic signals xd_i is y; a second type estimation step for calculating, from the probabilities p(y|xd_i) (i=1, ..., N), the probability p(y|x) that the type of an event corresponding to the acoustic signal x is y; and an occurrence direction estimation step for calculating, from the probabilities p(y|xd_i) (i=1, ..., N), an occurrence direction I of the event corresponding to the acoustic signal x.

Description

Acoustic event localization method, acoustic event localization device, and program

[0001] The present invention relates to acoustic event localization techniques.

[0002] Sound Event Localization and Detection (SELD) is a task that uses acoustic signals as input to estimate the type and direction of an event occurring in the surrounding environment. Deep neural networks (DNNs) that achieve sound event localization use training data consisting of multi-channel acoustic signals acquired using, for example, a first-order Ambisonics microphone, and a set of labels indicating the type of event corresponding to the acoustic signal and the direction of the event. However, the cost of collecting the large amount of training data required for this training is extremely high, and thus a sufficient amount of training data has not yet been obtained.

[0003] To address this problem of insufficient training data, Non-Patent Document 1 proposes a method that enables estimation of event types that are not included in the training data.

[0004] Kazuki Shimada, Kengo Uchida, Yuichiro Koyama, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji, Tatsuya Kawahara, "Zero- and Few-shot Sound Event Localization and Detection," Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.

[0005] However, the method of Non-Patent Document 1 still uses the conventional framework for estimating the direction of an event, and therefore the problem of a lack of acoustic signals labeled with an indication of the direction of an event remains unresolved.

[0006] Therefore, an object of the present invention is to provide an acoustic event localization technique based on learning using acoustic signals to which only labels indicating the type of event are attached.

[0007] One aspect of the present invention is that d1, ..., d N (where N is an integer greater than or equal to 2, and d1, …, d N are different from each other) is a vector used to represent the direction of occurrence of an event corresponding to an acoustic signal, and the acoustic event localization device converts the input acoustic signal x into a direction d i By beamforming to d_i (i=1, ..., N), and the acoustic event localization device generates the acoustic signal x d_i (i=1, …, N) to the corresponding acoustic signal x d_i The probability that the event type corresponding to x is y is p(y|x d_i ) (i=1, ..., N), and the acoustic event localization device calculates the probability p(y|x d_i ) (i=1, ..., N) to calculate a probability p(y|x) that the type of event corresponding to the acoustic signal x is y; and d_i ) (i=1, . . . , N), and calculating the direction of occurrence I of the event corresponding to the acoustic signal x from the corresponding acoustic signal x.

[0008] According to the present invention, it is possible to realize acoustic event localization based on learning using acoustic signals to which only labels indicating the type of event are attached.

[0009] FIG. 1 is a block diagram showing the configuration of a neural network learning device 100. FIG. 2 is a flowchart showing the operation of the neural network learning device 100. FIG. 3 is a block diagram showing the configuration of a sound event localization device 200. FIG. 4 is a flowchart showing the operation of the sound event localization device 200. FIG. 5 is a diagram showing an example of the functional configuration of a computer that realizes each device in an embodiment of the present invention.

[0010] Hereinafter, an embodiment of the present invention will be described in detail. Note that components having the same functions are given the same numbers and redundant explanations will be omitted.

[0011] Before describing each embodiment, the notation used in this specification will be explained.

[0012] ^ (caret) represents a superscript, e.g., x y^z Yes z is a superscript to x, and x y^z Yes z is a subscript to x. Also, _ (underscore) represents a subscript. For example, x y_z Yes z is a superscript to x, and x y_z Yes z is a subscript to x.

[0013] The superscripts "^" and "~" such as ^x and ~x for a certain letter x should be written directly above the "x", but due to restrictions on the notation in the specification, they are written as ^x and ~x.

[0014] <Technical Background> In an embodiment of the present invention, Multiple Instance Learning (MIL), a weak-label learning technique, is applied to acoustic event localization. This enables acoustic event localization based on learning using acoustic signals that are only labeled with an event type, thereby resolving the problem of insufficient training data in training deep neural networks that perform acoustic event localization.

[0015] <<1: Multiple Instance Learning>> Multiple instance learning can be used in classification problems where paired data of input x and output y cannot be used in estimating the posterior probability p(y|x) that the output is y∈Y given an input x∈X (where X and Y are a set of inputs and a set of outputs, respectively). In multiple instance learning, instead of paired data of input x and output y, N mutually different inputs x1, ..., x N A bag b={x1, …, x NWe deal with a situation where pair data of inputs x∈X and output y are given. Here, the posterior probability p(y|x) that the output is y∈Y given an input x∈X and the posterior probability p(y|b) that the output is y∈Y given a bag b⊆X, which is a set of inputs, are related by the following equation: In practice, instead of calculating the integral on the right hand side of equation (1), the following equation is used to calculate a finite sum over the bag b: where w i (i=1, …, N) is the posterior probability p(x i |b) and the posterior probability p(y|x i ) is called the weight.

[0016] Therefore, in multiple instance learning, the pair data of bag b and output y is used as training data, and the posterior probability p(y|x i ) is trained. For training the neural network, for example, binary cross entropy loss (BCE loss) can be used between the posterior probability p'(y|b) that the output is y∈Y under the condition that a bag b⊆X is given and the posterior probability p(y|b) that is the output of the neural network, which is calculated using training data.

[0017] <<2: Application of multiple instance learning to acoustic event localization>> A three-dimensional vector space R is used to represent the direction of occurrence of an event corresponding to an acoustic signal. 3 (where R is the set of real numbers) in the directions d1, …, d N (where N is an integer greater than or equal to 3, and d1, …, d N In other words, the direction of the event is determined by the direction d1, ..., d N It can be expressed as a linear combination of

[0018] Consider a pair of data consisting of a multi-channel audio signal x and an event type y corresponding to the audio signal x. In other words, y represents the type of event under the condition that the multi-channel audio signal x is given. Here, x d_i (i=1, …, N) is a multi-channel acoustic signal x in the direction d i Let b = {x d_1 , …, x d_N}, the following equation is obtained from equation (2): Therefore, using the pair data of the multi-channel acoustic signal x and the event type y corresponding to the acoustic signal x as training data, the posterior probability p(y|x d_i It is possible to train a neural network (i=1, …, N) that computes

[0019] Here, the direction I of an event corresponding to the multi-channel acoustic signal x is defined as the direction d1, ..., d N The posterior probability p(y|x d_i ) is the direction d i This assumption is based on the magnitude of the component in the direction d i The closer to the direction of the event, the greater the posterior probability p(y|x d_i ) is obtained. Based on this assumption, the direction of occurrence of the event I can be calculated using the following formula: In other words, the posterior probability p(y|x d_i ) is the direction d i The direction of the event, I, can be calculated by the above learning. d_i ) can be calculated directly from

[0020] In the above explanation, the directions d1, ..., d N is assumed to be a three-dimensional vector, but the directions d1, …, d N can be a two-dimensional vector. That is, we use a two-dimensional vector space R to represent the direction of occurrence of an event corresponding to an acoustic signal. 2(where R is the set of real numbers) in the directions d1, …, d N (where N is an integer greater than or equal to 2, and d1, …, d N are different from each other).

[0021] First Embodiment Neural Network Training Device 100 The neural network training device 100 trains a neural network NN by multiple instance training using a set of an audio signal x and an event type y corresponding to the audio signal x as training data. SELD Here, the acoustic signal x, which is an element of the training data, is a multi-channel acoustic signal acquired using, for example, an Ambisonics microphone. N (where N is an integer greater than or equal to 2, and d1, …, d N are different from each other) is used to represent the direction of an event corresponding to an acoustic signal. d_i (i=1, …, N) is an acoustic signal x in the direction d i The acoustic signal obtained by beamforming to d_i ) (i=1, …, N) into the acoustic signal x d_i If the probability that the type of event corresponding to is y, then the neural network NN SELD is the acoustic signal x d_1 , …, x d_N is input, and the probability p(y|x d_1 ), …, p(y|x d_N ) and outputs the acoustic signal x d_i From the probability p(y|x d_i ) is constructed as a neural network including neural networks (i=1, …, N) that calculate the acoustic signal x d_i (i=1, …, N) is the direction d obtained from the acoustic signal x by beamforming. i The single-channel acoustic signal is an enhancement signal of

[0022] The neural network training device 100 will be described below with reference to FIGS. 1 and 2. FIG. 1 is a block diagram showing the configuration of the neural network training device 100. FIG. 2 is a flowchart showing the operation of the neural network training device 100. As shown in FIG. 1, the neural network training device 100 includes an initialization unit 110, a beamforming unit 120, a first type estimation unit 130, a second type estimation unit 140, a parameter updating unit 150, a termination condition determination unit 160, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the neural network training device 100. The recording unit 190 records, for example, training data in advance.

[0023] The operation of the neural network learning device 100 will be described with reference to FIG.

[0024] In S110, the initialization unit 110 initializes the neural network NN SELD The initialization unit 110 initializes the model parameters of the model parameters by using, for example, random numbers. The initialization unit 110 also initializes a counter t that indicates the number of times learning has been performed. That is, the initialization unit 110 sets t=0.

[0025] In S120, the beamforming unit 120 receives as input a set of an acoustic signal x and an event type y corresponding to the acoustic signal x, which are learning data pre-recorded in the recording unit 190, and converts the acoustic signal x into a signal from a direction d i By beamforming to d_i (i=1, ..., N) is generated and output by the beamforming unit 120. d_i Any beamforming algorithm can be used to generate (i=1, ..., N).

[0026] In S130, the first type estimation unit 130 estimates the acoustic signal x generated in S120. d_i (i=1, …, N) as input, and the acoustic signal x d_i (i=1, …, N) to the acoustic signal x d_i The probability that the event type corresponding to x is y is p(y|x d_i) (i=1, ..., N) and outputs it. SELD It consists of:

[0027] In S140, the second type estimation unit 140 calculates the probability p(y|x d_i ) (i=1, …, N) as input, and the probability p(y|x d_i ) (i=1, …, N) to the acoustic signal x d_1 , …, x d_N The probability that the event type corresponding to x is y is p(y|{x d_1 , …, x d_N The second type estimation unit 140 calculates and outputs b={x d_1 , …, x d_N}, and then use equation (3) to determine the probability p(y|{x d_1 , …, x d_N}).

[0028] In S150, the parameter update unit 150 compares the combination of the acoustic signal x, which is the training data, and the event type y corresponding to the acoustic signal x with the probability p(y|{x d_1 , …, x d_N}) is input, and the acoustic signal x is generated using the event type y, which is an element of the training data. d_1 , …, x d_N The probability that the event type corresponding to x is y is p'(y|{x d_1 , …, x d_N}) and calculate the probability p(y|{x d_1 , …, x d_N}) and probability p'(y|{x d_1 , …, x d_N}) using the loss between the neural network NN SELD The parameter update unit 150 updates and outputs the model parameters of the model. The parameter update unit 150 can use binary cross-entropy loss as the loss. The parameter update unit 150 can also update the model parameters using, for example, backpropagation.

[0029] In S160, the termination condition determination unit 160 determines whether a predetermined termination condition is satisfied, and if the condition is satisfied, the neural network NN SELD If the model parameter is not satisfied, the value of the counter t is incremented by 1 and the process returns to step S120. The termination condition can be, for example, that the counter t is equal to or greater than T, where T is the upper limit of the number of learning times.

[0030] <<Acoustic Event Localization Device 200>> The acoustic event localization device 200 estimates, from an input acoustic signal x, the type of event corresponding to the acoustic signal x and the occurrence direction I of the event corresponding to the acoustic signal x. Here, the type of event corresponding to the acoustic signal x refers to the probability p(y|x) that the type of event corresponding to the acoustic signal x is y. Furthermore, the input acoustic signal x is, for example, a multi-channel acoustic signal acquired using an Ambisonics microphone.

[0031] The acoustic event localization device 200 will be described below with reference to Figs. 3 and 4. Fig. 3 is a block diagram showing the configuration of the acoustic event localization device 200. Fig. 4 is a flowchart showing the operation of the acoustic event localization device 200. As shown in Fig. 3, the acoustic event localization device 200 includes a beamforming unit 120, a first type estimation unit 130, a second type estimation unit 140, a direction of occurrence estimation unit 250, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for the processing of the acoustic event localization device 200. The recording unit 290 may record, for example, the neural network NN trained by the neural network training device 100. SELD The model parameters are recorded in advance. The model parameters are used to configure the first type estimation unit 130.

[0032] The operation of the sound event localization device 200 will be described with reference to FIG.

[0033] In S120, the beamforming unit 120 receives an acoustic signal x and converts the acoustic signal x into a beam of light in a direction d. i By beamforming to d_i (i=1, …, N) and output it.

[0034] In S130, the first type estimation unit 130 estimates the acoustic signal x generated in S120. d_i (i=1, …, N) as input, and the acoustic signal x d_i (i=1, …, N) to the acoustic signal x d_i The probability that the event type corresponding to x is y is p(y|x d_i ) (i=1, …, N) and output it.

[0035] In S140, the second type estimation unit 140 calculates the probability p(y|x d_i ) (i=1, …, N) as input, and the probability p(y|x d_i ) (i=1, …, N) to the acoustic signal x d_1 , …, x d_N The probability that the event type corresponding to x is y is p(y|{x d_1 , …, x d_N}) and calculate the probability p(y|{x d_1 , …, x d_N}) is output as the probability p(y|x) that the type of event corresponding to the acoustic signal x is y.

[0036] In S250, the direction of occurrence estimation unit 250 calculates the probability p(y|x d_i ) (i=1, …, N) as input, and the probability p(y|x d_i ) (i=1, . . . , N), the direction of occurrence I of the event corresponding to the acoustic signal x is calculated and output. The direction of occurrence estimation unit 250 calculates the direction of occurrence I using equation (4).

[0037] According to an embodiment of the present invention, it is possible to realize acoustic event localization based on learning using acoustic signals to which only labels indicating the type of event are attached.

[0038] <Additional Notes> The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0039] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0040] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0041] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 5, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0042] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0043] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0044] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its storage device and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of the server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0045] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0046] The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed.

Claims

1. d1, …, d N (where N is an integer greater than or equal to 2, and d1, …, d N are different from each other) is a vector used to represent the direction of occurrence of an event corresponding to an acoustic signal, and the acoustic event localization device converts the input acoustic signal x into a direction d i By beamforming to d_i (i=1, ..., N), and the acoustic event localization device generates a beamforming step of the acoustic signal x d_i (i=1, …, N) to the corresponding acoustic signal x d_i The probability that the event type corresponding to x is y is p(y|x d_i ) (i=1, ..., N), and the acoustic event localization device calculates the probability p(y|x d_i ) (i=1, ..., N) to calculate a probability p(y|x) that the type of event corresponding to the acoustic signal x is y; and d_i ) (i=1, . . . , N) to calculate an event occurrence direction I corresponding to the acoustic signal x.

2. An acoustic event localization method according to claim 1, wherein the first type estimation step is performed by a neural network trained by multiple instance learning using a set of an acoustic signal and the type of event corresponding to the acoustic signal as training data.

3. d1, …, d N (where N is an integer greater than or equal to 2, and d1, …, d N are different from each other) is a vector used to represent the direction of an event corresponding to an acoustic signal, and the input acoustic signal x is expressed as a direction d i By beamforming to d_i (i=1, ..., N), and a beamforming unit that generates the acoustic signal x d_i (i=1, …, N) to the corresponding acoustic signal x d_i The probability that the event type corresponding to x is y is p(y|x d_i ) (i=1, ..., N), and a first type estimator that calculates the probability p(y|x d_i ) (i=1, ..., N), a second type estimation unit that calculates a probability p(y|x) that the type of event corresponding to the acoustic signal x is y; d_i ) (i=1, . . . , N), and an occurrence direction estimation unit that calculates an occurrence direction I of an event corresponding to the acoustic signal x from the acoustic signal x (i=1, . . . , N).

4. A program for causing a computer to execute the acoustic event localization method according to claim 1 or 2.

Images