Learning device, acoustic event localization system, learning method, and program

Abstract

This technology provides acoustic event localization that can detect unknown classes by updating a pre-trained acoustic event localization model using a small number of unknown acoustic signals. [Solution] The learning device includes an embedding extraction unit that uses an extraction unit included in a learned acoustic event localization model to extract embeddings for information based on acoustic signals included in prototype creation data; a prototype creation unit that uses the embeddings extracted by the embedding extraction unit and the class information and angle information included in the prototype creation data to create a prototype for each combination of class and angle range; and an update unit that updates the learned acoustic event localization model by adding the prototypes created by the prototype creation unit to the classification unit included in the learned acoustic event localization model.

Description

[Technical Field]

[0001] The present invention relates to a system and method for performing sound event localization and detection (SELD). [Background technology]

[0002] Sound Event Detection (SED) is a task that estimates when and what happened in the surrounding environment, using single or multi-channel acoustic signals as input. Non-patent document 1 is known as a conventional technique for sound event detection.

[0003] Furthermore, acoustic event localization is a task that takes single or multi-channel acoustic signals as input and estimates when, where, and what happened in the surrounding environment. Non-patent document 2 is known as a conventional technique for acoustic event localization.

[0004] Figure 1 is a diagram illustrating an example of acoustic event localization. In this example, it is expected that the safety of the pedestrian 91 can be protected by having the headphones 92, which include a microphone and other sound-collecting device, detect the approach of a vehicle 93 approaching from behind, and notify the pedestrian 91 of the approach of the vehicle 93 and its direction through the headphones 92, which include a speaker. The acoustic event localization system 94 takes an acoustic signal as input and outputs an estimated value of the type of acoustic event and the direction of the sound source. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Jake Snell, Kevin Swersky, Richard S. Zemel, "Prototypical Networks for Few-shot Learning", 31st Conference on Neural Information Processing Systems (NIPS 2017) [Non-Patent Document 2] Sharath Adavanne, Archontis Politis, Joonas Nikunen, Tuomas Virtanen, "Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks", IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 1, p.34 - 48, MARCH 2019. [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] In an environment where a user uses an acoustic event detection system or an acoustic event localization system, unknown sound types (classes) that did not exist during model training may appear.

[0007] In acoustic event detection, few-shot learning and zero-shot learning, which are techniques for updating the system to detect newly occurring acoustic events by newly registering newly occurring acoustic events and the acoustic signals corresponding to the acoustic events in the user environment, are being considered.

[0008] In Few-shot Bioacoustic Event Detection (DCASE2023 Task5), a small number of data of unknown classes are registered in the learned model to detect unknown classes. In Non-Patent Document 1, the following prototype network is used.

[0009] In this prototype network, the embedding of the acoustic data to be registered is calculated using a pre-trained model, and the centroid of the embedding is used as the prototype. Also, at the time of estimation, an event is detected by searching for the prototype with the shortest distance. Figure 2 is a diagram for explaining the prototype and estimation of the prototype network. The few-shot prototypes c n (n = 1, 2, 3) are calculated as the average of the embeddings for each class. The zero-shot prototypes c n (n = 1, 2, 3) are generated by embedding the class metadata v n (n = 1, 2, 3). In the figure, x is the embedding corresponding to the acoustic signal to be estimated, and an event is detected by searching for the prototype with the shortest distance from this embedding.

[0010] However, in the acoustic event localization task that requires estimating both angle information and class information, it is difficult to directly apply a conventional prototype network (see Non-Patent Document 1). This is because in the embedding representation space, the embeddings obtained from acoustic data of the same class arriving from different angles do not necessarily belong to the same cluster. To confirm this, a preliminary experiment was conducted. In the preliminary experiment, SELDnet, which is the baseline of the DCASE2023 SELD Task, was used as the network, STARSS2023 was used as the dataset, t-SNE was used as the dimensionality reduction method to reduce 128 dimensions to 2 dimensions, and only the embeddings labeled with a single class were visualized. From this preliminary experiment, it was confirmed that the distribution of the same class is divided within the embedding space to form multiple clusters.

[0011] There is a need to develop an acoustic event localization system that can detect newly generated acoustic events in the user environment and corresponding acoustic signals by newly registering them to update the system. Here, such a problem setting is called zero-shot acoustic event localization.

[0012] The present invention aims to provide a learning device, an acoustic event localization system, a learning method, and a program that can learn an acoustic event localization model that can detect unknown classes of acoustic signals by updating a learned acoustic event localization model using a small number of unknown classes of acoustic signals. [Means for solving the problem]

[0013] To solve the above problems, according to one aspect of the present invention, a trained acoustic event localization model is a model that takes information based on an acoustic signal as input and outputs class information indicating an estimated value of the type of acoustic event corresponding to the acoustic signal, and angle information indicating an estimated value of the sound source direction of the acoustic event, and includes an extraction unit that extracts embeddings from information based on an acoustic signal, and a classification unit that estimates the type of acoustic event from the embeddings, and prototype creation data includes information based on an acoustic signal, class information, and angle information, and the class information included in the prototype creation data is an unknown class indicating a class not included in the acoustic events detectable by the trained acoustic event localization model, and the learning device includes an embedding extraction unit that extracts embeddings for information based on an acoustic signal included in the prototype creation data using the extraction unit included in the trained acoustic event localization model, a prototype creation unit that creates a prototype for each combination of class and angle range using the embeddings extracted by the embedding extraction unit and the class information and angle information included in the prototype creation data, and an update unit that updates the trained acoustic event localization model by adding the prototype created by the prototype creation unit to the classification unit included in the trained acoustic event localization model. [Effects of the Invention]

[0014] According to the present invention, by updating a pre-trained acoustic event localization model using data from a small number of unknown classes, it is possible to achieve acoustic event localization that can detect unknown classes. [Brief explanation of the drawing]

[0015] [Figure 1] A diagram illustrating an example of acoustic event localization. [Figure 2] A diagram illustrating the prototype and estimation of a prototype network. [Figure 3] Functional block diagram of the acoustic event localization system according to the first embodiment. [Figure 4] A diagram showing an example of the processing flow of the acoustic event localization system according to the first embodiment. [Figure 5] A figure showing the estimation results of the acoustic event localization system according to the first embodiment. [Figure 6] A diagram showing an example of a computer configuration to which this method is applied. [Modes for carrying out the invention]

[0016] Embodiments of the present invention will be described below. In the drawings used in the following description, components with the same function or steps that perform the same processing will be denoted by the same reference numeral, and redundant explanations will be omitted. In the following description, unless otherwise specified, the processing performed on each element of a vector or matrix will be applied to all elements of that vector or matrix.

[0017] <Key points of the first embodiment> To address the shortcomings of conventional methods, the prototype network will be improved as follows.

[0018] In the improved prototype network, for newly registered acoustic events, an acoustic signal is registered for each angle, and a prototype is created for each combination of class and angle range. This is expected to ensure that each prototype represents a single cluster for each combination of class and angle range.

[0019] <First Embodiment> Figure 3 shows a functional block diagram of the acoustic event localization system according to the first embodiment, and Figure 4 shows its processing flow.

[0020] The processing in the acoustic event localization system consists of a learning stage for learning an acoustic event localization model and an estimation stage for performing an acoustic event localization task on an acoustic signal to be estimated.

[0021] The acoustic event localization system includes a learning device 100 that executes the learning stage and an estimation device 200 that executes the estimation stage.

[0022] Furthermore, the learning device 100 includes a base model learning unit 110, an embedding extraction unit 120, a prototype creation unit 130, and an update unit 140.

[0023] The estimation device 200 includes an estimation unit 210.

[0024] The learning device 100 takes as inputs the base model learning data X L and the prototype creation data X P and uses the base model learning data X L to learn a base model, uses the prototype creation data X P to update the learned base model M L and outputs the updated learned base model as the acoustic event localization model M S to be used in the estimation stage.

[0025] The estimation device 200 receives the acoustic event localization model M S Furthermore, the estimation device 200 takes as an input the acoustic signal x to be estimated, uses the received acoustic event localization model M S to perform an acoustic event localization task, and outputs, as an estimation result, an estimated value y~=(e,d) of the type and sound source direction of the acoustic event corresponding to the acoustic signal x to be estimated.

[0026] The learning device 100 and the estimation device 200 are special devices configured by loading a special program onto a known or dedicated computer having, for example, a central processing unit (CPU) and main memory (RAM). The learning device 100 and the estimation device 200 execute their respective processes under the control of, for example, the central processing unit. Data input to the learning device 100 and the estimation device 200 and data obtained from each process are stored, for example, in main memory, and the data stored in main memory is read to the central processing unit as needed and used for other processes. Each processing unit of the learning device 100 and the estimation device 200 may be composed of hardware such as integrated circuits, at least in part. Each storage unit of the learning device 100 and the estimation device 200 can be composed of, for example, main memory such as RAM (Random Access Memory), or middleware such as a relational database or key-value store. However, each storage unit does not necessarily have to have the learning device 100 and estimation device 200 inside it. It may be configured as an auxiliary storage device consisting of a hard disk, optical disk, or semiconductor memory element such as flash memory, and may be provided outside the learning device 100 and estimation device 200.

[0027] First, let me explain the learning stages.

[0028] <Base Model Learning Unit 110> The base model training unit 110 uses the base model training data X L The input is the acoustic event localization model M L The trained acoustic event localization model M is learned (S110). L Outputs the following: Base model training data X L This includes an acoustic signal or acoustic features extracted from an acoustic signal (hereinafter collectively referred to as "information based on acoustic signals"), class information indicating the type of acoustic event, and angular information indicating the direction of the sound source of the acoustic event. The class information and angular information are used in the acoustic event localization model M.L This is the correct data for training. For example, an acoustic signal can be a single-channel or multi-channel acoustic signal. For example, if the acoustic signal is an M-channel acoustic signal, then the training data X for the base model is... L The information based on the acoustic signals contained in [X 1,L ,X 2,L ,…,X M,L ] and if each channel m (m=1,2,…,M) contains an acoustic signal for T time, then information X based on the acoustic signal of channel m m,L X m,L =[x 1,m,L ,x 2,m,L ,…,x T,m,L ] and the information based on the acoustic signal of channel m at time t (t=1,2,…,T) is x t,m,L That is the case.

[0029] Acoustic Event Localization Model M L This is a standard model capable of performing acoustic event localization, taking information based on acoustic signals as input and outputting estimated values ​​of class information and angle information. Based on prior art such as Non-Patent Document 2, the base model is trained using data X. L It can be learned using this method.

[0030] Acoustic Event Localization Model M L The system includes an extraction unit E that extracts an embedded z from information x based on an acoustic signal, and a classification unit C that estimates an acoustic event from the embedded z. If the output is y~, it can be expressed by the following equation.

[0031] y = M(x) = C(E(x)) (1) <Embedded extraction section 120> Prior to the extraction process, the embedded extraction unit 120 uses a pre-learned acoustic event localization model M L Receive.

[0032] The embedded extraction unit 120 extracts prototype creation data X P Information x based on the acoustic signal contained within p The input is the acoustic event localization model M LUsing the extraction unit E contained within, prototype creation data X is generated by the following equation. P Embedding z for P Extract (S120) and output.

[0033] z P =E(x p ) Note: Prototype creation data X P This includes information based on the acoustic signal, as well as class information and angle information corresponding to the acoustic signal. However, the class information is based on the acoustic event localization model M. L This is an unknown class that indicates a class not included in the acoustic events detectable by [the system]. Prototype creation data X used in the embedded extraction unit 120. P This includes information based on at least acoustic signals.

[0034] <Prototype Creation Section 130> The prototype creation unit 130 generates prototype creation data X P Embedding z for the acoustic signal contained within P And, prototype creation data X P The class information and angle information contained within are used as input, and the prototype creation data X P The class information and angle information contained within are used to determine the correct label Y. P It is used as such, and a prototype is created (S130) for each combination of class and angle range, and output. For each combination of class and angle range, the centroid of the embedding is found, and the found centroid is used as the prototype. The operation of finding the centroid of the embedding representation is the same as the derivation of the prototype in Non-Patent Literature 1, and is calculated as the average of the embeddings for each combination of class and angle range. For example, prototype creation data X P While the angle information included is in 1-degree increments, the angle range is set to 10-degree increments during prototype creation. The difference from conventional technology is that the prototype is derived for each angle range. The size of the angle range can be set according to the accuracy required during estimation; for example, if the four directions (front, back, left, and right) are divided, a 90-degree increment may be used. Also, the prototype creation data X used in the prototype creation unit 130 PThis includes at least class information and angle information.

[0035] <Update section 140> The update unit 140 displays prototypes and trained acoustic event localization models M for each combination of class and angle range. L The input is a trained acoustic event localization model M. L The classification unit included in the pre-trained acoustic event localization model M adds prototypes for each combination of class and angular range. L The updated (S140) trained acoustic event localization model M L Acoustic event localization model M that uses the estimation stage S Output as follows.

[0036] Next, I will explain the estimation stage.

[0037] <Estimation section 210> The estimation unit 210 uses the acoustic event localization model M S It receives the acoustic event localization model M, which is the acoustic signal x to be estimated as input. S Using this method, an acoustic event localization task is performed on the acoustic signal x (S210), and estimated class information and angular information y~=(e,d) are output as estimation results. However, the angular information output here indicates the angular range.

[0038] For example, acoustic event localization for the acoustic signal x to be estimated is performed in the same manner as in Non-Patent Document 1. That is, the combination of angular range and class to which the prototype closest to the embedding z(t)=E(x(t)) at each time t extracted from the data x=[x(1),x(2),…,x(t),…x(T')] belongs is taken as the estimation result.

[0039] <Effects> By using this configuration, it is possible to update a pre-trained acoustic event localization model with data from a small number of unknown classes, thereby achieving acoustic event localization that can detect unknown classes.

[0040] <Experimental Results> (Experiment preparation) To perform zero-shot training, we prepare a dataset in which the test data contains unknown classes.

[0041] Since the STARSS2023 dataset contains the same classes in both training and test data, in this experiment, we will remove the time domain of a specific class from the STARSS2023 dataset to use as training data for the base model, treat the removed class as an unknown class, and construct a dataset using the removed data as prototype data (zero-shot training data).

[0042] For example, the training data for the base model is created by removing the "Telephone" event from the training data based on DCASE2023 Task3.

[0043] The SELDnet model from the DCASE2023 Task3 baseline in Non-Patent Document 2 is used as the acoustic event localization model. During estimation, classification is performed using the prototype network without using the discriminative layer. Adam is used as the optimization algorithm, with 100 epochs and a batch size of 32.

[0044] Figure 5 shows the estimation results of the updated acoustic event localization model from this experiment. In the figure, Accuracy is the F1-score, which represents the F-value for event detection, and is a value between 0 and 1, with a value closer to 1 indicating better accuracy. In the figure, Le macro(°) represents the error in sound source direction estimation, with a value closer to 0° indicating better accuracy, and a value between 0 and 90° being considered accurate.

[0045] This embodiment demonstrates that the Telephone class not included during training can be identified with 77.1% accuracy. Note that the conventional method (Non-Patent Literature 2) lacks an estimation mechanism, and therefore its accuracy is undefined.

[0046] <Variation> The learning device 100 uses a pre-learned acoustic event localization model M L The base model training data X may be used as input and used in the embedding extraction unit 120 and the update unit 140. In this case, the learning device 100 uses the base model training data X L It does not require the input and does not need to include the base model learning unit 110.

[0047] <Other variations> The present invention is not limited to the embodiments and modifications described above. For example, the various processes described above may not only be performed sequentially as described, but may also be performed in parallel or individually as needed, depending on the processing capacity of the device performing the processes. Other modifications can be made as appropriate without departing from the spirit of the present invention.

[0048] <Hardware, programs, and recording media> The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0049] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0050] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0051] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 6, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0052] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0053] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0054] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed by a so-called ASP (Application Service Provider) type service, where the processing function is realized only by execution instructions and result acquisition, without transferring the program from the server computer to this computer. Furthermore, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. In this form, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the computer's processing).

[0055] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

[0056] <Other variations> The present invention is not limited to the embodiments and modifications described above. For example, the various processes described above may not only be performed sequentially as described, but may also be performed in parallel or individually as needed, depending on the processing capacity of the device performing the processes. Other modifications can be made as appropriate without departing from the spirit of the present invention.

Claims

1. A trained acoustic event localization model is a model that takes information based on an acoustic signal as input and outputs class information indicating an estimated value of the type of acoustic event corresponding to the acoustic signal, and angle information indicating an estimated value of the sound source direction of the acoustic event, and includes an extraction unit that extracts embeddings from information based on an acoustic signal, and a classification unit that estimates the type of acoustic event from the embeddings. The prototype creation data includes information based on acoustic signals, class information, and angle information, and the class information included in the prototype creation data is an unknown class that indicates a class not included in the acoustic events detectable by the trained acoustic event localization model. An embedding extraction unit that uses an extraction unit included in a pre-trained acoustic event localization model to extract embeddings for information based on acoustic signals included in prototype creation data, A prototype creation unit creates a prototype for each combination of class and angle range using the embedding extracted by the embedding extraction unit and the class information and angle information contained in the prototype creation data. The system includes an update unit that updates the learned acoustic event localization model by adding the prototype created by the prototype creation unit to the classification unit included in the learned acoustic event localization model. Learning device.

2. An acoustic event localization system including the learning device of claim 1, The system includes an estimation unit that performs an acoustic event localization task on an acoustic signal to be localized using the updated, previously trained acoustic event localization model. Acoustic event localization system.

3. A trained acoustic event localization model is a model that takes information based on an acoustic signal as input and outputs class information indicating an estimated value of the type of acoustic event corresponding to the acoustic signal, and angle information indicating an estimated value of the sound source direction of the acoustic event, and includes an extraction unit that extracts embeddings from information based on an acoustic signal, and a classification unit that estimates the type of acoustic event from the embeddings. The prototype creation data includes information based on acoustic signals, class information, and angle information, and the class information included in the prototype creation data is an unknown class that indicates a class not included in the acoustic events detectable by the trained acoustic event localization model. An embedding extraction step which uses the extraction unit included in a trained acoustic event localization model to extract embeddings for information based on acoustic signals included in prototype creation data, A prototype creation step, which creates a prototype for each combination of class and angle range using the embedding extracted in the embedding extraction step and the class information and angle information contained in the prototype creation data, The update step includes updating the learned acoustic event localization model by adding the prototype created in the prototype creation step to the classification unit included in the learned acoustic event localization model. Learning methods.

4. A program for causing a computer to function as a learning device according to claim 1.

Images