Sound event localization device, sound event localization method, and program

Abstract

This sound event localization device includes: a sound signal acquisition unit that acquires a sound signal from a multi-channel microphone array worn on a body; a sensor signal acquisition unit that acquires a sensor signal from a sensor that tracks the body; and an SELD unit that executes sound event localization on the basis of a deep learning model in which a combination of the sound signal and the sensor signal is input.

Description

Acoustic event localization device, acoustic event localization method, and program

[0001] The present disclosure relates to a sound event localization device, a sound event localization method, and a program for performing sound event localization and detection (SELD).

[0002] This disclosure focuses on Sound Event Localization and Detection (SELD), a technology that identifies the type, location, and time of an event occurring in the surrounding environment solely from acoustic signals obtained from a microphone array. This disclosure focuses in particular on Wearable SELD, which uses microphones built into a device worn by a person. If Wearable SELD can be realized, it will be possible to inform hearing-impaired people and users who cannot hear surrounding sounds while listening to music of their surroundings and provide a safe environment.

[0003] Many conventional SELD systems use first-order Ambisonics (FOA) signals as input and deep neural networks (DNNs) as a regression function for direction of arrival (DOA) and a classification function for SED to perform SELD inference. In recent years, SELD has become more complex, approaching the real world. State-of-the-art models have successfully performed SELD under conditions involving multiple moving sound sources and obstructions moving around the microphone. This technology is useful for applications that analyze and utilize events occurring in a room using fixed devices, such as smart speakers.

[0004] K. Nagatomo, M. Yasuda, et. al., “Wearable Seld Dataset: Dataset For Sound Event Localization And Detection Using Wearable Devices Around Head”, ICASSP2022

[0005] On the other hand, wearable SELD is a field that has received little research. Wearable SELD differs from conventional SELD using stationary devices in two major respects. First, the microphone array placement is restricted because it uses microphones worn by the user. Second, unlike stationary devices, the microphone array itself undergoes constant translational and rotational motion. To address the first issue, previous research has proposed a dataset for optimal microphone placement for SELD using a Head and Torso Simulator (HATS) or multiple microphones attached to headphones, and conducted extensive experiments (Non-Patent Document 1). However, the second issue has not yet been addressed. In particular, the rotational motion of the device itself due to the rotation of the body or head causes extremely high-speed relative motion of the sound source, which is thought to have a devastating impact on SELD performance.

[0006] Therefore, an object of the present disclosure is to provide a sound event localization device that does not lose accuracy when performing wearable SELD even when the wearer moves or rotates.

[0007] The acoustic event localization device of the present disclosure includes an acoustic signal acquisition unit, a sensor signal acquisition unit, and a SELD unit.

[0008] The acoustic signal acquisition unit acquires acoustic signals from a multi-channel microphone array attached to the body. The sensor signal acquisition unit acquires sensor signals from sensors that track the body. The SELD unit performs acoustic event localization based on a deep learning model that uses a combination of the acoustic signals and the sensor signals as input.

[0009] According to the acoustic event localization device of the present disclosure, when performing Wearable SELD, accuracy does not deteriorate even when the wearer moves or rotates.

[0010] 1 is a block diagram showing the functional configuration of a sound event localization apparatus according to a first embodiment; FIG. 2 is a flowchart showing the operation of the sound event localization apparatus according to a first embodiment; FIG. 3 is a block diagram showing the functional configuration of a sound event localization apparatus according to a first embodiment; FIG. 4 is a diagram showing an example of an earphone with a built-in microphone and a sensor; FIG. 5 is a flowchart showing the event detection operation of the sound event localization apparatus according to a first embodiment;

[0011] Hereinafter, embodiments of the present disclosure will be described in detail. Note that components having the same functions are assigned the same numbers, and redundant explanations will be omitted.

[0012] Humans can not only grasp the location and type of a sound source even when their heads are moving, but can also recognize the location of a sound source more accurately than when their heads are stationary. The acoustic event localization device of the first embodiment takes advantage of this fact.

[0013] This human ability relies on constantly knowing one's own rotational direction and angular velocity using the semicircular canals and visual information. Therefore, by incorporating a mechanism for observing one's own posture and position into Wearable SELD, this human ability can be imitated.

[0014] 1 and 2 , this embodiment discloses an acoustic event localization device 1 including an acoustic signal acquisition unit 101 that acquires acoustic signals from a multi-channel microphone array worn on the body, a sensor signal acquisition unit 102 that acquires sensor signals from sensors (e.g., a 6-axis acceleration sensor or a 9-axis geomagnetic / acceleration sensor) that track the body to mimic the human ability to perceive its own movement, and an SELD unit 501 that performs acoustic event localization based on a deep learning model that receives as input a combination of the acoustic signal and the sensor signal. The architecture of this system can be, for example, a DNN-based architecture such as that used in Non-Patent Document 1.

[0015] To tightly couple the analysis of acoustic signals and tracking information, it is necessary to consider their interrelationship during feature extraction. However, the appropriate feature extraction methods differ between acoustic signals and sensor signals. Therefore, it is desirable to have separate DNN architectures. An example of an architecture that meets this requirement is the architecture proposed to extract mutual features between audio and video, as shown in equations (1)-(5) in Reference Non-Patent Document 1.

[0016] (Reference Non-Patent Document 1: HRV Joze et. al., "MMTM: Multimodal Transfer Module for CNN Fusion", CVPR2020) <Formula when using the architecture of Reference Non-Patent Document 1> Number of frames obtained in the nth layer of DNN T Ａ , number of dimensions D Ａ , number of channels C Ａ The acoustic features of A ｎ ∈R ＴＡ×ＤＡ×ＣＡ (However, the superscript displayed after R should be T. Ａ ×D Ａ ×C Ａ ). Number of frames T Ｂ , number of dimensions D Ｂ , number of channels C Ｂ The sensor feature quantity of B ｎ ∈R ＴＢ×ＤＢ×ＣＢ (However, the superscript displayed after R should be T. Ｂ ×D Ｂ ×C Ｂ ). Furthermore, A ｎ , B ｎ The dimensions excluding the channel dimension are compressed by average pooling and expressed as S Ａ ∈R ＣＡ , S Ｂ ∈R ＣＢ (However, the superscripts displayed after R should be written as C. Ａ , C Ｂ ).

[0017] From these, the joint embedding representation Z is obtained according to the following formula: ｎ get. Here, [·,·] represents the combination of two vectors, and Wｎ is the linear transformation matrix, b ｎ is the bias term. ｎ is further separated according to the dimension of each modality (acoustic signal, sensor) as follows: Here, W Ａｎ , W Ｂｎ is the linear transformation matrix, b Ａｎ , b Ｂｎ is a bias term (the subscripts after W and b are actually A and B, respectively). ｎ , B ｎ ). The obtained E Ａｎ , E Ｂｎ Using the above, the original feature A is calculated as follows: ｎ , B ｎ (The subscripts after E are actually A and B. ｎ , B ｎ ). The acoustic features after weighting (hereinafter referred to as weighted acoustic features) are denoted as A ～ ｎ The weighted sensor feature (hereinafter referred to as the weighted sensor feature) is expressed as B ～ ｎ (where, ～ (The correct notation is that A and B should be displayed directly above A and B.) This makes it possible to incorporate the correlation between acoustic signals and sensor signals at the feature extraction stage.

[0018] The functional configuration of the acoustic event localization device of this embodiment will be described below with reference to Fig. 3. As shown in the figure, the acoustic event localization device 1 of this embodiment includes an audio signal acquisition unit 101, a sensor signal acquisition unit 102, an acoustic feature extraction unit 201, a sensor feature extraction unit 202, an inter-modality information exchange unit 203, a feature combination unit 204, an event detection unit 205, a sound source localization unit 206, a cost function calculation unit 401, an event detection label storage unit 301, and a sound source localization label storage unit 302. The SELD unit 501 described above corresponds to the acoustic feature extraction unit 201, the sensor feature extraction unit 202, the inter-modality information exchange unit 203, the feature combination unit 204, the event detection unit 205, the sound source localization unit 206, the cost function calculation unit 401, the event detection label storage unit 301, and the sound source localization label storage unit 302. In this embodiment, an earphone 9 incorporating a four-channel microphone 91 and a six-axis sensor 92 as shown in FIG. 4 is used. The four-channel microphone array 91 is used to observe acoustic signals. The six-axis sensor 92 is used to acquire sensor signals indicating tracking information obtained by tracking the body. While FIG. 4 shows an example of an earphone, this is not limiting. For example, a multi-channel microphone and a body tracking sensor can be incorporated into various body-worn devices, such as headphones or neck-worn devices. The multi-channel microphone is not limited to the four-channel microphone shown as an example, but can be any microphone with multiple channels, and the sensor is not limited to the six-axis sensor, but can have any number of axes.

[0019] Hereinafter, the event detection operation of each component of the sound event localization device 1 will be described with reference to FIG.

[0020] <Event Detection Operation> <Acoustic Signal Acquisition Unit 101> The acoustic signal acquisition unit 101 acquires an acoustic signal from the microphone array 91 (S101).

[0021] <Sensor Signal Acquisition Unit 102> The sensor signal acquisition unit 102 acquires a sensor signal from the sensor 92 (S102).

[0022] <Acoustic Feature Extraction Unit 201> The acoustic feature extraction unit 201 extracts acoustic features from an acoustic signal based on a deep learning model (S201). A DNN can be used as the deep learning model, and VGGish, for example, can be used as the DNN architecture.

[0023] <Sensor feature extraction unit 202> The sensor feature extraction unit 202 extracts sensor features from the sensor signal based on a deep learning model (S202). A DNN can be used as the deep learning model, and a Transformer, for example, can be used as the DNN architecture.

[0024] <Inter-modality information exchange unit 203> The inter-modality information exchange unit 203 exchanges acoustic features A ｎ and sensor feature B ｎ Based on the above equations (1)-(5), the joint embedding representation Z ｎ and weighted acoustic feature A, which is weighted based on the correlation between modalities. ～ ｎ and weighted sensor feature B ～ ｎ is obtained (S203).

[0025] <Feature Quantity Combining Unit 204> The feature quantity combining unit 204 uses the acoustic feature quantity extracted in step S201 and the sensor feature quantity extracted in step S202 to generate a weighted acoustic feature quantity A ～ ｎ and weighted sensor feature B ～ ｎ are combined in the channel dimension direction (S204).

[0026] <Event detection unit 205> The event detection unit 205 detects an event from the combined features based on the deep learning model (S205). For this, for example, an architecture with a linear layer and a Softmax activation function can be used.

[0027] <Sound source localization unit 206> The sound source localization unit 206 performs sound source localization from the combined features based on the deep learning model (S206). For this, for example, an architecture with a linear layer and a tanh activation function can be used.

[0028] <<Model Learning Operation>> Below, the model learning operation of each component of the sound event localization device 1 will be described with reference to Fig. 6. Note that the components and arrows represented by dashed dotted lines in Fig. 3 represent functional configurations that operate only during model learning operation, and transmission and reception of information that occurs only during model learning operation. Note that the sound signals and sensor signals acquired during model learning correspond to vectors in the training data {vector, label}. Note that the labels corresponding to the sound signals and sensor signals that correspond to the vectors are stored in the event detection label storage unit 301 and the sound source localization label storage unit 302, respectively.

[0029] <Cost function calculation unit 401> The cost function calculation unit 401 calculates a cost function using the event detected in step S205, the sound source localization in step S206, the label of the corresponding event stored in the event detection label storage unit 301, and the label of the corresponding sound source localization stored in the sound source localization label storage unit 302, and updates parameters of the deep learning models used in the acoustic feature extraction unit 201, the sensor feature extraction unit 202, the inter-modality information exchange unit 203, the feature combination unit 204, the event detection unit 205, and the sound source localization unit 206 based on the calculated cost function.

[0030] According to the acoustic event localization device 1 of this embodiment, the accuracy of the Wearable SELD is not degraded even when the wearer moves or rotates, which allows the Wearable SELD to operate robustly, so that the Wearable SELD can be used to inform hearing-impaired users and users who cannot hear surrounding sounds while listening to music of their surroundings, providing a safe environment.

[0031] <Additional Notes> The device disclosed herein may, for example, be a single hardware entity that includes an input unit to which a keyboard or the like can be connected, an output unit to which an LCD display or the like can be connected, a communication unit to which a communication device (e.g., a communication cable) capable of communicating with an external device of the hardware entity can be connected, a CPU (which may also include a central processing unit, cache memory, registers, etc.), memories such as RAM and ROM, an external storage device such as a hard disk, and buses connecting these input unit, output unit, communication unit, CPU, RAM, ROM, and external storage device so as to enable data exchange. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing recording media such as a CD-ROM. An example of a physical entity equipped with such hardware resources is a general-purpose computer.

[0032] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required for processing these programs (the programs may be stored in a ROM, which is a read-only storage device, for example, instead of an external storage device). Data obtained by processing these programs is stored in RAM, the external storage device, etc. as appropriate.

[0033] In a hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate, resulting in the CPU realizing a predetermined function (each component represented as a unit, means, etc.).

[0034] The present disclosure is not limited to the above-described embodiments, and modifications can be made as appropriate without departing from the spirit of the present disclosure. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capabilities of the devices that execute the processes or as needed.

[0035] As described above, when the processing functions of the hardware entities (apparatuses of the present disclosure) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program, and the processing functions of the hardware entities are realized on the computer by executing this program.

[0036] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 10020 of the computer 10000 shown in Figure 7, and operating the control unit 10010, input unit 10030, output unit 10040, etc.

[0037] The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories. Specifically, examples of magnetic recording devices include hard disk drives, flexible disks, and magnetic tapes; optical disks include DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable / Rewritable) / RWs; magneto-optical recording media include MOs (Magneto-Optical Discs), and semiconductor memories include EEP-ROMs (Electrically Erasable and Programmable-Read Only Memory).

[0038] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0039] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0040] In addition, in this embodiment, a hardware entity is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. An acoustic event localization device comprising: an acoustic signal acquisition unit that acquires acoustic signals from a multi-channel microphone array attached to the body; a sensor signal acquisition unit that acquires sensor signals from a sensor that tracks the body; and a SELD unit that performs acoustic event localization based on a deep learning model that inputs a combination of the acoustic signal and the sensor signal.

2. An acoustic event localization device as described in claim 1, wherein the SELD unit includes: an acoustic feature extraction unit that extracts acoustic features from the acoustic signal based on a deep learning model; a sensor feature extraction unit that extracts sensor features from the sensor signal based on a deep learning model; an inter-modality information exchange unit that acquires weighted acoustic features and weighted sensor features obtained by performing weighting on the acoustic features and the sensor features based on a joint embedding representation and a mutual relationship between modalities; a feature combination unit that uses the acoustic features and the sensor features to combine the weighted acoustic features and the weighted sensor features in the channel dimension direction; an event detection unit that detects an event from the combined features based on the deep learning model; and a sound source localization unit that performs sound source localization from the combined features based on the deep learning model.

3. An acoustic event localization apparatus according to claim 2, wherein the multi-channel microphone array and the sensor are both built into a body-worn device.

4. An acoustic event localization method executed by an acoustic event localization device, comprising: acquiring an acoustic signal from a multi-channel microphone array attached to the body; acquiring a sensor signal from a sensor that tracks the body; and performing acoustic event localization based on a deep learning model that uses as input a combination of the acoustic signal and the sensor signal.

5. A program for causing a computer to function as the acoustic event localization device according to any one of claims 1 to 3.