Sound event localization device and program

Abstract

This sound event localization device comprises: an on-vehicle distributed sensor signal acquisition unit that acquires a sensor signal, which is an observation signal from a sensor group, which is a plurality of sensors disposed so as to be distributed inside or outside a vehicle; a feature-amount extraction unit that extracts a feature amount from the sensor signal on the basis of a deep learning model; an event detection unit that detects an event from the feature amount on the basis of the deep learning model; and a sound source localization unit that localizes the sound source from the feature amount on the basis of the deep learning model.

Description

Acoustic event localization device and program

[0001] The present disclosure relates to a sound event localization device and program that executes sound event localization and detection (SELD).

[0002] This disclosure focuses on Sound Event Localization and Detection (SELD), a technology that identifies the type, location, and time of an event occurring in the surrounding environment solely from acoustic signals obtained from a microphone array. This disclosure particularly focuses on SELD that uses sensors attached inside and outside a vehicle.

[0003] Many conventional SELD systems use first-order Ambisonics (FOA) signals as input and a deep neural network (DNN) as a regression function for direction of arrival (DOA) and a classification function for SED to perform SELD inference. In recent years, SELD has become more complex, approaching the real world. State-of-the-art models have successfully performed SELD under conditions involving multiple moving sound sources and obstructions passing through the microphone. This technology is useful for applications that analyze and utilize events occurring indoors using fixed devices, such as smart speakers. However, due to the nature of Ambisonics microphones, installing them outside a vehicle is physically difficult, and there are no precedents for this technology.

[0004] Meanwhile, there is a conventional technology relating to a system that uses an in-vehicle microphone to detect an emergency vehicle and issue a warning to the driver (Non-Patent Document 1).

[0005] Shinichiro Otsuka, Hironori Hara, Shinji Ozawa, "Emergency Vehicle Presence and Direction Detection System Using In-Vehicle Microphones," IEEJ Trans. IA, Vol. 124, No. 4, 2004

[0006] The system in Non-Patent Document 1 performs detection using the fact that emergency vehicles emit specific frequencies, so it is a system specialized for emergency vehicles and cannot handle other sounds. Furthermore, the system in Non-Patent Document 1 also uses a rule-based algorithm for estimating the direction of sound, so it cannot handle environments outside the rules.

[0007] Therefore, an object of the present disclosure is to provide a sound event localization device that can be mounted on a vehicle and operates robustly in a real environment.

[0008] The acoustic event localization device of the present disclosure includes an on-board distributed sensor signal acquisition unit, a feature extraction unit, an event detection unit, and a sound source localization unit.

[0009] The on-board distributed sensor signal acquisition unit acquires sensor signals, which are observed signals from a sensor group consisting of multiple sensors distributed inside or outside the vehicle. The feature extraction unit extracts features from the sensor signals based on a deep learning model. The event detection unit detects events from the features based on the deep learning model. The sound source localization unit performs sound source localization from the features based on the deep learning model.

[0010] The acoustic event localization device of the present disclosure can be mounted on a vehicle and operates robustly in a real environment.

[0011] FIG. 1 is a diagram showing an example of the arrangement of a sensor group in a vehicle according to a first embodiment. FIG. 2 is a block diagram showing the functional configuration of a sound event localization device according to a first embodiment. FIG. 3 is a flowchart showing a model learning operation of the sound event localization device according to a first embodiment. FIG. 4 is a flowchart showing an event detection operation of the sound event localization device according to a first embodiment. FIG. 5 is a block diagram showing the functional configuration of a sound event localization device according to a second embodiment. FIG. 6 is a flowchart showing the operation of the sound event localization device according to a third embodiment. FIG. 7 is a block diagram showing the functional configuration of a sound event localization device according to a first modified example. FIG. 8 is a diagram showing an example of the functional configuration of a computer.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail. Note that components having the same functions are assigned the same numbers, and redundant explanations will be omitted.

[0013] The following describes a sound event localization device according to a first embodiment, which combines a sensor group distributed at multiple locations inside or outside a vehicle with a detection and localization system based on deep learning. As shown in Fig. 1, a sensor group 9 consisting of multiple sensors is distributed and located inside or outside the vehicle (or both inside and outside the vehicle). By using observed sensor signals as input and performing sound event localization using an algorithm based on deep learning, accuracy can be improved without introducing additional rules, provided that data can be collected within the expected driving range.

[0014] Although microphones are typically considered as sensors constituting the sensor group 9, cameras may also be used as other sensors. The sensor group 9 may also include both cameras and microphones. The sensor group 9 may also include any sensor (e.g., an acceleration sensor) as an auxiliary sensor for the microphone or camera. Sound event localization is generally thought to be performed using microphone signals, but in reality, sound event localization is possible using only video from a camera. For example, if a moving vehicle is captured in the video, it can be determined that a sound event related to this moving vehicle has occurred at a location identified by the video.

[0015] By using the technique of Non-Patent Document 1, it is possible to combine video information from a camera with audio information from a microphone. In addition, by using acceleration information from an acceleration sensor, it is possible to use information on the direction and speed of the vehicle, thereby reducing the influence of the Doppler effect and the movement of the sound source.

[0016] (Reference Non-Patent Document 1: GitHub, Inc., "DCASE2023 Sound Event Localization and Detection Track B Audio-Visual Inference", [online], [searched June 1, 2023], Internet <URL: https: / / github.com / sony / audio-visual-seld-dcase2023>) The functional configuration of the sound event localization device of this embodiment will be described below with reference to Figure 2. As shown in the figure, the sound event localization device 1 of this embodiment includes an on-board distributed sensor signal acquisition unit 101, a feature extraction unit 201, an event detection unit 202, a sound source localization unit 203, a cost function calculation unit 401, an event detection label storage unit 301, and a sound source localization label storage unit 302.

[0017] Below, we will explain the model learning operation of each component of the sound event localization device 1 with reference to Fig. 3. Note that the components and arrows shown with dashed dotted lines in Fig. 2 represent functional configurations that operate only during model learning operation, and transmission and reception of information that occurs only during model learning operation.

[0018] <Model Learning Operation> <On-board Distributed Sensor Signal Acquisition Unit 101> The on-board distributed sensor signal acquisition unit 101 acquires observed signals (hereinafter also referred to as sensor signals) from the sensor group 9 that correspond to vectors in the training data {vector, label} (S101). Note that the labels corresponding to the sensor signals that correspond to the vectors are assumed to be stored in the event detection label storage unit 301 and the sound source localization label storage unit 302, respectively.

[0019] <Feature Extraction Unit 201> The feature extraction unit 201 extracts features from sensor signals based on a deep learning model (DNN, etc.) (S201). If the sensor group 9 is a microphone, the feature extraction unit 201 extracts sound features, and if the sensor group 9 is a camera, the feature extraction unit 201 extracts video features. If the sensor group 9 is a combination of a microphone and a camera, the feature extraction unit 201 extracts sound and video features. If the sensor group 9 includes an acceleration sensor, the feature extraction unit 201 extracts acceleration features in addition to sound features or video features.

[0020] It should be noted that if learning is performed for the first time, the parameters of the deep learning model are set to predetermined initial values.

[0021] <Event detection unit 202> The event detection unit 202 detects an event from a feature amount based on a deep learning model (S202).

[0022] <Sound source localization unit 203> The sound source localization unit 203 performs sound source localization from the feature amount based on the deep learning model (S203).

[0023] <Cost function calculation unit 401> The cost function calculation unit 401 calculates a cost function using the event detected in step S202, the sound source localization in step S203, the label of the corresponding event stored in the event detection label storage unit 301, and the label of the corresponding sound source localization stored in the sound source localization label storage unit 302, and updates parameters of the deep learning models used in the feature extraction unit 201, the event detection unit 202, and the sound source localization unit 203 based on the calculated cost function.

[0024] Hereinafter, the event detection operation of each component of the sound event localization device 1 will be described with reference to FIG.

[0025] <Event Detection Operation> <In-Vehicle Distributed Sensor Signal Acquisition Unit 101> The in-vehicle distributed sensor signal acquisition unit 101 acquires a sensor signal (S101).

[0026] <Feature Extraction Unit 201> The feature extraction unit 201 extracts features from a sensor signal based on a deep learning model (S201).

[0027] <Event detection unit 202> The event detection unit 202 detects an event from a feature amount based on a deep learning model (S202).

[0028] <Sound source localization unit 203> The sound source localization unit 203 performs sound source localization from the feature amount based on the deep learning model (S203).

[0029] 5, the functional configuration of a sound event localization device according to a second embodiment will be described. The sound event localization device according to the second embodiment includes, in addition to the functions of the first embodiment, a function for reproducing an event sound inside the vehicle based on the type and direction of the event by using a detected event and sound source localization. As shown in the figure, the sound event localization device 2 according to the second embodiment includes an on-board distributed sensor signal acquisition unit 101, a feature extraction unit 201, an event detection unit 202, a sound source localization unit 203, a cost function calculation unit 401, an event detection label storage unit 301, a sound source localization label storage unit 302, a transmitted event sound calculation unit 501, and a transmitted event sound playback unit 502.

[0030] Hereinafter, with reference to FIG. 6, the operation of the constituent elements of the sound event localization device 2 that differ from those in the first embodiment will be described.

[0031] <Transmission Event Sound Calculation Unit 501> The transmission event sound calculation unit 501 calculates the transmission event sound, which is the event sound that is transmitted into the vehicle interior, based on the event detected in step S202 and the sound source localization in step S203 (S501). For example, the transmission event sound may be calculated by referencing a sound of the same type as the detected event type from a database in the transmission event sound calculation unit 501 and panning the sound based on the sound source localization result.

[0032] <Transparent event sound reproducing unit 502> The transparent event sound reproducing unit 502 reproduces the transparent event sound inside the vehicle (S502). For example, it is preferable to use speakers installed on the left and right of the headrests to reproduce the sound so as to present the type of sound and its position to the occupants of each seat.

[0033] 7, the functional configuration of an acoustic event localization device of Example 3 that, in addition to the functions of Example 1, executes active noise control (ANC) that can transmit a specific event by using a detected event and sound source localization. As shown in the figure, the acoustic event localization device 3 of this example includes an on-board distributed sensor signal acquisition unit 101, a feature extraction unit 201, an event detection unit 202, a sound source localization unit 203, a cost function calculation unit 401, an event detection label storage unit 301, a sound source localization label storage unit 302, an active noise control microphone signal acquisition unit 601, an active noise control secondary sound source signal calculation unit 701, and an active noise control secondary sound source signal reproduction unit 702.

[0034] Hereinafter, the operation of the constituent elements of the sound event localization device 3 that differ from those in the first embodiment will be described with reference to FIG.

[0035] <Active noise control microphone signal acquisition unit 601> The active noise control microphone signal acquisition unit 601 acquires an active noise control microphone signal, which is a microphone signal to be used for active noise control (S601). The microphones to be used for active noise control include all or any of the reference microphone, error microphone, and other microphones. The aforementioned sensor group 9 can be a microphone or a combination of a microphone and a camera, and the acoustic signals from the microphones included in the sensor group 9 can be used as the active noise control microphone signal (dashed arrow in FIG. 7). When only the acoustic signals from the microphones included in the sensor group 9 are used as the active noise control microphone signal, the active noise control microphone signal acquisition unit 601 may be omitted, as shown as a modified example in FIG. 9.

[0036] <Active noise control secondary sound source signal calculation unit 701> The active noise control secondary sound source signal calculation unit 701 calculates a secondary sound source signal that cancels noise but does not cancel specific sounds or sounds from specific directions (S701). The active noise control secondary sound source signal calculation unit 701 generates a secondary sound source that cancels noise but does not cancel specific sounds such as emergency vehicles or sounds from specific directions. To achieve this, the active noise control secondary sound source signal calculation unit 701 additionally uses all or any of the outputs of steps S202, S203, and S101. For example, it is preferable that the active noise control secondary sound source signal calculation unit 701 calculates the secondary sound source signal based on Filtered-X or DNN.

[0037] <Active noise control secondary sound source signal reproducing unit 702> The active noise control secondary sound source signal reproducing unit 702 reproduces the active noise control secondary sound source signal calculated in step S701 inside the vehicle (S702).

[0038] <Effects of Sound Event Localization Apparatus 1-3 of Example 1-3> According to the sound event localization apparatus 1 of Example 1, event detection and sound source localization that can be mounted on a vehicle and operates robustly in a real environment are realized.

[0039] The acoustic event localization device 2 of the second embodiment can grasp the location and type of events occurring around the vehicle while the vehicle is traveling, and can present necessary information in an emphasized manner inside the vehicle. This is particularly effective in vehicles with high soundproofing or in combination with active noise control.

[0040] According to the sound event localization device 3 of the third embodiment, by grasping the location and type of events occurring around the vehicle while the vehicle is traveling, it is possible to cancel out noise while presenting specific sounds without canceling them out, thereby increasing the comfort of the vehicle user and presenting only the necessary information inside the vehicle.

[0041] <Additional Notes> The device disclosed herein may, for example, be a single hardware entity that includes an input unit to which a keyboard or the like can be connected, an output unit to which an LCD display or the like can be connected, a communication unit to which a communication device (e.g., a communication cable) capable of communicating with an external device of the hardware entity can be connected, a CPU (which may also include a central processing unit, cache memory, registers, etc.), memories such as RAM and ROM, an external storage device such as a hard disk, and buses connecting these input unit, output unit, communication unit, CPU, RAM, ROM, and external storage device so as to enable data exchange. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing recording media such as a CD-ROM. An example of a physical entity equipped with such hardware resources is a general-purpose computer.

[0042] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required for processing these programs (the programs may be stored in a ROM, which is a read-only storage device, for example, instead of an external storage device). Data obtained by processing these programs is stored in RAM, the external storage device, etc. as appropriate.

[0043] In a hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate, resulting in the CPU realizing a predetermined function (each component represented as a unit, means, etc.).

[0044] The present disclosure is not limited to the above-described embodiments, and modifications can be made as appropriate without departing from the spirit of the present disclosure. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capabilities of the devices that execute the processes or as needed.

[0045] As described above, when the processing functions of the hardware entities (apparatuses of the present disclosure) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program, and the processing functions of the hardware entities are realized on the computer by executing this program.

[0046] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 10020 of the computer 10000 shown in Figure 10 and operating the control unit 10010, input unit 10030, output unit 10040, etc.

[0047] The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories. Specifically, examples of magnetic recording devices include hard disk drives, flexible disks, and magnetic tapes; optical disks include DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable / Rewritable) / RWs; magneto-optical recording media include MOs (Magneto-Optical Discs), and semiconductor memories include EEP-ROMs (Electrically Erasable and Programmable-Read Only Memory).

[0048] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0049] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0050] In addition, in this embodiment, a hardware entity is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. An acoustic event localization device including: an on-board distributed sensor signal acquisition unit that acquires sensor signals, which are observation signals from a sensor group that is a plurality of sensors distributed inside or outside a vehicle; a feature extraction unit that extracts features from the sensor signals based on a deep learning model; an event detection unit that detects an event from the features based on the deep learning model; and a sound source localization unit that performs sound source localization from the features based on the deep learning model.

2. An acoustic event localization device as claimed in claim 1, wherein the group of sensors is a microphone, or a camera, or a combination of a microphone and a camera, and the feature extraction unit extracts acoustic features, or video features, or audio and video features.

3. An acoustic event localization device according to claim 2, wherein the group of sensors is a combination of the microphones and the cameras and any other sensors.

4. An acoustic event localization device according to claim 1, comprising: a transmitted event sound calculation unit that calculates a transmitted event sound, which is an event sound that transmits into the vehicle interior based on the event and the sound source localization; and a transmitted event sound reproduction unit that reproduces the transmitted event sound.

5. An acoustic event localization device as claimed in claim 1, comprising: an active noise control microphone signal acquisition unit that acquires an active noise control microphone signal, which is a microphone signal to be used for active noise control; a secondary sound source signal calculation unit for active noise control that calculates a secondary sound source signal that cancels out noise and does not cancel out a specific sound or sound from a specific direction; and a secondary sound source signal reproduction unit for active noise control that reproduces the active noise control secondary sound source signal.

6. An acoustic event localization device according to claim 5, wherein the group of sensors is a microphone or a combination of a microphone and a camera, and acoustic signals from these microphones are used as the microphone signals for active noise control.

7. An acoustic event localization device according to claim 5, wherein the secondary sound source signal calculation unit for active noise control calculates a secondary sound source signal that cancels noise based on Filtered-X or DNN and does not cancel a specific sound or a sound from a specific direction.

8. A program for causing a computer to function as the acoustic event localization device according to any one of claims 1 to 7.

Images