Model learning device, model learning method, and program

Abstract

The present invention includes: a reconstruction cost processing unit that calculates a cost function of a spectrogram of echo sound data and a corresponding reconstructed spectrogram of the echo sound data, and updates an acoustic feature amount of the echo sound data and the reconstructed spectrogram; an acoustic feature amount combining unit that combines an acoustic feature amount of input batch data and the acoustic feature amount of the echo sound data, and updates the acoustic feature amount of the input batch data; a domain estimated cost processing unit that determines whether each sample configuring the input batch data is echoic data or anechoic data, calculates a cost function on the basis of a corresponding label, and updates the determination result; an acoustic event detection / orientation cost processing unit that estimates the type of acoustic event and the occurrence time on the basis of the combined acoustic feature amount, or estimates the direction of the acoustic event, calculates the cost function on the basis of the corresponding label, and updates the estimation result; and a cost calculation unit that calculates a cost function to be used in updating, as a weighted sum of calculation results of the cost function.

Description

[Technical Field]

[0001] The present invention relates to a model learning device, a model learning method, and a program for learning a model used for acoustic event detection or localization. [Background technology]

[0002] There are many different sounds around us, including human voices, the sounds of machines operating, animal cries, etc. Systems that can detect these sounds can be used in a variety of applications, such as autonomous driving that detects invisible dangers, crime detection in the dark, and pedestrian safety assistance.

[0003] An important technology for such applications is Sound Event Localization and Detection (SELD), which combines Direction of Arrival (DOA) estimation and Sound Event Detection (SED).

[0004] In recent SELD competitions, algorithms using deep learning (Deep Neural Networks: DNNs) for the DOA regression function and the SED classification function have achieved high performance. Training such DNNs requires a large amount of labeled data containing various types of sound events occurring in various directions around the microphone.

[0005] In the SELD problem configurations considered so far, systems have been trained using SELD datasets recorded in up to 11 environments, and performance has been evaluated using data recorded in the same environments.

[0006] On the other hand, in real applications, users will use the SELD system in any environment, so the system must be able to operate robustly even in environments not included in the training data (referred to as unknown environments in this specification).The easiest way to train a SELD system that can handle any environment is to record a complete dataset in many environments and use it for training, as in conventional SELD methods.

[0007] However, recording and labeling 3D sound data is extremely costly, and since sound directional information is affected by countless factors, such as the layout of objects in a room, the distance from the wall to the microphone, and building materials, it is not realistic to cover all possible combinations of these.

[0008] Non-Patent Document 1 studies a domain adaptation method for performing acoustic event detection or sound source direction estimation in an inference environment (target domain) that is different from an environment (source domain) where a sufficient amount of labeled training data can be acquired. In SED and acoustic scene classification, a domain adaptation method based on Domain Adversarial Training (DAT) has been proposed to bridge the gap between the training data and the target domain (inference environment).

[0009] In domain adaptation using DAT, a classifier is introduced to distinguish between features obtained from the source domain and the target domain. By training the feature extractor in an adversarial manner to deceive the classifier, it is expected that domain-independent features will be extracted. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] W. Wei et al., “A-crnn: A domain adaptation model for sound event detection,” in Proc. of IEEE Int. Conf. on Acoustics, Speech and Signal Process. (ICASSP), 2020. Summary of the Invention [Problem to be solved by the invention]

[0011] The method of Non-Patent Document 1 does not require labels of data in the target domain or requires only weak labels, but cannot be applied when the target domain is a completely unknown environment.

[0012] Therefore, an object of the present invention is to provide a model learning device that can learn a model that can operate robustly even in an unknown environment that is not included in the learning data. [Means for solving the problem]

[0013] The model learning device of the present invention includes a reconstruction cost processing unit, an acoustic feature combination unit, a domain estimation cost processing unit, an acoustic event detection and localization cost processing unit, and a cost calculation unit.

[0014] Data recorded in a reverberant environment is called reverberant data, data recorded in an anechoic chamber is called anechoic data, data combining reverberant and anechoic data is called input batch data, and data of reverberant sound in the same environment as the environment in which the reverberant data was obtained is called reverberant sound data.

[0015] The reconstruction cost processing unit calculates a cost function designed to bring the spectrogram of the reverberation sound data and the reconstructed spectrogram of the corresponding reverberation sound data closer together, and repeatedly executes a process of updating the acoustic features of the reverberation sound data and the reconstructed spectrogram of the corresponding reverberation sound data.

[0016] The acoustic feature combining unit combines the acoustic features of the input batch data with the acoustic features of the echo sound data, and repeatedly executes a process of updating the acoustic features of the input batch data.

[0017] The domain estimation cost processing unit repeatedly executes a process of determining whether each sample constituting the input batch data is echoed data or anechoic data, calculating a cost function based on the corresponding label, and updating the determination result.

[0018] The acoustic event detection and localization cost processor estimates the type and occurrence time of an acoustic event based on the combined acoustic features, or estimates its direction, calculates a cost function based on the corresponding label, and repeatedly updates the estimation results.

[0019] The cost calculation unit calculates a cost function to be used for updating as a weighted sum of the calculation results of the cost functions. [Effects of the Invention]

[0020] According to the model learning device of the present invention, it is possible to learn a model that can robustly operate even in an unknown environment that is not included in the learning data. [Brief explanation of the drawings]

[0021] [Figure 1] A diagram showing the two-stage inference procedure using EAR. [Figure 2] FIG. 1 is a block diagram showing the functional configuration of a model learning device according to a first embodiment. [Figure 3] 3 is a flowchart showing the operation of the model learning device according to the first embodiment. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of a reconstruction cost processing unit. [Figure 5] 10 is a flowchart showing the operation of a reconstruction cost processing unit. [Figure 6] FIG. 4 is a block diagram showing the functional configuration of an acoustic feature combination unit. [Figure 7] 10 is a flowchart showing the operation of an acoustic feature combination unit. [Figure 8] FIG. 2 is a block diagram showing the functional configuration of a domain estimation cost processing unit. [Figure 9] 10 is a flowchart showing the operation of a domain estimation cost processing unit. [Figure 10] FIG. 2 is a block diagram showing the functional configuration of an acoustic event detection and localization cost processing unit. [Figure 11] 10 is a first flowchart showing the operation of the acoustic event detection and localization cost processing unit. [Figure 12] 10 is a second flowchart showing the operation of the acoustic event detection and localization cost processing unit. [Figure 13] FIG. 2 is a diagram showing an example of the functional configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present invention will be described in detail. Components having the same functions are given the same numbers, and duplicated explanations will be omitted.

[0023] <Notation> The notation used in this specification is defined below. N known environments are denoted by {e1,...,e N}. The unknown environment ε* / ⊂ε (where / ⊂ is the negation of the set) is defined as an environment not included in ε. The signal observed in the known environment is defined as x ε , the signal observed in the unknown environment is x ε* Let the training data X ε ={x1,...,x K} to SELD's correct answer label Y ε ={y1,...,y K Let M denote the SELD estimation function that estimates the SED and DOA labels, where y contains both the SED and DOA labels, and y = {y SED ,y DOA}.

[0024] Furthermore, as a standard architecture, M is assumed to consist of a feature extractor F, an SED classifier C, and a DOA regression function D.

[0025] <Principle> The performance of a SELD system trained on data from a specific known environment degrades in unknown environments. This problem is called domain shift, and feature extractors trained in known environments do not work properly in unknown environments due to environmental influences such as noise and reverberation, causing feature statistics to shift and resulting in performance degradation.

[0026] To solve this problem, we use a technique called echo-aware feature refinement (EAR) that corrects features taking into account reverberation (see Figure 1).

[0027] The first step is to observe the echo in an unknown environment. The sound source for the echo observation is assumed to be the system's own sound, such as the system startup sound. The observed echo h ε* is a real vector z ε* Since the reverberation is expected to contain spatial cues about the surrounding environment, information about the unknown environment is embedded in z ε* It is expected to be embedded in

[0028] The second step is this z ε* First, we execute SELD using the feature extractor f ε* is the observed signal x containing acoustic events occurring in the surroundings. ε* From the feature value f ε* Here, to suppress environmental effects such as noise and reverberation, z ε* It utilizes spatial clues about the unknown environment embedded in the ε* Using f ε* The correction of f is given by the following formula (1): ε* and z ε* and correcting it with a correction function R.

[0029] f' ε* =R(f ε* ,z ε* )…(1) Finally, the obtained f' ε*Using this, SELD is performed using the SED classifier C and the DOA regression function D.

[0030] To perform this two-stage inference, it is necessary to train the feature extractor F, the reverberation encoder G, and the correction function R using only known environment data. For this training, we first train the reverberation encoder h observed in multiple known environments. ε and the acoustic event observation signal x ε Furthermore, similar pair data (h ε ,x ε ) to collect.

[0031] To train the system using this training data, we adopt a domain adaptation learning method based on DAT. However, since there is no data for the unknown environment ε*, which is the true target domain, the anechoic chamber is used as the source domain and other environments are used as the target domain. In the training, paired data (h ε ,x ε ) and calculate the corrected feature f' based on Eq. (1). ε The domain classifier H, which is used only during training, can be obtained as f' ε The inputs are classified into reverberant and anechoic sounds. Using the Gradient Reversal Layer (GRL) proposed in existing research (see Non-Patent Document 1), the reverberant sound encoder G, feature extractor F, and correction function R are adversarially trained to degrade the performance of domain classification by the domain classifier H, thereby obtaining a feature representation that does not distinguish between reverberant and anechoic sounds, i.e., an environment-invariant representation.

[0032] (Reference non-patent document 1: Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in Proc. of the 32nd Int. Conf. on Machine Learn. (ICML), 2015.) In particular, z ε The correction function R conditioned on z εThis has the effect of suppressing the influence of the environment at the feature level based on spatial cues of the surrounding environment (e.g., object placement) embedded in the image. [Example]

[0033] The functional configuration of the model learning device 1 of the first embodiment will be described below with reference to Fig. 2. As shown in Fig. 2, the model learning device 1 of the first embodiment includes an anechoic data storage unit 10A, an anechoic data storage unit 10B, an input batch data storage unit 10C, an echo data storage unit 10D, a reconstruction cost processing unit 11, an acoustic feature combination unit 12, a domain estimation cost processing unit 13, an acoustic event detection and localization cost processing unit 14, and a cost calculation unit 15. The operation of each component is shown in Fig. 3.

[0034] ≪Echoic data storage section 10A≫ The reverberant data storage unit 10A stores reverberant data. The reverberant data is data obtained by recording acoustic events in a reverberant environment. In this embodiment, the data was generated by convolving dry sources including 11 types of sound source classes with impulse responses collected in 96 environments.

[0035] <Anechoic data storage unit 10B> The anechoic data storage unit 10B stores anechoic data. The anechoic data is data obtained by recording acoustic events in an anechoic chamber. In this embodiment, the data is generated by convolving a dry source including 11 types of sound source classes with the impulse response collected in the anechoic chamber.

[0036] <Input batch data storage unit 10C> The input batch data storage unit 10C stores input batch data. The input batch data is a batch of echoic data and the same number of anechoic data. In this embodiment, 32 samples are prepared for each of the echoic and anechoic samples per iteration.

[0037] <<Echo sound data storage unit 10D>> The reverberation data storage unit 10D stores reverberation data. The reverberation data refers to data on reverberation in the same environment as the environment in which the reverberation data in the reverberation data storage unit 10A was acquired. In this embodiment, the data used was a recording of a swept-sine signal generated at an azimuth angle of 0°, an elevation angle of 0°, and a distance of 75 cm from the microphone.

[0038] <Reconstruction cost processing unit 11> The reconstruction cost processing unit 11 calculates a cost function designed to bring the spectrogram of the reverberation sound data and the reconstructed spectrogram of the corresponding reverberation sound data closer together, and repeatedly executes a process of updating the acoustic features of the reverberation sound data and the reconstructed spectrogram of the corresponding reverberation sound data (S11).

[0039] As shown in FIG. 4, the reconstruction cost processing unit 11 includes an echo spectrogram extraction unit 111, an echo encoder 112, an echo decoder 113, a reconstructed spectrogram storage unit 110A, and a reconstruction cost calculation unit 114.

[0040] The detailed operation of each component will be described with reference to FIGS.

[0041] <Reverberation spectrogram extraction unit 111> The reverberation spectrogram extraction unit 111 extracts a spectrogram from each sound sample of the reverberation data by Short-time Fourier transform (STFT) (S111).

[0042] <Reverberation Encoder 112> The reverberation encoder 112 extracts acoustic features from the spectrogram obtained in step S111 and obtains one real-valued embedding vector for each sample (S112). The reverberation encoder 112 corresponds to the reverberation encoder G described in the <Principle> section. In this embodiment, a multi-layer convolutional neural network (CNN) is used.

[0043] <Echo Decoder 113> The reverberation decoder 113 decodes the real-number embedding vector obtained in step S112 into a tensor of the same dimension as the spectrogram obtained in step S111 (S113). In this embodiment, a multi-layer convolutional neural network (CNN) is used.

[0044] <Reconstructed spectrogram storage unit 110A> Reconstructed spectrogram storage unit 110A stores the reconstructed spectrogram, which is the output of step S113 and is a tensor of the same dimensions as the output of step S111.

[0045] <Reconstruction cost calculation unit 114> The reconstruction cost calculation unit 114 calculates a cost function designed to bring the output of the reconstructed spectrogram storage unit 110A and the output of step S111 closer to each other (S114). In this embodiment, the mean square error (MSE) is used as the cost function.

[0046] <Acoustic feature combination unit 12> The acoustic feature combining unit 12 combines the acoustic feature of the input batch data with the acoustic feature of the echo sound data, and repeatedly executes the process of updating the acoustic feature of the input batch data (S12).

[0047] As shown in FIG. 6, the acoustic feature combination unit 12 includes a spectrogram extraction unit 121, an acoustic feature extractor 122, an echo embedding vector combination unit 123, and an acoustic feature correction unit .

[0048] The detailed operation of each component will be described with reference to FIGS.

[0049] <Spectrogram extraction unit 121> The spectrogram extraction unit 121 extracts a spectrogram from each audio sample of the input batch data by Short-time Fourier transform (STFT) (S121).

[0050] <Acoustic Feature Extractor 122> The acoustic feature extractor 122 extracts acoustic features from the spectrogram obtained in step S121 and obtains a time-series real-number embedding vector (S122). The acoustic feature extractor 122 corresponds to the acoustic feature extractor F described in <Principle>. In this example, a multi-layer convolutional neural network (CNN) is used.

[0051] <Echo embedding vector combining unit 123> The reverberation embedding vector combining unit 123 combines the reverberation embedding vector obtained in step S112 with each of the time-series embedding vectors of the acoustic features obtained in step S122 in a direction that increases the number of dimensions of the vector (S123).

[0052] <Acoustic feature correction unit 124> The acoustic feature correction unit 124 obtains corrected features from the vector obtained in step S123, which is a combination of the acoustic features and the embedding vector of the reverberation sound (S124). The acoustic feature correction unit 124 corrects the acoustic features using the correction function R described in <Principle>. The corrected features are time-series data with the same number of dimensions as the embedding vector of the acoustic features. In this example, a bidirectional LSTM is used.

[0053] <Domain estimation cost processing unit 13> The domain estimation cost processing unit 13 repeatedly executes the process of determining whether each sample constituting the input batch data is anechoic data or anechoic data, calculating a cost function based on the corresponding label, and updating the determination result (S13).

[0054] As shown in FIG. 8, the domain estimation cost processing unit 13 includes a gradient reversal unit 131, a domain classifier 132, a domain estimation cost calculation unit 133, and a domain estimation correct label storage unit 130A.

[0055] The detailed operation of each component will be described with reference to FIGS.

[0056] <Gradient reversal unit 131> The gradient inversion unit 131 inverts the sign of the error at this boundary when updating parameters based on the backpropagation algorithm using the cost function calculated in step S133 (S131). As a result, layers closer to the input than this are trained to increase the cost in step S133, and the reverberation encoder G, acoustic feature extractor F, and correction function R are trained adversarially to degrade the performance of domain classification by the domain classifier H.

[0057] <Domain Classifier 132> The domain classifier 132 determines whether each sample constituting the input batch data stored in the input batch data storage unit 10C is from the reverberant data storage unit 10A or the anechoic data storage unit 10B by binary classification (S132). The domain classifier 132 corresponds to the domain classifier H described in the <Principle> section. In this embodiment, it is implemented using a linear layer and a sigmoid activation function.

[0058] <Domain Estimation Cost Calculation Unit 133> The domain estimation cost calculation unit 133 calculates a cost function for the domain estimation task (S133) using the labels indicating whether each sample output from the reverberant data storage unit 10A and the anechoic data storage unit 10B is reverberant data or anechoic data, and the output of step S132. In this embodiment, binary cross entropy is used as the cost function.

[0059] <Domain estimation correct label storage unit 130A> The domain estimation correct label storage unit 130A stores a label indicating whether each sample output from the reverberant data storage unit 10A and the anechoic data storage unit 10B is reverberant data or anechoic data.

[0060] <Acoustic event detection and localization cost processing section 14> The acoustic event detection and localization cost processing unit 14 estimates the type and occurrence time of an acoustic event or its direction based on the combined acoustic features, calculates a cost function based on the corresponding label, and repeatedly executes the process of updating the estimation result (S14).

[0061] The sound event detection and localization cost processing unit 14 may perform both sound event detection and its cost calculation (S141-S142, see FIG. 11) and sound source direction estimation and its cost calculation (S143-S144, see FIG. 12), which will be described later, or may select and perform only one of them.

[0062] If only acoustic event detection and its cost calculation are selected and executed, the trained model will be a model that only performs acoustic event detection, and if only sound source direction estimation and its cost calculation are selected and executed, the trained model will be a model that only performs sound source direction estimation.If both are executed, the trained model will be a model that performs acoustic event detection and sound source direction estimation.

[0063] As shown in FIG. 10, the sound event detection and localization cost processing unit 14 includes a sound event detection unit 141, a sound event detection cost calculation unit 142, a sound event detection correct label storage unit 140A, a sound source direction estimation unit 143, a sound source direction estimation cost calculation unit 144, and a sound source direction estimation correct label storage unit 140B.

[0064] When sound event detection and its cost calculation processing are omitted, the sound event detection unit 141, sound event detection cost calculation unit 142, and sound event detection correct label storage unit 140A can be omitted.

[0065] When the sound source direction estimation and its cost calculation process are omitted, the sound source direction estimation unit 143, the sound source direction estimation cost calculation unit 144, and the sound source direction estimation correct label storage unit 140B can be omitted.

[0066] <Acoustic event detection unit 141> The sound event detection unit 141 estimates the type and occurrence time of the sound event based on the combined and corrected features obtained in step S124 (S141). In this example, this is implemented using a linear layer and a sigmoid activation function.

[0067] <Acoustic event detection cost calculation unit 142> The sound event detection cost calculation unit 142 calculates a cost function for the sound event detection task using the labels indicating the type and occurrence time of the sound event and the output of step S141. In this embodiment, binary cross entropy is used as the cost function.

[0068] <Acoustic event detection correct label storage unit 140A> The sound event detection correct label storage unit 140A stores labels indicating the type and occurrence time of a sound event.

[0069] <Sound source direction estimation unit 143> The sound source direction estimation unit 143 estimates the azimuth and elevation angles at which the acoustic event occurred based on the combined and corrected features obtained in step S124 (S143). In this embodiment, this is implemented using a linear layer and a hyperbolic tangent function.

[0070] <Sound source direction estimation cost calculation unit 144> The sound source direction estimation cost calculation unit 144 calculates a cost function of the sound source direction estimation task using the labels indicating the azimuth and elevation angles at which the acoustic event occurred and the output of step S143 (S144). In this embodiment, the mean square error (MSE) is used as the cost function.

[0071] <Sound source direction estimation correct label storage unit 140B> The sound source direction estimation correct label storage unit 140B stores labels indicating the azimuth and elevation angles at which sound events occur.

[0072] <Cost Calculation Section 15> The cost calculation unit 15 calculates a cost function used for parameter update as a weighted sum of the calculation results of the cost function (S15). In detail, the cost calculation unit 15 calculates a cost function for learning the entire system as a weighted sum of the outputs of steps S114, S133, S142, and S144. Based on the cost function calculated as a weighted sum, parameter update is repeatedly performed based on the backpropagation algorithm in each process (steps S112, S113, S122, S132, S141, and S143) that handles learning parameters of the entire system.

[0073] <SELD inference using trained systems> During inference, the anechoic data storage unit 10B, the reverberant sound encoder 112, the reconstructed spectrogram storage unit 110A, the gradient inversion unit 131, the domain classifier 132, and each cost function and label are not used. The input data to the input batch data storage unit 10C are acoustic signals acquired in real time or pre-recorded acoustic signals. The input data to the reverberant sound data storage unit 10D is reverberant sound data acquired at a certain time in the same environment as the input batch data storage unit 10C.

[0074] <Example of execution result> The execution results are shown in Table 1. Note that DE, FR, F, and ER represent DOE error, frame recall, F-measure, and error rate, respectively. Each indicator conforms to DCASE2019 Challenge task 3. [Table 1] Here, (A) is the case where data from the same environment was used during training and inference. (B) and (C) are cases where inference was performed in an environment different from that during training, but (B) did not use the adaptation method described in the examples, and (C) did use the adaptation method described in the examples. The results show that application of the present invention improves SELD performance in unknown environments not included in the training data.

[0075] <Surrounding environmental information> In the above-described embodiment, a model learning device is disclosed that can operate robustly even in unknown environments that are not included in the learning data by adapting the system by acquiring surrounding environmental information during inference and combining that information.

[0076] The surrounding environment information may be various information such as reverberation, noise, surrounding image information, GPS coordinates, etc., but the above embodiment has shown an example in which the surrounding environment information is acquired using reverberation in particular.

[0077] <Additional Notes> The device of the present invention may, for example, be a single hardware entity having an input unit to which a keyboard or the like can be connected, an output unit to which an LCD display or the like can be connected, a communication unit to which a communication device (e.g., a communication cable) capable of communicating with an external device can be connected, a CPU (which may also include a central processing unit, cache memory, registers, etc.), memories such as RAM and ROM, an external storage device such as a hard disk, and buses connecting these input unit, output unit, communication unit, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing to a recording medium such as a CD-ROM. A physical entity equipped with such hardware resources includes a general-purpose computer.

[0078] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required for processing these programs (the programs may be stored in a ROM, which is a read-only storage device, for example, instead of an external storage device). Data obtained by processing these programs is stored in RAM, the external storage device, etc. as appropriate.

[0079] In a hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate, resulting in the CPU realizing a predetermined function (each component represented as a unit, means, etc., above).

[0080] The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed.

[0081] As described above, when the processing functions of the hardware entities (apparatuses of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

[0082] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 10020 of the computer 10000 shown in Figure 13 and operating the control unit 10010, input unit 10030, output unit 10040, etc.

[0083] The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories. Specifically, examples of magnetic recording devices include hard disk drives, flexible disks, and magnetic tapes; optical disks include DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable) / RWs (Rewritable); magneto-optical recording media include MOs (Magneto-Optical discs), and semiconductor memories include EEP-ROMs (Electrically Erasable and Programmable-Read Only Memory).

[0084] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0085] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the received program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (such as data that is not a direct instruction to the computer but has properties that define computer processing).

[0086] In addition, in this embodiment, a hardware entity is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. Data obtained by recording an acoustic event in an environment with reverberation is called reverberant data, data obtained by recording an acoustic event in an anechoic chamber is called anechoic data, data obtained by combining the reverberant data and the anechoic data is called input batch data, and data obtained by reverberating sound in the same environment as the environment in which the reverberant data was obtained is called reverberant sound data. a reconstruction cost processing unit that calculates a cost function designed to bring the spectrogram of the reverberant sound data and the reconstructed spectrogram of the corresponding reverberant sound data closer to each other, and repeatedly executes a process of updating the acoustic feature of the reverberant sound data and the reconstructed spectrogram of the corresponding reverberant sound data; an acoustic feature combining unit that combines acoustic features of the input batch data and acoustic features of the reverberant sound data and repeatedly executes a process of updating the acoustic features of the input batch data; a domain estimation cost processing unit that repeatedly executes a process of determining whether each sample constituting the input batch data is echoed data or anechoic data, calculating a cost function based on the corresponding label, and updating the result of the determination; an acoustic event detection and localization cost processing unit that repeatedly executes a process of estimating the type and occurrence time of an acoustic event or its direction based on the combined acoustic features, calculating a cost function based on the corresponding label, and updating the estimation result; a cost calculation unit that calculates a cost function to be used for the update as a weighted sum of calculation results of the cost function; Model learning device.

2. 2. The model learning device according to claim 1, The reconstruction cost processing unit: a reverberation encoder G that extracts acoustic features from a spectrogram of the reverberation data; The acoustic feature combination unit an acoustic feature extractor F used to extract acoustic features of the input batch data; an acoustic feature correction unit that corrects the combined acoustic feature quantity by a correction function R; The domain estimation cost processing unit a domain classifier H that determines whether each sample constituting the input batch data is echoed data or anechoic data; The reverberation encoder G, the acoustic feature extractor F, and the correction function R are adversarially trained to degrade the performance of domain classification by the domain classifier H. Model learning device.

3. 3. The model learning device according to claim 2, the reconstruction cost processing unit, the acoustic feature combination unit, the domain estimation cost processing unit, and the acoustic event detection and localization cost processing unit perform the updating based on a backpropagation algorithm; The domain estimation cost processing unit inverts the sign of the error based on the backpropagation method. Model learning device.

4. Data obtained by recording an acoustic event in an environment with reverberation is called reverberant data, data obtained by recording an acoustic event in an anechoic chamber is called anechoic data, data obtained by combining the reverberant data and the anechoic data is called input batch data, and data obtained by reverberating sound in the same environment as the environment in which the reverberant data was obtained is called reverberant sound data. a reconstruction cost processing step of calculating a cost function designed to bring the spectrogram of the reverberant sound data and the reconstructed spectrogram of the corresponding reverberant sound data closer to each other, and repeatedly executing a process of updating the acoustic feature of the reverberant sound data and the reconstructed spectrogram of the corresponding reverberant sound data; an acoustic feature combining step of combining the acoustic features of the input batch data and the acoustic features of the reverberant sound data and repeatedly performing a process of updating the acoustic features of the input batch data; a domain estimation cost processing step for repeatedly determining whether each sample constituting the input batch data is anechoic data or anechoic data, calculating a cost function based on the corresponding label, and updating the result of the determination; an acoustic event detection and localization cost processing step for repeatedly estimating the type and occurrence time of an acoustic event or its direction based on the combined acoustic features, calculating a cost function based on the corresponding label, and updating the estimation result; a cost calculation step of calculating a cost function to be used for the update as a weighted sum of calculation results of the cost function; Model learning methods.

5. A program that causes a computer to function as the model learning device according to any one of claims 1 to 3.

Images