Model training device, model training method, and program

Abstract

Provided is a model training device that trains a model for distance estimation, by utilizing prior knowledge based on the acoustic power level of a sound source and the physics of a distance attenuation model from an observation point to each acoustic event. The model training device includes: a SELD cost function calculating unit that calculates a cost function between a SELD estimation result and a correct answer SELD label, and updates parameters of a model used for SELD estimation; and a distance cost function calculating unit that calculates an estimated value of the distance from an observation point to each acoustic event on the basis of acoustic power data, which are data to which acoustic power has been assigned as prior knowledge, calculates a cost function with the correct answer label of the distance, and, on the basis of the cost function, updates indoor acoustic characteristic parameters and parameters of a model used for acoustic power variation amount estimation.

Description

Model learning device, model learning method, and program

[0001] The present disclosure relates to a model learning device, a model learning method, and a program for learning a model for estimating a distance from an observation point to each acoustic event.

[0002] Conventionally, acoustic event distance estimation, which estimates the type of event that occurred in the surrounding environment and the distance to the sound source from an acoustic signal, has used a data-driven approach in which a deep neural network (DNN) that takes the acoustic signal as input is used as a regression function of the sound source distance and is trained using teacher labels of the sound source distance (Non-Patent Document 1).

[0003] DA Krause, A. Politis and A. Mesaros, “Sound Event Detection and Localization with Distance Estimation”, arXiv:2403.11827, 2024.

[0004] Sound sources often have a certain degree of acoustic power depending on their type. However, conventional methods estimate distance using a black-box regression function based on a trained system, which makes it impossible to explicitly consider the aforementioned properties of the sound source or constraints based on physical laws, resulting in a decrease in accuracy. In particular, for acoustic events whose acoustic power is clearly defined in advance, such as ambulance sirens, an estimation system that can utilize this prior knowledge is desirable, but such a system did not exist.

[0005] Therefore, an object of the present disclosure is to provide a model learning device that learns a model for distance estimation by utilizing prior knowledge based on physics of the acoustic power level of a sound source and a distance attenuation model from an observation point to each acoustic event.

[0006] The model learning device of the present disclosure includes a SELD cost function calculation unit and a distance cost function calculation unit.

[0007] The SELD cost function calculation unit calculates a cost function between the SELD estimation result using the acoustic signal data as training data and the correct SELD label corresponding to the acoustic signal data, and updates the parameters of the model used to estimate SELD based on the cost function.

[0008] The distance cost function calculation unit calculates an estimate of the distance from the observation point to each acoustic event based on the room acoustic characteristic parameters, the acoustic power fluctuations of each sound source, the sound pressure level of each sound source, and the acoustic power corresponding to each sound source class, which are calculated based on the acoustic signal data, which is training data, the acoustic event class labels, and the acoustic power data, which is data in which acoustic power is assigned as prior knowledge for each acoustic event class; calculates a cost function between the distance and the correct label; and updates the parameters of the model used to estimate the room acoustic characteristic parameters and the acoustic power fluctuations based on the cost function.

[0009] The model learning device of the present disclosure can learn a model for distance estimation by utilizing prior knowledge based on the physics of the acoustic power level of a sound source and a distance attenuation model from an observation point to each acoustic event.

[0010] 1 is a block diagram showing the functional configuration of a model learning device according to a first embodiment; 2 is a flowchart showing the operation of the model learning device according to a first embodiment; 3 is a block diagram showing the functional configuration of a sound event distance estimation device according to a second embodiment; 4 is a flowchart showing the operation of the sound event distance estimation device according to a second embodiment; 5 is a diagram showing an example of the functional configuration of a computer;

[0011] Hereinafter, embodiments of the present disclosure will be described in detail. Note that components having the same functions are assigned the same numbers, and redundant explanations will be omitted.

[0012] Hereinafter, we will describe a sound event distance estimation method that utilizes prior knowledge of sound power. This sound event distance estimation method combines the sound power corresponding to the type of sound source provided as prior knowledge with room acoustic characteristic parameters and the amount of sound power fluctuation learned from data, and estimates distance based on a sound attenuation model in the room.

[0013] <Sound attenuation model in a room> The attenuation model of an omnidirectional sound source in an anechoic space is expressed as follows.

[0014] P=W+10log{1 / (4πr 2 )}...(1) where P[dB] is the sound pressure level at an observation point r[m] away from the sound source, and W[dB] is the sound power of an omnidirectional point sound source. Rearranging this using log4π≒1.1 gives P=W-20logr-11...(2). It is known that the absolute value of the constant in the third term on the right-hand side becomes smaller in a room with reverberation due to reflection. Here, by representing this third term as the room acoustics parameter a[dB], the attenuation model of an omnidirectional sound source that takes into account sound absorption in a room sound field can be expressed as follows:

[0015] P = W - 20 log r - a (3) where a = 11 is typically used in an anechoic environment and a = 8 in semi-anechoic conditions. For distance estimation, solving equation (3) for r gives

[0016] r=10(WPa) / 20…(4) In practice, the sound power W of a sound source is unknown, but if the class c∈[1,...,C] representing the type of acoustic event is known, the typical sound power can be assumed as prior knowledge. The typical sound power of a sound source of class c is W c However, the actual sound power is W c It is not constant, but fluctuates over time. This fluctuation is called ΔW c [dB].

[0017] The sound pressure level P at the observation point includes the contributions of all sound sources that exist simultaneously. When multiple sound sources exist simultaneously, it is necessary to estimate the distance from the observation point for each sound source. Therefore, the sound pressure level P for each class is used for distance estimation. c To obtain this, the sound pressure level P is distributed to each sound source, and the posterior probability p^ of the existence of each class of sound source obtained as a result of the acoustic event detection is calculated. c Assume that the weights are calculated by P c is given as follows:

[0018] Pc =log{p^ c / (Σ C c=1 p^ c )10 P}...(5) Furthermore, the input signal X recorded by the microphone is the actual sound source signal that includes the signal amplification effect (gain) of the microphone. This microphone gain is defined as G [dB]. The proposed model for distance estimation that incorporates these effects into the above distance estimation equation (4) is formulated as follows:

[0019] r c =10(Wc+ΔWc-(Pc+G)-a) / 20…(6) where, r c indicates the distance of a sound source belonging to class c among the simultaneously existing sound sources. Note that this may be extended to estimation of multiple tracks as in Non-Patent Document 1. In this case, the distance of the sound source of track t and class c is r c,t is given as follows.

[0020] The distance between the observation point and the sound source may be expressed using at least the acoustic power of the sound source, the time variation of the acoustic power of the sound source, the sound pressure at the observation point, the characteristics of the sound collection equipment including the microphone used at the observation point, and the spatial characteristics including the sound source and the observation point.

[0021] <Hybrid distance estimation model> In the following, we will discuss a hybrid distance estimation model that combines a sound attenuation model in a room with modeling based on data on variables related to the attenuation model. From the previous section, the posterior probability p^ c , sound pressure P at the observation point, sound source power W c , the fluctuation of the sound source power ΔW c When the room acoustic parameter a and the microphone gain G are all known, distance estimation for each class is possible using equation (6).

[0022] However, in reality, it is difficult to measure the amount of fluctuation in the power of the sound source and the room acoustic characteristic parameters at the time of estimation, and they are unknown parameters.

[0023] Therefore, we use a data-driven approach to regress these unknown parameters.

[0024] The power of a sound source is expected to have some inherent characteristics depending on the type of sound source. For example, it is known that a woman's speaking voice is about 80 dB. However, since conversations contain intonations, the acoustic power of an actual sound source is unlikely to be a constant value and fluctuates over time. This is called the acoustic power fluctuation amount ΔW c However, this time fluctuation is expected to change depending on various factors such as the content of the conversation and individual differences, so this ΔW c It is very difficult to determine by measurement, etc. Therefore, a data-driven estimation is performed using the input acoustic signals given as training data. c It is thought that the range and trend will change depending on the class of the event that is occurring, but the learning data is also thought to contain information about the trends for each class.

[0025] Regarding the parameter a, which represents the acoustic characteristics of a room, in the SELD problem setting, the acoustic characteristics of the room in which an acoustic event is observed are generally unknown. However, it is known that it is possible to estimate the acoustic characteristics of a room using clues such as the amount of reverberation contained in the acoustic signal. Based on this, a is also estimated data-driven from the input acoustic signal. Note that the parameter a, which represents the acoustic characteristics, is not related to the event class, as it depends on the acoustic characteristics of the room. Therefore, when the input signal is X and the posterior probability of the existence of an acoustic event of class c obtained as a result of acoustic event detection is expressed as p^ c Then, each regression function can be given in the following form:

[0026] ΔW c =f Δc (X,p^ c ;w Δc )…(7) a=f a (X;w a )…(8) where f Δc , f a are ΔW c , a regression function, w Δc , w ais the parameter. Also, if the data is not recorded by the user, the microphone gain G is also an unknown parameter, but this is a constant parameter that does not depend on the input.

[0027] The specific construction method of these regression functions and the method of estimating each parameter are arbitrary, but examples include a deep learning model such as a convolutional neural network as the regression function, and an error backpropagation method in which the MSE loss between the estimated distance and the correct distance label is used as the cost function as the parameter estimation method.

[0028] This acoustic event distance estimation method makes it possible to estimate the acoustic event distance by effectively utilizing information on sound sources with specific acoustic power, particularly in situations where the room acoustic parameters fluctuate little. An example of such an example is the sound of an ambulance siren outdoors.

[0029] Furthermore, in conventional data-driven systems (which do not use prior knowledge based on physics, such as the time variation of acoustic power or sound attenuation models), estimation is performed using parameters learned from given data. However, it is unknown whether the properties of the learned parameters are applicable to the actual environment, and this also depends on the quality and quantity of the training data. Therefore, a system trained using data containing properties different from those of the actual environment may not work well in the actual environment. By utilizing prior knowledge based on physics that holds under real conditions, such as the time variation of acoustic power or sound attenuation models, it is expected that the system will work well in the actual environment even with a small amount of data.

[0030] On the other hand, the physical laws of sound still have challenges when it comes to applying them to simultaneous estimation of multiple different sound sources and moving sound sources, which are required in the SELD problem setting, so data-driven methods are superior in terms of dealing with such multiple and moving sound sources.

[0031] The functional configuration of a model learning device that learns a model that realizes the above-described sound event distance estimation method will be described below with reference to Fig. 1. Note that the sound event distance estimation task does not need to include sound source direction estimation, but in this example, a case in which sound source directions are simultaneously estimated will be dealt with as a problem setting similar to that in Non-Patent Document 1.

[0032] As shown in the figure, the model learning device 1 of this embodiment includes a learning data storage unit 101, a SELD estimation unit 201, an observation point total sound pressure level calculation unit 202, a room acoustic characteristic parameter estimation unit 203, an acoustic power fluctuation estimation unit 204, a microphone gain estimation unit 205, an individual sound source sound pressure level estimation unit 206, a sound source acoustic power extraction unit 207, a SELD cost function calculation unit 301, and a distance cost function calculation unit 302.

[0033] The details and operation of each component will be described below, with reference to FIG.

[0034] <Learning Data Storage Unit 101> The following data 102 to 106 are stored in advance in the learning data storage unit 101. These data are learning data that include acoustic signal data and complete labels for SELD, including distance and direction estimation.

[0035] Acoustic signal data 102: Acoustic signal data. Typically, a four-channel first-order Ambisonics signal is used. Note that the Ambisonics format for the acoustic signal data is just one example, and other data formats can also be used for the acoustic signal data.

[0036] Acoustic event class label 103: For example, it can be given as a multi-hot vector indicating the type of acoustic event included at each time point.

[0037] Sound source direction label 104: Indicates the direction of arrival of each event included in each time as seen from the microphone (observation point).

[0038] Sound source distance label 105: Indicates the distance from each microphone (observation point) to each sound event included at each time.

[0039] Acoustic power data 106: Typical acoustic power W of each class of sound source to be detected c For example, if it is a human voice, c ≒80 [dB], etc. Note that the typical range of time fluctuation of the acoustic power of each class may also be included in this data.

[0040] <SELD Estimation Unit 201> The SELD estimation unit 201 receives the acoustic signal data 102 as input, estimates the type and angle (SELD) of an acoustic event, and outputs the result (S201). For the estimation, a deep learning model equivalent to that described in Non-Patent Document 1 can be used, for example, but any other model structure may also be used.

[0041] <Observation point total sound pressure level calculation unit 202> The observation point total sound pressure level calculation unit 202 receives the acoustic signal data 102 as input, applies a window function to the acoustic signal data 102, and calculates the effective sound pressure for each frame. The observation point total sound pressure level calculation unit 202 calculates and outputs the total sound pressure level P at the observation point in each frame from the effective sound pressure and the reference sound pressure (pref=2.0×10^-5 [Pa]) (S202).

[0042] <Room acoustic characteristic parameter estimation unit 203> The room acoustic characteristic parameter estimation unit 203 receives the acoustic signal data 102 as input and estimates the room acoustic characteristic parameter a using, for example, a DNN model (S203). Note that in this embodiment, the estimation is performed as a parameter that is independent of time and sound source class due to its nature.

[0043] <Acoustic power fluctuation estimation unit 204> The acoustic power fluctuation estimation unit 204 receives the acoustic signal data 102 and the acoustic event class label 103 as input, and calculates the acoustic power fluctuation ΔW of each sound source using, for example, a DNN model. c is estimated (S204).

[0044] Although the acoustic event class label 103 cannot be used in the estimation device of the second embodiment, an estimated value based on the output of the SELD estimation unit 201 can be used, as will be described later. In addition, when the acoustic power data 106 includes a typical range of the time fluctuation of the acoustic power of each class, ΔW cmay be limited to this range for estimation.

[0045] <Microphone Gain Estimation Unit 205> The microphone gain estimation unit 205 estimates the value of the microphone gain as an independent parameter (S205). The microphone gain is assumed to be constant across all data.

[0046] If the microphone gain is known, step S205 can be omitted, and in this case the known microphone gain value is used in step S302.

[0047] <Each sound source sound pressure level estimator 206> Each sound source sound pressure level estimator 206 receives the SELD estimation result from the SELD estimator 201 and the output P from the observation point total sound pressure level calculator 202, and calculates the posterior probability p^ determined based on the SELD estimation result. c Using this, the sound pressure level P for class c is calculated according to equation (5). c is estimated (S206).

[0048] <Sound source acoustic power extraction unit 207> The sound source acoustic power extraction unit 207 extracts, from the acoustic power data 106, acoustic power given as prior knowledge corresponding to the class of the sound source present at each time given by the acoustic event class label 103, and outputs the acoustic power for all sound sources present at each time (S207).

[0049] <SELD cost function calculation unit 301> The SELD cost function calculation unit 301 calculates a cost function between the SELD estimation result obtained as the output of the SELD estimation unit 201 and the correct SELD label given by the acoustic event class label 103 and the sound source arrival direction label 104, and updates the parameters of the model used to estimate the SELD using the backpropagation method or the like based on the cost function (S301).

[0050] <Distance Cost Function Calculation Unit 302> The distance cost function calculation unit 302 calculates the room acoustic characteristic parameter a calculated based on the acoustic signal data 102, the acoustic event class label 103, and the acoustic power data 106, and the acoustic power fluctuation amount ΔW of each sound source. c , microphone gain G, sound pressure level of each sound source P c , the acoustic power W corresponding to each sound source classc Based on this, the estimated distance r from the observation point to each acoustic event is calculated using equation (6). c Calculate the cost function between the distance and the correct label, and then use the backpropagation algorithm or similar to calculate the room acoustic parameter a and the acoustic power fluctuation ΔW c The parameters of the model used to estimate are updated (S302).

[0051] For example, MSE loss can be used as the cost function. Note that when estimating multiple tracks, this can be achieved by extending the cost function to take into account the permutation of tracks in the same way as in Non-Patent Document 1.

[0052] Hereinafter, the functional configuration of a sound event distance estimation device that estimates the distance of each sound event in sound signal data to be estimated using the model learned in the first embodiment will be described with reference to FIG.

[0053] As shown in the figure, the acoustic event distance estimation device 2 of this embodiment includes an estimation target data storage unit 107, an acoustic power database 108, a SELD estimation unit 201, an acoustic event class estimation unit 401, an observation point total sound pressure level calculation unit 202, a room acoustic characteristic parameter estimation unit 203, an acoustic power variation estimation unit 204, a microphone gain estimation unit 205, an individual sound source sound pressure level estimation unit 206, a sound source acoustic power extraction unit 207, and a distance calculation unit 402. Note that the components other than the acoustic event class extraction unit 401 and the distance calculation unit 402 operate in the same way as in the first embodiment, and therefore description thereof will be omitted where appropriate.

[0054] Below, details of each component that differs from the first embodiment and operations that differ from the first embodiment will be described. For the operations of each component, please also refer to FIG.

[0055] <Inference Target Data Storage Unit 107> The estimation target data storage unit 107 stores in advance acoustic signal data 102A (with A to distinguish it from the learning data in the first embodiment) that is the estimation target of this device.

[0056] <Acoustic Power Database 108> In the acoustic power database 108, acoustic power data 106 used as prior knowledge is stored in advance, similarly to the first embodiment.

[0057] <Acoustic event class extraction unit 401> Because this device is an estimation device, the acoustic event class label 103 used in Example 1 does not exist (estimation must be performed in a state where the label is unknown). In this device, the acoustic event class extraction unit 401 extracts a class label of an acoustic event from the SELD estimation result that is the output of the SELD estimation unit 201 (S401).

[0058] <Distance Calculation Unit 402> The distance calculation unit 402 calculates the estimated value r of the distance from the observation point to each acoustic event using equation (6). ｃ This is similar to step S302 in that it calculates the following, but differs from step S302 in that it does not calculate the cost function or update the model parameters.

[0059] When outputting not only the distance but also the SELD estimation result, the output of the SELD estimation unit 201 can be used.

[0060] <Additional Notes> The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0061] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0062] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0063] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 10020 of the computer shown in Figure 5 and operating the control unit 10010, input unit 10030, output unit 10040, etc.

[0064] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0065] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0066] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0067] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. A model learning device including: a SELD cost function calculation unit that calculates a cost function between an SELD estimation result using acoustic signal data as learning data and a correct SELD label corresponding to the acoustic signal data, and updates parameters of a model used to estimate the SELD based on the cost function; and a distance cost function calculation unit that calculates an estimate of the distance from an observation point to each acoustic event based on room acoustic characteristic parameters, acoustic power fluctuations of each sound source, sound pressure levels of each sound source, and acoustic power corresponding to each sound source class, which are calculated based on the acoustic signal data as learning data, acoustic event class labels, and acoustic power data, which is data in which acoustic power is assigned as prior knowledge for each acoustic event class; calculates a cost function between the correct distance label; and updates parameters of a model used to estimate the room acoustic characteristic parameters and the acoustic power fluctuations based on the cost function.

2. A model learning method executed by a model learning device, comprising: a SELD cost function calculation step of calculating a cost function between an SELD estimation result using acoustic signal data as learning data and a correct SELD label corresponding to the acoustic signal data, and updating parameters of a model used for estimating the SELD based on the cost function; and a distance cost function calculation step of calculating an estimate of the distance from an observation point to each acoustic event based on room acoustic characteristic parameters, acoustic power fluctuations of each sound source, sound pressure levels of each sound source, and acoustic power corresponding to each sound source class, which are calculated based on the acoustic signal data as learning data, acoustic event class labels, and acoustic power data, which is data in which acoustic power is assigned as prior knowledge for each acoustic event class, calculating a cost function between the correct label of the distance, and updating parameters of a model used for estimating the room acoustic characteristic parameters and the acoustic power fluctuations based on the cost function.

3. A model learning method according to claim 2, comprising: a SELD estimation step for estimating SELD; an observation point total sound pressure level calculation step for calculating a total sound pressure level at an observation point; a room acoustic characteristic parameter estimation step for estimating the room acoustic characteristic parameters; an acoustic power fluctuation estimation step for estimating the acoustic power fluctuation of each of the sound sources; a microphone gain estimation step for estimating a microphone gain; and an individual sound source sound pressure level estimation step for estimating the sound pressure level of each of the sound sources.

4. The model learning method according to claim 3, wherein the step of estimating the sound pressure level of each sound source comprises: defining the total number of classes as C, class c∈[1,...,C], the total sound pressure level at the observation point as P, and determining the posterior probability p^ of the existence of a sound source of class c. c The posterior probability p^ determined based on the SELD estimation result c Using the above, the sound pressure level P of class c c P c =log{p^ c / (Σ C c=1 p^ c )10 P }, and the distance cost function calculation step estimates the room acoustic characteristic parameter as a, the acoustic power corresponding to each sound source class as W c , the acoustic power fluctuation amount of the sound source of each class is ΔW c , the microphone gain is G, and the estimated distance r from the observation point to each acoustic event is c , r c =10(Wc+ΔWc-(Pc+G)-a) / 20.

5. A program that causes a computer to function as the model learning device according to claim 1.

Images