Room impulse response simulation method and system based on generative adversarial network

Abstract

The invention belongs to the field of sound field environment modeling, and particularly discloses a room impulse response simulation method and system based on a generative adversarial network. The method comprises the following steps: carrying out convolution on a public room impulse response data set and a pure audio data set, and adding noise at the same time to obtain a reverberation audio training data set; inputting the constructed reverberation audio training data set into a physical parameter estimation network for pre-training, calculating a loss function of the network, and adjusting network parameters through gradient back propagation; and freezing the physical parameter estimation network, jointly training the generative adversarial network, taking the pre-trained physical parameter estimation network as a physical constraint module, and guiding a generator in the generative adversarial network to output a room impulse response conforming to a physical rule. According to the invention, the problem of distortion of the generated pulse response can be effectively reduced, and the generated pulse response is closer to real data.

Description

Technical Field

[0001] The present application relates to the field of sound field environment modeling, and more specifically, to a room impulse response simulation method and system based on a generative adversarial network. Background Art

[0002] The room impulse response (RIR) refers to the time-varying sound pressure recorded at the receiving point when a sound source emits an impulse signal in a room. It reflects the room's sound propagation, reflection, absorption, and attenuation characteristics, and is composed of early reflections and late reverberation. Early reflections are reflected sounds that arrive at the receiving point shortly after the sound source is emitted, enhancing the clarity and spatial perception of the sound. Late reverberation is a continuous sound field formed after multiple reflections, affecting the richness and duration of the sound. Room impulse responses have important applications in audio production, acoustic design, speech recognition, and other fields. By simulating and analyzing impulse responses, it is possible to optimize room acoustics, improve audio quality, and enhance voice communication performance.

[0003] However, traditional methods mainly simulate room impulse responses through pure signal processing, and the simulated room impulse response output is generally distorted. Therefore, how to solve the problem of simulated room impulse response distortion is a difficult problem in current research. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the purpose of this application is to provide a room impulse response simulation method and system based on a generative adversarial network, which can effectively reduce the problem of distortion of the generated impulse response and make the generated impulse response closer to the real data.

[0005] To achieve the above objectives, in a first aspect, the present application provides a room impulse response simulation method based on a generative adversarial network, comprising the following steps:

[0006] S10, convolving the public room impulse response dataset with the clean audio dataset and adding noise to obtain a reverberant audio training dataset;

[0007] S20, inputting the constructed reverberation audio training dataset into a physical parameter estimation network for pre-training, calculating the network's loss function, and adjusting the network parameters through gradient backpropagation. The parameters encode the physical laws of the room impulse response and implicitly model the mapping relationship between the room impulse response time domain waveform and the physical parameters;

[0008] S30, freeze the physical parameter estimation network, jointly train the generative adversarial network, use the pre-trained physical parameter estimation network as a physical constraint module, and guide the generator in the generative adversarial network to output a room impulse response that conforms to the laws of physics by complementing the loss function of the pre-trained physical parameter estimation network and the adversarial loss of the jointly trained generative adversarial network.

[0009] The beneficial effects of the present application are as follows: the room impulse response simulation method based on the generative adversarial network provided by the present application adds a physical parameter estimation network as a constraint module of the generator in the generative adversarial network model, and the module provides the generator with a deterministic gradient direction requirement T during the generator generation impulse response process. 60 (i.e., reverberation time), energy decay ratio, and waveform frequency band correspondence are close to their target values, which complements the stochastic gradient of the adversarial loss, alleviates the mode collapse, reduces the problem of distortion of the generated impulse response, and makes the generated impulse response closer to the real data.

[0010] As a further preference, in step S20, the main body of the physical parameter estimation network is a convolutional recurrent neural network.

[0011] As a further preference, in step S20, the pre-training and joint training processes both have four stages, including forward propagation, error calculation, back propagation and parameter update.

[0012] As a further preference, in step S30, the physical constraint module is divided into three two-dimensional convolutional layers, three pooling layers, one recurrent network layer and one fully connected output layer.

[0013] As a further preferred embodiment, in step S30, the physical constraint module includes a convolutional network module, a recurrent network module and a regression output layer.

[0014] As a further preferred embodiment, in step S20, the loss function L of the pre-trained physical parameter estimation network is phy The calculation formula is:

[0015]

[0016] Where λ1, λ2, and λ3 are the reverberation time, energy decay ratio, and waveform frequency band weight of the impulse response, respectively; Represents the reverberation time calculation function of the impulse response generated by the generator; Indicates the marked reverberation time; EDR (h gen ) is the energy decay ratio calculation function of the impulse response generated by the generator; EDR(h real ) is the energy attenuation ratio of the label; h gen The waveform frequency band for the generator to generate impulse responses; h realis the marked waveform frequency band; ‖*‖2 represents the L2 norm.

[0017] As a further preferred embodiment, in step S30, the loss function L of the joint training generation adversarial network is total The calculation formula is:

[0018] L total =L WGAN +L phy

[0019] Where, L WGAN The adversarial loss for training the generated adversarial network; the loss function L phy Loss function for pre-training the physical parameter estimation network.

[0020] As a further preference, the room impulse response simulation method adopts a neural network model, the neural network model utilizes a staged training model, and uses an Adam optimization algorithm to allow the model to converge.

[0021] In a second aspect, the present application provides a room impulse response simulation system based on a generative adversarial network, comprising:

[0022] The dataset generation module is used to convolve the public room impulse response dataset with the clean audio dataset and add noise to obtain the reverberant audio training dataset;

[0023] A pre-training module is used to input the constructed reverberant audio training dataset into the physical parameter estimation network for pre-training, while calculating the network's loss function and adjusting the network parameters through gradient backpropagation. The parameters encode the physical laws of the room impulse response and implicitly model the mapping relationship between the room impulse response time domain waveform and the physical parameters;

[0024] A joint training module is used to freeze the physical parameter estimation network, jointly train a generative adversarial network, use the pre-trained physical parameter estimation network as a physical constraint module, and guide the generator in the generative adversarial network to output a room impulse response that conforms to physical laws by complementing the loss function of the pre-trained physical parameter estimation network and the adversarial loss of the jointly trained generative adversarial network.

[0025] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of a room impulse response simulation method based on a generative adversarial network provided in an embodiment of the present application;

[0027] Figure 2This is a principle block diagram of a room impulse response simulation method based on a generative adversarial network provided in an embodiment of the present application;

[0028] Figure 3 This is a diagram of the physical constraint module architecture provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0030] It should be understood that, in the description of this application, the term "several" means at least one, such as one, two, etc., unless otherwise clearly and specifically defined; the term "plurality" means two or more, unless otherwise clearly and specifically defined; the terms "first" and "second" etc. are used to distinguish different objects, rather than to describe a specific order of objects; the term "and / or" includes any and all combinations of one or more related listed items.

[0031] Additionally, references throughout this specification to "one embodiment," "one embodiment," "an example," or similar language indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, appearances of the phrase "in one embodiment," "in one embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0032] After research, the present application found that the generative adversarial network is a simulation model that maps low-dimensional data into high-dimensional data, which includes a generator and a discriminator that are trained alternately with each other. Among them, the training goal of the generator is to map the noise vector from the distribution of noise to the distribution of target data, while the training goal of the discriminator is to distinguish whether the input is data generated by the generator or real data. During the training process, the generator is optimized in the direction of generating samples that are difficult for the discriminator to distinguish, while the discriminator is optimized in the direction of being able to distinguish between generated data and real data. The generator and the discriminator are in a state of adversarial game. After a period of alternating training, the data generated by the generator will be difficult to distinguish from the real data, so that the generator can be used for applications such as data simulation.

[0033] Based on this, in order to solve the problem of distortion of traditional simulated room impulse response, the present application provides a room impulse response simulation method based on a generative adversarial network. The method can adopt a neural network model, which mainly includes a generator, a physical constraint module and a discriminator.

[0034] like Figure 1 and 2As shown, the room impulse response simulation method provided by this embodiment includes steps S10 to S30, which are described in detail as follows:

[0035] S10, convolving the public room impulse response dataset with the clean audio dataset and adding noise to obtain a reverberant audio training dataset.

[0036] S20, inputs the constructed reverberation audio training dataset into the physical parameter estimation network for pre-training, calculates the loss function of the network at the same time, and adjusts the network parameters through gradient backpropagation. The parameters encode the physical laws of the room impulse response and implicitly model the mapping relationship between the time domain waveform of the room impulse response and the physical parameters.

[0037] In this embodiment, after adjusting the parameters of the pre-trained physical parameter estimation network, an auxiliary network that accurately predicts acoustic parameters can be obtained.

[0038] Specifically, the loss function L of the pre-trained physical parameter estimation network is phy The calculation formula is:

[0039]

[0040] Where λ1, λ2, and λ3 are the reverberation time, energy decay ratio, and waveform frequency band weight of the impulse response, respectively; Represents the reverberation time calculation function of the impulse response generated by the generator; Indicates the marked reverberation time; EDR (h gen ) is the energy decay ratio calculation function of the impulse response generated by the generator; EDR(h real ) is the energy attenuation ratio of the label; h gen The waveform frequency band for the generator to generate impulse responses; h real is the marked waveform frequency band; ‖*‖2 represents the L2 norm.

[0041] S30, freeze the physical parameter estimation network, that is, its parameters will no longer be transformed after the pre-training is completed, and then jointly train the generative adversarial network, use the pre-trained physical parameter estimation network as a physical constraint module, and guide the generator in the generative adversarial network to output a room impulse response that conforms to the laws of physics by complementing the loss function of the pre-trained physical parameter estimation network and the adversarial loss of the jointly trained generative adversarial network.

[0042] In this embodiment, in steps S20 and S30, the generative adversarial network (GAN) consists of a generator and a discriminator, trained through alternating collaborative training, forming a dynamic learning system. The core training goal of the generator is to map random vectors in the potential noise space to the target data distribution space, while the discriminator is dedicated to constructing a data authenticity discrimination boundary, distinguishing generated samples from real samples through probability estimation. Within the adversarial learning framework, the generator continuously optimizes its parameter space through a backpropagation algorithm to generate synthetic samples that can deceive the discriminator. Simultaneously, the discriminator synchronously updates its classification boundary to improve its ability to distinguish real from fake samples. This two-way adversarial game mechanism drives the model's continuous self-evolution. After sufficient iterations, the generator's output distribution reaches a Nash equilibrium with the real data distribution. At this point, the generated samples have statistical properties that are highly consistent with the real data, making them effective for downstream tasks such as data augmentation and feature learning. To address the physical irrationality issues encountered by general GANs, an acoustic parameter regression network (i.e., a physical parameter estimation network) is pre-trained and used as the physical constraint module of the generator. In addition, the discriminator also receives both the room impulse response waveform and the physical parameters to enhance the conditional constraints. The neural network model provided in this embodiment utilizes a staged training model and adopts the Adam optimization algorithm to allow the model to fully converge.

[0043] Specifically, the loss function L of the joint training generative adversarial network is total The calculation formula is:

[0044] L total =L WGAN +L phy

[0045] Where, L WGAN The adversarial loss for joint training of the generative adversarial network is to maximize the score of the generated data vector after passing through the discriminator, so that the distribution of the generated data is closer to the real data distribution, and the discriminator maximizes the distance between the distribution of the generated data and the distribution of the real data, so that the generated data and the real data are easier to distinguish; the loss function L phy Loss function for pre-training the physical parameter estimation network.

[0046] Specifically, the generative adversarial network provided in this embodiment includes a generator and a discriminator.

[0047] The input of the generator is the original time domain waveform, which consists of a fully connected layer and a transposed convolution layer. The transposed convolution layer is divided into 5 layers of one-dimensional convolution with a kernel size of 25. The upsampling step is 4, and there is no batch normalization to avoid destroying the dynamic range of the waveform. The first four layers are ReLU activated, and the last output layer is Tanh activated to limit the waveform range to [-1,1].

[0048] The input of the discriminator is a real or generated sample waveform. It consists of a one-dimensional convolutional layer and a front-end fully connected layer. The one-dimensional convolutional layer is divided into 5 layers of one-dimensional convolution with the same kernel size. The downsampling step is 4, and there is no batch normalization to avoid destroying the dynamic range of the waveform. After each convolution layer, the feature map is randomly shifted in the displacement range Uniform[-n,n], where n=2. Each layer is activated by LeakyReLU.

[0049] Compared with other existing technologies, the beneficial effects produced by this embodiment are:

[0050] This embodiment innovatively proposes a room impulse response simulation method based on a generative adversarial network, which overcomes the limitations of high reverberation data modeling and high distortion in traditional simulation methods. The generative adversarial network model of this embodiment incorporates a physical parameter estimation network as a constraint module of the generator. During the generator's impulse response generation process, this module provides the generator with a deterministic gradient direction requirement T. 60 The reverberation time, energy decay ratio, and waveform frequency band are aligned towards their target values, complementing the stochastic gradient of the adversarial loss and mitigating mode collapse. With the addition of physical constraints, the equilibrium point must not only match the generated distribution with the true distribution but also satisfy the intersection of physically feasible solution spaces.

[0051] In one embodiment, in steps S20 and S30, both the pre-training and joint training processes have four stages, including forward propagation, error calculation, back propagation, and parameter update.

[0052] In one embodiment, Figure 3 As shown, in step S30, the physical constraint module is divided into three two-dimensional convolutional layers, three pooling layers, one recurrent network layer and one fully connected output layer.

[0053] In one embodiment, in step S30, the physical constraint module includes a convolutional network module, a recurrent network module, and a regression output layer.

[0054] Based on the same inventive concept, the present application also provides a room impulse response simulation system based on a generative adversarial network, including a data set generation module, a pre-training module and a joint training module.

[0055] Among them, the dataset generation module is used to convolve the public room impulse response dataset with the clean audio dataset, and add noise to obtain the reverberant audio training dataset.

[0056] The pre-training module is used to input the constructed reverberant audio training dataset into the physical parameter estimation network for pre-training. At the same time, it calculates the loss function of the network and adjusts the network parameters through gradient backpropagation. The parameters encode the physical laws of the room impulse response and implicitly model the mapping relationship between the time domain waveform of the room impulse response and the physical parameters.

[0057] The joint training module is used to freeze the physical parameter estimation network and jointly train the generative adversarial network. The pre-trained physical parameter estimation network is used as the physical constraint module. By complementing the loss function of the pre-trained physical parameter estimation network and the adversarial loss of the jointly trained generative adversarial network, the generator in the generative adversarial network is guided to output a room impulse response that conforms to the laws of physics.

[0058] It should be noted that the functions of the modules provided in this embodiment can be found in the detailed description of the aforementioned method embodiment, and will not be described in detail in this embodiment.

[0059] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A room impulse response simulation method based on generative adversarial network, characterized in that: The steps include: S10, convolving the public room impulse response dataset with the clean audio dataset and adding noise to obtain a reverberant audio training dataset; S20, inputting the constructed reverberation audio training dataset into a physical parameter estimation network for pre-training, calculating the network's loss function, and adjusting the network parameters through gradient backpropagation. The parameters encode the physical laws of the room impulse response and implicitly model the mapping relationship between the room impulse response time domain waveform and the physical parameters; S30, freeze the physical parameter estimation network, jointly train the generative adversarial network, use the pre-trained physical parameter estimation network as a physical constraint module, and guide the generator in the generative adversarial network to output a room impulse response that conforms to the laws of physics by complementing the loss function of the pre-trained physical parameter estimation network and the adversarial loss of the jointly trained generative adversarial network.

2. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: In step S20, the main body of the physical parameter estimation network is a convolutional recurrent neural network.

3. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: In step S20, the pre-training and joint training processes both have four stages, including forward propagation, error calculation, back propagation and parameter update.

4. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: In step S30, the physical constraint module is divided into three two-dimensional convolutional layers, three pooling layers, one recurrent network layer and one fully connected output layer.

5. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: In step S30, the physical constraint module includes a convolutional network module, a recurrent network module and a regression output layer.

6. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: In step S20, the loss function L of the pre-trained physical parameter estimation network is phy The calculation formula is: Where λ1, λ2, and λ3 are the reverberation time, energy decay ratio, and waveform frequency band weight of the impulse response, respectively; Represents the reverberation time calculation function of the impulse response generated by the generator; Indicates the marked reverberation time; EDR (h gen ) is the energy decay ratio calculation function of the impulse response generated by the generator; EDR(h real ) is the energy attenuation ratio of the label; h gen The waveform frequency band for the generator to generate impulse responses; h real is the marked waveform frequency band; ‖*‖2 represents the L2 norm.

7. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: In step S30, the loss function L of the joint training generative adversarial network is total The calculation formula is: L total =L WGAN +L phy Where, L WGAN The adversarial loss for training the generated adversarial network; the loss function L phy Loss function for pre-training the physical parameter estimation network.

8. The room impulse response simulation method based on generative adversarial network according to claim 1, characterized in that: The room impulse response simulation method adopts a neural network model, the neural network model utilizes a staged training model, and adopts the Adam optimization algorithm to allow the model to converge.

9. A room impulse response simulation system based on a generative adversarial network, characterized in that: include: The dataset generation module is used to convolve the public room impulse response dataset with the clean audio dataset and add noise to obtain the reverberant audio training dataset; A pre-training module is used to input the constructed reverberant audio training dataset into the physical parameter estimation network for pre-training, while calculating the network's loss function and adjusting the network parameters through gradient backpropagation. The parameters encode the physical laws of the room impulse response and implicitly model the mapping relationship between the room impulse response time domain waveform and the physical parameters; A joint training module is used to freeze the physical parameter estimation network, jointly train a generative adversarial network, use the pre-trained physical parameter estimation network as a physical constraint module, and guide the generator in the generative adversarial network to output a room impulse response that conforms to physical laws by complementing the loss function of the pre-trained physical parameter estimation network and the adversarial loss of the jointly trained generative adversarial network.

Images