Sound signal processing method and model training method

Abstract

The present application provides a sound signal processing method and a model training method, wherein the sound signal processing method comprises: obtaining a multi-channel sound signal to be processed, realizing spatial and time-frequency characteristics based on the multi-channel sound signal, determining time-domain filter parameters corresponding to each channel by using a beamforming neural network in an end-to-end model, filtering and adding the time-domain filter parameters corresponding to each channel and the multi-channel sound signal to obtain a time-domain waveform of a single-channel sound signal, and performing short-time Fourier transform on the time-domain waveform of the single-channel sound signal to obtain a time-frequency graph of the single-channel sound signal; after determining a time-frequency mask based on a query sound signal by using a query-based sound source separation neural network in the end-to-end model according to the time-frequency graph of the single-channel sound signal, the target sound signal is determined according to the time-frequency graph of the single-channel sound signal and the time-frequency mask. Thus, the target sound signal after noise reduction, dereverberation and separation can be obtained, and the interference signal can be effectively suppressed.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and in particular to an acoustic signal processing method and a model training method. Background Technology

[0002] In array signal processing, signals acquired by multiple antennas are often weighted and combined to improve the signal-to-noise ratio (SNR) or signal-to-interference-plus-noise ratio (SINR) of the received signal. This weighted combining process is called beamforming. The basic principle of beamforming in physics is the principle of interference, specifically the coherent superposition of coherent signal sources. More simply, the superposition of signals in the same direction will produce a peak. It can be understood that since signals are coherent, they can coherently superimpose, while noise is incoherent and cannot coherently superimpose. Therefore, beamforming technology can suppress noise and interference in different directions, enhancing the signal in the target direction.

[0003] In related technologies, the time delay difference (ITD) and intensity difference (IID) of audio can be used for sound source separation. However, this method has spatial constraints, is limited by the single sound source assumption, and has many constraints, making it quite restrictive. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose an acoustic signal processing method that, under complex acoustic conditions, achieves source separation of dereverberation and interference sounds by employing two neural networks in an end-to-end model, without pre-setting various constraints or assumptions. This method demonstrates good applicability in practical applications and effectively solves the problem of sound source noise reduction.

[0006] The second objective of this invention is to propose a model training method.

[0007] The third objective of this invention is to provide an acoustic signal processing device.

[0008] The fourth objective of this invention is to provide a model training device.

[0009] The fifth objective of this invention is to provide an electronic device.

[0010] The sixth object of the present invention is to provide a computer-readable storage medium.

[0011] The seventh objective of this invention is to provide a computer program product.

[0012] To achieve the above objectives, a first aspect of the present invention provides a method for processing acoustic signals, comprising:

[0013] Acquire the multi-channel acoustic signal to be processed;

[0014] Based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, the beamforming neural network in the end-to-end model is used to determine the time-domain filter parameters corresponding to each channel.

[0015] The time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal are filtered and summed to obtain the time-domain waveform of the single-channel acoustic signal, and a short-time Fourier transform is performed on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal.

[0016] Based on the time-frequency diagram of the single-channel acoustic signal, a query-based sound source separation neural network in the end-to-end model is used to determine the time-frequency mask of the query-based acoustic signal.

[0017] The target acoustic signal is determined based on the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask.

[0018] Optionally, as a first possible implementation of the first aspect, the step of determining the time-domain filter parameters corresponding to each channel using a beamforming neural network in an end-to-end model based on the spatial and time-frequency characteristics of the multi-channel acoustic signal includes:

[0019] For any channel of the multi-channel acoustic signal, based on the spatial and time-frequency characteristics of the channel, determine the inter-channel characteristics and channel-specific characteristics of the channel;

[0020] The inter-channel features and channel-specific features of each channel are input into the beamforming neural network in the end-to-end model to obtain the time-domain filter parameters corresponding to each channel.

[0021] Optionally, as a second possible implementation of the first aspect, determining the inter-channel characteristics and channel-specific characteristics of any channel of the multi-channel acoustic signal based on the spatial and time-frequency characteristics of the channel includes:

[0022] For any channel of the multi-channel acoustic signal, the inter-channel characteristics of the channel are determined using the following formula:

[0023]

[0024] in, This represents the inter-channel characteristics of channel i; i represents the channel number; j represents... The time index of the signal; Indicates the context window signal of channel i around the current time frame; x 1 This indicates the signal of reference channel 1 in the current time frame; L represents the length of the time frame.

[0025] The L-dimensional features of the channel in the current time frame are converted into K-dimensional features using the following formula to obtain the channel-specific features:

[0026] R i =x i U

[0027] Among them, R i Indicates the channel-specific characteristics of channel i; x i U represents the signal of channel i in the current time frame; U represents the training transformation matrix.

[0028] Optionally, as a third possible implementation of the first aspect, the beamforming neural network includes an encoder layer, a temporal convolutional network layer, a 1×1 convolutional layer, and a gated output layer. The step of inputting the inter-channel features and channel-specific features of each channel into the beamforming neural network in the end-to-end model to obtain the temporal filter parameters corresponding to each channel includes:

[0029] For any given channel, the channel-specific features and channel-specific characteristics of the channel are concatenated to obtain the input data;

[0030] The input data is fed into the encoder layer, and after passing through the encoder layer, the temporal convolutional network layer and the 1×1 convolutional layer, at least one set of feature data is output.

[0031] The at least one set of feature data is input to the gated output layer to obtain the time-domain filter parameters corresponding to the channel.

[0032] Optionally, as a fourth possible implementation of the first aspect, determining the time-frequency mask based on the query-based acoustic signal using a query-based sound source separation neural network in an end-to-end model based on the time-frequency diagram of the single-channel acoustic signal includes:

[0033] A hierarchical self-attention model based on a moving window is used to extract low-dimensional features from the query acoustic signal; wherein the query acoustic signal is a pre-acquired single-channel, interference-free sound signal;

[0034] The time-frequency diagram and the low-dimensional features are input into a query-based sound source separation neural network in the end-to-end model to obtain a time-frequency mask based on the query sound signal.

[0035] Optionally, as a fourth possible implementation of the first aspect, the query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture, including at least one downsampling layer, at least one upsampling layer, and one convolutional layer. The step of inputting the time-frequency map and the low-dimensional features into the query-based sound source separation neural network in the end-to-end model to obtain a time-frequency mask based on the query sound signal includes:

[0036] The low-dimensional features are respectively input into at least one downsampling layer, at least one upsampling layer and a convolutional layer to obtain the feature dimensions corresponding to the at least one downsampling layer, at least one upsampling layer and a convolutional layer;

[0037] The time-frequency graph is input into a query-based sound source separation neural network in the end-to-end model to output a time-frequency mask based on the query sound signal, based on the time-frequency graph and the feature dimensions corresponding to each layer.

[0038] Optionally, as a fifth possible implementation of the first aspect, determining the target acoustic signal based on the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask includes...

[0039] Multiply the time-frequency diagram of the single-channel acoustic signal with the time-frequency mask to obtain the time-frequency diagram of the target acoustic signal;

[0040] A short-time inverse Fourier transform is performed on the time-frequency diagram of the target acoustic signal to obtain the time-frequency waveform of the target acoustic signal.

[0041] To achieve the above objectives, a second aspect of the present invention provides a model training method, comprising:

[0042] Acquire single-channel acoustic signals and generate multi-channel training acoustic signals based on the single-channel acoustic signals;

[0043] Based on the multi-channel training acoustic signal, the time-frequency waveform of the target acoustic signal is determined using an end-to-end model;

[0044] The average absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal is used as the loss function of the end-to-end model, and the end-to-end model is trained based on the loss function.

[0045] To achieve the above objectives, a third aspect of the present invention provides an acoustic signal processing apparatus, comprising:

[0046] The acquisition module is used to acquire the multi-channel acoustic signal to be processed;

[0047] The first determining module is used to determine the time-domain filter parameters corresponding to each channel based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, using a beamforming neural network in an end-to-end model.

[0048] The first processing module is used to filter and sum the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal, and to perform a short-time Fourier transform on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal.

[0049] The second determining module is used to determine the time-frequency mask of the query-based acoustic signal based on the time-frequency diagram of the single-channel acoustic signal using a query-based sound source separation neural network in an end-to-end model.

[0050] The third determining module is used to determine the target acoustic signal based on the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask.

[0051] Optionally, as a first possible implementation of the third aspect, the first determining module includes:

[0052] The determining unit is used to determine the inter-channel characteristics and channel-specific characteristics of any channel of the multi-channel acoustic signal based on the spatial and time-frequency characteristics of the channel.

[0053] The first processing unit is used to input the inter-channel features and channel-specific features of each channel into the beamforming neural network in the end-to-end model to obtain the time-domain filter parameters corresponding to each channel.

[0054] Alternatively, as a second possible implementation of the third aspect, the determining unit is further configured to:

[0055] For any channel of the multi-channel acoustic signal, the inter-channel characteristics of the channel are determined using the following formula:

[0056]

[0057] in, This represents the inter-channel characteristics of channel i; i represents the channel number; j represents... The time index of the signal; Indicates the context window signal of channel i around the current time frame; x 1 This indicates the signal of reference channel 1 in the current time frame; L represents the length of the time frame.

[0058] The L-dimensional features of the channel in the current time frame are converted into K-dimensional features using the following formula to obtain the channel-specific features:

[0059] R i =x i U

[0060] Among them, R i Indicates the channel-specific characteristics of channel i; x i U represents the signal of channel i in the current time frame; U represents the training transformation matrix.

[0061] Optionally, as a third possible implementation of the third aspect, the beamforming neural network includes an encoder layer, a temporal convolutional network layer, a 1×1 convolutional layer, and a gated output layer, and the first processing unit is further configured to:

[0062] For any given channel, the channel-specific features and channel-specific characteristics of the channel are concatenated to obtain the input data;

[0063] The input data is fed into the encoder layer, and after passing through the encoder layer, the temporal convolutional network layer and the 1×1 convolutional layer, at least one set of feature data is output.

[0064] The at least one set of feature data is input to the gated output layer to obtain the time-domain filter parameters corresponding to the channel.

[0065] Alternatively, as a fourth possible implementation of the third aspect, the second determining module includes:

[0066] The extraction unit is used to extract low-dimensional features of the query acoustic signal using a hierarchical self-attention model based on a moving window; wherein the query acoustic signal is a pre-acquired single-channel, interference-free sound signal;

[0067] The second processing unit is used to input the time-frequency diagram and the low-dimensional features into the query-based sound source separation neural network in the end-to-end model to obtain a time-frequency mask based on the query sound signal.

[0068] Optionally, as a fifth possible implementation of the third aspect, the query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture, including at least one downsampling layer, at least one upsampling layer, and one convolutional layer, wherein the second processing unit is further configured to:

[0069] The low-dimensional features are respectively input into at least one downsampling layer, at least one upsampling layer and a convolutional layer to obtain the feature dimensions corresponding to the at least one downsampling layer, at least one upsampling layer and a convolutional layer;

[0070] The time-frequency graph is input into a query-based sound source separation neural network in the end-to-end model to output a time-frequency mask based on the query sound signal, based on the time-frequency graph and the feature dimensions corresponding to each layer.

[0071] Optionally, as a sixth possible implementation of the third aspect, the third determining module is further configured to:

[0072] Multiply the time-frequency diagram of the single-channel acoustic signal with the time-frequency mask to obtain the time-frequency diagram of the target acoustic signal;

[0073] A short-time inverse Fourier transform is performed on the time-frequency diagram of the target acoustic signal to obtain the time-frequency waveform of the target acoustic signal.

[0074] To achieve the above objectives, a fourth aspect of the present invention provides a model training apparatus, comprising:

[0075] The second processing module is used to acquire single-channel acoustic signals and generate multi-channel training acoustic signals based on the single-channel acoustic signals.

[0076] The fourth determining module is used to determine the time-frequency waveform of the target acoustic signal based on the multi-channel training acoustic signal using an end-to-end model;

[0077] The training module is used to use the average absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal as the loss function of the end-to-end model, so as to train the end-to-end model based on the loss function.

[0078] To achieve the above objectives, a fifth aspect of the present invention provides an electronic device comprising:

[0079] At least one processor; and

[0080] A memory communicatively connected to the at least one processor; wherein,

[0081] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the aforementioned acoustic signal processing method of the first aspect, or to perform the aforementioned model training method of the second aspect.

[0082] To achieve the above objectives, a sixth aspect of the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the aforementioned acoustic signal processing method of the first aspect, or to execute the aforementioned model training method of the second aspect.

[0083] To achieve the above objectives, a seventh aspect of the present invention provides a computer program product, characterized in that it includes a computer program, which, when executed by a processor, implements the aforementioned acoustic signal processing method of the first aspect, or implements the aforementioned model training method of the second aspect.

[0084] The technical solutions provided by the embodiments of the present invention have the following beneficial effects:

[0085] By acquiring the multi-channel acoustic signals to be processed, and based on the spatial and time-frequency characteristics of the multi-channel acoustic signals, a beamforming neural network in an end-to-end model is used to determine the time-domain filter parameters corresponding to each channel. The time-domain filter parameters of each channel and the multi-channel acoustic signals are then filtered and summed to obtain the time-domain waveform of a single-channel acoustic signal. A short-time Fourier transform is then performed on the time-domain waveform of the single-channel acoustic signal to obtain its time-frequency diagram. Based on the time-frequency diagram of the single-channel acoustic signal, a query-based source separation neural network in the end-to-end model is used to determine the time-frequency mask based on the query acoustic signal. Finally, the target acoustic signal is determined based on the time-frequency diagram and the time-frequency mask of the single-channel acoustic signal. Therefore, under complex acoustic environment conditions, the use of two neural networks in an end-to-end model achieves source separation for dereverberation and interference sounds without pre-setting various constraints or assumptions, demonstrating good applicability in practical applications and effectively solving the problem of sound source noise reduction.

[0086] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0087] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0088] Figure 1 This is a schematic flowchart of an acoustic signal processing method provided in an embodiment of the present invention;

[0089] Figure 2 This is a schematic flowchart of another acoustic signal processing method provided in an embodiment of the present invention;

[0090] Figure 3 This is a schematic diagram of a beamforming neural network provided in an embodiment of the present invention;

[0091] Figure 4 This is a schematic diagram of the structure of a temporal convolutional network layer provided in an embodiment of the present invention;

[0092] Figure 5 This is a schematic flowchart of another acoustic signal processing method provided in an embodiment of the present invention;

[0093] Figure 6 This is a schematic diagram of a query-based sound source separation neural network provided in an embodiment of the present invention;

[0094] Figure 7 This is a schematic diagram illustrating the principle of a sound signal processing method in a specific scenario provided by an embodiment of the present invention.

[0095] Figure 8This is a flowchart illustrating a model training method provided in an embodiment of the present invention;

[0096] Figure 9 This is a schematic diagram of the structure of an acoustic signal processing device provided in an embodiment of the present invention;

[0097] Figure 10 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present invention;

[0098] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0099] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0100] In related technologies, the time delay difference (ITD) and intensity difference (IID) of audio can be used for sound source separation. However, this method has spatial constraints, is limited by the single sound source assumption, and has many constraints, making it quite restrictive.

[0101] To address this issue, embodiments of the present invention provide an acoustic signal processing method that enables the separation of reverberation and interference sources in complex acoustic environments by employing two neural networks in an end-to-end model. This method does not require pre-setting various constraints or assumptions, and has good applicability in practical applications, effectively solving the problem of noise reduction at sound sources.

[0102] The acoustic signal processing method and model training method of the present invention are described below with reference to the accompanying drawings.

[0103] It should be noted that the acoustic signal processing method of this embodiment can be executed by the acoustic signal processing device provided in this embodiment. This acoustic signal processing device can be an electronic device or can be configured within an electronic device. The electronic device can be any stationary or mobile computing device capable of data processing, such as a laptop, smartphone, wearable device, or other mobile computing device, or a desktop computer or other stationary computing device, or a server, or other types of computing devices, etc. This embodiment does not impose any limitations on this.

[0104] Similarly, the model training method of this embodiment can be executed by the model training device provided in this embodiment. This model training device can be an electronic device or can be configured within an electronic device.

[0105] Figure 1This is a schematic flowchart of an acoustic signal processing method provided in an embodiment of the present invention.

[0106] like Figure 1 As shown, the acoustic signal processing method includes the following steps:

[0107] Step 101: Obtain the multi-channel acoustic signal to be processed.

[0108] The multi-channel acoustic signal to be processed can be an acoustic signal acquired simultaneously by multiple sensors, such as an acoustic signal acquired by an array sensor.

[0109] In this embodiment, the sound signal processing device can acquire the multi-channel sound signal to be processed through various open, legal and compliant means. For example, the dialogue processing device can use multiple sensors to collect sound signals simultaneously to obtain the multi-channel sound signal to be processed, or it can acquire the multi-channel sound signal to be processed from other devices through network transmission or physical copying, or it can acquire the multi-channel sound signal to be processed through other open, legal and compliant means. This embodiment does not impose any restrictions on this.

[0110] It is understandable that, since the acquisition of sound signals is affected by reverberation, noise and other interference sources, the acquired multi-channel sound signals to be processed may contain reverberation, noise and interference. Therefore, it is necessary to perform dereverberation, noise reduction and separation on the acquired multi-channel sound signals to obtain reverberation-free and interference-free target sound signals.

[0111] Step 102: Based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, the beamforming neural network in the end-to-end model is used to determine the time-domain filter parameters corresponding to each channel.

[0112] In this embodiment, beamforming neural networks in an end-to-end model can be used to obtain the temporal filter parameters corresponding to each channel based on the spatial and temporal-frequency characteristics of the multi-channel acoustic signals. Optionally, the input to the beamforming neural network in the end-to-end model can be a multi-channel sound frame sequence, and the corresponding output is a frame-level sequence of beamforming temporal filter parameters corresponding to the target sound source. For example, suppose x i Let h represent the signal of channel i in the current time frame, with length L. Then, h can be obtained through a beamforming neural network. c i This is a beamforming time-domain filter parameter with a length of 2L+1, where L represents the length of the time frame, which can be understood as the number of sampling points contained in that time frame, c represents the sound source number, and h... c iThis can be understood as the time-domain filter parameters for channel i of sound source c. It's understandable that, since the multi-channel acoustic signal to be processed may contain multiple sound sources, the beamforming neural network can output the beamforming time-domain filter parameters corresponding to multiple sound sources.

[0113] Therefore, beamforming neural networks can be used to reduce reverberation and environmental noise in a sound field environment by utilizing information from multiple channels. It is understandable that, since the time-domain filter parameters corresponding to each channel can be determined using a beamforming neural network in an end-to-end model based on the spatial and temporal characteristics of the multi-channel acoustic signals, without prior knowledge of the spatial location of the sound source, it is suitable for reverberation reduction and noise reduction under complex and variable sound field conditions, exhibiting high applicability.

[0114] Step 103: Filter and sum the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal, and perform a short-time Fourier transform on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal.

[0115] In this embodiment, according to the filtering summation principle, the temporal filter parameters corresponding to each channel output by the beamforming neural network are convolved with and summed with the multi-channel acoustic signal to obtain the temporal waveform of the single-channel acoustic signal. Specifically, the temporal filter parameters (length 2L+1) can be convolved with and summed with the context window signal (length 3L, containing L sampling points of the current frame, L sampling points before it, and L sampling points after it) around the current time frame to obtain the temporal waveform of the single-channel acoustic signal. For example, suppose x i This represents the signal of channel i in the current time frame. This indicates that as the time index j increases, a signal of length L is extracted from the context window signal surrounding the current time frame using a sliding window method. It can also be denoted as... Given the time-domain filter parameters corresponding to each channel output by the beamforming neural network, the time-domain waveform of the target sound source, which is also the time-domain waveform of the single-channel sound signal to be obtained in this step, can be obtained using the following formula:

[0116]

[0117] Among them, y c This represents the time-domain waveform of the target sound source, which is also the time-domain waveform of the single-channel sound signal in this step; N represents the number of channels in the multi-channel signal. This is the convolution operator.

[0118] Understandably, based on the spatial and temporal characteristics of multi-channel acoustic signals, beamforming neural networks in end-to-end models can effectively reduce reverberation and environmental noise in the sound field environment, achieving denoising and noise reduction. They can also separate target sound sources from specific directions. However, the separated target sound sources may still contain other noise from the vicinity of the target sound source, such as the sound of equipment operation and other possible interference sound events (such as conversations, instantaneous impact sounds of equipment, etc.). Therefore, it is necessary to further separate the target sound source to separate the sound of machine operation and other possible interference sound events. In other words, it is also necessary to process the time-domain waveform of the obtained single-channel acoustic signal.

[0119] In this embodiment, after obtaining the time-domain waveform of the target sound source, that is, the time-domain waveform of the single-channel sound signal, a short-time Fourier transform can be performed on the obtained time-domain waveform to obtain a time-frequency diagram, so as to separate the sound source based on the time-frequency diagram.

[0120] Step 104: Based on the time-frequency diagram of the single-channel acoustic signal, determine the time-frequency mask of the query-based acoustic signal using a query-based sound source separation neural network in the end-to-end model.

[0121] In this embodiment, a query-based sound source separation neural network in an end-to-end model can be used to obtain a time-frequency mask based on the query sound signal, based on the time-frequency diagram of a single-channel sound signal. Thus, the sound source signal can be separated by estimating the time-frequency mask based on the query sound signal, thereby solving the sound source separation problem.

[0122] The process of separating sound source signals by estimating the time-frequency mask based on the query sound signal can be understood as follows: the sound emitted by a specific sound source has certain characteristics in time and frequency, such as being short and having a high frequency. At this time, the time-frequency mask is obtained by the query-based sound source separation neural network. Assuming that the time-frequency mask has a value of 1 in a short time range of high frequency bands, and a value of 0 in other time and frequency bands (an extreme example, in reality it should be a value between 0 and 1), then irrelevant signals can be set to 0 by multiplying the time-frequency plot of the single-channel sound signal with the time-frequency mask, thereby obtaining a target sound signal without reverberation or interference.

[0123] Therefore, a query-based sound source separation neural network can be used to separate the sound of the device operation from other possible interference sound events (such as conversations, momentary impacts of the device, etc.). Understandably, since the time-frequency mask of the query-based sound signal can be determined using the query-based sound source separation neural network in the end-to-end model based on the time-frequency diagram of the single-channel sound signal, it eliminates the need to know the time-frequency characteristics of the interference sound event in advance or make various assumptions about it, thus demonstrating good applicability in practical applications.

[0124] Step 105: Determine the target acoustic signal based on the time-frequency diagram and time-frequency mask of the single-channel acoustic signal.

[0125] In this embodiment, after obtaining the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask based on the query acoustic signal, the target acoustic signal can be determined based on these two data points. Specifically, the time-frequency diagram of the target acoustic signal can be obtained by multiplying the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask based on the query acoustic signal. Then, the target acoustic signal can be reconstructed by performing a short-time Fourier transform on the time-frequency diagram of the target acoustic signal, thus obtaining the time-frequency waveform of the target acoustic signal.

[0126] The acoustic signal processing method provided in this embodiment acquires the multi-channel acoustic signal to be processed. Based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, it uses a beamforming neural network in an end-to-end model to determine the time-domain filter parameters corresponding to each channel. Then, it filters and sums the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal. A short-time Fourier transform is then performed on the time-domain waveform of the single-channel acoustic signal to obtain its time-frequency diagram. Based on the time-frequency diagram of the single-channel acoustic signal, a query-based source separation neural network in the end-to-end model is used to determine the time-frequency mask based on the query acoustic signal. Finally, the target acoustic signal is determined based on the time-frequency diagram and the time-frequency mask of the single-channel acoustic signal. Therefore, under complex acoustic environment conditions, by using two neural networks in an end-to-end model to achieve reverberation and interference source separation, without pre-setting various constraints or assumptions, it has good applicability in practical applications and can effectively solve the problem of sound source noise reduction.

[0127] As can be seen from the above analysis, in this embodiment of the invention, the time-domain filter parameters corresponding to each channel can be determined using a beamforming neural network in an end-to-end model based on the spatial and time-frequency characteristics of the multi-channel acoustic signal. To clearly illustrate how the beamforming neural network in the end-to-end model is used to determine the time-domain filter parameters corresponding to each channel in this invention, this embodiment provides another acoustic signal processing method. Figure 2 This is a schematic flowchart of another acoustic signal processing method provided in an embodiment of the present invention.

[0128] like Figure 2 As shown, the acoustic signal processing method may include the following steps:

[0129] Step 201: Obtain the multi-channel acoustic signal to be processed.

[0130] It should be noted that the execution process of this step can refer to the execution process of step 101 in the previous embodiment, and the principle is the same, so it will not be repeated here.

[0131] Step 202: For any channel of the multi-channel acoustic signal, determine the inter-channel characteristics and channel-specific characteristics of the channel based on the spatial and time-frequency characteristics of the channel.

[0132] In this embodiment, the inter-channel characteristics and channel-specific characteristics of any channel of a multi-channel acoustic signal can be determined based on the spatial and temporal frequency characteristics of that channel. Thus, the inter-channel characteristics and channel-specific characteristics of each channel in the multi-channel acoustic signal can be obtained.

[0133] In one possible implementation of this embodiment, for any channel of the multi-channel acoustic signal, a frame-level normalized cross-correlation coefficient can be used as the inter-channel feature of that channel, as shown in the following formula:

[0134]

[0135] in, This represents the inter-channel characteristics of channel i; i represents the channel number; j represents... The time index of the signal; Indicates the context window signal of channel i around the current time frame; x 1 This indicates the signal of reference channel 1 in the current time frame; L represents the length of the time frame, which refers to the number of sampling points contained in the time frame.

[0136] It should be noted that, In this context, i represents the channel number, and j represents... The time index of the signal, as j increases, A signal segment of length L is extracted from the context window signal surrounding the current time frame using a sliding window method, thereby The length is 3L. It is the cosine similarity between the extracted signal and the current time frame signal of the reference channel.

[0137] Simultaneously, a linear transformation layer can be used to convert the L-dimensional features of this channel in the current time frame into K-dimensional features, obtaining the channel-specific features of this channel. The specific formula is as follows:

[0138] R i =x i U

[0139] Among them, R i Indicates the channel-specific characteristics of channel i; x i L represents the signal of channel i in the current time frame, with a length of L; U represents the training transformation matrix, which is equivalent to a fully connected neural network layer.

[0140] Step 203: Input the inter-channel features and channel-specific features of each channel into the beamforming neural network in the end-to-end model to obtain the time-domain filter parameters corresponding to each channel.

[0141] In this embodiment, after obtaining the inter-channel features and channel-specific features of each channel, the inter-channel features and channel-specific features of each channel can be input into the beamforming neural network in the end-to-end model to obtain the time-domain filter parameters corresponding to each channel.

[0142] To clearly illustrate how beamforming neural networks determine the time-domain filter parameters for each channel based on the inter-channel and channel-specific characteristics, this embodiment provides, for example... Figure 3 The diagram shows the structure of a beamforming neural network.

[0143] like Figure 3 As shown, the beamforming neural network includes an encoder layer, a temporal convolutional network layer, a 1×1 convolutional layer, and a gated output layer. Specifically, for each channel, the inter-channel features and channel-specific features are concatenated and input into the encoder layer of the beamforming neural network. After transformation by the encoder layer, temporal convolutional network layer, and 1×1 convolutional layer, at least one set of feature data is output, as shown in the following formula:

[0144]

[0145] in, This represents the output of a 1×1 convolutional layer; i represents the channel number; c represents the sound source number; C represents the total number of sound sources; R i Indicates the channel-specific characteristics of channel i; f i This represents the expression for j = 1, 2, ..., 2L+1. The vector formed; This represents a function mapping that includes an encoder layer, a temporal convolutional network layer, and a 1×1 convolutional layer.

[0146] Then, at least one set of output feature data is input into the gated output layer of the beamforming neural network to obtain the time-domain filter parameters corresponding to each channel, as shown in the following formula:

[0147]

[0148] in, Let represent the time-domain filter parameters for channel i for sound source c; W represents the weighting coefficient of the hyperbolic tangent function; b represents the deviation coefficient of the hyperbolic tangent function; V represents the weighting coefficient of the sigmoid function; q represents the deviation coefficient of the sigmoid function; and Θ represents the Hadamard product.

[0149] It should be noted that the encoder layer in a beamforming neural network can transform the input into an N-dimensional feature space through one-dimensional convolution operations and nonlinear activation functions.

[0150] To clearly illustrate the structure of the temporal convolutional network layer in the beamforming neural network, this embodiment also provides, as follows: Figure 4 The diagram shows the structure of a temporal convolutional network layer.

[0151] like Figure 4 As shown, each layer in the temporal convolutional network consists of one-dimensional convolutional blocks with progressively increasing dilation factors. These dilation factors grow exponentially to ensure a sufficiently large temporal context window is included. M convolutional blocks with dilation factors of 1, 2, ..., 2M-1 are repeated B times. Residual connections are added to each one-dimensional convolutional block, allowing the network to transfer information across layers.

[0152] A transformation-average-concatenation module is added after each one-dimensional convolutional block in the temporal convolutional network. Through inter-channel parameter sharing and global pooling, it adapts to changes in the number and arrangement of microphones while utilizing all microphone information. The transformation-average-concatenation module first transforms the features of each channel through a sub-module consisting of a fully connected layer and a non-linear activation function; this sub-module is shared across all channels. Next, the outputs of all channels are globally averaged and pooled before being input into another sub-module consisting of a fully connected layer and a non-linear activation function. The outputs of the first two modules are concatenated and input into a third sub-module consisting of a fully connected layer and a non-linear activation function to obtain channel-specific outputs. Residual connections are added between the input and output layers.

[0153] It should be noted that the output of the temporal convolutional network layer can be converted into C groups of N-dimensional features through a 1×1 convolutional layer, where C is the number of sound sources. Therefore, by inputting the C groups of K-dimensional features into a gated output layer, beamforming temporal filter parameters of length 2L+1 can be obtained.

[0154] Step 204: Filter and sum the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal, and perform a short-time Fourier transform on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal.

[0155] Step 205: Based on the time-frequency diagram of the single-channel acoustic signal, the query-based sound source separation neural network in the end-to-end model is used to determine the time-frequency mask of the query acoustic signal.

[0156] Step 206: Determine the target acoustic signal based on the time-frequency diagram and time-frequency mask of the single-channel acoustic signal.

[0157] It should be noted that the execution process of steps 204-206 can refer to the execution process of steps 103-105 in the previous embodiment, and the principle is the same, so it will not be repeated here.

[0158] The acoustic signal processing method provided in this embodiment determines the inter-channel characteristics and channel-specific characteristics of any channel of a multi-channel acoustic signal based on the channel's spatial and time-frequency features. These inter-channel and channel-specific characteristics are then input into a beamforming neural network in an end-to-end model to obtain the corresponding time-domain filter parameters for each channel. Thus, it is possible to use a beamforming neural network to determine the corresponding time-domain filter parameters for each channel based on its inter-channel and channel-specific characteristics.

[0159] As can be seen from the above analysis, in this embodiment of the invention, a query-based sound source separation neural network in an end-to-end model can be used to determine the time-frequency mask based on the query sound signal according to the time-frequency diagram of the single-channel sound signal. To clearly illustrate how the query-based sound source separation neural network is used to determine the time-frequency mask based on the query sound signal in this invention, this embodiment provides another sound signal processing method. Figure 4 This is a schematic flowchart of another acoustic signal processing method provided in an embodiment of the present invention.

[0160] like Figure 5 As shown, the acoustic signal processing method may include the following steps:

[0161] Step 501: Obtain the multi-channel acoustic signal to be processed.

[0162] Step 502: Based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, the beamforming neural network in the end-to-end model is used to determine the time-domain filter parameters corresponding to each channel.

[0163] Step 503: Filter and sum the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal, and perform a short-time Fourier transform on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal.

[0164] It should be noted that the execution process of steps 501-503 can refer to the execution process of steps 101-103 in the previous embodiment, and the principle is the same, so it will not be repeated here.

[0165] Step 504: Use a hierarchical self-attention model based on a moving window to extract low-dimensional features of the query acoustic signal.

[0166] In this embodiment, the query sound signal is a pre-acquired single-channel, interference-free sound signal.

[0167] In this implementation, a hierarchical self-attention model based on a moving window can be used to extract low-dimensional features of the query sound signal. These low-dimensional features are then input into the query-based sound source separation neural network in the end-to-end model and converted into the feature dimensions corresponding to each layer in the query-based sound source separation neural network.

[0168] Step 505: Input the time-frequency map and low-dimensional features into the query-based sound source separation neural network in the end-to-end model to obtain the time-frequency mask based on the query sound signal.

[0169] In this embodiment, the time-frequency map and low-dimensional features of a single-channel acoustic signal can be input into a query-based sound source separation neural network in an end-to-end model to obtain a time-frequency mask based on the query acoustic signal.

[0170] To clearly illustrate how the query-based sound source separation neural network obtains the time-frequency mask of the query-based sound signal based on the time-frequency plot and low-dimensional features of a single-channel sound signal, this embodiment provides, as follows: Figure 6 The diagram shows the structure of a query-based neural network for separating sound sources.

[0171] like Figure 6 As shown, the query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture, including at least one downsampling layer, at least one upsampling layer, and one convolutional layer. Specifically, the low-dimensional features of the query sound signal can be input into at least one downsampling layer, at least one upsampling layer, and one convolutional layer, respectively, to obtain the feature dimensions corresponding to at least one downsampling layer, at least one upsampling layer, and one convolutional layer. Thus, the time-frequency map of the single-channel sound signal is input into the query-based sound source separation neural network in the end-to-end model, and based on the time-frequency map of the single-channel sound signal and the feature dimensions corresponding to each layer, a time-frequency mask based on the query sound signal is output.

[0172] It should be noted that the query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture. Each layer of the U-shaped convolutional neural network is a two-dimensional convolutional layer, and each layer has its corresponding feature map. This feature map is obtained by inputting the output of the previous layer into the current layer for convolution. The output of each layer is obtained by adding the corresponding feature dimension of that layer to its corresponding feature map. Thus, the first half of the U-shaped convolutional neural network can continuously downsample and abstract through convolution and pooling. The second half continuously performs convolution and upsampling, concatenating the feature maps of the upsampled layers with the corresponding downsampled layers. This concatenated feature map is then subjected to further convolution and upsampling until an output feature map with the same size as the input signal is obtained, resulting in the time-frequency mask based on the query sound signal.

[0173] Step 506: Determine the target acoustic signal based on the time-frequency diagram and time-frequency mask of the single-channel acoustic signal.

[0174] It should be noted that the execution process of this step can refer to the execution process of step 101 in the previous embodiment, and the principle is the same, so it will not be repeated here.

[0175] The acoustic signal processing method provided in this embodiment extracts low-dimensional features of a single-channel acoustic signal using a hierarchical self-attention model based on a moving window. The time-frequency plot and low-dimensional features are then input into a query-based sound source separation neural network in an end-to-end model to obtain a time-frequency mask based on the query acoustic signal. Thus, a query-based sound source separation neural network can be used to determine a time-frequency mask based on the low-dimensional features and time-frequency plot of a single-channel acoustic signal.

[0176] To illustrate the above embodiments more clearly, examples are given below.

[0177] like Figure 7 As shown, multi-channel acoustic signals are input into a beamforming neural network and filtered and summed to obtain a single-channel acoustic signal with reduced noise and reverberation. The single-channel acoustic signal with reduced noise and reverberation and the query acoustic signal (a pre-acquired single-channel, interference-free sound signal) are then input into a query-based sound source separation neural network to obtain a single-channel acoustic signal with further separation and noise reduction. Thus, by using two neural networks in an end-to-end model, sound source separation for reverberation and interference can be achieved, effectively solving the sound source noise reduction problem.

[0178] As can be seen from the above analysis, the embodiments of the present invention require the use of an end-to-end model. It is understood that the end-to-end model used in the embodiments of the present invention is a pre-trained end-to-end model. To clearly illustrate the training process of the end-to-end model, the embodiments of the present invention provide a possible implementation of the model training method. Figure 8 This is a schematic flowchart of a model training method provided in an embodiment of the present invention.

[0179] like Figure 8 As shown, the model training method may include the following steps:

[0180] Step 801: Acquire a single-channel acoustic signal and generate a multi-channel training acoustic signal based on the single-channel acoustic signal.

[0181] In this embodiment, a single-channel acoustic signal can be pre-acquired using a unidirectional microphone under conditions where there are no other interfering sound sources. Then, an acoustic simulation method can be used to generate a multi-channel acoustic signal containing reverberation, noise, and interference based on the single-channel acoustic signal for use in training the end-to-end model.

[0182] Alternatively, acoustic simulation can be used to simulate the reverberant environment. The acquired single-channel reverberant sound signal is converted into a multi-channel sound signal with reverberation. White noise or interference sound events such as conversation or metal impact are added at random spatial locations in the simulated sound field with different signal-to-noise ratios. Finally, a multi-channel sound signal with reverberation, noise and interference is generated as the input to the end-to-end model.

[0183] Step 802: Based on the multi-channel training acoustic signal, the time-frequency waveform of the target acoustic signal is determined using an end-to-end model.

[0184] In this embodiment, the time-frequency waveform of the target acoustic signal can be determined using an end-to-end model based on multi-channel training acoustic signals. Optionally, the sound frame sequence of the multi-channel training acoustic signals can be first input into the beamforming neural network in the end-to-end model, and a single-channel acoustic signal can be obtained through filtering and weighting. Then, a hierarchical self-attention model with a moving window is used to extract low-dimensional features from the pre-acquired single-channel acoustic signals. It should be noted that, to ensure better training results, low-dimensional features from multiple pre-acquired single-channel acoustic signals can be extracted, and their low-dimensional features can be averaged to obtain the target low-dimensional features. The obtained target low-dimensional features and the single-channel acoustic signals are then input into a pre-trained sound source separation network to separate the target acoustic signal and obtain the time-frequency waveform of the target signal.

[0185] Alternatively, the target low-dimensional features can be obtained using the following formula:

[0186]

[0187] Among them, e q Represents the low-dimensional features of the target; N represents the total number of single-channel acoustic signals; i represents the single-channel acoustic signal number; This represents the low-dimensional features of the i-th single-channel acoustic signal.

[0188] Step 803: Use the average absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal as the loss function of the end-to-end model, and train the end-to-end model based on the loss function.

[0189] In this embodiment, the average absolute error between the time-frequency waveform of the target acoustic signal determined by the end-to-end model and the time-domain waveform of the pre-acquired single-channel acoustic signal can be used as the loss function of the end-to-end model, thereby enabling the training of the end-to-end model based on this loss function. Specifically, the loss function of the end-to-end model can be used as the objective function of the model. By differentiating the parameters according to the objective function in each parameter iteration process, the parameters that minimize the objective function can be obtained, thereby enabling the training of the end-to-end model.

[0190] Alternatively, the loss function of the end-to-end model can be expressed by the following formula:

[0191]

[0192] Where MAE() represents the mean absolute error function (MAE: mean absolute error); N represents the number of sampling points in the pre-acquired single-channel acoustic signal; Y represents the time-domain waveform of the pre-acquired single-channel acoustic signal; This represents the time-frequency waveform of the target acoustic signal determined by the end-to-end model.

[0193] The model training method provided in this embodiment acquires a single-channel acoustic signal and generates a multi-channel training acoustic signal based on the single-channel acoustic signal. Then, based on the multi-channel training acoustic signal, an end-to-end model is used to determine the time-frequency waveform of the target acoustic signal. The mean absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal is used as the loss function of the end-to-end model, and the end-to-end model is trained based on this loss function. Therefore, by training the end-to-end model, the trained end-to-end model can achieve the goals of dereverberation, noise reduction, and separation, thereby effectively enhancing the target acoustic signal and suppressing interference signals.

[0194] With the above Figures 1 to 7 Corresponding to the acoustic signal processing method provided in the embodiments, the present invention also provides an acoustic signal processing apparatus. Because the acoustic signal processing apparatus provided in the embodiments of the present invention is similar to the one described above... Figures 1 to 7 The sound signal processing method provided in the embodiments corresponds to the sound signal processing device provided in the embodiments of the present invention, and will not be described in detail in the embodiments of the present invention.

[0195] Figure 9 This is a schematic diagram of the structure of an acoustic signal processing device provided in an embodiment of the present invention.

[0196] like Figure 9 As shown, the acoustic signal processing device includes: an acquisition module 91, a first determination module 92, a first processing module 93, a second determination module 94, and a third determination module 95.

[0197] Acquisition module 91 is used to acquire the multi-channel acoustic signal to be processed;

[0198] The first determining module 92 is used to determine the time-domain filter parameters corresponding to each channel based on the spatial and time-frequency characteristics of the multi-channel acoustic signal and using a beamforming neural network in the end-to-end model.

[0199] The first processing module 93 is used to filter and sum the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal, and to perform a short-time Fourier transform on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal.

[0200] The second determining module 94 is used to determine the time-frequency mask of the query-based acoustic signal based on the time-frequency diagram of the single-channel acoustic signal using a query-based sound source separation neural network in the end-to-end model.

[0201] The third determining module 95 is used to determine the target acoustic signal based on the time-frequency diagram and time-frequency mask of the single-channel acoustic signal.

[0202] Furthermore, in one possible implementation of this invention, the first determining module 92 includes:

[0203] The determination unit is used to determine the inter-channel characteristics and channel-specific characteristics of any channel of a multi-channel acoustic signal based on the spatial and temporal-frequency characteristics of the channel.

[0204] The first processing unit is used to input the inter-channel features and channel-specific features of each channel into the beamforming neural network in the end-to-end model to obtain the time-domain filter parameters corresponding to each channel.

[0205] Furthermore, in one possible implementation of this invention, the determining unit is further configured to:

[0206] For any channel of a multi-channel acoustic signal, the inter-channel characteristics are determined using the following formula:

[0207]

[0208] in, This represents the inter-channel characteristics of channel i; i represents the channel number; j represents... The time index of the signal; Indicates the context window signal of channel i around the current time frame; x 1 This indicates the signal of reference channel 1 in the current time frame; L represents the length of the time frame.

[0209] The L-dimensional features of a channel in the current time frame are converted into K-dimensional features using the following formula to obtain the channel-specific features:

[0210] R i =x i U

[0211] Among them, R i Indicates the channel-specific characteristics of channel i; x iU represents the signal of channel i in the current time frame; U represents the training transformation matrix.

[0212] Furthermore, in one possible implementation of this invention, the beamforming neural network includes an encoder layer, a temporal convolutional network layer, a 1×1 convolutional layer, and a gated output layer. The first processing unit is further configured to:

[0213] For any given channel, the channel-specific features and channel-specific characteristics are concatenated to obtain the input data;

[0214] The input data is fed into the encoder layer, and after passing through the encoder layer, the temporal convolutional network layer and the 1×1 convolutional layer, at least one set of feature data is output.

[0215] At least one set of feature data is input into the gated output layer to obtain the time-domain filter parameters corresponding to the channel.

[0216] Furthermore, in one possible implementation of this invention, the second determining module 94 includes:

[0217] The extraction unit is used to extract low-dimensional features of a single-channel acoustic signal using a hierarchical self-attention model based on a moving window.

[0218] The second processing unit is used to input the time-frequency diagram and low-dimensional features into the query-based sound source separation neural network in the end-to-end model to obtain the time-frequency mask based on the query sound signal.

[0219] Furthermore, in one possible implementation of this invention, the query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture, including at least one downsampling layer, at least one upsampling layer, and one convolutional layer. The second processing unit is further configured to:

[0220] The low-dimensional features are input into at least one downsampling layer, at least one upsampling layer and a convolutional layer respectively to obtain the feature dimensions corresponding to at least one downsampling layer, at least one upsampling layer and a convolutional layer;

[0221] The time-frequency map is input into the query-based sound source separation neural network in the end-to-end model, and the time-frequency mask based on the query sound signal is output based on the time-frequency map and the feature dimensions corresponding to each layer.

[0222] Furthermore, in one possible implementation of this invention, the third determining module 95 is further configured to:

[0223] Multiply the time-frequency plot of the single-channel acoustic signal with the time-frequency mask to obtain the time-frequency plot of the target acoustic signal;

[0224] Perform a short-time inverse Fourier transform on the time-frequency diagram of the target acoustic signal to obtain the time-frequency waveform of the target acoustic signal.

[0225] The acoustic signal processing device provided in this embodiment acquires the multi-channel acoustic signal to be processed, and based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, uses a beamforming neural network in an end-to-end model to determine the time-domain filter parameters corresponding to each channel. Then, it filters and sums the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal. A short-time Fourier transform is then performed on the time-domain waveform of the single-channel acoustic signal to obtain its time-frequency diagram. Based on the time-frequency diagram of the single-channel acoustic signal, a query-based source separation neural network in an end-to-end model is used to determine the time-frequency mask based on the query acoustic signal. Finally, the target acoustic signal is determined based on the time-frequency diagram and the time-frequency mask of the single-channel acoustic signal. Therefore, under complex acoustic environment conditions, by using two neural networks in an end-to-end model to achieve reverberation and interference source separation, without pre-setting various constraints or assumptions, it has good applicability in practical applications and can effectively solve the problem of sound source noise reduction.

[0226] With the above Figure 8 Corresponding to the model training method provided in the embodiments, the present invention also provides a model training apparatus. Because the model training apparatus provided in the embodiments of the present invention is similar to the one described above... Figure 8 The model training method provided in the embodiments corresponds to the model training device provided in the embodiments of the present invention, and will not be described in detail in the embodiments of the present invention.

[0227] Figure 10 This is a schematic diagram of a model training device provided in an embodiment of the present invention.

[0228] like Figure 10 As shown, the model training device includes: a second processing module 11, a fourth determining module 12, and a training module 13.

[0229] The second processing module 11 is used to acquire single-channel acoustic signals and generate multi-channel training acoustic signals based on the single-channel acoustic signals.

[0230] The fourth determining module 12 is used to determine the time-frequency waveform of the target acoustic signal based on the multi-channel training acoustic signal and using an end-to-end model;

[0231] Training module 13 is used to use the average absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal as the loss function of the end-to-end model, so as to train the end-to-end model based on the loss function.

[0232] The model training device provided in this embodiment acquires a single-channel acoustic signal and generates a multi-channel training acoustic signal based on the single-channel acoustic signal. Then, based on the multi-channel training acoustic signal, an end-to-end model is used to determine the time-frequency waveform of the target acoustic signal. The mean absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal is used as the loss function of the end-to-end model, and the end-to-end model is trained based on this loss function. Therefore, by training the end-to-end model, the trained end-to-end model can achieve the goals of dereverberation, noise reduction, and separation, thereby effectively enhancing the target acoustic signal and suppressing interference signals.

[0233] To implement the above embodiments, the present invention also proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the acoustic signal processing method or model training method proposed in any of the above embodiments of the present invention.

[0234] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. It should be noted that... Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0235] like Figure 11 As shown, the electronic device may include: a housing 1101, a processor 1102, a memory 1103, a circuit board 1104, and a power supply circuit 1105. The circuit board 1104 is disposed inside the space enclosed by the housing 1101, and the processor 1102 and the memory 1103 are disposed on the circuit board 1104. The power supply circuit 1105 is used to supply power to various circuits or devices of the electronic device. The memory 1103 is used to store executable program code. The processor 1102 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 1103, and is used to execute the sound signal processing method or model training method proposed in any of the above embodiments of the present invention.

[0236] For details on the specific execution process of the above steps by processor 1102, and the steps further executed by processor 1102 through running executable program code, please refer to the present invention. Figure 1-8 The description of the illustrated embodiments will not be repeated here.

[0237] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the acoustic signal processing method or model training method proposed in any of the above embodiments of the present invention.

[0238] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the acoustic signal processing method or model training method proposed in any of the above embodiments of the present invention.

[0239] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0240] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0241] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0242] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0243] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0244] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0245] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0246] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for processing acoustic signals, characterized in that, include: Acquire the multi-channel acoustic signal to be processed; Based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, the beamforming neural network in the end-to-end model is used to determine the time-domain filter parameters corresponding to each channel. The time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal are filtered and summed to obtain the time-domain waveform of the single-channel acoustic signal, and a short-time Fourier transform is performed on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal. Based on the time-frequency diagram of the single-channel acoustic signal, a query-based sound source separation neural network in the end-to-end model is used to determine the time-frequency mask of the query-based acoustic signal. The target acoustic signal is determined based on the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask. Based on the spatial and time-frequency characteristics of the multi-channel acoustic signals, the beamforming neural network in the end-to-end model is used to determine the time-domain filter parameters corresponding to each channel, including: For any channel of the multi-channel acoustic signal, based on the spatial and time-frequency characteristics of the channel, determine the inter-channel characteristics and channel-specific characteristics of the channel; The inter-channel features and channel-specific features of each channel are input into the beamforming neural network in the end-to-end model to obtain the time-domain filter parameters corresponding to each channel; the determination of the inter-channel features and channel-specific features of any channel of the multi-channel acoustic signal based on the spatial and time-frequency features of the channel includes: For any channel of the multi-channel acoustic signal, the inter-channel characteristics of the channel are determined using the following formula: ，（ ） in, This represents the inter-channel characteristics of channel i; i represents the channel number; j represents... The time index of the signal; , This indicates the context window signal for channel i around the current time frame; This indicates the signal of reference channel 1 in the current time frame; L represents the length of the time frame. The L-dimensional features of the channel in the current time frame are converted into K-dimensional features using the following formula to obtain the channel-specific features: in, This represents the channel-specific characteristics of channel i; U represents the signal of channel i in the current time frame; U represents the training transformation matrix.

2. The method according to claim 1, characterized in that, The beamforming neural network includes an encoder layer, a temporal convolutional network layer, a 1×1 convolutional layer, and a gated output layer. The process of inputting the inter-channel features and channel-specific features of each channel into the beamforming neural network in the end-to-end model to obtain the temporal filter parameters corresponding to each channel includes: For any given channel, the channel-specific features and channel-specific characteristics of the channel are concatenated to obtain the input data; The input data is fed into the encoder layer, and after passing through the encoder layer, the temporal convolutional network layer and the 1×1 convolutional layer, at least one set of feature data is output. The at least one set of feature data is input to the gated output layer to obtain the time-domain filter parameters corresponding to the channel.

3. The method according to claim 1, characterized in that, The step of determining the time-frequency mask based on the query-based acoustic signal using a query-based sound source separation neural network in an end-to-end model, based on the time-frequency diagram of the single-channel acoustic signal, includes: A hierarchical self-attention model based on a moving window is used to extract low-dimensional features of the query sound signal; wherein the query sound signal is a pre-acquired single-channel, interference-free sound signal; The time-frequency diagram and the low-dimensional features are input into a query-based sound source separation neural network in the end-to-end model to obtain a time-frequency mask based on the query sound signal.

4. The method according to claim 3, characterized in that, The query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture, including at least one downsampling layer, at least one upsampling layer, and one convolutional layer. The step of inputting the time-frequency map and the low-dimensional features into the query-based sound source separation neural network in the end-to-end model to obtain a time-frequency mask based on the query sound signal includes: The low-dimensional features are respectively input into at least one downsampling layer, at least one upsampling layer and a convolutional layer to obtain the feature dimensions corresponding to the at least one downsampling layer, at least one upsampling layer and a convolutional layer; The time-frequency graph is input into a query-based sound source separation neural network in the end-to-end model to output a time-frequency mask based on the query sound signal, based on the time-frequency graph and the feature dimensions corresponding to each layer.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the target acoustic signal based on the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask includes... Multiply the time-frequency diagram of the single-channel acoustic signal with the time-frequency mask to obtain the time-frequency diagram of the target acoustic signal; A short-time inverse Fourier transform is performed on the time-frequency diagram of the target acoustic signal to obtain the time-frequency waveform of the target acoustic signal.

6. A model training method, characterized in that, include: Acquire single-channel acoustic signals and generate multi-channel training acoustic signals based on the single-channel acoustic signals; Based on the multi-channel training acoustic signal, the time-frequency waveform of the target acoustic signal is determined using an end-to-end model; The average absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal is used as the loss function of the end-to-end model, and the end-to-end model is trained based on the loss function, wherein the end-to-end model is applied in the acoustic signal processing method as claimed in claim 1.

7. A sound signal processing device, characterized in that, include: The acquisition module is used to acquire the multi-channel acoustic signal to be processed; The first determining module is used to determine the time-domain filter parameters corresponding to each channel based on the spatial and time-frequency characteristics of the multi-channel acoustic signal, using a beamforming neural network in an end-to-end model. The first processing module is used to filter and sum the time-domain filter parameters corresponding to each channel and the multi-channel acoustic signal to obtain the time-domain waveform of the single-channel acoustic signal, and to perform a short-time Fourier transform on the time-domain waveform of the single-channel acoustic signal to obtain the time-frequency diagram of the single-channel acoustic signal. The second determining module is used to determine the time-frequency mask of the query-based acoustic signal based on the time-frequency diagram of the single-channel acoustic signal using a query-based sound source separation neural network in an end-to-end model. The third determining module is used to determine the target acoustic signal based on the time-frequency diagram of the single-channel acoustic signal and the time-frequency mask; The first determining module includes: The determining unit is used to determine the inter-channel characteristics and channel-specific characteristics of any channel of the multi-channel acoustic signal based on the spatial and time-frequency characteristics of the channel. The first processing unit is configured to input the inter-channel features and channel-specific features of each channel into the beamforming neural network in the end-to-end model to obtain the temporal filter parameters corresponding to each channel; the determining unit is further configured to: For any channel of the multi-channel acoustic signal, the inter-channel characteristics of the channel are determined using the following formula: ，（ ） in, This represents the inter-channel characteristics of channel i; i represents the channel number; j represents... The time index of the signal; , This indicates the context window signal for channel i around the current time frame; This indicates the signal of reference channel 1 in the current time frame; L represents the length of the time frame. The L-dimensional features of the channel in the current time frame are converted into K-dimensional features using the following formula to obtain the channel-specific features: in, This represents the channel-specific characteristics of channel i; U represents the signal of channel i in the current time frame; U represents the training transformation matrix.

8. The apparatus according to claim 7, characterized in that, The beamforming neural network includes an encoder layer, a temporal convolutional network layer, a 1×1 convolutional layer, and a gated output layer. The first processing unit is further configured to: For any given channel, the channel-specific features and channel-specific characteristics of the channel are concatenated to obtain the input data; The input data is fed into the encoder layer, and after passing through the encoder layer, the temporal convolutional network layer and the 1×1 convolutional layer, at least one set of feature data is output. The at least one set of feature data is input to the gated output layer to obtain the time-domain filter parameters corresponding to the channel.

9. The apparatus according to claim 7, characterized in that, The second determining module includes: The extraction unit is used to extract low-dimensional features of the query sound signal using a hierarchical self-attention model based on a moving window; wherein the query sound signal is a pre-acquired single-channel, interference-free sound signal; The second processing unit is used to input the time-frequency diagram and the low-dimensional features into the query-based sound source separation neural network in the end-to-end model to obtain a time-frequency mask based on the query sound signal.

10. The apparatus according to claim 9, characterized in that, The query-based sound source separation neural network adopts a U-shaped convolutional neural network architecture, including at least one downsampling layer, at least one upsampling layer, and one convolutional layer. The second processing unit is further used for: The low-dimensional features are respectively input into at least one downsampling layer, at least one upsampling layer and a convolutional layer to obtain the feature dimensions corresponding to the at least one downsampling layer, at least one upsampling layer and a convolutional layer; The time-frequency graph is input into a query-based sound source separation neural network in the end-to-end model to output a time-frequency mask based on the query sound signal, based on the time-frequency graph and the feature dimensions corresponding to each layer.

11. The apparatus according to any one of claims 7-10, characterized in that, The third determining module is also used for Multiply the time-frequency diagram of the single-channel acoustic signal with the time-frequency mask to obtain the time-frequency diagram of the target acoustic signal; A short-time inverse Fourier transform is performed on the time-frequency diagram of the target acoustic signal to obtain the time-frequency waveform of the target acoustic signal.

12. A model training device, characterized in that, include: The second processing module is used to acquire single-channel acoustic signals and generate multi-channel training acoustic signals based on the single-channel acoustic signals. The fourth determining module is used to determine the time-frequency waveform of the target acoustic signal based on the multi-channel training acoustic signal using an end-to-end model; The training module is used to use the average absolute error between the time-frequency waveform of the target acoustic signal and the time-domain waveform of the single-channel acoustic signal as the loss function of the end-to-end model, so as to train the end-to-end model based on the loss function, wherein the end-to-end model is applied in the acoustic signal processing device as claimed in claim 7.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5, or to perform the method of claim 6.

14. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method of any one of claims 1-5, or to perform the method of claim 6.

15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5, or implements the method of claim 6.

Images