Self-adaptive speech enhancement method and device guided by acoustic environment perception

Abstract

The invention discloses a self-adaptive speech enhancement method and device guided by acoustic environment perception, and relates to the technical field of speech signal processing. The method comprises the following steps: carrying out time-frequency transformation on a multi-channel voice signal received by a microphone array to obtain a complex frequency spectrum; performing spatial sampling on the complex frequency spectrum, and obtaining a space-time frequency three-dimensional feature tensor through a group of fixed beam forming filters pointing to different spatial directions; inputting the space-time frequency three-dimensional feature tensor into an acoustic environment sensing network based on a Transform architecture, and extracting the last layer of network output of the acoustic environment sensing network as an acoustic environment coding vector; splicing the complex frequency spectrum and the acoustic environment coding vector, and inputting the spliced frequency spectrum and the spliced acoustic environment coding vector into a speech enhancement network to obtain a preliminarily enhanced speech spectrum; and splicing the preliminarily enhanced speech spectrum and the acoustic environment coding vector, and inputting the spliced speech spectrum and the spliced acoustic environment coding vector into a U-Net diffusion generation model to generate a final enhanced speech spectrum. According to the invention, the speech enhancement effect can be obviously improved.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to an acoustic environment-aware adaptive speech enhancement method and apparatus. Background Technology

[0002] With the widespread adoption of far-field voice interaction, microphone array voice enhancement has become a mainstream method for suppressing noise and reverberation, commonly used in applications such as conferencing systems and smart home voice interaction. However, noise and reverberation in practical applications affect the quality of the acquired voice. Therefore, voice enhancement has become a crucial preprocessing technique in voice interaction systems. Traditional solutions typically include a two-stage structure of "beamforming + post-filtering."

[0003] (1) Beamforming (such as MVDR, GSC) uses spatial information to enhance speech in the direction of the target;

[0004] (2) Post-filtering (such as Wiener filtering, deep time-frequency masking) further suppresses residual noise.

[0005] In recent years, deep learning-based speech enhancement methods (such as Conv-TasNet and DCCRN) and diffusion models have achieved better noise reduction effects in single-channel scenarios. However, in multi-channel far-field environments, beamformers are still sensitive to sound source orientation errors and reverberation, resulting in distortion of the target speech and poor speech enhancement effects. Summary of the Invention

[0006] To address the technical problems of target speech distortion and poor speech enhancement effects in existing technologies, this invention provides an acoustic environment-aware guided adaptive speech enhancement method and apparatus. The technical solution is as follows:

[0007] On the one hand, an acoustic environment-aware guided adaptive speech enhancement method is provided, which is implemented by an acoustic environment-aware guided adaptive speech enhancement device, and the method includes:

[0008] S1. Perform time-frequency transformation on the multi-channel speech signal received by the microphone array to obtain a complex spectrum;

[0009] S2. Spatial sampling is performed on the complex spectrum, and a set of fixed beamforming filters pointing in different spatial directions is used to obtain a spatial-time-frequency three-dimensional feature tensor.

[0010] S3. Input the spatial-temporal three-dimensional feature tensor into the acoustic environment perception network based on the Transformer architecture, and extract the output of the last layer of the acoustic environment perception network as the acoustic environment encoding vector.

[0011] S4. The complex spectrum is concatenated with the acoustic environment coding vector to form a first joint feature, which is then input into the speech enhancement network to obtain a preliminary enhanced speech spectrum;

[0012] S5. The preliminary enhanced speech spectrum is concatenated with the acoustic environment coding vector to form a second joint feature, which is then input into the U-Net diffusion generation model to generate the final enhanced speech spectrum.

[0013] On the other hand, an acoustic environment-aware guided adaptive speech enhancement device is provided, which is applied to an acoustic environment-aware guided adaptive speech enhancement method. The device includes:

[0014] The time-frequency conversion unit is used to perform time-frequency conversion on the multi-channel speech signal received by the microphone array to obtain a complex spectrum.

[0015] The sampling and filtering unit is used to spatially sample the complex spectrum and obtain a space-time-frequency three-dimensional feature tensor by passing it through a set of fixed beamforming filters pointing in different spatial directions.

[0016] The feature extraction unit is used to input the spatial-temporal three-dimensional feature tensor into the acoustic environment perception network based on the Transformer architecture, and extract the output of the last layer of the acoustic environment perception network as the acoustic environment encoding vector.

[0017] The speech enhancement unit is used to concatenate the complex spectrum with the acoustic environment coding vector to form a first joint feature, which is then input into the speech enhancement network to obtain a preliminary enhanced speech spectrum.

[0018] The diffusion generation unit is used to concatenate the preliminary enhanced speech spectrum with the acoustic environment coding vector to form a second joint feature, which is then input into the U-Net diffusion generation model to generate the final enhanced speech spectrum.

[0019] On the other hand, an acoustic environment-aware adaptive speech enhancement device is provided, the acoustic environment-aware adaptive speech enhancement device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the methods described above for acoustic environment-aware adaptive speech enhancement methods is implemented.

[0020] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described acoustic environment-aware guided adaptive speech enhancement methods.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0022] To address the issue of decreased adaptability to different acoustic environments in various application scenarios, this invention introduces acoustic environment coding. This involves a holistic perception of noise type, noise level, reverberation intensity, and target speech direction, extracting relevant representations to guide the speech enhancement model for targeted adaptation. To address the significant distortion in enhanced speech under strong noise conditions, this invention integrates multi-microphone spatial filtering with a generative enhancement model to reduce speech distortion. The acoustic environment representation is also used as one of the input vectors for the generative model to ensure adaptability under different acoustic environments. Through these two aspects, the speech enhancement algorithm can be significantly improved in various acoustic environments. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of an acoustic environment-aware adaptive speech enhancement method provided in an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of an acoustic environment-aware adaptive speech enhancement method provided in an embodiment of the present invention;

[0026] Figure 3 This is a block diagram of an acoustic environment-aware adaptive speech enhancement device provided in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an acoustic environment-aware adaptive speech enhancement device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0029] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0030] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0031] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0032] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0033] This invention provides an acoustic environment-aware adaptive speech enhancement method. This method can be implemented using an acoustic environment-aware adaptive speech enhancement device, which can be a terminal or a server. This invention is applicable to far-field speech interaction in high-noise, high-reverberation scenarios such as smart terminals, in-vehicle systems, conference systems, and robots. Functionally, this invention can improve the quality of speech perception and interaction in conference communication scenarios, enhance the accuracy of subsequent speech recognition, and ensure the accuracy of subsequent meeting minutes processing.

[0034] like Figure 1 The flowchart of the acoustic environment-aware guided adaptive speech enhancement method is shown below. Figure 2 The flowchart illustrates that the processing flow of this method may include the following steps:

[0035] S1. Perform time-frequency transformation on the multi-channel speech signal received by the microphone array to obtain a complex spectrum.

[0036] In one feasible implementation, the microphone array signal is first converted to a time-frequency domain signal using a short-time Fourier transform. This invention employs a Hamming window with a window length of 512 and a frame shift of 256 to extract the time-frequency plot of the signal from each microphone channel.

[0037] S2. Spatial sampling of the complex spectrum is performed, and a set of fixed beamforming filters pointing in different spatial directions are used to obtain the spatial-time-frequency three-dimensional feature tensor.

[0038] Optionally, the specific operations of S2 may include:

[0039] The 360-degree space is divided into N equal regions, and a fixed beamforming filter in the corresponding direction is used to filter the complex spectrum. The output of each filter determines the spatial sampling result. The value of N is positively correlated with the number of microphone arrays.

[0040] In one feasible implementation, the 360-degree space is divided into N equal regions, and each region is filtered using a fixed MVDR (Minimum Variance Distortionless Response) filter in the corresponding direction. The output of each filter is the spatial sampling result. The design of N is related to the number of microphones. Generally, the more microphones there are, the larger the value of N can be used to achieve more detailed spatial sampling. The disadvantage is that the computational load also increases proportionally.

[0041] Optionally, the number of microphone arrays is 5-8, and the value of N ranges from 12 to 36.

[0042] S3. Input the spatial-temporal three-dimensional feature tensor into the acoustic environment perception network based on the Transformer architecture, and extract the output of the last layer of the acoustic environment perception network as the acoustic environment encoding vector.

[0043] Among them, the acoustic environment perception network based on the Transformer architecture is the Transformer encoder, and its structure can adopt the structure commonly used in the existing technology. The embodiments of the present invention will not be described in detail here.

[0044] The Transformer encoder achieves compression and extraction of key environmental representations by encoding layer by layer from the first to the last, guided by a multi-task head. Therefore, this invention recommends using the output of the last layer as the representation to extract key information to the greatest extent possible.

[0045] The training process of the acoustic environment perception network based on the Transformer architecture is described below:

[0046] Optionally, during training, the acoustic environment perception network based on the Transformer architecture of S3 includes a Transformer encoder and a multi-task output head. The Transformer encoder is used to model the global dependencies of the input spatial-time-frequency three-dimensional feature tensor; the multi-task output head is connected to the output of the Transformer encoder and is used to simultaneously output the estimation results of noise type, signal-to-noise ratio, reverberation time, and sound source direction.

[0047] Optionally, the multi-tasking output header specifically includes:

[0048] The noise type classification head is used to output the probability distribution of multiple channel speech signals belonging to multiple preset noise types through a fully connected layer and a Softmax function.

[0049] The signal-to-noise ratio (SNR) regression head is used to output a normalized SNR estimate through a fully connected layer and a Sigmoid function.

[0050] The reverberation time regression head is used to output a normalized estimate of the reverberation time T60 through a fully connected layer and a sigmoid function.

[0051] The sound source direction classification head is used to output the probability distribution of a target sound source located in multiple preset spatial direction regions through a fully connected layer and a Softmax function.

[0052] In one feasible implementation, this application designs a multi-task head at the last layer of the traditional Transformer structure to optimize the corresponding loss function:

[0053] Noise type classification head: Soft-max classification head, supporting no fewer than 20 typical environments (white noise, street, factory, etc.);

[0054] Signal-to-noise ratio regression head: Sigmoid regression head, outputting continuous values ​​from -15 to 30 dB, using the average error loss function to guide its estimation;

[0055] Reverberation time regression head: Sigmoid regression head, predicts log10(T60), and uses the average error loss function to guide its estimation;

[0056] Sound source direction classification head: Soft-max classification head, which divides the space into M=18 directional regions and predicts the classification results.

[0057] The neural network with Transformer structure is used to realize the perception of acoustic environment. Its training objective is not the traditional "speech enhancement" or "single environmental parameter estimation". Instead, it takes four core acoustic environment parameters, namely noise type, signal-to-noise ratio (SNR), reverberation time T60 and target sound source direction (TDOA), as the multi-task output of the network. Through the multi-task loss function of the following formula (1), it back-propagates and forces the Transformer self-attention mechanism to automatically mine the joint representation coupled with the above parameters in the three-dimensional features of time-frequency-space.

[0058] Optionally, the loss function of the acoustic environment perception network based on the Transformer architecture during training is defined as:

[0059] (1)

[0060] in, Represents network parameters, , , and These represent the accuracy rates for noise type identification, signal-to-noise ratio estimation, reverberation time T60 identification, and target speech direction identification, respectively. and For cross-entropy loss, and This represents the mean square error loss.

[0061] The following is about , , and Each will be introduced separately:

[0062] 1. Noise type identification loss :

[0063] Task type: Multi-category task.

[0064] Network output: A probability vector processed by the Softmax function, where C is the total number of noise categories (≥20), p i This represents the probability that the model predicts the input speech to belong to the i-th type of noise.

[0065] Real label: A one-hot encoded vector in which only the element corresponding to the real noise class is 1, and the rest are 0.

[0066] Loss calculation: Cross-entropy loss is used. This loss measures the difference between the model's predicted probability distribution p and the true distribution y.

[0067] .

[0068] 2. Signal-to-noise ratio estimation loss :

[0069] Task type: Regression task.

[0070] Network output: A scalar o compressed by the Sigmoid function. SNR ∈(0,1).

[0071] Post-processing: Linearly map the network output to the target SNR range [SNR] min SNR max Given [-15, 30], we obtain the final predicted SNR value. .

[0072] =SNRmin+o SNR ×(SNRmax-SNRmin).

[0073] Real label: The true signal-to-noise ratio (SNR) of this audio segment.

[0074] Loss calculation: Mean squared error loss is used. This loss directly penalizes the squared difference between the predicted and the true values.

[0075] =(s- ) 2

[0076] 3. Reverberation time T60 recognition loss :

[0077] Task type: Regression task.

[0078] Network output: A scalar o compressed by the Sigmoid function. T60 ∈(0,1).

[0079] Post-processing: Due to the large dynamic range of T60 values, regression in the logarithmic space is usually more stable. Assume the true range of T60 is [T60...]. min T60 max The network actually predicts log. 10 The normalized value of (T60). The final predicted T60 value is:

[0080] =10 [log10(T60min)+ oT60×(log10(T60max)-log10(T60min))] .

[0081] Real tag: The actual reverberation time t of this audio segment.

[0082] Loss calculation: The mean squared error loss is calculated in the logarithmic domain to ensure balanced sensitivity to T60 values ​​of different magnitudes.

[0083] =(log 10 (t)-log 10 ( )) 2

[0084] 4. Loss in target speech direction recognition :

[0085] Task type: Multi-category task.

[0086] Network output: A probability vector processed by the Softmax function, where M=18, q j This represents the probability that the model predicts the sound source is located in the j-th direction region.

[0087] Real label: a one-hot encoded vector y doaIts dimension is M, and only the region index corresponding to the true direction of the sound source is 1.

[0088] Loss calculation: Cross-entropy loss is used.

[0089] .

[0090] S4. The complex spectrum is concatenated with the acoustic environment coding vector to form the first joint feature, which is then input into the speech enhancement network to obtain the preliminary enhanced speech spectrum.

[0091] The speech enhancement network can employ a DCCRN (Deep Complex Convolutional Recurrent Network) to adaptively compute enhancement filters and output an initial enhanced speech signal S(t,f).

[0092] In one feasible implementation, the complex spectrum X(t,f) is concatenated with the acoustic environment coding vector I(t,f) to obtain the first joint feature as Y(t,f) = [X(t,f), I(t,f)].

[0093] Existing DCCRN networks only use X(t,f) as the input vector, which only contains information about the distribution of the speech spectrum. This invention, in addition to this, also inputs I(t,f) from the vector Y(t,f) to explicitly carry information such as scene label, SNR, reverberation, and direction. This can provide the network with real-time clues about the application scenario, helping to obtain a more targeted enhanced audio signal S(t,f) based on the actual usage scenario.

[0094] S5. The initial enhanced speech spectrum is concatenated with the acoustic environment coding vector to form a second joint feature, which is then input into the U-Net diffusion generation model to generate the final enhanced speech spectrum.

[0095] Optionally, the network parameters of the U-Net diffusion generation model are 5 downsampled layers with [32, 64, 128, 256, 512] channels and symmetric upsampling.

[0096] In one feasible implementation, the initial enhanced speech spectrum S(t,f) is concatenated with the acoustic environment coding vector I(t,f) to obtain the second joint feature Y'(t,f) = [S(t,f), I(t,f)].

[0097] Because I(t,f) in this diffusion generation network provides acoustic environment cues, including the type and intensity of noise, this information can guide the network to generate the final speech signal more effectively, improving the model's adaptability to specific application scenarios. The model output is the final enhanced speech spectrum.

[0098] The experiment is briefly described below:

[0099] Based on a 6-microphone circular array (4cm radius, 16000Hz sampling rate) collected in the laboratory, 10 hours of test speech data were collected, totaling 36,000 samples, covering 15 types of noise (white noise, street noise, factory noise, fan noise, etc.), with SNR ranging from -5 dB to 20 dB, T60 distribution from 0.2 s to 0.8 s, and target orientation from 0° to 360°. Through the complete "acoustic environment perception-encoding splicing-condition generation" technology chain of this invention, compared to general solutions that remove environment encoding, under the same hardware and dataset, objective indicators show a 10% relative improvement in speech perception quality, an 8.5% relative improvement in speech intelligibility, and a 5.5% relative reduction in speech recognition word error rate.

[0100] To address the issue of decreased performance and robustness of speech enhancement models due to strong reverberation in practical applications, this invention improves speech enhancement performance primarily in the following two aspects:

[0101] (1) In view of the problem of decreased adaptability of acoustic environment in different application scenarios, this invention introduces acoustic environment coding to perceive the noise type, noise level, reverberation intensity and target speech direction as a whole, and extract relevant representations to guide the speech enhancement model to make targeted adaptations, which can significantly improve the speech enhancement effect of speech enhancement algorithm in different acoustic environments.

[0102] (2) In the case of strong noise environment, the enhanced speech has strong distortion. This invention integrates multi-microphone spatial filtering and generative enhancement model to reduce speech distortion. Acoustic environment representation is also used as one of the input vectors of generative model to ensure adaptability under different acoustic environments.

[0103] Figure 3 This is a block diagram of an acoustic environment-aware adaptive speech enhancement device provided in an embodiment of the present invention. This device is used in an acoustic environment-aware adaptive speech enhancement method. (Refer to...) Figure 3 The device includes a time-frequency conversion unit 310, a sampling and filtering unit 320, a feature extraction unit 330, a speech enhancement unit 340, and a diffusion generation unit 350. Wherein:

[0104] The time-frequency conversion unit 310 is used to perform time-frequency conversion on the multi-channel speech signal received by the microphone array to obtain a complex spectrum.

[0105] The sampling and filtering unit 320 is used to spatially sample the complex spectrum and obtain a spatial-time-frequency three-dimensional feature tensor by passing through a set of fixed beamforming filters pointing in different spatial directions.

[0106] The feature extraction unit 330 is used to input the spatial-temporal three-dimensional feature tensor into the acoustic environment perception network based on the Transformer architecture, and extract the output of the last layer of the acoustic environment perception network as the acoustic environment encoding vector.

[0107] The speech enhancement unit 340 is used to concatenate the complex spectrum with the acoustic environment coding vector to form a first joint feature, which is then input into the speech enhancement network to obtain a preliminary enhanced speech spectrum.

[0108] The diffusion generation unit 350 is used to concatenate the preliminary enhanced speech spectrum with the acoustic environment coding vector to form a second joint feature, which is then input into the U-Net diffusion generation model to generate the final enhanced speech spectrum.

[0109] Figure 4 This is a schematic diagram of the structure of an acoustic environment-aware adaptive speech enhancement device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, an acoustic environment-aware adaptive speech enhancement device may include the above-mentioned Figure 3 The illustrated acoustic environment-aware guided adaptive speech enhancement device. Optionally, the acoustic environment-aware guided adaptive speech enhancement device 410 may include a first processor 2001.

[0110] Optionally, the acoustically environment-aware adaptive speech enhancement device 410 may also include a memory 2002 and a transceiver 2003.

[0111] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0112] The following is combined with Figure 4 A detailed description of each component of the acoustic environment-aware adaptive speech enhancement device 410 is provided below:

[0113] The first processor 2001 is the control center of the acoustic environment perception-guided adaptive voice enhancement device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0114] Optionally, the first processor 2001 can perform various functions of the acoustically environment-aware adaptive speech enhancement device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0115] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0116] In a specific implementation, as one example, the acoustic environment-aware adaptive speech enhancement device 410 may also include multiple processors, such as... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0117] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0118] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the acoustically environment-aware adaptive speech enhancement device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0119] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0120] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0121] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the acoustically environment-aware adaptive speech enhancement device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0122] It should be noted that, Figure 4 The structure of the acoustically aware adaptive voice enhancement device 410 shown in the diagram does not constitute a limitation on the router. Actual acoustically aware adaptive voice enhancement devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0123] Furthermore, the technical effect of the acoustic environment perception-guided adaptive speech enhancement device 410 can be referred to the technical effect of the acoustic environment perception-guided adaptive speech enhancement method described in the above method embodiments, and will not be repeated here.

[0124] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0125] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0126] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0127] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0128] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0129] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive speech enhancement method guided by acoustic environment perception, characterized in that, The method includes: S1. Perform time-frequency transformation on the multi-channel speech signal received by the microphone array to obtain a complex spectrum; S2. Spatial sampling is performed on the complex spectrum, and a set of fixed beamforming filters pointing in different spatial directions is used to obtain a spatial-time-frequency three-dimensional feature tensor. S3. Input the spatial-temporal three-dimensional feature tensor into the acoustic environment perception network based on the Transformer architecture, and extract the output of the last layer of the acoustic environment perception network as the acoustic environment encoding vector. S4. The complex spectrum is concatenated with the acoustic environment coding vector to form a first joint feature, which is then input into the speech enhancement network to obtain a preliminary enhanced speech spectrum; S5. The preliminary enhanced speech spectrum is concatenated with the acoustic environment coding vector to form a second joint feature, which is then input into the U-Net diffusion generation model to generate the final enhanced speech spectrum.

2. The acoustic environment-aware guided adaptive speech enhancement method according to claim 1, characterized in that, The S2 process spatially samples the complex spectrum and obtains a space-time-frequency three-dimensional feature tensor by passing it through a set of fixed beamforming filters pointing in different spatial directions, including: The 360-degree space is divided into N equal regions, and the complex spectrum is filtered by fixed beamforming filters in the corresponding directions. The output of each filter determines the spatial sampling result. The value of N is positively correlated with the number of microphone arrays.

3. The acoustic environment-aware guided adaptive speech enhancement method according to claim 1, characterized in that, During training, the acoustic environment perception network based on the Transformer architecture of S3 includes a Transformer encoder and a multi-task output head; wherein, the Transformer encoder is used to model the global dependency relationship of the input spatial-time-frequency three-dimensional feature tensor. The multitasking output head is connected to the output terminal of the Transformer encoder and is used to simultaneously output the estimation results of the noise type, signal-to-noise ratio, reverberation time, and sound source direction.

4. The acoustic environment-aware guided adaptive speech enhancement method according to claim 3, characterized in that, The multi-task output header includes: A noise type classification head is used to output the probability distribution of the multi-channel speech signal belonging to a preset number of noise types through a fully connected layer and a Softmax function; The signal-to-noise ratio (SNR) regression head is used to output a normalized SNR estimate through a fully connected layer and a Sigmoid function. The reverberation time regression head is used to output a normalized estimate of the reverberation time T60 through a fully connected layer and a Sigmoid function; The sound source direction classification head is used to output the probability distribution of a target sound source located in multiple preset spatial direction regions through a fully connected layer and a Softmax function.

5. The acoustic environment-aware guided adaptive speech enhancement method according to claim 4, characterized in that, The loss function of the acoustic environment perception network to be trained is defined as: (1) in, Represents network parameters, , , and These represent the accuracy rates for noise type identification, signal-to-noise ratio estimation, reverberation time T60 identification, and target speech direction identification, respectively. and For cross-entropy loss, and This represents the mean square error loss.

6. The acoustic environment-aware guided adaptive speech enhancement method according to claim 2, characterized in that, The number of microphone arrays is 5-8, and the value of N ranges from 12 to 36.

7. The acoustic environment-aware guided adaptive speech enhancement method according to claim 1, characterized in that, The network parameters of the U-Net diffusion generation model are 5 downsampled layers with [32, 64, 128, 256, 512] channels and symmetrical upsampling.

8. An acoustic environment-aware adaptive speech enhancement device, wherein the acoustic environment-aware adaptive speech enhancement device is used to implement the acoustic environment-aware adaptive speech enhancement method as described in any one of claims 1-7, characterized in that, The device includes: The time-frequency conversion unit is used to perform time-frequency conversion on the multi-channel speech signal received by the microphone array to obtain a complex spectrum. The sampling and filtering unit is used to spatially sample the complex spectrum and obtain a space-time-frequency three-dimensional feature tensor by passing it through a set of fixed beamforming filters pointing in different spatial directions. The feature extraction unit is used to input the spatial-temporal three-dimensional feature tensor into the acoustic environment perception network based on the Transformer architecture, and extract the output of the last layer of the acoustic environment perception network as the acoustic environment encoding vector. The speech enhancement unit is used to concatenate the complex spectrum with the acoustic environment coding vector to form a first joint feature, which is then input into the speech enhancement network to obtain a preliminary enhanced speech spectrum. The diffusion generation unit is used to concatenate the preliminary enhanced speech spectrum with the acoustic environment coding vector to form a second joint feature, which is then input into the U-Net diffusion generation model to generate the final enhanced speech spectrum.

9. An acoustic environment-aware guided adaptive speech enhancement device, characterized in that, The acoustic environment-aware guided adaptive speech enhancement device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images