Voice signal processing method, device, equipment, medium and product

Abstract

The present application proposes a speech signal processing method, device, equipment, medium and product. The method obtains an original microphone signal, which includes a clean microphone signal and an interference signal, and determines a spectrum mask corresponding to the original microphone signal, wherein the spectrum mask characterizes the degree of retention of various types of signals in the original microphone signal; based on the original microphone signal and the spectrum mask, the interference signal in the original microphone signal is eliminated to obtain a clean microphone signal. The present application can distinguish and retain the key components of the clean microphone signal in the original microphone signal through the spectrum mask, while suppressing irrelevant noise and echo to improve speech clarity.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech signal processing method, device, equipment, medium and product. Background Art

[0002] In full-duplex real-time communication systems, such as teleconferencing, video conferencing, voice assistants, and smart home devices, clear voice quality is essential for effective interaction. These systems need to support not only high-quality voice transmission, but also strong anti-interference capabilities to ensure smooth communication between users. However, in actual application scenarios, the microphone not only picks up the voice of the near-end user, but also captures environmental noise and echo, which leads to a decrease in voice clarity and poor voice quality. Summary of the invention

[0003] Based on the above technical status, the present application proposes a speech signal processing method, device, equipment, medium and product, which can improve speech clarity and thus improve the speech quality during communication.

[0004] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0005] In a first aspect, the present application proposes a speech signal processing method, comprising: obtaining an original microphone signal, wherein the original microphone signal includes a clean microphone signal and an interference signal; determining a spectrum mask corresponding to the original microphone signal, wherein the spectrum mask characterizes the degree of retention of various types of signals in the original microphone signal; based on the original microphone signal and the spectrum mask, eliminating the interference signal in the original microphone signal to obtain the clean microphone signal.

[0006] In some implementations, the interference signal includes a background noise signal and an echo signal corresponding to a far-end reference signal; determining a spectrum mask corresponding to the original microphone signal, and eliminating the interference signal in the original microphone signal based on the original microphone signal and the spectrum mask to obtain the clean microphone signal, includes: superimposing the spectrum signals corresponding to the original microphone signal and the far-end reference signal to obtain a superimposed spectrum signal, and the far-end reference signal is used to assist a speech recognition model in identifying the echo signal in the original microphone signal; inputting the superimposed spectrum signal into a speech signal processing model, determining the spectrum mask corresponding to the superimposed spectrum signal by the speech signal processing model, and eliminating the interference signal in the original microphone signal based on the original microphone signal and the spectrum mask to obtain the clean microphone signal.

[0007] In some implementations, the speech signal processing model includes a feature extraction module, a decoder module and a complex ratio mask module; wherein, the speech processing model determines the spectrum mask corresponding to the superimposed spectrum signal, and based on the original microphone signal and the spectrum mask, eliminates the interference signal in the original microphone signal to obtain the clean microphone signal, including: extracting features from the superimposed spectrum signal through the feature extraction module to obtain the spectrum features corresponding to the superimposed spectrum signal; determining the spectrum mask corresponding to the superimposed spectrum signal based on the spectrum features through the decoder module; multiplying the original microphone signal and the spectrum mask through the complex ratio mask module to eliminate the interference signal in the original microphone signal to obtain the clean microphone signal.

[0008] In some implementations, the feature extraction module includes a frequency band merging unit and a feature extraction unit; wherein, the feature extraction module performs feature extraction on the superimposed spectrum signal to obtain a spectrum feature corresponding to the superimposed spectrum signal, including: compressing the spectrum signal within the first frequency band in the superimposed spectrum signal by the frequency band merging unit to obtain a compressed spectrum signal, the spectrum signal within the first frequency band being the spectrum signal corresponding to the interference signal; and performing feature extraction on the compressed spectrum signal by the feature extraction unit to obtain the spectrum feature.

[0009] In some implementations, the feature extraction unit includes an encoder and a bottleneck layer; wherein, performing feature extraction on the compressed spectrum signal by the feature extraction unit to obtain the spectrum feature includes: encoding the compressed spectrum signal by the encoder to obtain hidden spectrum features corresponding to the compressed spectrum signal; performing intra-frame and inter-frame feature extraction on the hidden spectrum features by the bottleneck layer to obtain spectrum features corresponding to the superimposed spectrum signal.

[0010] In some implementations, the decoder module includes a decoder and a frequency band splitting unit; wherein, the decoder module determines the spectrum mask corresponding to the superimposed spectrum signal based on the spectrum characteristics, including: decoding the spectrum characteristics by the decoder to obtain an initial spectrum mask; performing frequency band splitting on the initial spectrum mask by the frequency band splitting unit to obtain the spectrum mask, and the frequency resolution of the spectrum mask is the same as the frequency resolution of the superimposed spectrum signal.

[0011] In some implementations, the spectrum signal of the original microphone signal includes multiple time-frequency blocks; wherein, the original microphone signal and the spectrum mask are multiplied by a complex ratio mask module to eliminate the interference signal in the original microphone signal to obtain the clean microphone signal, including: for each time-frequency block in the multiple time-frequency blocks, the complex ratio mask module is used to multiply the time-frequency block and its corresponding adjacent time-frequency blocks with the spectrum mask value at the corresponding position in the spectrum mask to obtain the reconstructed spectrum signal corresponding to the time-frequency block; the reconstructed spectrum signals corresponding to each of the multiple time-frequency blocks are converted to the time domain to obtain the clean microphone signal.

[0012] In some implementations, the speech signal processing model is trained by the following steps: obtaining original microphone signal samples including various types of signal samples, the various types of signal samples including clean microphone signal samples and interference signal samples, the original microphone signal samples corresponding to labeled clean microphone signals; determining a spectrum mask sample corresponding to the original microphone signal sample through the speech signal processing model to be trained, and based on the original microphone signal samples and the spectrum mask sample, eliminating the interference signal samples in the original microphone signal samples to obtain a predicted clean microphone signal, the spectrum mask sample characterizing the degree of retention of various types of signal samples in the original microphone signal samples; determining a signal prediction loss based on the difference between the predicted clean microphone signal and the labeled clean microphone signal, and converging the speech signal processing model to be trained based on the signal prediction loss to obtain a speech signal processing model.

[0013] In some implementations, the obtaining of original microphone signal samples including various types of signal samples includes: obtaining clean microphone signal samples and clean far-end reference signal samples; superimposing a far-end background noise signal and an echo signal on the clean far-end reference signal samples to obtain initial far-end reference signal samples; superimposing a near-end background noise signal on the clean microphone signal samples to obtain initial microphone signal samples; and obtaining the original microphone signal samples based on the initial far-end reference signal samples and the initial microphone signal samples.

[0014] The second aspect of the present application proposes a speech signal processing device, comprising: an acquisition unit, used to acquire an original microphone signal, wherein the original microphone signal includes a clean microphone signal and an interference signal; a determination unit, used to determine a spectrum mask corresponding to the original microphone signal, wherein the spectrum mask characterizes the degree of retention of various types of signals in the original microphone signal; and an elimination unit, used to eliminate the interference signal in the original microphone signal based on the original microphone signal and the spectrum mask to obtain the clean microphone signal.

[0015] The third aspect of the present application proposes an electronic device, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the speech signal processing method described in the first aspect and any one of the implementation methods of the first aspect by running the program in the memory.

[0016] In a fourth aspect, the present application proposes a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the speech signal processing method described in the first aspect and any one of the implementation methods of the first aspect is implemented.

[0017] In a fifth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the speech signal processing method described in the first aspect and any one of the implementation methods of the first aspect.

[0018] The speech signal processing method, device, equipment, medium and product proposed in the present application determine the spectrum mask corresponding to the original microphone signal, which spectrum mask characterizes the degree of retention of various signals (including clean microphone signals and interference signals) in the original microphone signal, and eliminates the interference signal in the original microphone signal based on the original microphone signal and the spectrum mask, thereby obtaining a clean microphone signal. Since the spectrum mask indicates which frequency components in the original microphone signal should be retained and which frequency components should be suppressed, the spectrum mask can distinguish and retain the key components in the original microphone signal, while suppressing irrelevant interference signals, thereby improving the clarity of the speech signal. In real-time communication systems such as telephone conferences, video conferences, and voice assistants, the speech signal after removing the interference signal can ensure that the participants can clearly hear each other's voices, improve communication efficiency, and be more easily accurately recognized by the automatic speech recognition (ASR) system, thereby improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 A flowchart of a speech signal processing method provided in an embodiment of the present application.

[0021] Figure 2 A schematic diagram of the framework of the speech signal processing model provided in the embodiment of the present application.

[0022] Figure 3 A schematic diagram of a flow chart for determining a clean microphone signal through a speech signal processing model provided in an embodiment of the present application.

[0023] Figure 4 A schematic diagram of the framework of a feature extraction module provided in an embodiment of the present application.

[0024] Figure 5 A schematic diagram of the structure of the bottleneck layer provided in an embodiment of the present application.

[0025] Figure 6 A schematic diagram of the framework of another feature extraction module provided in an embodiment of the present application.

[0026] Figure 7 A schematic diagram of the structure of a decoder module provided in an embodiment of the present application.

[0027] Figure 8 A schematic diagram of the structure of another decoder module provided in an embodiment of the present application.

[0028] Fig. 9 A flowchart of the training process of the speech signal processing model provided in an embodiment of the present application.

[0029] Fig.10 A schematic diagram of a process for acquiring original microphone signal samples provided in an embodiment of the present application.

[0030] Fig.11 A schematic diagram of the structure of a speech signal processing device provided in an embodiment of the present application.

[0031] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solution proposed in the embodiments of the present application is applicable to any speech signal processing application scenario, such as telephone conferencing, video conferencing, in-vehicle voice interaction systems, smart home devices, etc.

[0033] In a telephone conference or video conference scenario, multiple participants are located in different locations and communicate through telephone or video conferencing software. Each location may have different background noise (such as air conditioning sound, keyboard typing, etc.), and due to different room acoustic characteristics, different degrees of echo will be generated.

[0034] In the in-vehicle voice interaction system, when the driver uses the voice assistant to make calls, navigate, control vehicle functions, etc. while driving, there will be environmental noise in the car (such as wind noise, engine noise, tire friction noise, etc.) and echo (due to reflection inside the car).

[0035] In the smart home device scenario, when users control various devices in the home (such as lights, curtains, audio, etc.) through smart speakers or smart assistants, there will be background noise (such as TV sound, kitchen sound, etc.) and echo (due to furniture layout) in the home environment.

[0036] Applying the technical solution of the embodiment of the present application to any application scenario of voice signal processing can improve the quality of voice signals, thereby improving the smoothness of user voice communication.

[0037] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] Before introducing this application solution, the relevant technologies are first introduced:

[0040] In a full-duplex real-time communication system, acoustic echo cancellation (AEC) and noise suppression (NS) can effectively eliminate echoes and noise, thereby improving voice quality during communication.

[0041] Acoustic Echo Cancellation (AEC) refers to the echo that is generated when the sound played by the speaker is picked up again by the microphone. In full-duplex communication, this echo will have a significant impact on the user experience. The goal of AEC is to eliminate this echo and ensure that the receiving end hears only the clear voice of the near-end user.

[0042] Noise Suppression (NS) refers to the fact that in actual application scenarios, environmental noise such as background conversation, traffic noise, and wind noise will affect the quality of voice communication. NS aims to eliminate these background noises and ensure the clarity of voice signals.

[0043] At present, the methods for processing AEC tasks and NS tasks mainly include: traditional methods, neural network-based methods, and comprehensive methods. In traditional methods, classic noise suppression methods (such as Wiener filters) are usually used to estimate the noise power spectrum and adjust the spectral components of the speech signal to suppress noise. In AEC, the normalized least mean squares (NLMS) adaptive filter is used to minimize the error signal by gradually adjusting the filter coefficients.

[0044] The method based on deep neural network (DNN) uses large-scale data to train models for NS tasks and AEC tasks respectively, learns complex noise and echo features, and then applies the trained models to NS tasks and AEC tasks. This method can handle more complex noise and echo patterns, but requires a lot of computing resources.

[0045] The comprehensive method uses a neural network to combine NS and AEC for processing, while removing echo and environmental noise. This method can improve the voice quality in complex scenarios to a certain extent, but it still needs further optimization in dual-talk scenarios (i.e., there is both far-end echo and near-end speech).

[0046] In summary, these methods still need to be further improved in improving speech quality. In addition, neural network-based methods and comprehensive methods require a lot of computing resources and have high computational complexity, which not only affects the efficiency of speech signal processing, but also makes it difficult to achieve real-time processing in resource-constrained environments (such as mobile devices).

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] The present application embodiment first proposes a method for processing a speech signal. Figure 1 As shown, the method includes steps S101 to S103:

[0049] S101. Acquire an original microphone signal, where the original microphone signal includes a clean microphone signal and an interference signal.

[0050] In this embodiment, the original microphone signal refers to the sum of all sounds actually picked up by the microphone, including clean microphone signals and interference signals. The clean microphone signal refers to an ideal speech signal that is not affected by interference signals. Interference signals include background noise signals and echo signals corresponding to far-end reference signals.

[0051] Specifically, the far-end reference signal refers to the signal transmitted from the far end, which is played by the near-end speaker after a series of transmission and processing. In this process, the far-end reference signal may experience factors such as room reverberation and spatial propagation, causing the signal to change. When the signal is captured by the near-end microphone, it is no longer the original far-end reference signal, but a signal that is different from the original far-end reference signal. The signal captured by the microphone is the original microphone signal at the near end, which contains the components of the far-end reference signal and may also contain other sounds at the near end, such as the speech of the near-end personnel, background noise, etc.

[0052] For example, assume that in a teleconference system, user A is talking to remote user B through a microphone in a relatively quiet room. User B's voice is played through a speaker while user A is speaking.

[0053] When user A speaks in a completely silent or nearly silent environment (without background noise or echo interference), the clear speech of user A captured by the microphone is the clean microphone signal.

[0054] When there is air conditioning sound, wind sound, or distant conversation in the room, etc., these sounds are regarded as background noise signals.

[0055] When the voice of user B is played out through the speaker, it is picked up by the microphone again after being reflected in the room, which is an echo signal. For example, user B says, "Can you hear me?" This sentence is played out through the speaker, reflected in the room, and picked up by the microphone again.

[0056] The far-end reference signal refers to the signal transmitted from the far end. After a series of propagation and processing, the signal is played out by the near-end speaker. For example, the voice of user B is transmitted to the device of user A through the network and played out through the speaker. Suppose user B says, "Can you hear me?" This sentence is the far-end reference signal. When the sentence is played out through the speaker, it is picked up by the microphone again after being reflected in the room, and becomes an echo signal.

[0057] In general, a clean microphone signal refers to a voice signal that only includes the clear speech of user A. Interference signals include background noise (such as the sound of an air conditioner) and the echo signal of the far-end reference signal (the echo of user B's voice). All sounds actually picked up by the microphone, including user A's speech, background noise, and the echo of user B's speech, are the original microphone signals.

[0058] S102: Determine a spectrum mask corresponding to the original microphone signal, where the spectrum mask represents the degree of preservation of various types of signals in the original microphone signal.

[0059] The spectrum mask is a two-dimensional matrix in which each element corresponds to a specific frequency component and time frame, and its value determines the degree of preservation of the corresponding frequency component in the spectrum signal of the original microphone signal.

[0060] S103 . Eliminate interference signals in the original microphone signal based on the original microphone signal and the spectrum mask to obtain a clean microphone signal.

[0061] In some embodiments, step S102 and step S103 may be implemented by a speech signal processing model.

[0062] Specifically, the spectral signals corresponding to the original microphone signal and the remote reference signal can be superimposed to obtain a superimposed spectral signal, and the remote reference signal is used to assist the speech recognition model in identifying the echo signal in the original microphone signal; the superimposed spectral signal is input into the speech signal processing model, and the speech signal processing model determines the spectral mask corresponding to the superimposed spectral signal, and based on the original microphone signal and the spectral mask, eliminates the interference signal in the original microphone signal to obtain a clean microphone signal.

[0063] The specific process of implementing step S102 and step S103 by using the speech signal processing model is introduced below in conjunction with the accompanying drawings:

[0064] Figure 2 The following is a schematic diagram of the framework of the speech signal processing model provided in the embodiment of the present application. Figure 2 As shown in the figure, x(n) represents the remote reference signal, and y(n) represents the original microphone signal. By performing short-time Fourier transform (STFT) on the remote reference signal x(n) and the original microphone signal y(n), they can be converted into the frequency domain to obtain the spectrum signal Y(f, t) corresponding to the original microphone signal and the spectrum signal X(f, t) corresponding to the remote reference signal, and the spectrum signal Y(f, t) corresponding to the original microphone signal and the spectrum signal X(f, t) corresponding to the remote reference signal are superimposed to obtain a superimposed spectrum signal.

[0065] After obtaining the superimposed spectrum signal, the spectrum mask corresponding to the superimposed spectrum signal can be determined through the speech signal processing model, and based on the original microphone signal and the spectrum mask, the interference signal in the original microphone signal can be eliminated to obtain a clean microphone signal.

[0066] like Figure 2 As shown, the speech signal processing model includes a feature extraction module 21, a decoder module 22 and a complex ratio mask module 23; the feature extraction module 21 is used to extract spectrum features from the superimposed spectrum signal; the decoder module 22 is used to determine the spectrum mask based on the spectrum features; the complex ratio mask module 23 is used to multiply the original microphone signal and the spectrum mask to eliminate the interference signal in the original microphone signal to obtain a clean microphone signal.

[0067] Specifically, the specific implementation process of obtaining a clean microphone signal through a speech signal processing model can be as follows:

[0068] Figure 3 A schematic diagram of a process for determining a clean microphone signal through a speech signal processing model provided in an embodiment of the present application. Figure 3 As shown, the spectrum mask corresponding to the superimposed spectrum signal is determined by the speech signal processing model, and based on the original microphone signal and the spectrum mask, the interference signal in the original microphone signal is eliminated to obtain a clean microphone signal, including the following steps S301 to S303:

[0069] S301 . Extract features from the superimposed spectrum signal through the feature extraction module 21 to obtain spectrum features corresponding to the superimposed spectrum signal.

[0070] There are many ways to extract features from the superimposed spectrum signal through the feature extraction module 21 to obtain the spectrum features corresponding to the superimposed spectrum signal, which can be specifically as follows:

[0071] Figure 4 A schematic diagram of a feature extraction module provided in an embodiment of the present application. Figure 4 As shown, the feature extraction module 21 includes a band merging (Band Merge, BM) unit 210 and a feature extraction unit 211. The band merging (Band Merge, BM) unit 210 is used to compress the superimposed spectrum signal to obtain a compressed spectrum signal, and the spectrum signal within the first frequency band is the spectrum signal corresponding to the interference signal; the feature extraction unit 211 is used to extract features from the compressed spectrum signal to obtain spectrum features. Specifically, the band merging unit 210 compresses the spectrum signal within the first frequency band in the superimposed spectrum signal to obtain a compressed spectrum signal.

[0072] The feature extraction module performs feature extraction on the superimposed spectrum signal to obtain spectrum features corresponding to the superimposed spectrum signal, including: compressing the spectrum signal within the first frequency band in the superimposed spectrum signal by the frequency band merging unit 210 to obtain a compressed spectrum signal; and performing feature extraction on the compressed spectrum signal by the feature extraction unit 211 to obtain spectrum features.

[0073] In this embodiment, the superimposed spectrum signal includes multiple frequency band signals. The information carried by different frequency band signals in the superimposed spectrum signal is not balanced. For example, the human voice frequency is mainly concentrated in 0-2KHz, and the high frequency band carries relatively less information. Before processing the superimposed spectrum signal, the redundant information in the high frequency band of the superimposed spectrum signal is compressed by the BM module, which can reduce the amount of calculation. Specifically, the equivalent rectangular bandwidth (ERB) scale can be used to divide the frequency band. For example, the superimposed spectrum signal includes 321 frequency bands. Since the human voice frequency is mainly concentrated in 80 frequency bands within the range of 0-2 KHz, the 80 frequency bands corresponding to the low frequency band (0-2KHz) can be kept unchanged, and the remaining 241 high frequency bands (2-8KHz) can be compressed to 81, thereby compressing the 321 frequency bands of the superimposed spectrum signal to 161 frequency bands, which can reduce half of the signal dimension, thereby reducing the subsequent amount of calculation, so that a lightweight speech signal processing model can be realized.

[0074] Continue reading Figure 4 The feature extraction unit 211 includes an encoder 2110 and a bottleneck layer 2111. The encoder 2110 is used to extract features from the compressed spectrum signal to obtain hidden spectrum features corresponding to the compressed spectrum signal; the bottleneck layer 2111 is used to perform intra-frame and inter-frame feature extraction on the hidden spectrum features corresponding to the compressed spectrum signal to obtain spectrum features corresponding to the superimposed spectrum signal.

[0075] The feature extraction unit 211 performs feature extraction on the compressed spectrum signal to obtain spectrum features, including: performing feature extraction on the compressed spectrum signal through the encoder 2110 to obtain hidden spectrum features corresponding to the compressed spectrum signal; performing intra-frame and inter-frame feature extraction on the hidden spectrum features corresponding to the compressed spectrum signal through the bottleneck layer 2111 to obtain spectrum features corresponding to the superimposed spectrum signal.

[0076] The encoder 2110 includes a plurality of convolution modules, such as 3. Each convolution module includes a two-dimensional convolution layer, a batch normalization layer, and a PReLU activation function layer connected in sequence. The plurality of convolution modules encode the compressed spectrum signal in sequence (convolution operation and downsampling operation) to convert it into a "code" or "hidden representation". The hidden representation or code can reflect the key attribute information in the compressed spectrum signal, and its frequency resolution is lower than the frequency resolution of the compressed spectrum signal.

[0077] For example, for the 161 frequency bands of the compressed spectrum signal, the frequency resolution is reduced to 81, 41, and 41 bands after three convolution modules. The last convolution module does not perform downsampling in order to ensure a certain frequency resolution, which is conducive to the subsequent signal recovery.

[0078] The number of channels of the convolution layers of the three convolution modules in encoder 2110 is set to {32, 64, 64}, respectively, and the convolution kernel size and stride in the time and frequency dimensions are {(1, 3), (1, 3), (1, 3)} and {(1, 2), (1, 2), (1, 1)}, respectively. The convolution kernel size in the time dimension is set to 1, so that the past time frame cache can be discarded in streaming inference, reducing the amount of calculation during inference.

[0079] The bottleneck layer can use a dual-path gated recurrent unit (DPGRU).

[0080] Figure 5 This is a schematic diagram of the bottleneck layer structure provided in the embodiment of the present application. Figure 5 As shown, the bottleneck layer 2111 includes an intra-frame feature extraction unit (Intra GRU) 41 and an inter-frame feature extraction unit (Inter GRU) 42. The intra-frame feature extraction unit 41 is used to perform intra-frame feature extraction on hidden spectrum features corresponding to each time frame of the compressed spectrum signal to obtain intra-frame spectrum features corresponding to each time frame of the compressed spectrum signal; the inter-frame feature extraction unit 42 is used to perform inter-frame spectrum feature extraction on intra-frame spectrum features corresponding to each time frame of the compressed spectrum signal to obtain spectrum features corresponding to the compressed spectrum signal.

[0081] The bottleneck layer 2111 is used to perform intra-frame and inter-frame feature extraction on the hidden spectrum features corresponding to the compressed spectrum signal to obtain the spectrum features corresponding to the superimposed spectrum signal, including: performing intra-frame spectrum feature extraction on the hidden spectrum features corresponding to each time frame of the compressed spectrum signal by the intra-frame feature extraction unit 41 to obtain the intra-frame spectrum features corresponding to each time frame of the compressed spectrum signal; performing inter-frame spectrum feature extraction on the intra-frame spectrum features corresponding to each time frame of the compressed spectrum signal by the inter-frame feature extraction unit 42 to obtain the spectrum features corresponding to the compressed spectrum signal.

[0082] In this embodiment, the intra-frame feature extraction unit (Intra GRU) 41 can model the spectral features of a single time frame, and the inter-frame feature extraction unit (Inter GRU) 42 can capture the dependencies between adjacent frames, thereby reducing the impact on causality during reasoning. In addition, this embodiment uses a dual-path gated recurrent unit as a bottleneck layer. Since it has fewer gates and a lower number of parameters, it can reduce the number of parameters and computational costs of the entire model, making the speech signal processing model lightweight while maintaining model performance and efficiency.

[0083] Continue reading Figure 5 , the intra-frame feature extraction unit 41 includes a bidirectional gated recurrent unit (Bi-GRU), a fully connected layer and a normalization layer; the inter-frame feature extraction unit 42 includes a unidirectional gated recurrent unit GRU, a fully connected layer and a normalization layer. The bidirectional GRU is used to model the spectrum of a single time frame, that is, to process the spectrum features of each time frame in the frequency domain. In this way, the spectrum information within each time frame can be fully utilized to capture the correlation in the frequency domain. The unidirectional GRU is used to capture the dependency between adjacent frames, that is, to capture the changes between frames on the time axis. In this way, the causality of the model can be maintained during reasoning, that is, only the current and previous frame information is used for prediction.

[0084] The bottleneck layer compresses the input features through a combination of bidirectional GRU and unidirectional GRU to generate a compact and information-rich feature representation. These features are fused together to form a comprehensive feature representation that includes both features in the frequency domain and features of changes on the time axis. This helps to remove redundant information and retain the most core features.

[0085] The above embodiment introduces a feature extraction module including a frequency band merging unit. However, in some implementations, the feature extraction module may not include a frequency band merging unit, as follows:

[0086] Figure 6 A schematic diagram of another feature extraction module provided in an embodiment of the present application. Figure 6As shown, the feature extraction module 21 includes a feature extraction unit 211, which is used to extract features from the superimposed spectrum signal to obtain spectrum features corresponding to the superimposed spectrum signal.

[0087] Among them, the feature extraction unit 211 includes an encoder 2110 and a bottleneck layer 2111; the encoder 2110 is used to extract features from the superimposed spectrum signal to obtain hidden spectrum features corresponding to the superimposed spectrum signal; the bottleneck layer 2111 is used to perform intra-frame and inter-frame feature extraction on the hidden spectrum features corresponding to the superimposed spectrum signal to obtain spectrum features corresponding to the superimposed spectrum signal.

[0088] The feature extraction unit 211 performs feature extraction on the superimposed spectrum signal to obtain spectrum features, including: performing feature extraction on the superimposed spectrum signal through the encoder 2110 to obtain hidden spectrum features corresponding to the superimposed spectrum signal; performing intra-frame and inter-frame feature extraction on the hidden spectrum features corresponding to the superimposed spectrum signal through the bottleneck layer 2111 to obtain spectrum features corresponding to the superimposed spectrum signal.

[0089] Continue reading Figure 3 After step S301, this embodiment may further include the following step S302.

[0090] S302 : Determine, by the decoder module 22 , a spectrum mask corresponding to the superimposed spectrum signal based on the spectrum characteristics.

[0091] Figure 7 A schematic diagram of the structure of a decoder module provided in an embodiment of the present application. Figure 7 and Figure 4 Correspondingly, the decoder module 22 includes a decoder 221 and a frequency band segmentation unit 222. The decoder 221 is used to determine an initial spectrum mask based on the spectrum feature, and the frequency resolution of the initial spectrum mask is the same as the frequency resolution of the compressed spectrum signal. The frequency band segmentation unit 222 is used to perform frequency band segmentation based on the initial spectrum mask to obtain a spectrum mask. The spectrum mask finally obtained has the same frequency resolution corresponding to the spectrum signal of the original microphone signal.

[0092] Among them, the far-end reference signal is used to help the decoder module 22 identify the spectrum signal corresponding to the echo signal during the decoding process, so that the echo signal can be accurately eliminated. For example, the decoder module 22 compares the spectrum characteristics of the far-end reference signal with the spectrum characteristics in the original microphone signal to identify the spectrum components corresponding to the echo signal. With the help of the far-end reference signal, the decoder module can accurately identify the spectrum components corresponding to the echo signal, thereby generating a spectrum mask. The mask selectively suppresses the echo signal in the spectrum domain and retains the near-end voice signal, thereby improving the clarity of the voice signal and improving the voice quality.

[0093] The decoder module 22 determines the spectrum mask corresponding to the superimposed spectrum signal based on the spectrum characteristics, including: decoding the spectrum characteristics through the decoder 221 to obtain an initial spectrum mask; performing frequency band segmentation on the initial spectrum mask through the frequency band segmentation unit 222 to obtain a spectrum mask, and the frequency resolution of the spectrum mask is the same as the frequency resolution of the superimposed spectrum signal.

[0094] That is, the spectrum mask output by the decoder 221 is a spectrum mask after the resolution is compressed, and needs to be further restored to the same frequency resolution as the input signal of the encoder through the frequency band division unit 222 .

[0095] In this embodiment, the parameter setting of the decoder 221 is symmetrical to that of the encoder, and a two-dimensional deconvolution layer is also used to restore the frequency resolution of the superimposed spectrum signal, so as to restore the frequency resolution of the spectrum feature to 161.

[0096] Figure 8 A schematic diagram of the structure of another decoder module provided in an embodiment of the present application. Figure 8 and Figure 5 Correspondingly, Figure 5 Correspondingly, the decoder module 22 includes a decoder 221. The decoder 221 is used to determine a spectrum mask based on the spectrum feature, and the frequency resolution of the spectrum mask is the same as the frequency resolution of the original microphone signal. Then, the decoder module 22 determines the spectrum mask corresponding to the superimposed spectrum signal based on the spectrum feature, including: decoding the spectrum feature through the decoder 221 to obtain the spectrum mask.

[0097] In some embodiments, in order to reduce information loss during the downsampling process, the encoder 211 and the decoder 221 can also be jump-connected to help retain key details in the signal and minimize information loss. At the same time, the jump connection can also promote more efficient gradient propagation during back propagation, which is helpful for model training.

[0098] Continue reading Figure 3 After step S302, this embodiment may further include the following step S303.

[0099] S303 : Multiply the original microphone signal and the spectrum mask by the complex ratio mask module 23 to eliminate the interference signal in the original microphone signal and obtain a clean microphone signal.

[0100] The spectrum mask can be considered as a two-dimensional matrix, in which each element corresponds to a specific frequency component and time point. The spectrum mask is a matrix with the same spectrum dimension as the spectrum signal corresponding to the original microphone signal, and its values ​​are between 0 and 1. When the mask value corresponding to an element is close to 1, it means that the corresponding spectrum component will be completely retained; when the mask value corresponding to an element is close to 0, it means that the corresponding spectrum component will be suppressed or removed.

[0101] The spectrum mask can be precisely adjusted for each frequency component in each time frame, that is, it can design a mask based on the spectral characteristics of the echo signal and the noise signal so that these unwanted components are suppressed. At the same time, the spectrum mask can also retain important features in the original microphone signal, thereby ensuring that the processed signal still retains the quality of the original speech.

[0102] The spectral mask has small complex values ​​at the locations of the echo and noise spectra, and large complex values ​​at the locations of the clean microphone signal spectrum. Applying the spectral mask M(f, t) to the spectrum Y(f, t) of the original microphone signal gives the reconstructed spectrum signal S(f, t) corresponding to the clean microphone signal.

[0103] Since the mask has a smaller value at the echo and noise positions, these components are suppressed, while the mask has a larger value at the speech position, so the speech components are retained. In this way, the CRM module can effectively remove the echo and noise components and restore a high-quality near-end speech signal, that is, a clean microphone signal.

[0104] In order to obtain a more stable output and avoid problems such as destructive interference caused by directly performing element multiplication on the estimated time-frequency block, the following embodiments may also be used:

[0105] Among them, the spectrum signal of the original microphone signal includes multiple time-frequency blocks. The original microphone signal and the spectrum mask are multiplied by the complex ratio mask module 23 to eliminate the interference signal in the original microphone signal and obtain a clean microphone signal, including: for each time-frequency block in the multiple time-frequency blocks, the complex ratio mask module multiplies the time-frequency block and its corresponding adjacent time-frequency blocks with the spectrum mask value at the corresponding position in the spectrum mask to obtain the reconstructed spectrum signal corresponding to the time-frequency block; the reconstructed spectrum signals corresponding to each of the multiple time-frequency blocks are converted to the time domain to obtain a clean microphone signal. The process can be expressed as the following formula:

[0106] ;

[0107] In the formula, I and L represent the fusion window size in the time frame direction and frequency direction respectively. For example, I can be 2 or 3, indicating the fusion of 2 or 3 adjacent time frames in the time direction; L also represents the fusion window size in the frequency direction similarly.

[0108] Assume there is a time-frequency graph Y(f, t), where f represents the frequency component and t represents the time frame. If I=1 and L=1 are selected, it means that one adjacent time frame will be fused in the time direction and one adjacent frequency component will be fused in the frequency direction.

[0109] By padding extra zeros around the spectral signal Y(f, t) of the original microphone signal, each time-frequency block can be correctly reconstructed, thereby maintaining causality in the calculation. This approach improves the robustness of echo cancellation and noise suppression by incorporating contextual information from adjacent time frames and frequency bands, thereby improving the accuracy and stability of the recovered near-end speech signal.

[0110] Fusion of information from adjacent blocks can improve the robustness of the processing results and reduce errors caused by inaccurate estimation of a single time-frequency block. In addition, fusion of information from adjacent blocks can smooth the processing results, avoid mutations and destructive interference, and make the processed signal more natural. In this way, noise and echo can be removed more effectively to obtain high-quality clean near-end speech signals.

[0111] The above embodiment introduces the application process of the speech signal processing model. The speech signal processing model needs to be trained before application. The training process of the speech signal processing model will be introduced below:

[0112] Fig. 9 The flowchart of the training process of the speech signal processing model provided in the embodiment of the present application is as follows. Fig. 9 As shown, the training process includes the following steps S401 to S403:

[0113] S401: Obtain original microphone signal samples including various signal samples.

[0114] Among them, various types of signal samples include clean microphone signal samples and interference signal samples, and the original microphone signal samples correspond to labeled clean microphone signals.

[0115] In this embodiment, the original microphone signal sample refers to the sum of all sounds actually picked up by the microphone, including clean microphone signal samples and interference signal samples. Clean microphone signal samples refer to ideal speech signal samples that are not affected by interference signals. Interference signals include background noise signal samples and echo signal samples corresponding to far-end reference signals. For the definition of far-end reference signals, please refer to the introduction of relevant contents in the aforementioned embodiments, which will not be repeated here.

[0116] In some embodiments, obtaining original microphone signal samples including various types of signal samples includes the following steps a1 to a4:

[0117] Step a1: Obtain clean microphone signal samples and clean far-end reference signal samples.

[0118] Here, the clean microphone signal sample refers to the clear speech signal of the near-end user that the microphone should pick up ideally without noise and echo interference. Usually, these signals are clean speech signals recorded in a laboratory environment or synthesized by other means.

[0119] Clean far-end reference signal samples refer to ideal speech signals transmitted from the far end without being affected by any environmental noise or echo. This signal is used as a supervisory signal when training a speech signal processing model to help the model learn how to recover clean speech signals from original microphone signals with noise and echo.

[0120] Clean far-end reference signal samples can be obtained by recording the voice signal of the far-end user in a laboratory environment using professional recording equipment without background noise and echo. Or, clean voice signals can be generated by synthesis technology. For example, a text-to-speech (TTS) system can be used to generate synthesized speech, which usually does not contain background noise and echo. Or, clean voice clips can be selected from high-quality voice databases as far-end reference signal samples. These databases usually contain processed voice signals without noise and echo.

[0121] Step a2: superimpose the far-end background noise signal and the echo signal on the clean far-end reference signal sample to obtain an initial far-end reference signal sample.

[0122] Step a3: superimpose the near-end background noise signal on the clean microphone signal sample to obtain an initial microphone signal sample.

[0123] Step a4: obtaining original microphone signal samples based on the initial far-end reference signal samples and the initial microphone signal samples.

[0124] Fig.10 Schematic diagram of the process of obtaining the original microphone signal sample provided in the embodiment of the present application. Fig.10As shown in the figure, the clean microphone signal samples and the clean far-end reference signal samples are first superimposed with the noise signal according to the random signal-to-noise ratio (SNR), and convolved with the random room impulse response (RIR) signal to obtain the far-end speech signal and near-end speech signal with noisy and reverberant audio in the random range of [-5dB, 25dB].

[0125] Afterwards, a bandpass filter is applied to the far-end speech signal to simulate the bad acoustic characteristics of the speaker, and random spectrum shaping is applied to the far-end speech signal after passing through the bandpass filter, that is, a second-order FIR filter is applied to the signal.

[0126] ;

[0127] Where r1, r2, r3 and r4 are the coefficients of the FIR filter, which are randomly selected between [-0.5, 0.5].

[0128] Random spectrum shaping is also applied to the near-end speech signal to enhance robustness and simulate the impact of audio signals during transmission.

[0129] Finally, the far-end speech signal after spectrum shaping is convolved with the random delay RIR to obtain the echo signal. The RIR signal is randomly filled with zeros ranging from 0 to 1.25 seconds at the beginning and then convolved with the speech signal to simulate the transmission and processing delay of the signal.

[0130] The echo signal and the near-end speech signal after spectrum shaping are superimposed according to the random signal-to-echo ratio (SER) to obtain the final original microphone signal sample, and the random range is [-15dB, 20dB].

[0131] By randomly setting the far-end reference signal samples or the near-end microphone signal samples to 0, three scenarios of far-end single talk, near-end single talk and double talk can be simulated.

[0132] Through the above data generation and enhancement methods, the real environment can be better simulated to ensure that the model can maintain stable performance in various complex scenarios.

[0133] S402. Determine the spectrum mask sample corresponding to the original microphone signal sample through the speech signal processing model to be trained, and eliminate the interference signal sample in the original microphone signal sample based on the original microphone signal sample and the spectrum mask sample to obtain a predicted clean microphone signal. The spectrum mask sample characterizes the retention degree of various signal samples in the original microphone signal sample.

[0134] Among them, the speech signal processing model to be trained includes a feature extraction module to be trained, a decoder module to be trained and a complex ratio mask module to be trained; the feature extraction module to be trained is used to extract spectrum feature samples from superimposed spectrum signal samples; the decoder module to be trained is used to determine the spectrum mask sample based on the spectrum feature samples; the complex ratio mask module is used to multiply the original microphone signal sample and the spectrum mask sample to eliminate the interference signal sample in the original microphone signal sample to obtain a predicted clean microphone signal.

[0135] The superimposed spectrum signal samples are obtained by superimposing the spectrum signal samples corresponding to the original microphone signal samples and the remote reference signal samples. The spectrum signal samples corresponding to the original microphone signal samples and the remote reference signal samples can be obtained by performing short-time Fourier transform on the original microphone signal samples and the remote reference signal samples.

[0136] Specifically, the spectrum mask sample corresponding to the original microphone signal sample is determined by the speech signal processing model to be trained, and based on the original microphone signal sample and the spectrum mask sample, the interference signal sample in the original microphone signal sample is eliminated to obtain a predicted clean microphone signal, including the following steps b1 to b3:

[0137] Step b1: extracting features from the superimposed spectrum signal samples through the feature extraction module to be trained to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples.

[0138] There are multiple implementation methods for extracting features from the superimposed spectrum signal samples through the feature extraction module to be trained to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples, which can be specifically as follows:

[0139] In some implementations, the feature extraction module to be trained includes a frequency band merging unit to be trained and a feature extraction unit to be trained; the frequency band merging unit to be trained is used to compress the superimposed spectrum signal samples to obtain compressed spectrum signal samples; the feature extraction unit to be trained is used to extract features from the compressed spectrum signal samples to obtain spectrum feature samples. When the frequency band merging unit to be trained is used to compress the superimposed spectrum signal samples, it compresses the spectrum signal samples within the first frequency band of the superimposed spectrum signal samples, and the spectrum signal samples within the first frequency band are the spectrum signal samples corresponding to the interference signal samples.

[0140] The feature extraction module to be trained is used to perform feature extraction on the superimposed spectrum signal samples to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples, including: compressing the spectrum signal samples within the first frequency band in the superimposed spectrum signal samples by the frequency band merging unit to be trained to obtain compressed spectrum signal samples; and extracting features from the compressed spectrum signal samples by the feature extraction unit to be trained to obtain spectrum feature samples.

[0141] Among them, the feature extraction unit to be trained includes an encoder to be trained and a bottleneck layer to be trained; the encoder to be trained is used to extract features from compressed spectrum signal samples to obtain hidden spectrum feature samples corresponding to the compressed spectrum signal samples; the bottleneck layer to be trained is used to perform intra-frame and inter-frame feature extraction on hidden spectrum feature samples corresponding to the compressed spectrum signal samples to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples.

[0142] The feature extraction unit to be trained is used to perform feature extraction on the compressed spectrum signal samples to obtain spectrum feature samples, including: performing feature extraction on the compressed spectrum signal samples by the encoder to be trained to obtain hidden spectrum feature samples corresponding to the compressed spectrum signal samples; performing intra-frame and inter-frame feature extraction on the hidden spectrum feature samples corresponding to the compressed spectrum signal samples by the bottleneck layer to be trained to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples.

[0143] The specific structure of the encoder to be trained can be found in the introduction to the structure of the encoder in the aforementioned embodiment, which will not be described in detail here.

[0144] In this embodiment, the bottleneck layer to be trained includes an internal frame feature extraction unit to be trained and an inter-frame feature extraction unit to be trained. The internal frame feature extraction unit to be trained is used to perform intra-frame feature extraction on hidden spectrum feature samples corresponding to each time frame of the compressed spectrum signal sample to obtain intra-frame feature samples corresponding to each time frame of the compressed spectrum signal sample; the inter-frame feature extraction unit to be trained is used to perform inter-frame feature extraction on intra-frame feature samples corresponding to each time frame of the compressed spectrum signal sample to obtain spectrum feature samples corresponding to the compressed spectrum signal sample.

[0145] Then, intra-frame and inter-frame feature extraction is performed on hidden spectrum feature samples corresponding to the compressed spectrum signal samples through the bottleneck layer to be trained to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples, including: intra-frame feature extraction is performed on hidden spectrum feature samples corresponding to each time frame of the compressed spectrum signal samples through the internal frame feature extraction unit to be trained to obtain intra-frame feature samples corresponding to each time frame of the compressed spectrum signal samples; inter-frame feature extraction is performed on intra-frame feature samples corresponding to each time frame of the compressed spectrum signal samples through the inter-frame feature extraction unit to be trained to obtain spectrum feature samples corresponding to the compressed spectrum signal samples.

[0146] In some implementations, the feature extraction module to be trained includes a feature extraction unit to be trained, which is used to perform feature extraction on the superimposed spectrum signal samples to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples.

[0147] Among them, the feature extraction unit to be trained includes an encoder to be trained and a bottleneck layer to be trained; the encoder to be trained is used to perform feature extraction on the superimposed spectrum signal samples to obtain hidden spectrum feature samples corresponding to the superimposed spectrum signal samples; the bottleneck layer to be trained is used to perform intra-frame and inter-frame feature extraction on the hidden spectrum feature samples corresponding to the superimposed spectrum signal samples to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples.

[0148] The feature extraction unit to be trained is used to perform feature extraction on the superimposed spectrum signal samples to obtain spectrum feature samples, including: performing feature extraction on the superimposed spectrum signal samples by the encoder to be trained to obtain hidden spectrum feature samples corresponding to the superimposed spectrum signal samples; performing intra-frame and inter-frame feature extraction on the hidden spectrum feature samples corresponding to the superimposed spectrum signal samples by the bottleneck layer to be trained to obtain spectrum feature samples corresponding to the superimposed spectrum signal samples.

[0149] After step b1, this embodiment may further include the following step b2.

[0150] Step b2: determining a spectrum mask sample corresponding to the superimposed spectrum signal sample based on the spectrum feature sample by the decoder module to be trained.

[0151] In some implementations, the decoder module to be trained includes a decoder to be trained and a frequency band segmentation unit to be trained. The decoder to be trained is used to determine an initial spectrum mask sample based on the spectrum feature samples, and the frequency resolution of the initial spectrum mask sample is the same as the frequency resolution of the compressed spectrum signal sample. The frequency band segmentation unit to be trained is used to perform frequency band segmentation based on the initial spectrum mask sample to obtain a spectrum mask sample. The spectrum mask sample finally obtained has the same frequency resolution corresponding to the spectrum signal sample of the original microphone signal sample.

[0152] The spectrum mask sample corresponding to the superimposed spectrum signal sample is determined based on the spectrum feature samples by the decoder module to be trained, including: decoding the spectrum feature samples by the decoder to be trained to obtain an initial spectrum mask sample; performing frequency band segmentation on the initial spectrum mask sample by the frequency band segmentation unit to be trained to obtain a spectrum mask sample, and the frequency resolution of the spectrum mask sample is the same as the frequency resolution of the superimposed spectrum signal sample.

[0153] In this embodiment, the parameter setting of the decoder to be trained is symmetrical with that of the encoder, and a two-dimensional deconvolution layer is also used to restore the frequency resolution of the superimposed spectrum signal.

[0154] In some implementations, the decoder module to be trained includes a decoder to be trained. The decoder to be trained is used to determine a spectrum mask sample based on the spectrum feature samples, and the frequency resolution of the spectrum mask sample is the same as the frequency resolution of the original microphone signal sample. Then, the spectrum mask sample corresponding to the superimposed spectrum signal sample is determined by the decoder module to be trained based on the spectrum feature samples, including: decoding the spectrum feature samples by the decoder to be trained to obtain the spectrum mask sample.

[0155] In some embodiments, in order to reduce information loss during the downsampling process, the encoder to be trained and the decoder to be trained can also be jump-connected to help retain key details in the signal and minimize information loss. At the same time, the jump connection can also promote more efficient gradient propagation during backpropagation, which is helpful for model training.

[0156] After step b2, this embodiment may further include the following step b3.

[0157] Step b3: multiplying the original microphone signal samples and the spectrum mask samples by the complex ratio mask module to be trained to eliminate the interference signal samples in the original microphone signal samples to obtain a predicted clean microphone signal.

[0158] The specific implementation process of step b3 is similar to the specific implementation process of step S303. For the specific implementation process of step b3, please refer to the introduction of step S303, which will not be repeated here.

[0159] Steps b1 to b3 are to process the original microphone signal samples through the speech signal processing model to be trained to obtain a predicted clean microphone signal. This process includes the identification and suppression of noise and echo, in order to generate a result that is as close to the real clean microphone signal as possible.

[0160] Continue reading Fig. 9 , after step S402, step S403 may also be included.

[0161] S403: Determine a signal prediction loss based on the difference between the predicted clean microphone signal and the labeled clean microphone signal, and converge the speech signal processing model to be trained based on the signal prediction loss to obtain a speech signal processing model.

[0162] By comparing the difference between the predicted clean microphone signal and the labeled clean microphone signal, the signal prediction loss can be calculated. During the training process, the signal prediction loss is gradually reduced by continuously adjusting the parameters of the model. Specifically, the back propagation algorithm can be used to calculate the gradient of the signal prediction loss relative to the model parameters, and the model parameters can be updated according to a certain learning rate to minimize the signal prediction loss. The signal prediction loss can use loss functions such as Mean Squared Error (MSE) and Root Mean Squared Error (RMSE). These loss functions can quantify the gap between the predicted signal and the true signal, and are used to guide the model training process.

[0163] The training process usually consists of multiple iterations, in which the model processes the training data multiple times and adjusts the parameters based on the value feedback of the loss function. This process continues until the model converges, that is, the signal prediction loss is less than the preset loss.

[0164] Through the above training process, a speech signal processing model that can effectively handle noise and echo interference can be obtained. In practical applications, this model can recover a speech signal close to the real clean microphone signal from the original microphone signal with noise and echo.

[0165] In summary, the technical solution proposed in this application has the following beneficial effects:

[0166] (1) The superimposed spectrum signal is compressed through the frequency band merging module, especially the high-frequency part containing more noise components, which can reduce its frequency resolution, thereby reducing the computational complexity of subsequent processing. The compressed spectrum signal is then downsampled through the encoder to further reduce the computational complexity. The features output by the encoder are further extracted through the bottleneck layer to generate highly abstract feature representations, namely spectral features, to further reduce the number of features, significantly reducing the number of parameters and computing requirements, and realizing a lightweight end-to-end neural network model that can efficiently process AEC and NS tasks at the same time, suitable for resource-constrained devices, and maintain good performance in complex scenarios.

[0167] (2) The superimposed spectrum signal is compressed through the frequency band merging module, especially the high-frequency part containing more noise components, thereby reducing the impact of noise. The encoder extracts key features useful for AEC and NS tasks from the compressed spectrum signal. The bottleneck layer further extracts features from the features output by the encoder to generate a highly abstract feature representation, namely the spectrum feature. The spectrum feature contains the most valuable information for removing echo and noise. The decoder upsamples the spectrum features output by the bottleneck layer to restore the frequency resolution close to the spectrum signal of the original microphone signal. This process allows the feature information to be distributed in the frequency domain, preparing for subsequent complex ratio mask processing. In the process of generating the spectrum mask, the neural network or other machine learning model will learn the characteristics of the input signal. For example, it will learn the spectral distribution law of echo and noise. Therefore, when applied to speech signal processing, it can automatically identify and eliminate noise signals and echo signals to obtain clear and clean microphone signals.

[0168] (3) When processing acoustic echo cancellation (AEC) and noise suppression (NS) tasks, the original microphone signal and the far-end reference signal are superimposed, and then certain intermediate features are extracted and shared through the model to achieve collaborative work between tasks. The model can process AEC and NS tasks simultaneously within the same framework, avoiding repeated calculations and reducing the computational burden of the model.

[0169] Corresponding to the above-mentioned speech signal processing method, the present application embodiment also proposes a speech signal processing device, such as Fig.11 As shown, the device includes: an acquisition unit 110, a determination unit 111 and an elimination unit 112; wherein the acquisition unit 110 is used to acquire an original microphone signal, wherein the original microphone signal includes a clean microphone signal and an interference signal; the determination unit 111 is used to determine a spectrum mask corresponding to the original microphone signal, wherein the spectrum mask characterizes the degree of retention of various types of signals in the original microphone signal; the elimination unit 112 is used to eliminate the interference signal in the original microphone signal based on the original microphone signal and the spectrum mask to obtain the clean microphone signal.

[0170] In some embodiments, the interference signal includes a background noise signal and an echo signal corresponding to a far-end reference signal; the determination unit 111 determines a spectrum mask corresponding to the original microphone signal, and based on the original microphone signal and the spectrum mask, eliminates the interference signal in the original microphone signal to obtain the clean microphone signal, including: superimposing the spectrum signals corresponding to the original microphone signal and the far-end reference signal to obtain a superimposed spectrum signal, and the far-end reference signal is used to assist a speech recognition model in identifying the echo signal in the original microphone signal; inputting the superimposed spectrum signal into a speech signal processing model, and the speech signal processing model determines the spectrum mask corresponding to the superimposed spectrum signal, and based on the original microphone signal and the spectrum mask, eliminates the interference signal in the original microphone signal to obtain the clean microphone signal.

[0171] In some embodiments, the speech signal processing model includes a feature extraction module, a decoder module and a complex ratio mask module; wherein the determination unit 111 determines the spectrum mask corresponding to the superimposed spectrum signal by the speech processing model, and eliminates the interference signal in the original microphone signal based on the original microphone signal and the spectrum mask to obtain the clean microphone signal, including: performing feature extraction on the superimposed spectrum signal through the feature extraction module to obtain the spectrum feature corresponding to the superimposed spectrum signal; determining the spectrum mask corresponding to the superimposed spectrum signal based on the spectrum feature through the decoder module; multiplying the original microphone signal and the spectrum mask through the complex ratio mask module to eliminate the interference signal in the original microphone signal to obtain the clean microphone signal.

[0172] In some embodiments, the feature extraction module includes a frequency band merging unit and a feature extraction unit; wherein the determination unit 111 performs feature extraction on the superimposed spectrum signal through the feature extraction module to obtain a spectrum feature corresponding to the superimposed spectrum signal, including: compressing the spectrum signal within the first frequency band in the superimposed spectrum signal through the frequency band merging unit to obtain a compressed spectrum signal, the spectrum signal within the first frequency band being the spectrum signal corresponding to the interference signal; performing feature extraction on the compressed spectrum signal through the feature extraction unit to obtain the spectrum feature.

[0173] In some embodiments, the feature extraction unit includes an encoder and a bottleneck layer; wherein the determination unit 111 performs feature extraction on the compressed spectrum signal through the feature extraction unit to obtain the spectrum feature, including: encoding the compressed spectrum signal through the encoder to obtain hidden spectrum features corresponding to the compressed spectrum signal; performing intra-frame and inter-frame feature extraction on the hidden spectrum features through the bottleneck layer to obtain spectrum features corresponding to the superimposed spectrum signal.

[0174] In some embodiments, the decoder module includes a decoder and a frequency band splitting unit; wherein the determination unit 111 determines the spectrum mask corresponding to the superimposed spectrum signal based on the spectrum characteristics through the decoder module, including: decoding the spectrum characteristics through the decoder to obtain an initial spectrum mask; performing frequency band splitting on the initial spectrum mask through the frequency band splitting unit to obtain the spectrum mask, and the frequency resolution of the spectrum mask is the same as the frequency resolution of the superimposed spectrum signal.

[0175] In some embodiments, the spectrum signal of the original microphone signal includes multiple time-frequency blocks; wherein the determination unit 111 multiplies the original microphone signal and the spectrum mask through a complex ratio mask module to eliminate the interference signal in the original microphone signal to obtain the clean microphone signal, including: for each time-frequency block in the multiple time-frequency blocks, multiplying the time-frequency block and its corresponding adjacent time-frequency blocks with the spectrum mask value at the corresponding position in the spectrum mask through a complex ratio mask module to obtain a reconstructed spectrum signal corresponding to the time-frequency block; converting the reconstructed spectrum signals corresponding to each of the multiple time-frequency blocks to the time domain to obtain the clean microphone signal.

[0176] In some embodiments, the speech signal processing model is trained by the following steps: obtaining original microphone signal samples including various types of signal samples, wherein the various types of signal samples include clean microphone signal samples and interference signal samples, and the original microphone signal samples correspond to labeled clean microphone signals; determining a spectrum mask sample corresponding to the original microphone signal sample through the speech signal processing model to be trained, and based on the original microphone signal samples and the spectrum mask sample, eliminating the interference signal samples in the original microphone signal samples to obtain a predicted clean microphone signal, wherein the spectrum mask sample characterizes the degree of retention of various types of signal samples in the original microphone signal samples; determining a signal prediction loss based on the difference between the predicted clean microphone signal and the labeled clean microphone signal, and converging the speech signal processing model to be trained based on the signal prediction loss to obtain a speech signal processing model.

[0177] In some embodiments, the acquisition unit 110 acquires original microphone signal samples including various types of signal samples, including: acquiring clean microphone signal samples and clean far-end reference signal samples; superimposing a far-end background noise signal and an echo signal on the clean far-end reference signal samples to obtain an initial far-end reference signal sample; superimposing a near-end background noise signal on the clean microphone signal samples to obtain an initial microphone signal sample; and obtaining the original microphone signal sample based on the initial far-end reference signal sample and the initial microphone signal sample.

[0178] The speech signal processing device provided in this embodiment belongs to the same application concept as the speech signal processing method provided in the above embodiments of this application, can execute the speech signal processing method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the speech signal processing method. For the technical details not described in detail in this embodiment, please refer to the specific processing content of the speech signal processing method provided in the above embodiments of this application, which will not be repeated here.

[0179] The functions implemented by each unit in the above speech signal processing device can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0180] It should be understood that the units in the above speech signal processing device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in the form of a processor calling software, and the remaining part can be implemented in the form of a hardware circuit.

[0181] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0182] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0183] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0184] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to a data input component via the interface circuit of the control device.

[0185] The data input component specifically refers to a functional component capable of inputting or collecting voice data, such as a microphone, etc.

[0186] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0187] The processor in the control device is also a circuit with signal processing capability, which executes the voice signal processing method introduced in the above embodiment. The specific implementation of the processor can refer to the above processor implementation, and the embodiment of the present application is not strictly limited.

[0188] Another embodiment of the present application also provides an electronic device, see Fig.12As shown, the device includes:

[0189] Memory 200 and processor 210;

[0190] The memory 200 is connected to the processor 210 and is used to store programs;

[0191] The processor 210 is used to implement the speech signal processing method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0192] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0193] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0194] A bus may include a pathway that transfers information between components of a computer system.

[0195] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0196] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0197] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0198] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0199] Output device 240 may include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0200] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0201] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement each step of any speech signal processing method provided in the above embodiments of the present application.

[0202] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the speech signal processing method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the description of the embodiment of the above speech signal processing method.

[0203] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech signal processing method described in any of the above embodiments of this specification.

[0204] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0205] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the speech signal processing method described in any of the above embodiments of this specification.

[0206] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0207] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0208] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0209] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0210] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0211] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0213] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0214] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0215] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0216] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech signal processing method, characterized in that: include: Determine the spectrum mask corresponding to the original microphone signal through the speech signal processing model, and based on the original microphone signal and the spectrum mask, eliminate the interference signal in the original microphone signal to obtain a clean microphone signal; the spectrum mask is complex domain information, which characterizes the degree of retention of various types of signals in the original microphone signal; The speech signal processing model compresses the spectrum signal in the superimposed spectrum signal within the first frequency band through a frequency band merging unit to obtain a compressed spectrum signal, and determines the spectrum mask corresponding to the superimposed spectrum signal by extracting the spectrum features of the compressed spectrum signal; The spectrum signal within the first frequency band is the spectrum signal corresponding to the interference signal, and the superimposed spectrum signal is obtained by superimposing the spectrum signals corresponding to the original microphone signal and the remote reference signal.

2. The method according to claim 1, characterized in that The speech signal processing model includes a feature extraction module, a decoder module and a complex ratio mask module; The method includes determining a spectrum mask corresponding to an original microphone signal through a speech signal processing model, and eliminating an interference signal in the original microphone signal based on the original microphone signal and the spectrum mask to obtain a clean microphone signal, including: The feature extraction module extracts features from the superimposed spectrum signal to obtain spectrum features corresponding to the superimposed spectrum signal; the frequency band merging unit compresses the spectrum signal within the first frequency band in the superimposed spectrum signal to obtain a compressed spectrum signal, and the spectrum mask corresponding to the superimposed spectrum signal is determined by extracting the spectrum features of the compressed spectrum signal; Determining, by the decoder module, a spectrum mask corresponding to the superimposed spectrum signal based on the spectrum feature; The original microphone signal is multiplied by the spectrum mask through a complex ratio mask module to eliminate interference signals in the original microphone signal and obtain the clean microphone signal.

3. The method according to claim 2, characterized in that The feature extraction module includes the frequency band merging unit and a feature extraction unit; The feature extraction module extracts features from the superimposed spectrum signal to obtain spectrum features corresponding to the superimposed spectrum signal, including: The frequency spectrum signal in the superimposed frequency spectrum signal within the first frequency band is compressed by the frequency band merging unit to obtain a compressed frequency spectrum signal, wherein the frequency spectrum signal within the first frequency band is a frequency spectrum signal corresponding to the interference signal; The feature extraction unit extracts features from the compressed spectrum signal to obtain the spectrum features.

4. The method according to claim 3, characterized in that: The feature extraction unit includes an encoder and a bottleneck layer; The feature extraction unit extracts features from the compressed spectrum signal to obtain the spectrum features, including: Encoding the compressed spectrum signal by the encoder to obtain hidden spectrum features corresponding to the compressed spectrum signal; The hidden spectrum features are extracted intra-frame and inter-frame through the bottleneck layer to obtain spectrum features corresponding to the superimposed spectrum signal.

5. The method according to claim 4, characterized in that The decoder module includes a decoder and a frequency band splitting unit; Wherein, determining, by the decoder module based on the spectrum feature, a spectrum mask corresponding to the superimposed spectrum signal comprises: Decoding the spectrum feature by the decoder to obtain an initial spectrum mask; The frequency band segmentation unit performs frequency band segmentation on the initial frequency band mask to obtain the frequency band mask, and the frequency resolution of the frequency band mask is the same as the frequency resolution of the superimposed spectrum signal.

6. The method according to claim 2, characterized in that The spectrum signal of the original microphone signal includes a plurality of time-frequency blocks; The method of multiplying the original microphone signal and the spectrum mask by a complex ratio mask module to eliminate interference signals in the original microphone signal to obtain the clean microphone signal includes: For each time-frequency block in the multiple time-frequency blocks, multiply the time-frequency block and its corresponding adjacent time-frequency block by the spectrum mask value at the corresponding position in the spectrum mask through a complex ratio mask module to obtain a reconstructed spectrum signal corresponding to the time-frequency block; The reconstructed spectrum signals corresponding to each of the multiple time-frequency blocks are converted into the time domain to obtain the clean microphone signal.

7. The method according to any one of claims 1 to 6, characterized in that: The speech signal processing model is trained by the following steps: Acquire original microphone signal samples including various signal samples, wherein the various signal samples include clean microphone signal samples and interference signal samples, and the original microphone signal samples correspond to labeled clean microphone signals; Determine a spectrum mask sample corresponding to the original microphone signal sample by a speech signal processing model to be trained, and eliminate interference signal samples in the original microphone signal sample based on the original microphone signal sample and the spectrum mask sample to obtain a predicted clean microphone signal, wherein the spectrum mask sample characterizes the degree of retention of various signal samples in the original microphone signal sample; Based on the difference between the predicted clean microphone signal and the labeled clean microphone signal, a signal prediction loss is determined, and based on the signal prediction loss, the speech signal processing model to be trained is converged to obtain a speech signal processing model.

8. The method according to claim 7, characterized in that The obtaining of original microphone signal samples including various types of signal samples comprises: Acquire clean microphone signal samples and clean far-end reference signal samples; Superimposing a far-end background noise signal and an echo signal on the clean far-end reference signal sample to obtain an initial far-end reference signal sample; Superimposing a near-end background noise signal on the clean microphone signal sample to obtain an initial microphone signal sample; The original microphone signal samples are obtained based on the initial far-end reference signal samples and the initial microphone signal samples.

9. A speech signal processing device, characterized in that: include: An acquisition unit, configured to acquire an original microphone signal, wherein the original microphone signal includes a clean microphone signal and an interference signal; A determination unit, configured to determine a spectrum mask corresponding to the original microphone signal through a speech signal processing model, and eliminate interference signals in the original microphone signal based on the original microphone signal and the spectrum mask to obtain the clean microphone signal; the spectrum mask is complex domain information, characterizing the degree of retention of various types of signals in the original microphone signal; The speech signal processing model compresses the spectrum signal in the superimposed spectrum signal within the first frequency band through a frequency band merging unit to obtain a compressed spectrum signal, and determines the spectrum mask corresponding to the superimposed spectrum signal by extracting the spectrum features of the compressed spectrum signal; The spectrum signal within the first frequency band is the spectrum signal corresponding to the interference signal, and the superimposed spectrum signal is obtained by superimposing the spectrum signals corresponding to the original microphone signal and the remote reference signal.

10. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 8.

Images