Echo cancellation method and device based on positive and negative sample comparative implicit modeling
Through the method based on contrast implicit modeling of positive and negative samples, the problems of echo residual and speech damage in existing echo cancellation methods are solved, and efficient echo cancellation is achieved, reducing the computational complexity and cost.
Patent Information
- Application Number
- CN202411039490.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-31
AI Technical Summary
Existing echo cancellation methods are prone to serious echo residual problems and proximal speech damage problems, and the model's redundancy and computational cost are high.
Using a method based on positive and negative samples to compare implicit modeling, short-time Fourier transform is performed by obtaining the mixed signal and remote signal of the target full-duplex voice communication system, extracting the time spectrum information and self-attention based on preset comparison Force strategy acquires speech information and echo information, builds speech prediction models, generates mapping representations, and generates predicted speech signals through phase reconstruction subnetwork to eliminate echoes.
It avoids the excessive computational complexity caused by additional modeling echoes, maintains model performance, effectively solves the problems of echo residuals and speech damage, and reduces the degree of model redundancy and computing cost.
Smart Images

Figure CN119068894A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and in particular to an echo cancellation method and device based on implicit modeling of positive and negative sample contrast. Background Art
[0002] In a full-duplex voice communication system, the voice of the remote user (i.e., the far-end signal) is transmitted through the communication link and then played by the local speaker. The local microphone captures this voice and transmits the mixed signal formed by it and the local user's voice (i.e., the near-end voice) to the remote end, so that the remote user can hear his or her own voice. This phenomenon is called acoustic echo, which is formed by the echo path between the speaker and the microphone after the received far-end signal is played by the speaker. The echo signal will seriously affect the user's communication experience. In order to improve the communication quality, the echo signal needs to be removed from the mixed signal. Traditional echo cancellation methods use mathematical models and adaptive algorithms to estimate the acoustic echo path, and then estimate the echo signal and remove it from the local mixed signal. However, traditional echo cancellation methods have limited modeling capabilities for nonlinear echoes and cannot effectively remove nonlinear echoes.
[0003] In recent years, deep learning has been widely used in the field of echo cancellation due to its powerful ability to model nonlinear relationships. Compared with traditional adaptive echo cancellation methods, deep learning-based echo cancellation methods have achieved significant improvements in both echo cancellation and near-end speech retention. Existing deep learning-based echo cancellation methods receive far-end signals and mixed signals and output near-end speech. Most models usually only model near-end speech without modeling echo signals. However, due to the similarity between echo and speech, the method of only modeling near-end speech will lead to serious echo residual problems and near-end speech damage problems. Some models adopt the strategy of simultaneously modeling near-end speech and echo, which has achieved good performance improvement. However, the additional modeling of echo increases the computational complexity of the model, and the echo component is not a component that the system wants to retain. Therefore, the solution of synchronously modeling the echo component and the speech component will increase the redundancy of the model and generate excessive computational costs.
[0004] In summary, the existing echo cancellation methods are prone to cause serious echo residual problems and near-end speech damage problems, and the model has high redundancy and computational cost, which need to be solved urgently. Summary of the invention
[0005] The present application provides an echo cancellation method and device based on implicit modeling of positive and negative sample contrast, so as to solve the problems that the existing echo cancellation methods are prone to cause serious echo residual problems and near-end speech damage problems, and the redundancy of the model and the high computational cost.
[0006] The first aspect of the present application provides an echo cancellation method based on positive and negative sample contrast implicit modeling, comprising the following steps: obtaining a mixed signal and a far-end signal of a target full-duplex voice communication system, and performing a short-time Fourier transform on the mixed signal and the far-end signal to obtain time-frequency spectrum information corresponding to the mixed signal and the far-end signal, respectively, and extracting an initial representation corresponding to the time-frequency spectrum information; obtaining an intermediate representation corresponding to the initial feature and speech information and echo information in a latent space according to a preset contrastive self-attention strategy, and determining positive samples and negative samples through the speech information and the echo information, and based on The positive samples, the negative samples and the preset contrastive learning loss function are used to construct a speech prediction model, and the intermediate representation is input into the speech prediction model to generate a mapping representation; the mapping representation is decoded to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and a subnetwork is reconstructed based on the predicted speech amplitude spectrum, the time-frequency spectrum information and the preset phase to obtain the estimated phase of the target full-duplex voice communication system, and the predicted speech amplitude spectrum and the estimated phase are inversely short-time Fourier transformed to generate a predicted speech signal of the target full-duplex voice communication system with the echo information eliminated.
[0007] Optionally, in one embodiment of the present application, the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space are obtained according to a preset contrastive self-attention strategy, including: decoupling the initial representation to obtain a decoupling result, and performing matrix multiplication processing on the query data and key data in the decoupling result to generate a feature spectrum; obtaining the intermediate representation based on the feature spectrum and the value data in the decoupling result; performing a mask generation operation on the feature spectrum to obtain a mask corresponding to the feature spectrum, and generating the speech information and the echo information according to the mask and a preset feature threshold.
[0008] Optionally, in one embodiment of the present application, the decoding of the mapping representation to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system includes: decoding the mapping representation based on a preset jump connection strategy and a preset decoding network layer to generate a mask corresponding to the mapping representation; multiplying the mask corresponding to the mapping representation by the amplitude spectrum in the time-frequency spectrum information of the mixed signal point by point to obtain the predicted speech amplitude spectrum of the target full-duplex voice communication system.
[0009] Optionally, in one embodiment of the present application, the estimated phase of the target full-duplex voice communication system is obtained based on the predicted speech amplitude spectrum, the time-frequency spectrum information and the preset phase reconstruction subnetwork, and the predicted speech amplitude spectrum and the estimated phase are inversely short-time Fourier transformed to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information, including: merging the predicted speech amplitude spectrum and the time-frequency spectrum information to obtain a merging result, and inputting the merging result to the preset phase reconstruction subnetwork to output the estimated phase of the target full-duplex voice communication system; performing an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to obtain a target speech waveform; and constructing a joint loss function based on a preset amplitude loss function, a preset SI-SNR loss function and a preset contrast learning loss function to obtain the predicted speech signal of the target full-duplex voice communication system through the joint loss function, the predicted speech amplitude spectrum and the target speech waveform.
[0010] The second aspect of the present application provides an echo cancellation device based on positive and negative sample contrast implicit modeling, including: an extraction unit, used to obtain a mixed signal and a far-end signal of a target full-duplex voice communication system, and perform a short-time Fourier transform on the mixed signal and the far-end signal to obtain time-frequency spectrum information corresponding to the mixed signal and the far-end signal, respectively, and extract an initial representation corresponding to the time-frequency spectrum information; a modeling module, used to preset a contrastive self-attention strategy to obtain an intermediate representation corresponding to the initial feature and voice information and echo information in a latent space, and determine positive samples and negative samples through the voice information and the echo information, and based on A speech prediction model is constructed based on the positive samples, the negative samples and the preset contrast learning loss function, and the intermediate representation is input into the speech prediction model to generate a mapping representation; a generation module is used to decode the mapping representation to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and based on the predicted speech amplitude spectrum, the time-frequency spectrum information and the preset phase reconstruction subnetwork, an estimated phase of the target full-duplex voice communication system is obtained, and an inverse short-time Fourier transform is performed on the predicted speech amplitude spectrum and the estimated phase to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information.
[0011] Optionally, in one embodiment of the present application, the modeling module includes: a decoupling unit, used to decouple the initial representation to obtain a decoupling result, and perform matrix multiplication processing on the query data and key data in the decoupling result to generate a feature spectrum; an acquisition unit, used to acquire the intermediate representation based on the feature spectrum and the value data in the decoupling result; a mask unit, used to perform a mask generation operation on the feature spectrum to obtain a mask corresponding to the feature spectrum, and generate the voice information and the echo information according to the mask and a preset feature threshold.
[0012] Optionally, in one embodiment of the present application, the generation module includes: a decoding unit, used to decode the mapping representation based on a preset jump connection strategy and a preset decoding network layer to generate a mask corresponding to the mapping representation; a multiplication unit, used to point-by-point multiply the mask corresponding to the mapping representation with the amplitude spectrum in the time-frequency spectrum information of the mixed signal to obtain the predicted speech amplitude spectrum of the target full-duplex voice communication system.
[0013] Optionally, in one embodiment of the present application, the generation module also includes: a merging unit, used to merge the predicted speech amplitude spectrum and the time-frequency spectrum information to obtain a merging result, and input the merging result into the preset phase reconstruction subnetwork to output the estimated phase of the target full-duplex voice communication system; a transformation unit, used to perform an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to obtain a target speech waveform; a prediction unit, used to construct a joint loss function based on a preset amplitude loss function, a preset SI-SNR loss function and a preset contrast learning loss function, so as to obtain the predicted speech signal of the target full-duplex voice communication system through the joint loss function, the predicted speech amplitude spectrum and the target speech waveform.
[0014] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the echo cancellation method based on implicit modeling of contrast between positive and negative samples as described in the above embodiment.
[0015] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned echo cancellation device based on implicit modeling of contrast between positive and negative samples.
[0016] The fifth aspect of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned echo cancellation method based on implicit modeling of contrast between positive and negative samples.
[0017] Therefore, the embodiments of the present application have the following beneficial effects:
[0018] The embodiment of the present application can obtain the mixed signal and the far-end signal of the target full-duplex voice communication system, and perform short-time Fourier transform on the mixed signal and the far-end signal to obtain the time-frequency spectrum information corresponding to the mixed signal and the far-end signal respectively, and extract the initial representation corresponding to the time-frequency spectrum information; according to the preset contrastive self-attention strategy, the intermediate representation corresponding to the initial feature and the voice information and echo information in the latent space are obtained, and the positive sample and the negative sample are determined by the voice information and the echo information, and the voice prediction model is constructed based on the positive sample, the negative sample and the preset contrastive learning loss function, and the intermediate representation is input into the voice prediction model to generate a mapping representation; the mapping representation is decoded to generate the predicted voice amplitude spectrum of the target full-duplex voice communication system, and the estimated phase of the target full-duplex voice communication system is obtained based on the predicted voice amplitude spectrum, the time-frequency spectrum information and the preset phase reconstruction subnetwork, and the predicted voice amplitude spectrum and the estimated phase are inversely short-time Fourier transformed to generate the predicted voice signal of the target full-duplex voice communication system that eliminates the echo information. The contrastive implicit modeling method of the present application can avoid the excessively high computational complexity caused by the additional modeling echo and maintain the effect of the model performance. This solves the problems that existing echo cancellation methods easily lead to serious echo residual problems and near-end speech damage, as well as the high redundancy of the model and high computational cost.
[0019] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0021] Figure 1 A flowchart of an echo cancellation method based on implicit modeling of positive and negative sample contrast according to an embodiment of the present application;
[0022] Figure 2 A schematic diagram of the execution logic of an echo cancellation method based on implicit modeling of positive and negative sample contrast provided by an embodiment of the present application;
[0023] Figure 3 A schematic diagram of a coding module flow chart provided for an embodiment of the present application;
[0024] Figure 4 A schematic diagram of a comparative self-attention module flow chart provided for one embodiment of the present application;
[0025] Figure 5A schematic diagram of a time-frequency-GRU layer flow chart provided for an embodiment of the present application;
[0026] Figure 6 A schematic diagram of a decoding module flow chart provided for an embodiment of the present application;
[0027] Figure 7 A schematic diagram of a phase reconstruction subnetwork process provided for an embodiment of the present application;
[0028] Figure 8 is an example diagram of an echo cancellation device based on implicit modeling of positive and negative sample contrast according to an embodiment of the present application;
[0029] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0030] Among them, 10-an echo cancellation device based on implicit modeling of contrast between positive and negative samples; 100-an extraction module, 200-a modeling module, 300-a generation module; 901-a memory, 902-a processor, 903-a communication interface. DETAILED DESCRIPTION
[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0032] The following describes an echo cancellation method and device based on implicit modeling of positive and negative sample contrast according to an embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides an echo cancellation method based on positive and negative sample contrast implicit modeling, in which the mixed signal and the far-end signal of the target full-duplex voice communication system are obtained, and the mixed signal and the far-end signal are subjected to short-time Fourier transform to obtain the time-frequency spectrum information corresponding to the mixed signal and the far-end signal respectively, and the initial representation corresponding to the time-frequency spectrum information is extracted; the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space are obtained according to the preset contrastive self-attention strategy, and the positive sample and the negative sample are determined by the speech information and the echo information, and a speech prediction model is constructed based on the positive sample, the negative sample and the preset contrastive learning loss function, and the intermediate representation is input into the speech prediction model to generate a mapping representation; the mapping representation is decoded to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and the estimated phase of the target full-duplex voice communication system is obtained based on the predicted speech amplitude spectrum, the time-frequency spectrum information and the preset phase reconstruction subnetwork, and the predicted speech amplitude spectrum and the estimated phase are subjected to inverse short-time Fourier transform to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information. The comparative implicit modeling method of the present application can avoid the excessive computational complexity caused by the additional modeling echo and maintain the effect of model performance. Thus, the existing echo cancellation method easily leads to serious echo residual problems and near-end speech damage problems, and the redundancy of the model and high computational cost are solved.
[0033] Specifically, Figure 1 A flowchart of an echo cancellation method based on implicit modeling of positive and negative sample contrast provided in an embodiment of the present application.
[0034] like Figure 1 As shown, the echo cancellation method based on implicit modeling of positive and negative sample contrast includes the following steps:
[0035] In step S101, a mixed signal and a far-end signal of a target full-duplex voice communication system are acquired, and short-time Fourier transform is performed on the mixed signal and the far-end signal to obtain time-frequency spectrum information corresponding to the mixed signal and the far-end signal, respectively, and an initial representation corresponding to the time-frequency spectrum information is extracted.
[0036] The embodiment of the present application can first obtain the mixed signal and the far-end signal of the target full-duplex voice communication system, and use short-time Fourier transform on the far-end signal y(n) and the mixed signal x(n). The organization method of each signal is (batch size, time length), and the corresponding time-frequency spectra Y and X are obtained respectively, and the amplitude spectrum Y m , X m Input into the encoding module, such as Figure 2As shown, the initial representation W0 is obtained, each amplitude spectrum is organized as (batch size, channel dimension, frequency dimension, time dimension), and the initial representation is organized as (batch size, channel dimension, frequency dimension, time dimension).
[0037] Among them, the encoding module contains four convolution blocks, each of which contains a 2D convolution layer, a batch normalization layer and a PReLU activation function, and the output of each convolution block is fed to the decoding module through jump connections.
[0038] As a feasible method, for remote signals and mixed signals, the batch size of the embodiment of the present application in the training phase can be set to 16, the signal length is 4s, and the sampling frequency is 16KHz; the batch size in the inference phase can be set to 3, the signal length is 10s, and the sampling frequency is 16KHz. Before each input, the segment will be cut into multiple 4s segments, and the part less than 4s will be filled with zeros; the frame length and frame shift of the short-time Fourier transform are 512 and 256 respectively, the window length of the fast Fourier transform is 512, and the time-frequency spectrum Y and X after the short-time Fourier transform have a channel dimension of 2, which are the real part and the imaginary part respectively, a frequency dimension of 257, and a time dimension of 251.
[0039] In the embodiment of the present application, the amplitude spectrum Y of each signal can be obtained by calculating the square sum of the real part and the imaginary part of each signal and then taking the square root. m , X m ; Amplitude spectrum Y m , X m The data is merged into a channel dimension of 2 and input into the encoding module to generate the initial representation. Figure 3 As shown in the figure, it consists of 4 convolutional blocks with similar structures. Each convolutional block contains a 2D convolutional layer, a batch normalization layer, an activation function and a skip connection.
[0040] Specifically, the input channels of the four convolutional layers are 2, 16, 32, and 64, respectively, and the output channels are 16, 32, 64, and 128, respectively. The convolution kernel size of each convolutional layer is (5, 3), the step size is (2, 1), and the padding is (2, 2). The first dimension is the frequency dimension, and the second dimension is the time dimension. In particular, the embodiment of the present application uses causal convolution, and the number of channels of each batch normalization is consistent with the output channel of the corresponding previous convolutional layer. The activation functions of the four convolutional blocks are unified as the PReLU function, and the jump connection in each layer of the convolutional block transmits the output of the current convolution block to the corresponding transposed convolution block in the decoding module.
[0041] In the encoding phase, due to the existence of the convolutional layer, the frequency dimension of the input features will be continuously compressed. Specifically, the frequency dimension of the initial input amplitude spectrum stack is 257, and the frequency dimensions of the output features of the 4-layer convolutional block are 129, 65, 33, and 17 respectively. The time dimension and batch size remain unchanged, and the time dimension is fixed to 251. The encoding module will eventually send the output initial feature W0 to the contrastive self-attention module. W0 is organized as (batch size, channel dimension, frequency dimension, time dimension). The batch size is 16 in the training phase and 3 in the inference phase. The channel dimension is 128, the frequency dimension is 17, and the time dimension is 251.
[0042] Therefore, the embodiment of the present application obtains time-frequency domain features by subjecting the mixed signal and the remote signal to short-time Fourier transform, and obtains the initial representation using a coding module composed of multiple convolution blocks, thereby providing a reliable data basis for the generation of subsequent mapping representations, etc.
[0043] In step S102, the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space are obtained according to the preset contrastive self-attention strategy, and the positive samples and negative samples are determined by the speech information and echo information. Based on the positive samples, negative samples and the preset contrastive learning loss function, a speech prediction model is constructed, and the intermediate representation is input into the speech prediction model to generate a mapping representation.
[0044] Furthermore, the embodiments of the present application can obtain the intermediate representation and the speech information and echo information in the latent space by presetting the contrastive self-attention strategy, that is, the contrastive self-attention module, and the two are respectively regarded as positive samples and negative samples. By contrasting the learning loss function, the difference between the positive and negative samples will be improved, thereby achieving the purpose of implicit modeling; thereafter, the embodiments of the present application also need to input the intermediate representation into the time-frequency-GRU layer to obtain the mapping representation.
[0045] Optionally, in one embodiment of the present application, an intermediate representation corresponding to the initial feature and speech information and echo information in the latent space are obtained according to a preset contrastive self-attention strategy, including: decoupling the initial representation to obtain a decoupling result, and performing matrix multiplication processing on the query data and key data in the decoupling result to generate a feature spectrum; obtaining an intermediate representation based on the feature spectrum and the value data in the decoupling result; performing a mask generation operation on the feature spectrum to obtain a mask corresponding to the feature spectrum, and generating speech information and echo information according to the mask and a preset feature threshold.
[0046] In the actual implementation process, the embodiment of the present application sends the initial feature W0 obtained above to the contrastive self-attention module. The process of the contrastive self-attention module is as follows: Figure 4 shown.
[0047] Specifically, in an embodiment of the present application, first, W0 passes through a convolution block and is decoupled into three components of self-attention: query Q, key K, and value V. The structure of the convolution block is similar to the convolution block of the encoding module mentioned above, including a 2D convolution layer, a batch normalization layer, and an activation function, wherein the 2D convolution layer is used to expand the channel dimension of W0 to three times the original to facilitate subsequent decoupling; in addition, the activation function can select the PReLU function, after which W0 will be reshaped and divided into query Q, key K, and value V, and the three variables are organized in the same way as (batch size * time dimension, frequency dimension, channel dimension); then, the transpose of query Q and key K is matrix multiplied to obtain the feature spectrum F:
[0048] F=QK T
[0049] The feature spectrum F is organized as (batch size * time dimension, frequency dimension, frequency dimension). This process allows the speech component to strengthen its weight, making it easier to distinguish it from the echo component later. Then, the feature spectrum F will be sent to two operation areas:
[0050] Among them, the first operation area is to distinguish the echo component area and the speech component area in F. In the echo cancellation task, most of the negative related information comes from the acoustic echo, and the acoustic echo has the characteristics of being similar to speech and is difficult to be eliminated. Therefore, the speech information and echo information can be regarded as positive sample information and negative sample information (i.e., positive and negative sample features). Since the speech components are aggregated by matrix multiplication, the weights are more prominent; therefore, the embodiment of the present application can identify the two through a discriminant module, which includes a linear layer and a Sigmoid function. After the feature spectrum F passes through the discriminant module, its value will be limited to (0, 1), and a threshold α is set for judgment. The area greater than or equal to α is regarded as a positive sample area, and the area less than α is regarded as a negative sample area; thus, the embodiment of the present application obtains two masks, namely the positive sample mask and the negative sample mask, and the two masks are multiplied point by point with the feature spectrum to obtain the weights of the positive sample information and the negative sample information respectively; then, the embodiment of the present application introduces contrastive learning loss to improve the difference between positive and negative samples. Through this implicit modeling method, echo and speech information are modeled synchronously, and the formula of contrastive learning loss is as follows:
[0051]
[0052] Where N is the number of contrastive self-attention modules, which can be set to 4, and F i and F j are positive sample information and negative sample information, n and m represent the number of regions occupied by speech information and echo information, respectively, and ε is a constant set to 1e-6. c will continue to decline, Fi will continue to improve, and F j It will continue to decrease, thereby increasing the difference between positive and negative samples, thereby achieving the purpose of modeling speech information and echo information.
[0053] The second operation area is to perform subsequent self-attention operations. After passing the feature spectrum F through the Softmax function, it is matrix multiplied with the value V and passed through a convolution layer with a convolution kernel of (1, 1) to obtain the intermediate feature W0. ′ , W0 ′ The organization is (batch size, channel dimension, frequency dimension, time dimension).
[0054] Furthermore, the embodiment of the present application can convert the output W0 of the contrastive self-attention module into ′ Sent to the frequency-time-GRU module, the process of the frequency-time-GRU module is as follows Figure 5 shown.
[0055] Specifically, the module consists of a frequency-GRU layer, a time-GRU layer, and a layer normalization. The frequency-GRU layer and the time-GRU layer process frequency domain information and time information respectively. The composition structure of the frequency-GRU layer is a reshaping module, a unidirectional GRU layer, an activation function, a linear layer, a residual connection, and a reshaping module. The submodules of the frequency-GRU layer are as follows:
[0056] The reshape module can be used to reshape the organization of the input W to facilitate the processing of subsequent modules. Therefore, the reshape module will reshape the organization of W into (batch size * time dimension, frequency dimension, channel dimension);
[0057] The GRU layer is used to process frequency dimension information. The number of hidden layers is 1, and the output dimension of the hidden layer is 128. The activation function is the ReLU function.
[0058] The input and output dimensions of the linear layer are both 128;
[0059] The residual connection part adds the reshaped features to the output features of the linear layer, and then sends them to the second reshaping module. The task of this reshaping module is to restore the organization of the output features of the residual connection to (batch size, channel dimension, frequency dimension, time dimension). The output features of the second reshaping module are sent to the time-GRU layer to process the time information.
[0060] In the embodiment of the present application, the structure of the time-GRU layer is similar to that of the frequency-GRU layer, except that the first reshaping module of the time-GRU layer reshapes the organization of the original features into (batch size * frequency dimension, time dimension, channel dimension), and the GRU layer structure of the time-GRU layer is similar to that of the GRU layer of the frequency-GRU layer, but focuses on processing time dimension information. The output features of the time-GRU layer are normalized by the layer to obtain a new intermediate representation W1. Repeat the above steps 4 times, and the output of the previous frequency-time-GRU module is used as the input of the next contrastive self-attention module to obtain the final feature W4. W n The organization method is (batch size, channel dimension, frequency dimension, time dimension).
[0061] Therefore, the embodiments of the present application can model the echo component and the speech component in the mixed signal separately, and the implicit modeling method avoids the redundant calculation amount of the explicit modeling echo module.
[0062] In step S103, the mapping representation is decoded to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and a subnetwork is reconstructed based on the predicted speech amplitude spectrum, time-frequency spectrum information and a preset phase to obtain an estimated phase of the target full-duplex voice communication system, and an inverse short-time Fourier transform is performed on the predicted speech amplitude spectrum and the estimated phase to generate a predicted speech signal of the target full-duplex voice communication system with echo information eliminated.
[0063] Furthermore, the embodiments of the present application also need to obtain the amplitude mask of the predicted speech signal through a decoding module, and calculate the estimated amplitude of the predicted speech signal (i.e., the predicted speech amplitude spectrum), and pass it together with the time-frequency domain features of the mixed signal and the far-end signal through a phase reconstruction subnetwork to obtain the estimated phase of the predicted speech signal, and obtain the predicted target speech signal (i.e., the predicted speech signal) through an inverse short-time Fourier transform.
[0064] Optionally, in one embodiment of the present application, the mapping representation is decoded to generate a predicted speech amplitude spectrum of a target full-duplex voice communication system, including: based on a preset jump connection strategy and a preset decoding network layer, the mapping representation is decoded to generate a mask corresponding to the mapping representation; the mask corresponding to the mapping representation is point-by-point multiplied with the amplitude spectrum in the time-frequency spectrum information of the mixed signal to obtain the predicted speech amplitude spectrum of the target full-duplex voice communication system.
[0065] It should be noted that after the intermediate representation W4 in the embodiment of the present application is sent to the decoding module, the predicted target speech amplitude spectrum can be obtained. The execution flow of the decoding module is as follows Figure 6As shown in the figure, the entire decoding module consists of 4 transposed convolution blocks, each of which contains a connection layer, a 2D transposed convolution layer, a batch normalization layer, and an activation function.
[0066] Specifically, the connection layer stacks the skip connection output of the corresponding convolution block in the encoding block with the current input in the channel dimension, and the channel dimension is expanded to twice the input channel. The ath transposed convolution block of the decoding module corresponds to the ath-to-last convolution block of the encoding module, that is, the features of the ath-to-last convolution block output of the encoding module are sent to the ath transposed convolution block of the decoding module when they are sent to the next layer, and are connected to the input of the transposed convolution block. The 2D transposed convolution layers in the 4 transposed convolution blocks have input channels set to 256, 128, 64, and 32, respectively, and output channels set to 64, 32, 16, and 1. The transposed convolution kernel size of each transposed convolution layer is (5, 3), the stride is (2, 1), and the padding is (2, 2). The first dimension is the frequency dimension, and the second dimension is the time dimension. In particular, we use causal transposed convolution. The number of channels of each batch normalization is consistent with the output channels of the corresponding transposed convolution layer. The activation function of the previous transposed convolution block is the PReLU function, and the activation function of the last transposed convolution block is the Sigmoid function, which is used to generate the predicted amplitude mask. Since the transposed convolution is the inverse operation of the convolution process, the frequency dimension of the intermediate representation is in a state of continuous expansion during the decoding stage. The frequency dimension of the initial input intermediate representation is 17, and the output feature frequency dimensions of the 4-layer transposed convolution block are 33, 65, 129, and 257 respectively. The time dimension and batch size remain unchanged, and the time dimension is fixed to 251.
[0067] The mask generated by the last transposed convolution block is compared with the amplitude spectrum Y of the mixed signal m Perform point-by-point multiplication to obtain the predicted target speech amplitude spectrum The organization is (batch size, channel dimension, frequency dimension, time dimension), and the channel dimension is 1.
[0068] Optionally, in one embodiment of the present application, based on the predicted speech amplitude spectrum, time-frequency spectrum information and a preset phase reconstruction subnetwork, an estimated phase of the target full-duplex voice communication system is obtained, and the predicted speech amplitude spectrum and the estimated phase are inversely short-time Fourier transformed to generate a predicted speech signal of the target full-duplex voice communication system with echo information eliminated, including: combining the predicted speech amplitude spectrum and time-frequency spectrum information to obtain a combined result, and inputting the combined result into a preset phase reconstruction subnetwork to output the estimated phase of the target full-duplex voice communication system; performing an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to obtain a target speech waveform; based on a preset amplitude loss function, a preset SI-SNR loss function and a preset contrast learning loss function, a joint loss function is constructed to obtain the predicted speech signal of the target full-duplex voice communication system through the joint loss function, the predicted speech amplitude spectrum and the target speech waveform.
[0069] In the specific implementation process, the embodiment of the present application can obtain the predicted target speech amplitude spectrum The real and imaginary parts of the time-frequency spectrum of the mixed signal Y r , Y i , and the real and imaginary parts of the time-frequency spectrum of the far-end signal X r , X i , a total of 5 features are merged in the channel dimension to obtain the input feature I (i.e., the merged result) of the phase reconstruction subnetwork. The organization of I is similar to The same as above, but the channel dimension is 5. The process of the phase reconstruction subnetwork is as follows Figure 7 shown.
[0070] Specifically, first, the input feature I of the embodiment of the present application can pass through two 2D convolutional layers. The convolution kernel size of the first convolutional layer is (7, 1), the padding size is (3, 0), the input channel is 5, and the output channel is 8; the convolution kernel size of the second convolutional layer is (25, 5), the padding size is (12, 4), and the input channel and output channel are both 8; secondly, the feature will pass through three repeated convolution blocks. In the convolution block, the input feature passes through a 2D convolutional layer, layer normalization, a 2D convolutional layer, layer normalization, and a residual connection in sequence. The convolution kernel sizes of the two convolutional layers are (5, 5) and (25, 1), respectively, and the padding sizes are (2, 4) and (12, 0), respectively; the three convolutional blocks share parameters to reduce the complexity of the model; finally, the feature will pass through a 2D convolutional layer and a normalization layer to obtain a predicted speech phase spectrum. The convolution kernel size of the convolution layer is (1, 1), the input channel is 8, and the output channel is 2, representing the real and imaginary parts of the phase, respectively.
[0071] Afterwards, according to the predicted speech amplitude spectrum Predicted speech phase spectrum Transform to obtain the real and imaginary parts of the predicted speech The transformation formula is as follows:
[0072]
[0073] The target speech waveform is obtained by inverse short-time Fourier transform.
[0074] Furthermore, the embodiments of the present application may design a joint loss function, which includes amplitude loss (i.e., amplitude loss function), SI-SNR (scale-invariant signal-to-noise ratio) loss (i.e., SI-SNR loss function) and contrastive learning loss. mag Use the MSE function to calculate the predicted speech amplitude spectrum and the target speech amplitude spectrum S m The difference between the two; SI-SNR loss is used to train the predicted speech amplitude spectrum The calculation process and phase reconstruction subnetwork are as follows:
[0075]
[0076] Among them, s and denote the clean and estimated time domain waveforms respectively, 〈·,·〉 denotes the dot product between two vectors, and represents the Euclidean distance.
[0077] Contrastive learning loss can use the above echo information and speech information, and the formula is as follows:
[0078]
[0079] The mathematical expression of the joint loss function is as follows:
[0080] Loss = L SI-SNR +λ1L mag +λ2L c
[0081] Among them, λ1 and λ2 are 0.5 and 5 respectively.
[0082] Therefore, the embodiment of the present application obtains the amplitude mask of the predicted speech signal by utilizing the decoding module to obtain the estimated amplitude of the predicted speech signal, and then obtains the estimated phase of the predicted speech signal, so as to obtain the predicted target speech signal through the inverse short-time Fourier transform, so as to effectively reduce the computational complexity and redundancy of the model.
[0083] According to the echo cancellation method based on positive and negative sample contrast implicit modeling proposed in the embodiment of the present application, by obtaining the mixed signal and the far-end signal of the target full-duplex voice communication system, and performing short-time Fourier transform on the mixed signal and the far-end signal, the time-frequency spectrum information corresponding to the mixed signal and the far-end signal is obtained respectively, and the initial representation corresponding to the time-frequency spectrum information is extracted; according to the preset contrastive self-attention strategy, the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space are obtained, and the positive sample and the negative sample are determined by the speech information and the echo information, and based on the positive sample, the negative sample and the preset contrast learning loss function, a speech prediction model is constructed, and the intermediate representation is input into the speech prediction model to generate a mapping representation; the mapping representation is decoded to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and based on the predicted speech amplitude spectrum, the time-frequency spectrum information and the preset phase reconstruction subnetwork, the estimated phase of the target full-duplex voice communication system is obtained, and the predicted speech amplitude spectrum and the estimated phase are inversely short-time Fourier transformed to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information. The comparative implicit modeling method of the present application can avoid the excessive computational complexity caused by the additional modeling echo and maintain the effect of the model performance.
[0084] Next, an echo cancellation device based on implicit modeling of contrast between positive and negative samples proposed in an embodiment of the present application is described with reference to the accompanying drawings.
[0085] Figure 8 It is a block diagram of an echo cancellation device based on implicit modeling of positive and negative sample contrast according to an embodiment of the present application.
[0086] like Figure 8 As shown, the echo cancellation device 10 based on implicit modeling of positive and negative sample contrast includes: an extraction module 100 , a modeling module 200 and a generation module 300 .
[0087] Among them, the extraction module 100 is used to obtain the mixed signal and the far-end signal of the target full-duplex voice communication system, and perform short-time Fourier transform on the mixed signal and the far-end signal to obtain the time-frequency spectrum information corresponding to the mixed signal and the far-end signal respectively, and extract the initial representation corresponding to the time-frequency spectrum information.
[0088] The modeling module 200 is used to preset a contrastive self-attention strategy to obtain the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space, and determine the positive sample and the negative sample through the speech information and the echo information, and build a speech prediction model based on the positive sample, the negative sample and the preset contrastive learning loss function, and input the intermediate representation into the speech prediction model to generate a mapping representation.
[0089] The generation module 300 is used to decode the mapping representation to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and reconstruct the subnetwork based on the predicted speech amplitude spectrum, time-frequency spectrum information and a preset phase to obtain an estimated phase of the target full-duplex voice communication system, and perform an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to generate a predicted speech signal of the target full-duplex voice communication system that eliminates echo information.
[0090] Optionally, in one embodiment of the present application, the modeling module 200 includes: a decoupling unit, an acquisition unit and a mask unit.
[0091] The decoupling unit is used to decouple the initial representation to obtain a decoupling result, and to perform matrix multiplication processing on the query data and key data in the decoupling result to generate a characteristic spectrum.
[0092] The acquisition unit is used to acquire the intermediate representation based on the characteristic spectrum and the value data in the decoupling result.
[0093] The mask unit is used to perform a mask generation operation on the feature spectrogram to obtain a mask corresponding to the feature spectrogram, and generate voice information and echo information according to the mask and a preset feature threshold.
[0094] Optionally, in one embodiment of the present application, the generation module 300 includes: a decoding unit and a multiplication unit.
[0095] The decoding unit is used to decode the mapping representation based on a preset jump connection strategy and a preset decoding network layer to generate a mask corresponding to the mapping representation.
[0096] The multiplication unit is used to multiply the mask corresponding to the mapping representation and the amplitude spectrum in the time-frequency spectrum information of the mixed signal point by point to obtain the predicted speech amplitude spectrum of the target full-duplex voice communication system.
[0097] Optionally, in one embodiment of the present application, the generation module 300 further includes: a merging unit, a transform unit and a prediction unit.
[0098] Among them, the merging unit is used to merge the predicted speech amplitude spectrum and time-frequency spectrum information to obtain a merging result, and input the merging result into a preset phase reconstruction subnetwork to output the estimated phase of the target full-duplex voice communication system.
[0099] The transform unit is used to perform inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to obtain the target speech waveform.
[0100] The prediction unit is used to construct a joint loss function based on a preset amplitude loss function, a preset SI-SNR loss function and a preset contrast learning loss function, so as to obtain a predicted speech signal of a target full-duplex voice communication system through the joint loss function, the predicted speech amplitude spectrum and the target speech waveform.
[0101] It should be noted that the above explanation of the embodiment of the echo cancellation method based on implicit modeling of contrast between positive and negative samples is also applicable to the echo cancellation device based on implicit modeling of contrast between positive and negative samples in this embodiment, which will not be described in detail here.
[0102] According to the echo cancellation device based on positive and negative sample contrast implicit modeling proposed in the embodiment of the present application, it includes an extraction module, which is used to obtain the mixed signal and the far-end signal of the target full-duplex voice communication system, and perform short-time Fourier transform on the mixed signal and the far-end signal to obtain the time-frequency spectrum information corresponding to the mixed signal and the far-end signal respectively, and extract the initial representation corresponding to the time-frequency spectrum information; a modeling module, which is used to preset a contrastive self-attention strategy to obtain the intermediate representation corresponding to the initial feature and the voice information and echo information in the latent space, and determine the positive sample and the negative sample through the voice information and the echo information, and construct a voice prediction model based on the positive sample, the negative sample and the preset contrast learning loss function, and input the intermediate representation into the voice prediction model to generate a mapping representation; a generation module, which is used to decode the mapping representation to generate a predicted voice amplitude spectrum of the target full-duplex voice communication system, and reconstruct a subnetwork based on the predicted voice amplitude spectrum, the time-frequency spectrum information and the preset phase to obtain the estimated phase of the target full-duplex voice communication system, and perform an inverse short-time Fourier transform on the predicted voice amplitude spectrum and the estimated phase to generate a predicted voice signal of the target full-duplex voice communication system that eliminates the echo information. The comparative implicit modeling method of the present application can avoid the excessive computational complexity caused by the additional modeling echo and maintain the effect of the model performance.
[0103] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0104] A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .
[0105] When the processor 902 executes the program, the echo cancellation method based on implicit modeling of contrast between positive and negative samples provided in the above embodiment is implemented.
[0106] Furthermore, the electronic device further comprises:
[0107] The communication interface 903 is used for communication between the memory 901 and the processor 902 .
[0108] The memory 901 is used to store computer programs that can be executed on the processor 902 .
[0109] The memory 901 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0110] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0111] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.
[0112] The processor 902 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0113] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the echo cancellation method based on implicit modeling of contrast between positive and negative samples as described above.
[0114] The embodiment of the present application further provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned echo cancellation method based on implicit modeling of contrast between positive and negative samples.
[0115] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0116] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0117] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0118] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0119] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0120] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0121] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0122] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. An echo cancellation method based on implicit modeling of positive and negative sample contrast, characterized in that: The following steps are involved: Acquire a mixed signal and a far-end signal of a target full-duplex voice communication system, perform short-time Fourier transform on the mixed signal and the far-end signal, obtain time-frequency spectrum information corresponding to the mixed signal and the far-end signal, respectively, and extract an initial representation corresponding to the time-frequency spectrum information; According to a preset contrastive self-attention strategy, an intermediate representation corresponding to the initial feature and speech information and echo information in a latent space are obtained, and positive samples and negative samples are determined by the speech information and the echo information, and a speech prediction model is constructed based on the positive samples, the negative samples and a preset contrastive learning loss function, and the intermediate representation is input into the speech prediction model to generate a mapping representation; The mapping representation is decoded to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and an estimated phase of the target full-duplex voice communication system is obtained based on the predicted speech amplitude spectrum, the time-frequency spectrum information and a preset phase reconstruction subnetwork, and an inverse short-time Fourier transform is performed on the predicted speech amplitude spectrum and the estimated phase to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information.
2. The method according to claim 1, characterized in that The method of obtaining the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space according to the preset contrastive self-attention strategy includes: Decoupling the initial representation to obtain a decoupling result, and performing matrix multiplication processing on the query data and key data in the decoupling result to generate a characteristic spectrum; Based on the characteristic spectrum and the value data in the decoupling result, obtaining the intermediate representation; A mask generation operation is performed on the feature spectrogram to obtain a mask corresponding to the feature spectrogram, and the voice information and the echo information are generated according to the mask and a preset feature threshold.
3. The method according to claim 2, characterized in that The decoding of the mapping representation to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system includes: Based on a preset skip connection strategy and a preset decoding network layer, decoding the mapping representation to generate a mask corresponding to the mapping representation; The mask corresponding to the mapping representation is multiplied point by point with the amplitude spectrum in the time-frequency spectrum information of the mixed signal to obtain the predicted speech amplitude spectrum of the target full-duplex voice communication system.
4. The method according to claim 3, characterized in that The method reconstructs a subnetwork based on the predicted speech amplitude spectrum, the time-frequency spectrum information and the preset phase to obtain an estimated phase of the target full-duplex voice communication system, and performs an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information, including: Combining the predicted speech amplitude spectrum and the time-frequency spectrum information to obtain a combined result, and inputting the combined result into the preset phase reconstruction subnetwork to output an estimated phase of the target full-duplex voice communication system; Performing an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to obtain a target speech waveform; Based on a preset amplitude loss function, a preset SI-SNR loss function and a preset contrast learning loss function, a joint loss function is constructed to obtain a predicted speech signal of the target full-duplex voice communication system through the joint loss function, the predicted speech amplitude spectrum and the target speech waveform.
5. An echo cancellation device based on implicit modeling of positive and negative sample contrast, characterized in that: include: An extraction module is used to obtain a mixed signal and a far-end signal of a target full-duplex voice communication system, and perform short-time Fourier transform on the mixed signal and the far-end signal to obtain time-frequency spectrum information corresponding to the mixed signal and the far-end signal, respectively, and extract an initial representation corresponding to the time-frequency spectrum information; A modeling module, for obtaining the intermediate representation corresponding to the initial feature and the speech information and echo information in the latent space by presetting a contrastive self-attention strategy, and determining positive samples and negative samples by the speech information and the echo information, and constructing a speech prediction model based on the positive samples, the negative samples and a preset contrastive learning loss function, and inputting the intermediate representation into the speech prediction model to generate a mapping representation; A generation module is used to decode the mapping representation to generate a predicted speech amplitude spectrum of the target full-duplex voice communication system, and to reconstruct a subnetwork based on the predicted speech amplitude spectrum, the time-frequency spectrum information and a preset phase to obtain an estimated phase of the target full-duplex voice communication system, and to perform an inverse short-time Fourier transform on the predicted speech amplitude spectrum and the estimated phase to generate a predicted speech signal of the target full-duplex voice communication system that eliminates the echo information.
6. The device according to claim 5, characterized in that The modeling module includes: A decoupling unit, used for decoupling the initial representation to obtain a decoupling result, and performing matrix multiplication processing on the query data and key data in the decoupling result to generate a characteristic spectrum; An acquisition unit, configured to acquire the intermediate representation based on the characteristic spectrum and the value data in the decoupling result; A mask unit is used to perform a mask generation operation on the feature spectrogram to obtain a mask corresponding to the feature spectrogram, and generate the voice information and the echo information according to the mask and a preset feature threshold.
7. The device according to claim 6, characterized in that The generation module comprises: A decoding unit, configured to decode the mapping representation based on a preset skip connection strategy and a preset decoding network layer to generate a mask corresponding to the mapping representation; The multiplication unit is used to multiply the mask corresponding to the mapping representation by the amplitude spectrum in the time-frequency spectrum information of the mixed signal point by point to obtain the predicted speech amplitude spectrum of the target full-duplex voice communication system.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the echo cancellation method based on implicit modeling of contrast between positive and negative samples as described in any one of claims 1 to 4.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the echo cancellation method based on positive and negative sample contrast implicit modeling as described in any one of claims 1 to 4.
10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the echo cancellation method based on positive and negative sample contrast implicit modeling as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Deep clustering speech separation method based on logarithmic magnitude spectrum and inter-ear phase difference
CN112216301A
Echo cancellation method and device, electronic equipment and readable storage medium
CN112634933A
End-to-end real-time speech synthesis method
CN113409759A
Multi-feature fusion echo cancellation method and system based on self-attention transformation network
CN113870874A
Method for restoration of target speech based upon speech section detection under stationary noise
JP2005084244A
Cited By
Digital self-interference cancellation method and device based on enhanced neural network structure
CN120049908A
