Voice processing methods and electronic devices
By updating the speech processing model based on the speech interaction state in a full-duplex speech interaction scenario, echoes and noise are eliminated, solving the problem of inaccurate speech processing caused by module cascading in existing technologies and achieving higher speech extraction accuracy.
Patent Information
- Application Number
- CN202611195804.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-07
AI Technical Summary
In full-duplex voice interaction scenarios, when the echo cancellation module and noise suppression module are cascaded in series, it is difficult to accurately extract the user's voice, resulting in poor voice processing accuracy.
By acquiring the initial speech signal and the playback reference signal, the speech interaction state is determined, and the speech processing model is updated based on this state. Echo signals and noise signals are eliminated, and an end-to-end processing method is adopted to reduce the accumulation of errors and module interference caused by multi-level module processing.
It improves the accuracy of speech processing, can accurately extract user input speech, and reduces interference and error accumulation between modules.
Smart Images

Figure CN122695983B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of speech processing, and in particular relates to a speech processing method and an electronic device. Background Technology
[0002] In a full-duplex voice interaction scenario, the device can play music while simultaneously picking up audio. In this case, the device's microphone may simultaneously pick up three types of signals: the echo returned by the device's playback through the acoustic loop, ambient noise, and the voice of the nearby user. The voice of the nearby user is the speech that needs to be extracted.
[0003] In existing technologies, echo cancellation and noise suppression modules are cascaded in series. The echo cancellation and noise suppression modules repeatedly process the speech captured by the device's microphone. During speech processing, the residual echo output by the echo cancellation module is passed to the noise suppression module. The noise suppression module struggles to distinguish the residual echo from real ambient noise, while the distortion of user speech caused by noise suppression is easily mistaken for non-echo signals and attempted to be eliminated by the echo cancellation module. This mutual interference between the two modules makes it difficult to extract accurate user speech, resulting in poor speech processing accuracy. Summary of the Invention
[0004] This application provides a voice processing method and an electronic device that can accurately extract user-input voice and improve the accuracy of voice processing.
[0005] In a first aspect, embodiments of this application provide a voice processing method, including: Acquire an initial voice signal and a playback reference signal from the target device, wherein the initial voice signal includes an echo signal and a noise signal, and the echo signal is related to the playback reference signal; Determine the voice interaction state of the target device, wherein the voice interaction state indicates the signal type contained in the initial voice signal; The voice processing model is updated based on the voice interaction state to obtain the updated voice processing model. The initial voice signal and the playback reference signal are then input into the updated voice processing model to obtain the target voice. The voice processing model is used to eliminate the echo signal and the noise signal.
[0006] Optionally, the speech processing model includes multiple normalization layers; updating the speech processing model based on the speech interaction state to obtain the updated speech processing model includes: For each of the normalization layers, the statistical parameters and affine parameters of the normalization layer are determined based on the voice interaction state; The normalization layer is updated based on the statistical parameters and the affine parameters to obtain the updated normalization layer; Based on the updated normalization layer corresponding to each of the multiple normalization layers, the speech processing model is updated to obtain the updated speech processing model.
[0007] Optionally, the voice interaction state includes a first candidate interaction state and a second candidate interaction state; determining the statistical parameters and affine parameters of the normalization layer based on the voice interaction state includes: Determine the first confidence level corresponding to the first candidate interaction state and the second confidence level corresponding to the second candidate interaction state; Based on the first candidate interaction state, determine the first statistical parameter and the first affine parameter; Based on the second candidate interaction state, determine the second statistical parameter and the second affine parameter; Based on the first confidence level and the second confidence level, the first statistical parameter and the second statistical parameter are weighted and summed to obtain the statistical parameters of the normalized layer; Based on the first confidence level and the second confidence level, the first affine parameter and the second affine parameter are weighted and summed to obtain the affine parameters of the normalized layer.
[0008] Optionally, the step of inputting the initial speech signal and the playback reference signal into the updated speech processing model to eliminate the echo signal and the noise signal to obtain the target speech includes: Based on the playback reference signal, target signal features, echo signal features, and noise signal features are extracted from the initial speech signal; The target signal features, the echo signal features, and the noise signal features are fused to obtain fused features; The fused features are input into the updated speech processing model to eliminate the echo signal and the noise signal, thereby obtaining the target speech.
[0009] Optionally, the updated speech processing model includes a complex encoder, a cyclic unit, and a complex decoder; the step of inputting the fused features into the updated speech processing model to eliminate the echo signal and the noise signal to obtain the target speech includes: The complex encoder is used to downsample the fused features to obtain complex frequency domain features; The complex frequency domain features are input into the cyclic unit to obtain the time-series complex features; The complex frequency domain features and the temporal complex features are input into the complex decoder to obtain the complex spectrum; The complex spectrum is converted into target speech.
[0010] Optionally, the speech processing model is trained in the following manner: Acquire sample speech signals, sample reference signals, sample noise signals, and multiple preset speech interaction states; Based on the sample reference signal, the sample echo signal is determined; For each preset voice interaction state, the sample voice signal, the sample echo signal, and the sample noise signal are mixed based on the preset voice interaction state to obtain the initial sample voice signal; The initial speech signal and the reference signal are input into the speech processing model to obtain the processed speech signal. Based on the sample processed speech and the sample speech signal, train the speech processing model to obtain the updated speech processing model corresponding to the preset speech interaction state; The updated voice processing models corresponding to the multiple preset voice interaction states are integrated to obtain a voice processing model.
[0011] Optionally, the step of training the speech processing model based on the sample processed speech and the sample speech signal to obtain the updated speech processing model corresponding to the preset speech interaction state includes: Based on the processed speech and the sample speech signal, determine the speech quality loss; Based on the preset voice interaction state and the sample processed voice, determine the echo suppression loss; Based on the speech quality loss and the echo suppression loss, the speech processing model is trained to obtain the updated speech processing model corresponding to the preset speech interaction state.
[0012] Optionally, the step of inputting the initial sample speech signal and the sample reference signal into the speech processing model to obtain the processed sample speech includes: The target speech extraction module is used to extract target signal features from the initial speech signal of the sample. The echo signal extraction module extracts sample echo signal features from the initial speech signal of the sample based on the sample reference signal; The noise signal extraction module is used to extract sample noise signal features from the initial speech signal of the sample. The sample target signal features, the sample echo signal features, and the sample noise signal features are fused to obtain sample fusion features, and the sample fusion features are input into the speech processing model to obtain sample processed speech.
[0013] Optionally, training the speech processing model based on the speech quality loss and the echo suppression loss to obtain the updated speech processing model corresponding to the preset speech interaction state includes: Based on the difference between the sample target signal features and the corresponding vector features of the sample speech signal, a first loss function is determined; A second loss function is determined based on the difference between the features of the sample echo signal and the corresponding vector features of the sample echo signal; Based on the difference between the sample noise signal features and the corresponding vector features of the sample noise signal, a third loss function is determined; Based on the speech quality loss, the echo suppression loss, the first loss function, the second loss function, and the third loss function, the speech processing model, the target speech extraction module, the echo signal extraction module, and the noise signal extraction module are jointly trained to obtain the updated speech processing model corresponding to the preset speech interaction state.
[0014] Optionally, determining the sample echo signal based on the sample reference signal includes: Get multiple preset extension methods; Select the target extension method from the multiple preset extension methods; The sample reference signal is nonlinearly extended based on the target extension method to obtain the sample echo signal.
[0015] Secondly, embodiments of this application provide a voice processing device, including: A signal acquisition unit is used to acquire an initial voice signal and a playback reference signal from a target device. The initial voice signal includes an echo signal and a noise signal, and the echo signal is related to the playback reference signal. A state determination unit is used to determine the voice interaction state of the target device, wherein the voice interaction state indicates the signal type contained in the initial voice signal; The model processing unit is used to update the speech processing model based on the speech interaction state to obtain the updated speech processing model, and input the initial speech signal and the playback reference signal into the updated speech processing model to obtain the target speech. The speech processing model is used to eliminate the echo signal and the noise signal.
[0016] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the voice processing method as described in any of the first aspects.
[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech processing method as described in any of the first aspects.
[0018] This application embodiment determines a playback reference signal for the target device based on the initial voice signal received by the target device. The echo signal in the initial voice signal is related to the playback reference signal and is the echo returned by the playback reference signal through the acoustic loop. Then, the voice interaction state of the target device is determined. The voice interaction state indicates the signal types contained in the initial voice signal, such as the signal types corresponding to various combinations of echo signals, noise signals, and target voice. The emphasis of voice processing differs under different voice interaction states. For example, when the user is not speaking, voice processing focuses on noise reduction and echo suppression; when the user is speaking, voice processing includes not only noise reduction and echo suppression but also highlighting the user's voice. Therefore, this application embodiment sets different model parameters for different voice processing states. After determining the voice interaction state of the target device, the voice processing model can be updated based on the voice interaction state to obtain an updated voice processing model. The initial voice signal and the playback reference signal are input into the updated voice processing model to eliminate the echo and noise signals in the initial voice signal, thus obtaining the target voice. The updated speech processing model is matched with the current speech interaction state of the target device, thereby improving the accuracy of target speech extraction. Furthermore, this embodiment of the application performs end-to-end processing of the initial speech signal and the playback reference signal through the updated speech processing model, with simultaneous noise suppression and echo cancellation. This reduces error accumulation and module interference issues caused by multi-level module processing, enabling accurate extraction of the user-input target speech from the initial speech signal and improving the accuracy of speech processing. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a system architecture diagram of the speech processing method according to the embodiments of this application; Figure 2 This is a general flowchart of a speech processing method according to an embodiment of this application; Figure 3 This is a schematic diagram of a speech processing method according to an embodiment of this application; Figure 4 This is another schematic diagram of a speech processing method according to an embodiment of this application; Figure 5 This is a schematic diagram of normalization layer update according to an embodiment of this application; Figure 6 This is a schematic diagram illustrating the determination of fusion features according to an embodiment of this application; Figure 7 This is a schematic diagram of a voice processing module according to an embodiment of this application; Figure 8 This is a schematic diagram of the target speech separation effect according to an embodiment of the present application, wherein (a) is a schematic diagram of speech processing model loss calculation, and (b) is a schematic diagram of target speech separation quality; Figure 9 This is a schematic diagram of the structure of a voice processing device according to an embodiment of this application; Figure 10 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0022] It should be understood that, when used in this application specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.
[0023] It should also be understood that the term “and / or” as used in this application specification means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.
[0024] As used in this application specification, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."
[0025] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0027] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application shall be interpreted as follows: Full Duplex: Both communicating parties can send and receive data simultaneously. The two channels for sending and receiving are completely independent and do not interfere with each other, allowing bidirectional transmission at the same time.
[0028] Noise suppression: While preserving human voices, suppress steady-state background noise and improve the signal-to-noise ratio.
[0029] Echo cancellation: Achieves clear speech or recordings by identifying and suppressing repeating sound waves in audio.
[0030] In a full-duplex voice interaction scenario, the device can play music while simultaneously picking up audio. In this case, the device's microphone may simultaneously pick up three types of signals: the echo returned by the device's playback through the acoustic loop, ambient noise, and the voice of the nearby user. The voice of the nearby user is the speech that needs to be extracted.
[0031] In related technologies, an echo cancellation module and a noise suppression module are cascaded in series. The echo cancellation module and the noise suppression module repeatedly process the speech picked up by the device's microphone. During speech processing, the residual echo output by the echo cancellation module is passed to the noise suppression module. The noise suppression module struggles to distinguish the residual echo from real ambient noise, while the distortion of user speech caused by noise suppression is easily mistaken for non-echo signals and attempted to be eliminated by the echo cancellation module. This mutual interference between the two modules makes it difficult to extract accurate user speech, resulting in poor speech processing accuracy.
[0032] Based on this, embodiments of this application provide a voice processing method and an electronic device. The voice processing method provided by embodiments of this application can accurately extract user-input voice, thereby improving the accuracy of voice processing.
[0033] Figure 1 This is a system architecture diagram of the speech processing method according to the embodiments of this application. Figure 1 This includes terminal 140, Internet 130, gateway 120, server 110, etc.
[0034] Terminal 140 is a device that receives the initial voice signal and also plays the reference signal. Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.
[0035] Server 110 refers to a computer system capable of providing voice processing services. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). In a distributed system, multiple servers 110 are interconnected as computing and storage units to transmit data and coordinate task processing using shared communication lines. Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.
[0036] A gateway, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions.
[0037] The embodiments of this application can be applied in various scenarios, such as full-duplex interaction scenarios like smart speakers, in-vehicle voice systems, mobile phone calls, and wearable devices.
[0038] It should be emphasized that the embodiments of this application are applicable to various application scenarios, such as full-duplex interaction scenarios like smart speakers, in-vehicle voice systems, mobile phone calls, and wearable devices. In related technologies, the echo cancellation module and the noise suppression module are cascaded in series. For the speech picked up by the device's microphone, the echo cancellation module and the noise suppression module repeatedly process the speech picked up by the device's microphone. During speech processing, the residual echo output by the echo cancellation module is passed to the noise suppression module. The noise suppression module has difficulty distinguishing between the residual echo and real environmental noise, while the distortion of user speech caused by noise suppression is easily regarded as a non-echo signal by the echo cancellation module and attempted to be eliminated. The two modules interfere with each other, making it difficult to extract accurate user speech, resulting in poor speech processing accuracy. Some embodiments of this application provide a speech processing method and electronic device that can accurately extract user input speech and improve the accuracy of speech processing.
[0039] The voice processing method of this application embodiment can be performed on the terminal 140, on the server 110, or partly on the terminal 140 and partly on the server 110.
[0040] like Figure 2 As shown, a speech processing method according to one embodiment of this application may include: Step 210: Obtain the initial voice signal and playback reference signal from the target device. The initial voice signal includes an echo signal and a noise signal. The echo signal is related to the playback reference signal. Step 220: Determine the voice interaction status of the target device. The voice interaction status indicates the signal type contained in the initial voice signal. Step 230: Update the speech processing model based on the speech interaction state to obtain the updated speech processing model. Input the initial speech signal and the playback reference signal into the updated speech processing model to obtain the target speech. The speech processing model is used to eliminate echo signals and noise signals.
[0041] In step 210, the target device is a voice interaction device used in a full-duplex scenario. The target device can be a mobile phone, smart speaker, smart wearable device, in-vehicle voice playback device, etc. Since the target device is used for voice interaction in a full-duplex scenario, the target device includes at least a microphone and a speaker.
[0042] The initial voice signal is the voice signal received by the target device through its microphone. The playback reference signal is the voice signal played by the target device through its speaker.
[0043] The initial speech signal contains an echo signal and a noise signal. The noise signal comes from the ambient noise of the target device's environment, while the echo signal is the sound returning through the acoustic loop after the playback reference signal is played through the speaker. Therefore, the echo signal is related to the playback reference signal.
[0044] In step 220, the voice interaction state can also be considered as a full-duplex state, and the voice interaction state indicates the type of signal contained in the initial voice signal.
[0045] Voice interaction states can be categorized into far-end (speaker playback) and near-end (microphone corresponding to a nearby user) activities into far-end one-way, near-end one-way, two-way, and noise only. In far-end one-way, the initial voice signal indicates that it contains echo and noise. In near-end one-way, the initial voice signal indicates that it contains user input (the user's voice, i.e., the target voice) and noise. In two-way, the initial voice signal indicates that it contains user input (the user's voice, i.e., the target voice), echo, and noise. In noise only, the initial voice signal indicates that it contains noise.
[0046] It should be noted that the voice interaction state can be determined based on at least one of the following: playback activity detection, near-end voice activity detection, far-end playback flag, and a preemptive state machine. The far-end playback flag determines whether the target device is playing a playback reference signal, while playback activity detection determines whether the target device is actually playing the playback reference signal. Together, they determine whether the target device's speaker is effectively playing audio, thus identifying the presence of an echo source. If there is no effective playback at the far end, the echo signal in the initial voice signal is 0. Near-end voice activity detection determines whether there is valid user speech in the target device's microphone signal, identifying the presence of near-end target speech. The preemptive state machine can perform timing smoothing and anti-jitter filtering on the near-end and far-end activity detection results to stably output the final voice interaction state. Additionally, noise signals exist in all voice interaction states.
[0047] In step 230, the parameters used by the speech processing model differ under different speech interaction states. The updated speech processing model parameters correspond to the speech interaction states; it is the model obtained by adjusting the parameters of the speech processing model based on the speech interaction states.
[0048] The focus of speech processing differs depending on the voice interaction state. For example, when the user is not speaking on the target device, speech processing emphasizes noise reduction and echo suppression. However, when the user is speaking on the target device, speech processing includes not only noise reduction and echo suppression but also highlighting the user's voice. Therefore, different voice interaction states correspond to different updated speech processing models.
[0049] It should be noted that the parameters of only a single module or a few modules of the speech processing model are adaptively adjusted according to the speech interaction state. This allows the internal parameters to be adaptively adjusted by a single model according to different speech interaction states, without the need to set up an independent model for each speech interaction state. This improves the applicability of speech processing while reducing the memory usage of the speech processing model.
[0050] The initial speech signal and the playback reference signal are input into the updated speech processing model. Based on the playback reference signal, the updated speech processing model eliminates the echo signal and noise signal to obtain the target speech.
[0051] Assume the target device is a mobile phone, and the user is user A. User A and user B communicate through the target device. The reference speech signal is user B's input speech, the echo signal is the echo generated by user B's input speech, and the noise signal is the noise of the environment where the target device is located. If user A is on the street, the noise signal includes car horns, pedestrian conversations, etc. The target speech is user A's input speech. When the voice interaction state is detected as two-way, the speech processing model is updated to obtain the updated speech processing model. The initial speech signal and the playback reference signal are input into the updated speech processing model, and the resulting target speech contains only user A's input speech.
[0052] Reference Figure 3 and Figure 4 Related technologies repeatedly process the initial speech signal (received by the device's microphone) through cascaded echo cancellation and noise suppression modules. This application, however, updates the parameters of the speech processing model based on the target device's voice interaction state, and then uses the updated model to process the initial speech signal and the playback reference signal to obtain the target speech. The speech processing model is an end-to-end model, capable of outputting the target speech in one step, reducing cascading errors and improving the accuracy of the target speech.
[0053] In the embodiments described above, steps 210 to 230, based on the initial speech signal received by the target device, determine the playback reference signal for the target device. The echo signal in the initial speech signal is related to the playback reference signal and is the echo returned by the playback reference signal through the acoustic loop. Then, the speech interaction state of the target device is determined. The speech interaction state indicates the signal types contained in the initial speech signal, such as the signal types corresponding to various combinations of echo signals, noise signals, and target speech. The emphasis of speech processing differs under different speech interaction states. For example, when the target device corresponds to a user who is not speaking, speech processing focuses on noise reduction and echo suppression; while when the target device corresponds to a user speaking, speech processing includes not only noise reduction and echo suppression but also highlighting the user's speech. Therefore, this application embodiment sets different model parameters for the speech processing model for different speech interaction states. After determining the speech interaction state of the target device, the speech processing model can be updated based on the speech interaction state to obtain the updated speech processing model. The initial speech signal and the playback reference signal are input into the updated speech processing model to eliminate the echo signal and noise signal in the initial speech signal, thus obtaining the target speech. The updated speech processing model is matched with the current speech interaction state of the target device, thereby improving the accuracy of target speech extraction. Furthermore, this embodiment of the application performs end-to-end processing of the initial speech signal and the playback reference signal through the updated speech processing model, with simultaneous noise suppression and echo cancellation. This reduces error accumulation and module interference issues caused by multi-level module processing, enabling accurate extraction of the user-input target speech from the initial speech signal and improving the accuracy of speech processing.
[0054] In one embodiment, the speech processing model includes multiple normalization layers, and step 230 includes: Step 310: For each normalization layer, determine the statistical parameters and affine parameters of the normalization layer based on the voice interaction state; Step 320: Update the normalization layer based on statistical parameters and affine parameters to obtain the updated normalization layer; Step 330: Based on the updated normalization layers corresponding to each of the multiple normalization layers, update the speech processing model to obtain the updated speech processing model.
[0055] In step 310, the normalization layer constrains the distribution of the input features to a stable range, accelerating convergence and improving training stability. In short, the normalization layer makes the observed data distribution as stable as possible.
[0056] The speech processing model contains multiple normalization layers, which are set in each layer of the speech processing model, such as the encoder and decoder.
[0057] The normalization layer in this application is a conditional normalization layer, which normalizes the data based on preset mean and standard deviation, and then reconstructs the normalized data. The statistical parameters of the normalization layer are the mean and variance. The affine parameters are the scaling and translation values used in the reconstruction process.
[0058] The statistical and affine parameters of the normalized layer are determined based on the voice interaction state.
[0059] Under four different voice interaction states—far-end single-talk, near-end single-talk, dual-talk, and noise-only—the data distribution of speech features in the initial speech signal, such as mean, variance, and dynamic range, varies significantly. The normalization layer, through statistical and affine parameters, can adjust the feature distribution, thereby achieving transformation rules that match different signal distributions to different feature distributions. Furthermore, compared to convolutional layers and recurrent units, the normalization layer has smaller parameters; thus, only a small number of parameters follow the voice interaction state switching, resulting in minimal parameter growth.
[0060] It should be noted that the statistical parameters and affine parameters of the normalization layer corresponding to different voice interaction states are determined based on training.
[0061] In step 320, the updated normalized layer is the network layer corresponding to the voice interaction state, and the statistical parameters and affine parameters of the updated normalized layer correspond to the voice interaction state.
[0062] In step 330, after determining the updated normalization layer corresponding to the voice interaction state, the normalization layer in the voice processing model is replaced with the corresponding updated normalization layer to obtain the updated voice processing model.
[0063] Reference Figure 5 The initial speech signal corresponds to four speech interaction states: far-end single-talk, near-end single-talk, two-talk, and noise only. The initial speech signal exhibits different signal types depending on the speech interaction state. Therefore, the data distribution of the speech features of the initial speech signal varies significantly across different speech interaction states. The normalization layer can adjust the feature distribution; therefore, corresponding statistical and affine parameters can be selected based on the speech interaction state, and the normalization layer can be updated based on these parameters, thereby updating the speech processing model.
[0064] In the embodiments described in steps 310 to 330 above, the speech processing model includes multiple normalization layers. Only the statistical and affine parameters of the normalization layers change with the speech interaction state. Compared with other network layers, the normalization layers have fewer parameters, thereby reducing the number of parameters that need to be updated in each speech processing step and reducing the corresponding data storage, thus improving speech processing efficiency. Furthermore, the normalization layers can adjust the feature distribution, allowing the speech processing model to correspond to the speech interaction state. Based on the speech interaction state, the corresponding speech processing model can accurately process the corresponding data, improving the accuracy of speech processing.
[0065] In one embodiment, the voice interaction state includes a first candidate interaction state and a second candidate interaction state, and step 310 includes: Step 410: Determine the first confidence level corresponding to the first candidate interaction state and the second confidence level corresponding to the second candidate interaction state; Step 420: Based on the first candidate interaction state, determine the first statistical parameter and the first affine parameter; Step 430: Based on the second candidate interaction state, determine the second statistical parameter and the second affine parameter; Step 440: Based on the first confidence level and the second confidence level, perform a weighted summation of the first statistical parameter and the second statistical parameter to obtain the statistical parameters of the normalized layer; Step 450: Based on the first confidence level and the second confidence level, perform a weighted summation of the first affine parameter and the second affine parameter to obtain the affine parameters of the normalized layer.
[0066] The voice interaction state is determined by at least one of the following: playback activity detection, near-end voice activity detection, far-end playback flag, and call-blocking state machine. Typically, multiple candidate interaction states and their corresponding confidence levels are provided. If the confidence level of the candidate interaction state with the highest confidence level is greater than a preset confidence threshold, then that candidate interaction state is used as the initial voice interaction state for the voice signal.
[0067] It should be noted that the preset confidence threshold can be set as needed. Assuming the preset confidence threshold is 0.75, and the predicted confidence of near-end single-talk is 0.8, and the confidence of two-talk is 0.2, then near-end single-talk is determined to be a voice interaction state.
[0068] The first candidate interaction state and the second candidate interaction state are two different states that the voice interaction state may have. The first candidate interaction state and the second candidate interaction state are jointly given by at least one of the following: playback activity detection, near-end voice activity detection, far-end playback flag and call-blocking state machine.
[0069] It should be noted that, to improve voice processing efficiency, the initial voice signal is typically 1 second, but can also be set to 3 seconds or 5 seconds. In this case, the initial voice signal may have two or more voice interaction states. For example, during a voice call, the first 2 seconds of the initial voice signal are user A speaking (near-end one-way), and the last 3 seconds of the initial voice signal are user B speaking (far-end one-way).
[0070] It should be noted that, in addition to the first and second candidate interaction states, the given candidate interaction states may also include a third and a fourth candidate interaction state.
[0071] In step 410, the first confidence level is the credibility of the target device's voice interaction state as the first candidate interaction state. The higher the first confidence level, the higher the probability of using the first candidate interaction state as the voice interaction state. Similarly, the second confidence level is the credibility of the target device's voice interaction state as the second candidate interaction state. The higher the second confidence level, the higher the probability of using the second candidate interaction state as the voice interaction state.
[0072] If the given candidate interaction states only include the first candidate interaction state and the second candidate interaction state, then the sum of the first confidence level and the second confidence level is 1.
[0073] In step 420, the first statistical parameter and the first affine parameter are the parameters of the normalization layer corresponding to the first candidate interaction state. Through training, it is determined that when the voice interaction state of the target device is the first candidate interaction state, the normalization layer possesses the first statistical parameter and the first affine parameter.
[0074] In step 430, the second statistical parameter and the second affine parameter are the parameters of the normalization layer corresponding to the second candidate interaction state. Through training, it is obtained that when the voice interaction state of the target device is the second candidate interaction state, the normalization layer possesses the second statistical parameter and the second affine parameter.
[0075] The initial speech signal has different characteristics under different voice interaction states, and therefore the feature distribution of the initial speech signal is also different. Since the statistical parameters and affine parameters of the normalization layer are trained using data from the corresponding voice interaction states, the first statistical parameter and the second statistical parameter are different from each other, as are the first affine parameter and the second affine parameter.
[0076] In step 440, the statistical parameters of the normalization layer are the weighted sum of the first statistical parameter and the second statistical parameter. In the process of calculating the weighted sum, the weight of the first statistical parameter is determined based on the first confidence level, and the weight of the second statistical parameter is determined based on the second confidence level.
[0077] It should be noted that a preset confidence threshold is usually set. If the confidence of the candidate interaction state with the highest confidence is greater than the preset confidence threshold, then the candidate interaction state is used as the voice interaction state of the initial voice signal. If both the first confidence and the second confidence are less than the preset confidence threshold, then the first statistical parameter and the second statistical parameter are weighted and summed based on the first confidence and the second confidence to obtain the statistical parameters of the normalization layer.
[0078] Assuming the first statistical parameter is x, the second statistical parameter is y, the first confidence level is 0.6, the second confidence level is 0.4, and the preset confidence threshold is 0.75, then the statistical parameter of the normalized layer is 0.6x + 0.4y.
[0079] In step 450, the affine parameters of the normalization layer are the weighted sum of the first affine parameter and the second affine parameter. In the calculation of the weighted sum, the weight of the first affine parameter is determined based on the first confidence level, and the weight of the second affine parameter is determined based on the second confidence level.
[0080] It should be noted that a preset confidence threshold is usually set. If the confidence of the candidate interaction state with the highest confidence is greater than the preset confidence threshold, then this candidate interaction state is used as the initial speech interaction state, thereby determining the affine parameters of the normalization layer. If both the first confidence and the second confidence are less than the preset confidence threshold, then the first and second affine parameters are weighted and summed based on the first and second confidence to obtain the affine parameters of the normalization layer.
[0081] Assuming the first affine parameter is p, the second affine parameter is q, the first confidence level is 0.7, the second confidence level is 0.3, and the preset confidence threshold is 0.75, then the affine parameter of the normalized layer is 0.7p + 0.3q.
[0082] It should be noted that, referring to Figure 5 Assuming the given candidate interaction states for voice interaction only include a first candidate interaction state and a second candidate interaction state, and these two states are two of the following: far-end single-talk, near-end single-talk, dual-talk, or noise only. When the confidence levels corresponding to the first and second candidate interaction states are both low, a hybrid parameter is used as the parameter for the normalization layer. Specifically, based on the first and second confidence levels, the first and second statistical parameters are weighted and summed to obtain the statistical parameters of the normalization layer. Similarly, the affine parameters of the normalization layer are obtained.
[0083] It should be noted that fixed default parameters can also be set according to the training data. When the confidence scores of multiple candidate interaction states are low, the default parameters are used as the statistical parameters and affine parameters of the normalization layer.
[0084] It should be noted that a pre-set confidence threshold may not be necessary. Regardless of the values of the first and second confidence levels, the statistical and affine parameters of the normalization layer can be determined through the steps described above.
[0085] In the embodiments described in steps 410 to 450 above, the predicted value of the voice interaction state of the target device includes two values: a first candidate interaction state and a second candidate interaction state. The probability that the voice interaction state is the first candidate interaction state is represented by a first confidence level, and the probability that the voice interaction state is the second candidate interaction state is represented by a second confidence level. Furthermore, the first candidate interaction state and the second candidate interaction state correspond to different normalization parameters. In this case, based on the first confidence level and the second confidence level, the first statistical parameter and the second statistical parameter can be weighted and summed to obtain the statistical parameters of the normalization layer. Similarly, based on the first confidence level and the second confidence level, the first affine parameter and the second affine parameter can be weighted and summed to obtain the affine parameters of the normalization layer. The statistical parameters and affine parameters of the normalization layer integrate the parameters of the two candidate interaction states, thus making it suitable for situations where the initial voice signal is long and contains multiple voice interaction states. This improves the applicability of voice processing, facilitates the processing of long initial voice signals, and improves voice processing efficiency compared to the segmentation processing of a single voice interaction state.
[0086] In one embodiment, step 230 includes: Step 510: Based on the playback reference signal, extract target signal features, echo signal features, and noise signal features from the initial speech signal; Step 520: Fuse the target signal features, echo signal features, and noise signal features to obtain fused features; Step 530: Input the fused features into the updated speech processing model to eliminate echo and noise signals and obtain the target speech.
[0087] In step 510, the target signal features are the vector feature representations of the target speech. The echo signal features are the vector feature representations of the echo signal. The noise signal features are the vector feature representations of the noise signal.
[0088] Reference Figure 6 The target signal features can be extracted from the initial speech signal by the target speech extraction module. The noise signal features can be extracted from the initial speech signal by the noise signal extraction module. The echo signal, however, is the echo returned by the playback reference signal played by the target device through the acoustic loop; therefore, the echo signal features need to be extracted from the initial speech signal by the echo signal extraction module based on the playback reference signal.
[0089] It should be noted that the target speech extraction module, noise signal extraction module, and echo signal extraction module are all derived through training. These modules can be speech extraction modules, neural networks, converter models, large language models, etc.
[0090] In step 520, the fused feature is the result of fusing the target signal feature, echo signal feature, and noise signal feature. Specifically, the target signal feature, echo signal feature, and noise signal feature are concatenated to obtain the fused feature. In this way, the information of the target signal feature, echo signal feature, and noise signal feature is completely preserved in the fused feature, without information cancellation, and different features occupy different positions in the fused feature, making it easy to distinguish the three types of features in subsequent processing.
[0091] In step 530, after obtaining the fusion features, the updated speech processing model is used to process the fusion features to eliminate echo and noise signals in the initial speech signal, thereby obtaining the target speech. (Refer to...) Figure 6 The outputs of each module, namely the target signal features, echo signal features, and noise signal features, are fused to obtain fused features, which are then used as the input to the updated speech processing model.
[0092] It should be noted that the target speech extraction module refers to the feature representation used to extract the target speech, i.e., the target signal features, not the final estimator. It initially highlights the distinguishable features of the target speech from the aliased initial speech signal, providing a foundation for subsequent signal separation. The final high-fidelity reconstruction is completed by the speech processing model. Furthermore, in the initial speech signal, the target speech, echo signal, and noise signal spectra are highly aliased, and the target speech extraction module is insufficient for robust separation. Specifically, the target speech, echo signal, and noise signal have highly overlapping spectra, especially in the case of dual-talk, low signal-to-noise ratio, and nonlinear echo. The preliminary target signal features obtained solely by the target speech extraction module are unreliable under all conditions; robust separation requires the fusion and mutual verification of target signal features, echo signal features, and noise signal features. Moreover, the training of the target speech extraction module is only to ensure that the target signal feature extraction function of the module is implemented effectively; it does not guarantee that the speech signal corresponding to the target signal features output by a single channel during inference is the perfect target speech. Therefore, the fused features need to be input into the updated speech processing model to eliminate echo signals and noise signals to obtain the target speech.
[0093] In the embodiments described in steps 510 to 530, target signal features, echo signal features, and noise signal features are extracted from the initial speech signal. At this stage, the target signal features are only a rough representation of the target speech and cannot accurately correspond to the target speech. Therefore, it is necessary to fuse the target signal features, echo signal features, and noise signal features to obtain fused features. These fused features are then processed by an updated speech processing model to obtain the target speech. During the processing of the speech processing model, the target signal features, echo signal features, and noise signal features indicate a general separation direction for the corresponding speech signals. Based on this, the target speech can be robustly separated from the initial speech signal, thereby improving the accuracy of speech processing.
[0094] In one embodiment, the updated speech processing model includes a complex encoder, a cyclic unit, and a complex decoder, and step 530 includes: Step 610: Use a complex encoder to downsample the fused features to obtain complex frequency domain features; Step 620: Input the complex frequency domain features into the cyclic unit to obtain the time-series complex features; Step 630: Input the complex frequency domain features and time-series complex features into the complex decoder to obtain the complex spectrum; Step 640: Convert the complex spectrum into the target speech.
[0095] In step 610, the complex encoder is used to downsample the fused features to obtain complex frequency domain features.
[0096] Complex frequency domain features are the downsampling results of fused features, and also the sampling results of fused features in the frequency domain.
[0097] Reference Figure 7 Complex encoders are typically configured in multiple ways, forming a hierarchical stacked structure. For each complex encoder, it receives the output or fused features of the previous complex encoder, processes its real and imaginary parts simultaneously through complex convolution to mine local time-frequency acoustic features, and performs frequency domain downsampling to obtain complex frequency domain features.
[0098] Furthermore, for multiple complex encoders, shallow complex encoders have high feature resolution and focus on extracting high-frequency details, such as consonants, subtle harmonics in speech, and nonlinear echo harmonics. Mid-level complex encoders fuse local features to capture cross-band correlation patterns and can distinguish between the fundamental frequency of human voice and the spectral structure of echoes. Deep complex encoders have the lowest feature resolution and the largest receptive field, extracting global high-level semantic features to characterize the long-term spectral variation patterns of speech and echoes.
[0099] When there are multiple complex encoders, the complex frequency domain characteristics usually refer to the output of the last complex encoder.
[0100] It should be noted that if a real-number encoder is used, phase information is usually discarded, and features can only be extracted based on the amplitude spectrum, making it difficult to separate the target speech.
[0101] In step 620, the complex frequency domain features extracted by the complex encoder are intra-frame frequency domain features, but the connection between consecutive audio frames is not visible. The recurrent unit is a module unit responsible for temporal modeling and mining the dependencies between consecutive audio frames. The temporal complex features are the vector feature representation of the temporal dependencies of the complex frequency domain features captured by the recurrent unit.
[0102] The cyclic unit is specifically a complex cyclic unit, in which the real and imaginary parts of the complex frequency domain features are included in the calculation, maintaining the complex representation and providing features containing contextual information for the complex decoder, thereby improving the stability of echo suppression.
[0103] In step 630, the complex decoder corresponds to the complex encoder. The complex decoder can upsample the complex frequency domain features and temporal complex features, and reconstruct the complex representation at the original resolution, i.e., the complex spectrum. The complex spectrum is the upsampling result of the complex frequency domain features and temporal complex features by the complex decoder; it is also the spectrum of the target speech in the complex domain, and the complex representation of the target speech in the time-frequency domain. The complex spectrum contains a real part and an imaginary part, and equivalently carries amplitude and phase.
[0104] It should be noted that the output of the complex decoder may be a complex spectrum or a complex mask. Complex masking is a time-frequency domain filtering operator. Multiplying the complex mask by the complex spectrum corresponding to the initial speech signal at each time frequency point yields the complex spectrum of the target speech.
[0105] It should be noted that, referring to Figure 7 With N complex encoders, N complex decoders are also configured, with a one-to-one correspondence between the complex decoders and complex encoders. Each complex decoder is connected to its corresponding complex encoder, enabling skip connections between them. For each complex decoder, the process combines the output from the corresponding complex encoder (i.e., the complex frequency domain features) with the output from the previous complex decoder, and uses the output from the last complex decoder as the complex code for the target speech.
[0106] For the first complex decoder among multiple complex decoders, it processes the complex frequency domain features and temporal complex features output by the first complex encoder and inputs the output results to the next complex decoder.
[0107] A skip connection is established between the complex encoder and the complex decoder, directly transmitting features from each complex encoder to the corresponding complex decoder. During downsampling by multiple complex encoders, high-resolution time-frequency details such as speech consonants and nonlinear echo harmonics are easily lost. By multiplexing shallow multi-scale complex features through the skip connection, the complex decoder simultaneously possesses global temporal context information (temporal complex features) and local fine-grained frequency domain features (complex frequency domain features) from the cyclic unit output during the spectrum reconstruction stage, improving the recovery of weak human voices and harmonic structures. Simultaneously, complete complex information is transmitted throughout the process, ensuring that the speech processing model can achieve joint estimation of the target speech amplitude and phase.
[0108] In step 640, the target speech is the result of a complex spectrum transformation. Specifically, an inverse Fourier transform is performed on the complex spectrum to obtain the target speech.
[0109] In the embodiments described in steps 610 to 640 above, the updated speech processing model includes a complex encoder, a recurrent unit, and a complex decoder. The complex encoder is used to downsample the fused features to obtain complex frequency domain features. The recurrent unit can capture the temporal dependencies of the complex frequency domain features, thereby obtaining temporal complex features. The complex encoder and complex decoder are connected in a skip connection. The complex decoder processes the complex frequency domain features and the temporal complex features, enabling the complex decoder to simultaneously possess global temporal context information (temporal complex features) and local fine-grained frequency domain features (complex frequency domain features) during the spectrum reconstruction stage, thereby obtaining a complex spectrum. This achieves joint estimation of the amplitude and phase of the target speech, improving the accuracy of the separated target speech and thus improving the accuracy of speech processing. In addition, the entire processing of the updated speech processing model is performed in the complex domain, enabling more robust separation of target speech in low signal-to-noise ratio scenarios with negative signal-to-noise ratios.
[0110] In one embodiment, the speech processing model is trained in the following manner: Step 710: Acquire sample speech signal, sample reference signal, sample noise signal, and multiple preset speech interaction states; Step 720: Determine the sample echo signal based on the sample reference signal; Step 730: For each preset voice interaction state, based on the preset voice interaction state, mix the sample voice signal, sample echo signal and sample noise signal to obtain the initial voice signal of the sample. Step 740: Input the initial speech signal and the reference signal into the speech processing model to obtain the processed speech signal; Step 750: Based on the sample processed speech and sample speech signals, train the speech processing model to obtain the updated speech processing model corresponding to the preset speech interaction state. Step 760: Integrate the updated speech processing models corresponding to the multiple preset speech interaction states to obtain the speech processing model.
[0111] In step 710, the sample speech signal is the speech data known during the training of the speech processing model, and it is also the user speech received by the target device at the near end. The sample speech signal corresponds to the target speech.
[0112] The sample reference signal is the speech signal played by the device during the training of the speech processing model. The sample reference signal corresponds to the playback reference signal.
[0113] The sample noise signal is the noise data known during the training of the speech processing model; the sample noise signal corresponds to the noise signal. The difference is that the noise signal in the initial speech signal is unknown, while the sample noise signal is known.
[0114] To enable the trained speech processing model to separate the target speech from the initial speech signal in each speech interaction state, the model needs to be trained using data from each speech interaction state. Preset speech interaction states are the pre-defined speech interaction states that require training. Multiple preset speech interaction states can include far-end single-talk, near-end single-talk, dual-talk, and noise-only.
[0115] In step 720, the sample echo signal is the result of processing the sample reference signal. Specifically, the sample reference signal can be processed by imitating an acoustic circuit to obtain the sample echo signal.
[0116] It should be noted that the processing of the sample reference signal can be linear or nonlinear, and the specific settings can be configured as needed.
[0117] In step 730, for each preset voice interaction state, the sample voice signal, sample echo signal and sample noise signal are mixed based on the preset voice interaction state to obtain the initial voice signal of the sample.
[0118] The preset voice interaction states can include far-end single-talk, near-end single-talk, dual-talk, and noise only. Assuming the preset voice interaction state is far-end single-talk, the sample echo signal and sample noise signal are mixed to obtain the initial sample voice signal. Assuming the preset voice interaction state is near-end single-talk, the sample voice signal and sample noise signal are mixed to obtain the initial sample voice signal. Assuming the preset voice interaction state is dual-talk, the sample voice signal, sample echo signal, and sample noise signal are mixed to obtain the initial sample voice signal. Assuming the preset voice interaction state is noise only, the sample noise signal can be directly used as the initial sample voice signal.
[0119] In step 740, the sample processed speech is the result of the speech processing model processing the initial sample speech signal and the sample reference signal, and it is also the sample speech signal separated by the speech processing model from the initial sample speech signal.
[0120] In step 750, the sample processed speech is the prediction result of the speech processing model on the sample speech signal. The training objective of the speech processing model is to make the output result, i.e., the sample processed speech, infinitely close to, or even equal to, the sample speech signal. Therefore, it is necessary to train the speech processing model based on the difference between the sample processed speech and the sample speech signal to obtain the updated speech processing model corresponding to the preset speech interaction state.
[0121] It should be noted that during the training of the speech processing model, for the updated speech processing model under different preset speech interaction states, it is necessary to ensure that only the parameters of the normalization layer of each updated speech processing model are different, while the parameters of the remaining parts of the complex encoder, recurrent unit, and complex decoder remain the same.
[0122] In step 760, the speech processing model is the integrated result of the updated speech processing models corresponding to multiple preset speech interaction states.
[0123] It should be noted that the statistical and affine parameters of the normalization layer of the speech processing model are variable parameters that change with the speech interaction state, thereby enabling the processing of initial speech signals for different speech interaction states through a single speech processing model.
[0124] The embodiments of steps 710 to 760 above are for training a speech processing model. During the training process, the sample speech signal, sample reference signal, sample noise signal, and sample echo signal are all known data. To enable the trained speech processing model to process initial speech signals under different speech interaction states, multiple preset speech interaction states are set. Based on these preset states, the sample speech signal, sample echo signal, and sample noise signal are mixed to obtain the initial sample speech signal, which matches the actual preset speech interaction state. Then, the initial sample speech signal and the sample reference signal are input into the speech processing model to obtain the processed sample speech. The processed sample speech is the prediction result of the speech processing model on the sample speech signal. Therefore, the speech processing model can be trained based on the processed sample speech and the sample speech signal, so that the updated speech output by the speech processing model corresponding to the preset speech interaction state is infinitely close to, or even equal to, the sample speech signal, thereby improving the accuracy of the processed sample speech output by the speech processing model and improving the accuracy of speech processing. Furthermore, by integrating the updated speech processing models corresponding to multiple preset speech interaction states, a single speech processing model is obtained to process the initial speech signals of different speech interaction states, reducing the model's footprint. Moreover, the speech processing model is an end-to-end processing model, allowing noise suppression and echo cancellation to be processed simultaneously, reducing error accumulation and module interference issues caused by multi-level module processing, and improving the accuracy of speech processing.
[0125] In one embodiment, step 750 includes: Step 810: Determine the speech quality loss based on the processed speech and sample speech signals; Step 820: Determine the echo suppression loss based on the preset voice interaction state and the sample processed voice; Step 830: Based on speech quality loss and echo suppression loss, train the speech processing model to obtain the updated speech processing model corresponding to the preset speech interaction state.
[0126] In step 810, the speech quality loss is a loss function related to the target speech or the sample processed speech. The speech quality loss is set to improve the intelligibility and listenability of the target speech.
[0127] Speech quality loss is specifically determined based on the difference between the processed speech and the original speech signal. Various methods exist for calculating speech quality loss, such as scale-invariant signal-to-noise ratio loss, spectral amplitude loss, complex spectrum reconstruction loss, and differentiable proxy loss based on perceptual evaluation of speech quality (PESQ) or short-time objective intelligibility (STOI). One of these methods can be chosen to calculate speech quality loss, or two or more can be selected and calculated using a weighted summation.
[0128] In step 820, the echo suppression loss is a loss function associated with the echo signal or the sample echo signal. The echo suppression loss is set to suppress the echo signal in the initial speech signal and also to suppress the sample echo signal in the initial sample speech signal, so as to reduce the residual echo in the generated target speech or sample processed speech.
[0129] There are various methods for calculating echo suppression loss, which can be selected based on the current preset voice interaction state. Examples include residual echo power constraint, dual-talk protection storage, residual echo harmonic band constraint, and echo component masking supervision. Residual echo power constraint is only applied in the preset voice interaction state of far-end single-talk and noise-only. In this state, the ideal value for the sample processed speech should be silence, so the output is directly constrained, i.e., the residual energy of the sample processed speech is minimized. Dual-talk protection loss is applied in the dual-talk state, used to constrain echo suppression while ensuring that the near-end speech is not distorted. The specific dual-talk protection loss can be determined based on the difference between the sample processed speech and the sample speech signal. In this embodiment, echo signal extraction module extracts echo signal features from the initial speech signal. Similarly, during the training phase, the echo signal extraction module can extract sample echo signal features from the initial sample speech signal. Echo component masking supervision acts on the echo signal extraction module, and it is calculated based on the difference between the sample echo signal features and the corresponding feature vectors of the sample echo signal. The echo harmonic frequency band residual constraint is used to constrain the sample-processed speech. The frequency band set B containing the sample echo signal is determined by the sample reference signal. In the final output sample-processed speech, the residual energy falling within frequency band set B and related to the echo is measured and minimized. In the preset voice interaction state of a single-talk session at the far end, B can be directly constrained to 0. However, in the preset voice interaction state of a two-talk session, the sample reference signal is used as a reference, and only the residual echo signal related to the sample reference, i.e., the sample echo signal, is measured. The sample echo signal is mainly used to locate the frequency band and measure the residual echo in the sample-processed speech.
[0130] The calculation method for echo suppression loss can be selected from any one or more of the following as needed: residual echo power constraint, dual-talk protection storage, residual constraint of echo harmonic frequency band, and echo component masking supervision.
[0131] In step 830, the speech processing model is trained based on speech quality loss and echo suppression loss to obtain the updated speech processing model corresponding to the preset speech interaction state.
[0132] The speech processing model is trained simultaneously based on speech quality loss and echo suppression loss. This is done by enhancing the sample speech signal in the initial speech signal through speech quality loss and suppressing the sample echo signal in the initial speech signal to reduce the residual echo in the processed speech and make it closer to the original speech signal.
[0133] It should be noted that, referring to Figure 3 The overall training function of the speech processing model can be determined based on the weighted sum of speech quality loss and echo suppression loss. During the calculation of the overall training function, the speech quality loss and echo suppression loss can be set according to requirements. For models with higher requirements for speech quality, the weight of the speech quality loss is higher; for models that focus more on echo suppression, the weight of the echo suppression loss is higher. This weighting achieves a balanced optimization of speech quality and echo suppression.
[0134] In the embodiments described in steps 810 to 830 above, the speech processing model is trained jointly based on speech quality loss and echo suppression loss. The speech quality loss is determined based on the difference between the processed speech and the original speech signal, and is set to improve the intelligibility and listenability of the target speech. The echo suppression loss is determined based on a preset speech interaction state and the processed speech, and is set to suppress residual echoes in the processed speech. The combination of speech quality loss and echo suppression loss, used to train the speech processing model, enhances the original speech signal while suppressing the echo signal, thereby improving the performance and accuracy of the speech processing model. Furthermore, it reduces multi-module interference and cascade error accumulation caused by serial cascading, allowing the target speech to be output directly from a single model, further improving the accuracy of speech processing.
[0135] In one embodiment, step 740 includes: Step 910: Extract the target signal features from the initial speech signal of the sample using the target speech extraction module; Step 920: Use the echo signal extraction module to extract the sample echo signal features from the initial speech signal of the sample based on the sample reference signal; Step 930: Extract sample noise signal features from the initial speech signal of the sample using the noise signal extraction module; Step 940: Fuse the sample target signal features, sample echo signal features, and sample noise signal features to obtain sample fusion features, and input the sample fusion features into the speech processing model to obtain the sample processed speech.
[0136] In step 910, the target speech extraction module is the module that extracts target signal features from the initial speech signal, and also the module that extracts sample target signal features from the initial sample speech signal.
[0137] In step 920, the echo signal extraction module is the module that extracts echo signal features from the initial speech signal, and it is also the module that extracts sample echo signal features from the sample initial speech signal. Since the sample echo signal is generated based on the sample reference signal, the echo signal extraction module specifically extracts sample echo signal features from the sample initial speech signal based on the sample reference signal.
[0138] In step 930, the noise signal extraction module is the module that extracts noise signal features from the initial speech signal, and also the module that extracts sample noise signal features from the initial speech signal of the sample.
[0139] In step 940, the sample fusion feature is the result of fusing the sample target signal feature, the sample echo signal feature, and the sample noise signal feature. Specifically, the sample target signal feature, the sample echo signal feature, and the sample noise signal feature are concatenated to obtain the sample fusion feature.
[0140] After obtaining the sample fusion features, the sample fusion features are processed using a speech processing model to obtain the target speech.
[0141] In the embodiments described in steps 910 to 940 above, sample target signal features, sample echo signal features, and sample noise signal features are extracted from the initial speech signal. At this point, the sample target signal features indicate a coarse feature of the sample speech signal and cannot accurately correspond to the sample speech signal. Therefore, the fusion result of the sample target signal features, sample echo signal features, and sample noise signal features is used as the fusion feature, and the sample fusion feature is input into the speech processing model to obtain the sample processed speech. During the processing of the speech processing model, the sample target signal features, sample echo signal features, and sample noise signal features indicate a general separation direction for the corresponding speech signal. Based on this, the sample processed speech can be robustly separated from the initial sample speech signal, thereby improving the accuracy of speech processing.
[0142] In one embodiment, step 830 includes: Step 1010: Determine the first loss function based on the difference between the sample target signal features and the corresponding vector features of the sample speech signal; Step 1020: Determine the second loss function based on the difference between the features of the sample echo signal and the corresponding vector features of the sample echo signal; Step 1030: Determine the third loss function based on the difference between the features of the sample noise signal and the corresponding vector features of the sample noise signal; Step 1040: Based on speech quality loss, echo suppression loss, first loss function, second loss function and third loss function, jointly train the speech processing model, target speech extraction module, echo signal extraction module and noise signal extraction module to obtain the updated speech processing model corresponding to the preset speech interaction state.
[0143] In step 1010, the first loss function is determined based on the difference between the features of the sample target signal and the corresponding vector features of the sample speech signal. The first loss function constrains the target speech extraction module so that the features of the sample target signal extracted by the target speech extraction module are infinitely close to the vector features of the sample speech signal. Specifically, the first loss function can be a near-end speech masking loss, used to constrain the output of the target speech extraction module to approximate the known near-end speech, i.e., the sample speech signal.
[0144] In step 1020, the second loss function is determined based on the difference between the features of the sample echo signal and the corresponding vector features of the sample echo signal. The second loss function constrains the echo signal extraction module so that the features of the sample echo signal extracted by the echo signal extraction module are infinitely close to the vector features of the sample echo signal. Specifically, the second loss function can be an echo masking loss, constraining the features of the sample echo signal to approximate the masking or spectrum corresponding to the known sample echo signal.
[0145] In step 1030, the third loss function is determined based on the difference between the features of the sample noise signal and the corresponding vector features of the sample noise signal. This third loss function is used to train the noise signal extraction module. Specifically, the third loss function can be a noise masking loss, which constrains the features of the sample noise signal to approximate the masking or spectrum corresponding to the known sample noise signal.
[0146] In step 1040, after determining the first loss function, the second loss function, and the third loss function, the first loss function, the second loss function, and the third loss function can be superimposed on the speech quality loss and echo suppression loss. Based on the superposition result, the speech processing model, the target speech extraction module, the echo signal extraction module, and the noise signal extraction module are jointly trained to obtain the updated speech processing model corresponding to the preset speech interaction state. The superposition can be performed by weighted summation.
[0147] Reference Figure 6During the training phase, the target speech extraction module, echo signal extraction module, and noise signal extraction module are trained simultaneously using a multi-task head. The task head is a part of the deep learning model and is responsible for training these three modules.
[0148] It should be noted that, in addition to the first loss function, the second loss function, and the third loss function, a hybrid consistency loss can also be set to constrain the sum of the reconstructed sample target signal features, sample echo signal features, and sample noise signal features to be approximately equal to the initial sample speech signal, so as to ensure separation consistency.
[0149] In addition to the speech quality loss and echo suppression loss, the embodiments of steps 1010 to 1040 above also include a first loss function, a second loss function, and a third loss function. The first loss function is mainly used to train the target speech extraction module, the second loss function is mainly used to train the echo signal extraction module, and the third loss function is mainly used to train the noise signal extraction module. This ensures that the features extracted by each module are close to the features of the corresponding speech signal, thereby improving the accuracy of the separation direction indicated by the sample target signal features, sample echo signal features, and sample noise signal features as the corresponding speech signal during the speech processing model, and thus improving the accuracy of speech processing.
[0150] In one embodiment, step 720 includes: Step 1110: Obtain multiple preset extension methods; Step 1120: Select the target extension method from a variety of preset extension methods; Step 1130: Perform nonlinear extension on the sample reference signal based on the target extension method to obtain the sample echo signal.
[0151] In step 1110, the preset extension method is an extension method based on the sample reference signal to generate the sample echo signal, which is also a method of simulating acoustic circuit.
[0152] To improve the ability of speech processing to handle nonlinear echoes, the default extension method is usually nonlinear extension, such as harmonic processing, polynomial or Volterra extension, speaker volume or state conditions, learnable nonlinear encoders, etc.
[0153] In step 1120, the target expansion method is one of several preset expansion methods. A preset expansion method can be randomly selected as the target expansion method, or the target expansion methods can be selected sequentially, so that the sample echo signals of the training data are signals generated by multiple preset expansion methods, thereby improving the applicability of the speech processing model in processing nonlinear echoes.
[0154] In step 1130, after determining the target expansion method, the sample reference signal can be nonlinearly expanded based on the target expansion method to obtain the sample echo signal.
[0155] Reference Figure 6 The sample echo signal input to the echo signal extraction module is a nonlinear extension of the playback reference signal.
[0156] The embodiments of steps 1110 to 1130 above perform nonlinear extension on the sample reference signal to obtain the sample echo signal. In this way, the obtained sample echo signal is a nonlinear echo. The speech processing model trained based on the nonlinear echo can suppress the nonlinear echo, so as to more robustly separate the target speech in nonlinear distortion scenarios such as mobile phones or small speakers, thereby improving the accuracy of speech processing.
[0157] In one embodiment, reference is made to Figure 8 In (a) of this application, the speech processing model is trained based on a weighted combination of speech quality loss and noise suppression loss. (Refer to...) Figure 8 In (b) of this paper, the related technologies can achieve accurate separation of target speech under high signal-to-noise ratio (SNR) conditions, but the separation effect is poor under low SNR or even negative SNR conditions. However, the embodiments of this application can achieve robust separation of target speech under high SNR, low SNR, and negative SNR conditions, thereby improving the accuracy of speech processing.
[0158] Additionally, refer to Figure 9 This application provides a voice processing device 1100, including: The signal acquisition unit 901 is used to acquire an initial voice signal and a playback reference signal from the target device. The initial voice signal includes an echo signal and a noise signal, and the echo signal is related to the playback reference signal. The status determination unit 902 is used to determine the voice interaction status of the target device. The voice interaction status indicates the signal type contained in the initial voice signal. The model processing unit 903 is used to update the speech processing model based on the speech interaction state to obtain the updated speech processing model, and input the initial speech signal and the playback reference signal into the updated speech processing model to obtain the target speech. The speech processing model is used to eliminate echo signals and noise signals.
[0159] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0161] This application also provides an electronic device, see embodiments thereof. Figure 10 The electronic device 1000 includes: at least one processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on at least one processor 1001. When the processor 1001 executes the computer program 1003, it implements the steps in any of the above-described method embodiments.
[0162] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.
[0163] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.
[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0166] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0167] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0169] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A speech processing method, characterized in that, include: Acquire an initial voice signal and a playback reference signal from the target device, wherein the initial voice signal includes an echo signal and a noise signal, and the echo signal is related to the playback reference signal; Determine the voice interaction state of the target device, wherein the voice interaction state indicates the signal type contained in the initial voice signal, and the voice interaction state includes a first candidate interaction state and a second candidate interaction state. Determine the first confidence level corresponding to the first candidate interaction state and the second confidence level corresponding to the second candidate interaction state; Based on the first candidate interaction state, determine the first statistical parameter and the first affine parameter; Based on the second candidate interaction state, determine the second statistical parameter and the second affine parameter; Based on the first confidence level and the second confidence level, the first statistical parameter and the second statistical parameter are weighted and summed to obtain the statistical parameter; Based on the first confidence level and the second confidence level, the first affine parameter and the second affine parameter are weighted and summed to obtain the affine parameter; The speech processing model is updated based on the statistical parameters and the affine parameters to obtain the updated speech processing model. The initial speech signal and the playback reference signal are then input into the updated speech processing model to obtain the target speech. The speech processing model is used to eliminate the echo signal and the noise signal.
2. The speech processing method according to claim 1, characterized in that, The speech processing model includes multiple normalization layers; updating the speech processing model based on the statistical parameters and the affine parameters to obtain the updated speech processing model includes: For each of the normalization layers, the normalization layer is updated based on the statistical parameters and the affine parameters to obtain the updated normalization layer; Based on the updated normalization layer corresponding to each of the multiple normalization layers, the speech processing model is updated to obtain the updated speech processing model.
3. The speech processing method according to claim 1, characterized in that, The step of inputting the initial speech signal and the playback reference signal into the updated speech processing model to obtain the target speech includes: Based on the playback reference signal, target signal features, echo signal features, and noise signal features are extracted from the initial speech signal; The target signal features, the echo signal features, and the noise signal features are fused to obtain fused features; The fused features are input into the updated speech processing model to obtain the target speech.
4. The speech processing method according to claim 3, characterized in that, The updated speech processing model includes a complex encoder, a cyclic unit, and a complex decoder; the step of inputting the fused features into the updated speech processing model to obtain the target speech includes: The complex encoder is used to downsample the fused features to obtain complex frequency domain features; The complex frequency domain features are input into the cyclic unit to obtain the time-series complex features; The complex frequency domain features and the temporal complex features are input into the complex decoder to obtain the complex spectrum; The complex spectrum is converted into target speech.
5. The speech processing method according to claim 1, characterized in that, The speech processing model was trained in the following way: Acquire sample speech signals, sample reference signals, sample noise signals, and multiple preset speech interaction states; Based on the sample reference signal, the sample echo signal is determined; For each preset voice interaction state, the sample voice signal, the sample echo signal, and the sample noise signal are mixed based on the preset voice interaction state to obtain the initial sample voice signal; The initial speech signal and the reference signal are input into the speech processing model to obtain the processed speech signal. Based on the sample processed speech and the sample speech signal, train the speech processing model to obtain the updated speech processing model corresponding to the preset speech interaction state; The updated voice processing models corresponding to the multiple preset voice interaction states are integrated to obtain a voice processing model.
6. The speech processing method according to claim 5, characterized in that, The step of training the speech processing model based on the sample processed speech and the sample speech signal to obtain the updated speech processing model corresponding to the preset speech interaction state includes: Based on the processed speech and the sample speech signal, determine the speech quality loss; Based on the preset voice interaction state and the sample processed voice, determine the echo suppression loss; Based on the speech quality loss and the echo suppression loss, the speech processing model is trained to obtain the updated speech processing model corresponding to the preset speech interaction state.
7. The speech processing method according to claim 6, characterized in that, The step of inputting the initial sample speech signal and the sample reference signal into the speech processing model to obtain the processed sample speech includes: The target speech extraction module is used to extract target signal features from the initial speech signal of the sample. The echo signal extraction module extracts sample echo signal features from the initial speech signal of the sample based on the sample reference signal; The noise signal extraction module is used to extract sample noise signal features from the initial speech signal of the sample. The sample target signal features, the sample echo signal features, and the sample noise signal features are fused to obtain sample fusion features, and the sample fusion features are input into the speech processing model to obtain sample processed speech.
8. The speech processing method according to claim 7, characterized in that, The step of training the speech processing model based on the speech quality loss and the echo suppression loss to obtain the updated speech processing model corresponding to the preset speech interaction state includes: Based on the difference between the sample target signal features and the corresponding vector features of the sample speech signal, a first loss function is determined; A second loss function is determined based on the difference between the features of the sample echo signal and the corresponding vector features of the sample echo signal; Based on the difference between the sample noise signal features and the corresponding vector features of the sample noise signal, a third loss function is determined; Based on the speech quality loss, the echo suppression loss, the first loss function, the second loss function, and the third loss function, the speech processing model, the target speech extraction module, the echo signal extraction module, and the noise signal extraction module are jointly trained to obtain the updated speech processing model corresponding to the preset speech interaction state.
9. The speech processing method according to claim 5, characterized in that, The step of determining the sample echo signal based on the sample reference signal includes: Get multiple preset extension methods; Select the target extension method from the multiple preset extension methods; The sample reference signal is nonlinearly extended based on the target extension method to obtain the sample echo signal.
10. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device causes the electronic device to implement the voice processing method as described in any one of claims 1 to 9.
Citation Information
Patent Citations
Echo cancellation method and system based on neural network double-talk detection
CN115457928A
Echo cancellation algorithm
CN119007737A
