Model training and speech enhancement method, device and equipment based on double tasks

Abstract

The invention discloses a training method of a speech enhancement model based on double tasks, and a speech enhancement method, device and equipment. Although a traditional voice enhancement task does not need to register voice signals to provide effective information, the consistency of a model input structure is ensured by designing invalid registered voice signals. Both a traditional speech enhancement sample and a personalized speech enhancement sample have the same input form when entering the model, and a traditional speech enhancement task and a personalized speech enhancement task are seamlessly integrated into a single model, so that the complexity of model architecture design is reduced, and the design efficiency is improved. According to the method, the model can conveniently adopt a unified network layer structure to process the two types of samples, parameter sharing and cooperative training among different tasks are achieved, the universality and generalization ability of the model on different tasks are improved, and deployment memory and resource waste are reduced.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and speech enhancement, and in particular to a training method for a dual-task-based speech enhancement model and a speech enhancement method, device and equipment. Background Art

[0002] With the widespread adoption of smart home devices and the increasing use of multi-speaker conferencing systems in business, education, and other scenarios, voice data processing faces unprecedented challenges. In practical applications, interference factors such as overlapping speech, ambient noise, and room reverberation severely impact voice quality, reducing both intelligibility and clarity. For example, in multi-speaker scenarios, the traditional "cocktail party problem" reveals that humans have the ability to focus on a specific speaker and flexibly shift attention in noisy environments. However, replicating this ability in machines remains a major challenge in the field of voice processing.

[0003] To address these issues, both traditional and personalized speech enhancement technologies have been developed. Traditional speech enhancement technologies focus on removing noise and reverberation to improve speech intelligibility; personalized speech enhancement technologies leverage registered speech signals as auxiliary information to accurately extract and enhance the target speaker's speech data in complex multi-speaker environments. In recent years, the Deep Noise Suppression Challenge and the URGENT Challenge have significantly advanced the development of these two technologies. In particular, the URGENT Challenge, by proposing a baseline model that emphasizes universality, robustness, and generalization, has effectively promoted the application of dual-task speech enhancement models in complex scenarios, including those with varying environmental conditions, sampling rates, and microphone configurations.

[0004] However, current research still has significant shortcomings. On the one hand, traditional speech enhancement and personalized speech enhancement require separate training models. Taking traditional speech enhancement models such as DPCCN, BSRNN, Sepformer, and DeepFilterNet as examples, their personalized versions (sDPCCN, pBSRNN, X-Sepformer, and pDeepFilterNet) require additional branches to process the registered speech signal. This requires training different weight sets and deploying independent models, significantly increasing memory and computational overhead and reducing industrial application efficiency. On the other hand, there is a lack of effective methods to integrate the two into a unified model, and existing models lack robustness when dealing with diverse tasks and complex application scenarios. Therefore, studying a unified model framework that can take into account both traditional speech enhancement and personalized speech enhancement tasks has become a key issue that needs to be addressed. Summary of the Invention

[0005] The present application provides a training method for a dual-task speech enhancement model and a speech enhancement method, apparatus, device and medium, which are used to solve the problem that the existing execution of traditional speech enhancement and personalized speech enhancement tasks requires different models to process, resulting in large memory and computing overhead.

[0006] In a first aspect, the present application provides a training method for a dual-task speech enhancement model, the method comprising:

[0007] Obtain any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein each speech sample also corresponds to a standard clean speech signal, each speech sample includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech corresponding to each traditional speech enhancement sample is the same invalid registration speech signal, and each personalized speech enhancement sample corresponds to its own personalized registration speech signal;

[0008] Obtaining, by an original personalized speech enhancement model, a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal;

[0009] Based on the predicted clean speech signal and the standard clean speech signal, the original personalized speech enhancement model is iteratively trained to obtain a trained dual-task speech enhancement model, which has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0010] In a second aspect, the present application further provides a speech enhancement method based on the above-mentioned model, the method comprising:

[0011] Obtaining a voice signal to be processed;

[0012] Determining a target registration voice signal for the voice signal to be processed; wherein, if the voice signal to be processed is a traditional voice enhancement signal, the target registration voice signal is a pre-configured invalid registration voice signal; if the voice signal to be processed is a personalized voice enhancement signal, the target registration voice signal is a pre-configured personalized registration voice signal;

[0013] A clean speech signal after speech enhancement is determined based on the speech signal to be processed and the target registered speech signal through a pre-trained dual-task speech enhancement model.

[0014] In a third aspect, the present application further provides a method and apparatus for training a dual-task speech enhancement model, the apparatus comprising:

[0015] An acquisition unit is configured to acquire any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein each of the speech samples also corresponds to a standard clean speech signal, each of the speech samples includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech signals corresponding to each of the traditional speech enhancement samples are the same invalid registration speech signal, and each of the personalized speech enhancement samples corresponds to its own personalized registration speech signal;

[0016] a processing unit, configured to obtain, by using an original personalized speech enhancement model, a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal;

[0017] A training unit is used to iteratively train the original personalized speech enhancement model based on the predicted clean speech signal and the standard clean speech signal to obtain a trained dual-task speech enhancement model, wherein the dual-task speech enhancement model has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0018] In a fourth aspect, the present application further provides a speech enhancement device based on the above-mentioned model, the device comprising:

[0019] An acquisition module, used for acquiring a speech signal to be processed;

[0020] a determination module, configured to determine a target registration voice signal for the voice signal to be processed; wherein, if the voice signal to be processed is a traditional voice enhancement signal, the target registration voice signal is a pre-configured invalid registration voice signal; and if the voice signal to be processed is a personalized voice enhancement signal, the target registration voice signal is a pre-configured personalized registration voice signal;

[0021] The processing module is used to determine a clean speech signal after speech enhancement based on the speech signal to be processed and the target registered speech signal through a pre-trained dual-task speech enhancement model.

[0022] In a fifth aspect, the present application provides a computer device comprising a processor, wherein the processor is configured to implement the steps of the training method of the dual-task based speech enhancement model as described above when executing a computer program stored in a memory, or to implement the steps of the speech enhancement method as described above.

[0023] In a sixth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training method of the dual-task-based speech enhancement model as described above, or implements the steps of the speech enhancement method as described above.

[0024] The beneficial effects of this application are as follows:

[0025] 1. A pre-collected dual-task dataset containing a large number of speech samples, corresponding registered speech signals, and standard clean speech signals provides rich and comprehensive data support for model training. The speech samples cover both traditional and personalized speech enhancement samples, enabling the model to learn the characteristics of different speech enhancement tasks. This adapts to various scenarios, such as single-speaker distorted speech processing and extracting the target speaker's voice from complex mixed speech, greatly broadening the model's application scope.

[0026] 2. By assigning the same invalid registration voice signal to traditional speech enhancement samples, the independence and purity of the task is ensured without interfering with the traditional speech enhancement task processing logic, allowing the model to focus on optimizing the processing of distorted speech signals from a single speaker and improving speech clarity. Personalized speech enhancement samples, corresponding to their own personalized registration voice signals, provide the model with a reference to the target speaker's unique voice features in the dual-input, single-output personalized speech enhancement task. This helps the model accurately extract the target speaker's voice from noisy speech signals, meeting the needs of specific speech enhancement processing in complex multi-speaker environments.

[0027] 3. Although traditional speech enhancement tasks do not require registered speech signals to provide effective information, the design of an invalid registered speech signal ensures the consistency of the model input structure. Whether traditional speech enhancement samples or personalized speech enhancement samples, they have the same input format when entering the model. This seamlessly integrates traditional speech enhancement tasks and personalized speech enhancement tasks into a single model. This reduces the complexity of model architecture design and facilitates the model to process both types of samples using a unified network layer structure. This enables parameter sharing and collaborative training between different tasks, improves the model's versatility and generalization capabilities across different tasks, and saves deployment memory and resource waste.

[0028] 4. The trained dual-task speech enhancement model has both traditional speech enhancement and personalized speech enhancement capabilities. It can perform both traditional and personalized speech enhancement tasks simultaneously, reducing the memory and computing overhead required to simultaneously meet both tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0030] Figure 1 A schematic diagram of the training process of a dual-task speech enhancement model provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the training process of a dual-task speech enhancement model provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a speech enhancement process provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a specific speech enhancement process provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the structure of a dual-task speech enhancement model training device provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of a speech enhancement device provided in an embodiment of the present application;

[0036] Figure 7 It is a structural diagram of a computer device provided in an optional embodiment of the present application. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0038] In order to reduce the memory and computing overhead required to perform both traditional speech enhancement and personalized speech enhancement tasks, the present application provides a training method for a dual-task-based speech enhancement model and a speech enhancement method, apparatus, device, and medium.

[0039] Example 1:

[0040] This application provides a training method for a dual-task speech enhancement model. Figure 1 A schematic diagram of a training process for a dual-task speech enhancement model provided in an embodiment of the present application, the process comprising:

[0041] S101: Obtain any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein, each of the speech samples also corresponds to a standard clean speech signal, each of the speech samples includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech corresponding to each of the traditional speech enhancement samples is the same invalid registration speech signal, and each of the personalized speech enhancement samples corresponds to its own personalized registration speech signal.

[0042] In the present application, the training method of the dual-task based speech enhancement model is applied to a computer device, which can be an intelligent terminal, such as a computer, a robot, etc., or a server, such as an application server, a business server, etc.

[0043] In order to obtain a high-performance dual-task speech enhancement model, this application pre-collected a dual-task dataset, which was used as the basis for model training, and finally obtained a trained dual-task speech enhancement model. The dual-task dataset serves as the basic data source for the entire model training and contains a large number of speech samples and their corresponding registered speech signals and standard clean speech signals. Among them, the format of the speech samples can be common PCM, WAV, etc., and the registered speech signals and standard clean speech signals are also stored in an adapted audio format.

[0044] In order to realize the parallel processing of traditional speech enhancement tasks and personalized speech enhancement tasks by the model, this application divides the speech samples into traditional speech enhancement samples and personalized speech enhancement samples. Among them, the traditional speech enhancement task is a single-input and single-output task. Without relying on the registered speech signal, the model can process the distorted speech signal of a single speaker to improve speech clarity, remove noise and reverberation or other interference factors. For the personalized speech enhancement task, it is a dual-input and single-output task. The model needs to use the registered speech signal as an auxiliary signal to extract the speech signal of the target speaker from the noisy speech signal. Based on this, if you want to realize the model to perform personalized speech enhancement tasks, you need to use a two-input and one-output personalized speech enhancement model as the original personalized speech enhancement model. On this basis, although the traditional speech enhancement task does not require the registration voice signal to provide effective information in processing logic, in order to avoid increasing the complexity of model design and the difficulty of training due to excessive differences in data input, and to improve the versatility and efficiency of model training, it is possible to design invalid registration voice signals to achieve the independence and purity of the traditional speech enhancement task without interfering with the traditional speech enhancement samples, so that it can focus on the processing and optimization of the distorted speech signals of a single speaker, and ensure the consistency of the model input.

[0045] In a possible implementation, the invalid registration voice signal is determined as follows:

[0046] When the original personalized speech enhancement model is a model that performs splicing processing on the speech sample and the registration speech signal, the invalid registration speech signal is an all-0 speech signal;

[0047] When the original personalized speech enhancement model is a model that performs multiplication processing on the speech sample and the registration speech signal, the invalid registration speech signal is an all-1 speech signal.

[0048] Specifically, for the original personalized speech enhancement model that uses splicing processing, the all-0 speech signal is used as an invalid registration speech signal for the traditional speech enhancement sample. This is because in the process of splicing the speech sample with the registration speech signal, the all-0 speech signal will not change the original characteristics and data distribution of the speech sample. After splicing, it only exists as filler data and will not introduce additional interference information for the traditional speech enhancement task, thereby ensuring that the model can process the traditional speech enhancement sample according to the predetermined single-input single-output logic. At the same time, the operation of splicing the personalized registration speech signal with the personalized speech enhancement sample maintains formal consistency, so that the model can receive and process the two types of samples in a unified manner at the data input level, reducing the complexity of the model architecture design and facilitating parameter sharing and collaborative training of the model between different tasks.

[0049] When the original personalized speech enhancement model uses multiplication processing, the all-one speech signal has unique advantages as an invalid registration speech signal. In the multiplication operation, after the all-one speech signal is multiplied with the traditional speech enhancement sample, the original numerical value of the speech sample will not be changed. This is equivalent to directly passing the original speech sample data, and will not have any substantial impact on the processing process of the traditional speech enhancement task, thus ensuring the independence of the traditional speech enhancement task. This design ensures that the data processing flow and form of traditional speech enhancement samples and personalized speech enhancement samples are consistent when entering the model for multiplication processing, which is conducive to the unified design and optimization of the internal network structure of the model. In addition, the all-one speech signal, as an invalid registration speech signal, can be compared with the personalized registration speech signal under the multiplication operation mechanism during the model training process, helping the model to better learn and distinguish valid registration speech features from invalid data, thereby improving the model's sensitivity and extraction ability for registration speech features in personalized speech enhancement tasks.

[0050] In one possible implementation, each personalized speech enhancement sample includes at least noisy speech signals from multiple speakers, ensuring that the trained dual-task speech enhancement model can effectively extract the target speaker's clean speech signal from the noisy multi-speaker speech signals. This allows the model to provide high-quality personalized speech enhancement services even in complex, noisy environments with multiple speakers, significantly improving the model's robustness. Optionally, the personalized speech enhancement samples may also include one or more of the following: noisy speech signals from a single speaker, clean speech signals from a single speaker, or clean speech signals from multiple speakers. For a noisy single-speaker speech signal, the model learns the target speaker's specific speech features, such as unique timbre, intonation, and pronunciation habits, contained in the corresponding personalized registration speech signal. This is then accurately matched and integrated with the noisy speech signal. This allows the model to effectively suppress background noise and other interfering signals, accurately extract the target speaker's speech content, and improve the speech enhancement effect. For example, in an intelligent customer service scenario, the customer service agent's speech may be affected by ambient noise. Through this training approach, the model can specifically enhance the customer service agent's speech, allowing users to hear the service content more clearly. For clean speech signals from a single speaker, this data can facilitate the model's learning of standard speech feature references for the target speaker's pronunciation content without noise interference, preventing the model from being disturbed by pronunciation content and affecting the speech enhancement effect. For clean speech signals from multiple speakers, this type of data provides the model with rich multi-speaker speech feature comparison information, helping the model to deeply learn the differences and commonalities in speech features between different speakers. By analyzing the clean speech signals of multiple speakers, the model can construct a more comprehensive and accurate speaker feature space, thereby more accurately distinguishing the speech of different speakers when processing noisy multi-speaker speech signals. For example, in a multi-speaker conversation scenario, the model can quickly identify the target speaker's speech fragments based on the feature differences learned from the clean speech signals of multiple speakers, and effectively separate the interfering speech of other speakers.

[0051] It should be noted that the speech of the target speaker in the personalized speech enhancement sample is different from the personalized registration speech of the corresponding target speaker, so as to improve the speech enhancement effect and robustness of the model for different pronunciation contents.

[0052] In one example, to ensure that the trained dual-task speech enhancement model possesses good adaptability and generalization capabilities, and can effectively handle speech enhancement tasks in various scenarios, the speech samples in the dual-task training dataset are as comprehensive and diverse as possible. From the scenario perspective, these can include everyday communication scenarios, such as multi-person conversations in a family or meetings in an office; they can also include complex environments, such as speech in a bustling market or the mixed speech of airport terminal announcements. From the voice type perspective, the speech samples can include human speech of different genders, ages, and accents, and in a variety of forms, including natural conversations, speeches, and phone recordings. Furthermore, considering the practical requirements of speech enhancement tasks, the speech samples in the dataset are artificially enriched with different types of noise, such as white noise, traffic noise, and industrial noise, to simulate realistic and complex speech environments.

[0053] In a possible implementation, the voice sample and its corresponding registration voice signal may be obtained from one or more of the following sources:

[0054] Acquisition of public datasets: Acquire from public speech data resource libraries, such as Libri2Mix and VoiceBankDEMAND. These public datasets provide a rich variety of noisy speech signals for speech enhancement algorithms, covering different speakers and different language scenarios. Screen speech samples that meet the requirements from the public datasets, and match them with corresponding registration voice signals according to the requirements of this application. For example, for traditional speech enhancement samples, invalid registration voice signals are uniformly assigned; for personalized speech enhancement samples, personalized registration voice signals are assigned according to the characteristics of the samples. Taking VoiceBank DEMAND as an example, the VoiceBank DEMAND dataset is a noisy speech signal library for speech enhancement algorithms disclosed by the Speech Technology Research Center of the University of Edinburgh, which also contains clean speech signals corresponding to each noisy speech signal. When determining the noisy speech signal, it can be obtained by adding noise to the clean speech signal. The clean speech signal comes from the VoiceBank corpus, which consists of 28 speakers from England, including 14 males and 14 females. Each speaker speaks approximately 400 sentences, totaling 11,542 clean speech signals. The data sampling rate is 16kHz and is resampled to 8k. The noise data used to add noise to the clean speech signal comes from eight real noises in the Demand database, covering scenes such as home, office, public places, traffic, and streets. The above noise data is superimposed on the clean speech signal at four different signal-to-noise ratios (15dB, 10dB, 5dB, and 0dB), forming a total of 32 different noise conditions, which means that each speaker has approximately 10 different sentences under each condition. During synthesis, the noise data is added to the waveform of the clean speech signal using the ITU-TP.56 method, and silence segments exceeding 200 milliseconds are trimmed at the beginning and end of each sentence.

[0055] Real-world acquisition: Utilize microphone arrays and portable recording devices to capture speech signals in real-life and work scenarios. For example, a microphone array can be deployed in a conference room to record speech signals from a multi-speaker meeting as speech samples. During the acquisition process, a clean speech signal can be recorded for each speaker as their personalized registration speech signal. For the acquisition of traditional speech enhancement samples, speech can be recorded in a fixed background noise environment, and the registration speech signal can be uniformly disabled.

[0056] Partners: We collaborate with professional voice data collection companies and research institutions to obtain voice data collected or organized by them. These institutions have professional data collection equipment and technology and can provide high-quality voice samples and registered voice data that meet specific requirements.

[0057] In some possible implementations, after obtaining the original noisy speech signal and its registered speech signal, a series of preprocessing tasks can be performed on them to facilitate subsequent model processing and improve training efficiency. The preprocessing tasks may include, but are not limited to, one or more of the following:

[0058] Format conversion: Convert noisy speech signals and registered speech signals of different formats into a format that the model can recognize and process. For example, convert them into a WAV format with 16-bit quantization, mono, and a sampling rate of 16kHz to eliminate compatibility issues caused by format differences.

[0059] Duration adjustment: Uniformly adjust the duration of the noisy speech signal and the registered speech signal to ensure that the duration of each speech sample is equal. For example, this can be achieved through audio editing or padding with silence. For example, the end of a shorter noisy speech signal can be padded with silence to make it consistent with the duration of the longest noisy speech signal, ensuring consistency in data processing during training.

[0060] In one possible implementation, for each noisy speech signal, because the speech length is different, all speech segments that are uniformly segmented and cropped into preset standard speech lengths (such as 4 seconds, 10 seconds, etc.) are used as speech samples, so that the duration of each speech sample is the pre-configured standard speech length. For example, if the length of the noisy speech signal is 12s, it is cut into 3 4s long speech segments, and each speech segment is determined as a speech sample for training. If it is a traditional speech enhancement task, a 4s long all-0 registration speech signal can be used for training. According to the setting of the number of batch speech items, the speech segments are grouped into batches for model training. When forming a small batch data training model, the traditional speech enhancement samples of the traditional speech enhancement task and the personalized speech enhancement samples of the personalized speech enhancement task have a fifty percent probability of being selected, so two types of speech samples will appear in one batch at the same time. The difference is that the different types of speech samples correspond to different registration speech signals.

[0061] Normalization: Normalize the volume of the noisy and registered speech signals to the same level to prevent volume differences from affecting model training. Specifically, an energy-based normalization method can be used to ensure that the average energy of all speech data reaches a set standard value.

[0062] Labeling: Label each noisy speech signal with the corresponding standard clean speech signal, clearly distinguish between traditional speech enhancement samples and personalized speech enhancement samples, assign a unified invalid registered speech identifier to traditional speech enhancement samples, and assign each personalized speech enhancement sample its own personalized registered speech identifier, to facilitate accurate identification and processing of different types of samples during subsequent model training.

[0063] Through the above-mentioned data collection, acquisition and preprocessing methods, a high-quality, compliant dual-task training dataset can be obtained, providing a reliable data basis for the training of the dual-task speech enhancement model, enabling the model to learn a rich variety of speech features and enhancement patterns in the subsequent training process, thereby improving the model's speech enhancement capabilities and generalization performance.

[0064] S102: Obtaining a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal using an original personalized speech enhancement model.

[0065] S103: Based on the predicted clean speech signal and the standard clean speech signal, the original personalized speech enhancement model is iteratively trained to obtain a trained dual-task speech enhancement model, wherein the dual-task speech enhancement model has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0066] After obtaining the dual-task dataset based on the above embodiment, the original personalized speech enhancement model can be iteratively trained based on the speech samples in the dual-task dataset. In each iterative training process, multiple steps are required. The following describes any iterative process:

[0067] Obtain any speech sample and its corresponding registration speech signal from the dual-task dataset, and input the speech sample and its corresponding registration speech signal into the original personalized speech enhancement model. By processing the input speech sample and the registration speech signal through the original personalized speech enhancement model, a predicted clean speech signal corresponding to the speech sample can be obtained. Then, based on the predicted clean speech signal and the standard clean speech signal corresponding to the speech sample, a speech loss value is determined. The speech loss value can be determined by means of mean square error, perceptual loss, spectrometer loss, etc. Based on the speech loss value, the parameters in the original personalized speech enhancement model of the current iteration are adjusted to obtain a trained personalized speech enhancement model. The trained model is a dual-task speech enhancement model, which has both traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0068] In a possible implementation, the iteratively training the original personalized speech enhancement model based on the predicted clean speech signal and the standard clean speech signal includes:

[0069] Determining a correlation coefficient between the predicted clean speech signal and the standard clean speech signal;

[0070] determining an adjustment factor based on the correlation coefficient and the total energy of the standard clean speech signal;

[0071] determining an adjusted speech signal according to the adjustment factor and the standard clean speech signal;

[0072] Determining a noise signal according to the predicted clean speech signal and the standard clean speech signal;

[0073] Obtaining a ratio between the total energy of the adjusted speech signal and the total energy of the noise signal;

[0074] determining a speech loss value based on a base 10 logarithm of the ratio;

[0075] The original personalized speech enhancement model is iteratively trained according to the speech loss value.

[0076] The correlation coefficient between the predicted clean speech signal and the standard clean speech signal can reflect the degree of similarity between the model's prediction and the actual speech in terms of overall trends and changing patterns. For example, determining the correlation coefficient by taking the inner product of the predicted clean speech signal and the standard clean speech signal as an example measures the degree of linear correlation between the two, and its value range is [-1, 1]. The closer its absolute value is to 1, the stronger the correlation between the predicted clean speech signal and the standard clean speech signal; the closer it is to 0, the weaker the correlation. In speech enhancement scenarios, determining the correlation can provide a preliminary assessment of whether the model is capturing the essential characteristics and patterns of the speech signal. A low correlation indicates significant deviations in the model's processing of the speech signal, making subsequent adjustments and optimizations particularly important. Because the correlation alone cannot fully reflect the importance of the standard clean speech signal and the actual predictions of the model, the total energy of the standard clean speech signal reflects its strength and importance. Combining this with the correlation coefficient can comprehensively consider the similarity between the predicted clean speech signal and the standard clean speech signal, as well as the characteristics of the standard clean speech signal itself. Therefore, after obtaining the correlation coefficient based on the above embodiment, an adjustment factor can be determined based on the correlation coefficient and the total energy of the standard clean speech signal. This adjustment factor can more accurately reflect the degree of discrepancy between the current model's prediction and the real speech, providing a quantitative basis for subsequent model adjustments. With the adjustment factor, the standard clean speech signal can be scaled according to the adjustment factor to obtain an adjusted speech signal. The quantified result of the adjustment factor can then be applied to real data, allowing the standard clean speech signal to dynamically adjust according to the model's prediction. Furthermore, the noise signal can be determined based on the predicted clean speech signal and the standard clean speech signal, thereby identifying the discrepancy between the model's prediction and the real speech—that is, the noise and error that the model failed to accurately account for. The ratio of the total energy of the adjusted speech signal to the total energy of the noise signal is obtained. This ratio reflects the relative strength of the adjusted speech signal relative to the noise signal. A larger ratio indicates that the clean speech signal predicted by the model is closer to the standard clean speech signal and contains relatively less noise. The speech loss value is determined based on the base-10 logarithm of the ratio: loss = 10log10(γ). Finally, the model parameters are updated based on the determined speech loss value to obtain the trained dual-task speech enhancement model.

[0077] In one example, the speech loss value is determined by the following formula:

[0078] A= <X est ,X clean >

[0079] E clean =||X clean || 2

[0080]

[0081] X target =αX clean

[0082] E t arg et =||X t arg et || 2

[0083] X noise =X est -X clean

[0084] E noise =||X noise || 2

[0085]

[0086] Among them, X est Represents the predicted clean speech signal, X clean represents a standard clean speech signal, <·> represents the inner product operation, A represents the correlation coefficient, ||·|| 2 Represents the square of the vector norm, E clean represents the energy of the standard clean speech signal, α represents the adjustment factor, E t arg et Represents the total energy of the adjusted speech signal, X target represents the adjusted speech signal, X noise represents the noise signal, E noise It represents the total energy of the noise signal, and loss represents the speech loss value, which is equivalent to calculating the orthogonal projection signal-to-noise ratio of the adjusted speech signal in two dimensions: the standard clean speech signal and the noise signal. Therefore, loss is not affected by the signal energy level and can provide a relatively more objective evaluation of the enhancement performance.

[0087] Since the dual-task dataset for training the original personalized speech enhancement model contains a large number of speech samples, the above operation is performed on each speech sample. When the preset convergence conditions are met, the training of the original personalized speech enhancement model is completed, thereby obtaining a trained dual-task speech enhancement model.

[0088] The preset convergence condition may be that the sum of the loss values corresponding to the speech samples in the current iteration dual-task dataset reaches a minimum or stabilizes, or the number of iterations for training the original personalized speech enhancement model reaches a set maximum number of iterations, etc. This setting can be flexibly adjusted in specific implementations and is not specifically limited here.

[0089] As a possible implementation method, when conducting model training, the speech samples in the dual-task dataset can be divided into a training set, a validation set, and a test set. The original personalized speech enhancement model is first trained based on the training set, and then the reliability of the trained personalized speech enhancement model is verified based on the validation set. The performance of the trained student model is tested based on the test set.

[0090] For example, multiple public data sets can be used to construct a test set to ensure the reliability and representativeness of the test results. For example, the standard clean speech signal of the test set is selected from the speech of two English-accented speakers (one male and one female) in the VoiceBank corpus, which has high speech quality and standard pronunciation; the noise data comes from the Demand database, covering five different scene noises such as home, office, traffic and street, and four slightly higher SNR values of 17.5dB, 12.5dB, 7.5dB and 2.5dB are selected to form 20 different noise conditions. Each speaker has about 20 different sentences under each condition, and the final test set contains 842 voices. All noises are added according to a unified procedure to ensure consistency in data processing. This dataset fully simulates a complex speech environment through a variety of speakers, scene noises and signal-to-noise ratio settings, providing strong support for fully evaluating the generalization ability of the model.

[0091] In addition, the internationally available English audio dataset Libri2Mix is also suitable for training and testing this model. This English audio dataset, built on the LibriSpeech corpus, focuses on speech separation tasks and includes dual-speaker mixed speech. The training set is derived from the train-clean-100 subset, encompassing 13,900 utterances from 251 speakers, totaling 58 hours. The validation and test sets are derived from the dev-clean and test-clean subsets, respectively, each containing 3,000 utterances from 40 speakers, totaling 11 hours. For speech mixing, Libri2Mix randomly selects speech segments from different speakers and normalizes them to LUFS (loudness units relative to full scale). The segments are uniformly sampled between -25dB and -33dB LUFS. During training, speech samples are perturbed by 0.8x and 1.2x speed to effectively increase sample diversity. The resulting speech signal has a mean signal-to-noise ratio of 0dB and a standard deviation of 4.1dB under clean conditions, and is clipped at a maximum amplitude of 0.9 to prevent overload distortion. Using the mix_clean version of Libri2Mix, we can focus on personalized speech enhancement (PSE) tasks for two-speaker mixed speech. Its rich speaker diversity and high-quality speech data provide an excellent experimental foundation for PSE tasks, helping to more accurately evaluate the model's personalized speech enhancement performance in complex multi-speaker scenarios.

[0092] The beneficial effects of this application are as follows:

[0093] 1. A pre-collected dual-task dataset containing a large number of speech samples, corresponding registered speech signals, and standard clean speech signals provides rich and comprehensive data support for model training. The speech samples cover both traditional and personalized speech enhancement samples, enabling the model to learn the characteristics of different speech enhancement tasks. This adapts to various scenarios, such as single-speaker distorted speech processing and extracting the target speaker's voice from complex mixed speech, greatly broadening the model's application scope.

[0094] 2. By assigning the same invalid registration voice signal to traditional speech enhancement samples, the independence and purity of the task is ensured without interfering with the traditional speech enhancement task processing logic, allowing the model to focus on optimizing the processing of distorted speech signals from a single speaker and improving speech clarity. Personalized speech enhancement samples, corresponding to their own personalized registration voice signals, provide the model with a reference to the target speaker's unique voice features in the dual-input, single-output personalized speech enhancement task. This helps the model accurately extract the target speaker's voice from noisy speech signals, meeting the needs of specific speech enhancement processing in complex multi-speaker environments.

[0095] 3. Although traditional speech enhancement tasks do not require registered speech signals to provide effective information, the design of an invalid registered speech signal ensures the consistency of the model input structure. Whether traditional speech enhancement samples or personalized speech enhancement samples, they have the same input format when entering the model. This seamlessly integrates traditional speech enhancement tasks and personalized speech enhancement tasks into a single model. This reduces the complexity of model architecture design and facilitates the model to process both types of samples using a unified network layer structure. This enables parameter sharing and collaborative training between different tasks, improves the model's versatility and generalization capabilities across different tasks, and saves deployment memory and resource waste.

[0096] 4. The trained dual-task speech enhancement model has both traditional speech enhancement and personalized speech enhancement capabilities. It can perform both traditional and personalized speech enhancement tasks simultaneously, reducing the memory and computing overhead required to simultaneously meet both tasks.

[0097] Example 2:

[0098] The following describes the training method of the dual-task speech enhancement model provided by the present application through specific embodiments. Figure 2 A schematic diagram of a training process for a dual-task speech enhancement model provided in an embodiment of the present application, the process includes:

[0099] S201: Collect speech samples contained in the dual-task dataset and perform annotation processing on the collected speech samples to obtain a registered speech signal and a standard clean speech signal corresponding to the speech samples.

[0100] The voice samples include traditional voice enhancement samples and personalized voice enhancement samples. The registration voice signals corresponding to the traditional voice enhancement samples are all the same invalid registration voice signal, and the personalized voice enhancement samples correspond to their own personalized registration voice signals.

[0101] In a possible implementation, each personalized speech enhancement sample includes noisy speech signals of multiple speakers, clean speech signals of multiple speakers, and a noisy speech signal of a single speaker.

[0102] Optionally, the personalized speech enhancement sample also includes a clean speech signal of a single speaker.

[0103] In a possible implementation, an invalid registration voice signal is determined as follows:

[0104] When the original personalized speech enhancement model is a model that concatenates the speech sample and the registration speech signal, the invalid registration speech signal is an all-0 speech signal;

[0105] When the original personalized speech enhancement model is a model that performs multiplication processing on the speech sample and the registration speech signal, the invalid registration speech signal is an all-1 speech signal.

[0106] In a possible implementation, the length of each voice sample is a preconfigured standard voice length.

[0107] S202: Obtain any speech sample in the dual-task dataset and its corresponding registration speech signal.

[0108] S203: Obtain the voice signal corresponding to the voice sample based on the voice sample and the registered voice signal through the original personalized voice enhancement model.

[0109] S204: Determine a correlation coefficient between the predicted clean speech signal and the standard clean speech signal.

[0110] S205: Determine an adjustment factor based on the correlation coefficient and the total energy of the standard clean speech signal.

[0111] S206: Determine an adjusted speech signal according to the adjustment factor and the standard clean speech signal.

[0112] S207: Determine a noise signal according to the predicted clean speech signal and the standard clean speech signal.

[0113] S208: Obtain the ratio between the adjusted total energy of the speech signal and the total energy of the noise signal.

[0114] S209: Determine the speech loss value according to the logarithm with base 10 of the ratio.

[0115] S210: Iteratively train the original personalized speech enhancement model according to the speech loss value to obtain a trained dual-task speech enhancement model. The dual-task speech enhancement model has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0116] The original personalized speech enhancement model was trained on a dual-task dataset. Because this dataset contains a large number of speech samples, the process of inputting the model, calculating the loss, and adjusting the parameters must be repeated for each sample. For example, the training process uses the Adam optimizer with an initial learning rate of 0.0005, combined with the StepLR learning rate scheduler for optimization. During the first 100 epochs of training, the learning rate is multiplied by 0.98 every two epochs to achieve gradual decay. During the final 20 epochs, the learning rate is multiplied by 0.9 to fine-tune the model parameters. Furthermore, gradient clipping is applied to limit the maximum L2 norm of the gradient to 1 to prevent gradient explosion and ensure a stable training process. The entire training process lasts for a maximum of 120 epochs. When the preset convergence criteria are met, the original personalized speech enhancement model is considered trained, resulting in a dual-task speech enhancement model with both traditional and personalized speech enhancement capabilities.

[0117] To comprehensively evaluate the model's performance on personalized speech enhancement and traditional speech enhancement tasks, this application uses three evaluation metrics: SI-SDR (Signal-to-Distortion Ratio, dB), PESQ (Speech Quality Evaluation, fraction), and STOI (Speech Intelligibility, %). SI-SDR measures the improvement in the signal-to-distortion ratio, accurately quantifying the separation quality of the prediction results; PESQ evaluates the perceived quality of the enhanced speech based on a psychoacoustic model; and STOI evaluates the intelligibility of the speech signal by calculating the correlation between the speech spectrograms of the clean and enhanced speech signals.

[0118] Based on the aforementioned training configuration, evaluation metrics, and datasets, we conducted comparative experiments comparing the dual-task speech enhancement model (using USEF-PNet as an example) with independent models corresponding to traditional speech enhancement and personalized speech enhancement (using SEF-PNet as examples). Under identical settings and training data conditions, the two models were fairly compared. The relevant experimental results are shown in the following table:

[0119]

[0120] As shown in the table above, when the training set is Libri2Mix, SEF-PNet achieves SISNR / PESQ / STOI = 13.00 / 3.01 / 89.71 on the test set, while USEF-PNet achieves SISNR / PESQ / STOI = 13.22 / 3.01 / 90.07 on the same test set, showing a slight improvement. When the training set is VoiceBank DEMAND, SEF-PNet achieves SISNR / PESQ / STOI = 19.21 / 3.59 / 0.95 on the corresponding test set, while USEF-PNet achieves SISNR / PESQ / STOI = 19.89 / 3.59 / 0.95 on the same test set. These results fully demonstrate that the dual-task speech enhancement model proposed in this application performs better than the independent model, confirming that traditional speech enhancement and personalized speech enhancement tasks are complementary and can improve each other's performance, further highlighting the superiority of the dual-task speech enhancement model.

[0121] Example 3:

[0122] This application also provides a method for speech enhancement based on a model trained in any of the above embodiments. Figure 3 A schematic diagram of a speech enhancement process provided in an embodiment of the present application, the process includes:

[0123] S301: Acquire a voice signal to be processed.

[0124] S302: Determine a target registered voice signal for the voice signal to be processed; if the voice signal to be processed is a traditional voice enhancement signal, the target registered voice signal is a pre-configured invalid registered voice signal; if the voice signal to be processed is a personalized voice enhancement signal, the target registered voice signal is a pre-configured personalized registered voice signal.

[0125] S303: Determine a clean speech signal after speech enhancement based on the speech signal to be processed and the target registered speech signal using a pre-trained dual-task speech enhancement model.

[0126] The speech enhancement method provided in this application is applied to a computer device, which can be a smart device or a server. The computer device performing speech enhancement in this application can be the same as or different from the computer device performing the dual-task speech enhancement model training described above.

[0127] In a possible implementation, the dual-task speech enhancement model training method is generally performed in an offline manner. After the dual-task speech enhancement model is obtained, the dual-task speech enhancement model can be saved in the computer device for speech enhancement.

[0128] In this embodiment, the computer device performs the speech enhancement task mainly including three core steps: obtaining the speech signal to be processed, determining the target registration speech signal, and generating a clean speech signal using the dual-task speech enhancement model.

[0129] The first step is to obtain the voice signal to be processed. Computer devices can obtain voice signals to be processed in a variety of ways. In the context of smart devices, such as smartwatches and smartphones, the built-in microphones in these devices serve as audio acquisition modules, capturing ambient sound in real time, converting it into electrical signals, and then performing analog-to-digital conversion to generate digital voice signals to be processed. For example, when a user uses a voice assistant on a noisy street, the microphone will capture voice signals that contain various interferences, such as vehicle noise and human voices. For servers, obtaining voice signals to be processed typically involves receiving data uploaded from multiple terminal devices. For example, in a customer service call center system, each customer service agent terminal uploads the voice calls between customers and customer service representatives to the server, and the voice calls received by the server serve as the voice signals to be processed.

[0130] The next step is to determine the target registration voice signal. This determination is based on the type of voice signal being processed. If the voice signal being processed is a traditional voice enhancement signal, the system uses a pre-configured invalid registration voice signal as the target registration voice signal. This invalid registration voice signal is a segment of speech that contains no valid semantic information, such as silence or interference at a specific frequency. This serves to unify the inputs of the traditional voice enhancement task within the dual-task voice enhancement model during the speech enhancement process, enabling the model to receive and process input data from different tasks in a unified manner. If the voice signal being processed is a personalized voice enhancement signal, the system selects one of multiple pre-configured personalized registration voice signals as the target registration voice signal. Personalized registration voice signals are customized to the user's specific needs or characteristics, such as voice signals tailored to the user's voiceprint or frequently used voice commands. In a smart home system, if a user wishes the device to respond only to their voice commands, a pre-recorded voice sample containing frequently used commands will be used as the personalized registration voice signal.

[0131] In one possible implementation, for a dual-task speech enhancement model that uses splicing processing, the pre-configured invalid registered speech signal is an all-0 speech signal. In the process of splicing the speech signal to be processed and the target registered speech signal, the all-0 speech signal will not change the original characteristics and data distribution of the speech signal to be processed. The all-0 speech signal only exists as filler data after splicing, and will not introduce additional interference information for the traditional speech enhancement task, thereby ensuring that the model can process the traditional speech enhancement sample according to the predetermined single-input single-output logic. For a dual-task speech enhancement model that uses multiplication processing, the pre-configured invalid registered speech signal is an all-1 speech signal. In the multiplication operation, after the all-1 speech signal is multiplied by the traditional speech enhancement signal, the original value of the speech signal will not be changed, which is equivalent to directly transmitting the original speech signal, and will not have any substantial impact on the processing process of the traditional speech enhancement task, thereby ensuring the independence of the traditional speech enhancement task.

[0132] In a possible implementation, when determining the type of the speech signal to be processed, the determination may be made according to the following methods:

[0133] A. Identification information carried by the signal: The voice signal to be processed can be assigned a specific identification field during the collection or transmission process. For example, in the audio acquisition module program of a smart device, when the user chooses to turn on the "personalized voice enhancement mode", the collected voice signal will be added with a custom label (such as "personalized_voice signal"); and the signal collected in the default "traditional voice enhancement mode" is marked as "traditional_voice signal". After receiving the voice signal, the server or smart device can quickly determine the signal type by parsing the identification field. Similarly, in some communication protocols, a field is specially set to identify the processing requirements of the voice signal, thereby distinguishing between traditional and personalized voice enhancement signals.

[0134] B. User input instructions or configuration information: This is determined based on the user's operating instructions on the device. Taking a smart home central control device as an example, the user can send instructions through a mobile phone app to explicitly specify that the current voice input should be processed using "personalized voice enhancement." In this case, the voice signal received by the device is a personalized voice enhancement signal. If the user does not make special settings, the device processes the received voice signal according to the default traditional voice enhancement mode. In addition, the default type of voice enhancement can also be pre-set in the device's system configuration file. When there are no special instructions, the type of voice signal to be processed is determined based on the contents of the configuration file.

[0135] C. Feature Analysis of Voice Signals: This is done by analyzing and identifying the acoustic features of voice signals. Personalized voice enhancement signals often contain unique acoustic features associated with a specific user, such as a specific tone, speaking rate, and spectral distribution. Voiceprint recognition technology can be used to extract the voiceprint features of the voice signal to be processed and compare them with pre-stored user-personalized voiceprint templates. If the similarity reaches a certain threshold, it is determined to be a personalized voice enhancement signal; if the similarity is low and the signal features conform to common general speech patterns, it is determined to be a traditional voice enhancement signal. The type of voice signal can also be predicted by analyzing features such as the frequency distribution and energy changes of the voice signal, combined with a classification model trained using a machine learning algorithm.

[0136] D. Signal source and usage scenario: Determine the type based on the source of the voice signal and the intended usage scenario. For example, voice signals collected in a professional recording studio are usually processed using traditional voice enhancement methods for audio post-production to remove environmental noise and optimize sound quality. In smart wearable devices, voice signals collected for user interactions with personal voice assistants are mostly processed to meet the user's personalized interaction needs and tend to be processed using personalized voice enhancement. For example, in a customer service center, traditional voice enhancement is used to ensure call clarity for recordings of ordinary customer consultations. For recordings of VIP customers, personalized voice enhancement may be used to better analyze the customer's voice characteristics and emotions. Therefore, the type of signal can be preliminarily determined based on the signal source and scenario.

[0137] In one possible implementation, the pre-configured personalized registration voice signal that matches the identity information can be determined as the target registration voice signal based on the identity information collected when the voice signal to be processed is collected. Alternatively, the target registration voice signal can be determined based on the voiceprint matching results between the voice signal to be processed and the pre-configured personalized registration voice signals.

[0138] Exemplarily, determining the target registration voice signal of the voice signal to be processed includes:

[0139] Performing voiceprint matching on the to-be-processed voice signal and the pre-configured personalized registration voice signals;

[0140] If there is any personalized registration voice signal that matches the voiceprint of the voice signal to be processed, determining that the voice signal to be processed is a personalized voice enhancement signal, and determining the matching personalized registration voice signal as the target registration voice signal;

[0141] Otherwise, it is determined that the speech signal to be processed belongs to the traditional speech enhancement signal, and the invalid registration speech signal is determined as the target registration speech signal.

[0142] After acquiring the voice signal to be processed, the computer device can perform voiceprint matching on it with each pre-configured personalized registration voice signal. Voiceprint matching identifies and compares the speaker's identity characteristics based on the unique acoustic features in the voice signal. In actual operation, the system first extracts features from the voice signal to be processed and each personalized registration voice signal. Common feature extraction methods include Mel-Frequency Cepstral Coefficients (MFCC) and Linear Prediction Cepstral Coefficients (LPCC). These methods can convert voice signals into representative feature vectors for subsequent similarity comparison. For example, in a smart home system, personalized registration voice signals for multiple family members are pre-configured. When a smart speaker receives a voice signal to be processed, it extracts features from the signal to obtain a feature vector. This feature vector is then compared one by one with the feature vector corresponding to the personalized registration voice signal of each family member.

[0143] If, during the voiceprint matching process, any personalized registered voice signal exists that matches the voiceprint of the voice signal to be processed, the system will determine that the voice signal to be processed is a personalized voice enhancement signal. Voiceprint matching typically sets a similarity threshold. When the feature vector similarity between the voice signal to be processed and a personalized registered voice signal exceeds this threshold, they are considered to have a voiceprint match. Once determined to be a personalized voice enhancement signal, the system will identify the matching personalized registered voice signal as the target registered voice signal. Taking a smart speaker as an example, if the voice signal to be processed is detected to match the voiceprint of the father's personalized registered voice signal at home, then the father's personalized registered voice signal will be used as the target registered voice signal. The advantage of this is that in the subsequent voice enhancement process, the model can better preserve the unique characteristics of the father's voice based on this personalized reference signal, while removing background noise and improving the clarity and intelligibility of the voice.

[0144] If no matching signal is found after performing voiceprint matching on all pre-configured personalized registration voice signals, the system will determine that the voice signal to be processed is a traditional voice enhancement signal. In this case, the system will determine the pre-configured invalid registration voice signal as the target registration voice signal.

[0145] Finally, after receiving the processed speech signal and the target registered speech signal, the computer device quickly calls the saved dual-task speech enhancement model. Using the pre-trained dual-task speech enhancement model, the model determines a clean speech signal after speech enhancement based on the processed and target registered speech signals. During training, the dual-task speech enhancement model acquires the ability to simultaneously perform speech enhancement and feature recognition by learning from a large number of speech data pairs. During processing, the dual-task speech enhancement model analyzes and processes the input processed speech signal and the target registered speech signal, combining the knowledge learned from the dual-task task to remove noise while enhancing the key features of the speech signal. For example, in an in-vehicle voice interaction system, the processed speech signal may contain engine noise and conversations within the vehicle, while the target registered speech signal is a pre-set personalized speech sample. The dual-task speech enhancement model can remove noise while accurately identifying the driver's voice commands and outputting a clear, clean speech signal for the vehicle's intelligent system to subsequently interpret and execute the commands.

[0146] It should be noted that the training method of the dual-task speech enhancement model has been described in the above embodiments 1-2 and will not be described in detail here.

[0147] Example 4:

[0148] The following describes the speech enhancement method provided by this application through specific embodiments. Figure 4 A specific speech enhancement process diagram provided in an embodiment of the present application includes:

[0149] S401: Acquire a voice signal to be processed.

[0150] S402: Determine whether there is a personalized registration voice signal corresponding to the voice signal to be processed among the pre-configured personalized registration voice signals. If so, execute S403; otherwise, execute S404.

[0151] In one possible implementation, a voiceprint matching method can be used to determine whether a personalized registration voice signal corresponding to the voice signal to be processed exists among the pre-configured personalized registration voice signals. If a personalized registration voice signal matching the voiceprint of the voice signal to be processed exists, then it is determined that a personalized registration voice signal corresponding to the voice signal to be processed exists among the pre-configured personalized registration voice signals. If no personalized registration voice signal matching the voiceprint of the voice signal to be processed exists, then it is determined that a personalized registration voice signal corresponding to the voice signal to be processed does not exist among the pre-configured personalized registration voice signals.

[0152] S403: Determine the personalized registration voice signal corresponding to the voice signal to be processed as the target registration voice signal.

[0153] S404: Determine the pre-configured invalid registration voice signal as the target registration voice signal.

[0154] S405: Determine a clean speech signal after speech enhancement based on the speech signal to be processed and the target registered speech signal using a pre-trained dual-task speech enhancement model.

[0155] Example 5:

[0156] Based on the same inventive concept, the present application also provides a training device for a dual-task speech enhancement model. Figure 5 A schematic diagram of a dual-task speech enhancement model training device provided in an embodiment of the present application, comprising:

[0157] An acquisition unit 51 is configured to acquire any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein each speech sample also corresponds to a standard clean speech signal, each speech sample includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech signals corresponding to each traditional speech enhancement sample are the same invalid registration speech signal, and each personalized speech enhancement sample corresponds to its own personalized registration speech signal;

[0158] A processing unit 52 is configured to obtain a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal using the original personalized speech enhancement model;

[0159] The training unit 53 is used to iteratively train the original personalized speech enhancement model based on the predicted clean speech signal and the standard clean speech signal to obtain a trained dual-task speech enhancement model, which has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0160] The training method device of the dual-task speech enhancement model in this embodiment is presented in the form of a functional module, where the module refers to an invalid integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0161] The further functional description of each of the above modules and units is the same as that of the corresponding embodiments 1-2 above and will not be repeated here.

[0162] Example 6:

[0163] Based on the same inventive concept, the present application also provides a speech enhancement device, Figure 6This is a schematic diagram of the structure of a speech enhancement device provided in an embodiment of the present application, the device comprising:

[0164] An acquisition module 61 is used to acquire a speech signal to be processed;

[0165] a determination module 62 for determining a target registration voice signal for the voice signal to be processed; wherein, if the voice signal to be processed is a conventional voice enhancement signal, the target registration voice signal is a pre-configured invalid registration voice signal; and if the voice signal to be processed is a personalized voice enhancement signal, the target registration voice signal is a pre-configured personalized registration voice signal;

[0166] The processing module 63 is configured to determine a clean speech signal after speech enhancement based on the speech signal to be processed and the target registered speech signal using a pre-trained dual-task speech enhancement model.

[0167] The training method device of the dual-task speech enhancement model in this embodiment is presented in the form of a functional module, where the module refers to an invalid integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0168] The further functional description of each of the above modules and units is the same as that of the corresponding embodiments 3-4 above and will not be repeated here.

[0169] Example 7:

[0170] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present application. Figure 7 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The six components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of a GUI on an external input / output device (such as, a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0171] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0172] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0173] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of a computer device for displaying a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0174] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0175] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0176] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0177] Example 8:

[0178] Based on the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the following steps:

[0179] Obtain any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein each speech sample also corresponds to a standard clean speech signal, each speech sample includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech corresponding to each traditional speech enhancement sample is the same invalid registration speech signal, and each personalized speech enhancement sample corresponds to its own personalized registration speech signal;

[0180] Obtaining, by an original personalized speech enhancement model, a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal;

[0181] Based on the predicted clean speech signal and the standard clean speech signal, the original personalized speech enhancement model is iteratively trained to obtain a trained dual-task speech enhancement model, which has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

[0182] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the training method of the dual-task-based speech enhancement model, the implementation of the above-mentioned computer-readable storage medium can refer to Examples 1-2 of the method, and the repeated parts will not be repeated.

[0183] Example 9:

[0184] Based on the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the following steps:

[0185] Obtaining a voice signal to be processed;

[0186] Determining a target registration voice signal for the voice signal to be processed; wherein, if the voice signal to be processed is a traditional voice enhancement signal, the target registration voice signal is a pre-configured invalid registration voice signal; if the voice signal to be processed is a personalized voice enhancement signal, the target registration voice signal is a pre-configured personalized registration voice signal;

[0187] A clean speech signal after speech enhancement is determined based on the speech signal to be processed and the target registered speech signal through a pre-trained dual-task speech enhancement model.

[0188] Since the principle of solving the problem by the above computer-readable storage medium is similar to that of the speech enhancement method, the implementation of the above computer-readable storage medium can refer to Examples 3-4 of the method, and the repeated parts will be omitted.

[0189] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A training method for a dual-task speech enhancement model, characterized in that: The method comprises: Obtain any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein each speech sample also corresponds to a standard clean speech signal, each speech sample includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech corresponding to each traditional speech enhancement sample is the same invalid registration speech signal, and each personalized speech enhancement sample corresponds to its own personalized registration speech signal; Obtaining, by an original personalized speech enhancement model, a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal; Based on the predicted clean speech signal and the standard clean speech signal, the original personalized speech enhancement model is iteratively trained to obtain a trained dual-task speech enhancement model, which has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

2. The method according to claim 1, wherein Each personalized speech enhancement sample includes noisy speech signals of multiple speakers.

3. The method according to claim 1, wherein The length of each voice sample is a pre-configured standard voice length.

4. The method according to claim 1, wherein The iteratively training the original personalized speech enhancement model based on the predicted clean speech signal and the standard clean speech signal includes: Determining a correlation coefficient between the predicted clean speech signal and the standard clean speech signal; determining an adjustment factor based on the correlation coefficient and the total energy of the standard clean speech signal; determining an adjusted speech signal according to the adjustment factor and the standard clean speech signal; Determining a noise signal according to the predicted clean speech signal and the standard clean speech signal; Obtaining a ratio between the total energy of the adjusted speech signal and the total energy of the noise signal; determining a speech loss value based on a base 10 logarithm of the ratio; The original personalized speech enhancement model is iteratively trained according to the speech loss value.

5. The method according to claim 1, wherein The invalid registration voice signal is determined as follows: When the original personalized speech enhancement model is a model that performs splicing processing on the speech sample and the registration speech signal, the invalid registration speech signal is an all-0 speech signal; When the original personalized speech enhancement model is a model that performs multiplication processing on the speech sample and the registration speech signal, the invalid registration speech signal is an all-1 speech signal.

6. A method for speech enhancement based on a model trained by the method according to any one of claims 1 to 5, characterized in that: The method comprises: Obtaining a voice signal to be processed; Determining a target registration voice signal for the voice signal to be processed; wherein, if the voice signal to be processed is a traditional voice enhancement signal, the target registration voice signal is a pre-configured invalid registration voice signal; if the voice signal to be processed is a personalized voice enhancement signal, the target registration voice signal is a pre-configured personalized registration voice signal; A clean speech signal after speech enhancement is determined based on the speech signal to be processed and the target registered speech signal through a pre-trained dual-task speech enhancement model.

7. The method according to claim 6, wherein The determining of the target registration voice signal of the voice signal to be processed includes: Performing voiceprint matching on the to-be-processed voice signal and the pre-configured personalized registration voice signals; If there is any personalized registration voice signal that matches the voiceprint of the voice signal to be processed, determining that the voice signal to be processed is a personalized voice enhancement signal, and determining the matching personalized registration voice signal as the target registration voice signal; Otherwise, it is determined that the speech signal to be processed belongs to the traditional speech enhancement signal, and the invalid registration speech signal is determined as the target registration speech signal.

8. A training device for a dual-task speech enhancement model, characterized in that: The device comprises: An acquisition unit is configured to acquire any speech sample in the dual-task dataset and its corresponding registration speech signal; wherein each of the speech samples also corresponds to a standard clean speech signal, each of the speech samples includes a traditional speech enhancement sample and a personalized speech enhancement sample, the registration speech signals corresponding to each of the traditional speech enhancement samples are the same invalid registration speech signal, and each of the personalized speech enhancement samples corresponds to its own personalized registration speech signal; a processing unit, configured to obtain, by using an original personalized speech enhancement model, a predicted clean speech signal corresponding to the speech sample based on the speech sample and the registered speech signal; A training unit is used to iteratively train the original personalized speech enhancement model based on the predicted clean speech signal and the standard clean speech signal to obtain a trained dual-task speech enhancement model, wherein the dual-task speech enhancement model has traditional speech enhancement capabilities and personalized speech enhancement capabilities.

9. A speech enhancement device based on a model trained by the method according to any one of claims 1 to 5, characterized in that: The device comprises: An acquisition module, used for acquiring a speech signal to be processed; a determination module, configured to determine a target registration voice signal for the voice signal to be processed; wherein, if the voice signal to be processed is a traditional voice enhancement signal, the target registration voice signal is a pre-configured invalid registration voice signal; and if the voice signal to be processed is a personalized voice enhancement signal, the target registration voice signal is a pre-configured personalized registration voice signal; The processing module is used to determine a clean speech signal after speech enhancement based on the speech signal to be processed and the target registered speech signal through a pre-trained dual-task speech enhancement model.

10. A computer device, characterized in that: The computer device includes a processor, which is used to implement the steps of the training method of the dual-task speech enhancement model as described in any one of claims 1 to 5 above when executing the computer program stored in the memory, or to implement the steps of the speech enhancement method as described in claim 6 above.

Images