Audio enhancement method and related equipment
By constructing a speech enhancement model based on WavLM-large model and WWF model, the noise reduction problem in the Alzheimer's disease detection field is solved, and the accuracy of the detection system is improved. Avoid feature loss.
Patent Information
- Application Number
- CN202411664672.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art has not fully met the noise reduction needs in the field of Alzheimer's disease detection, especially in the case of small data volume and lack of paired clean speech, resulting in noise impact and 'Clever Hans' effect, which in turn affects the detection results. accuracy.
By building a preset speech enhancement model based on WavLM-large model and WWF model, and using the preset audio data set to build a training set and a test set, the target speech enhancement model is trained, and the enhanced audio data set is obtained, which solves the noise reduction problem And avoid the loss of effective features.
It effectively eliminates the noise and ‘Clever Hans’ effect of the audio data set to be tested, improves the performance and accuracy of the Alzheimer’s disease detection system, and avoids feature loss problems caused by denoising.
Smart Images
Figure CN119446165A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of audio processing technology, and in particular, relates to an audio enhancement method and related equipment. Background Art
[0002] Alzheimer's disease (AD) has become a common health problem, mainly manifested by a progressive decline in cognitive function, which seriously affects the daily life of patients. At present, existing drug treatments cannot cure Alzheimer's disease, which highlights the importance of early diagnosis and intervention. However, current clinical examination methods are often expensive and time-consuming, bringing huge social and economic burdens to patients and their families. Studies have shown that language dysfunction (such as speech dysfluency and slow speech speed) can be used as potential symptoms in the early stages of Alzheimer's disease. Bertini et al. proposed a speech-based auxiliary Alzheimer's detection system and obtained excellent accuracy results. However, in actual application scenarios, the widespread environmental noise in the room will undoubtedly affect the smoothness and other qualities of the speech, thereby affecting the accuracy of speech feature extraction, and ultimately affecting the accuracy of the results of Alzheimer's disease-related research based on speech. For example, the Pitt Corpus, the most widely used public dataset in the field of Alzheimer's disease detection, has been recently shown by Liu to have a significant impact on the results of Alzheimer's disease detection due to the presence of background noise, resulting in the so-called "Clever Hans" effect, which affects the accuracy of the results of the existing Alzheimer's disease detection system trained based on the Pitt Corpus. Therefore, speech enhancement of speech collected from different audio collection environments in the field of Alzheimer's disease detection is one of the keys to improving the performance of the Alzheimer's disease detection system.
[0003] The purpose of speech enhancement is to extract pure speech from noisy speech while ensuring the quality and intelligibility of speech. In recent years, supervised learning-based methods have been widely used in the field of speech enhancement, which requires a large amount of paired data of clean and noisy speech. The UNet model proposed by Ronneberger is a traditional supervised speech enhancement method. Its unique U-shaped architecture and jump connection can effectively retain the high-frequency part and nonlinear characteristics of speech signals, thereby achieving excellent speech enhancement effects. In subsequent studies, Defossez et al. proposed to use the long short-term memory (LSTM) network in the bottleneck layer to simulate temporal information. At the same time, Kong et al. introduced the self-attention mechanism into the bottleneck layer of UNet and named the model CleanUNet, which achieved advanced results on commonly used speech enhancement datasets. However, existing speech enhancement technologies have failed to fully meet the noise reduction needs in the field of Alzheimer's disease detection, and mainly face the following problems: First, the small amount of data in the field of Alzheimer's disease detection and the lack of paired clean speech limit the training capacity of existing speech enhancement models; second, the existing speech enhancement technology library in Python may delete information related to the detection of Alzheimer's disease, thereby affecting the accuracy of subsequent diagnosis. Summary of the invention
[0004] The embodiment of the present application provides an audio enhancement method, which can solve the problem in the prior art that effective features are deleted during audio data enhancement, resulting in poor audio data enhancement results.
[0005] In a first aspect, an embodiment of the present application provides an audio enhancement method, the method comprising:
[0006] Construct training and test sets based on the preset audio dataset;
[0007] Using the training set and the test set, a preset speech enhancement model constructed based on the WavLM-large model and the WWF model is trained to obtain a target speech enhancement model;
[0008] The audio data set to be tested is input into the target speech enhancement model to obtain an enhanced audio data set.
[0009] Optionally, the preset speech enhancement model includes: a speech enhancement encoder unit, a speech enhancement bottleneck layer unit, a feature processing layer unit, a speech enhancement decoder unit and a skip connection unit;
[0010] The feature processing layer unit includes a WWF model, and the WWF model is used to integrate the output features of each sublayer in the speech enhancement bottleneck layer by a weighted method, and output the integrated features;
[0011] The speech enhancement encoder unit and the speech enhancement bottleneck layer unit are set according to the model structure of the WavLM-large model, and initialization parameters of the speech enhancement encoder unit and the speech enhancement bottleneck layer unit are set by the WavLM-large model;
[0012] The skip connection unit is used to maintain the skip connection between the speech enhancement encoder unit and the speech enhancement encoder unit;
[0013] The speech enhancement decoder unit is used to restore the integrated features output by the feature processing layer unit to the original speech signal.
[0014] Optionally, the step of training a preset speech enhancement model constructed based on the WavLM-large model and the WWF model by using the training set and the test set to obtain a target speech enhancement model includes:
[0015] Inputting the training set into the preset speech enhancement model to obtain a training result;
[0016] Based on the training result, updating the initialization parameters of the speech enhancement encoder unit and the speech enhancement bottleneck layer unit through the WavLM-large model, and obtaining the speech enhancement model to be tested based on the updated speech enhancement encoder unit and the speech enhancement bottleneck layer unit;
[0017] The test set is input into the speech enhancement model to be tested to obtain a target speech enhancement model.
[0018] Optionally, the step of inputting the audio data set to be tested into the target speech enhancement model to obtain the enhanced audio data set includes:
[0019] Inputting the audio data set to be tested into the speech enhancement encoder unit in the target speech enhancement model to obtain encoded data for the audio data set to be tested;
[0020] Inputting the encoded data into the speech enhancement bottleneck layer unit in the target speech enhancement model to obtain integrated features of each audio data in the audio data set to be tested;
[0021] inputting the integrated features into a speech enhancement decoder unit in the target speech enhancement model;
[0022] Sending the encoded data to a speech enhancement decoder unit in the target speech enhancement model through a skip connection unit in the target speech enhancement model;
[0023] Through the speech enhancement decoder unit in the target speech enhancement model, an enhanced audio data set corresponding to the audio data set to be tested is obtained according to the integrated features and the encoded data.
[0024] Optionally, the mathematical representation of the WWF model is:
[0025]
[0026] Where N is the total number of transformer encoder layers, h i is the feature set output by the i-th encoder layer, ω i is the learnable weight of the ith feature set.
[0027] In another aspect, the present application provides an audio enhancement device, the device comprising:
[0028] A first acquisition module, used to construct a training set and a test set according to a preset audio data set;
[0029] A training module, used to train a preset speech enhancement model constructed based on the WavLM-large model and the WWF model through the training set and the test set to obtain a target speech enhancement model;
[0030] The enhancement module is used to input the audio data set to be tested into the target speech enhancement model to obtain an enhanced audio data set.
[0031] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.
[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0033] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0034] By constructing a training set and a test set based on a preset audio data set; using the training set and the test set, the preset speech enhancement model constructed based on the WavLM-large model and the WWF model is trained to obtain a target speech enhancement model; the audio data set to be tested is input into the target speech enhancement model to obtain an enhanced audio data set. This solves the noise reduction problem in the field of audio detection in scenarios where the data volume is small and there is a lack of paired clean speech, eliminates the noise and "Clever Hans" effect of the audio data set to be tested, and avoids the problem of effective feature loss in the audio data set to be tested due to denoising. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0036] Figure 1 is a flowchart of an audio enhancement method provided by an embodiment of the present application;
[0037] Figure 2 is a framework diagram of an audio enhancement method provided by an embodiment of the present application;
[0038] Figure 3 is a schematic diagram of the structure of an audio enhancement device provided by an embodiment of the present application;
[0039] Figure 4 is a structural block diagram of an electronic device in one embodiment;
[0040] Figure 5 FIG. 4 is a structural block diagram of a computer storage medium in another embodiment. DETAILED DESCRIPTION
[0041] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0042] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0043] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0044] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0045] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0046] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0047] Figure 1 A schematic flow chart of an audio enhancement method provided by the present application is shown. An embodiment of the present application provides an audio enhancement method, comprising:
[0048] S101, constructing a training set and a test set according to a preset audio data set;
[0049] Exemplarily, a paired speech (clean-noise) dataset in the field of speech enhancement, the Deep NoiseSuppression (DNS) dataset, is obtained and a training set and a test set are set.
[0050] S102, training a preset speech enhancement model constructed based on the WavLM-large model and the WWF model through the training set and the test set to obtain a target speech enhancement model;
[0051] S103: Input the audio data set to be tested into the target speech enhancement model to obtain an enhanced audio data set.
[0052] Exemplarily, an Alzheimer's disease detection dataset is obtained, and specifically, Pitt Corpus is used as a data source.
[0053] By constructing a training set and a test set based on a preset audio data set; using the training set and the test set, the preset speech enhancement model constructed based on the WavLM-large model and the WWF model is trained to obtain a target speech enhancement model; the audio data set to be tested is input into the target speech enhancement model to obtain an enhanced audio data set. This solves the noise reduction problem in the field of audio detection in scenarios where the data volume is small and there is a lack of paired clean speech, eliminates the noise and "Clever Hans" effect of the audio data set to be tested, and avoids the problem of effective feature loss in the audio data set to be tested due to denoising.
[0054] In a possible implementation, Figure 2 As shown, the preset speech enhancement model includes: a speech enhancement encoder unit, a speech enhancement bottleneck layer unit, a feature processing layer unit, a speech enhancement decoder unit and a jump connection unit;
[0055] The feature processing layer unit includes a WWF model, and the WWF model is used to integrate the output features of each sublayer in the speech enhancement bottleneck layer by a weighted method, and output the integrated features;
[0056] The speech enhancement encoder unit and the speech enhancement bottleneck layer unit are set according to the model structure of the WavLM-large model, and initialization parameters of the speech enhancement encoder unit and the speech enhancement bottleneck layer unit are set by the WavLM-large model;
[0057] The skip connection unit is used to maintain the skip connection between the speech enhancement encoder unit and the speech enhancement encoder unit;
[0058] The speech enhancement decoder unit is used to restore the integrated features output by the feature processing layer unit to the original speech signal.
[0059] Exemplarily, the speech enhancement encoder and the speech enhancement bottleneck layer follow the WavLM-large model structure, that is, the structure of wavlm-large includes two parts, an encoder and a transformer encoder, and the speech enhancement encoder and the speech enhancement bottleneck layer structure also include two parts, an encoder and a transformer encoder, and are initialized with the parameters of the pre-trained WavLM-large model and fine-tuned when training the speech enhancement model. It is used to extract important information from speech, which is equivalent to denoising. The feature processing layer part includes a weighted feature fusion module (WWF), which integrates the output features of each sublayer in the speech enhancement bottleneck layer through learnable weights to obtain a comprehensive feature representation. The jump connection part transfers the feature information of different levels in the speech enhancement encoder to the speech enhancement decoder. The speech enhancement decoder restores the feature representation based on the output of the feature processing layer to the feature representation of the original speech signal.
[0060] In a possible implementation, the step of training a preset speech enhancement model constructed based on the WavLM-large model and the WWF model by using the training set and the test set to obtain a target speech enhancement model includes:
[0061] Inputting the training set into the preset speech enhancement model to obtain a training result;
[0062] Based on the training result, updating the initialization parameters of the speech enhancement encoder unit and the speech enhancement bottleneck layer unit through the WavLM-large model, and obtaining the speech enhancement model to be tested based on the updated speech enhancement encoder unit and the speech enhancement bottleneck layer unit;
[0063] The test set is input into the speech enhancement model to be tested to obtain a target speech enhancement model.
[0064] Exemplarily, fine-tuning is to use the training set DNS dataset to optimize the parameters of the unfrozen layers in the WavLM-large pre-trained model, learn and obtain effective features of speech enhancement information (outputs of the encoder and bottleneck layer units).
[0065] In a possible implementation, the step of inputting the audio data set to be tested into the target speech enhancement model to obtain the enhanced audio data set includes:
[0066] Inputting the audio data set to be tested into the speech enhancement encoder unit in the target speech enhancement model to obtain encoded data for the audio data set to be tested;
[0067] Inputting the encoded data into the speech enhancement bottleneck layer unit in the target speech enhancement model to obtain integrated features of each audio data in the audio data set to be tested;
[0068] inputting the integrated features into a speech enhancement decoder unit in the target speech enhancement model;
[0069] Sending the encoded data to a speech enhancement decoder unit in the target speech enhancement model through a skip connection unit in the target speech enhancement model;
[0070] Through the speech enhancement decoder unit in the target speech enhancement model, an enhanced audio data set corresponding to the audio data set to be tested is obtained according to the integrated features and the encoded data.
[0071] In a possible implementation, the mathematical representation of the WWF model is:
[0072]
[0073] Where N is the total number of transformer encoder layers, h i is the feature set output by the i-th encoder layer, ω i is the learnable weight of the ith feature set.
[0074] In a possible implementation, the silent segment of each Pitt Corpus speech is extracted according to the timestamp information of the Pitt Corpus to form a Pitt Corpus Silence (PCSI) data set, which is input into the Alzheimer's disease classification system after noise reduction through the speech enhancement system. Compared with the result of directly classifying the PCSI data set without noise reduction, the accuracy rate is greatly improved from 84.34% to 58.47%. At the same time, the noise-reduced Pitt Corpus is used by the Alzheimer's disease classification system for Alzheimer's disease classification prediction, and the accuracy rate reaches 88.96%. The results show that the speech enhancement system can not only effectively eliminate the "Clever Hans" problem of Pitt Corpus caused by background noise, but also can largely avoid the problem of deleting Alzheimer's disease detection related information due to excessive noise reduction.
[0075] In another aspect, the present application provides an audio enhancement device, the device comprising:
[0076] The first acquisition module 201 is used to construct a training set and a test set according to a preset audio data set;
[0077] A training module 202 is used to train a preset speech enhancement model constructed based on the WavLM-large model and the WWF model through the training set and the test set to obtain a target speech enhancement model;
[0078] The enhancement module 203 is used to input the audio data set to be tested into the target speech enhancement model to obtain an enhanced audio data set.
[0079] Thirdly, Figure 4 As shown, an embodiment of the present application provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: constructing a training set and a test set according to a preset audio data set; training a preset speech enhancement model constructed based on a WavLM-large model and a WWF model through the training set and the test set to obtain a target speech enhancement model; and inputting the audio data set to be tested into the target speech enhancement model to obtain a step of enhancing the audio data set.
[0080] Fourthly, Figure 5 As shown, an embodiment of the present application provides a computer-readable storage medium 400, which stores a computer program 411. When the computer program is executed by a processor, the following steps are implemented: a training set and a test set are constructed according to a preset audio data set; a preset speech enhancement model constructed based on a WavLM-large model and a WWF model is trained through the training set and the test set to obtain a target speech enhancement model; the audio data set to be tested is input into the target speech enhancement model to obtain an enhanced audio data set.
[0081] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0082] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment / terminal equipment and methods can be implemented in other ways. For example, the device / network equipment / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0084] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. An audio enhancement method, characterized in that: The method comprises: Construct training and test sets based on the preset audio dataset; Using the training set and the test set, a preset speech enhancement model constructed based on the WavLM-large model and the WWF model is trained to obtain a target speech enhancement model; The audio data set to be tested is input into the target speech enhancement model to obtain an enhanced audio data set.
2. The audio enhancement method according to claim 1, wherein: The preset speech enhancement model includes: a speech enhancement encoder unit, a speech enhancement bottleneck layer unit, a feature processing layer unit, a speech enhancement decoder unit and a skip connection unit; The feature processing layer unit includes a WWF model, and the WWF model is used to integrate the output features of each sublayer in the speech enhancement bottleneck layer by a weighted method, and output the integrated features; The speech enhancement encoder unit and the speech enhancement bottleneck layer unit are set according to the model structure of the WavLM-large model, and initialization parameters of the speech enhancement encoder unit and the speech enhancement bottleneck layer unit are set by the WavLM-large model; The skip connection unit is used to maintain the skip connection between the speech enhancement encoder unit and the speech enhancement encoder unit; The speech enhancement decoder unit is used to restore the integrated features output by the feature processing layer unit to the original speech signal.
3. The audio enhancement method according to claim 2, wherein: The step of training the preset speech enhancement model constructed based on the WavLM-large model and the WWF model through the training set and the test set to obtain the target speech enhancement model includes: Inputting the training set into the preset speech enhancement model to obtain a training result; Based on the training result, updating the initialization parameters of the speech enhancement encoder unit and the speech enhancement bottleneck layer unit through the WavLM-large model, and obtaining the speech enhancement model to be tested based on the updated speech enhancement encoder unit and the speech enhancement bottleneck layer unit; The test set is input into the speech enhancement model to be tested to obtain a target speech enhancement model.
4. The audio enhancement method according to claim 2, wherein: The step of inputting the audio data set to be tested into the target speech enhancement model to obtain the enhanced audio data set includes: Inputting the audio data set to be tested into the speech enhancement encoder unit in the target speech enhancement model to obtain encoded data for the audio data set to be tested; Inputting the encoded data into the speech enhancement bottleneck layer unit in the target speech enhancement model to obtain integrated features of each audio data in the audio data set to be tested; inputting the integrated features into a speech enhancement decoder unit in the target speech enhancement model; Sending the encoded data to a speech enhancement decoder unit in the target speech enhancement model through a skip connection unit in the target speech enhancement model; Through the speech enhancement decoder unit in the target speech enhancement model, an enhanced audio data set corresponding to the audio data set to be tested is obtained according to the integrated features and the encoded data.
5. The audio enhancement method according to claim 1, wherein: The mathematical representation of the WWF model is: Where N is the total number of transformer encoder layers, h i is the feature set output by the i-th encoder layer, ω i is the learnable weight of the ith feature set.
6. An audio enhancement device, characterized in that: The device comprises: A first acquisition module, used to construct a training set and a test set according to a preset audio data set; A training module, used to train a preset speech enhancement model constructed based on the WavLM-large model and the WWF model through the training set and the test set to obtain a target speech enhancement model; The enhancement module is used to input the audio data set to be tested into the target speech enhancement model to obtain an enhanced audio data set.
7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.