A method, system, storage medium and program product for voice enhancement in a mask

Abstract

A method, system, storage medium and program product for speech enhancement in a mask, relating to the field of speech analysis or synthesis, the method comprising: collecting a first audio sample of a plurality of target objects in a mask-wearing state and a second audio sample in a mask-not-wearing state; determining the spectrum difference characteristics under the mask effect according to the first audio sample and the second audio sample; training a model based on the spectrum difference characteristics to obtain a speech enhancement model; obtaining a real-time audio signal of a target user in a work scene, and performing noise reduction preprocessing on the real-time audio signal to obtain a preprocessed audio signal; inputting the preprocessed audio signal into the speech enhancement model to obtain an enhanced audio signal; adjusting the enhanced audio signal according to a preset user configuration to obtain an output audio signal, and outputting the target speech as the output audio signal. Implementation of this application can improve the speech quality of the sound emitted in the mask and ensure smooth communication when multiple people work together.

Description

Technical Field

[0001] The present application relates to the field of speech analysis or synthesis, and in particular to a method, system, storage medium and program product for in-mask speech enhancement. Background Art

[0002] With the rapid development of industrial production, emergency rescue and other fields, the use of various types of protective masks is becoming more and more widespread. In special work scenarios such as fire rescue sites, underwater working environments, and industrial anti-toxicity, workers need to wear professional protective masks to ensure personal safety. In these scenarios, clear voice communication is of great significance to work efficiency and safety. However, the changes in the acoustic environment generated during the wearing of the mask and the complex environmental noise have brought severe challenges to voice communication.

[0003] In related technologies, the speech enhancement scheme mainly uses a noise reduction microphone array to collect speech signals and uses traditional digital signal processing algorithms to suppress noise. Generally, a microphone array is used to obtain the target speech and environmental noise, and then classic algorithms such as Wiener filtering or spectral subtraction are used to reduce noise on the signal. Finally, the gain of different frequency bands is adjusted through an equalizer to improve the clarity of the speech signal.

[0004] However, the speech enhancement methods in the related art are difficult to fully consider the impact of the special environment inside the mask on the sound, which can easily cause speech signal attenuation and distortion. When multiple people work together, speech distortion will hinder communication. Summary of the invention

[0005] The present application provides a method, system, storage medium and program product for enhancing speech in a mask, which are used to improve the speech quality of speech emitted in a mask and ensure smooth communication when multiple people work together.

[0006] In the first aspect, the present application provides a method for speech enhancement inside a mask, which is applied to a speech enhancement system, and the method includes: collecting a first audio sample of multiple target objects in a mask-wearing state and a second audio sample of a mask-not-wearing state; determining the spectrum difference characteristics under the mask effect based on the first audio sample and the second audio sample; the spectrum difference characteristics include frequency response offset and amplitude attenuation; performing model training based on the spectrum difference characteristics to obtain a speech enhancement model; obtaining a real-time audio signal of a target user in a work scene, and performing noise reduction preprocessing on the real-time audio signal to obtain a preprocessed audio signal; inputting the preprocessed audio signal into the speech enhancement model to obtain an enhanced audio signal; adjusting the enhanced audio signal according to a preset user configuration to obtain an output audio signal, and outputting the target speech with the output audio signal.

[0007] In the above embodiment, the speech enhancement system collects audio samples with and without a mask, analyzes the spectral difference characteristics, establishes a speech enhancement model, and combines real-time noise reduction preprocessing and user configuration to achieve precise enhancement of the speech inside the mask. By determining the actual impact of the mask on the speech, the naturalness and clarity of the enhanced speech are ensured, thereby improving the quality of language communication when wearing a mask.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of inputting the preprocessed audio signal into a speech enhancement model to obtain an enhanced audio signal specifically includes: collecting environmental parameters of the working scene; the environmental parameters include temperature parameters, humidity parameters, atmospheric pressure parameters and noise level parameters; when the working scene is determined to be in an abnormal state according to the environmental parameters, obtaining the pre-stored compensation parameters corresponding to the abnormal state; adjusting the processing parameters of the speech enhancement model according to the environmental parameters and the compensation parameters to obtain a speech enhancement optimization model; inputting the preprocessed audio signal into the speech enhancement optimization model to obtain an enhanced audio signal.

[0009] In the above embodiment, the speech enhancement system dynamically adjusts the processing parameters of the speech enhancement model by monitoring environmental parameters and applying compensation parameters under abnormal conditions, so that the system can adapt to various complex working environments and ensure the stability and reliability of the speech enhancement effect in different environments.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the step of determining the spectral difference characteristics under the mask effect based on the first audio sample and the second audio sample specifically includes: collecting material influence parameters of multiple different types of masks, and constructing a mask material feature library based on the material influence parameters; obtaining the vocal feature data of the target object, and constructing an acoustic mapping model based on the mask material feature library and the vocal feature data; obtaining the wearing type of the mask and the identity information of the target user in real time, and determining the spectral compensation parameters according to the acoustic mapping model; determining the spectral difference characteristics based on the first audio sample, the second audio sample and the spectral compensation parameters.

[0011] In the above embodiment, the speech enhancement system achieves precise adaptation to different types of masks by establishing a mask material feature library and an acoustic mapping model, and combining real-time wearing information for spectral compensation. This is suitable for the problem that different mask materials have different effects on speech, thereby improving the versatility of the system.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the step of determining the spectrum difference characteristics based on the first audio sample, the second audio sample and the spectrum compensation parameters specifically includes: performing time-frequency domain decomposition on the first audio sample and the second audio sample to obtain time-varying spectrum characteristics; inputting the time-varying spectrum characteristic matrix into a preset spectrum mapping model to obtain spectrum envelope characteristics and harmonic structure characteristics; correcting the spectrum envelope characteristics and harmonic structure characteristics according to the spectrum compensation parameters to obtain spectrum optimization characteristics; analyzing the spectrum optimization characteristics to determine the energy distribution characteristics and phase characteristics of the spectrum optimization characteristics in different frequency bands; and calculating the frequency response offset and amplitude attenuation according to the energy distribution characteristics and phase characteristics to obtain spectrum difference characteristics.

[0013] In the above embodiment, the speech enhancement system performs multi-dimensional analysis on the speech signal through time-frequency domain decomposition and spectrum mapping model, realizes accurate extraction of spectrum difference features, and ensures the training quality of the speech enhancement model.

[0014] In combination with some embodiments of the first aspect, in some embodiments, before the step of adjusting the enhanced audio signal according to the preset user configuration to obtain the output audio signal, and outputting the target voice as the output audio signal, the method also includes: constructing a user configuration template library including multiple scene templates; collecting the voice habit data of the target user, and matching the target template from the user configuration template library according to the voice habit data; receiving the real-time scene requirements input by the user, and adjusting the template parameters of the target template according to the real-time scene requirements, and determining the preset user configuration.

[0015] In the above embodiment, the voice enhancement system achieves personalized customization of voice enhancement effects by establishing a scene template library and dynamically matching it with user voice habits, thereby meeting the differentiated needs of different users in various scenarios and improving user experience.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the steps of collecting the voice habit data of the target user and matching the target template from the user configuration template library according to the voice habit data specifically include: performing frame analysis on the real-time voice of the target user, extracting acoustic features including fundamental frequency trajectory, resonance peak distribution and sound intensity change characteristics; detecting the speaking speed, pause pattern and intonation change rules of the real-time voice, and generating the prosodic features of the target user; constructing a user voice portrait of the target user according to the acoustic features and prosodic features, and generating voice habit data; inputting the voice habit data into the template matching engine, and calculating the similarity score with each scene template in the user configuration template library; and selecting the scene template with the highest similarity score as the target template.

[0017] In the above embodiment, the speech enhancement system constructs a user speech portrait by analyzing the acoustic and prosodic features of the user's speech, thereby achieving the accuracy of template matching, ensuring the matching degree between the speech enhancement effect and the user's personal characteristics, and improving the applicability of the system.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the steps of adjusting the enhanced audio signal according to a preset user configuration to obtain an output audio signal, and outputting the target speech as the output audio signal, the method also includes: constructing a speech quality assessment model including multiple feedback indicators; collecting speech feedback data of the target user in real time, and inputting the speech feedback data into the speech quality assessment model to obtain multiple quality indicator scores; when any one of the multiple quality indicator scores does not meet the preset indicator requirements, determining quality optimization suggestions for the multiple quality indicator scores; and adjusting the processing parameters of the speech enhancement model according to the quality optimization suggestions.

[0019] In the above embodiment, the speech enhancement system achieves continuous improvement of speech enhancement effect by constructing a speech quality assessment model, collecting feedback in real time and performing optimization adjustments. The long-term stability of system performance is ensured through a complete quality monitoring and optimization mechanism.

[0020] In a second aspect, an embodiment of the present application provides a speech enhancement system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the speech enhancement system to perform the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a speech enhancement system, enables the speech enhancement system to perform the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on a speech enhancement system, the speech enhancement system executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] It can be understood that the speech enhancement system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] 1. Since the first audio sample in the mask state and the second audio sample in the non-wearing state are compared and analyzed, and the spectrum difference features are extracted for model training, the frequency response offset and amplitude attenuation characteristics caused by the mask on the speech can be accurately captured, which effectively solves the problem of speech distortion caused by the mask effect that cannot be dealt with by simple signal processing in the existing technology, and then realizes the precise enhancement and high-quality restoration of the speech in the mask, ensuring the naturalness and clarity of the enhanced speech, and improving the quality of language communication when wearing a mask.

[0026] 2. Since the time-frequency domain decomposition technology is used to process the audio samples, and the spectrum envelope characteristics and harmonic structure characteristics are extracted through the spectrum mapping model, and then optimized and corrected in combination with the spectrum compensation parameters, the energy distribution and phase characteristics of the speech signal in each frequency band can be fully and accurately obtained, which effectively solves the problem of insufficient extraction of complex spectrum change features caused by the mask in the existing technology, and then realizes the accurate characterization of speech features under the mask effect, providing a more reliable training basis for the speech enhancement model, and ensuring the accuracy and effect of subsequent enhancement processing.

[0027] 3. Since the frame analysis technology is used to extract acoustic features such as fundamental frequency trajectory and resonance peak distribution, and the user voice portrait is constructed in combination with rhythmic features such as speaking rate and pause pattern, it can comprehensively and accurately describe the user's personalized voice characteristics, effectively solving the problem of unsatisfactory enhancement effect caused by insufficient template matching accuracy in the existing technology, and thus achieving accurate matching of voice enhancement effect with user's personal characteristics, thereby improving the personalization level and applicability of voice enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the method for speech enhancement in a mask in an embodiment of the present application;

[0029] Figure 2 is another flow chart of the method for in-mask speech enhancement in an embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of a physical device structure of a speech enhancement system in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0032] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0033] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.

[0034] In a fire rescue detachment, firefighters need to wear positive pressure air respirator masks throughout the process of handling hazardous chemical accidents. During combat deployment, a firefighter found that wearing a mask caused serious voice distortion and ambiguity, and team members often needed to confirm instructions repeatedly, especially in the case of huge noise in the fire environment. Not only did it reduce the rescue efficiency, but it could also affect rescue safety due to inaccurate information transmission. When multiple rescue teams work together, the voice distortion caused by the mask also increases the difficulty of communication and easily leads to operational errors.

[0035] In the related art, the masked voice enhancement processing can be achieved by using a traditional voice enhancement algorithm, such as a simple equalizer adjustment or fixed parameter noise suppression. The following introduces the scenario of using the masked voice enhancement method in the related art.

[0036] The fire brigade initially tried to use a simple loudspeaker to solve this problem. They purchased a batch of portable loudspeakers and asked firefighters to wear them on their combat uniforms. However, this solution has several problems: the loudspeaker can only amplify the sound and cannot solve the problem of sound quality distortion caused by the mask; secondly, in dangerous areas such as flammable and explosive areas, electronic equipment may pose a safety hazard; different types of masks (such as positive pressure air respirators and gas masks) have different effects on voice, and uniform volume amplification is difficult to adapt to the needs of different scenarios.

[0037] By adopting the in-mask speech enhancement method in the embodiment of the present application, by constructing a speech enhancement model, the speech features blocked by the mask are accurately restored, ensuring the optimization of the speech enhancement effect. The following introduces the scenario in which the in-mask speech enhancement method in the present application is used.

[0038] The present application provides a speech enhancement system that can collect voice samples of firefighters, establish a speech enhancement model, and then automatically adjust speech enhancement parameters based on the model. For example, when conducting fire rescue, the speech enhancement system recognizes that the firefighter is wearing a positive pressure air respirator mask and is in a high noise environment, and will focus on enhancing the mid- and high-frequency speech components attenuated by the mask. The system will automatically improve the speech clarity according to the environmental noise so that the instructions can be accurately conveyed.

[0039] The speech enhancement system is mainly composed of a microphone array built into the mask, an environmental sensor group (including temperature, humidity, air pressure and noise detectors) and a pressure sensor array to form a data acquisition end, which is connected to a data processing end (a portable microcomputer or a remote server, etc.) through wireless or wired communication; the data processing end collects audio samples when the mask is worn and not worn, combines the mask material feature library and the acoustic mapping model, establishes spectral difference features, and then trains a speech enhancement model; in actual applications, the system monitors environmental parameters and mask wearing status in real time, dynamically adjusts processing parameters, and performs personalized optimization based on user configuration templates and voice habit data, and finally outputs enhanced voice signals, thereby realizing adaptive processing of different types of masks, usage environments and user characteristics.

[0040] It can be seen that the in-mask speech enhancement method in the embodiment of the present application can not only improve the speech clarity, but also effectively solve the distortion problem caused by the traditional fixed parameter processing solution, thereby achieving a more natural and personalized speech enhancement effect.

[0041] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , which is a flow chart of the method for in-mask speech enhancement in an embodiment of the present application.

[0042] S101. Collect first audio samples of multiple target objects in a mask-wearing state and second audio samples of multiple target objects in a mask-not-wearing state.

[0043] Among them, the target object refers to the test person who needs to collect voice samples, which may include people of different genders, ages and voice characteristics; the first audio sample refers to the voice data collected in a controlled environment wearing a mask, which is used to characterize the impact of the mask on the voice; the second audio sample refers to the voice data in the same environment without wearing a mask, which is used as a comparison benchmark.

[0044] Before training the model, the speech enhancement system needs to build a training data set first. Specifically, the system uses a high-fidelity microphone array to collect speech data of multiple target subjects reading preset texts in two states in a professional recording studio environment. During the collection process, the system controls key parameters such as environmental noise, sampling frequency, and signal-to-noise ratio to ensure the quality of sample data. At the same time, the system also records detailed metadata for each sample, including speaker information, mask type, collection environment parameters, etc.

[0045] S102: Determine frequency spectrum difference characteristics under the mask effect according to the first audio sample and the second audio sample.

[0046] Among them, the mask effect refers to the acoustic impact of the mask on speech propagation; the spectrum difference characteristics refer to the change characteristics of the speech signal in the frequency domain before and after wearing the mask, including the frequency response offset and amplitude attenuation; the frequency response offset indicates the degree of offset of the energy distribution in each frequency band; the amplitude attenuation is used to indicate the attenuation of the signal strength.

[0047] After obtaining paired audio samples, the speech enhancement system needs to perform spectral feature analysis. Specifically, the system first preprocesses and frames the audio samples, and then converts the signal to the frequency domain through short-time Fourier transform. The system compares and analyzes the spectral features before and after wearing the mask, calculates the energy difference and phase change of each frequency band, and thus obtains the characteristic description of the mask effect. These features will serve as an important basis for subsequent model training.

[0048] In some embodiments, the determination of the spectrum difference characteristics can be achieved in a variety of ways: Optionally, the system can use spectrogram analysis technology to extract frequency domain features, then use wavelet transform to perform time-frequency joint analysis, and extract harmonic structure features through cepstrum analysis. Optionally, the system can also establish an acoustic transfer function model, then calculate the degree of deviation of the frequency response curve, and analyze the attenuation characteristics of the signal envelope. It is understandable that other feature extraction methods can also be used to determine the spectrum difference, which is not limited here.

[0049] S103: Perform model training based on the spectrum difference features to obtain a speech enhancement model.

[0050] Among them, the speech enhancement model refers to a mathematical model used to improve the speech quality inside the mask, and its training parameters refer to the adjustable weights and biases in the model.

[0051] The speech enhancement system needs to build an enhancement model based on the extracted features. Specifically, the system uses a deep learning framework, takes spectral difference features as input, takes the features of unmasked speech as the target, and optimizes the model parameters through multiple rounds of iterative training. During the training process, the system dynamically adjusts the learning rate, uses cross-validation to ensure the generalization ability of the model, and uses an early stopping strategy to prevent overfitting.

[0052] Specifically, the speech enhancement model adopts a multi-stage processing architecture, including a feature extraction network, a mask compensation encoder, and a speech reconstruction decoder.

[0053] The feature extraction network first transforms the input masked speech signal x(t) into a time-frequency spectrum X(ω, t), and then extracts the time-frequency features through a multi-layer convolutional neural network. The mask compensation encoder learns the influence of the mask on the speech and establishes a nonlinear mapping of the mask acoustic transfer function H(ω).

[0054] That is, assuming that the original clear speech signal is s(t), the acoustic transfer function of the mask is h(t), and the ambient noise is n(t), the observed speech signal x(t) with the mask can be expressed as: x(t)=h(t)*s(t)+n(t); where * represents the convolution operation; in the frequency domain, this relationship can be expressed as: X(ω)=H(ω)S(ω)+N(ω). The goal of speech enhancement is to restore the original speech s(t) from the observed signal x(t).

[0055] The speech reconstruction decoder restores clear speech based on the mask compensation information. The objective function of the entire model can be expressed as: L=λ 1 Lrecon+λ 2 Lspec+λ 3 Lperc;

[0056] Where Lrecon represents the time domain reconstruction loss, which is used to ensure the overall restoration quality of the signal; Lspec is the spectrum loss, which ensures the accurate reconstruction of the frequency domain features; Lperc is the perceptual loss, which is used to maintain the naturalness of the speech. 1 , 2 , 3 is the weight coefficient, and the optimal value is determined by grid search on the validation set.

[0057] The setting of the perceptual loss Lperc is based on the characteristics of the human auditory system and adopts a multi-level feature comparison method. The specific calculation formula is: Lperc=∑ᵢαᵢ||φᵢ(x̂)-φᵢ(x)|| 2 ²+β||G(x̂)-G(x)|| 1 ;

[0058] Where φᵢ represents the i-th layer feature extractor of the pre-trained acoustic model, x̂ represents the enhanced speech, and x represents the target clear speech. G represents the Gammatone filter bank, which is used to simulate the frequency selectivity of the human cochlea. αᵢ and β are the weight coefficients of features at different levels.

[0059] This loss function can be decomposed into three key components:

[0060] 1. Acoustic feature loss: Extract multi-level features through a pre-trained acoustic model and calculate the mean square error between the reconstructed signal and the target signal at each feature layer. The pre-trained model uses an acoustic encoder trained on large-scale speech recognition tasks to ensure that the extracted features are speech-perceptually relevant.

[0061] 2. Auditory perception loss: The signal is analyzed using the Gammatone filter bank. The center frequency of the filter is set according to the frequency resolution characteristics of the human ear. The calculation formula is:

[0062] g(t)=at^(n-1)exp(-2πbt)cos(2πfct+φ);

[0063] Where a, b, n are filter parameters, and fc is the center frequency.

[0064] 3. Temporal consistency loss: ensure the temporal continuity of the enhanced speech and avoid artifacts. The calculation formula is: Ltc=||∇t(x̂)-∇t(x)|| 1 ; where ∇t represents the time domain gradient operator.

[0065] In addition, for the processing of different voices under the same type of mask, the model introduces a speaker adaptation layer. This layer contains an attention mechanism to capture the differences in acoustic features of different speakers. The attention calculation formula is: A(Q, K, V) = softmax(QK^T / √d)V;

[0066] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d is the feature dimension. Through this mechanism, the model can automatically adjust the compensation parameters according to the voice characteristics of different speakers.

[0067] During the data collection phase, we collected speech data from 50 speakers wearing three typical masks (gas mask, fire mask, diving mask), with each speaker reading 100 standard texts. The sampling rate was set to 48kHz, and professional recording equipment was used to ensure that the signal-to-noise ratio was greater than 40dB. At the same time, metadata such as speaker information, mask type, and wearing status of each sample were recorded, which was used for subsequent multi-task learning.

[0068] For the preprocessing of training data, we first use the formula y(n)=x(n)-0.97x(n-1) for pre-emphasis, and then use the Hamming window for framing (frame length 25ms, frame shift 12.5ms). Each frame of signal is subjected to a 512-point FFT to obtain a complex spectrum, which is then mapped to an 80-dimensional Mel frequency scale. The processed features retain the detailed information of the speech while reducing the computational complexity.

[0069] The model training adopts a phased strategy. In the first phase, only the basic acoustic feature extraction network is trained, and the optimization goal is to minimize the reconstruction error. In the second phase, a speaker adaptation layer is added, and contrastive learning loss is introduced to enhance the model's ability to recognize speaker differences. In the third phase, the overall fine-tuning is performed, and all loss functions are optimized at the same time. The training uses the Adam optimizer, with an initial learning rate of 0.001 and a batch size of 32. The early stopping mechanism is started when the performance of the validation set no longer improves.

[0070] In practice, the model processes the speech of new speakers through an online adaptive mechanism. The system first extracts the speaker's acoustic feature vector, and then updates the attention weights in the adaptive layer to achieve rapid adaptation. This approach avoids the need to retrain the model for each new speaker, improving the practicality of the system.

[0071] In order to further improve the generalization ability of the model, data enhancement technology can also be introduced. The training data can be expanded by adding different types of environmental noise, adjusting the mask wearing position, etc. At the same time, the mixed expert system (Mixture of Experts) architecture is adopted to allow different expert networks to process different types of speech features respectively, and finally fuse the output results of each expert through the gating network. The mixed expert system adopts a dynamic routing mechanism, which includes multiple expert networks and a gating network. The forward propagation process of the system can be expressed as: y=∑ᵢg(x,θg)ᵢ·E(x,θᵢ);

[0072] Among them, g(x, θg) is the gated network, which outputs the weight of each expert; E(x, θᵢ) is the output of the i-th expert network. The design of the gated network adopts the attention mechanism:

[0073] g(x, θg)=softmax(W·h(x)+b);

[0074] Where h(x) is the hidden layer representation of the input feature. The expert networks are designed with different architectures, targeting: 1. Spectral structure expert: responsible for processing the harmonic structure of speech; 2. Formant expert: focusing on the reconstruction of vocal tract features; 3. Rhythm expert: processing the rhythmic features of speech; 4. Noise expert: responsible for processing environmental noise. Each expert network is equipped with an independent set of parameters, which are optimized through end-to-end training.

[0075] The system also introduces a load balancing mechanism to ensure balanced utilization of the expert network:

[0076] Lbalance = CV(∑ᵢg(x,θg)ᵢ); where CV represents the coefficient of variation, which is used to measure the uniformity of the distribution of the frequency of experts’ usage.

[0077] During the testing phase, the model was able to achieve real-time processing. The input speech signal first passed through the feature extraction network to obtain basic acoustic features. Then the compensation parameters were adjusted through the speaker adaptation layer, and finally the reconstruction decoder generated the enhanced speech. The entire processing delay was controlled within 100ms, meeting the needs of real-time calls. Through a large number of tests and verifications, the model can achieve stable enhancement effects under different speaker and mask combinations, reducing speech distortion by more than 40% on average.

[0078] S104: Acquire a real-time audio signal of a target user in a work scene, and perform noise reduction preprocessing on the real-time audio signal to obtain a preprocessed audio signal.

[0079] Among them, the working scene refers to the actual usage environment; the real-time audio signal refers to the currently collected voice data; the noise reduction preprocessing refers to the process of removing background noise; the preprocessed audio signal is used to represent the voice data that has been preliminarily processed.

[0080] In practical applications, the speech enhancement system needs to process the speech collected on site. Specifically, the system collects the user's speech in real time through a microphone, performs frame division and windowing on the signal, and then uses an adaptive filtering algorithm to remove environmental noise. At the same time, the system will perform signal normalization and spectrum smoothing to ensure the stability of subsequent processing.

[0081] S105: Input the preprocessed audio signal into a speech enhancement model to obtain an enhanced audio signal.

[0082] The enhanced audio signal refers to the speech signal processed by the model.

[0083] The speech enhancement system needs to use a trained model to process speech. Specifically, the system converts the pre-processed audio signal into the feature format required by the model, performs inference calculations through the speech enhancement model, and obtains enhanced speech features. Then, the system reconstructs these features into time domain signals and performs post-processing to ensure the continuity of the audio.

[0084] S106: Adjust and enhance the audio signal according to the preset user configuration to obtain an output audio signal, and output the target speech using the output audio signal.

[0085] Among them, the preset user configuration refers to the processing parameters customized by the user; the output audio signal refers to the final processed speech result; and the target speech refers to the enhanced speech output by the system.

[0086] The speech enhancement system needs to be personalized before outputting the results. Specifically, the system makes final adjustments to the enhanced audio signal based on the configuration parameters preset by the user, including optimization of volume, pitch, speech speed, etc. The system ensures that the adjusted speech maintains clarity and meets the personalized needs of the user, and finally outputs the enhanced speech through the audio device.

[0087] In some embodiments, audio output adjustment can be achieved in a variety of ways: Optionally, the system can automatically select an output mode based on the scenario, and then use an equalizer to adjust the frequency response curve, and control the volume range through dynamic compression. Optionally, the system can also use a user preference model for personalized adjustment, dynamically optimize through real-time collection of feedback, and then adapt the output parameters according to the characteristics of the device. It is understandable that other adjustment methods can also be used to optimize the output effect, which is not limited here.

[0088] In the above embodiment, the system mainly adjusts parameters through preset scene templates. In practical applications, the system can also continuously optimize the processing model through deep learning technology, automatically update the parameter configuration according to user feedback, and achieve more intelligent adaptive adjustment. The following is a supplement to the scenario of this embodiment.

[0089] The voice enhancement system can automatically switch to the optimal configuration according to different rescue types and different environmental characteristics. For example, during fire rescue, the system will dynamically adjust the enhancement parameters according to the air respirator cylinder pressure and remaining time; in the handling of hazardous chemical accidents, the system will automatically adjust the processing strategy according to the protection level and environmental danger level; when conducting cross-regional joint rescue, the system can also optimize the voice quality according to the communication network status. The system has also established a voice quality feedback mechanism to continuously optimize the processing parameters by collecting firefighters' actual combat experience.

[0090] After combining the above scenarios, the following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the method for in-mask speech enhancement in an embodiment of the present application.

[0091] S201. Collect first audio samples of multiple target objects in a mask-wearing state and second audio samples of multiple target objects in a mask-not-wearing state.

[0092] Referring to step S101 , the speech enhancement system collects a first audio sample and a second audio sample.

[0093] S202, collecting material influence parameters of various types of masks, and constructing a mask material feature library based on the material influence parameters.

[0094] Among them, the material influencing parameters represent the factors affecting the physical properties of the mask on sound propagation, including the mask material damping coefficient, cavity resonance frequency, vent position and sealing edge characteristics; the mask material damping coefficient refers to the degree of absorption of sound wave energy by the material; the cavity resonance frequency represents the natural vibration frequency of the internal space of the mask; the vent position distribution is used to represent the spatial layout characteristics of the breathable area; the sealing edge characteristics refer to the acoustic sealing performance of the mask in the contact area with the face.

[0095] The speech enhancement system needs to obtain the mask material characteristics before building an acoustic model. Specifically, the system uses professional acoustic testing equipment to measure the acoustic parameters of different types of masks, including the material's sound wave transmittance, reflectivity and other physical properties. After the system standardizes the measured parameters, it establishes a structured feature database and a unique acoustic feature file for each mask type.

[0096] S203: Acquire the vocal feature data of the target object, and construct an acoustic mapping model based on the mask material feature library and the vocal feature data.

[0097] Among them, the vocal feature data represents a quantitative description of the speaker's individual speech characteristics, including vocal tract length, pronunciation method and timbre characteristics; vocal tract length refers to the length of the acoustic channel from the vocal cords to the oral cavity; pronunciation method represents the vocal tract adjustment characteristics during the speech production process; timbre characteristics are used to represent the speaker's voice personality characteristics.

[0098] The speech enhancement system needs to establish an acoustic mapping relationship before performing spectrum compensation. Specifically, the system first collects the acoustic characteristics of the target object, including parameters such as vocal tract resonance characteristics and fundamental frequency changes. These characteristics are then correlated with the data in the mask material feature library to build a nonlinear mapping model to predict how the mask affects the speech of different speakers.

[0099] It should be noted that the acoustic mapping model is a multi-task learning framework used to establish a mapping relationship between mask material characteristics and speech acoustic features. The model contains a shared feature extraction layer and two dedicated task branches. The feature extraction layer uses a multi-layer perceptron to extract a high-dimensional representation of mask material parameters (such as density, thickness, breathability, etc.); the first task branch predicts the acoustic attenuation characteristics of the mask for different frequency bands and outputs the frequency-related transfer function by regression; the second task branch predicts the changes in the resonance characteristics of the vocal tract and outputs the corrected resonance peak frequency and bandwidth. The model training adopts multi-task joint optimization, and the losses of the two tasks are combined by weighted summation. The weight coefficient determines the optimal combination through grid search. In order to improve the generalization ability of the model, the training data covers a variety of mask materials and different speaker combinations. The model completes the training after the mapping accuracy and acoustic parameter prediction error on the validation set reach the preset standard.

[0100] The interaction between the acoustic mapping model and the mask material feature library is based on a dynamic parameter adaptation mechanism. By establishing a nonlinear mapping relationship between the material feature vector and the acoustic transfer function, the conversion from feature space to acoustic parameter space is realized. In actual operation, the system can collect the contact state data between the mask and the face in real time through the pressure sensor array to construct the mask deformation tensor. This tensor is subjected to tensor operations with the static parameters in the material feature library to obtain a real-time acoustic impedance distribution. The system inputs this time-varying impedance distribution into the acoustic mapping model and dynamically updates the acoustic transfer function. This parameter adaptation method based on a physical model ensures that the system can accurately capture the impact of slight changes in the mask wearing state on the acoustic characteristics. For example, when firefighters adjust the position of the positive pressure air respirator mask or replace the gas cylinder during the rescue process, the system can sense the changes in the mask air tightness and positive pressure state in real time, and adjust the acoustic compensation parameters accordingly to maintain the stability of the speech enhancement effect.

[0101] S204: Acquire the wearing type of the mask and the identity information of the target user in real time, and determine the spectrum compensation parameters according to the acoustic mapping model.

[0102] Among them, the wearing type indicates the specific usage and status of the mask; the identity information refers to the user's personal characteristic identifier; the spectrum compensation parameter is used to represent the correction coefficient for a specific user and mask combination.

[0103] The speech enhancement system needs to determine personalized compensation parameters before processing real-time speech. Specifically, the system uses sensors to detect the wearing position and sealing condition of the mask, and recognizes the current user's identity. The system inputs this information into the acoustic mapping model and calculates the spectrum compensation parameters for the current situation for subsequent signal processing.

[0104] S205: Determine a spectrum difference feature according to the first audio sample, the second audio sample, and the spectrum compensation parameter.

[0105] The frequency spectrum difference feature indicates the influence of the mask on the frequency domain characteristics of the speech signal; the first audio sample and the second audio sample are used to provide a comparison benchmark.

[0106] The speech enhancement system needs to comprehensively analyze multi-dimensional information before the final feature extraction. Specifically, the system performs spectral analysis on the paired audio samples, combines the previously determined spectral compensation parameters, and obtains a more accurate description of the spectral difference characteristics through difference calculation and feature fusion, providing a basis for subsequent model optimization.

[0107] In some embodiments, the speech enhancement system performs time-frequency domain decomposition on the first audio sample and the second audio sample to obtain time-varying spectrum features; inputs the time-varying spectrum feature matrix into a preset spectrum mapping model to obtain spectrum envelope features and harmonic structure features; corrects the spectrum envelope features and harmonic structure features according to spectrum compensation parameters to obtain spectrum optimization features; analyzes the spectrum optimization features to determine the energy distribution characteristics and phase characteristics of the spectrum optimization features in different frequency bands; calculates the frequency response offset and amplitude attenuation based on the energy distribution characteristics and phase characteristics to obtain spectrum difference features.

[0108] Among them, time-frequency domain decomposition refers to the process of analyzing the signal in both time and frequency dimensions; time-varying spectrum features refer to the spectrum features that change with time; spectrum envelope features refer to the features that describe the overall shape of the spectrum; harmonic structure features are used to represent the periodic components of sound; spectrum optimization features represent the corrected spectrum features; energy distribution features refer to the distribution of energy in each frequency band; and phase features are used to represent the phase information of the signal.

[0109] The speech enhancement system needs to perform in-depth spectrum analysis before determining the spectral difference characteristics. Specifically, the system first performs a short-time Fourier transform on the paired audio samples to obtain a time-frequency representation. Then, the spectral features are extracted through a pre-trained mapping model and optimized according to the spectral compensation parameters. The system further analyzes the energy and phase characteristics of the optimized features and finally calculates a quantitative description of the spectral differences.

[0110] It should be noted that the preset spectrum mapping model inputs a large number of paired unmasked / masked speech sample pairs during training, obtains time-frequency features through short-time Fourier transform, and establishes spectrum mapping relationships. The training goal is to minimize the spectrum difference before and after mapping while maintaining the naturalness of the speech. Data from multiple speakers and multiple mask types are used during training to ensure the generalization ability of the model. The preset spectrum mapping model adopts a deep neural network architecture, which includes three main modules: feature extraction, mapping conversion, and reconstruction. The feature extraction module is responsible for separating the spectrum envelope and harmonic structure, the mapping conversion module learns the influence of the mask on these features, and the reconstruction module generates the target spectrum by comprehensively optimizing the features. By inputting the time-varying spectrum feature matrix of masked speech in real time, the preset spectrum mapping model can output spectrum envelope features and harmonic structure features for subsequent spectrum compensation and optimization.

[0111] S206: Perform model training based on the spectrum difference features to obtain a speech enhancement model.

[0112] Referring to step S103, the speech enhancement system trains a speech enhancement model.

[0113] S207: Acquire a real-time audio signal of a target user in a work scene, and perform noise reduction preprocessing on the real-time audio signal to obtain a preprocessed audio signal.

[0114] Referring to step S104, the speech enhancement system determines the pre-processed audio signal of the user.

[0115] S208: Collect environmental parameters of the working scene.

[0116] Among them, environmental parameters represent the physical environment status indicators of the working scene, including temperature parameters, humidity parameters, atmospheric pressure parameters and noise level parameters; temperature parameters refer to the numerical characteristics of ambient temperature; humidity parameters represent the degree of moisture content in the air; atmospheric pressure parameters are used to represent the air pressure state; noise level parameters refer to the intensity indicators of ambient noise.

[0117] The speech enhancement system needs to evaluate the environmental conditions before performing speech processing. Specifically, the system collects environmental data in real time through multiple sensors, including data from temperature and humidity detectors, barometers, and noise detectors. The system performs standardization and anomaly detection on the collected data, establishes a state description of the current environment, and provides an environmental reference for subsequent processing.

[0118] S209: When it is determined according to the environmental parameters that the working scene is in an abnormal state, obtain a pre-stored compensation parameter corresponding to the abnormal state.

[0119] The abnormal state refers to the situation where the environmental parameters exceed the normal working range; the compensation parameter refers to the adjustment coefficient used to correct the abnormal influence.

[0120] The speech enhancement system needs to start the compensation mechanism when it detects an abnormal environment. Specifically, the system compares the real-time environmental parameters with the preset thresholds. When an abnormality is found, the corresponding compensation parameters are retrieved from the system database. These parameters are obtained through a large number of experiments and data analysis, and can effectively offset the negative impact of environmental abnormalities.

[0121] S210: Adjust the processing parameters of the speech enhancement model according to the environmental parameters and the compensation parameters to obtain a speech enhancement optimization model.

[0122] Among them, the processing parameters represent the core operating parameters of the speech enhancement model, including the frequency response compensation coefficient, the signal-to-noise ratio threshold and the gain adjustment factor; the frequency response compensation coefficient refers to the weight factor of the frequency domain correction; the signal-to-noise ratio threshold is used to represent the judgment criterion for noise suppression; the gain adjustment factor represents the control parameter of the signal amplification; the speech enhancement optimization model refers to the improved model after parameter adjustment.

[0123] The speech enhancement system needs to adjust the model parameters according to the environmental conditions. Specifically, the system comprehensively analyzes the environmental parameters and compensation parameters, and dynamically adjusts the core processing parameters of the model, including modifying the frequency response characteristics, updating the noise judgment criteria, and adjusting the gain control strategy. Through these adjustments, the system generates an optimized model that adapts to the current environment.

[0124] Among them, the abnormal processing of environmental parameters adopts a multi-level adaptive compensation mechanism, and the system can intelligently respond to abnormal working conditions by constructing a mapping relationship between the environmental state vector and the compensation strategy. The system first establishes an environmental state assessment model, which organizes environmental parameters such as temperature, humidity, and air pressure into state vectors, and inputs them into the anomaly detector after dimensionality reduction through principal component analysis. The anomaly detector calculates the degree of deviation between the current state and the normal working condition based on the Gaussian mixture model, and quantifies the degree of abnormality into multiple levels. The system configures a corresponding compensation model library for each abnormality level. These compensation models are obtained through training with a large amount of experimental data and can accurately describe the influence of changes in environmental parameters on acoustic characteristics. When an environmental abnormality is detected, the system selects the corresponding compensation model according to the abnormality level and dynamically adjusts the core parameters of the speech enhancement model through an adaptive filter. At the same time, the system will start the environmental parameter trend analysis, predict the parameter change trend, adjust the processing strategy in advance, and prevent the sudden change of the processing effect caused by the drastic change of the environment. For example, when the system is used in a high temperature and high humidity environment, the compensation mechanism can automatically adjust the frequency response compensation curve according to the changes in the acoustic characteristics of the air to maintain the stability of the speech enhancement effect.

[0125] S211. Input the preprocessed audio signal into a speech enhancement optimization model to obtain an enhanced audio signal.

[0126] Among them, the enhanced audio signal refers to the optimized speech after model processing.

[0127] The speech enhancement system needs to process real-time speech after completing model optimization. Specifically, the system converts the preprocessed audio signal into a format that the model can process and inputs it into the optimized speech enhancement model. The model enhances the signal according to the current parameter settings and outputs a speech signal with improved quality.

[0128] S212: Build a user configuration template library including multiple scene templates.

[0129] Among them, the scene template represents a combination of parameters preset for different usage scenarios; the user configuration template library refers to a database that stores multiple scene configurations; the template parameters of the scene template include volume gain, noise reduction level, speech clarity and sound quality balance; volume gain indicates the degree of sound amplification; noise reduction level is used to indicate the intensity of noise suppression; speech clarity refers to the degree of speech recognition; sound quality balance indicates the balance of energy in each frequency band.

[0130] The voice enhancement system needs to establish a configuration library that adapts to different scenarios. Specifically, the system pre-sets corresponding parameter templates for different application scenarios such as fire, outdoor, and underwater. Each template contains a complete set of audio processing parameters. These parameters have been professionally debugged and verified in practice, and can provide the best voice effects for different scenarios.

[0131] S213: Collect the voice habit data of the target user, and match the target template from the user configuration template library according to the voice habit data.

[0132] Among them, voice habit data represents the user's personal speaking characteristics, including speaking speed, pitch range and pronunciation characteristics; speaking speed refers to the speed of the voice rhythm; pitch range represents the range of changes in sound frequency; pronunciation characteristics are used to represent an individual's unique way of speaking; target template refers to the scene template that best matches the user's characteristics.

[0133] The speech enhancement system needs to select the appropriate configuration template based on the user's characteristics. Specifically, the system collects the user's speaking characteristics through speech analysis, including data on speech speed, range, tone, and other dimensions. The system calculates the matching degree between these characteristics and the configuration in the template library, and selects the template that best suits the current user as the basic configuration.

[0134] In some embodiments, the speech enhancement system performs frame analysis on the real-time speech of the target user, extracts acoustic features including fundamental frequency trajectory, resonance peak distribution, and sound intensity change characteristics; detects the speaking speed, pause pattern, and intonation change rules of the real-time speech, and generates the prosodic features of the target user; constructs a user speech portrait of the target user based on the acoustic features and prosodic features, and generates speech habit data; inputs the speech habit data into the template matching engine, calculates the similarity score with each scene template in the user-configured template library; and selects the scene template with the highest similarity score as the target template.

[0135] Among them, acoustic features represent the basic acoustic parameters of speech signals; fundamental frequency trajectory refers to the curve of pitch change over time; resonance peak distribution represents the resonance characteristics of the vocal tract; sound intensity change characteristics are used to describe the dynamic characteristics of volume; prosodic features refer to the rhythm and intonation characteristics of speech; user voice portrait represents the set of voice features of the speaker; and template matching engine refers to the processing module for similarity calculation.

[0136] The speech enhancement system needs to establish a user profile before selecting a configuration template. Specifically, the system extracts multi-dimensional features from real-time speech, including acoustic and rhythmic analysis. The system integrates these features into a user speech profile, compares it with the configuration in the template library through the matching engine, and selects the best matching template as the basic configuration.

[0137] It should be noted that the template matching engine is a big data model. During training, the template matching engine collects a large amount of user voice data and corresponding optimal configuration parameters to establish a mapping relationship between user features and processing parameters. The training goal of the template matching engine is to maximize the comprehensive score of feature similarity and configuration applicability. The template matching engine includes a feature comparison module and a similarity calculation module. The feature comparison module processes the matching degree of acoustic features and prosodic features, and the similarity calculation module comprehensively evaluates the overall matching degree of multi-dimensional features. By inputting the user's voice habit data, the template matching engine can perform feature matching with the scene template in the template library and output the target template with the highest similarity score.

[0138] S214: receiving a real-time scene requirement input by a user, adjusting template parameters of a target template according to the real-time scene requirement, and determining a preset user configuration.

[0139] Among them, the real-time scenario requirements represent the user's current usage environment and requirements; the preset user configuration represents the finalized personalized parameter set.

[0140] The voice enhancement system needs to optimize configuration parameters according to actual needs. Specifically, the system receives the scene requirements input by the user through the interface or voice, analyzes the characteristics of the requirements and compares them with the current template parameters. The system adjusts the relevant parameters in the template according to the requirements to form a personalized configuration that is more in line with the current scenario, ensuring that the voice enhancement effect meets user needs.

[0141] S215: Adjust and enhance the audio signal according to the preset user configuration to obtain an output audio signal, and output the target speech as the output audio signal.

[0142] Referring to step S106, the speech enhancement system outputs the target speech.

[0143] In some embodiments, the speech enhancement system constructs a speech quality assessment model including multiple feedback indicators; the feedback indicators include speech clarity index, acoustic feature fidelity and subjective satisfaction score; the speech feedback data of the target user is collected in real time, and the speech feedback data is input into the speech quality assessment model to obtain multiple quality indicator scores; when any one of the multiple quality indicator scores does not meet the preset indicator requirements, quality optimization suggestions for the multiple quality indicator scores are determined; and the processing parameters of the speech enhancement model are adjusted according to the quality optimization suggestions.

[0144] Among them, feedback indicators represent the measurement criteria for evaluating speech quality, including speech clarity index, acoustic feature fidelity and subjective satisfaction score; speech quality assessment model refers to the mathematical model used to quantitatively evaluate speech quality; quality indicator scores are used to represent the evaluation results of various indicators; quality optimization suggestions refer to improvement plans for quality problems.

[0145] The speech enhancement system needs to continuously evaluate the processing effect during operation. Specifically, the system establishes a comprehensive quality evaluation system to collect and analyze user feedback data in real time. When the evaluation results show that certain indicators are not up to standard, the system will generate specific optimization suggestions and adjust the model parameters accordingly to improve the processing effect.

[0146] It should be noted that the speech quality assessment model inputs paired data of subjective and objective evaluations during training, including expert scores and objective indicators. The training data covers speech samples in different scenarios and their quality assessment results. The training goal of the speech quality assessment model is to make the model prediction score match the expert score to the greatest extent possible, while ensuring the balance of multiple quality dimensions. The speech quality assessment model adopts a multi-layer perceptron structure and integrates multiple evaluation indicators, including clarity, naturalness, coherence and other dimensions. The speech quality assessment model includes a feature extraction layer, an evaluation calculation layer and a score output layer. By inputting the processed speech signal in real time, the speech quality assessment model can output multiple quality indicator scores for evaluating the enhancement effect and guiding parameter adjustment.

[0147] In the embodiments of the present application, since environmental perception technology, acoustic mapping modeling, user feature analysis and adaptive processing algorithms are combined, and deep learning technology is used to realize feature extraction and parameter optimization, it is possible to make real-time dynamic adjustments according to different mask types, wearing states, usage environments and user habits, effectively solving the technical problems such as blind processing, fixed parameters and poor adaptability existing in traditional speech enhancement solutions, thereby achieving more accurate speech feature compensation, more natural sound restoration, more personalized parameter configuration and more intelligent scene adaptation, and improving communication efficiency and accuracy when wearing protective equipment.

[0148] The following describes the speech enhancement system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of a physical device structure of a speech enhancement system in an embodiment of the present application.

[0149] It should be noted that Figure 3 The structure of the voice enhancement system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0150] like Figure 3As shown, the speech enhancement system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through the bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0151] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0152] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0153] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0155] Specifically, the speech enhancement system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the in-mask speech enhancement method provided by the above embodiment is implemented.

[0156] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the speech enhancement system described in the above embodiment; or may exist independently without being assembled into the speech enhancement system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the speech enhancement system, the speech enhancement system implements the in-mask speech enhancement method provided in the above embodiment.

[0157] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0158] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0159] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A method for in-mask speech enhancement, characterized in that: Applied to a speech enhancement system, the method comprises: Collecting a first audio sample of a plurality of target objects in a mask-wearing state and a second audio sample of a plurality of target objects in a mask-not-wearing state; Determine the frequency spectrum difference characteristics under the mask effect according to the first audio sample and the second audio sample; the frequency spectrum difference characteristics include frequency response offset and amplitude attenuation; the step of determining the frequency spectrum difference characteristics under the mask effect according to the first audio sample and the second audio sample specifically includes: collecting material influence parameters of multiple different types of masks, and constructing a mask material feature library based on the material influence parameters; obtaining the vocal feature data of the target object, and constructing an acoustic mapping model based on the mask material feature library and the vocal feature data; obtaining the wearing type of the mask and the identity information of the target user in real time, and determining the frequency spectrum compensation parameters according to the acoustic mapping model; determine the frequency spectrum difference characteristics according to the first audio sample, the second audio sample and the frequency spectrum compensation parameters; Performing model training based on the frequency spectrum difference features to obtain a speech enhancement model; Acquire a real-time audio signal of a target user in a work scene, and perform noise reduction preprocessing on the real-time audio signal to obtain a preprocessed audio signal; Inputting the preprocessed audio signal into the speech enhancement model to obtain an enhanced audio signal; The enhanced audio signal is adjusted according to a preset user configuration to obtain an output audio signal, and the target speech is outputted using the output audio signal.

2. The method according to claim 1, characterized in that The step of inputting the preprocessed audio signal into the speech enhancement model to obtain an enhanced audio signal specifically comprises: Collecting environmental parameters of the working scene; the environmental parameters include temperature parameters, humidity parameters, atmospheric pressure parameters and noise level parameters; When it is determined according to the environmental parameters that the working scene is in an abnormal state, obtaining a pre-stored compensation parameter corresponding to the abnormal state; Adjusting the processing parameters of the speech enhancement model according to the environmental parameters and the compensation parameters to obtain a speech enhancement optimization model; The preprocessed audio signal is input into the speech enhancement optimization model to obtain an enhanced audio signal.

3. The method according to claim 1, characterized in that The step of determining the spectrum difference feature according to the first audio sample, the second audio sample and the spectrum compensation parameter specifically includes: Decomposing the first audio sample and the second audio sample in the time-frequency domain to obtain time-varying spectral features; Inputting the time-varying spectrum feature matrix into a preset spectrum mapping model to obtain spectrum envelope features and harmonic structure features; Correcting the spectrum envelope feature and the harmonic structure feature according to the spectrum compensation parameter to obtain a spectrum optimization feature; Analyze the spectrum optimization features to determine energy distribution features and phase features of the spectrum optimization features in different frequency bands; According to the energy distribution characteristics and the phase characteristics, the frequency response offset and the amplitude attenuation are calculated to obtain the frequency spectrum difference characteristics.

4. The method according to claim 1, characterized in that: Before the step of adjusting the enhanced audio signal according to the preset user configuration to obtain an output audio signal, and outputting the target speech with the output audio signal, the method further includes: Build a user configuration template library including multiple scenario templates; Collecting the voice habit data of the target user, and matching the target template from the user configuration template library according to the voice habit data; A real-time scene requirement input by a user is received, and template parameters of the target template are adjusted according to the real-time scene requirement to determine a preset user configuration.

5. The method according to claim 4, characterized in that The step of collecting the voice habit data of the target user and matching the target template from the user configuration template library according to the voice habit data specifically includes: Performing frame analysis on the real-time speech of the target user to extract acoustic features including fundamental frequency trajectory, formant distribution and sound intensity change characteristics; Detecting the speech rate, pause pattern and intonation change rule of the real-time speech to generate the prosodic features of the target user; constructing a user voice profile of the target user according to the acoustic features and the prosodic features, and generating voice habit data; Inputting the voice habit data into a template matching engine to calculate a similarity score with each scene template in the user-configured template library; The scene template with the highest similarity score is selected as the target template.

6. The method according to claim 1, characterized in that After the step of adjusting the enhanced audio signal according to the preset user configuration to obtain an output audio signal, and outputting the target speech with the output audio signal, the method further includes: Construct a speech quality assessment model that includes multiple feedback indicators; Collecting the target user's voice feedback data in real time, and inputting the voice feedback data into the voice quality assessment model to obtain multiple quality indicator scores; When any one of the multiple quality indicator scores does not meet the preset indicator requirement, determining quality optimization suggestions for the multiple quality indicator scores; The processing parameters of the speech enhancement model are adjusted according to the quality optimization suggestion.

7. A speech enhancement system, characterized in that: The speech enhancement system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the speech enhancement system to perform the method described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a speech enhancement system, the speech enhancement system is caused to perform the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program product runs on a speech enhancement system, the speech enhancement system is caused to perform the method according to any one of claims 1 to 6.

Images