Intelligent communication equipment voice noise reduction method and system based on artificial intelligence

Abstract

The invention belongs to the field of voice noise reduction processing, and discloses an intelligent communication equipment voice noise reduction method and system based on artificial intelligence. Comprising intelligent communication equipment, and a microphone array is deployed on the intelligent communication equipment to record real-time noisy voice data of a user; inputting the recorded noisy voice data into the edge layer, and obtaining the voice data after noise reduction through real-time processing; the method comprises the following steps: pre-constructing a dialect voice database, adding a scene evolution layer, a voice change layer and a noise mixing layer in a conditional adversarial network cGAN to expand the dialect voice database, forming a data sample, and performing three-dimensional marking on each piece of voice data in the data sample; an input, output and multi-task learning mechanism is designed based on an end-to-end voice enhancement model, pre-training is carried out by using a data sample, and a shared encoder is arranged in the multi-task learning mechanism and is used for combining a task R1, a task R2 and a task R3 to realize voice noise reduction of intelligent communication equipment.

Description

Technical Field

[0001] The present invention relates to the field of speech noise reduction processing, and more specifically, to a speech noise reduction method and system for intelligent communication equipment based on artificial intelligence. Background Art

[0002] Voice distortion and sound quality degradation are common problems in the process of voice noise reduction for smart communication devices, especially when combined with the special scenario of dialects, the problem may be further amplified. During the noise reduction process, especially in a low signal-to-noise ratio environment, some algorithms may over-filter certain frequency components in the voice signal, causing the voice to sound unnatural (such as "robotic voice" or "hollow"). This problem may be more prominent when users use dialects, because the voice characteristics of dialects (such as pitch, timbre, and harmonic structure) are significantly different from Mandarin or other standard languages.

[0003] This is because most noise reduction algorithms are trained based on large-scale datasets of Mandarin, English or other standard languages. These datasets often lack sufficient dialect samples, resulting in insufficient modeling of the characteristics of non-standard speech.

[0004] In addition, noise reduction algorithms usually distinguish speech from noise through spectral analysis, but in low signal-to-noise ratio environments, the harmonics or detail components of speech signals may overlap with the spectral features of noise. Certain unique frequency components in dialects (such as high-frequency tones or low-frequency glottal sounds) are more likely to be misjudged as noise and filtered out, and may not adapt well to such rapidly changing speech patterns, resulting in loss of details, sounding fuzzy or incomplete, and affecting the accurate transmission of semantics.

[0005] In view of this, the present invention proposes a voice noise reduction system for intelligent communication equipment based on artificial intelligence to solve the above problems. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a voice noise reduction system for intelligent communication devices based on artificial intelligence, comprising an intelligent communication device, on which a microphone array is deployed to collect real-time noisy voice data of a user; The recorded noisy speech data is input into the edge layer, where a pre-trained noise reduction algorithm is deployed. The noise-reduced speech data is obtained through real-time processing and output. The hierarchical deployment of the noise reduction algorithm specifically includes: pre-building a dialect speech database, adding a scene evolution layer, a sound change layer, and a noise mixing layer in the conditional adversarial network cGAN to expand the dialect speech database, form data samples, and perform three-dimensional annotation on each speech data in the data sample; Based on the end-to-end speech enhancement model, the input, output and multi-task learning mechanisms are designed, and data samples are used for pre-training. The multi-task learning mechanism combines deep learning structure with generative adversarial network GAN for speech reconstruction. A shared encoder is set in the multi-task learning mechanism for joint tasks R1, R2 and R3. Task R1 designs an IRM estimator based on U-Net, and uses the IRM estimator to train the cIRM model for noise suppression, and adds a lightweight signal-to-noise ratio estimator; Task R2 jointly supervises the retention of dialect features based on the feature embedding layer, tone protection module, timbre protection module and semantic-assisted denoising module; Task R3 reconstructs speech through a generative adversarial network.

[0007] Preferably, the method of deploying a microphone array on an intelligent communication device to collect user voice data includes: According to the size of the intelligent communication device and the computing resources of the edge layer, a suitable array and spacing are selected. The microphone has a high sampling rate and a wide frequency range, is equipped with an analog-to-digital converter with a high dynamic range, and is connected to the edge layer of the intelligent communication device using an interface.

[0008] Preferably, the data sample collection method includes: Record the user's dialect voice data in a low signal-to-noise ratio environment and build a dialect voice database covering the low signal-to-noise ratio environment; The architecture of the conditional adversarial network cGAN includes a generator and a discriminator. The speech data in the dialect speech database is input into the generator. The scene evolution layer, sound change layer and noise mixing layer are added to the generator. The generator and the discriminator are trained in turn by alternating training. After the training is completed, the discriminator is fixed and the generator is used to generate new low signal-to-noise ratio dialect speech data. The generated data is used to expand the existing dialect speech database to form data samples. A three-dimensional blank dataset is set up, where the three dimensions include multi-granularity acoustic features, noise features and semantic features. The multi-granularity acoustic features include the phoneme level, pronunciation method, rhythmic features and timbre of the speech data. The noise features include noise events, noise components and time-frequency distribution. The semantic features include word-level annotations, sentence-level annotations and paragraph-level annotations. The three-dimensional data of each speech data in the data sample is extracted, and the data is filled into the blank dataset to form a three-dimensional annotation for each speech data.

[0009] Preferably, the method of adding a scene evolution layer, a sound change layer and a noise mixing layer in the generator comprises: The input of the scene evolution layer is the preset scene label. According to the scene label, the corresponding noise fragment is extracted from the dialect speech database, and the scene speech is output through random mixing. Random mixing includes single scene isolation and multi-scene superposition. Single scene isolation is to directly output the selected noise fragment, and multi-scene superposition is to linearly or nonlinearly superimpose the noise fragments of multiple scenes according to the preset weight distribution mechanism of each scene. The probability distribution mechanism means that a weight value range is preset for each scenario, and the sum of the weights is less than or equal to 1 when multiple scenarios are superimposed; The input of the sound change layer is the output of the speech encoder and the sound change control parameters. According to the phonetic characteristics of the dialect, the sound change rules are defined to form a rule-based sound change model. Using the dialect speech data with sound change annotations, a sequence-to-sequence model is trained to predict and generate sound changes, forming a data-driven sound change model. The rule-based sound change model and the data-driven sound change model are combined to control the degree and type of sound change by adjusting the sound change parameters, and the sound-changed speech feature sequence is output, which is recorded as the sound-changed speech. The input of the noise mixing layer is the scene speech and the sound-changed speech, which are mixed according to the specified signal-to-noise ratio, and the speech feature sequence with noise and sound changes is output, which is recorded as the new low signal-to-noise ratio dialect speech data.

[0010] Preferably, the design of the noise reduction algorithm further includes: Design a noise reduction algorithm based on an end-to-end speech enhancement model, including input, output, and multi-task learning mechanism. Set the input as the time-frequency spectrum of noisy speech. The multi-task learning mechanism uses a deep learning structure for time-frequency processing and combines it with a generative adversarial network (GAN) for speech reconstruction. The output is a time-frequency spectrum of pure speech after noise reduction. The input noisy speech is transformed into a time-frequency spectrum through short-time Fourier transform (STFT), and the output clean speech time-frequency spectrum after noise reduction is reconstructed into a time domain signal through inverse STFT to form the noise-reduced speech data; The loss function is designed to be the weighted sum of noise reduction loss, speech reconstruction loss, dialect feature loss and naturalness loss. The noise reduction loss is used to measure the noise reduction effect in a low signal-to-noise ratio environment. The speech reconstruction loss is used to evaluate the perceptual quality of the speech data after noise reduction. The dialect feature loss is used to measure the damage of the model to the tone and timbre. The naturalness loss is used to measure the naturalness of the speech data after noise reduction. A loss priority rule is added, that is, two thresholds are preset to meet the requirement that the speech reconstruction loss is greater than the threshold and the dialect feature loss is less than the threshold. Use data samples to pre-train the denoising algorithm until the loss function value meets the requirements or the number of iterations is reached. Then, the pre-trained denoising algorithm is obtained and deployed at the edge layer to process real-time voice data.

[0011] Preferably, the multi-task learning mechanism adopts a deep learning structure for time-frequency processing, and the method for combining a generative adversarial network GAN to perform speech reconstruction includes: Set up a shared encoder to extract deep feature representation of the spectrogram, use a deep learning structure, and output the encoded feature map; The preset task-specific headers include Task R1, Task R2, and Task R3. Task R1 is used for noise suppression. A lightweight signal-to-noise ratio estimator is added after the shared encoder and before the IRM estimator in Task R1 to evaluate the signal-to-noise ratio level of the input speech. Task R2 is used to preserve dialect features. By introducing the supervision information of dialect features, key features are retained and not filtered out. Task R3 is used for speech reconstruction. It reconstructs the details of speech through a generative adversarial network, including using the generator of the generative adversarial network to reconstruct the harmonics, tones, and rapidly changing pronunciation details of the speech, and the discriminator enhances the naturalness of the speech. The dialect features output by Task R2 are introduced as a guide.

[0012] Preferably, the method for noise suppression in task R1 comprises: Task R1 designs an IRM estimator based on U-Net, whose input is the feature map output by the shared encoder. The structure includes an encoder, a decoder, and an output layer. The encoder is similar or identical to the shared encoder and contains multiple downsampling modules. The decoder is symmetrical to the encoder and contains multiple upsampling modules. The encoder and the decoder are jump-connected. The output layer is set to a 1x1 convolutional layer. The Sigmoid activation function is used to map the feature map to the IRM mask. The size of the output IRM mask is the same as the input STFT spectrum map. The loss function is mean square error or binary cross entropy. The IRM model is trained using noisy speech and clean speech data until convergence, and the output is the trained IRM model and the predicted IRM mask.

[0013] The output feature map of the shared encoder and the predicted IRM mask are used as the input of the cIRM model. The cIRM estimator is designed based on U-Net, and its structure is similar to the IRM model, but IRM fusion is added after the jump connection. The fusion method is to splice the predicted IRM with the output feature map of the encoder as the input of the cIRM encoder. In each upsampling module of the decoder, the predicted IRM is fused with the upsampled feature map. The cIRM contains real and imaginary parts. The loss function is set and the output is the predicted cIRM mask. The predicted cIRM mask is applied to the STFT spectrum of the noisy speech to obtain the STFT spectrum of the denoised speech.

[0014] Preferably, a lightweight signal-to-noise ratio estimator is added after the shared encoder and before the IRM estimator in the task R1 to evaluate the signal-to-noise ratio level of the input speech, including: The input of the signal-to-noise ratio estimator is set to the feature map output by the shared encoder. The structure includes global average pooling, a fully connected layer, an activation function, and an output layer. The output layer uses a single-output fully connected layer, and the output is the estimated SNR value. The loss function is set to mean square error to calculate the mean square error between the estimated SNR value and the true SNR value. The STFT spectrogram of the noisy speech is input into the shared encoder, and the output feature map of the shared encoder is input into the signal-to-noise ratio estimator to calculate the MSE loss between the estimated SNR value and the true SNR value. The weight of the signal-to-noise ratio estimator is updated using the back propagation algorithm. In the IRM estimator, the estimated SNR value is input as an additional input into the IRM estimator together with the original input.

[0015] Preferably, the task R2 is used to retain dialect features. By introducing the supervision information of dialect features, the method of retaining key features from being filtered out includes: It includes feature embedding layer, tone protection module, timbre protection module and semantic-assisted noise reduction module; The input of the feature embedding layer is noisy speech data + dialect label; the dialect label is converted into a vector of fixed dimension, recorded as the dialect embedding vector; this dialect embedding vector is concatenated with the speech feature vector or fused through the attention mechanism to form a new vector, which will be used as the input of the subsequent noise reduction algorithm; The input of the tone protection module is noisy speech data. It uses a pre-trained dialect tone classifier and outputs the tone category probability distribution and tone contour features for each time frame. In the process of noise reduction, the tone contour features of the output speech are constrained by the loss function so that the similarity between them and the tone contour features of the tone protection module meets the requirements; The input of the timbre protection module is noisy speech data. The pre-trained timbre feature extractor is used to output a timbre feature vector of fixed dimension. A term is added to the loss function of the denoising process to penalize the difference between the timbre feature vectors before and after denoising. The input of semantic-assisted denoising is noisy speech data. Using a pre-trained dialect speech recognition model, the output is text transcription of the speech and information at the phoneme / syllable level. Based on the speech recognition results, tone changes, syllable boundaries and key phonemes are extracted; and protection is performed separately based on the recognition results.

[0016] A method for reducing voice noise of intelligent communication equipment based on artificial intelligence, comprising: Step S1: deploying a microphone array on the intelligent communication device to collect the user's real-time noisy voice data; Step S2: designing the input, output and multi-task learning mechanism of the noise reduction algorithm based on the end-to-end speech enhancement model; Step S3: Use data samples for pre-training, obtain the trained denoising algorithm, and deploy it to the edge layer; Step S4: input the collected noisy speech data into the edge layer, and output the noise-reduced speech data through real-time processing.

[0017] The technical effects and advantages of the speech noise reduction system for intelligent communication equipment based on artificial intelligence of the present invention are as follows: 1. By pre-building a dialect speech database and using conditional generative adversarial networks (cGAN) to expand the data, the system can effectively process various dialects and retain dialect characteristics. Through the scene evolution layer, it can adapt to complex noise environments with single and multiple scenes superimposed, which is more in line with actual usage scenarios.

[0018] 2. Data collection and algorithm training designed specifically for low signal-to-noise ratio environments enable the system to maintain good noise reduction effects in noisy environments. Through three-dimensional annotation (acoustic features, noise features, and semantic features) and multi-task learning mechanisms, the system can simultaneously achieve noise reduction, retain dialect features, and reconstruct high-quality speech. The pre-trained noise reduction algorithm is deployed at the edge layer to achieve real-time processing, reduce latency, and improve user experience.

[0019] 3. The sound change layer and dialect feature retention mechanism ensure that the noise reduction process does not destroy the key features of the speech, especially the tone, timbre and semantic information. Through the signal-to-noise ratio estimator, the noise reduction strategy can be automatically adjusted according to different signal-to-noise ratio levels to avoid speech distortion caused by excessive noise reduction. Through the cIRM model (complex ideal ratio mask) processing, it can process amplitude and phase information at the same time, which is better than the traditional method of processing only amplitude. The three tasks (R1 noise suppression, R2 dialect feature retention, R3 speech reconstruction) work together, and through the shared encoder and specific task head design, a balance between noise reduction and feature retention is achieved. Using semantic information to guide the noise reduction process can more accurately protect important speech components and reduce the loss of semantic information.

[0020] In summary, the system effectively reduces noise while retaining the characteristics of dialects, solving the problem that traditional noise reduction systems easily filter out key dialect features when processing dialects. At the same time, it achieves real-time, high-quality speech enhancement through multi-task learning and edge computing, improving the user experience of smart communication equipment in complex noisy environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a structural schematic diagram of a voice noise reduction system for intelligent communication equipment based on artificial intelligence of the present invention; Figure 2 A hierarchical schematic diagram of a noise reduction algorithm in a voice noise reduction system of an intelligent communication device based on artificial intelligence of the present invention; Figure 3 The present invention is a schematic diagram of the steps of a method for reducing voice noise of an intelligent communication device based on artificial intelligence. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Example 1, please refer to Figure 1 and Figure 2 As shown, the present embodiment describes an intelligent communication device voice noise reduction system based on artificial intelligence, and the main design contents are: Voice distortion and sound quality degradation are common problems in the process of voice noise reduction for smart communication devices, especially when combined with the special scenario of dialects, the problem may be further amplified. Specifically: During the noise reduction process, especially in low signal-to-noise ratio environments, some algorithms may over-filter certain frequency components in the voice signal, causing the voice to sound unnatural (such as "robotic" or "hollow"). This problem may be more prominent when the user speaks in a dialect, because the voice characteristics of the dialect (such as pitch, timbre, harmonic structure) are significantly different from Mandarin or other standard languages.

[0024] For example: Tone loss: Many dialects (such as Cantonese and Hokkien) have complex tone systems. The noise reduction algorithm may mistakenly filter out certain high-frequency or low-frequency tone components as noise, causing the speech to sound flat or lose semantic information.

[0025] Changes in timbre: The unique timbre of dialects (such as nasal and laryngeal sounds) may be mistaken by the algorithm as noise or non-speech components, causing the speech to sound unnatural or even "robotic."

[0026] Loss of details: Certain fast-changing pronunciation details in dialects (such as entering tones and light tones) may be smoothed out by the algorithm, resulting in fuzzy speech or semantic confusion.

[0027] Hollow feeling: In a low signal-to-noise ratio (SNR) environment, the noise reduction algorithm may over-suppress the background noise, resulting in "holes" in the spectrum of the speech signal. This phenomenon is particularly obvious in dialects because the spectral distribution of dialects may be quite different from the standard speech in the algorithm training data.

[0028] This is because most noise reduction algorithms (such as deep learning-based speech enhancement models) are trained based on large-scale datasets of Mandarin, English or other standard languages. These datasets often lack sufficient dialect samples, resulting in the algorithm's inadequate modeling of non-standard speech features (such as dialect tones and harmonics).

[0029] Denoising algorithms usually distinguish speech from noise through spectral analysis, but in low signal-to-noise ratio environments, the harmonics or detail components of speech signals may overlap with the spectral features of noise. Certain unique frequency components in dialects (such as high-frequency tones or low-frequency guttural sounds) are more likely to be misjudged as noise and filtered out.

[0030] To suppress noise, many algorithms use smoothing techniques (such as spectral subtraction, Wiener filtering, or masking in deep learning), which may weaken the rapidly changing features in the speech signal. The rapidly changing pronunciation details such as entering tone and light tone in dialects are particularly susceptible to impact.

[0031] Many dialects (such as Cantonese, which has 9 tones and Mandarin, which has only 4) rely on high-frequency or low-frequency tone information to express semantics, and noise reduction algorithms may not be able to accurately retain this information. For example, the common contrast between "unvoiced" and "voiced" in Wu dialects (such as Shanghainese), "entering" syllables in Cantonese, and "erhua" in Sichuan dialect may all be mistaken by the algorithm as noise or non-speech components. The speed and rhythm of dialects may be different from Mandarin. For example, some dialects (such as Northeastern Chinese) are spoken at a faster speed, and noise reduction algorithms may not adapt well to this rapidly changing speech pattern, resulting in loss of details.

[0032] In a low signal-to-noise ratio environment, the energy of the speech signal is seriously masked by the noise. In order to improve the signal-to-noise ratio, the noise reduction algorithm often adopts a more aggressive filtering strategy. This strategy is more likely to cause speech distortion when processing dialects, because the speech characteristics of the dialect may not match the algorithm's "speech template".

[0033] The noise-reduced speech sounds blurry or incomplete. For example, the loss of tones in Cantonese may make words such as "chicken" and "machine" indistinguishable, affecting the accurate transmission of semantics.

[0034] In scenarios such as meetings, phone calls, or voice assistants, the voices of dialect users may be over-processed, making them difficult for the caller or the voice recognition system to understand.

[0035] The timbre and intonation of a dialect are important components of its cultural and emotional expression. Excessive filtering by noise reduction algorithms may make the speech sound mechanical or lose emotion. For example, after the "erhua sound" in Sichuan dialect is filtered out, the speech may lose its intimacy.

[0036] Users may feel that the voice is "robotic", reducing their comfort level during calls or voice interactions.

[0037] For users who rely on dialects for daily communication, voice distortion may directly affect communication efficiency, especially in noisy environments (such as vegetable markets and subway stations).

[0038] In voice assistants or smart devices (such as smart speakers), improper handling of dialects by noise reduction algorithms may lead to a decrease in speech recognition rate and reduce convenience of use.

[0039] Dialects are not only a means of communication, but also a carrier of cultural heritage and emotional expression. The distortion of noise reduction algorithms may weaken the cultural characteristics of dialects and affect users' cultural identity and usage experience.

[0040] Based on this, a voice noise reduction method for intelligent communication equipment based on artificial intelligence is designed, including an intelligent communication equipment, on which a microphone array is deployed to collect the user's voice data; The recorded noisy speech data is input into the edge layer, and the noise-reduced speech data is obtained through real-time processing and output; Among them, the edge layer is deployed on the intelligent communication device with an end-to-end generation architecture, which can be installed through the chip. The purpose of the end-to-end generation architecture is to generate pure speech directly from noisy speech to avoid the destruction of dialect characteristics in the intermediate processing steps. The hierarchical deployment of the noise reduction algorithm specifically includes: pre-building a dialect speech database, adding a scene evolution layer, a sound change layer, and a noise mixing layer in the conditional adversarial network cGAN to expand the dialect speech database, form a data sample, and perform three-dimensional annotation on each speech data in the data sample; Based on the end-to-end speech enhancement model, the input, output and multi-task learning mechanisms are designed, and data samples are used for pre-training. The multi-task learning mechanism combines deep learning structure with generative adversarial network GAN for speech reconstruction. A shared encoder is set in the multi-task learning mechanism for joint tasks R1, R2 and R3. Task R1 designs an IRM estimator based on U-Net, and uses the IRM estimator to train the cIRM model for noise suppression, and adds a lightweight signal-to-noise ratio estimator; Task R2 jointly supervises the retention of dialect features based on the feature embedding layer, tone protection module, timbre protection module and semantic-assisted denoising module; Task R3 reconstructs speech through a generative adversarial network.

[0041] The method of deploying a microphone array on an intelligent communication device to collect user voice data includes: According to the size of the intelligent communication device and the computing resources of the edge layer, the appropriate array and spacing are selected. The array includes linear array, circular array, planar array and three-dimensional array, etc. The spacing generally includes narrow spacing, wide spacing and mixed spacing. Generally speaking, the larger the array, the better the performance. The microphone selects a high sampling rate (such as 48kHz or higher) and a wide frequency range to ensure that the complete spectrum information of the voice signal, including dialects, can be fully captured, covering high-frequency tones, low-frequency throat sounds and other key audio features. To further improve the pickup effect, the sound source positioning and beamforming are achieved through the spatial distribution of multiple microphones, thereby enhancing the ability to extract the target voice signal. Equipped with a high dynamic range analog-to-digital converter (ADC), and connected to the edge layer of the intelligent communication device using an interface (such as I2S, SPI or USB). To ensure the high fidelity of the audio signal during the digitization process. At the same time, according to the real-time monitoring of the ambient noise level, the microphone gain is dynamically adjusted to ensure that the signal quality is effectively improved in a low signal-to-noise ratio environment to avoid signal saturation or loss of details.

[0042] The methods used to collect data samples include; Low signal-to-noise ratio environments are the core challenge for noise reduction algorithms, so it is necessary to construct training data specifically for low signal-to-noise ratio scenarios while ensuring coverage of dialect diversity.

[0043] Record the user's dialect voice data in a low signal-to-noise ratio environment. For example, record the voice data of dialect users in noisy environments (such as streets, factories, and subway stations) to ensure that the data contains different types of noise (stationary noise, non-stationary noise, and sudden noise). Collect voice data of major dialects (such as Cantonese, Wu, Minnan, Sichuanese, and Northeastern dialect). The recorded voice data is uploaded voluntarily by smart device users (subject to privacy protection regulations). Enrich the dialect samples in low signal-to-noise ratio environments and build a dialect voice database covering low signal-to-noise ratio environments (-10dB to 5dB); The architecture of the preset conditional adversarial network cGAN includes a generator and a discriminator. The speech data in the dialect speech database is input into the generator. A scene evolution layer, a sound change layer and a noise mixing layer are added to the generator. The generator and the discriminator are trained in turn by alternating training. When the training is completed, the discriminator is fixed and the generator is used to generate new low signal-to-noise ratio dialect speech data. The generated data is used to expand the existing dialect speech database to form data samples. The goal of the above processing is to build a generative model that can generate low signal-to-noise ratio dialect speech data with specific scene noise and sound change characteristics, so as to expand the existing dialect speech data set and improve the performance of tasks such as speech recognition and speech enhancement in low signal-to-noise ratio environments. Conditional Generative Adversarial Network (cGAN), cGAN is a special GAN ​​that adds conditional information to the input of both the generator and the discriminator. In this scheme, the conditional information includes scene labels and sound change parameters. Deep learning models such as convolutional neural networks (CNN), recurrent neural networks (RNN, such as LSTM, GRU) or Transformer can be used to build generators and discriminators.

[0044] When speech data is input into the generator, the generator is designed with an input layer for receiving pure dialect speech signals (which can be represented as STFT spectrograms, MFCC features or other acoustic features), and a speech encoder for encoding the input speech signals into a low-dimensional latent representation, which can use structures such as CNN, RNN or Transformer.

[0045] In the model training phase, an alternating training method is adopted to train the generator and the discriminator in turn. For the generator loss, an adversarial loss is used to encourage the generator to generate more realistic samples, making it difficult for the discriminator to distinguish. Wasserstein distance, least squares loss, etc. can be used. Other loss functions can also be introduced, such as: cycle consistency loss (to ensure that the speech generated by the generator can reconstruct the original pure speech after the discriminator and generator cycle), perceptual loss (using a pre-trained speech recognition model to extract features and calculate the distance between the generated speech and the real speech in the feature space). Discriminator loss is used to encourage the discriminator to distinguish between real samples and generated samples more accurately. Use Adam, SGD or other commonly used optimization algorithms. You can preset a validation set and use the validation set to adjust the model's hyperparameters, such as learning rate, batch size, signal-to-noise ratio range, sound change parameters, etc.

[0046] A three-dimensional blank dataset is set up, where the three dimensions include multi-granularity acoustic features, noise features and semantic features. The multi-granularity acoustic features include the phoneme level, pronunciation method, rhythmic features and timbre of the speech data. The noise features include noise events, noise components and time-frequency distribution. The semantic features include word-level annotations, sentence-level annotations and paragraph-level annotations. The three-dimensional data of each speech data in the data sample is extracted, and the data is filled into the blank dataset to form a three-dimensional annotation for each speech data.

[0047] Among them, phoneme-level annotation not only marks the tone, but also the initial consonant and final consonant of each syllable (as well as the initial, middle and final consonants of the final consonant, if the dialect has a distinction). This is crucial for capturing the subtle pronunciation differences of dialects. Pronunciation mode annotation is used to mark the pronunciation mode of phonemes (such as aspirated / unaspirated, voiceless / voiced, stop / fricative / nasal, etc.). This is particularly useful for distinguishing special sound changes in dialects (such as the voiceless and unvoiced opposition in Wu dialect). Prosodic feature annotation is used to mark prosodic information such as stress, intonation, pauses, etc. This information is more easily interfered with under low signal-to-noise ratio, but it is important for semantic understanding and speech naturalness. The refinement of timbre annotation can use more professional phonetic terms to describe timbre, such as "bright", "deep", "hoarse", "nasal", etc., and try to keep the annotation consistent.

[0048] Noise event annotation: For sudden noise (such as car horns and knocking sounds), its start time, end time, type (such as "short horn", "continuous horn"), source (such as car), etc. Noise component analysis: For steady-state noise, in addition to the type (such as "white noise", "pink noise"), its main components (such as "mainly human voice" and "mainly traffic noise") are also annotated. Time-frequency distribution refinement: More sophisticated time-frequency analysis tools (such as wavelet transform) can be used to describe the time-frequency distribution of noise.

[0049] Word-level annotation is the basic annotation of speech recognition results. Sentence-level annotation is to annotate the intention and emotion of the sentence. Chapter-level annotation can annotate the topic and turn of the conversation.

[0050] Add scene evolution layer, sound change layer and noise mixing layer to the generator, the method includes: The input of the scene evolution layer is the preset scene label, which is processed using a one-hot vector or an embedded vector. The scene label can be manually marked without duplication according to different scenes. According to the scene label, the corresponding noise fragment is extracted from the dialect speech database, and the scene speech is output by random mixing. Random mixing includes single scene isolation and multi-scene superposition. Single scene isolation is to directly output the selected noise fragment, and multi-scene superposition is to linearly or nonlinearly superimpose the noise fragments of multiple scenes according to the preset weight distribution mechanism for each scene; the mixed scene speech has the same dimension and length as the input speech data.

[0051] The probability distribution mechanism means that a weight value range is preset for each scenario, and the sum of the weights is less than or equal to 1 when multiple scenarios are superimposed; Specifically, we build a dataset containing noises from various scenes, such as street noise (cars, crowds, horns), restaurant noise (conversation sounds, clashes of cutlery), and office noise (keyboard sounds, air conditioner sounds, phone rings). We randomly select one or more scenes, assign a weight to each scene (to simulate the intensity of different scenes), and add the noises from multiple scenes according to the weights. We can also randomly crop, change the speed, and change the pitch of the noise samples to increase the diversity of the noise.

[0052] The input of the sound change layer is the output of the speech encoder and the sound change control parameters. According to the phonetic characteristics of the dialect, the sound change rules are defined, such as consonant weakening, vowel nasalization, tone drift, etc. Specifically, for slurred speech, the simulated consonant pronunciation is unclear, for example, / t / is pronounced as / d / , / k / Pronounce as / g / , for swallowing, simulate the omission of certain syllables or phonemes, for example, pronounce "we" as "I", for connected reading, simulate the unclear transition between syllables, and for tone change, simulate the slight shift or instability of dialect tones, so as to form a rule-based sound change model; use dialect speech data with sound change annotations to train a sequence-to-sequence model (for example, an encoder-decoder model based on an attention mechanism) for predicting and generating sound changes, so as to form a data-driven sound change model; combine the rule-based sound change model with the data-driven sound change model, control the degree and type of sound change by adjusting sound change parameters (for example, sound change probability, sound change intensity, etc.), and output a sequence of speech features that have undergone sound change, recorded as sound-changed speech; The input of the noise mixing layer is the scene speech and the sound-changed speech, which are mixed according to the specified signal-to-noise ratio. Additive mixing, convolution mixing or other mixing methods can be used to output a speech feature sequence with noise and sound changes, which is recorded as the new low signal-to-noise ratio dialect speech data.

[0053] Design a noise reduction algorithm specifically for low signal-to-noise ratio environments, focusing on solving the problems of speech distortion, sound quality degradation, and misprocessing of dialect features. The algorithm uses a deep learning framework combined with adaptive strategies and multi-task learning.

[0054] A pre-built dialect speech database solves the problem of insufficient dialect processing in traditional noise reduction systems. The three-dimensional annotation system (acoustic, noise, semantic) provides comprehensive feature protection guidance. The feature embedding layer integrates dialect label information into the noise reduction process to enhance the ability to retain dialect features. The scene evolution layer, sound change layer and noise mixing layer in the cGAN architecture form a complete data enhancement pipeline. The single-scene isolation and multi-scene superposition mechanism simulates the real usage environment. The combination of rule-based and data-driven sound change models comprehensively covers the dialect speech change characteristics.

[0055] The design of the noise reduction algorithm also includes: Design a noise reduction algorithm based on an end-to-end speech enhancement model, including input, output, and multi-task learning mechanisms. Set the input as the time-frequency spectrum of noisy speech. The multi-task learning mechanism uses a deep learning structure (such as Deep Complex U-Net, Transformer, Conformer, CRNN, etc.) for time-frequency processing, combined with a generative adversarial network GAN (such as SEGAN, MetricGAN, etc., which are specially designed for speech enhancement) for speech reconstruction, and outputs a time-frequency spectrum of pure speech after noise reduction. The input noisy speech is transformed into a time-frequency spectrum through short-time Fourier transform (STFT), and the output clean speech time-frequency spectrum after noise reduction is reconstructed into a time domain signal through inverse STFT to form the noise-reduced speech data; The end-to-end model can directly map the time-frequency spectrum of noisy speech to the time-frequency spectrum of clean speech, avoiding the error accumulation between multiple modules in traditional methods. Using deep learning for time-frequency processing and combining GAN for speech reconstruction can effectively learn complex noise patterns and generate high-quality speech. Adding a lightweight SNR estimator helps the model to adaptively process speech with different signal-to-noise ratios. Introducing auxiliary noise reduction technology to protect dialect characteristics can improve the performance of the model.

[0056] The loss function is designed to be the weighted sum of denoising loss, speech reconstruction loss, dialect feature loss and naturalness loss. The denoising loss is used to measure the denoising effect in a low signal-to-noise ratio environment. The mean square error (MSE) or signal log spectral distance (LSD) can be used as the loss function of the denoising task. The speech reconstruction loss is used to evaluate the perceptual quality of the denoised speech data. Specifically, the perceptual loss is introduced and the pre-trained speech recognition model (such as HuBERT or Wav2Vec) is used to evaluate the perceptual quality of the denoised speech data. The dialect feature loss is used to measure the model's damage to tone and timbre. Specifically, a special tone loss and timbre loss are designed. For example, the output of the tone classifier is used to calculate the tone consistency loss. The naturalness loss is used to measure the naturalness of the speech data after noise reduction. Specifically, the discriminator of the generative adversarial network is used to optimize the naturalness of the enhanced speech to avoid "robotic voice" or "hollowness". A loss priority rule is added, that is, two thresholds are preset to meet the requirement that the speech reconstruction loss is greater than the threshold and the dialect feature loss is less than the threshold. The goal is to increase the weights of the speech reconstruction loss and the dialect feature loss in a low signal-to-noise ratio environment to ensure the priority retention of the speech signal.

[0057] The above loss calculation can be based on the content of three-dimensional annotation. For example, the noise reduction loss is used to measure the noise reduction effect in a low signal-to-noise ratio environment. The noise events, noise components and time-frequency distribution in the noise features can be used as comprehensive considerations. For example, the phoneme level, pronunciation method, rhythmic features and timbre in the multi-granular acoustic features can be used to determine whether the denoised speech data is still consistent with the input, and the dialect features have not changed. Loss calculation based on three-dimensional annotation (noise features, multi-granular acoustic features) can more finely evaluate model performance.

[0058] Use data samples to pre-train the denoising algorithm until the loss function value meets the requirements or the number of iterations is reached. Then, the pre-trained denoising algorithm is obtained and deployed at the edge layer to process real-time voice data.

[0059] An innovative architecture that combines a shared encoder with three tasks (R1, R2, and R3) achieves a balance between noise reduction and feature retention. Task R1 effectively suppresses noise through the IRM and cIRM estimators of the U-Net structure. Task R2 focuses on retaining dialect features to avoid filtering of key speech features. Task R3 uses GAN technology to reconstruct speech details and improve naturalness.

[0060] The multi-task learning mechanism uses a deep learning structure for time-frequency processing and combines it with the generative adversarial network GAN for speech reconstruction. The methods include: Set up a shared encoder to extract deep feature representations of the spectrogram. These features should be able to capture the common information of speech and noise, and provide a processing basis for subsequent tasks R1, R2, and R3. Use a deep learning structure, such as a multi-layer convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer layer, or a combination thereof (such as Conformer). Output the encoded feature map; The preset task-specific headers include Task R1, Task R2, and Task R3. Task R1 is used for noise suppression. It removes background noise by learning time-frequency masks, focusing on optimizing the mask accuracy in low signal-to-noise ratio environments. A lightweight signal-to-noise ratio estimator (such as a CNN or RNN-based model) is added after the shared encoder in Task R1 and before the IRM estimator to evaluate the signal-to-noise ratio level of the input speech. Specifically, it includes: Task R1 designs an IRM estimator based on U-Net, whose input is the feature map output by the shared encoder. The structure includes an encoder, a decoder, and an output layer. The encoder is similar or identical to the shared encoder (the number of layers can be reduced) and contains multiple downsampling modules (e.g., convolution layer + pooling layer), which gradually reduces the spatial resolution of the feature map while increasing the number of channels. The decoder is symmetrical to the encoder and contains multiple upsampling modules (e.g., transposed convolution layer or upsampling layer + convolution layer), which gradually restores the spatial resolution of the feature map while reducing the number of channels. The encoder and the decoder are skip-connected. The skip connection is to concatenate or add the output feature map of each downsampling module in the encoder with the input feature map of the corresponding upsampling module in the decoder, which helps to pass low-level detail information to the decoder and improve the mask accuracy. The output layer is set to a 1x1 convolution layer, and the sigmoid activation function is used to map the feature map to the IRM mask, where the size of the output IRM mask is the same as the input STFT spectrum map; The loss function is mean square error or binary cross entropy. The IRM model is trained using noisy speech and clean speech data until convergence. The output is the trained IRM model (denoted as IRM_Model) and the predicted IRM mask.

[0061] The output feature map of the shared encoder and the predicted IRM mask are used as the input of the cIRM model. The cIRM estimator is designed based on U-Net, and its structure is similar to that of the IRM model. Specifically, the encoder and decoder are similar to the IRM model, but the encoder may need some adjustments (for example, increasing the number of channels) to adapt to the complex characteristics of cIRM, and the output layer of the decoder needs to predict the real and imaginary parts of the complex mask. The encoder and decoder are jump-connected, and the output layer structure is similar to that of the IRM model, that is, it is usually a 1x1 convolution layer followed by a Sigmoid activation function to map the feature map to the IRM mask. The size of the output IRM mask is the same as the input STFT spectrogram, but IRM fusion is added after the jump connection. The fusion method is to concatenate the predicted IRM with the output feature map of the encoder as the input of the cIRM encoder. In each upsampling module of the decoder, the predicted IRM is fused with the upsampled feature map (for example, the fusion can be an attention mechanism, a weighted average), and the function is to predict the complex ideal ratio mask (cIRM), which contains real and imaginary parts: cIRM = (real part)cIRM_real + j (Imaginary part) cIRM_imag, j represents a number, set the loss function, the loss function can use the MSE loss in the complex domain, or calculate the MSE loss of the real part and the imaginary part separately, and then sum them up weightedly. During the training process, the weights of the trained IRM model can be used to initialize the weights of the cIRM model (except the output layer), and then the cIRM model can be fine-tuned using noisy speech and clean speech data, and the output is the predicted cIRM mask (including the real part and the imaginary part); The predicted cIRM mask (complex number) is applied to the STFT spectrum of the noisy speech to obtain the STFT spectrum of the denoised speech; the cIRM model processes both amplitude and phase information, which is superior to the traditional method that only processes amplitude.

[0062] A lightweight SNR estimator (such as a CNN or RNN-based model) is added after the shared encoder in Task R1 and before the IRM estimator to evaluate the SNR level of the input speech. This mainly solves three key problems of traditional speech noise reduction systems in dealing with low SNR environments: 1. Traditional noise reduction systems focus only on the amplitude spectrum and ignore phase information, resulting in a decrease in the quality of reconstructed speech; 2. In a high-noise environment, the traditional IRM mask is not accurate enough to accurately distinguish between noise and speech; 3. There is a lack of evaluation mechanism for the noise level of the input signal, and the noise reduction strategy is not flexible enough.

[0063] After the shared encoder and before the IRM estimator is the best location for the following reasons: The shared encoder has extracted deep features of the input speech, which are more suitable for SNR estimation than the original STFT spectrograms because they have been abstracted and denoised to a certain extent.

[0064] The estimated SNR can be used as prior information to guide the IRM estimator to work better. For example, for speech with low SNR, the IRM estimator can perform noise suppression more aggressively; for speech with high SNR, the IRM estimator can perform noise suppression more conservatively to avoid damaging the speech.

[0065] Adding an SNR estimator before the IRM estimator can avoid SNR estimation for each time-frequency unit, thereby improving computational efficiency.

[0066] Including: setting the input of the signal-to-noise ratio estimator to the feature map output by the shared encoder, the structure includes global average pooling, a fully connected layer, an activation function and an output layer, the output layer uses a single-output fully connected layer, and the output is the estimated SNR value (in decibels); The feature map output by the shared encoder is globally averaged pooled in the time and frequency dimensions, and the feature map is converted into a vector. Each element of the vector represents the average activation value of a channel. The purpose is to compress the spatial information into a global feature vector, which is representative of the SNR.

[0067] Use one or two fully connected layers (also called dense layers) to process the vector output by global average pooling. The first fully connected layer can map the feature vector to a lower dimensional space (for example, 64 or 128 units). If two fully connected layers are used, the second fully connected layer can further map the features to a lower dimensional space (for example, 16 or 32 units). The role of the fully connected layer is to learn a nonlinear mapping from the global average pooling features to the SNR value.

[0068] Use an activation function after each fully connected layer, such as ReLU (Rectified Linear Unit) or LeakyReLU. The role of the activation function is to introduce nonlinearity and enhance the expressiveness of the model.

[0069] The loss function is set to mean square error to calculate the mean square error between the estimated SNR value and the true SNR value. The STFT spectrogram of the noisy speech is input into the shared encoder, and the output feature map of the shared encoder is input into the signal-to-noise ratio estimator to calculate the MSE loss between the estimated SNR value and the true SNR value. The weight of the signal-to-noise ratio estimator is updated using the back propagation algorithm. In the IRM estimator, the estimated SNR value is input as an additional input into the IRM estimator together with the original input.

[0070] The double-layer mask architecture is designed. First, a real-valued mask is generated through the IRM estimator to provide basic information for the subsequent cIRM. Then, a complex mask is generated through the cIRM estimator, and amplitude and phase information are processed at the same time. The cascade processing of the two-layer mask improves the noise reduction accuracy and speech quality. The predicted IRM mask is fused with the encoder feature map to provide prior knowledge for cIRM. The IRM information is fused at each layer of the decoder to guide the cIRM generation process. Fine fusion is achieved through the attention mechanism or weighted averaging to improve the mask quality.

[0071] The encoder-decoder symmetric structure effectively extracts and reconstructs time-frequency features. The skip connection mechanism retains low-level detail information and avoids the loss of important features. The lightweight signal-to-noise ratio estimator design reduces computational overhead. The shared encoder architecture saves model parameters and computing resources.

[0072] Task R2 is used to preserve dialect features. By introducing supervisory information of dialect features, key features (such as tone, timbre, etc.) are retained and not filtered out.

[0073] It includes feature embedding layer, tone protection module, timbre protection module and semantic-assisted noise reduction module; The input of the feature embedding layer is (original) noisy speech data + dialect label (for example, "Cantonese" or an integer ID representing Cantonese); the dialect label is converted into a vector of fixed dimension (for example, using one-hot encoding or learning an embedding matrix), which is recorded as the dialect embedding vector; this dialect embedding vector is concatenated with the speech feature vector (for example, MFCC, Mel spectrum) or fused through the attention mechanism to form a new vector, which will be used as the input of the subsequent denoising model. The purpose is to make the model aware that it is currently processing a specific dialect (for example, Cantonese), so as to adjust its denoising strategy and avoid misprocessing the unique features of Cantonese.

[0074] The input of the tone protection module is (raw) noisy speech data, and a pre-trained dialect tone classifier is used (for example, trained on a large amount of Cantonese data on an LSTM or Transformer-based model). The output is the tone category probability distribution (for example, the probability of the 9 Cantonese tones) and tone contour features (for example, fundamental frequency F0 and its changes) for each time frame.

[0075] During the denoising process, the tone contour features of the output speech are constrained by a loss function (e.g., mean square error (MSE) can be used) so that the similarity with the tone contour features of the tone protection module meets the requirements (e.g., 95% similarity is required to ensure that the performance is as consistent as possible). The purpose is to ensure that the spectral features and change patterns of the 9 tones of the dialect (Cantonese) are preserved after denoising.

[0076] The input of the timbre protection module is (original) noisy speech data. A pre-trained timbre feature extractor (e.g., an autoencoder based on WaveNet, or a speaker recognition model pre-trained on a large amount of speech data) is used to output a timbre feature vector of fixed dimension (e.g., i-vector, x-vector). An item is added to the loss function of the denoising process to penalize the difference between the timbre feature vectors before and after denoising. For example, cosine similarity or Euclidean distance can be used as part of the loss function. A similarity requirement can also be set. If the timbre difference exceeds a threshold, the penalty is increased. This prevents the denoising process from changing the timbre characteristics of the speech and maintains the speaker's personalized characteristics.

[0077] The input of semantic-assisted denoising is (raw) noisy speech data. Using a pre-trained dialect speech recognition model (e.g., Wav2Vec 2.0, Conformer), the output is a text transcription of the speech (possibly with a timestamp), phoneme / syllable-level information (e.g., syllable boundaries, tone information). Based on the speech recognition results, the tone changes (identifying which syllables have tone changes (e.g., "chicken" and "machine")), syllable boundaries (determining the start and end time of each syllable) and key phonemes (identifying phonemes that are important for semantic understanding (e.g., stops and fricatives in Cantonese)) are extracted. Based on the recognition results, they are protected separately. For example, for tone change protection, the role of the tone protection module can be enhanced near the syllable boundaries of the tone changes to ensure that the tone differences are not smoothed out. For syllable boundary protection, the denoising intensity can be reduced near the syllable boundaries to prevent the transition between syllables from being destroyed. For key phoneme protection, the retention of these phoneme features can be enhanced in the time-frequency region corresponding to the key phonemes.

[0078] By using semantic information, the noise reduction process can be more finely controlled to ensure that speech components important for semantic expression are not filtered out. Through multi-level and multi-angle constraints, it is ensured that the intonation, timbre and semantic characteristics of Cantonese are retained to the greatest extent while noise reduction is being performed. It can effectively improve the quality and intelligibility of dialect (Cantonese) speech noise reduction in low signal-to-noise ratio environments.

[0079] Task R3 is used for speech reconstruction, which reconstructs speech details (such as intonation and harmonics) by generating adversarial networks to avoid the "empty feeling".

[0080] The generator of the generative adversarial network is used to reconstruct the harmonics, intonation and fast-changing pronunciation details of the speech, and the discriminator is used to enhance the naturalness of the speech; The dialect features output by Task R2 are used as a guide to ensure that the enhanced speech is as close to the sound quality of the real dialect as possible. For example, the fundamental frequency (F0) estimator is used to extract the fundamental frequency information of the speech, and the fundamental frequency is forcibly retained during the reconstruction process to avoid tone loss. The formant analysis is used to enhance the timbre characteristics of the speech to avoid timbre changes.

[0081] In a low signal-to-noise ratio environment, the fundamental frequency and resonance peak characteristics of the speech are enhanced first to avoid the "hollow feeling" while reducing excessive filtering of high-frequency noise.

[0082] Example 2, please refer to Figure 1 and Figure 3 As shown, the part not described in detail in this embodiment is described in Example 1, which provides a method for voice noise reduction of an intelligent communication device based on artificial intelligence, including: Step S1: deploying a microphone array on the intelligent communication device to collect the user's real-time noisy voice data; Step S2: designing the input, output and multi-task learning mechanism of the noise reduction algorithm based on the end-to-end speech enhancement model; Step S3: Use data samples for pre-training, obtain the trained denoising algorithm, and deploy it to the edge layer; Step S4: input the collected noisy speech data into the edge layer, and output the noise-reduced speech data through real-time processing.

[0083] Through the innovative multi-task learning architecture and dialect feature protection mechanism, the limitations of traditional noise reduction systems in processing dialects are effectively solved, and the voice quality and user experience of smart communication devices in complex noise environments are significantly improved.

[0084] Embodiment 3. This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the above-mentioned artificial intelligence-based intelligent communication device voice noise reduction system is implemented.

[0085] Since the electronic device introduced in this embodiment is an electronic device used to implement a voice noise reduction system for an intelligent communication device based on artificial intelligence in the embodiment of this application, based on the voice noise reduction system for an intelligent communication device based on artificial intelligence introduced in the embodiment of this application, the technical personnel of this field can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not introduced in detail here. As long as the technical personnel of this field implement the electronic device used by the voice noise reduction system for an intelligent communication device based on artificial intelligence in the embodiment of this application, it belongs to the scope of protection of this application.

[0086] The above formulas are all dimensionless and numerical calculations. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0087] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A voice noise reduction system for intelligent communication equipment based on artificial intelligence, characterized in that: Including intelligent communication equipment, on which a microphone array is deployed to collect real-time noisy voice data of users; The recorded noisy speech data is input into the edge layer, where a pre-trained noise reduction algorithm is deployed. The noise-reduced speech data is obtained through real-time processing and output. The hierarchical deployment of the noise reduction algorithm specifically includes: pre-building a dialect speech database, adding a scene evolution layer, a sound change layer, and a noise mixing layer in the conditional adversarial network cGAN to expand the dialect speech database, form data samples, and perform three-dimensional annotation on each speech data in the data sample; Based on the end-to-end speech enhancement model, the input, output and multi-task learning mechanisms are designed, and data samples are used for pre-training. The multi-task learning mechanism combines deep learning structure with generative adversarial network GAN for speech reconstruction. A shared encoder is set in the multi-task learning mechanism for joint tasks R1, R2 and R3. Task R1 designs an IRM estimator based on U-Net, and uses the IRM estimator to train the cIRM model for noise suppression, and adds a lightweight signal-to-noise ratio estimator; Task R2 jointly supervises the retention of dialect features based on the feature embedding layer, tone protection module, timbre protection module and semantic-assisted denoising module; Task R3 reconstructs speech through a generative adversarial network.

2. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 1, characterized in that: The method of deploying a microphone array on an intelligent communication device to collect user voice data includes: According to the size of the intelligent communication device and the computing resources of the edge layer, a suitable array and spacing are selected. The microphone has a high sampling rate and a wide frequency range, is equipped with an analog-to-digital converter with a high dynamic range, and is connected to the edge layer of the intelligent communication device using an interface.

3. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 2, characterized in that: The data sample collection method includes: Record the user's dialect voice data in a low signal-to-noise ratio environment and build a dialect voice database covering the low signal-to-noise ratio environment; The architecture of the conditional adversarial network cGAN includes a generator and a discriminator. The speech data in the dialect speech database is input into the generator. The scene evolution layer, sound change layer and noise mixing layer are added to the generator. The generator and the discriminator are trained in turn by alternating training. After the training is completed, the discriminator is fixed and the generator is used to generate new low signal-to-noise ratio dialect speech data. The generated data is used to expand the existing dialect speech database to form data samples. A three-dimensional blank dataset is set up, where the three dimensions include multi-granularity acoustic features, noise features and semantic features. The multi-granularity acoustic features include the phoneme level, pronunciation method, rhythmic features and timbre of the speech data. The noise features include noise events, noise components and time-frequency distribution. The semantic features include word-level annotations, sentence-level annotations and paragraph-level annotations. The three-dimensional data of each speech data in the data sample is extracted, and the data is filled into the blank dataset to form a three-dimensional annotation for each speech data.

4. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 3 is characterized in that: The method of adding a scene evolution layer, a sound change layer and a noise mixing layer in the generator includes: The input of the scene evolution layer is the preset scene label. According to the scene label, the corresponding noise fragment is extracted from the dialect speech database, and the scene speech is output through random mixing. Random mixing includes single scene isolation and multi-scene superposition. Single scene isolation is to directly output the selected noise fragment, and multi-scene superposition is to linearly or nonlinearly superimpose the noise fragments of multiple scenes according to the preset weight distribution mechanism of each scene. The probability distribution mechanism means that a weight value range is preset for each scenario, and the sum of the weights is less than or equal to 1 when multiple scenarios are superimposed; The input of the sound change layer is the output of the speech encoder and the sound change control parameters. According to the phonetic characteristics of the dialect, the sound change rules are defined to form a rule-based sound change model. Using the dialect speech data with sound change annotations, a sequence-to-sequence model is trained to predict and generate sound changes, forming a data-driven sound change model. The rule-based sound change model and the data-driven sound change model are combined to control the degree and type of sound change by adjusting the sound change parameters, and the sound-changed speech feature sequence is output, which is recorded as the sound-changed speech. The input of the noise mixing layer is the scene speech and the sound-changed speech, which are mixed according to the specified signal-to-noise ratio, and the speech feature sequence with noise and sound changes is output, which is recorded as the new low signal-to-noise ratio dialect speech data.

5. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 4, characterized in that: The design of the noise reduction algorithm also includes: Design a noise reduction algorithm based on an end-to-end speech enhancement model, including input, output, and multi-task learning mechanism. Set the input as the time-frequency spectrum of noisy speech. The multi-task learning mechanism uses a deep learning structure for time-frequency processing and combines it with a generative adversarial network (GAN) for speech reconstruction. The output is a time-frequency spectrum of pure speech after noise reduction. The input noisy speech is transformed into a time-frequency spectrum through short-time Fourier transform (STFT), and the output clean speech time-frequency spectrum after noise reduction is reconstructed into a time domain signal through inverse STFT to form the noise-reduced speech data; The loss function is designed to be the weighted sum of noise reduction loss, speech reconstruction loss, dialect feature loss and naturalness loss. The noise reduction loss is used to measure the noise reduction effect in a low signal-to-noise ratio environment. The speech reconstruction loss is used to evaluate the perceptual quality of the speech data after noise reduction. The dialect feature loss is used to measure the damage of the model to the tone and timbre. The naturalness loss is used to measure the naturalness of the speech data after noise reduction. A loss priority rule is added, that is, two thresholds are preset to meet the requirement that the speech reconstruction loss is greater than the threshold and the dialect feature loss is less than the threshold. Use data samples to pre-train the denoising algorithm until the loss function value meets the requirements or the number of iterations is reached. Then, the pre-trained denoising algorithm is obtained and deployed at the edge layer to process real-time voice data.

6. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 5, characterized in that: The multi-task learning mechanism adopts a deep learning structure to perform time-frequency processing, and the method of combining a generative adversarial network (GAN) to perform speech reconstruction includes: Set up a shared encoder to extract deep feature representation of the spectrogram, use a deep learning structure, and output the encoded feature map; The preset task-specific headers include Task R1, Task R2, and Task R3; Task R1 is used for noise suppression. A lightweight signal-to-noise ratio estimator is added after the shared encoder and before the IRM estimator in Task R1 to evaluate the signal-to-noise ratio level of the input speech. Task R2 is used to preserve dialect features. By introducing the supervision information of dialect features, key features are retained and not filtered out. Task R3 is used for speech reconstruction. It reconstructs the details of speech through a generative adversarial network, including using the generator of the generative adversarial network to reconstruct the harmonics, tones, and rapidly changing pronunciation details of the speech, and the discriminator enhances the naturalness of the speech. The dialect features output by Task R2 are introduced as a guide.

7. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 6, characterized in that: The method for noise suppression in task R1 includes: Task R1 designs an IRM estimator based on U-Net, whose input is the feature map output by the shared encoder. The structure includes an encoder, a decoder, and an output layer. The encoder is similar or identical to the shared encoder and contains multiple downsampling modules. The decoder is symmetrical to the encoder and contains multiple upsampling modules. The encoder and the decoder are jump-connected. The output layer is set to a 1x1 convolutional layer. The Sigmoid activation function is used to map the feature map to the IRM mask. The size of the output IRM mask is the same as the input STFT spectrum map. The loss function is mean square error or binary cross entropy. The IRM model is trained using noisy speech and clean speech data until convergence. The output is the trained IRM model and the predicted IRM mask. The output feature map of the shared encoder and the predicted IRM mask are used as the input of the cIRM model. The cIRM estimator is designed based on U-Net, and its structure is similar to the IRM model, but IRM fusion is added after the jump connection. The fusion method is to splice the predicted IRM with the output feature map of the encoder as the input of the cIRM encoder. In each upsampling module of the decoder, the predicted IRM is fused with the upsampled feature map. The cIRM contains real and imaginary parts. The loss function is set and the output is the predicted cIRM mask. The predicted cIRM mask is applied to the STFT spectrum of the noisy speech to obtain the STFT spectrum of the denoised speech.

8. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 7, characterized in that: A lightweight signal-to-noise ratio estimator is added after the shared encoder in the task R1 and before the IRM estimator to evaluate the signal-to-noise ratio level of the input speech, including: The input of the signal-to-noise ratio estimator is set to the feature map output by the shared encoder. The structure includes global average pooling, a fully connected layer, an activation function, and an output layer. The output layer uses a single-output fully connected layer, and the output is the estimated SNR value. The loss function is set to mean square error to calculate the mean square error between the estimated SNR value and the true SNR value. The STFT spectrogram of the noisy speech is input into the shared encoder, and the output feature map of the shared encoder is input into the signal-to-noise ratio estimator to calculate the MSE loss between the estimated SNR value and the true SNR value. The weight of the signal-to-noise ratio estimator is updated using the back propagation algorithm. In the IRM estimator, the estimated SNR value is input as an additional input into the IRM estimator together with the original input.

9. The voice noise reduction system for intelligent communication equipment based on artificial intelligence according to claim 8, characterized in that: The task R2 is used to retain dialect features, by introducing the supervision information of dialect features, retaining the key features from being filtered out. include: It includes feature embedding layer, tone protection module, timbre protection module and semantic-assisted noise reduction module; The input of the feature embedding layer is noisy speech data + dialect label; the dialect label is converted into a vector of fixed dimension, recorded as the dialect embedding vector; this dialect embedding vector is concatenated with the speech feature vector or fused through the attention mechanism to form a new vector, which will be used as the input of the subsequent noise reduction algorithm; The input of the tone protection module is noisy speech data. It uses a pre-trained dialect tone classifier and outputs the tone category probability distribution and tone contour features for each time frame. In the process of noise reduction, the tone contour features of the output speech are constrained by the loss function so that the similarity between them and the tone contour features of the tone protection module meets the requirements; The input of the timbre protection module is noisy speech data. The pre-trained timbre feature extractor is used to output a timbre feature vector of fixed dimension. A term is added to the loss function of the denoising process to penalize the difference between the timbre feature vectors before and after denoising. The input of semantic-assisted denoising is noisy speech data. Using a pre-trained dialect speech recognition model, the output is text transcription of the speech and information at the phoneme / syllable level. Based on the speech recognition results, tone changes, syllable boundaries and key phonemes are extracted; and protection is performed separately based on the recognition results.

10. A method for voice noise reduction of intelligent communication equipment based on artificial intelligence, applied to the voice noise reduction system of intelligent communication equipment based on artificial intelligence according to any one of claims 1 to 9, characterized in that: The method for reducing voice noise of intelligent communication equipment based on artificial intelligence includes: Step S1: deploying a microphone array on the intelligent communication device to collect the user's real-time noisy voice data; Step S2: designing the input, output and multi-task learning mechanism of the noise reduction algorithm based on the end-to-end speech enhancement model; Step S3: Use data samples for pre-training, obtain the trained denoising algorithm, and deploy it to the edge layer; Step S4: input the collected noisy speech data into the edge layer, and output the noise-reduced speech data through real-time processing.

Images