Ultra-lightweight speech enhancement neural network and method based on network architecture search

Abstract

The application discloses a kind of based on network architecture search's super-lightweight speech enhancement neural network and method.The neural network includes encoder, enhancer and decoder, and the encoder includes: spectrum compression module, for the compressed spectrum of noisy speech in frequency dimension;Down-sampling convolution module is used for feature extraction and frequency dimension down-sampling to feature map;Grouped depthwise separable convolution module is used for feature extraction and feature enhancement to feature map;Enhancer includes: two-way grouped recurrent neural network, for time-frequency domain modeling to feature map;The decoder includes: grouped depthwise separable convolution module, for feature extraction and feature enhancement to feature map;Up-sampling convolution module is used for feature reconstruction and frequency dimension up-sampling to feature map;Spectrum expansion module is used for the expansion of frequency dimension to feature map.The application can realize high-performance speech enhancement with lower computational cost overhead.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of deep learning and speech enhancement, and particularly relates to a super-lightweight speech enhancement neural network system based on network architecture search and a method thereof. BACKGROUND

[0002] In daily life, speech signals are inevitably contaminated by various kinds of interference, such as background noise, echo and reverberation. Speech enhancement aims to effectively recover speech signals from interference such as noise, and has important applications in audio-visual systems and intelligent speech interaction. In recent years, deep learning models have been widely used in speech enhancement, which learns the nonlinear mapping relationship from mixed speech to clean speech directly through data-driven way, and has achieved much better performance than traditional methods in many scenarios. However, the current state-of-the-art speech enhancement neural network has a problem of high memory and computing resource consumption, which makes it difficult to deploy on end-side devices which are widely used in applications. In the scene where computing resources are limited and real-time requirements are high, lightweight neural networks are needed. Typical lightweight design strategies include depthwise separable convolution, grouped convolution and channel shuffle, and lightweight attention mechanism, etc. Neural network architecture search technology is also a lightweight network design strategy.

[0003] Lightweight neural network design in the field of speech enhancement is attracting more and more attention from academia and industry. The literature (H.-S. Choi, S. Park, J. H. Lee, H. Heo, D. Jeon, and K. Lee, "Real-time denoising and dereverberation with tiny recurrent u-net," in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021: IEEE, pp. 5789-5793.) combines strategies such as depth separable convolution to optimize the convolution recurrent neural network, achieving comparable enhancement performance to large models while significantly reducing parameter and computational complexity. However, there is still room for optimization in terms of model complexity and enhancement performance. The literature (Lei Yang, Wei Liu, Ruijie Meng, Gunwoo Lee, Soonho Baek, and Han-Gil Moon, "Fspen: an ultra-lightweight network for real-time speech enhancement," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10671-10675.) proposes sub-band segmentation coding and path expansion modeling to improve the modeling ability of lightweight models, with a computational complexity as low as 89 MMACs.Document (Xiaobin Rong, Tianchi Sun, Xu Zhang, Yuxiang Hu, Changbao Zhu, and Jing Lu,“Gtcrn: A speech enhancement model requiring ultralow computational resources,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 971-975.) proposes to use a double-branch grouped convolution structure to reduce the computational amount of the model, and also uses sub-band feature extraction and timing cycle attention, etc. Plug-and-play modules, effectively improve the performance of the model without increasing the additional computational amount of the model. The model achieves quite good speech enhancement performance with as low as 39 MMACs. However, the performance of these ultra-lightweight speech enhancement models still needs to be further improved.

[0004] Different from the way of manual design, neural architecture search (NAS for short) is a technical means that can automatically design network models. The main goal of NAS is to automatically search for the most ideal network architecture in the predefined search space by using a specific search algorithm. Although NAS has shown excellent performance in image classification, object detection, and semantic segmentation, it has received little attention in the field of speech enhancement. The literature (J.-H. Lee, J.-H. Chang, J.-M. Yang, and H.-G. Moon, “Nas-tasnet: Neural architecture search for time-domain speech separation,” IEEE Access, vol. 10, pp. 56031-56043, 2022.) explores the application of NAS on a classic speech separation neural network Conv-TasNet, achieving higher performance with fewer parameters. However, the final network obtained still has relatively high complexity and does not belong to ultra-lightweight network. Another literature (A. Vafeiadis, N. van de Waterlaat, C. Castel, B. Defraene, G. Daalderop, S. Vogel, and C. Macours, “Ultra-low memory speech denoising using quantization-aware neural architecture search,” in 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1-6) proposes a quantization-aware NAS technology to design a speech enhancement neural network with extremely low resource consumption. However, its design space is based on a simple convolutional recurrent encoder-decoder architecture family, so the final speech enhancement is only at a general level. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application proposes an ultra-lightweight speech enhancement neural network and method based on network architecture search, which can achieve high-performance speech enhancement with low computational overhead.

[0006] The technical scheme of the neural network of the present application is as follows:

[0007] The network architecture search-based super-lightweight speech enhancement neural network comprises an encoder, an enhancer and a decoder, the encoder comprises: a spectrum compression module for compressing the frequency dimension of the noisy speech spectrum; a down-sampling convolution module for feature extraction and frequency dimension down-sampling of the feature map; and a grouped depthwise separable convolution module for feature extraction and feature enhancement of the feature map.

[0008] The enhancer comprises a two-way grouped recurrent neural network for time-frequency domain modeling of the feature map.

[0009] The decoder comprises a grouped depthwise separable convolution module for feature extraction and feature enhancement of the feature map; an up-sampling convolution module for feature reconstruction and frequency dimension up-sampling of the feature map; and a spectrum expansion module for frequency dimension expansion of the feature map.

[0010] The application further provides a design method of the network architecture search-based super-lightweight speech enhancement neural network, and specifically comprises the following steps:

[0011] S1, constructing a modularized network architecture search space;

[0012] S2, sampling in the search space using a controller network;

[0013] S3, evaluating the network architecture obtained by sampling in step S2 on a data set to obtain a speech quality index;

[0014] S4, updating the controller network in a reinforcement learning manner by taking the speech quality index and the network architecture calculation amount as rewards;

[0015] S5, repeating steps S2, S3 and S4 for several rounds until the sampling result converges;

[0016] S6, applying the searched best neural network architecture to a speech enhancement task.

[0017] Further, step S1 specifically comprises the following sub-steps:

[0018] S11, constructing a network architecture prototype in a manner that the encoder, the enhancer and the decoder are sequentially connected;

[0019] S12, generating a modularized local search space based on the internal structure of the encoder, comprising: the type of the convolution module, the channel number of the convolution module and the convolution kernel size of the convolution module;

[0020] S13, determining the architecture of the enhancer and the decoder according to the architecture of the encoder;

[0021] S14, constructing a complete network, so that each network architecture corresponds to a serialized representation.

[0022] Further, the step S2 specifically comprises the following sub-steps:

[0023] S21, design a controller network based on long short-term memory unit, the output of each unit constitutes a hidden state of the serialized representation of the network architecture in the search space;

[0024] S22, the complete sequence representation output by the controller network is deserialized to obtain the searched network architecture.

[0025] Further, the step S4 specifically comprises the following sub-steps:

[0026] S41, the speech quality index and the network architecture calculation amount are integrated by a multi-objective fusion method to obtain a reward;

[0027] S42, the gradient is estimated and the parameters of the controller network are updated using the proximal policy optimization algorithm.

[0028] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0029] (1) The present application first designs a higher-performance ultra-lightweight neural network prototype: group convolution and group recurrent neural network are used to greatly reduce the parameter quantity and calculation amount of the network; dilated convolution is used to expand the receptive field in the time dimension and improve the network's ability to capture long-range dependencies in time series; a series of nonlinear activation functions are used to improve the network's nonlinear expression ability; a lightweight polynomial attention module is used to improve the network's ability to capture feature map energy distribution.

[0030] (2) The present application uses the above-mentioned neural network as a prototype and applies NAS technology on a finely designed search space, thereby successfully achieving more outstanding speech enhancement performance.

[0031] (3) The calculation amount of the system of the present application is equivalent to that of the current optimal ultra-lightweight speech enhancement neural network, but it can achieve higher performance. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The structure diagram of the ultra-lightweight speech enhancement neural network in the embodiment of the present application;

[0033] Figure 2 The structure diagram of the polynomial attention module in the embodiment of the present application;

[0034] Figure 3 The structure diagram of the down-sampling convolution module in the embodiment of the present application;

[0035] Figure 4 The structure diagram of the up-sampling convolution module in the embodiment of the present application;

[0036] Figure 5 This is a structural diagram of the grouped depth separation convolution module in an embodiment of the present invention;

[0037] Figure 6 This is a flowchart of the reinforcement learning-based architecture search algorithm of the present invention;

[0038] Figure 7 This is a diagram of the controller network structure in an embodiment of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will be fully and clearly described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0040] The structural diagram of the ultra-lightweight speech enhancement neural network system designed in this invention is shown below. Figure 1 As shown, the system includes a spectrum compression module for compressing the frequency spectrum of noisy speech; a downsampling convolution module for feature extraction and frequency downsampling of the feature map; a grouped deep separation convolution module for feature extraction and enhancement; a grouped recurrent neural network module for time-frequency domain modeling of the feature map; an upsampling convolution module for feature reconstruction and frequency upsampling of the feature map; and a spectrum expansion module for expanding the frequency dimension of the feature map. The spectrum compression module, downsampling convolution module, and grouped deep separation convolution module form the encoder; the grouped recurrent neural network module forms the enhancer; and the grouped deep separation convolution module, upsampling convolution module, and spectrum expansion module form the decoder. A skip connection is added between the encoder and decoder. The encoder transmits its output to the enhancer, where the dual-path grouped recurrent neural network models the feature map in the time-frequency domain and then outputs the result to the decoder.

[0041] The system is obtained by applying a network architecture search technology on the basis of a neural network prototype. The neural network prototype is designed as follows according to experience: the encoder is sequentially connected by one spectrum compression module, two down-sampling convolution modules and three grouped deep separation convolution modules; the enhancer is connected by two grouped recurrent neural network modules; the decoder is the mirror image of the encoder and is sequentially connected by three grouped deep separation convolution modules, two up-sampling convolution modules and one spectrum expansion module; and the jump connection is added between the encoder and the decoder. After the network architecture search technology is applied in the application, the connection relationship in the encoder is improved to spectrum compression module-down-sampling convolution module-grouped deep separation convolution module-down-sampling convolution module-grouped deep separation convolution module-grouped deep separation convolution module, and the connection relationship in the decoder is improved to grouped deep separation convolution module-grouped deep separation convolution module-up-sampling convolution module-grouped deep separation convolution module-up-sampling convolution module-spectrum expansion module.

[0042] In the embodiment, the spectrum compression module and the spectrum expansion module keep each frequency point in the 0kHz-2kHz frequency band unchanged, and use an equivalent rectangular bandwidth filter bank to realize feature transformation on each frequency point in the 2kHz-8kHz frequency band.

[0043] The down-sampling convolution module includes a two-dimensional grouped convolution layer, a batch normalization layer, a step non-linear activation function and a polynomial attention module. As shown in Figure 3 In the embodiment, the down-sampling convolution module is composed of a batch normalization layer and a step non-linear activation layer added after the two-dimensional grouped convolution layer, wherein the two-dimensional grouped convolution layer has a grouping number of 2, a frequency stride of 2, and a proper padding is set according to the size of the convolution kernel, so that the feature map realizes 2 times down-sampling in the frequency dimension after passing through the convolution layer.

[0044] In the embodiment, the step non-linear activation function is composed of a learnable affine transformation and a ReLU activation function with learnable scaling coefficients and learnable bias coefficients of multiple terms. Assuming that the input feature map is wherein is a real set, and C, T and F represent the sizes of the feature dimension, the time domain dimension and the frequency domain dimension, respectively. The scaling coefficient and the bias coefficient of the affine transformation are defined as and The affine transformation is

[0045] A(X)=a⊙X+b

[0046] wherein is an element-wise point multiplication. The scaling coefficient and the bias coefficient of the polynomial non-linear activation are defined as and wherein N is a positive integer representing the number of non-linear activation terms. The output of the step non-linear activation function is

[0047]

[0048] Where the coefficients a, b, γ i ,β i All of these are learnable parameters that are updated with gradients during training.

[0049] In this embodiment, the polynomial attention module is as follows: Figure 2 As shown, the system consists of pooling layers, a first-order attention module, and a second-order attention module. The first-order attention module includes a gated recurrent unit layer, a fully connected layer, and a sigmoid activation function; the second-order attention module consists of two fully connected layers. The input feature map of dimension C×T×F is pooled in the frequency dimension using a pooling layer to achieve squared average pooling, adjusting the dimension to C×T×1. The first-order attention module then provides the attention mask value, which is multiplied by the input feature map to obtain the first-order output. The second-order attention module's first fully connected layer is followed by a ReLU6 activation function, multiplied by itself, and then passed through a second fully connected layer to obtain the second-order output. The first-order and second-order outputs are weighted and summed using a learnable coefficient to obtain the final output.

[0050] In this embodiment, the upsampling convolution module is as follows: Figure 4 As shown, it consists of a two-dimensional grouped transposed convolutional layer followed by a batch normalization layer and a series nonlinear activation layer. The stride of the two-dimensional grouped transposed convolutional layer in terms of frequency is set to 2, and appropriate padding is set according to the size of the convolutional kernel, so that the feature map achieves upsampling of the frequency dimension by 2 times after passing through the convolutional layer.

[0051] In this embodiment, the grouping depth separable convolution module is as follows: Figure 5 As shown, it includes a two-dimensional grouped pointwise convolutional layer, a first batch normalization layer (BN), a first-level nonlinear activation function, a depthwise convolutional layer, a second batch normalization layer (BN), a second-level nonlinear activation function, a multinomial attention module, and a channel shuffling operation module. The two-dimensional grouped pointwise convolutional layer refers to a two-dimensional convolutional layer with 2 groups and a kernel size of 1x1; the depthwise convolutional layer refers to a two-dimensional convolutional layer with the number of groups equal to the number of channels.

[0052] In this embodiment, the grouped recurrent neural network module consists of two recurrent neural networks. After the input feature map dimension is adjusted to F×T×C, it is split in two along the feature dimension. The two recurrent neural networks are then used in parallel to model the temporal relationship. Finally, the two recurrent neural networks are concatenated along the channel dimension and the channels are shuffled before output.

[0053] In this embodiment, the output of the spectral expansion module of the speech enhancement neural network is passed through the Sigmoid activation function to obtain the spectral amplitude masking value between (0,1).

[0054] The application also provides a method for designing an ultra-light voice enhancement neural network system based on a network architecture search technology, as shown in the following steps. Figure 6

[0055] S1, constructing a modular network architecture search space; specifically including the following sub-steps:

[0056] S11, constructing a network architecture prototype: constructing a network architecture prototype in the connection mode of "encoder-enhancer-decoder", and adding additive skip connections between the encoder and the decoder to improve the modeling ability and training stability of the network;

[0057] S12, generating a modular local search space based on the internal structure of the encoder: the internal structure of the encoder is composed of two convolution modules, namely down-sampling convolution modules and grouped depth separable convolution modules, wherein the frequency dimension of the output feature map of the down-sampling convolution module is half of the input feature map; the frequency dimension of the output feature map of the grouped depth separable convolution module is the same as the size of the input feature map.

[0058] The encoder contains 5 convolution modules, and the selection of the convolution modules generates a search space, including: the type of the convolution module, the channel number of the convolution module and the convolution kernel size of the convolution module. Specifically, the type of the convolution module includes two optional items of down-sampling convolution module and grouped depth separable convolution module; the channel number of the convolution module includes seven optional items of 12, 16, 24, 28, 32 and 36; the convolution kernel size of the convolution module includes three optional items of (1, 5), (2, 5) and (3, 3);

[0059] S13, determining the architecture of the enhancer and the decoder according to the architecture of the encoder;

[0060] S14, constructing a complete network, so that each network architecture corresponds to a serialized representation, that is, each network structure in the search space has a unique representation, such as using different numerical strings to represent the network architecture.

[0061] According to the method of steps S11-S14, a modular search space based on the internal structure of the encoder can be constructed.

[0062] S2, sampling using a controller network; specifically including the following sub-steps:

[0063] S21, designing a controller network based on a long short-term memory unit, as shown in the following figure: Figure 7 ​The number of units of the controller network and the number of nodes of the search space are the same. Each long short-term memory unit outputs a one-hot encoding representation of a structure parameter at each time, and after passing through an embedding layer, it is used as the input of the next long short-term memory unit. This process is repeated until the one-hot encoding representation of the structure parameter of the last node in the search space is output. The outputs of all units form a serialized representation of the network architecture in the search space;

[0064] S22, the complete sequence representation output by the controller is deserialized to obtain the searched network architecture.

[0065] S3, evaluate the sampled network architecture on the dataset to obtain a speech quality indicator; specifically including the following sub-steps:

[0066] S31, train the searched network architecture on the speech enhancement training dataset, and stop after a certain number of training rounds;

[0067] S32, evaluate the trained network architecture on the speech enhancement verification dataset, and take the average of the speech quality evaluation indicators (PESQ) on the verification set in the last several rounds as the final quality indicator.

[0068] S4, update the controller network in a reinforcement learning manner by taking the speech quality evaluation indicator and the network architecture computation as the reward; specifically including the following sub-steps:

[0069] S41, use a reinforcement learning algorithm to construct a network architecture search algorithm: set the one-hot representation of the structure parameter in the network architecture search space as the action output by the controller network, and set the one-hot representation of the structure parameter generated by the controller as the state of the reinforcement learning. The reward is obtained by integrating the speech quality evaluation indicator and the neural network computation in a multi-objective fusion manner. Assuming that the speech quality indicator is P, the neural network computation is M, and the expected size of the computation is M t , the reward of reinforcement learning is defined as Where λ<0 is a hyperparameter that balances the speech quality and neural network computation;

[0070] S42, estimate the gradient and update the parameters of the controller network using the proximal policy optimization algorithm.

[0071] S5, repeat steps S2, S3, and S4 for several rounds until the sampling result converges, and obtain the optimal neural network architecture.

[0072] S6, apply the optimal neural network architecture to the speech enhancement task; specifically including the following sub-steps:

[0073] S61, perform short-time Fourier transform on the speech to be enhanced to obtain the speech spectrum, and then take the logarithmic amplitude to obtain the input feature of the neural network;

[0074] S62, the input feature is passed through the neural network to obtain a speech spectrum mask value consistent with the input shape, and the mask value is multiplied element by element with the amplitude of the speech spectrum to obtain an enhanced speech amplitude spectrum;

[0075] S63, the enhanced speech amplitude spectrum is combined with the phase spectrum of the original speech to obtain an enhanced speech complex spectrum, and an inverse short-time Fourier transform is performed to obtain an enhanced speech.

[0076] The specific settings of the embodiment are as follows: the speech enhancement object is audio data with a sampling rate of 16 kHz, and the audio data is converted into time-frequency spectrum data after STFT, and the logarithmic amplitude spectrum is taken as the input of the network. The frame length and frame shift of STFT are set to 512 (32 ms) and 256 (16 ms) respectively, and the window function is a square-root Hanning window. The frequency spectrum compression module keeps 65 frequency bands within 0 kHz-2 kHz unchanged, and maps 192 frequency bands within 2 kHz-8 kHz into 64 frequency bands.

[0077] The settings of the network architecture search are as follows: the search space includes 5 convolution modules in the encoder, the structure parameters of each module include the type of the convolution module, the channel number of the convolution module and the convolution kernel size of the convolution module, so there are 15 search nodes in total. Specifically, the type of the convolution module includes two optional items of down-sampling convolution module and grouped depthwise separable convolution module; the channel number of the convolution module includes seven optional items of 12, 16, 24, 28, 32, 36; and the convolution kernel size of the convolution module includes three optional items of (1, 5), (2, 5) and (3, 3). The controller network of the network architecture search contains 15 long short-term memory units in total, the hidden layer unit is 100, and the embedding layer unit is 32. After each search of a network architecture, 30 rounds of training are performed on the speech enhancement training data set, and the average value of the perceptual evaluation quality (PESQ) of the speech on the verification machine in the last 3 rounds is taken as the speech quality index. When calculating the reward, the expected calculation amount is set to 30 MMACs, and λ is set to -0.06 to balance the speech quality and the calculation amount of the neural network.

[0078] The number of network architectures sampled by the embodiment each time is 32, a total of 60 rounds are searched, and a total of 1920 network architectures are searched. The network structure with the best performance finally searched is as follows:

[0079] Table 1 Network architecture search results

[0080] Structural parameters First convolution module Second convolution module Third convolution module Fourth convolution module Fifth convolution module Type Downsampling Grouped depth separation Downsampling Grouped depth separation Grouped depth separation Channel number 12 16 12 32 16 Convolution kernel size (2,5) (1,5) (1,5) (2,5) (3,3)

[0081] According to the above parameter settings, the parameter amount of the neural network is 124K, and the calculation amount per second is 32MMACs.

[0082] To demonstrate the effect of the present application, experiments were conducted on a simulated data set generated using the DNS5 database, with GTCRN (Xiaobin Rong, Tianchi Sun, Xu Zhang, Yuxiang Hu, Changbao Zhu, and Jing Lu, "Gtcrn: A speech enhancement model requiring ultralow computational resources," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 971-975.) selected as the baseline model for comparison. The evaluation indicators of enhancement performance include scale-invariant signal-to-noise ratio (SISNR), perceptual evaluation of speech quality (PESQ), and non-intrusive speech quality evaluation indicators DNSMOS, including signal preservation quality SIG, noise suppression quality BAK, and overall quality OVRL, all of which are higher the better. The comparison results are shown in Table 2.

[0083] Table 2 Comparison results

[0084]

[0085]

[0086] From the results in Table 2, it is proved that the neural network system of the present application can achieve better enhancement performance with less computational amount than GTCRN. It is worth noting that although the parameter amount of the system of the present application is significantly increased compared with GTCRN, in actual application deployment, this degree of increase in parameter amount will hardly bring any negative impact. Therefore, the present application can realize high-performance speech enhancement with lower parameter amount and computational amount overhead.

Claims

1. An ultra-lightweight speech enhancement device based on network architecture search neural network, comprising an encoder, an enhancer and a decoder, characterized in that, The encoder comprises: a spectrum compression module for compressing the noisy speech spectrum in the frequency dimension; a down-sampling convolution module for feature extraction and down-sampling in the frequency dimension; and a grouped depthwise separable convolution module for feature extraction and feature enhancement. The enhancer comprises: a two-path grouped recurrent neural network for time-frequency domain modeling of the feature map. The decoder comprises: a grouped depthwise separable convolution module for feature extraction and feature enhancement; an up-sampling convolution module for feature reconstruction and up-sampling in the frequency dimension; and a spectrum expansion module for expanding the frequency dimension of the feature map. The specific structure of the encoder comprises, sequentially connected, a spectrum compression module, a down-sampling convolution module, a grouped depthwise separable convolution module, a down-sampling convolution module, a grouped depthwise separable convolution module, and a grouped depthwise separable convolution module; and the specific structure of the decoder comprises, sequentially connected, a grouped depthwise separable convolution module, a grouped depthwise separable convolution module, an up-sampling convolution module, a grouped depthwise separable convolution module, an up-sampling convolution module, and a spectrum expansion module. The down-sampling convolution module comprises a two-dimensional convolution layer, a batch normalization layer, a polynomial non-linear activation function, and a polynomial attention module, wherein the two-dimensional convolution layer realizes down-sampling in the frequency dimension of the feature map with a stride of 2.

2. The ultra-lightweight speech enhancement device based on network architecture search neural network according to claim 1, characterized in that, The polynomial non-linear activation function comprises a sum of a learnable affine transformation and a plurality of ReLU activation functions, and each ReLU activation function has a learnable scaling coefficient and a bias coefficient.

3. The ultra-lightweight speech enhancement device based on network architecture search neural network according to claim 1, characterized in that, The polynomial attention module comprises a two-order attention module, wherein the first-order attention module comprises a pooling layer, a gated recurrent unit layer, and a full connection layer; the second-order attention module comprises two full connection layers; the input feature map obtains an attention mask value through the first-order attention module, and is multiplied with the input feature map to obtain a first-order output; the input feature map is multiplied with itself after passing through the first layer full connection layer in the second-order attention module and passing through an activation function, and then passes through the second layer full connection layer to obtain a second-order output; the first-order output and the second-order output are weighted and summed with a learnable coefficient to obtain a final output.

4. The ultra-lightweight speech enhancement device based on network architecture search neural network according to claim 1, characterized in that, The grouped depthwise separable convolution module comprises a point-by-point convolution layer with a grouping number of 2, a first batch normalization layer, a first polynomial non-linear activation function, a depthwise convolution layer, a second batch normalization layer, a second polynomial non-linear activation function, a polynomial attention module, and a channel shuffle operation module.

5. The method of claim 1, wherein the network architecture search neural network based design method of an ultra-lightweight speech enhancement device is characterized by, Specifically comprising the following steps: S1, constructing a modular network architecture search space; S2, sampling in the search space using a controller network; S3, evaluating the network architecture obtained by sampling in step S2 on a data set to obtain a speech quality index; S4, updating the controller network in a reinforcement learning manner by taking the speech quality index and the network architecture calculation amount as rewards; S5, repeating steps S2, S3, and S4 for several rounds until the sampling result converges; S6, applying the best neural network architecture searched to a speech enhancement task.

6. The method of designing according to claim 5, wherein, The step S1 specifically comprises the following sub-steps: S11, constructing a network architecture prototype in a manner that the encoder, the enhancer and the decoder are connected in sequence; S12, generating a modular local search space based on the internal structure of the encoder, including: the type of convolution module, the number of channels of the convolution module and the size of the convolution kernel of the convolution module; S13, determining the architecture of the enhancer and the decoder according to the architecture of the encoder; S14, constructing a complete network so that each network architecture corresponds to a serialized representation.

7. The method of designing according to claim 6, wherein, The step S2 specifically includes the following sub-steps: S21, designing a controller network based on a long short-term memory unit, and the output of each unit constitutes a hidden state of the serialized representation of the network architecture in the search space; S22, de-serializing the complete sequence representation output by the controller network to obtain the searched network architecture.

8. The method of designing according to claim 7, wherein, The step S4 specifically includes the following sub-steps: S41, synthesizing the speech quality index and the network architecture calculation amount by means of multi-target fusion to obtain a reward; S42, estimating the gradient and updating the parameters of the controller network by using a proximal policy optimization algorithm.

Images