Ultra-lightweight speech enhancement neural network and method based on network architecture search

Abstract

The invention discloses an ultra-lightweight speech enhancement neural network and method based on network architecture search. The neural network comprises an encoder, an enhancer and a decoder, and the encoder comprises a frequency spectrum compression module used for compressing a noisy speech frequency spectrum in a frequency dimension; the down-sampling convolution module is used for performing feature extraction and frequency dimension down-sampling on the feature map; the grouping depth separation convolution module is used for performing feature extraction and feature enhancement on the feature map; the intensifier comprises a two-way packet recurrent neural network which is used for carrying out time-frequency domain modeling on a feature map; the decoder comprises a grouping depth separation convolution module which is used for performing feature extraction and feature enhancement on the feature map; the up-sampling convolution module is used for performing feature reconstruction and frequency dimension up-sampling on the feature map; and the frequency spectrum expansion module is used for expanding the frequency dimension of the feature map. According to the invention, high-performance speech enhancement can be realized with relatively low calculation overhead.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and speech enhancement, and specifically relates to an ultra-lightweight speech enhancement neural network system and method based on network architecture search. Background Art

[0002] In daily life, speech signals are always inevitably contaminated by various interferences, such as background noise, echo, and reverberation. Speech enhancement aims to effectively restore speech signals from interferences such as noise, and has important applications in audio and video systems and intelligent voice interaction. In recent years, deep learning models have been widely used in speech enhancement. Through data-driven methods, deep neural networks can directly learn the nonlinear mapping relationship from mixed speech to clean speech, and have achieved performance far exceeding that of traditional methods in many scenarios. However, the most advanced speech enhancement neural networks currently have problems with excessive memory and computing resource consumption, which makes them difficult to deploy on widely used end-side devices. In scenarios where computing resources are limited and real-time requirements are high, lightweight neural networks are needed. Typical lightweight design strategies include depthwise separable convolution, grouped convolution and channel shuffle, and lightweight attention mechanisms. Neural network architecture search technology is also a lightweight network design strategy.

[0003] The design of lightweight neural networks in the field of speech enhancement has attracted more and more attention from academia and industry. The literature (H.-S. Choi, S. Park, J. H. Lee, H. Heo, D. Jeon, and K. Lee, "Real-time denoising and dereverberation with tiny recurrent u-net," in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021: IEEE, pp. 5789-5793.) combines strategies such as deep separable convolution to optimize convolutional recurrent neural networks, achieving enhancement performance comparable to that of large models while significantly reducing the number of parameters and computational complexity. However, there is room for further optimization of the network in terms of both model complexity and enhancement performance. The literature (Lei Yang, Wei Liu, Ruijie Meng, Gunwoo Lee, SoonhoBaek, and Han-Gil Moon, "Fspen: an ultra-lightweight network for real-time speech enhancement," in ICASSP2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10671-10675.) proposed modeling methods such as subband splitting coding and path extension to improve the modeling capability of the lightweight model. The computational complexity of the proposed model is as low as 89 MMACs.The paper (Xiaobin Rong, Tianchi Sun, Xu Zhang, Yuxiang Hu, Changbao Zhu, and Jing Lu, "Gtcrn: A speech enhancement model requiring ultralow computational resources," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 971-975.) proposed to use a dual-branch grouped convolution structure to reduce the amount of computation of the model. It also used plug-and-play modules such as sub-band feature extraction and temporal recurrent attention to effectively improve the performance of the model without increasing the amount of additional computation of the model. The model achieved quite good speech enhancement performance with a computational cost as low as 39 MMACs. However, the performance of these ultra-lightweight speech enhancement models needs to be further improved.

[0004] Different from the manual design method, neural network architecture search (NAS for short) is a technical means to automatically design network models. The main goal of NAS is to automatically search for the most ideal network architecture in a pre-defined search space with a specific search algorithm. Although NAS has shown excellent performance in fields such as image classification, object detection, and semantic segmentation, it has received little attention in the field of speech enhancement. The literature (J.-H.Lee, J.-H.Chang, J.-M.Yang, and H.-G.Moon, "Nas-tasnet: Neural architecture search for time-domain speech separation," IEEE Access, vol. 10, pp. 56 031-56 043, 2022.) explored the application of NAS on a classic speech separation neural network Conv-TasNet, achieving higher performance with fewer parameters. However, the final network still has a very high complexity and is not an ultra-lightweight network. Another paper (A. Vafeiadis, N. van de Waterlaat, C. Castel, B. Defraene, G. Daalderop, S. Vogel, and C. Macours, "Ultra-low memory speech denoising using quantization-aware neural architecture search," in 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1-6) proposes a quantization-aware NAS technology, which aims to design a speech enhancement neural network with extremely low resource consumption. However, its design space is based on a simple convolutional cyclic encoder-decoder architecture family, so the speech enhancement achieved is only average. Summary of the invention

[0005] In view of the deficiencies in the above-mentioned prior art, the present invention proposes an ultra-lightweight speech enhancement neural network and method based on network architecture search, which can achieve high-performance speech enhancement with low computational overhead.

[0006] The technical solution adopted by the neural network of the present invention is:

[0007] An ultra-lightweight speech enhancement neural network based on network architecture search, including an encoder, an enhancer and a decoder, wherein the encoder includes: a spectrum compression module for compressing the noisy speech spectrum in the frequency dimension; a downsampling convolution module for extracting features from feature maps and downsampling the frequency dimension; a grouped depth separation convolution module for extracting features from feature maps and enhancing features;

[0008] The enhancer includes: a two-way grouping recurrent neural network for modeling the feature map in the time and frequency domain;

[0009] The decoder includes: a grouped depth separation convolution module for extracting and enhancing features of a feature map; an upsampling convolution module for reconstructing features of a feature map and upsampling the frequency dimension; and a spectrum expansion module for expanding the frequency dimension of the feature map.

[0010] The present invention also provides a design method for the ultra-lightweight speech enhancement neural network based on network architecture search, which specifically includes the following steps:

[0011] S1, building a modular network architecture search space;

[0012] S2, uses the controller network to sample in the search space;

[0013] S3, evaluating the network architecture sampled in step S2 on the data set to obtain a speech quality index;

[0014] S4, combining the voice quality index and the network architecture calculation amount as a reward to update the controller network in a reinforcement learning manner;

[0015] S5, repeat steps S2, S3, and S4 for several rounds until the sampling results converge;

[0016] S6, applies the searched optimal neural network architecture to the speech enhancement task.

[0017] Furthermore, the step S1 specifically includes the following sub-steps:

[0018] S11, construct a network architecture prototype by connecting the encoder, enhancer and decoder in sequence;

[0019] S12, generating a modular local search space based on the internal structure of the encoder, including: the type of convolution module, the number of channels of the convolution module, and the convolution kernel size of the convolution module;

[0020] S13, determining the architecture of the enhancer and the decoder according to the architecture of the encoder;

[0021] S14, build a complete network so that each network architecture corresponds to a serialized representation.

[0022] Furthermore, the step S2 specifically includes the following sub-steps:

[0023] S21, design a controller network based on long short-term memory units, where the output of each unit constitutes a hidden state of the sequential representation of the network architecture in the search space;

[0024] S22, deserialize the complete sequence representation output by the controller network to obtain the searched network architecture.

[0025] Furthermore, the step S4 specifically includes the following sub-steps:

[0026] S41, rewards are given by integrating voice quality indicators and network architecture calculations through multi-objective fusion;

[0027] S42, estimates the gradient and updates the parameters of the controller network using a proximal policy optimization algorithm.

[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0029] (1) The present invention first designs a higher-performance ultra-lightweight neural network prototype: grouped convolution and grouped recurrent neural networks are used to significantly reduce the number of network parameters and computational complexity; dilated convolution is used to expand the receptive field in the time dimension and improve the network's ability to capture long-range temporal dependencies; a series of nonlinear activation functions is used to improve the network's nonlinear expression capabilities; and a lightweight polynomial attention module is used to improve the network's ability to capture the energy distribution of feature maps.

[0030] (2) The present invention uses the above-mentioned neural network as a prototype and applies NAS technology on a carefully designed search space, thereby successfully achieving more outstanding speech enhancement performance.

[0031] (3) The computational complexity of the system of the present invention is comparable to that of the currently optimal ultra-lightweight speech enhancement neural network, but it can achieve higher performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a structural diagram of an ultra-lightweight speech enhancement neural network in an embodiment of the present invention;

[0033] Figure 2 This is a structural diagram of a polynomial attention module in an embodiment of the present invention;

[0034] Figure 3 1 is a structural diagram of a downsampling convolution module in an embodiment of the present invention;

[0035] Figure 4 Schematic diagram of the upsampling convolution module in an embodiment of the present invention;

[0036] Figure 5 It is a structural diagram of a grouped depth separation convolution module in an embodiment of the present invention;

[0037] Figure 6 This is a flow chart of the architecture search algorithm based on reinforcement learning of the present invention;

[0038] Figure 7 4 is a diagram of a controller network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical scheme of the present invention is described in detail and clearly in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The structure diagram of an ultra-lightweight speech enhancement neural network system designed by the present invention is as follows: Figure 1 As shown, the system includes a spectrum compression module for compressing the noisy speech spectrum in the frequency dimension; a downsampling convolution module for extracting features from the feature map and downsampling the frequency dimension; a grouped deep separation convolution module for extracting features from the feature map and enhancing the features; a grouped recurrent neural network module for modeling the feature map in the time-frequency domain; an upsampling convolution module for reconstructing features from the feature map and upsampling the frequency dimension; and a spectrum expansion module for expanding the frequency dimension of the feature map. The spectrum compression module, the downsampling convolution module and the grouped deep separation convolution module form an encoder, the grouped recurrent neural network module forms an enhancer, the grouped deep separation convolution module, the upsampling convolution module and the spectrum expansion module form a decoder, and a jump connection is added between the encoder and the decoder. The encoder transmits the output result to the enhancer, and the dual-path grouped recurrent neural network of the enhancer models the feature map in the time-frequency domain, and then outputs the result to the decoder.

[0041] The above system is obtained by applying the network architecture search technology on the basis of the neural network prototype. The neural network prototype is designed empirically as follows: the encoder is composed of 1 spectrum compression module, 2 downsampling convolution modules and 3 grouped depth separation convolution modules connected in sequence; the enhancer is composed of 2 grouped recurrent neural network modules connected in sequence; the decoder is a mirror image of the encoder, composed of 3 grouped depth separation convolution modules, 2 upsampling convolution modules and 1 spectrum expansion module connected in sequence; at the same time, a jump connection is added between the encoder and the decoder. After the network architecture search technology is applied in the present invention, the connection relationship in the encoder is improved to spectrum compression module-downsampling convolution module-grouped depth separation convolution module-downsampling convolution module-grouped depth separation convolution module-grouped depth separation convolution module, and the connection relationship in the decoder is improved to grouped depth separation convolution module-grouped depth separation convolution module-upsampling convolution module-grouped depth separation convolution module-upsampling convolution module-spectrum expansion module.

[0042] In this embodiment, the spectrum compression module and the spectrum expansion module keep the frequency points in the 0kHz-2kHz frequency band unchanged, and use an equivalent rectangular bandwidth filter group to achieve feature transformation for the frequency points in the 2kHz-8kHz frequency band.

[0043] The downsampling convolution module includes a two-dimensional grouped convolution layer, a batch normalization layer, a series of nonlinear activation functions, and a polynomial attention module. Figure 3 As shown, the downsampling convolution module in this embodiment is composed of a batch normalization layer and a series nonlinear activation layer added after a two-dimensional grouped convolution layer, wherein the number of groups in the two-dimensional grouped convolution layer is 2, the stride in frequency is set to 2, and appropriate padding is set according to the size of the convolution kernel, so that the feature map can be downsampled by 2 times the frequency dimension after passing through the convolution layer.

[0044] In this embodiment, the series nonlinear activation function consists of a learnable affine transformation and multiple ReLU activation functions with learnable scaling coefficients and learnable bias coefficients. Assume that the input feature map is in is a set of real numbers, C, T, F represent the size of feature dimension, time domain dimension and frequency domain dimension respectively. The scaling coefficient and bias coefficient of affine transformation are defined as and Then the affine transformation is

[0045] A(X)=a⊙X+b

[0046] Where ⊙ is the element-by-element dot product. The scaling coefficients and bias coefficients for multiple nonlinear activations are defined as and Where N is a positive integer representing the number of nonlinear activation items. Then the output of the series nonlinear activation function is

[0047]

[0048] The coefficients a, b, and γ i ,β i These are all learnable parameters and are updated with gradients during training.

[0049] In this embodiment, the polynomial attention module is as follows Figure 2 As shown in the figure, it consists of a pooling layer, a first-order attention module, and a second-order attention module. The first-order attention module includes a gated recurrent unit layer, a fully connected layer, and a Sigmoid activation function; the second-order attention module includes two fully connected layers. The input feature map with a dimension of C×T×F is subjected to square average pooling of energy in the frequency dimension through a pooling layer, and the dimension is adjusted to C×T×1. The attention mask value is obtained through the first-order attention module and multiplied with the input feature map to obtain the first-order output; after passing through the first fully connected layer in the second-order attention module, it is multiplied by itself after passing through the ReLU6 activation function, and then passed through the second fully connected layer to obtain the second-order output. The first-order output and the second-order output are weighted and summed with a learnable coefficient to obtain the final output.

[0050] In this embodiment, the upsampling convolution module is as follows Figure 4 As shown in the figure, it consists of a two-dimensional grouped transposed convolution layer followed by a batch normalization layer and a series nonlinear activation layer, where the stride of the two-dimensional grouped transposed convolution layer in frequency is set to 2, and appropriate padding is set according to the size of the convolution kernel, so that the feature map can be upsampled by 2 times in the frequency dimension after passing through the convolution layer.

[0051] In this embodiment, the grouped depth-separable convolution module is as follows: Figure 5 As shown, it includes a two-dimensional grouped point-by-point convolution layer, a first batch normalization layer BN, a first-order nonlinear activation function, a depth convolution layer, a second batch normalization layer BN, a second-order nonlinear activation function, a polynomial attention module, and a channel shuffle operation module. The two-dimensional grouped point-by-point convolution layer refers to a two-dimensional convolution layer with a group number of 2 and a convolution kernel size of 1x1; the depth convolution layer refers to a two-dimensional convolution layer with a group number equal to the number of channels.

[0052] In this embodiment, the grouped recurrent neural network module is composed of two recurrent neural networks. After the input feature map dimension is adjusted to F×T×C, it is divided into two in the feature dimension, and after the temporal relationship is modeled in parallel through two recurrent neural networks, it is spliced ​​together in the channel dimension and output after channel shuffling.

[0053] In this embodiment, the output of the spectrum expansion module of the speech enhancement neural network is passed through the Sigmoid activation function to obtain a spectrum amplitude masking value output between (0, 1).

[0054] The present invention also provides a method for designing an ultra-lightweight speech enhancement neural network system based on network architecture search technology, such as Figure 6 As shown, the following steps are included:

[0055] S1, building a modular network architecture search space; specifically includes the following sub-steps:

[0056] S11, build a network architecture prototype: build a network architecture prototype with the connection mode of "encoder-booster-decoder", and add additive jump connections between the encoder and decoder to improve the modeling ability and training stability of the network;

[0057] S12, generates a modular local search space based on the internal structure of the encoder: the encoder is composed of two convolution modules, a downsampling convolution module and a grouped depthwise separable convolution module. The frequency dimension of the output feature map of the downsampling convolution module is half of the input feature map; the frequency dimension of the output feature map of the grouped depthwise separable convolution module is the same as the size of the input feature map.

[0058] The encoder contains 5 convolution modules, and the selection of convolution modules generates a search space, including: the type of convolution module, the number of channels of the convolution module, and the convolution kernel size of the convolution module. Specifically, the type of convolution module includes two options: downsampling convolution module and grouped depth-separable convolution module; the number of channels of the convolution module includes seven options: 12, 16, 24, 28, 32, and 36; the convolution kernel size of the convolution module includes three options: (1,5), (2,5), and (3,3);

[0059] S13, determining the architecture of the enhancer and the decoder according to the architecture of the encoder;

[0060] S14, build a complete network so that each network architecture corresponds to a serialized representation, that is, each network structure in the search space has a unique representation, such as using different digital strings to represent the network architecture.

[0061] According to the method of steps S11-S14 above, a modular search space based on the internal structure of the encoder can be constructed.

[0062] S2, sampling using the controller network; specifically includes the following sub-steps:

[0063] S21, design a controller network based on long short-term memory units, such as Figure 7As shown. The number of units in the controller network is the same as the number of nodes in the search space. At each moment, the long short-term memory unit outputs a unique hot encoding representation of the structural parameters, and after passing through the embedding layer, it serves as the input of the long short-term memory unit at the next moment. This cycle repeats until the structural parameter representation of the last node in the search space is output. The outputs of all units constitute the serialized representation of the network architecture in the search space;

[0064] S22, deserialize the complete sequence representation output by the controller to obtain the searched network architecture.

[0065] S3, evaluating the sampled network architecture on the data set to obtain a voice quality index; specifically comprising the following sub-steps:

[0066] S31, training the searched network architecture on the speech enhancement training data set, and stopping after a certain number of rounds of training;

[0067] S32, evaluate the trained network architecture on the speech enhancement verification data set, and take the average of the speech quality evaluation index (PESQ) of the last several rounds on the verification set as the final quality index.

[0068] S4, comprehensively considering the speech quality evaluation index and the network architecture calculation amount as a reward to update the controller network in a reinforcement learning manner; specifically, it includes the following sub-steps:

[0069] S41, using reinforcement learning algorithm to build a search algorithm for network architecture: set the one-hot representation of the structural parameters in the network architecture search space as the action output by the controller network, set the one-hot representation of the structural parameters generated by the controller as the state of reinforcement learning, and obtain rewards by integrating the speech quality evaluation index and the neural network calculation amount through multi-objective fusion. Assume that the speech quality index is P, the neural network calculation amount is M, and the expected size of the calculation amount is M t , then the reward of reinforcement learning is defined as Where λ<0 is a hyperparameter that balances speech quality and neural network computation;

[0070] S42, estimates the gradient and updates the parameters of the controller network using a proximal policy optimization algorithm.

[0071] S5, repeat steps S2, S3, and S4 for several rounds until the sampling results converge and the optimal neural network architecture is obtained.

[0072] S6, applying the optimal neural network architecture to a speech enhancement task; specifically comprising the following sub-steps:

[0073] S61, performing short-time Fourier transform on the speech to be enhanced to obtain a speech spectrum, and then taking the logarithmic amplitude to obtain the input feature of the neural network;

[0074] S62, obtaining a speech spectrum masking value consistent with the input shape through a neural network, performing element-by-element dot multiplication of the masking value and the amplitude of the speech spectrum to obtain an enhanced speech amplitude spectrum;

[0075] S63, combining the enhanced speech amplitude spectrum with the original speech phase spectrum to obtain an enhanced speech complex spectrum; and then performing an inverse short-time Fourier transform to obtain the enhanced speech.

[0076] The specific setting of this embodiment is: the speech enhancement object is audio data with a sampling rate of 16kHz, and after the audio data is converted into time-frequency spectrum data through STFT, the logarithmic amplitude spectrum is taken as the input of the network. The frame length and frame shift of STFT are set to 512 (32ms) and 256 (16ms) respectively, and the window function is the square root Hanning window. The spectrum compression module keeps the 65 frequency bands within 0kHz-2kHz unchanged, and maps the 192 frequency bands within 2kHz-8kHz into 64 frequency bands.

[0077] The network architecture search is set as follows: the search space includes 5 convolution modules in the encoder, and the structural parameters of each module include the type of convolution module, the number of channels of the convolution module, and the convolution kernel size of the convolution module, so there are 15 search nodes in total. Specifically, the type of convolution module includes two options: downsampling convolution module and group depth separation convolution module; the number of channels of the convolution module includes seven options: 12, 16, 24, 28, 32, and 36; the convolution kernel size of the convolution module includes three options: (1,5), (2,5), and (3,3). The controller network of the network architecture search contains a total of 15 long short-term memory units, 100 hidden layer units, and 32 embedding layer units. After each network architecture is searched out, it is trained on the speech enhancement training dataset for 30 rounds, and the average of the speech perceptual evaluation quality (PESQ) of the last three rounds on the verification machine is taken as the speech quality indicator. When calculating the reward, the expected computational effort is set to 30 MMACs, and λ is set to -0.06 to balance the speech quality and neural network computational effort.

[0078] In this embodiment, the number of network architectures sampled each time is 32, and a total of 60 rounds of searches are performed, with a total of 1920 network architectures. The network structure with the best performance finally searched out is as follows:

[0079] Table 1 Network architecture search results

[0080] Structural parameters The first convolution module The second convolution module The third convolution module The fourth convolution module Fifth convolution module type Downsampling Packet depth separation Downsampling Packet depth separation Group depth can be separated Number of channels 12 16 12 32 16 Convolution kernel size (2,5) (1,5) (1,5) (2,5) (3,3)

[0081] According to the above parameter settings, the number of parameters of the neural network is 124K and the number of calculations per second is 32MMACs.

[0082] In order to illustrate the effect of the present invention, experiments were carried out on a simulated data set generated by using the DNS5 database, and GTCRN (Xiaobin Rong, Tianchi Sun, Xu Zhang, Yuxiang Hu, Changbao Zhu, and Jing Lu, "Gtcrn: A speech enhancement model requiring ultralow computational resources," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 971-975.) was selected as the baseline model for comparison. The evaluation indicators of the enhanced performance include scale-invariant signal-to-noise ratio (SISNR), perceptual evaluation of speech quality (PESQ) and non-intrusive speech quality evaluation index DNSMOS, including signal retention quality SIG, noise suppression quality BAK and overall quality OVRL, and the higher these indicators are, the better. The comparison results are shown in Table 2.

[0083] Table 2 Comparison results

[0084]

[0085]

[0086] The results in Table 2 prove that the neural network system of the present invention can achieve better enhancement performance with a smaller amount of calculation than GTCRN. It is worth pointing out that although the number of system parameters of the present invention is significantly increased compared with GTCRN, in actual application deployment, this degree of increase in the number of parameters will hardly bring any negative impact. Therefore, the present invention can achieve high-performance speech enhancement with a lower amount of parameters and computational overhead.

Claims

1. An ultra-lightweight speech enhancement neural network based on network architecture search, including an encoder, an enhancer and a decoder, characterized in that: The encoder includes: a spectrum compression module for compressing the noisy speech spectrum in the frequency dimension; a downsampling convolution module for extracting features from the feature map and downsampling the frequency dimension; a grouped depth separation convolution module for extracting features from the feature map and enhancing features; The enhancer includes: a two-way grouping recurrent neural network for modeling the feature map in the time and frequency domain; The decoder includes: a grouped depth separation convolution module for extracting and enhancing features of a feature map; an upsampling convolution module for reconstructing features of a feature map and upsampling the frequency dimension; and a spectrum expansion module for expanding the frequency dimension of the feature map.

2. The ultra-lightweight speech enhancement neural network based on network architecture search according to claim 1, characterized in that: The specific structure of the encoder is a spectrum compression module, a downsampling convolution module, a grouped depth separation convolution module, a downsampling convolution module, a grouped depth separation convolution module, and a grouped depth separation convolution module connected in sequence; the specific structure of the decoder is a grouped depth separation convolution module, a grouped depth separation convolution module, an upsampling convolution module, a grouped depth separation convolution module, an upsampling convolution module, and a spectrum expansion module connected in sequence.

3. The ultra-lightweight speech enhancement neural network based on network architecture search according to claim 1, characterized in that: The downsampling convolution module includes a two-dimensional convolution layer, a batch normalization layer, a series nonlinear activation function and a polynomial attention module, wherein the two-dimensional convolution layer downsamples the frequency dimension of the feature map with a stride of 2.

4. The ultra-lightweight speech enhancement neural network based on network architecture search according to claim 3, characterized in that: The series nonlinear activation function includes a learnable affine transformation and the sum of multiple ReLU activation functions, and each ReLU activation function has a learnable scaling coefficient and bias coefficient.

5. The ultra-lightweight speech enhancement neural network based on network architecture search according to claim 3, characterized in that: The polynomial attention module includes a two-order attention module, wherein the first-order attention module includes a pooling layer, a gated recurrent unit layer and a fully connected layer; the second-order attention module includes two fully connected layers; the input feature map obtains an attention masking value through the first-order attention module, and multiplies it with the input feature map to obtain a first-order output; after the input feature map passes through the first fully connected layer in the second-order attention module, it passes through an activation function and multiplies it with itself, and then passes through the second fully connected layer to obtain a second-order output; the first-order output and the second-order output are weighted and summed with a learnable coefficient to obtain the final output.

6. The ultra-lightweight speech enhancement neural network based on network architecture search according to claim 1, characterized in that: The grouped deep separation convolution module includes a point-by-point convolution layer with a grouping number of 2, a first batch normalization layer, a first order nonlinear activation function, a deep convolution layer, a second batch normalization layer, a second order nonlinear activation function, a polynomial attention module and a channel shuffling operation module.

7. The design method of an ultra-lightweight speech enhancement neural network based on network architecture search as claimed in claim 1, characterized in that: The specific steps include: S1, building a modular network architecture search space; S2, uses the controller network to sample in the search space; S3, evaluating the network architecture sampled in step S2 on the data set to obtain a speech quality index; S4, combining the voice quality index and the network architecture calculation amount as a reward to update the controller network in a reinforcement learning manner; S5, repeat steps S2, S3, and S4 for several rounds until the sampling results converge; S6, applies the searched optimal neural network architecture to the speech enhancement task.

8. The design method according to claim 7, characterized in that: The step S1 specifically includes the following sub-steps: S11, construct a network architecture prototype by connecting the encoder, enhancer and decoder in sequence; S12, generating a modular local search space based on the internal structure of the encoder, including: the type of convolution module, the number of channels of the convolution module, and the convolution kernel size of the convolution module; S13, determining the architecture of the enhancer and the decoder according to the architecture of the encoder; S14, build a complete network so that each network architecture corresponds to a serialized representation.

9. The design method according to claim 8, characterized in that: The step S2 specifically includes the following sub-steps: S21, design a controller network based on long short-term memory units, where the output of each unit constitutes a hidden state of the sequential representation of the network architecture in the search space; S22, deserialize the complete sequence representation output by the controller network to obtain the searched network architecture.

10. The design method according to claim 9, characterized in that: The step S4 specifically includes the following sub-steps: S41, rewards are given by integrating voice quality indicators and network architecture calculations through multi-objective fusion; S42, estimates the gradient and updates the parameters of the controller network using a proximal policy optimization algorithm.

Images