Model deployment method and device, electronic equipment and computer program product

Abstract

The invention discloses a model deployment method and device, electronic equipment and a computer program product. The method comprises the following steps: acquiring model training parameters supported by model deployment equipment; according to the model training parameters, a voice sub-model is intercepted in a voice enhancement streaming neural network of a preset voice model, and each residual module in the voice enhancement streaming neural network at least comprises a scalable linear layer, a scalable attention mechanism layer and a scalable normalization layer; training a preset voice model and the voice sub-model according to pre-configured sample data to obtain a target voice model trained based on the preset voice model and a target voice sub-model trained based on the voice sub-model; and allocating a target voice sub-model to the model deployment device, and allocating a target voice model to the cloud server. The technical problem that in the prior art, a speech enhancement model is difficult to dynamically adapt to and deploy on equipment with different computing capabilities is solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a model deployment method, apparatus, electronic device, and computer program product. Background Technology

[0002] With the rapid development of deep neural networks, state-of-the-art performance has been achieved in many fields, including speech processing. Speech enhancement (SE) is a key task in audio signal processing, aiming to improve the quality and intelligibility of speech signals that have been damaged by ambient noise, room reverberation, amplitude clipping, or packet loss during transmission. Speech enhancement algorithms are widely deployed, from consumer devices such as headphones and smartphones to high-performance cloud computing platforms, with significant differences in the computational resource limitations of each platform.

[0003] Traditional methods face significant challenges when deploying neural networks for speech enhancement across different platforms. While large-model-based speech enhancement algorithms offer excellent performance, their high computational cost makes them impractical for edge devices with limited computing resources, such as headphones or smartphones. These devices typically require models with a computational cost below 1 billion multiply-accumulate operations per second (GMACs / s). Conversely, smaller models designed for low-resource devices, while having lower computational overhead, often underperform models deployed on high-end platforms (supporting tens to hundreds of GMACs / s).

[0004] To address this issue, various techniques have been explored to adapt neural networks to different computational budgets. For example, Neural Architecture Search (NAS), proposed by Poyser and Breckon (Pattern Recognition, 2024), attempts to find the optimal architecture that satisfies specific constraints, but this typically requires significant computational resources. Model pruning and knowledge distillation are often used to generate smaller models from large networks, but they usually require fine-tuning or retraining for each target configuration, which is costly. For example, models such as LLaMA and Whisper offer multiple size variants, but each variant must be trained independently, leading to inefficient development and deployment.

[0005] Another approach is the early exit strategy, which allows intermediate layers of the neural network to output predictions, thereby reducing the network depth during the inference phase. However, the early exit strategy primarily adjusts the depth while keeping the network width constant, limiting the granularity of scalability. Techniques such as Dropout and model ensembles inspire the dynamic activation of neurons, but these methods are primarily used during the training phase to prevent overfitting or simulate ensembles, rather than for achieving runtime scalability.

[0006] In the context of speech enhancement, existing solutions typically focus on specific subtasks, such as denoising, dereverberation, declipping, or packet loss concealment. These subtasks are closely related and can usually be uniformly modeled as different forms of noise in the time-frequency domain. However, existing techniques often design independent models for each subtask, requiring separate training and optimization. This fragmented approach increases development complexity and resource consumption, especially when models need to be deployed across multiple platforms.

[0007] Furthermore, the AI ​​Flow paradigm emphasizes the collaborative deployment of models across heterogeneous edge devices, highlighting the necessity of a single network that can adapt to different computing capabilities without retraining. Current methods still lack a unified architecture that can effectively support multiple speech enhancement tasks and dynamically scale across different hardware platforms, making efficient deployment from low-resource devices to high-performance servers difficult.

[0008] There is currently no effective solution to the problem that existing speech enhancement models have difficulty dynamically adapting to deployment on devices with different computing capabilities. Summary of the Invention

[0009] This invention provides a model deployment method, apparatus, electronic device, and computer program product to at least solve the technical problem in the prior art that speech enhancement models are difficult to dynamically adapt to deployment on devices with different computing capabilities.

[0010] According to one aspect of the present invention, a model deployment method is provided, comprising: acquiring model training parameters supported by a model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; and extracting a speech sub-model from a preset speech model based on the model training parameters, wherein the preset speech model employs a speech enhancement streaming neural network, the speech enhancement streaming neural network including at least: a plurality of cascaded residual modules, each residual module including at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes, and the speech sub-model employs a speech enhancement streaming neural network. The speech enhancement streaming neural network is a segment extracted from a high-flow neural network. The speech enhancement streaming neural network includes at least: residual sub-modules connected in series according to a pre-set number of residuals, the number of residuals being determined based on a pre-set model depth; the residual sub-modules being configured according to the pre-set model width and the pre-set number of attention heads; training a pre-set speech model and a speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the pre-set speech model and a target speech sub-model trained based on the speech sub-model, wherein during training, the parameter updates and learning progress of the pre-set speech model and the speech sub-model are synchronized; the target speech sub-model is allocated to the model deployment device and to the cloud server.

[0011] Optionally, extracting a speech sub-model from a preset speech model based on the model training parameters includes: determining the number of residual sub-modules required in the speech enhancement streaming neural network based on the preset model depth in the model training parameters, thereby obtaining the number of residuals; extracting the cascaded residual modules in the speech enhancement streaming neural network according to the number of residuals, thereby obtaining multiple residual modules to be adjusted; adjusting the scalable linear layer, the scalable attention mechanism layer, and the scalable normalization layer in each residual module to be adjusted according to the preset model width and the preset number of attention heads in the model training parameters, thereby obtaining the residual sub-module; and determining the speech sub-model based on the cascaded residual sub-modules according to the number of residuals, wherein the speech enhancement streaming neural network used in the speech sub-model is the cascaded residual sub-modules according to the number of residuals.

[0012] Optionally, adjusting the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module to be adjusted, based on the preset model width and preset number of attention heads in the model training parameters, to obtain the residual submodule includes: determining the number of activated neurons in the scalable linear layer based on the preset model width in the model training parameters, to obtain the activation count; determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, to obtain the activation head count; determining a subset of weight vectors in the scalable normalization layer based on the preset model width in the model training parameters, wherein the subset of weight vectors is truncated from multiple weight vectors in a pre-set weight set, and the weight vectors are used to adjust the combination method between neurons in the scalable linear layer; and configuring each residual module to be adjusted according to the activation count, the number of activation heads, and the subset of weight vectors to obtain the residual submodule.

[0013] Optionally, after determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, and obtaining the number of activated heads, the method further includes: obtaining the total feature dimension of the features to be processed by the preset speech model; and determining the feature dimension allocated to each attention head when performing feature processing based on the ratio of the total feature dimension to the number of activated heads.

[0014] Optionally, before training the preset speech model and speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech sub-model trained based on the speech sub-model, the method further includes: acquiring clean speech samples; performing degradation processing on the clean speech samples according to at least one pre-set degradation method to obtain speech distortion samples, wherein the degradation method includes at least: noise, reverberation, clipping, and packet loss; and combining the clean speech samples and the corresponding speech distortion samples to obtain the sample data.

[0015] Optionally, performing degradation processing on the clean speech sample according to at least one pre-set degradation method to obtain a speech distortion sample includes: randomly combining multiple degradation methods according to a pre-configured selection probability to obtain a degradation strategy combination, wherein each degradation method has a pre-set corresponding selection probability; and performing degradation processing on the clean speech sample according to the degradation strategy combination to obtain a speech distortion sample.

[0016] Optionally, the clean speech sample is degraded according to at least one pre-set degradation method to obtain a speech distortion sample, including at least one of the following: randomly adding pre-acquired noise speech samples to the clean speech sample according to a preset signal-to-noise ratio to obtain the speech distortion sample; adding reverberation to the clean speech sample using pre-acquired house impulse response samples to obtain the speech distortion sample, wherein the house impulse response samples are used to describe the propagation characteristics of sound in an enclosed space; clipping sound waves in the clean speech sample whose waveform amplitude exceeds a preset amplitude threshold to obtain the speech distortion sample, wherein the preset amplitude threshold is determined based on the waveform amplitude of the waveform distribution in the clean speech sample that is not lower than a preset quantile; or simulating packet loss in the clean speech sample using a Markov chain to obtain the speech distortion sample.

[0017] According to another aspect of the present invention, a model deployment apparatus is also provided, comprising: an acquisition module, configured to acquire model training parameters supported by a model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; and a truncating module, configured to truncate a speech sub-model from a preset speech model based on the model training parameters, wherein the preset speech model employs a speech enhancement streaming neural network, and the speech enhancement streaming neural network includes at least: a plurality of cascaded residual modules, each of the residual modules including at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes, and the speech sub-model employs a scalable linear layer for controlling the model width, a preset number of attention heads, and a preset number of attention heads pre-configured for the model deployment device; the speech sub-model employs a scalable linear ... The speech enhancement streaming neural network is a segment extracted from a high-flow neural network. The speech enhancement streaming neural network includes at least: residual sub-modules connected in series according to a pre-set number of residuals, the number of residuals being determined based on a pre-set model depth; the residual sub-modules being configured according to the pre-set model width and the pre-set number of attention heads; a training module, used to train the pre-set speech model and the speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the pre-set speech model and a target speech sub-model trained based on the speech sub-model, wherein during training, the parameter updates and learning progress of the pre-set speech model and the speech sub-model are synchronized; and a deployment module, used to allocate the target speech sub-model to the model deployment device and to the cloud server.

[0018] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described model deployment method through the computer program.

[0019] According to another aspect of the present invention, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of the above-described model deployment method.

[0020] In the embodiments described above, the model training parameters are determined based on the resources supported by the model deployment device. Then, a matching speech enhancement streaming neural network sub-network is extracted from the preset complete speech enhancement streaming neural network according to the model training parameters to obtain a speech sub-model of the preset speech model. The speech enhancement streaming neural network is designed with multiple cascaded residual modules. Each residual module is equipped with a scalable linear layer, a scalable attention mechanism layer, and a scalable normalization layer. This allows the model to flexibly adjust its structure and computational complexity according to actual hardware conditions without affecting the overall learning progress and parameter updates. Furthermore, by training the preset speech model and its extracted speech sub-models synchronously, the performance consistency of the model is guaranteed on various types of hardware, thereby achieving the technical effect of seamless model deployment from low-resource devices to high-performance servers. This solves the problem in the prior art that speech enhancement models are difficult to dynamically adapt to deployment on devices with different computing capabilities. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1 This is a flowchart of a model deployment method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the structure of a speech enhancement streaming neural network according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the implementation process of an SEFlow network according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a model deployment device according to an embodiment of the present invention;

[0026] Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0030] AI Flow: The AI-empowered Flow Networking Paradigm is a new network architecture centered on data flow and driven by AI. Through edge-cloud collaboration, it leverages AI technology to achieve a closed loop of network perception, decision-making, control, and optimization. AI capabilities are no longer confined to the application layer but are embedded in various layers such as the data link layer, network layer, and transport layer to achieve intelligent scheduling, resource allocation, and service assurance.

[0031] Speech enhancement (SE) refers to the process of recovering clearer, more intelligible, or more natural speech from damaged or noise-contaminated speech signals in noisy environments through signal processing or deep learning techniques.

[0032] Whisper is a large-scale automatic speech recognition (ASR) model developed by OpenAI based on the encoder-decoder Transformer architecture. It has multilingual speech recognition, speech translation, and language recognition capabilities, and can achieve near-human level speech-to-text and multilingual translation in complex accent and noisy environments.

[0033] Attention mechanism: This refers to a mechanism in neural networks that mimics human attention allocation behavior. It allows the model to dynamically focus on the more important parts of the input information, rather than processing all information equally. The core idea is to "selectively focus" on the parts of the input information most relevant to the current task. Essentially, this is achieved by calculating attention weights to weight the contributions of different pieces of information.

[0034] Deep learning model deployment refers to the process of integrating and deploying the deep learning model obtained during the training phase into a real system or device to achieve real-time prediction, inference, or service functions.

[0035] According to an embodiment of the present invention, a model deployment method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] Figure 1 This is a flowchart of a model deployment method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step S102: Obtain the model training parameters supported by the model deployment device, wherein the model training parameters include at least: the preset model depth, preset model width and preset number of attention heads pre-configured for the model deployment device;

[0038] Step S104: Based on the model training parameters, extract a speech sub-model from the preset speech model. The preset speech model uses a speech enhancement streaming neural network, which includes at least: multiple residual modules connected in series. Each residual module includes at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes. The speech sub-model uses a speech enhancement streaming neural network extracted from the speech enhancement streaming neural network. The speech enhancement streaming neural network includes at least: residual sub-modules connected in series according to a preset number of residuals. The number of residuals is determined according to the preset model depth, and the residual sub-modules are residual modules configured according to the preset model width and the preset number of attention heads.

[0039] Step S106: Train the preset speech model and speech sub-model based on the pre-configured sample data to obtain the target speech model trained based on the preset speech model and the target speech sub-model trained based on the speech sub-model. During the training process, the parameter updates and learning progress of the preset speech model and the speech sub-model are kept synchronized.

[0040] Step S108: Assign a target speech sub-model to the model deployment device and assign a target speech model to the cloud server.

[0041] In the embodiments described above, the model training parameters are determined based on the resources supported by the model deployment device. Then, a matching speech enhancement streaming neural network sub-network is extracted from the preset complete speech enhancement streaming neural network according to the model training parameters to obtain a speech sub-model of the preset speech model. The speech enhancement streaming neural network is designed with multiple cascaded residual modules. Each residual module is equipped with a scalable linear layer, a scalable attention mechanism layer, and a scalable normalization layer. This allows the model to flexibly adjust its structure and computational complexity according to actual hardware conditions without affecting the overall learning progress and parameter updates. Furthermore, by training the preset speech model and its extracted speech sub-models synchronously, the performance consistency of the model is guaranteed on various types of hardware, thereby achieving the technical effect of seamless model deployment from low-resource devices to high-performance servers. This solves the problem in the prior art that speech enhancement models are difficult to dynamically adapt to deployment on devices with different computing capabilities.

[0042] In step S102 above, the model deployment device can be a device with computing power, such as a computer terminal, smart earphone, mobile phone terminal, tablet terminal, and server, etc. The model training parameters supported by the model deployment device can be determined according to the amount of resources that the model deployment device can provide. The more resources that the model deployment device can provide, the stronger its computing power will be, and thus the larger the model size that can be deployed on the model deployment device will be.

[0043] In step S104 above, the preset speech model adopts a speech enhancement streaming neural network with multiple residual modules connected in series. In this speech enhancement streaming neural network, each residual module can be used for feature analysis. The more residual modules connected in series, the deeper the preset speech model is. Based on the preset model depth in the model training parameters, the corresponding number of residual modules can be extracted from the preset speech model to adjust the model depth and obtain a speech sub-model that can be applied to the model deployment device.

[0044] In step S104 above, each residual module includes: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes. Based on the preset model width and preset number of attention heads in the model training parameters, the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module can be adjusted to obtain the adjusted residual sub-module.

[0045] Figure 2 This is a schematic diagram of the structure of a speech enhancement streaming neural network according to an embodiment of the present invention, as shown below. Figure 2 As shown, the speech enhancement streaming neural network, also known as the SEFlow network, is a causal, sampling rate-independent unified speech enhancement model based on the BS-RoFormer architecture. The input is a damaged speech signal in the time-frequency domain, which is mapped to an embedded representation by a bandwidth encoder. After residual block processing, the decoder generates an enhanced complex spectrogram.

[0046] Optionally, such as Figure 2 As shown, the residual modules in the speech enhancement streaming neural network include: FlexLinear (also known as a scalable linear layer), FlexAttention (also known as a scalable attention mechanism layer), and FlexRMSNorm (also known as a scalable normalization layer).

[0047] Optionally, such as Figure 2 As shown, FlexLinear (also known as a scalable linear layer) can dynamically adjust the number of input and output neurons. The width control of the linear layer is decomposed into the control of the input width and the output width. A layer with... Input and A linear layer with one output computes each output as a weighted sum of all inputs plus a bias term. Clearly, Each of the outputs depends on its own set of learnable parameters. Therefore, by using the weight matrix... Slices are Slice and set the bias vector Slices are To control the output width of the linear layer, where Regarding input width control, this invention finds that in both cases, features are formed by summing the contributions of multiple neurons. Inspired by the principle of early termination only activating early layers, this invention activates only early neurons (i.e., the early termination neurons)... neurons, of which The input width is controlled by [a certain component]. The two components mentioned above constitute the proposed FlexLinear, as shown below. Figure 2 As shown, the computational complexity scales quadratically with the width.

[0048] Optionally, such as Figure 2 As shown, FlexAttention (also known as the scalable attention mechanism layer) is built on top of FlexLinear and supports dynamically adjusting the number of attention heads. Assuming a standard multi-head attention mechanism has... Each head has a dimension of [number]. The input is ,in, For sequence length, For the embedding dimension, the proposed FlexAttention sets the maximum number of heads to... and selectively use only The number of heads. A flexible attention mechanism is implemented through slice queries and key-value matrices, suitable for various sequence modeling tasks.

[0049] Optionally, such as Figure 2 As shown, FlexRMSNorm (also known as a scalable normalization layer) adjusts the weight vector in the normalization layer. This is to adapt to different input dimensions and ensure model stability under varying widths. Given an input tensor... ,( The calculation method for FlexRMSNorm is as follows:

[0050] ;

[0051] in, yes The sliced ​​version, only the first few slices are retained. There are 1 element, ⊙ represents the Hadamard product (element-wise multiplication), and It is a small constant used to prevent the denominator from being zero.

[0052] Optionally, the SEFlow network supports the idea of ​​early exit, which adjusts the network depth by selecting the first few residual blocks and combines this with width adjustment to form a highly scalable sub-network.

[0053] Optionally, the SEFlow network supports VAD auxiliary tasks by adding a lightweight VAD decoder and using FlexFusionAttention to aggregate information along the frequency dimension to generate speech activity predictions, thereby improving performance as an auxiliary task.

[0054] Optionally, in the SEFlow network, the decoder is based on the BS-RoFormer design, the first linear layer is replaced with FlexLinear, and a complex mapping strategy (instead of complex masking) is used to generate the output spectrogram, supporting tasks such as packet loss.

[0055] The SEFlow network described in this application reduces the number of parameters and computational cost, supports dynamic inference pruning, adapts to different hardware deployment requirements (such as mobile / cloud), meets lightweight requirements, and improves computational efficiency. The input width control borrows the "early exit" concept, allowing the model to selectively activate some neurons based on the complexity of the input features. This may enhance the ability to quickly process simple features, or capture multi-scale information through different combinations, exhibiting structural flexibility and satisfying multi-scale feature learning. For simple tasks, only a few heads are needed to capture key associations, reducing computational cost. For complex tasks, more heads are activated as needed, enhancing feature interaction capabilities, balancing performance and efficiency, and allowing for flexible adjustment of the number of heads in multi-head attention.

[0056] The embodiments described above in this application, while ensuring the model's generalization ability and handling of various speech enhancement tasks, greatly improve the model's adaptability and deployment efficiency, reduce the risk of performance degradation due to differences in device computing power, and make real-time speech enhancement possible.

[0057] The SEFlow network architecture and deployment method described in the above embodiments of this application significantly improve the flexibility and efficiency of speech enhancement models. By dynamically adjusting the model depth, width, and number of attention heads, the SEFlow network can seamlessly adapt to a wide range of hardware environments, from low-end edge devices to high-performance cloud servers, without the need for additional fine-tuning or retraining, thus greatly reducing the cost and complexity of model deployment. In practical applications, the SEFlow network can not only handle various speech enhancement tasks, including denoising, dereverberation, de-clipping, and packet loss concealment, but also automatically prune the model size according to the device's computing power, ensuring real-time performance and resource utilization.

[0058] In step S106 above, the preset speech model and the speech sub-model extracted from the preset speech model adopt dynamic sub-network sampling and synchronous random seeding during the training process, which optimizes the distributed training efficiency and enables the preset speech model and the speech sub-model to be trained with the same random variables. This ensures that the training results of the preset speech model and the speech sub-model are consistent. As a result, in the model application stage, the trained target speech sub-model and the target speech model can produce the same or similar output content for the same input content.

[0059] In step S106 above, the preset speech model and speech sub-model trained based on the SEFlow network can be applied to speech enhancement tasks. Each sample data used in the training model includes a clean speech sample and a speech distortion sample obtained by degrading based on the clean speech sample.

[0060] Alternatively, clean speech samples can be collected from the DNS5 Speech, WSJ, LibriTTS, VCTK, and EARS datasets, totaling approximately 800 hours of raw audio.

[0061] Optionally, the collected raw audio can be cleaned using DNSMOS scoring (retaining samples with scores >3.0) for DNS5Speech and WSJ, and DNS5 Speech can be deduplicated, resulting in approximately 400,000 clean speech segments with an average duration of 4 seconds, totaling approximately 440 hours.

[0062] It should be noted that distorted speech samples can be obtained by processing clean speech samples through degradation methods such as adding noise, reverberation, clipping, and packet loss.

[0063] Optionally, approximately 270,000 noise segments were collected from datasets such as DNS5 Noise, WHAM, FMA, FSD50K, ESC-50, UrbanSound8K, TAU, CochlScene, and ICME'24 ASC, after which samples containing significant speech content were removed.

[0064] Alternatively, reverberation can be achieved based on the room impulse response (RIR).

[0065] It should be noted that the room impulse response section uses Pyroomacoustics to simulate 1.44 million RIRs, covering small, medium, and large room types (200 rooms in each type). 24 microphones are randomly placed in each room, generating 100 RIRs at different sound source locations. Room parameters include height (2-5m), wall absorption coefficient (0.2-0.8), length and width (3-10m for small rooms, 10-30m for medium rooms, and 30-50m for large rooms), with reverberation time T_60 ranging from 0.03 to 2.98 seconds.

[0066] The embodiments described above in this application simulate complex interferences (such as environmental noise, equipment limitations, and network transmission problems) in actual speech transmission and recording by superimposing various degradation types such as noise, reverberation, clipping, and packet loss onto clean speech samples. This enables the model to learn general feature representations across scenarios, rather than overfitting to a single degradation type, thereby improving the model's generalization ability.

[0067] The embodiments described above support unified multi-task training: sub-tasks such as denoising, dereverberation, de-clipping, and packet loss concealment are abstracted into a unified "speech degradation-enhancement" problem. Through hybrid distortion data enhancement, shared features of different tasks (such as time-frequency domain anomalies) are exposed. Simultaneously, independent training costs are reduced. This invention generates multi-task training samples in a single data enhancement process, enabling a single model to handle cross-task tasks and saving over 70% of repetitive training costs. Experimental results show that even the smallest speech sub-model significantly outperforms unprocessed input, while the overall model's performance approaches that of state-of-the-art speech enhancement models, demonstrating strong cross-platform deployment capabilities and real-time application potential.

[0068] The technical solution claimed in this application, particularly on resource-constrained devices, enables the SEFlow network to provide efficient speech enhancement services, improving the quality of voice communication in application scenarios such as voice assistants, teleconferencing, and online education. Whether for consumer devices or enterprise servers, a model version suitable for their computing resources can be found, ensuring a consistent and high-quality user experience. Furthermore, due to its unified architecture and multi-tasking capabilities, the SEFlow network reduces the workload of model development and maintenance, opening up new directions for research and application in the field of speech enhancement.

[0069] As an optional embodiment, extracting a speech sub-model from a preset speech model based on model training parameters includes: determining the number of residual sub-modules required in the speech enhancement streaming neural network based on the preset model depth in the model training parameters, thus obtaining the number of residuals; extracting cascaded residual modules in the speech enhancement streaming neural network according to the number of residuals, thus obtaining multiple residual modules to be adjusted; adjusting the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module to be adjusted according to the preset model width and preset number of attention heads in the model training parameters, thus obtaining residual sub-modules; and determining the speech sub-model based on the cascaded residual sub-modules according to the number of residuals, wherein the speech enhancement streaming neural network used in the speech sub-model is a cascaded residual sub-module according to the number of residuals.

[0070] In this embodiment, the specific structure of the speech enhancement streaming neural network SEFlow is dynamically adjusted based on the model training parameters, including the number of residual sub-modules and the parameter configuration within each module. First, the required number of residual sub-modules is determined according to the preset model depth, which directly affects the network depth. Then, a corresponding number of residual modules are extracted from the SEFlow network to form a sequence of modules to be adjusted. Next, based on the preset model width and the preset number of attention heads, the FlexLinear (i.e., scalable linear layer), FlexAttention (i.e., scalable attention mechanism layer), and FlexRMSNorm (i.e., scalable normalization layer) in each residual module to be adjusted are adjusted, thereby changing the network width and the scale of the attention mechanism. This parameter-based dynamic pruning method not only greatly improves the deployment efficiency of the model on different hardware platforms, but also ensures that the model can maintain excellent speech enhancement performance whether on low-resource devices or in high-resource environments by fully covering the parameter updates of different sub-models during the training phase.

[0071] The embodiments described above in this application, based on the preset model depth, preset model width, and preset number of attention heads in the model training parameters, adjust the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in the preset speech model. This enables the accurate generation of speech sub-models that meet specific computing resource requirements from the full model (such as the preset speech model), achieving flexible adaptation from extremely low computing costs (as low as 0.34 GMACs / s) to high-performance computing environments (up to 24.71 GMACs / s).

[0072] The technical solution claimed in this application allows the SEFlow network to automatically select the optimal model configuration based on the computing power of the actual deployment platform and the specific needs of the speech enhancement task during runtime. This ensures both real-time performance and resource conservation, while also effectively improving speech quality, especially demonstrating significant advantages when handling complex degradation scenarios (such as multiple noises, reverberation, and packet loss). In this way, the SEFlow network successfully transcends the limitations of hardware platforms, becoming a truly scalable and unified speech enhancement solution.

[0073] As an optional embodiment, based on the preset model width and preset number of attention heads in the model training parameters, the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module to be adjusted are adjusted to obtain the residual sub-module. This includes: determining the number of activated neurons in the scalable linear layer based on the preset model width in the model training parameters, thus obtaining the activation count; determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, thus obtaining the activation head count; determining a subset of weight vectors in the scalable normalization layer based on the preset model width in the model training parameters, wherein the weight vector subset is truncated from multiple weight vectors in a pre-set weight set, and the weight vectors are used to adjust the combination method between neurons in the scalable linear layer; and configuring each residual module to be adjusted according to the activation count, the number of activation heads, and the weight vector subset to obtain the residual sub-module.

[0074] In this embodiment, the SEFlow network employs a dynamic adjustment mechanism to flexibly control the parameters of the FlexLinear, FlexAttention, and FlexRMSNorm layers within the residual modules, achieving dynamic adaptation of model width and the number of attention heads. First, the number of activated neurons in the linear layers is determined based on a preset model width, thereby adjusting the input and output widths. Second, the number of activated attention heads is determined based on a preset number of attention heads, optimizing the computational efficiency of the multi-head attention mechanism. Third, a subset of weight vectors in the normalization layer is determined using the preset model width, ensuring model stability under different widths. Finally, based on the above configuration, each residual module is correspondingly adjusted to a residual sub-module, enabling SEFlow to dynamically sample sub-networks during the training phase, significantly improving the model's scalability and hardware adaptability. This mechanism not only adjusts the model's computational complexity according to changes in hardware resources but also allows for flexible switching between different speech enhancement tasks without additional fine-tuning or retraining, greatly improving model deployment efficiency and resource utilization efficiency. In this way, SEFlow can provide efficient speech enhancement processing even on resource-constrained edge devices, while on powerful cloud servers, it can unleash its full potential to achieve optimal enhancement results.

[0075] As an optional embodiment, after determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, the method further includes: obtaining the total feature dimension of the features to be processed by the preset speech model; and determining the feature dimension allocated to each attention head when performing feature processing based on the ratio of the total feature dimension to the number of activated heads.

[0076] In this embodiment, the FlexAttention layer (also known as the scalable attention mechanism layer) in the SEFlow network dynamically determines the actual number of activated attention heads H based on a preset number of attention heads Hmax. This process first involves obtaining the total dimension D of the features to be processed; then, by calculating the ratio of D to H, the specific feature dimension D_head = D / H allocated to each attention head during feature processing is determined. This method ensures that even when adjusting the number of attention heads, the model can maintain effective analysis and processing of features, ensuring consistent performance and computational efficiency whether using fewer attention heads for lightweight processing on resource-constrained devices or using all attention heads for refined feature interaction in high-resource environments.

[0077] The embodiments described above in this application enable the SEFlow network to maintain the model's generalization ability and multi-task processing efficiency under different computational budgets by dynamically allocating the dimension of the attention head. It can be efficiently deployed on a wide range of hardware platforms without additional model training or fine-tuning, while providing appropriate attention mechanism support for simple and complex tasks, optimizing resource utilization and inference speed.

[0078] As an optional embodiment, before training a preset speech model and a speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech sub-model trained based on the speech sub-model, the method further includes: acquiring clean speech samples; performing degradation processing on the clean speech samples according to at least one pre-set degradation method to obtain speech distortion samples, wherein the degradation method includes at least: noise, reverberation, clipping, and packet loss; and combining the clean speech samples and the corresponding speech distortion samples to obtain sample data.

[0079] In this embodiment, the training process for the speech enhancement neural network is meticulously planned. First, clean speech samples are collected and subjected to diverse degradation processing, including but not limited to noise introduction, reverberation simulation, amplitude clipping, and packet loss simulation, generating a series of speech distortion samples. This series of steps ensures the richness and diversity of the training data, enabling the model to have broader adaptability to various speech degradation scenarios in the real world. Subsequently, the original clean speech samples are paired with their corresponding degradation versions to form a sample dataset for model training. This dataset allows the SEFlow network to access speech samples more closely aligned with real-world applications during training. Therefore, when deployed to hardware platforms with varying computing capabilities, whether resource-constrained edge devices or computationally abundant cloud servers, it can exhibit excellent speech enhancement effects, effectively resolving the contradiction between model performance and computing resources in existing technologies, and achieving a dual optimization of computational efficiency and speech quality.

[0080] In the above embodiments of this application, during the training process, the model not only learns how to restore speech affected by a single degradation factor, but also masters the ability to cope with multiple complex degradations at the same time, which greatly improves the generalization performance and practicality of the model. Furthermore, in the subsequent training stage, the strategy of dynamic sub-network sampling and synchronous random seeding further ensures the flexibility and efficiency of the model in dealing with various speech degradation problems, enabling the SEFlow network to achieve high-performance speech enhancement tasks on different hardware configurations.

[0081] As an optional embodiment, degrading clean speech samples according to at least one pre-set degradation method to obtain speech distortion samples includes: randomly combining multiple degradation methods according to pre-configured selection probabilities to obtain a degradation strategy combination, wherein each degradation method has a pre-set corresponding selection probability; and degrading clean speech samples according to the degradation strategy combination to obtain speech distortion samples.

[0082] In this embodiment, a dynamic combination of degradation strategies is employed in the process of degrading clean speech samples to obtain speech distortion samples. Multiple degradation methods are defined, including but not limited to adding noise, reverberation, clipping, and packet loss simulation, each assigned a specific selection probability. During the training data preparation phase, degradation strategies are randomly combined based on these selection probabilities to form composite degradation patterns, which are then applied to the clean speech samples. This not only increases the diversity of the training data but also ensures that the model can learn to cope with various degradation scenarios. This dynamic random combination method optimizes model training efficiency while ensuring a reasonable distribution of different degradation types in the dataset, avoiding excessive bias towards any single degradation type, thereby improving the overall robustness and applicability of the model.

[0083] It should be noted that, in actual implementation, the random combination of multiple degradation methods improves the generalization performance of the model, enabling the SEFlow network to more effectively recover speech quality when faced with speech signals of unknown or mixed degradation types.

[0084] As an optional embodiment, clean speech samples are degraded according to at least one pre-set degradation method to obtain speech distortion samples, including at least one of the following: randomly adding pre-acquired noise speech samples to clean speech samples according to a preset signal-to-noise ratio to obtain speech distortion samples; adding reverberation to clean speech samples using pre-acquired house impulse response samples to obtain speech distortion samples, wherein the house impulse response samples are used to describe the propagation characteristics of sound in enclosed spaces; clipping sound waves in clean speech samples whose waveform amplitude exceeds a preset amplitude threshold to obtain speech distortion samples, wherein the preset amplitude threshold is determined based on waveform amplitudes in clean speech samples whose waveform distribution is not lower than a preset quantile; and simulating packet loss in clean speech samples using a Markov chain to obtain speech distortion samples.

[0085] In this embodiment, noise addition, clipping, and packet loss simulation, by controlling the signal-to-noise ratio, amplitude threshold, and packet loss probability, increase the diversity of the training data. The use of R-room impulse response (RIR) samples simulates the sound propagation characteristics under different environments, covering room types from small to large. This data augmentation method not only improves the model's ability to handle environmental noise, equipment limitations, and network transmission issues, but also promotes the model's unified learning of multi-task processing—namely, denoising, dereverberation, de-clipping, and packet loss concealment—by randomly combining multiple degradation types, reducing the complexity and cost of designing and training models independently for different tasks. Through this embodiment, the model can effectively identify and process various speech degradation types in practical applications, significantly improving the quality and efficiency of speech enhancement.

[0086] Optionally, this application can further improve the robustness and adaptability of the model by adjusting the specific parameters of the above-mentioned degradation methods or introducing more types of degradation (such as nonlinear distortion, spectral changes, etc.).

[0087] Optionally, degradation processing can randomly add noisy speech samples to clean speech samples with a signal-to-noise ratio of [-5, 20] dB.

[0088] Optionally, the degradation process can use RIR to generate point source noise with a 50% probability.

[0089] Alternatively, degradation processing can be simulated by limiting waveform amplitudes above the 90th percentile to simulate clipping.

[0090] Alternatively, degradation can be simulated using Markov chains to handle packet loss.

[0091] Optionally, in the case of degradation processing, the dwell and transition probabilities for each degradation mode are 0.95 and 0.05, respectively, and the maximum continuous loss duration is 1-10 packets (10ms per packet).

[0092] Optionally, in the case of degradation processing, the random combination of each degradation mode only includes noise, reverberation, clipping and packet loss, and is applied to clean speech with random combinations of probabilities 1.0, 0.5, 0.3 and 0.3 to enhance the model's ability to generalize to multiple degradation types.

[0093] This invention also provides a preferred embodiment, which proposes a scalable and unified neural network architecture, namely the SEFlow network architecture, specifically designed for speech enhancement tasks. This architecture dynamically adjusts the computational complexity and structure of the network without fine-tuning or retraining, adapting to diverse hardware platform requirements ranging from low-resource edge devices (such as headphones and smartphones) to high-performance cloud servers. Simultaneously, it implements multiple speech enhancement tasks (such as denoising, dereverberation, de-clipping, and packet loss concealment) through a single model, improving the model's versatility and deployment efficiency, overcoming the limitations of existing models designed for single tasks or fixed hardware configurations.

[0094] As an optional embodiment, the architecture of the SEFlow network includes the following steps:

[0095] 1) Data preprocessing and augmentation: Dataset preparation: By aggregating diverse clean speech, noise and RIR data, and combining dynamic data augmentation strategies (randomly combining noise, reverberation, clipping and packet loss), approximately 440 hours of training data were constructed, covering a variety of degradation scenarios.

[0096] 2) Model Architecture: Based on the SEFlow network, the model utilizes innovative FlexLinear, FlexAttention, and FlexRMSNorm modules, combined with an early exit strategy, to construct a highly scalable unified architecture in both depth and width. This architecture can handle various speech enhancement tasks and adapt to different hardware platforms. Performance is enhanced by BS-RoFormer-based dual-path sequence modeling, VAD-assisted tasks, and complex mapping decoding. Dynamic subnetwork sampling and synchronous random seeds optimize training efficiency, and Mel-scale subband segmentation achieves sampling rate independence, providing an efficient solution for cross-platform and cross-task speech enhancement.

[0097] 3) Training Process: SEFlow's training component optimizes distributed training efficiency through dynamic sub-network sampling and synchronous random seeding, and improves performance by combining loss functions (complex spectrogram, amplitude spectrogram, VAD). The model supports multiple sampling rates, has efficient training configuration, and shows significant differences in parameter count and computational complexity between the full model and sub-networks, providing a flexible and efficient training solution for cross-platform and cross-task speech enhancement.

[0098] 4) Inference and Device Deployment: SEFlow achieves dynamic inference speeds from 0.19 GMACs / s to 24.71 GMACs / s through scalable modules (FlexLinear, FlexAttention, FlexRMSNorm) and an early exit strategy, adapting to diverse hardware from edge devices to cloud servers. The model supports multiple sampling rates and, combined with VAD-assisted tasks and complex mapping decoding, uniformly handles various speech enhancement tasks. The overall model performance is close to state-of-the-art, with the smallest subnetwork still significantly outperforming unprocessed input, demonstrating efficient cross-platform deployment capabilities and enhanced real-time application potential.

[0099] The SEFlow neural network architecture described in this application is scalable in terms of computational complexity and can uniformly handle a variety of speech enhancement tasks. This architecture should be able to be seamlessly deployed across different platforms without fine-tuning or retraining, while maintaining performance comparable to current state-of-the-art task-specific models.

[0100] Figure 3 This is a schematic diagram illustrating the implementation flow of an SEFlow network according to an embodiment of the present invention, as shown below. Figure 3 As shown, for low-resource devices (such as headphones), select... =2, =1 (corresponding to) =64), computational cost 0.34 GMACs / s, parameters 1.76M; for high-resource devices (such as cloud servers), use the complete model ( =12, =4), computational cost 24.71 GMACs / s, parameters 27.19M. The input to the SEFlow network is noisy speech. The signal is converted to a frequency domain signal via STFT; the frequency band encoder f divides the signal into K=41 sub-bands, which are then mapped to features. ,pass Extract features from each residual block and activate early exit points. Decoder f converts the features into enhanced speech. Output time-domain waveform.

[0101] The embodiments described above in this application feature lightweight design and hardware adaptability. Through dynamic parameter pruning (such as slicing the input and output widths of linear layers and adjusting the number of attention heads), the number of model parameters and computational load are significantly reduced, supporting flexible deployment on heterogeneous hardware such as mobile devices (as low as 0.34 GMACs / s) and the cloud (24.71 GMACs / s). The number of parameters ranges from 1.76M (minimal model) to 27.19M (full model), covering two orders of magnitude. Subnetworks are activated on demand during inference, avoiding redundant computation of the full model on simple tasks, improving resource utilization by more than 30%, and significantly improving computational efficiency.

[0102] The embodiments described above in this application support multi-scale feature processing and adaptive mechanisms, and support dynamic control of input width. By adopting the idea of ​​"early exit," it can selectively activate preceding neurons to achieve hierarchical processing of input feature complexity. For simple features, only a small number of neurons are activated for rapid processing, reducing latency (e.g., voice wake-word detection); for complex features, multiple sets can be combined to capture cross-scale information (e.g., time-frequency domain details of reverberant speech); and the same model can simultaneously learn feature representations of different granularities, improving the generalization ability to diverse inputs (e.g., near-field and far-field speech, audio with different signal-to-noise ratios), exhibiting structural flexibility.

[0103] The above embodiments of this application indicate a dynamic balancing strategy for the attention mechanism, which can achieve task-adaptive head number adjustment. For simple tasks (such as low-noise speech denoising), using only 1-2 attention heads can reduce the computational load by 50% to 75% while maintaining real-time performance. For complex tasks (such as strong reverberation + packet loss speech repair), the full number of heads (such as 4 heads) can be activated, and long-distance dependency modeling can be enhanced through multi-head interaction, improving objective indicators (such as SDR) by 2 to 3 dB. This avoids the "overcomputation of simple tasks" or "insufficient modeling of complex tasks" caused by a fixed number of heads, maintaining optimal inference efficiency in different scenarios and achieving a balance between performance and efficiency.

[0104] The embodiments described above in this application simulate complex interferences (such as environmental noise, equipment limitations, and network transmission problems) in actual speech transmission and recording by superimposing various degradation types such as noise, reverberation, clipping, and packet loss. This enables the model to learn general feature representations across scenarios, rather than overfitting to a single degradation type.

[0105] According to an embodiment of the present invention, a model deployment device embodiment is also provided. It should be noted that the model deployment device can be used to execute the model deployment method in the embodiment of the present invention, and the model deployment method in the embodiment of the present invention can be executed in the model deployment device.

[0106] Figure 4 This is a schematic diagram of a model deployment device according to an embodiment of the present invention, such as... Figure 4 As shown, the device may include: an acquisition module 42, used to acquire model training parameters supported by the model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; and a truncating module 44, used to truncate a speech sub-model from a preset speech model based on the model training parameters, wherein the preset speech model employs a speech enhancement streaming neural network, and the speech enhancement streaming neural network includes at least: multiple residual modules connected in series, each residual module including at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes, and the speech sub-model adopts a speech enhancement streaming neural network. The speech enhancement streaming neural network is a segment extracted from the neural network. The speech enhancement streaming neural network includes at least: residual sub-modules connected in series according to a preset number of residuals, the number of residuals being determined according to a preset model depth, and the residual sub-modules being residual modules configured according to a preset model width and a preset number of attention heads; a training module 46, used to train a preset speech model and a speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech sub-model trained based on the speech sub-model, wherein, during the training process, the parameter updates and learning progress of the preset speech model and the speech sub-model are kept synchronized; and a deployment module 48, used to allocate the target speech sub-model to the model deployment device and allocate the target speech model to the cloud server.

[0107] It should be noted that the acquisition module 42 in this embodiment can be used to execute step S102 in this application embodiment, the interception module 44 in this embodiment can be used to execute step S104 in this application embodiment, the training module 46 in this embodiment can be used to execute step S106 in this application embodiment, and the deployment module 48 in this embodiment can be used to execute step S108 in this application embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0108] In the embodiments described above, the model training parameters are determined based on the resources supported by the model deployment device. Then, a matching speech enhancement streaming neural network sub-network is extracted from the preset complete speech enhancement streaming neural network according to the model training parameters to obtain a speech sub-model of the preset speech model. The speech enhancement streaming neural network is designed with multiple cascaded residual modules. Each residual module is equipped with a scalable linear layer, a scalable attention mechanism layer, and a scalable normalization layer. This allows the model to flexibly adjust its structure and computational complexity according to actual hardware conditions without affecting the overall learning progress and parameter updates. Furthermore, by training the preset speech model and its extracted speech sub-models synchronously, the performance consistency of the model is guaranteed on various types of hardware, thereby achieving the technical effect of seamless model deployment from low-resource devices to high-performance servers. This solves the problem in the prior art that speech enhancement models are difficult to dynamically adapt to deployment on devices with different computing capabilities.

[0109] As an optional embodiment, the truncation module includes: a first determining unit, used to determine the number of residual sub-modules required in the speech enhancement streaming neural network based on the preset model depth in the model training parameters, thereby obtaining the number of residuals; a truncation unit, used to truncate the concatenated residual modules in the speech enhancement streaming neural network according to the number of residuals, thereby obtaining multiple residual modules to be adjusted; an adjusting unit, used to adjust the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module to be adjusted according to the preset model width and preset number of attention heads in the model training parameters, thereby obtaining the residual sub-module; and a second determining unit, used to determine the speech sub-model based on the concatenated residual sub-modules according to the number of residuals, wherein the speech enhancement streaming neural network used in the speech sub-model is a concatenated residual sub-module according to the number of residuals.

[0110] As an optional embodiment, the adjustment unit includes: a first determining subunit, used to determine the number of activated neurons in the scalable linear layer based on the preset model width in the model training parameters, to obtain the activation count; a second determining subunit, used to determine the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, to obtain the activation head count; a third determining subunit, used to determine a subset of weight vectors in the scalable normalization layer based on the preset model width in the model training parameters, wherein the subset of weight vectors is truncated from multiple weight vectors in a pre-set weight set, and the weight vectors are used to adjust the combination method between neurons in the scalable linear layer; and a configuration subunit, used to configure each residual module to be adjusted according to the activation count, the activation head count, and the subset of weight vectors, to obtain a residual submodule.

[0111] As an optional embodiment, the apparatus further includes: an acquisition unit, configured to determine the number of attention heads activated in the scalable attention mechanism layer based on a preset number of attention heads in the model training parameters, and after obtaining the number of activated heads, obtain the total feature dimension of the features to be processed by the preset speech model; and a third determination unit, configured to determine the feature dimension allocated to each attention head when performing feature processing based on the ratio of the total feature dimension to the number of activated heads.

[0112] As an optional embodiment, the apparatus further includes: an acquisition submodule, configured to acquire clean speech samples before training a preset speech model and a speech submodel based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech submodel trained based on the speech submodel; a degradation submodule, configured to perform degradation processing on the clean speech samples according to at least one pre-set degradation method to obtain speech distortion samples, wherein the degradation method includes at least: noise, reverberation, clipping, and packet loss; and a combination submodule, configured to combine the clean speech samples and the corresponding speech distortion samples to obtain sample data.

[0113] As an optional embodiment, the degradation submodule includes: a combination unit, used to randomly combine multiple degradation methods according to a pre-configured selection probability to obtain a degradation strategy combination, wherein each degradation method has a pre-set corresponding selection probability; and a degradation unit, used to perform degradation processing on clean speech samples according to the degradation strategy combination to obtain speech distortion samples.

[0114] As an optional embodiment, the degradation submodule includes at least one of the following: a noise unit, used to randomly add pre-acquired noise speech samples to clean speech samples according to a preset signal-to-noise ratio to obtain speech distortion samples; a reverberation unit, used to add reverberation to clean speech samples using pre-acquired house impulse response samples to obtain speech distortion samples, wherein the house impulse response samples are used to describe the propagation characteristics of sound in an enclosed space; a clipping unit, used to clip sound waves in clean speech samples whose waveform amplitude exceeds a preset amplitude threshold to obtain speech distortion samples, wherein the preset amplitude threshold is determined based on waveform amplitudes in clean speech samples whose waveform distribution is not lower than a preset quantile; and a packet loss unit, used to simulate packet loss in clean speech samples using a Markov chain to obtain speech distortion samples.

[0115] Embodiments of the present invention can provide an electronic device, which can be a computer terminal, and the computer terminal can be any one of a group of computer terminal devices. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0116] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0117] In this embodiment, the computer terminal described above can execute program code for the following steps in the model deployment method: obtaining model training parameters supported by the model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; based on the model training parameters, extracting a speech sub-model from a preset speech model, wherein the preset speech model employs a speech enhancement streaming neural network, and the speech enhancement streaming neural network includes at least: multiple residual modules connected in series, each residual module including at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes. The phonological model employs a speech enhancement streaming neural network extracted from a speech enhancement streaming neural network. This sub-network includes at least: residual sub-modules connected in series according to a pre-set number of residuals, the number of residuals being determined based on a pre-set model depth; and residual sub-modules configured according to a pre-set model width and a pre-set number of attention heads. The model and its sub-modules are trained using pre-configured sample data to obtain a target speech model trained based on the pre-set speech model and a target speech sub-model trained based on the speech sub-model. During training, the parameter updates and learning progress of the pre-set speech model and the speech sub-model are synchronized. The target speech sub-model is assigned to the model deployment device, and the target speech model is assigned to the cloud server.

[0118] Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of the present invention, such as... Figure 5 As shown, the computer terminal 50 may include one or more (only one is shown in the figure) processors 52 and memory 54.

[0119] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model deployment method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned model deployment method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal 50 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring model training parameters supported by the model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; based on the model training parameters, extracting a speech sub-model from a preset speech model, wherein the preset speech model employs a speech enhancement streaming neural network, and the speech enhancement streaming neural network includes at least: multiple cascaded residual modules, each residual module including at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes; and so on. The phonological model employs a speech enhancement streaming neural network extracted from a speech enhancement streaming neural network. This sub-network includes at least: residual sub-modules connected in series according to a pre-set number of residuals, the number of residuals being determined based on a pre-set model depth; and residual sub-modules configured according to a pre-set model width and a pre-set number of attention heads. The model and its sub-modules are trained using pre-configured sample data to obtain a target speech model trained based on the pre-set speech model and a target speech sub-model trained based on the speech sub-model. During training, the parameter updates and learning progress of the pre-set speech model and the speech sub-model are synchronized. The target speech sub-model is assigned to the model deployment device, and the target speech model is assigned to the cloud server.

[0121] Optionally, the processor may also execute program code for the following steps: determining the number of residual sub-modules required in the speech enhancement streaming neural network based on the preset model depth in the model training parameters, thus obtaining the number of residuals; in the speech enhancement streaming neural network, extracting the cascaded residual modules according to the number of residuals, thus obtaining multiple residual modules to be adjusted; adjusting the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module to be adjusted according to the preset model width and preset number of attention heads in the model training parameters, thus obtaining the residual sub-module; and determining the speech sub-model based on the cascaded residual sub-modules according to the number of residuals, wherein the speech enhancement streaming neural network used in the speech sub-model is a cascaded residual sub-module according to the number of residuals.

[0122] Optionally, the processor may also execute program code for the following steps: determining the number of activated neurons in the scalable linear layer based on the preset model width in the model training parameters, thus obtaining the activation count; determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, thus obtaining the activation head count; determining a subset of weight vectors in the scalable normalization layer based on the preset model width in the model training parameters, wherein the subset of weight vectors is truncated from multiple weight vectors in a pre-set weight set, and the weight vectors are used to adjust the combination of neurons in the scalable linear layer; and configuring each residual module to be adjusted according to the activation count, the number of activation heads, and the subset of weight vectors, thus obtaining a residual submodule.

[0123] Optionally, the processor may also execute program code that performs the following steps: obtains the total feature dimension of the features to be processed by the preset speech model; and determines the feature dimension assigned to each attention head when performing feature processing based on the ratio of the total feature dimension to the number of activation heads.

[0124] Optionally, the processor may also execute program code for the following steps: acquiring clean speech samples; performing degradation processing on the clean speech samples according to at least one pre-set degradation method to obtain speech distortion samples, wherein the degradation method includes at least: noise, reverberation, clipping and packet loss; and combining the clean speech samples and the corresponding speech distortion samples to obtain sample data.

[0125] Optionally, the processor may also execute program code that performs the following steps: randomly combining multiple degradation methods according to a pre-configured selection probability to obtain a degradation strategy combination, wherein each degradation method has a pre-set corresponding selection probability; and performing degradation processing on clean speech samples according to the degradation strategy combination to obtain speech distortion samples.

[0126] Optionally, the processor may also execute program code for the following steps: randomly adding pre-acquired noisy speech samples to clean speech samples according to a preset signal-to-noise ratio to obtain speech distortion samples; adding reverberation to clean speech samples using pre-acquired house impulse response samples to obtain speech distortion samples, wherein the house impulse response samples are used to describe the propagation characteristics of sound in a closed space; clipping sound waves in clean speech samples whose waveform amplitude exceeds a preset amplitude threshold to obtain speech distortion samples, wherein the preset amplitude threshold is determined based on waveform amplitudes in clean speech samples whose waveform distribution is not lower than a preset quantile; and simulating packet loss in clean speech samples using a Markov chain to obtain speech distortion samples.

[0127] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 50 may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0128] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing the hardware related to the terminal device. The computer program can be stored in a non-volatile medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0129] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the model deployment method provided in the above embodiments.

[0130] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0131] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining model training parameters supported by the model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; and extracting a speech sub-model from a preset speech model based on the model training parameters, wherein the preset speech model employs a speech enhancement streaming neural network, and the speech enhancement streaming neural network includes at least: multiple residual modules connected in series, each residual module including at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes. The speech sub-model adopts a speech enhancement streaming neural network extracted from the speech enhancement streaming neural network. The speech enhancement streaming neural network includes at least: residual sub-modules connected in series according to a preset number of residuals, the number of residuals being determined according to the preset model depth, and residual sub-modules being configured according to the preset model width and the preset number of attention heads; training the preset speech model and speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech sub-model trained based on the speech sub-model. During the training process, the parameter updates and learning progress of the preset speech model and the speech sub-model are kept synchronized; the target speech sub-model is allocated to the model deployment device and the target speech model is allocated to the cloud server.

[0132] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining the number of residual sub-modules required in the speech enhancement streaming neural network based on the preset model depth in the model training parameters, thereby obtaining the number of residuals; in the speech enhancement streaming neural network, extracting the cascaded residual modules according to the number of residuals, thereby obtaining multiple residual modules to be adjusted; adjusting the scalable linear layer, scalable attention mechanism layer, and scalable normalization layer in each residual module to be adjusted according to the preset model width and preset number of attention heads in the model training parameters, thereby obtaining the residual sub-module; determining the speech sub-model based on the cascaded residual sub-modules according to the number of residuals, wherein the speech enhancement streaming neural network used by the speech sub-model is a cascaded residual sub-module according to the number of residuals.

[0133] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining the number of activated neurons in the scalable linear layer based on the preset model width in the model training parameters, thus obtaining the activation count; determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, thus obtaining the activation head count; determining a subset of weight vectors in the scalable normalization layer based on the preset model width in the model training parameters, wherein the subset of weight vectors is truncated from multiple weight vectors in a pre-set weight set, and the weight vectors are used to adjust the combination of neurons in the scalable linear layer; configuring each residual module to be adjusted according to the activation count, the activation head count, and the subset of weight vectors, thus obtaining a residual submodule.

[0134] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining the total feature dimension of the features to be processed by the preset speech model; and determining the feature dimension allocated to each attention head when performing feature processing based on the ratio of the total feature dimension to the number of activation heads.

[0135] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring clean speech samples; performing degradation processing on the clean speech samples according to at least one preset degradation method to obtain speech distortion samples, wherein the degradation method includes at least: noise, reverberation, clipping and packet loss; and combining the clean speech samples and the corresponding speech distortion samples to obtain sample data.

[0136] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: randomly combining multiple degradation methods according to a pre-configured selection probability to obtain a degradation strategy combination, wherein each degradation method has a pre-set corresponding selection probability; and performing degradation processing on clean speech samples according to the degradation strategy combination to obtain speech distortion samples.

[0137] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: randomly adding pre-acquired noisy speech samples to clean speech samples according to a preset signal-to-noise ratio to obtain speech distortion samples; adding reverberation to clean speech samples using pre-acquired house impulse response samples to obtain speech distortion samples, wherein the house impulse response samples are used to describe the propagation characteristics of sound in enclosed spaces; clipping sound waves in clean speech samples whose waveform amplitude exceeds a preset amplitude threshold to obtain speech distortion samples, wherein the preset amplitude threshold is determined based on waveform amplitudes in clean speech samples whose waveform distribution is not lower than a preset quantile; and simulating packet loss in clean speech samples using a Markov chain to obtain speech distortion samples.

[0138] Embodiments of the present invention also provide a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the steps of the model deployment method provided in the above embodiments.

[0139] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0140] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned non-volatile storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0145] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A model deployment method, characterized in that, include: Obtain the model training parameters supported by the model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; Based on the model training parameters, a speech sub-model is extracted from a preset speech model. The preset speech model employs a speech enhancement streaming neural network, which includes at least: multiple cascaded residual modules. Each residual module includes at least: a scalable linear layer for controlling model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes. The speech sub-model is a speech enhancement streaming neural network extracted from the speech enhancement streaming neural network. This sub-network includes at least: residual sub-modules cascaded according to a pre-set number of residuals, the number of residuals being determined based on the preset model depth. Each residual sub-module is configured according to the preset model width and the preset number of attention heads. The preset speech model and speech sub-model are trained based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech sub-model trained based on the speech sub-model. During the training process, the parameter updates and learning progress of the preset speech model and the speech sub-model are kept synchronized. The target speech sub-model is assigned to the model deployment device, and the target speech model is assigned to the cloud server.

2. The method according to claim 1, characterized in that, Based on the model training parameters, extracting a speech sub-model from the preset speech model includes: Based on the preset model depth in the model training parameters, the number of residual sub-modules required in the speech enhancement streaming neural network is determined, and the number of residuals is obtained. In the speech enhancement streaming neural network, the serially connected residual modules are truncated according to the number of residuals to obtain multiple residual modules to be adjusted. Based on the preset model width and the preset number of attention heads in the model training parameters, the scalable linear layer, the scalable attention mechanism layer and the scalable normalization layer in each residual module to be adjusted are adjusted to obtain the residual sub-module; The speech sub-model is determined based on the residual sub-modules connected in series according to the number of residuals, wherein the speech enhancement streaming neural network used in the speech sub-model is the residual sub-modules connected in series according to the number of residuals.

3. The method according to claim 2, characterized in that, Based on the preset model width and preset number of attention heads in the model training parameters, the scalable linear layer, the scalable attention mechanism layer, and the scalable normalization layer in each residual module to be adjusted are adjusted to obtain the residual sub-module, which includes: Based on the preset model width in the model training parameters, the number of activated neurons in the scalable linear layer is determined to obtain the activation count; Based on the preset number of attention heads in the model training parameters, the number of activated attention heads in the scalable attention mechanism layer is determined, and the number of activated heads is obtained. Based on the preset model width in the model training parameters, a subset of weight vectors in the scalable normalized layer is determined, wherein the subset of weight vectors is truncated from multiple weight vectors in a preset weight set, and the weight vectors are used to adjust the combination of neurons in the scalable linear layer. Each residual module to be adjusted is configured according to the number of activations, the number of activation heads, and the subset of weight vectors to obtain the residual submodule.

4. The method according to claim 1, characterized in that, After determining the number of activated attention heads in the scalable attention mechanism layer based on the preset number of attention heads in the model training parameters, the method further includes: Obtain the total feature dimension of the features to be processed by the preset speech model; Based on the ratio of the total feature dimension to the number of activation heads, the feature dimension allocated to each attention head during feature processing is determined.

5. The method according to claim 1, characterized in that, Before training the preset speech model and speech sub-model based on pre-configured sample data to obtain the target speech model trained based on the preset speech model and the target speech sub-model trained based on the speech sub-model, the method further includes: Obtain clean speech samples; The clean speech sample is degraded according to at least one pre-set degradation method to obtain a speech distortion sample, wherein the degradation method includes at least: noise, reverberation, clipping and packet loss; The clean speech sample and the corresponding distorted speech sample are combined to obtain the sample data.

6. The method according to claim 5, characterized in that, The clean speech samples are degraded according to at least one pre-set degradation method to obtain speech distortion samples, including: Multiple degradation methods are randomly combined according to a pre-configured selection probability to obtain a degradation strategy combination, wherein each degradation method has a pre-set corresponding selection probability; The clean speech samples are subjected to degradation processing according to a combination of degradation strategies to obtain distorted speech samples.

7. The method according to claim 5, characterized in that, The clean speech samples are degraded according to at least one pre-set degradation method to obtain speech distortion samples, including at least one of the following: According to a preset signal-to-noise ratio, a pre-acquired noisy speech sample is randomly added to the clean speech sample to obtain the speech distortion sample; Using pre-acquired house impulse response samples, reverberation is added to the clean speech sample to obtain the speech distortion sample, wherein the house impulse response samples are used to describe the propagation characteristics of sound in an enclosed space; The sound waves in the clean speech sample whose waveform amplitude exceeds a preset amplitude threshold are clipped to obtain the speech distortion sample. The preset amplitude threshold is determined based on the waveform amplitude in the clean speech sample whose waveform distribution is not lower than a preset quantile. The distorted speech sample is obtained by simulating packet loss in the clean speech sample using a Markov chain.

8. A model deployment device, characterized in that, include: The acquisition module is used to acquire model training parameters supported by the model deployment device, wherein the model training parameters include at least: a preset model depth, a preset model width, and a preset number of attention heads pre-configured for the model deployment device; The truncation module is used to truncate a speech sub-model from a preset speech model based on the model training parameters. The preset speech model employs a speech enhancement streaming neural network, which includes at least: multiple cascaded residual modules. Each residual module includes at least: a scalable linear layer for controlling the model width, a scalable attention mechanism layer for selecting the number of attention heads, and a scalable normalization layer for maintaining model stability when the model width changes. The speech sub-model is a speech enhancement streaming neural network truncated from the speech enhancement streaming neural network. The speech enhancement streaming neural network includes at least: residual sub-modules cascaded according to a preset number of residuals, the number of residuals being determined based on the preset model depth. The residual sub-modules are configured according to the preset model width and the preset number of attention heads. The training module is used to train the preset speech model and the speech sub-model based on pre-configured sample data to obtain a target speech model trained based on the preset speech model and a target speech sub-model trained based on the speech sub-model. During the training process, the parameter updates and learning progress of the preset speech model and the speech sub-model are kept synchronized. The deployment module is used to allocate the target speech sub-model to the model deployment device and to allocate the target speech model to the cloud server.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the model deployment method of any one of claims 1 to 7 through the computer program.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the model deployment method according to any one of claims 1 to 7.

Images