Audio processing method, computing device, server, storage medium, and program product

Abstract

The application provides an audio processing method, a computing device, a server, a storage medium and a program product, which relates to the field of artificial intelligence, inputs a mixed audio into a mixed audio speech separation model, converts the mixed audio into audio features; performs focus linear attention-based processing on the audio features to obtain mixed hidden layer features; performs speech separation on the mixed hidden layer features to obtain single-speaker hidden layer features of each speaker; generates speech of each speaker according to the single-speaker hidden layer features of each speaker; using the focus linear attention mechanism, the complexity of attention calculation of the speech separation model can be reduced from quadratic complexity O(N 2 ) to linear complexity O(N), thereby reducing the calculation complexity and memory occupation of the speech separation model and improving the inference efficiency of the speech separation model; especially when processing long speech sequences, the calculation complexity and memory occupation are greatly reduced, and the inference efficiency is greatly improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an audio processing method, computing device, server, storage medium, and program product. Background Technology

[0002] Speech separation refers to extracting each individual speech source (i.e., speech) from a single-channel audio mixture to solve the cocktail party problem, playing a crucial role in downstream speech processing and applications.

[0003] With the development of artificial neural network technology, the introduction of neural networks into speech separation tasks has led to significant progress. Speech separation models have also gradually been divided into time-domain models and time-frequency domain models. Existing speech separation models have achieved excellent performance. However, unlike other speech sequence tasks, speech separation requires processing extremely long audio sequences. Modeling extremely long speech sequences results in extremely high computational complexity and slow inference efficiency.

[0004] Existing speech separation models employ methods such as fixed receptive fields, chunking, skipping, and downsampling to shorten the feature sequence length when processing long-range global information. However, this leads to the loss of some global information. Furthermore, even with downsampling techniques, these solutions still have a computational complexity of O(N^2) when introducing attention mechanisms. 2 Specifically, when processing long audio sequences, the processing time of speech separation models increases proportionally to the square of the input sequence length N. Therefore, existing attention-based speech separation models suffer from high computational complexity, low inference efficiency, and high memory consumption when processing long speech sequences. Summary of the Invention

[0005] This application provides an audio processing method, computing device, server, storage medium, and program product to solve the problems of high computational complexity, low inference efficiency, and high memory usage in existing speech separation models when processing long speech sequences.

[0006] In a first aspect, this application provides an audio processing method, including:

[0007] Obtain a mixed audio of multiple speakers' speech; input the mixed audio into a speech separation model based on focused linear attention and perform the following speech separation processing: convert the mixed audio into audio features; process the audio features based on focused linear attention to obtain mixed hidden layer features; perform speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; generate the speech of each speaker based on the individual speaker hidden layer features of each speaker.

[0008] Secondly, this application provides an audio processing method, including:

[0009] In response to a call to the speech separation model, acquire the mixed audio of the mixed multi-speaker speech to be processed;

[0010] The mixed audio is input into the speech separation model for speech separation processing to obtain the speech of each speaker; wherein the speech separation processing includes: converting the mixed audio into audio features; performing gating-focused linear attention processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain the individual speaker hidden layer features of each speaker; and generating the speech of each speaker based on the individual speaker hidden layer features of each speaker.

[0011] Output the voice recordings of each speaker.

[0012] Thirdly, this application provides an audio processing method, including:

[0013] Obtain the mixed conference audio to be separated, wherein the mixed conference audio is a mixed audio of multiple speakers' speech;

[0014] The mixed conference audio is input into a speech separation model for speech separation processing to obtain the conference speech of each speaker; wherein the speech separation processing includes: converting the mixed conference audio into audio features; performing gating-focused linear attention processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain the individual speaker hidden layer features of each speaker; and generating the conference speech of each speaker based on the individual speaker hidden layer features of each speaker.

[0015] Output the conference audio of each speaker.

[0016] Fourthly, this application provides a computing device, comprising:

[0017] A memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the methods provided by any of the foregoing aspects.

[0018] Fifthly, this application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the server to perform the methods provided in any of the foregoing aspects.

[0019] Sixthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method provided in any of the foregoing aspects.

[0020] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the foregoing aspects.

[0021] The audio processing method, computing device, server, storage medium, and program product provided in this application utilize a speech separation model based on focused linear attention to achieve speech separation. Specifically, a mixed audio of multiple speakers is acquired, and the mixed audio is input into a speech separation model for the following speech separation processing: the mixed audio is converted into audio features; the audio features are processed based on focused linear attention to obtain mixed hidden layer features; speech separation is performed on the mixed hidden layer features to obtain individual speaker hidden layer features; and the individual speaker hidden layer features are used to generate the speech of each speaker. Since the computational complexity of the focused linear attention mechanism is linear O(N), using the focused linear attention mechanism in the speech separation model reduces the complexity of attention calculation in the speech separation model from the original quadratic complexity O(N). 2 The computational complexity is reduced to linear O(N), which reduces the computational complexity and memory usage of the speech separation model and improves the inference efficiency of the speech separation model. When the speech separation model processes long speech sequences (i.e., when N is large), the computational complexity and memory usage are further reduced, and the inference efficiency is greatly improved. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A schematic diagram of the speech separation system architecture provided in this application;

[0024] Figure 2 A flowchart of an audio processing method provided as an exemplary embodiment of this application;

[0025] Figure 3 A flowchart illustrating the gating and focusing linear attention process provided in an exemplary embodiment of this application;

[0026] Figure 4 A schematic diagram of the structure of a gated focusing linear attention block provided in an exemplary embodiment of this application;

[0027] Figure 5A schematic diagram of the structure of a time-domain speech separation model based on gated focusing linear attention blocks, provided as an exemplary embodiment of this application;

[0028] Figure 6 A flowchart of speech separation processing provided as an exemplary embodiment of this application;

[0029] Figure 7 A schematic diagram of the structure of a time-frequency domain speech separation model based on a gated focusing linear attention block provided in an exemplary embodiment of this application;

[0030] Figure 8 A flowchart of speech separation processing provided as another exemplary embodiment of this application;

[0031] Figure 9 A flowchart of an audio processing method provided as another exemplary embodiment of this application;

[0032] Figure 10 A flowchart of an audio processing method provided as another exemplary embodiment of this application;

[0033] Figure 11 This is a structural block diagram of a computing device according to an embodiment of this application;

[0034] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application.

[0035] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0038] First, let me explain the terms used in this application:

[0039] SepReformer is an improved temporal speech separation model based on a pure Transformer architecture. It achieves speech separation using an asymmetric strategy with decoupled encoders and decoders. The encoder analyzes features and divides its output into segments corresponding to the number of speakers to be separated. The weight-shared decoder reconstructs the separated sequences. Using a common speaker segmentation layer, the reconstruction decoder, based on a U-Network architecture, also utilizes features from the intermediate encoder. Instead of dividing and processing the feature sequence into blocks, it achieves higher computational efficiency and better performance by directly processing long sequences through global and local transformer blocks.

[0040] TF-Locoformer is a speech separation model based on the Transformer architecture and incorporating convolutional layers for local modeling. This model achieves speech separation by introducing convolutional layers for local modeling and utilizing the Transformer's self-attention mechanism to capture global patterns. This model avoids the cyclical nature of traditional RNNs, improving the parallelism and scalability of model computation.

[0041] Current speech separation models are divided into time-domain models and time-frequency domain models. Early time-frequency domain models used deep clustering and permutation-invariant training (PIT) methods to solve the speech separation problem in the time-frequency domain. A more recent and advanced time-frequency domain model, TF-GridNet, achieves better performance by combining a Long Short-Term Memory (LSTM) network with an attention mechanism. Building on this, a new time-frequency domain model, TF-locoformer, further introduces the Transformer architecture to improve speech separation performance.

[0042] In temporal model research, the Convolutional Temporal Speech Separation Network (ConvTasNet) introduces neural network structures such as Temporal Convolutional Network (TCN) and LSTM into temporal models. Furthermore, the Dual-Path Recurrent Neural Network (DPRNN) and the Dual-Path Transformer Network (DPTNet) utilize LSTM and Transformer networks respectively in their dual-path modeling schemes. The Sepformer model is an advanced temporal speech separation model that achieves higher computational efficiency and better performance by replacing the Recurrent Neural Network (RNN) in DPRNN with an attention-based Transformer.

[0043] While LSTM-based speech separation models have linear computational complexity, they suffer from the inability to effectively capture global information. Transformer-based speech separation models offer higher performance and better computational efficiency. However, the calculation method for the attention mechanism in Transformer-based speech separation models can be expressed as follows: (1)

[0044]

[0045] Where X represents the feature sequence, and N represents the length of the feature sequence X. q W k and W v Let be a linear mapping matrix. Let i represent the index of the feature sequence.

[0046] Current speech separation models typically use a softmax attention mechanism as the kernel function for the introduced attention mechanism. exp represents the exponential function, and both Q and K have dimensions of N×d, where N is the length of the feature sequence and d is the feature dimension. Matrix operations (QK) are performed during attention calculation. T The computational complexity of is O(N). 2 When processing long audio sequences, the processing time of speech separation models increases proportionally to the square of the input sequence length N. This results in high computational complexity, low inference efficiency, and high memory consumption for speech separation models when processing long speech sequences.

[0047] The kernel function used in the linear attention mechanism is: The calculation method of the linear attention mechanism can be expressed as the following formula (2):

[0048]

[0049] First calculate by applying the distributive property of multiplication. The calculation method for the linear attention mechanism is adjusted from formula (2) to the following formula (3):

[0050]

[0051] Attention is calculated based on formula (3), and the computational complexity of linear attention is O(N).

[0052] However, existing linear attention mechanisms suffer from a trade-off between accuracy and performance. Using simple activation functions like ReLU (Rectified Linear Unit) and ELU (Exponential Linear Unit) fails to effectively model the long-range relationship between Q and K, leading to overly flat attention weights and a sharp performance drop. Using complex kernel functions may result in increased computational complexity. Compared to Softmax Attention, linear attention suffers from an overly flat attention weight matrix, making it difficult to effectively aggregate important features from long sequences for different Q (query features). Furthermore, in long-range relationships such as speech separation, linear attention may fail to effectively model feature diversity.

[0053] In this embodiment, Focused Linear Attention (FLA) is introduced into the speech separation task. Focused Linear Attention uses a new kernel function (called the focusing function) to enhance the model's ability to focus on different features. The focusing function can be expressed as the following formula (4):

[0054]

[0055] Where, x **p This indicates that the elements in feature x are calculated to the power of p. p is the focus factor, and its value can be set and adjusted according to the needs of the actual application scenario. By setting an appropriate focus factor p, similar Q (query features) and K (key features) can be brought closer together, while dissimilar Q and K can be separated. This allows linear attention to approximate the probability distribution of softmax attention, especially the sharp distribution, thereby achieving the modeling effect for long-range important speech features.

[0056] Compared to traditional Vanilla Linear Attention, Focused Linear Attention (FLA) maintains linear complexity and reduces computational complexity compared to the quadratic complexity of Softmax Attention, while still being able to model long-range speech features. Efficient speech separation models can be built based on Focused Linear Attention (FLA), reducing the inference complexity of speech separation models.

[0057] This application provides a speech separation model based on focused linear attention, replacing the Softmax Attention used in the speech separation model with focused linear attention (FLA). Focused linear attention adds a mechanism to the existing focused linear attention model; FLA is a linear attention mechanism with very low computational complexity. This reduces the complexity of attention calculation in the speech separation model from quadratic complexity O(N^2) to O(N^2). 2 The computational complexity is reduced to linear O(N), achieving both parallel training and linear inference time growth. This significantly reduces the computational complexity, inference efficiency, and memory usage of attention-based speech separation models when processing long speech sequences.

[0058] This application provides an audio processing method that uses a speech separation model based on focused linear attention to achieve speech separation. Specifically, it acquires mixed audio of multiple speakers' speech and inputs the mixed audio into the speech separation model for the following speech separation processing: converting the mixed audio into audio features; processing the audio features based on focused linear attention to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the speech of each speaker based on the individual speaker hidden layer features. By using the focused linear attention mechanism, the complexity of attention calculation in the speech separation model can be reduced from quadratic complexity O(N^2) to O(N^2). 2 The computational complexity is reduced to linear O(N), which reduces the computational complexity and memory usage of the speech separation model and improves the inference efficiency of the speech separation model. When the speech separation model processes long speech sequences, the computational complexity and memory usage are significantly reduced, and the inference efficiency is significantly improved.

[0059] Because speech separation models significantly reduce computational complexity and memory usage when processing long speech sequences, they greatly improve inference efficiency. When performing speech separation, the original audio can be divided into longer audio segments, and speech separation can be performed on each of these longer audio segments. The longer audio segments provide the model with more long-range information, which can improve the accuracy of speech separation.

[0060] Figure 1This is a schematic diagram of the speech separation system architecture provided in this application. Figure 1 As shown, the system architecture includes edge devices and servers. The edge devices and servers have a communication link that allows them to communicate with each other.

[0061] The server is a computing device deployed in the cloud or locally, such as a cloud cluster. The server stores a speech separation model based on focused linear attention and is responsible for separating the speech of mixed audio from multiple speakers using this model to obtain the speech of each speaker.

[0062] The edge device can be an electronic device running an upstream application / system that generates mixed audio of mixed multi-speaker speech, or an electronic device used by the user. Specifically, it can be a hardware device with network communication, computing, and information display functions, including but not limited to smartphones, tablets, desktop computers, local servers, and cloud servers. When speech separation is required, the edge device sends the mixed audio of the mixed multi-speaker speech to be separated to the server.

[0063] The server receives mixed audio from the terminal device, which contains multiple speakers' speech. It then inputs the mixed audio into a speech separation model for speech separation processing to obtain the speech of each speaker. Specifically, the speech separation process includes: converting the mixed audio into audio features; performing linear attention-based processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain the individual speaker hidden layer features for each speaker; and generating the speech of each speaker based on their individual speaker hidden layer features.

[0064] Furthermore, the server returns the voice recordings of each speaker to the end-user device. Further, the end-user device can perform speech-to-text conversion, translation, assisted dubbing, and audio restoration on the voice recordings of each speaker according to the needs of the upstream application / system; specific limitations are not specified here.

[0065] It should be noted that the solution in this embodiment can be applied to speech separation of mixed audio data generated in various scenarios. For example, in an online conferencing system, it performs real-time speech separation on multi-speaker, overlapping conference audio to obtain the speech of a single speaker, and generates real-time subtitles and translated text based on the single speaker's speech, thereby improving the accuracy of real-time subtitle generation and translation. Of course, this solution can also be applied to offline speech separation of conference audio.

[0066] For example, in a voice assistant, a user's voice input in a noisy environment with multiple speakers may include the voices of other speakers in the noisy environment. This solution can separate the user's clean speech from the voice input. Based on the clean speech of the target speaker, user commands can be accurately recognized, thereby reducing the false wake-up rate.

[0067] For example, in a customer service quality inspection scenario, this solution can separate the overlapping audio of customer service and user calls, obtaining the customer service representative's voice and the user's voice. Based on the customer service representative's voice, emotion analysis or speech-to-text transcription can be performed, and based on the user's voice, emotion analysis or speech-to-text transcription can be performed, thereby improving the efficiency of speech-to-text transcription and emotion analysis.

[0068] For example, in film and television post-production scenarios, this solution can separate a single audio track from a recorded mixed audio track, which can assist in dubbing and audio restoration.

[0069] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0070] Figure 2 This is a flowchart illustrating an exemplary embodiment of the audio processing method provided in this application. The execution entity in this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0071] Step S201: Obtain the mixed audio of multiple speakers' speech.

[0072] The server acquires the mixed audio to be separated into multiple speakers (referred to as speech). The mixed audio can be mixed audio data generated in various scenarios, including but not limited to: conference audio with multiple speakers overlapping, user voice input in a noisy environment with multiple people talking, communication records between customer service and users, and mixed audio tracks recorded in movie scenes.

[0073] The solution in this embodiment aims to perform speech separation on mixed audio containing multiple speakers to obtain the speech of each speaker.

[0074] It should be noted that a single speech separation process is typically applied to a relatively short audio segment. If the mixed audio to be separated is long, it can be divided into multiple shorter audio segments. Speech separation is performed on one audio segment at a time to obtain the speech segments of each speaker. By sequentially concatenating the speech segments of the same speaker, the speaker's complete speech can be obtained.

[0075] Step S202: Perform speech separation processing on the mixed audio input based on the focused linear attention speech separation model to obtain the speech of each speaker.

[0076] In this embodiment, an end-to-end speech separation model based on focused linear attention is used to achieve speech separation. Specifically, mixed audio is input into the speech separation model, and the model performs speech separation processing on the input mixed audio to obtain the speech of each speaker.

[0077] For example, the speech separation model includes a focused linear attention block, which is used to process audio features based on focused linear attention. Any type of focused linear attention (FLA) block can be used to process audio features based on focused linear attention; no specific limitation is made here.

[0078] The speech separation model in this embodiment can be obtained as follows: Take any attention-based speech separation model as the base model, and replace the quadratic complexity attention blocks (such as SoftmaxAttention) in the base model with focused linear attention blocks to obtain a speech separation model based on focused linear attention. Training this focused linear attention-based speech separation model yields the speech separation model used in this embodiment. When the base model is a time-domain model, the speech separation model in this embodiment is a time-domain speech separation model. When the base model is a time-frequency domain model, the speech separation model in this embodiment is a time-frequency domain speech separation model.

[0079] In this step, the speech separation processing based on the focused linear attention speech separation model involves the following steps:

[0080] Step S2021: Convert the mixed audio into audio features.

[0081] In this step, for the mixed audio input to the speech separation model, the speech separation model converts the mixed audio into audio features. These converted audio features are discrete audio feature sequences, also known as audio token sequences, which can be represented as X, where X has a dimension of F×T, denoted as X∈R. F×T Where F represents the dimension of the audio feature, and T represents the length of the audio feature sequence, i.e., the number of frames of the audio feature.

[0082] The speech separation model includes an input layer that converts mixed audio into audio features.

[0083] In one alternative implementation, the speech separation model is a temporal model, and the input layer may include one-dimensional convolution, GELU (Gaussian Error Linear Units) activation, a linear layer, and layer normalization. This input layer converts the mixed input audio into audio features.

[0084] In one optional implementation, the speech separation model is a time-frequency domain model, and the input layer is used to implement Short-Time Fourier Transform (STFT) and feature transformation. The input layer performs a STFT on the mixed audio to convert the one-dimensional audio into a feature sequence X, and then transforms the feature sequence X into hidden layer features to obtain the audio features.

[0085] It should be noted that the speech separation model used in this embodiment can be applied to various basic models for speech separation. Different basic models may use different input layers, and the methods for converting mixed audio into audio features through the input layer will also be different. No specific limitations are made here.

[0086] Step S2022: Perform focus-based linear attention processing on the audio features to obtain hybrid hidden layer features.

[0087] In this step, the speech separation model uses focused linear attention blocks to process audio features based on focused linear attention, thereby modeling global information with linear complexity and obtaining hidden layer features containing global information.

[0088] To distinguish it from the hidden features obtained after speech separation, the hidden features obtained in this step are called mixed hidden features, while the hidden features obtained after speech separation are called single speaker hidden features.

[0089] In one optional implementation, the speech separation model is a temporal model, connected to an encoding module after the input layer. The encoding module consists of at least one stacked layer of encoding blocks. Each encoding block includes one or more attention blocks with different attention mechanisms. In this embodiment, the encoding blocks of the speech separation model include focused linear attention blocks for processing based on focused linear attention.

[0090] In this step, the audio features output from the input layer are fed into the encoding module for encoding. During this encoding process, the audio features are processed using a focused linear attention block. The hidden layer features output from the last layer of the encoding module are used as the hybrid hidden layer features.

[0091] In one optional implementation, the speech separation model is a time-frequency domain model, connected to a time-frequency domain modeling module after the input layer. The time-frequency domain modeling module consists of at least one stacked layer of time-frequency domain modeling modules. Each layer includes a frequency domain modeling module and a time domain modeling module. The time domain modeling module is based on gated linear attention and includes a linear attention block, enabling linear attention-based processing.

[0092] In this step, the audio features output from the input layer are fed into the time-frequency domain modeling module for modeling. During this modeling process, the audio features are processed using a focused linear attention block. The hidden layer features output from the last layer of the time-frequency domain modeling module are used as the hybrid hidden layer features.

[0093] It should be noted that the speech separation model used in this embodiment can be applied to various basic models for speech separation. The process of encoding / modeling to obtain hybrid hidden layer features may differ for different basic models, and no specific limitation is made here.

[0094] Step S2023: Perform speech separation on the mixed hidden layer features to obtain the individual speaker hidden layer features of each speaker.

[0095] Both time-domain and time-frequency domain speech separation models have speech separators. Speech separators are used to separate mixed hidden features into speech, outputting the hidden features corresponding to a single speaker, which are called single-speaker hidden features.

[0096] It should be noted that the speech separation model used in this embodiment can be applied to various basic models for speech separation. Different basic models may use different speech separators, and the methods for speech separation by mixing hidden layer features will also be different. No specific limitations are made here.

[0097] Step S2024: Generate the speech of each speaker based on the single speaker hidden layer features of each speaker.

[0098] In this step, the speech separation model converts the single-speaker hidden layer features of each speaker into speech signals to obtain the speech of each speaker.

[0099] In one optional implementation, the speech separation model is a temporal model, with a decoding module and an output layer connected after the speech separator. The decoding module includes multiple decoding branches, each corresponding to a different speaker. Different decoding branches have the same architecture. Each decoding branch consists of at least one layer of stacked decoding blocks. Each decoding block includes an attention block with one or more different attention mechanisms. In this embodiment, the decoding blocks of the speech separation model include focused linear attention blocks for processing based on focused linear attention.

[0100] In this step, the speech separator outputs the individual speaker hidden features for each speaker, which are then input into different decoding branches for decoding. During this decoding process, the individual speaker hidden features are processed using focused linear attention blocks. The hidden features output from the last layer of each decoding branch are used as the decoded features for each speaker.

[0101] Furthermore, the decoded features of each speaker are converted into speech signals through the output layer to obtain the speech of each speaker. For example, the output layer may include modules such as one-dimensional deconvolution, which are responsible for reconstructing the hidden layer features of the separated individual speakers into speech signals to obtain the speech of the individual speaker.

[0102] It should be noted that the speech separation model used in this embodiment can be applied to various time-domain basic models for speech separation. Different time-domain basic models may use different output layers, and the methods for converting decoded features into speech signals through the output layer will also be different. No specific limitations are made here.

[0103] In one optional implementation, the speech separation model is a time-frequency domain model, with an output layer connected after the speech separator. The output layer is used to implement the inverse short-time Fourier transform (iSTFT). By performing the inverse short-time Fourier transform on the individual speaker hidden features of each speaker through the output layer, the individual speaker hidden features of each speaker are converted into speech signals, thus obtaining the speech of each speaker.

[0104] In this embodiment, after obtaining the voice recordings of each speaker, the server can output the voice recordings of each speaker. For example, it can send the voice recordings of each speaker to the end device; or it can push the voice recordings of each speaker to a designated device according to a preset push rule, etc. The specific output method is not limited here. The preset push rule can be configured and adjusted according to actual application needs, and is not specifically limited here.

[0105] The scheme in this embodiment performs the following speech separation processing on the mixed audio input to the speech separation model, which contains multiple speakers' voices: converting the mixed audio into audio features; processing the audio features based on focused linear attention to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the speech of each speaker based on the individual speaker hidden layer features. By using a focused linear attention mechanism, the complexity of attention calculation in the speech separation model can be reduced from quadratic complexity O(N^2) to O(N^2). 2The computational complexity is reduced to linear O(N), which reduces the computational complexity and memory usage of the speech separation model and improves the inference efficiency of the speech separation model. When the speech separation model processes long speech sequences, the computational complexity and memory usage are significantly reduced, and the inference efficiency is significantly improved.

[0106] In addition, since the speech separation model significantly reduces computational complexity and memory usage when processing long speech sequences, it greatly improves inference efficiency. When performing speech separation, the original audio can be divided into longer audio segments, and speech separation can be performed on each of the longer audio segments. The longer audio segments provide the model with more long-range information, which can improve the accuracy of speech separation.

[0107] In an optional embodiment, the focused linear attention block in the speech separation model can use a gated focused linear attention (Gated FLA) block. The speech separation model includes a gated focused linear attention block. The gated focused linear attention block is used to process audio features using gated focused linear attention. Specifically, the gated focused linear attention block includes a focused linear attention module and a gating module. The focused linear attention module is used to: perform focused linear attention calculations on the audio features to obtain initial hidden layer features. The gating module is used to: calculate the gating coefficients corresponding to the initial hidden layer features based on the audio features. Further, based on the initial hidden layer features and the gating coefficients, a mixed hidden layer feature is determined. For example, the initial hidden layer feature is multiplied by the gating coefficient to obtain the mixed hidden layer feature.

[0108] Figure 3 This is a flowchart of the gating and focusing linear attention processing provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of the gated focusing linear attention block provided in this embodiment. Figure 3 As shown, the aforementioned step S2022 can be implemented using the following steps:

[0109] Step S301: Process the audio features based on focused linear attention to obtain the initial hidden layer features.

[0110] In this step, the audio features of the input gated linear attention (Gated FLA) block (which can be the initial audio features or the hidden features output by the intermediate layer between the input and output layers) are processed by the linear attention module in the gated FLA block to obtain the initial hidden features.

[0111] The focused linear attention module within the gated focused linear attention (Gated FLA) block can be implemented using any type of focused linear attention (FLA) block, for example... Figure 4 The linear attention module shown here is not specifically limited.

[0112] like Figure 4 As shown, the focused linear attention module includes multiple attention heads (h in the figure represents the number of attention heads) and linear layers. Each attention head includes three linear layers, a focused linear attention layer, and a depthwise convolution (DWC) module. The three linear layers are used to linearly map the audio features of the input Gated FLA block into query features (Q), key features (K), and value features (V).

[0113] The focused linear attention layer performs focused linear attention calculations on the query feature (Q), key feature (K), and value feature (V), outputting the attention feature after the focused linear attention calculation. Specifically, as follows... Figure 4 As shown, the focused linear attention layer includes a kernel function, addition, matrix multiplication, and division. Specifically, the focused linear attention layer uses the focus function of formula (4) as the kernel function to perform the attention calculation of formula (3) and output attention features.

[0114] The Deep Convolution (DWC) module employs a one-dimensional deep convolution module with a one-dimensional convolution kernel. The DWC module performs deep convolution on the audio features of the input Gated FLA block to obtain an attention matrix with the same full rank as the original Softmax Attention.

[0115] Traditional linear attention modules use two-dimensional deep convolutional modules (DWC2d, with a kernel size of k*k) to focus on local modeling information of value features in two-dimensional images. In speech separation tasks, the deep convolution (DWC) module is adjusted to a one-dimensional deep convolutional module (DWC1d, with a kernel size of k) to model local feature information between one-dimensional speech features.

[0116] Compared to Softmax Attention, linear attention, besides constructing sharper attention weights, also needs to supplement the diversity of feature weights. The upper bound of the rank of linear attention depends on the number of features N and the feature dimension d: In speech separation tasks, d is often much smaller than N (for example, in a speech separation model, inputting 1 second of audio results in d=16 and N=2000 for the hidden layer features). This leads to a much smaller feature diversity in linear attention compared to conventional Softmax Attention. Traditional linear attention uses two-dimensional deep convolutional modules to supplement the rank of the attention weights.

[0117] In this embodiment, the linear attention module utilizes a one-dimensional deep convolution (DWC) module. This enhances the feature diversity of the attention weights and supplements φ(Q)φ(K) by using a DWC1d attention matrix with the same full rank as the original Softmax Attention. T The missing rank will be used to supplement the final attention matrix to the original upper bound rank; on the other hand, it can be used as a local attention mechanism to enhance the information of different speech features by using the neighboring features in the time domain or time-frequency domain.

[0118] Furthermore, the full-rank attention matrix output by the depthwise convolution (DWC) module is added to the attention features output by each attention head of the linear attention layer to obtain the full-rank attention features corresponding to each attention head.

[0119] The full-rank attention features corresponding to each attention head are concatenated and processed by a linear layer to obtain the initial hidden layer features of the focused linear attention module.

[0120] Step S302: Calculate the gating coefficients corresponding to the initial hidden layer features based on the audio features.

[0121] In this step, the gating module in the Gated FLA block calculates the gating coefficients corresponding to the initial hidden features based on the audio features of the input Gated FLA block (which can be the initial audio features or the hidden features output from the intermediate layer between the input and output layers).

[0122] The gating coefficients corresponding to the initial hidden features are matrices with the same dimensions as the initial hidden features. The values ​​in the matrix represent the weighting coefficients corresponding to features of different dimensions in the initial hidden features.

[0123] Step S302 and step S301 can be performed in parallel to improve the efficiency of audio processing.

[0124] To further enhance the modeling capability of focused linear attention, a gating mechanism is added to improve the output of the focused linear attention module. The gating module can use a gated multi-layer perceptron (Gated MLP), which includes a normalization layer, a linear layer, and an activation function. The audio features of the input Gated FLA block are sequentially processed through the normalization layer, the linear layer, and the activation function to obtain the gating coefficients corresponding to the initial hidden layer features.

[0125] Step S303: Determine the hybrid hidden layer features based on the initial hidden layer features and gating coefficients.

[0126] In this step, the initial hidden layer features are multiplied by the gating coefficients to obtain the output features of the GatedFLA block.

[0127] By introducing a gating mechanism, we can further utilize the features of the current speech to control the global feature interaction results.

[0128] In this embodiment, the speech separation model uses gated linear attention (GLA) blocks to process audio features based on gated linear attention. This not only reduces the complexity of attention calculation in the speech separation model from quadratic complexity O(N)... 2 The computational complexity is reduced to linear O(N), thereby reducing the computational complexity and memory usage of the speech separation model and improving the inference efficiency of the speech separation model. Moreover, by introducing a gating mechanism, the information of the current features can be better utilized to control the interaction of global information, which enhances the performance of the focus attention mechanism.

[0129] In one alternative embodiment, the speech separation model based on gated focused linear attention blocks can be a time-domain model. Figure 5 This is a schematic diagram of the structure of a temporal speech separation model based on gated focusing linear attention blocks, provided as an exemplary embodiment of this application. (See diagram below.) Figure 5 As shown, the speech separation model includes: a first input layer, an encoder based on gated focusing linear attention, a first speech separator, a decoder based on gated focusing linear attention, and a first output layer.

[0130] The first input layer is the input layer of the speech separation model. This input layer may include one-dimensional convolution, GELU activation, linear layers, and layer normalization. This input layer converts the mixed input audio into audio features.

[0131] like Figure 5As shown, the encoder based on gated focusing linear attention comprises (R+1) layers (represented by "×R+1" in the figure) of stacked coding blocks. Each coding block consists of multiple stacked structural units, including local coding modules (Local Transformers) and global coding modules (Global FLA Transformers) based on the gated focusing linear attention mechanism. The number of stacked structural units including local and global coding modules in a coding block is denoted as B. E The first R-layer coding block also includes a downsampling module, which is the last layer of the coding block. The first R-layer coding block is used to implement feature encoding and downsampling processing. The last layer coding block may not include a downsampling module (the downsampling module is shown with a dashed border in the diagram, indicating that not every coding block contains a downsampling module).

[0132] For example, the downsampling module can be implemented using downsampling convolutional operations. For instance, the downsampling module may include a one-dimensional depthwise convolutional (Dconv) layer with a stride of 2 and a kernel size of 5, batch normalization, and GeLU activation. In the encoder, the feature sequence is downsampled R times consecutively, from sequence length T to T / 2R.

[0133] The Global FLA Transformer, based on a gated linear attention mechanism, includes a gated linear attention block (Gated FLA) and a gated convolutional feed-forward network (GCFN). The gated FLA is responsible for modeling global information with linear complexity; its implementation principle is described in the previous embodiments and will not be repeated here. The GCFN is responsible for modeling local information using depthwise convolutions and pointwise convolutions.

[0134] The Local Transformer module includes a Convolutional Local Attention (CLA) block and a Gated Convolutional Feedforward Neural Network (GCFN). The CLA block contains larger convolutional kernels responsible for modeling longer-scale speech features.

[0135] The first speech separator includes two linear layers and a normalization layer, both equipped with gated linear units (GLUs) activation.

[0136] In a gated, focused linear attention encoder, the audio features before downsampling are input into the first speaker splitter for initial separation of the speaker's speech features in each stacked coding block. Early separation of speech features from different speakers allows features at different scales to be transmitted to the decoder in advance.

[0137] The decoder based on gated focusing linear attention includes multiple decoding branches corresponding to different speakers. Each decoding branch is used to perform gated focusing linear attention-based decoding processing on the second hidden layer features of any speaker to obtain the decoded features of that speaker.

[0138] Each decoding branch consists of stacked decoding blocks of R layers. One of these decoding blocks includes: a global decoding module based on a gated linear attention mechanism, a local decoding module, and a cross-speaker decoding module. For example... Figure 5 As shown, a decoding block can be composed of multiple stacked basic units. Each basic unit includes a global decoding module, a local decoding module, and a cross-speaker decoding module based on a gated focusing linear attention mechanism. The number of the aforementioned basic units stacked in a decoding block can be denoted as B. D .

[0139] The global decoding module includes a gated linear attention block and a gated convolutional feedforward neural network. The local decoding module includes a convolutional local attention block and a gated convolutional feedforward neural network. Global decoding modules at the same layer in different decoding branches share parameters, as do local decoding modules at the same layer in different decoding branches. The global decoding module has the same structure as the global encoding module in the encoder, and the local decoding module has the same structure as the local encoding module in the encoder; these details will not be elaborated further here.

[0140] The cross-speaker decoding module (CS-Transformer) includes a self-attention block and a feedforward neural network. During the decoding separation process between the global and local decoding modules, where decoding branch weights are shared, speech elements may be incorrectly concentrated in other speaker channels. Therefore, to effectively recover distorted speech elements, the cross-speaker decoding module performs attention operations on the speaker dimension, learning to identify interference components in opposing speaker sequences within the same time frame, thus improving the accuracy of speech separation.

[0141] Each decoded block has a preprocessing module before the stacked structure, which includes upsampling, concatenation, and linear layers. The upsampling module can be implemented using upsampling convolution operations with a stride of 2. In the decoder, the feature sequence is upsampled R times consecutively, from sequence length T / 2R to T.

[0142] For any decoding block in any decoding branch, the input features of the current decoding block are upsampled by the preprocessing layer (the input features of the first decoding block are the features separated by the first speech separator from the output features of the (R+1)th coding block). The upsampled result is concatenated with the input features of the current decoding block and processed by a linear layer before being input to the subsequent global decoding module. This process then sequentially passes through the global decoding module, the local decoding module, and the cross-speaker decoding module to obtain the output features of that decoding block. The output features of the previous decoding block serve as the input features of the next decoding block. The output features of the final decoding block are the speaker's decoded features.

[0143] It should be noted that the different decoding branches in the decoder have the same structure. Figure 5 The structure of one of the decoding branches is used as an example for illustration. Figure 5 The speech separation model shown is a modification of the temporal model SepReformer. The quadratic complexity attention block (Efficient Global Attention) in SepReformer is replaced with a gated linear attention block (Gated FLA). The specific implementation principles of the rest remain unchanged and will not be elaborated here. SepReformer's Efficient Global Attention uses downsampled multi-head self-attention (MHSA). Although downsampling reduces the sequence length to 1 / 2R, the computational complexity is still quadratic O(N). 2 This scheme uses gated linear attention blocks (Gated FLA) to model long-range features without downsampling, and can also reduce the computational complexity to linear complexity O(N), thereby reducing the computational complexity and memory usage of the speech separation model and improving the inference efficiency of the speech separation model.

[0144] In an alternative embodiment, Figure 5 The gated linear attention block (GatedFLA) in the speech separation model shown can be replaced with a linear attention block (FLA) to obtain a temporal speech separation model based on linear attention blocks. This model includes a first input layer, a linear attention-based encoder, a first speech separator, a linear attention-based decoder, and a first output layer. The speech separation model based on linear attention blocks achieves speech separation, and also reduces the complexity of attention calculation in the speech separation model from quadratic complexity O(N). 2 The computational complexity is reduced to linear O(N), thereby reducing the computational complexity and memory usage of the speech separation model and improving the inference efficiency of the speech separation model.

[0145] Figure 6 This is a flowchart of the speech separation process provided in this embodiment. Based on... Figure 5 The speech separation model shown is as follows: Figure 6 As shown, in step S202 above, the specific steps for performing speech separation processing on the mixed audio input based on the focused linear attention speech separation model are as follows:

[0146] Step S601: Input the mixed audio into the first input layer and convert the mixed audio into the first audio feature.

[0147] In this step, the mixed audio is input into the first input layer of the speech separation model. The first input layer converts the mixed audio into audio features, obtaining the audio features of the mixed audio, which are denoted here as the first audio features. The implementation principle of the first input layer can be found in the relevant content of the aforementioned embodiments, and will not be repeated here.

[0148] Step S602: The first audio feature is encoded using an encoder based on gated focusing linear attention to obtain the first hidden layer feature.

[0149] In this step, the first audio feature output from the first input layer is input to the encoder. The encoder performs gating-focused linear attention-based encoding processing on the first audio feature, outputting a hybrid hidden layer feature (denoted as the first hidden layer feature), which includes the features in each coding block of the encoder before downsampling. The implementation principle of the encoder is described in the relevant content of the aforementioned embodiments, and will not be repeated here.

[0150] Step S603: Using the first speech separator, perform speech separation on the first hidden layer features to obtain the second hidden layer features of each speaker.

[0151] In this step, the first hidden layer feature output by the encoder is input to the first speech separator. The first speech separator performs speech separation on the first hidden layer feature to obtain the individual speaker hidden layer feature of each speaker, which is denoted as the second hidden layer feature. The implementation principle of the first speech separator is described in the relevant content of the aforementioned embodiment, and will not be repeated here.

[0152] Furthermore, through steps S604-S605, the speech of each speaker is generated based on the single speaker hidden layer features (i.e., the second hidden layer features) of each speaker.

[0153] Step S604: The second hidden layer features of each speaker are decoded using a gated focusing linear attention method to obtain the decoded features of each speaker.

[0154] The second hidden layer features of each speaker separated by the first speech separator are input into different decoding branches of the decoder. The second hidden layer features of the speakers are decoded through the decoding branches to obtain the decoded features of the speakers. The implementation principle of the decoder is described in the relevant content of the aforementioned embodiments, and will not be repeated here.

[0155] Step S605: Convert the decoded features of each speaker into the speech of each speaker through the first output layer.

[0156] The decoded features of the speaker output from each decoding branch of the decoder are input to the first output layer. The first output layer converts the decoded features of each speaker into speech signals to obtain the speech of each speaker. The implementation principle of the first output layer is described in the relevant content of the aforementioned embodiments and will not be repeated here.

[0157] This embodiment provides a temporal speech separation model based on gated linear attention (Gated FLA). This model enables speech separation of mixed audio from multiple speakers, reducing the complexity of attention calculations in the speech separation model from quadratic complexity O(N^2). 2 The computational complexity is reduced to linear O(N), lowering the computational complexity and memory usage of the speech separation model while improving its inference efficiency. This is particularly beneficial when processing long speech sequences, significantly reducing computational complexity and memory usage and improving inference efficiency. Furthermore, because the computational complexity and memory usage are significantly reduced and inference efficiency is greatly improved when processing long speech sequences, the original audio can be segmented into longer audio segments for speech separation. These longer segments provide the model with more long-range information, thus improving the accuracy of speech separation.

[0158] In an optional embodiment, the speech separation model based on gated focused linear attention blocks can be a time-frequency domain model. Figure 7 This is a schematic diagram of the structure of a time-frequency domain speech separation model based on gated focusing linear attention blocks, provided as an exemplary embodiment of this application. Figure 7 As shown, the speech separation model includes: a second input layer, a time-frequency domain modeling module based on gated focusing linear attention, a second speech separator, and a second output layer.

[0159] The second input layer is the input layer of the speech separation model. This input layer may include a Short-Time Fourier Transform (STFT) layer and a feature transformation layer. The STFT layer performs a Short-Time Fourier Transform (STFT) on the input mixed audio, converting the mixed audio into a feature sequence X∈R. F×TFurthermore, the feature sequence X is converted into audio features (denoted as the second audio feature) through a feature transformation layer. The dimensions of the second audio feature are B×T×F×D, where B is the batch size, T represents the number of time frames of the feature, F represents the frequency domain dimension, and D represents the feature dimension.

[0160] For example, the feature transformation layer can use two-dimensional convolution (Conv2D) and global layer normalization (gLN).

[0161] like Figure 7 As shown, the time-frequency domain modeling module based on gated focusing linear attention includes B layers of stacked time-frequency domain modeling modules. Each layer of time-frequency domain modeling modules includes a frequency domain modeling module and a time domain modeling module based on gated focusing linear attention.

[0162] The frequency modeling module includes: a pair of convolutional layers, a normalization layer, and a multi-head self-attention (MHSA) block. The pair of convolutional layers can be a pair of convolutionally gated linear units (Conv-SwiGLUs). Conv-SwiGLU is a convolutional layer that uses a hybrid activation function combining the Swish activation function and gated linear units (GLUs). Conv-SwiGLU uses the SwiGLU activation function, combining the Swish activation function and GLUs, and performs better than using only the Swish activation function. Placing ConvSwiGLU before and after the attention mechanism allows for stronger local modeling.

[0163] The temporal module based on gated linear attention includes: two convolutional layers in pairs, a normalization layer, and a gated linear attention block (Gated FLA). The two convolutional layers in pairs can be a pair of convolutional gated linear units (Conv-SwiGLU).

[0164] It should be noted that frequency modeling treats F as the sequence length for modeling, while temporal modeling treats T as the sequence length for modeling. Since the value of F is fixed, the inference time of the frequency modeling part is linear. Therefore... Figure 7In the speech separation model shown, the frequency modeling module still uses the MHSA block. In an alternative embodiment, the MHSA block in the frequency modeling module can also be replaced with a gated focused linear attention block (Gated FLA).

[0165] The second audio feature output from the second input layer is processed by the time-frequency domain construction module stacked in layer B to obtain the third hidden layer feature.

[0166] The second speech separator can use a two-dimensional deconvolutional layer (Deconv2D) to separate the fourth hidden layer features of each speaker by deconvolving the features of the third hidden layer.

[0167] The second output layer is an Inverse Short-Time Fourier Transform (iSTFT) layer. The iSTFT layer performs an inverse short-time fourier transform (iSTFT) on the fourth hidden layer features of each speaker, converting the fourth hidden layer features of each speaker into speech signals, thus obtaining the speech of each speaker.

[0168] It should be noted that, Figure 7 The speech separation model shown is a modification of the time-frequency domain model TF-locoformer. The quadratic complexity attention block (MHSA in Temporal modeling, or MHSA in both Frequency modeling and Temporal modeling) in TF-locoformer is replaced with a gated linear attention block (Gated FLA). The specific implementation principles of the remaining parts remain unchanged and will not be elaborated here. Furthermore, although MHSA in TF-locoformer uses STFT to reduce the sequence length of speech features, its computational complexity is still O(N). 2 This scheme uses gated linear attention blocks (Gated FLA) to reduce the quadratic complexity of the original MHSA computation to O(N). 2 The computational complexity is reduced to linear O(N), thereby reducing the computational complexity and memory usage of the speech separation model and improving the inference efficiency of the speech separation model.

[0169] In an alternative embodiment, Figure 7The gated linear attention block (GatedFLA) in the speech separation model can be replaced with a focused linear attention block (FLA) to obtain a time-frequency domain speech separation model based on focused linear attention blocks. This model consists of a second input layer, a time-frequency domain modeling module based on focused linear attention, a second speech separator, and a second output layer. The time-frequency domain speech separation model based on focused linear attention blocks achieves speech separation, and the complexity of attention calculation in the speech separation model can be reduced from quadratic complexity O(N). 2 The computational complexity is reduced to linear O(N), thereby reducing the computational complexity and memory usage of the speech separation model and improving the inference efficiency of the speech separation model.

[0170] Figure 8 This is a flowchart of the speech separation process provided in this embodiment. Based on... Figure 7 The speech separation model shown is as follows: Figure 8 As shown, in step S202 above, the specific steps for performing speech separation processing on the mixed audio input based on the focused linear attention speech separation model are as follows:

[0171] Step S801: Input the mixed audio into the second input layer and convert the mixed audio into a second audio feature.

[0172] In this step, the mixed audio is input into the second input layer of the speech separation model. The mixed audio is then converted into audio features, denoted as the second audio features. The implementation principle of the second input layer is detailed in the aforementioned embodiments and will not be repeated here.

[0173] Step S802: Through the time-frequency domain modeling module, the second audio features are processed based on self-attention and gated focusing linear attention to obtain the third hidden layer features.

[0174] In this step, the second audio features output from the second input layer are input to the time-frequency domain modeling module. The time-frequency domain modeling module processes the second audio features based on self-attention and gated focusing linear attention to obtain hybrid hidden layer features, denoted as the third hidden layer features. The implementation principle of the time-frequency domain modeling module is described in the relevant content of the aforementioned embodiments, and will not be repeated here.

[0175] Step S803: Using the second speech separator, perform speech separation on the third hidden layer features to obtain the fourth hidden layer features of each speaker.

[0176] In this step, the second hidden layer features output by the time-frequency domain modeling module are input into the second speech separator. The second speech separator performs speech separation on the third hidden layer features to obtain the individual speaker hidden layer features of each speaker, which are denoted as the fourth hidden layer features. The implementation principle of the second speech separator is described in the relevant content of the aforementioned embodiments, and will not be repeated here.

[0177] Step S804: Convert the fourth hidden layer features of each speaker into the speech of each speaker through the second output layer.

[0178] In this step, the fourth hidden layer features of each speaker are converted into speech signals through the second output layer to obtain the speech of each speaker. The implementation principle of the second output layer is described in the relevant content of the aforementioned embodiments, and will not be repeated here.

[0179] This embodiment provides a time-frequency domain speech separation model based on gated linear attention (Gated FLA). This model enables speech separation of mixed audio from multiple speakers, reducing the complexity of attention calculations in the speech separation model from quadratic complexity O(N^2). 2 The computational complexity is reduced to linear O(N), lowering the computational complexity and memory usage of the speech separation model while improving its inference efficiency. This is particularly beneficial when processing long speech sequences, significantly reducing computational complexity and memory usage and improving inference efficiency. Furthermore, because the computational complexity and memory usage are significantly reduced and inference efficiency is greatly improved when processing long speech sequences, the original audio can be segmented into longer audio segments for speech separation. These longer segments provide the model with more long-range information, thus improving the accuracy of speech separation.

[0180] Figure 9 A flowchart illustrating an exemplary embodiment of this application. Figure 9 As shown, the specific steps of this method are as follows:

[0181] Step S901: In response to the call request to the speech separation model, obtain the mixed audio of the mixed multi-speaker speech to be processed.

[0182] Based on the speech separation model provided in any of the foregoing embodiments, this embodiment can deploy the speech separation model on a server. The server can provide an application programming interface (API) for the speech separation model to one or more end devices. The end devices call the speech separation model according to the API to achieve speech separation.

[0183] In practical applications, when speech separation is required, the edge device can send a request to the server to invoke the speech separation model, using the mixed audio of the multi-speaker speech to be processed as input parameters, based on the speech separation model's API. This request contains the mixed audio of the multi-speaker speech to be processed.

[0184] The server responds to a call request for a speech separation model by retrieving the mixed audio of the mixed multi-speaker speech to be processed from the call request.

[0185] Step S902: Perform speech separation processing on the mixed audio input speech separation model to obtain the speech of each speaker.

[0186] In this embodiment, the speech separation process includes: converting the mixed audio into audio features; performing gated linear attention-based processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the speech of each speaker based on the individual speaker hidden layer features. For the specific implementation principle, please refer to the relevant content of the foregoing embodiments, which will not be repeated here.

[0187] Step S903: Output the voice of each speaker.

[0188] In this step, the server returns the voice recordings of each speaker to the end device that sent the call request.

[0189] The scheme in this embodiment performs the following speech separation processing on the mixed audio input to the speech separation model, which contains multiple speakers' voices: converting the mixed audio into audio features; processing the audio features based on focused linear attention to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the speech of each speaker based on the individual speaker hidden layer features. By using a focused linear attention mechanism, the complexity of attention calculation in the speech separation model can be reduced from quadratic complexity O(N^2) to O(N^2). 2 The computational complexity is reduced to linear O(N), which reduces the computational complexity and memory usage of the speech separation model and improves the inference efficiency of the speech separation model. When the speech separation model processes long speech sequences, the computational complexity and memory usage are significantly reduced, and the inference efficiency is significantly improved.

[0190] In addition, since the speech separation model significantly reduces computational complexity and memory usage when processing long speech sequences, it greatly improves inference efficiency. When performing speech separation, the original audio can be divided into longer audio segments, and speech separation can be performed on each of the longer audio segments. The longer audio segments provide the model with more long-range information, which can improve the accuracy of speech separation.

[0191] The solution in this embodiment can be applied to speech separation of mixed audio data generated in various scenarios. The following description uses speech separation of conference audio as an example.

[0192] Figure 10 A flowchart illustrating an exemplary embodiment of this application. Figure 10 As shown, the specific steps of this method are as follows:

[0193] Step S1001: Obtain the mixed conference audio to be separated. The mixed conference audio is a mixed audio of multiple speakers' voices.

[0194] The mixed conference audio can be either real-time audio generated by the online conferencing system or pre-recorded audio. Conference audio often contains multiple speakers, and is a mixed audio stream where the voices of these speakers are superimposed together.

[0195] It should be noted that a single speech separation process is typically applied to a relatively short audio segment. If the mixed audio to be separated is long, it can be divided into multiple shorter audio segments. Speech separation is performed on one audio segment at a time to obtain the speech segments of each speaker. By sequentially concatenating the speech segments of the same speaker, the speaker's complete speech can be obtained.

[0196] Step S1002: Perform speech separation processing on the mixed conference audio input speech separation model to obtain the conference speech of each speaker.

[0197] In this embodiment, the speech separation process includes: converting the mixed conference audio into audio features; performing gating-focused linear attention processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the conference speech for each speaker based on the individual speaker hidden layer features for each speaker.

[0198] The implementation principle of this step can be found in the relevant content of the foregoing embodiments, and will not be repeated here.

[0199] Step S1003: Output the conference audio of each speaker.

[0200] In this step, the server outputs the conference audio of each speaker.

[0201] For example, the server can store the conference audio of each speaker to a specified storage location. The specified storage location can be set according to the actual application requirements of the conference system, and is not specifically limited here.

[0202] For example, the server can push conference audio from each speaker to designated devices according to preset push rules; the output method is not specifically limited here. The preset push rules can be configured and adjusted according to actual application needs, and are not specifically limited here.

[0203] In this embodiment, the mixed conference audio with multiple speakers is input into a speech separation model and processed as follows: the mixed conference audio is converted into audio features; the audio features are processed based on focused linear attention to obtain mixed hidden layer features; speech separation is performed on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and the conference audio for each speaker is generated based on the individual speaker hidden layer features. By using a focused linear attention mechanism, the complexity of attention calculation in the speech separation model can be reduced from quadratic complexity O(N^2) to O(N^2). 2 The computational complexity is reduced to linear O(N), which reduces the computational complexity and memory usage of the speech separation model and improves the inference efficiency of the speech separation model. When the speech separation model processes long speech sequences, the computational complexity and memory usage are significantly reduced, which greatly improves the efficiency of speech separation in conferences.

[0204] In addition, since the speech separation model significantly reduces computational complexity and memory usage when processing long speech sequences, it greatly improves inference efficiency. When performing speech separation, the original audio can be divided into longer audio segments, and speech separation can be performed on each of the longer audio segments. The longer audio segments provide the model with more long-range information, which can improve the accuracy of speech separation.

[0205] Figure 11 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 11 As shown, the computing device 1100 may include one or more (only one is shown in the figure) processors 1101 and memory 1102. The memory 1102 stores computer programs / instructions, and the processor 1101 executes the computer programs / instructions. When the computer programs / instructions are executed by the processor 1101, they implement the technical solutions provided in any of the aforementioned method embodiments. Their specific functions and the technical effects they can achieve are similar and will not be repeated here.

[0206] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, mobile phones, tablet computers, or other portable smart terminals, and the computing device may have the model in the above embodiments of this application pre-installed.

[0207] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0208] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0209] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 12 As shown, the server includes a memory 1201 and a processor 1202. The memory 1201 stores computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1202 is communicatively connected to the memory 1201 and executes the computer-executable instructions stored in the memory 1201 to implement the technical solutions provided in any of the above method embodiments. Their specific functions and the technical effects they achieve are similar and will not be repeated here.

[0210] Optional, such as Figure 12 As shown, the server also includes other components such as a firewall 1203, a load balancer 1204, a communication component 1205, and a power supply component 1206. Figure 12 The diagram only shows a portion of the components and does not imply that the server only includes... Figure 12 The components shown. Figure 12 This example uses a cloud server deployed in the cloud as an example, but the server can also be deployed locally. This embodiment does not make any specific limitations here.

[0211] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0212] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the server to perform the technical solution provided in any of the above method embodiments. The specific functions and the technical effects that can be achieved are not described here.

[0213] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solution of the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided in any of the aforementioned method embodiments.

[0214] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0215] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in at least one processor.

[0216] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0217] The aforementioned storage device can be object storage service (OSS).

[0218] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0219] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / Long Term Evolution (LTE), fifth-generation (5G), or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), infrared, Ultra Wide Band (UWB), Bluetooth, and other technologies.

[0220] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0221] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0222] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0223] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0224] The order of the embodiments described above is merely for illustrative purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0225] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0226] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0227] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An audio processing method, characterized in that, include: Acquire mixed audio of multiple speakers' speech; The mixed audio input is processed by a speech separation model based on focused linear attention as follows: Convert the mixed audio into audio features; The audio features are processed using focused linear attention to obtain hybrid hidden layer features; Speech separation is performed on the hybrid hidden layer features to obtain the individual speaker hidden layer features of each speaker; The speech of each speaker is generated based on the single-speaker hidden layer features of each speaker.

2. The method according to claim 1, characterized in that, The speech separation model includes a focused linear attention block. The focused linear attention block is used to process the audio features based on focused linear attention.

3. The method according to claim 1, characterized in that, The speech separation model includes a gated focused linear attention block, which comprises a focused linear attention module and a gating module. The processing of the audio features based on focused linear attention to obtain hybrid hidden layer features includes: Specifically, the audio features are processed by the focused linear attention module to obtain the initial hidden layer features. The gating module calculates the gating coefficients corresponding to the initial hidden layer features based on the audio features. Based on the initial hidden layer features and the gating coefficients, the hybrid hidden layer features are determined.

4. The method according to claim 3, characterized in that, The depth convolution module in the focused linear attention module is a one-dimensional depth convolution module.

5. The method according to any one of claims 1-4, characterized in that, The speech separation model includes: a first input layer, a focused linear attention-based encoder, a first speech separator, a focused linear attention-based decoder, and a first output layer. The step of performing speech separation processing on the mixed audio input based on a focused linear attention speech separation model includes: The mixed audio is input into the first input layer and converted into a first audio feature; The encoder performs a focused linear attention-based encoding process on the first audio feature to obtain a first hidden layer feature, which is the hybrid hidden layer feature. The first speech separator performs speech separation on the first hidden layer features to obtain the second hidden layer features of each speaker. The decoder performs a focused linear attention-based decoding process on the second hidden layer features of each speaker to obtain the decoded features of each speaker. The first output layer converts the decoded features of each speaker into their speech.

6. The method according to claim 5, characterized in that, The encoder comprises multiple stacked coding blocks, each including a local coding module and a global coding module based on a focused linear attention mechanism. The global encoding module includes: a focused linear attention block and a gated convolutional feedforward neural network; The local coding module includes: a convolutional local attention block and a gated convolutional feedforward neural network.

7. The method according to claim 5, characterized in that, The decoder includes multiple decoding branches corresponding to speakers. One of the decoding branches is used to perform a focused linear attention-based decoding process on the second hidden layer features of any speaker to obtain the decoded features of the speaker. Each of the decoding branches includes multi-layered stacked decoding blocks, the decoding blocks including: a global decoding module based on a focused linear attention mechanism, a local decoding module, and a cross-speaker decoding module; The global decoding module includes: a focused linear attention block and a gated convolutional feedforward neural network; The local decoding module includes: a convolutional local attention block and a gated convolutional feedforward neural network; The cross-speaker decoding module includes: a self-attention block and a feedforward neural network; Global decoding modules at the same layer in different decoding branches share parameters, and local decoding modules at the same layer in different decoding branches also share parameters.

8. The method according to any one of claims 1-4, characterized in that, The speech separation model includes: a second input layer, a time-frequency domain modeling module based on focused linear attention, a second speech separator, and a second output layer. The step of performing speech separation processing on the mixed audio input based on a focused linear attention speech separation model includes: The mixed audio is input into the second input layer and converted into a second audio feature; The second audio feature is processed by the time-frequency domain modeling module based on self-attention and focused linear attention to obtain the third hidden layer feature, which is the hybrid hidden layer feature. The second speech separator performs speech separation on the third hidden layer features to obtain the fourth hidden layer features of each speaker; The fourth hidden layer features of each speaker are converted into the speech of each speaker through the second output layer.

9. The method according to claim 8, characterized in that, The time-frequency domain modeling module includes a frequency domain modeling module and a time domain modeling module based on focused linear attention. The time domain modeling module includes a convolutional layer and a focused linear attention block.

10. An audio processing method, characterized in that, include: In response to a call to the speech separation model, acquire the mixed audio of the mixed multi-speaker speech to be processed; The mixed audio is input into the speech separation model for speech separation processing to obtain the speech of each speaker; The speech separation process includes: converting the mixed audio into audio features; performing gated linear attention-based processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the speech of each speaker based on the individual speaker hidden layer features. Output the voice recordings of each speaker.

11. An audio processing method, characterized in that, include: Obtain the mixed conference audio to be separated, wherein the mixed conference audio is a mixed audio of multiple speakers' speech; The mixed conference audio input speech separation model is subjected to speech separation processing to obtain the conference speech of each speaker; The speech separation process includes: converting the mixed conference audio into audio features; performing gating-focused linear attention processing on the audio features to obtain mixed hidden layer features; performing speech separation on the mixed hidden layer features to obtain individual speaker hidden layer features for each speaker; and generating the conference audio for each speaker based on the individual speaker hidden layer features. Output the conference audio of each speaker.

12. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein when the computer programs / instructions are executed by the processor, they implement the method according to any one of claims 1-11.

13. A server, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the server to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.

Images