Audio-oriented speaker separation method and device based on deep neural network, and medium

Abstract

The invention belongs to the technical field of deep learning, and particularly relates to an audio-oriented speaker separation method and device based on a deep neural network, and a medium, which can improve the quality and definition of separated voice signals of various applications in an audio processing and communication system. Relates to a lightweight human body posture estimation algorithm and a video stream-oriented human body posture key point smoothing method, and solves the defects of decoupling of phase and amplitude of signals, suboptimum of voice separation time-frequency representation, long delay during spectrogram calculation and the like in the prior art. On the basis that the processing speed not lower than the general video frame rate is provided, the accuracy of attitude estimation is guaranteed, a smoothing method is provided for the problem that jitter errors are prone to occurring when human body estimation is conducted on the video stream, the jitter errors of key points of figure instances in the video stream are effectively reduced, and the accuracy of human body estimation is improved. And the accuracy and smoothness of human body estimation are improved.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and speech processing technology, specifically relating to a speaker separation method, device, and medium based on deep neural networks for audio. Background Technology

[0002] In noisy, multi-speaker environments, the human auditory system can focus on and identify specific sound sources—a phenomenon known as the "cocktail party effect." However, enabling machines to possess similar capabilities is an extremely challenging task. Traditional speaker separation methods primarily rely on signal processing techniques such as spectral subtraction, independent component analysis (ICA), and models based on computational auditory scene analysis. These methods typically assume that sound sources are statistically independent or that prior information is known. However, in complex real-world scenarios, their separation performance is limited, and they struggle to handle overlapping speech and non-stationary noise, often resulting in unsatisfactory output speech quality and intelligibility.

[0003] In recent years, the rise of deep learning has brought revolutionary breakthroughs to speech separation tasks. Unlike traditional methods that rely on fixed rules, deep learning models can automatically learn and extract robust acoustic feature representations from massive amounts of speech data. Through end-to-end training, these models can directly map clean speech from individual speakers in mixed audio, significantly improving the accuracy and robustness of separation. Deep learning-based speech separation can be achieved mainly through several approaches: first, by enhancing the speech features of the target speaker and suppressing other interferences, thereby improving its clarity and intelligibility; second, by directly suppressing background noise and the voices of other speakers to improve the signal-to-noise ratio of the target speech; and third, in multi-speaker scenarios, by separating the mixed audio into multiple independent mono streams, each containing only the speech of one speaker.

[0004] The advantages of deep learning in this field are mainly reflected in the following aspects: First, deep neural networks possess powerful nonlinear modeling capabilities, enabling them to learn complex structures and patterns in speech signals, thereby achieving more refined separation. Second, thanks to modern computing hardware, models can be trained using large-scale datasets, resulting in excellent generalization performance and adaptability to diverse acoustic environments. Finally, deep learning frameworks have good compatibility, allowing for flexible integration with traditional signal processing techniques or other machine learning modules to form hybrid systems that further enhance overall performance. Despite significant progress, designing more efficient, lightweight deep neural network models suitable for low-computing-power devices remains an important direction in current research and applications. Summary of the Invention

[0005] In view of this, the present invention provides a deep neural network-based speaker separation method, device, and medium for audio, which can improve the quality and clarity of separated speech signals in various applications of audio processing and communication systems.

[0006] To achieve the objectives of this invention, the following technical solutions are provided.

[0007] A deep neural network-based speaker separation method for audio, comprising: Encoder processing steps: The input mixed speech waveform is segmented into overlapping short segments through a one-dimensional convolution operation, and the short segments are mapped to a high-dimensional feature space to obtain a high-dimensional feature representation; The separation module processing steps are as follows: The high-dimensional feature representation is received and processed using a convolutional separation module constructed based on a temporal convolutional network; the convolutional separation module contains multiple stacked one-dimensional dilated convolutional blocks, each with a different dilation rate; in the convolutional separation module, depthwise separable convolution is used instead of standard convolution, and residual connections and skip connections are introduced; finally, a multiplication mask is estimated for each target speech source through point convolutional layers. Decoder processing steps: Perform element-wise multiplication on the multiplication mask and the high-dimensional feature representation output by the encoder to obtain the masked features. Then, reconstruct the masked features back to the time-domain waveform through a one-dimensional transpose convolution operation. Finally, sum the overlapping reconstructed segments and output the separated single speech source signal.

[0008] The specific operations of the depthwise separable convolution include: Depth convolution stage: Apply a convolution kernel independently to each channel of the input feature map to perform a convolution operation and generate a depth feature map; Point convolution stage: After the depth convolution stage, all channels of the depth feature map are linearly combined using 1x1 convolution kernels to mix channel information and generate the final output feature map.

[0009] The specific operations of the one-dimensional dilated convolutional block include: For the input one-dimensional sequence, perform dilated convolution operation using a one-dimensional convolution kernel with a predetermined dilation rate; A nonlinear activation function is applied after the dilated convolution operation; Optionally, the input of the dilated convolution operation can be added to the output after the activation function to form a residual connection; Multiple one-dimensional dilated convolutional blocks with different dilation rates are stacked.

[0010] The residual connection is used to directly pass the input of the one-dimensional dilated convolution block to its output for addition.

[0011] The skip connection is used to sum the outputs of all or part of the one-dimensional dilated convolutional blocks as the final feature output of the convolution separation module.

[0012] The training process includes a step of applying normalization after each or a specific layer of the convolutional separation module to stabilize the training process.

[0013] The present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the audio-oriented speaker separation method based on deep neural networks described in the present invention.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio-oriented speaker separation method based on a deep neural network as described in the present invention.

[0015] Beneficial effects 1. This invention relates to a lightweight human pose estimation algorithm and a smoothing method for key points of human pose in video streams, addressing shortcomings in existing technologies such as decoupling of signal phase and amplitude, suboptimal time-frequency representation of speech separation, and long delays in calculating spectrograms. This invention is applicable to low-computing-power devices, providing processing speeds no less than typical video frame rates while ensuring the accuracy of pose estimation. Furthermore, it proposes a smoothing method to address the jitter error problem that easily occurs in human pose estimation of video streams, effectively reducing jitter errors in key points of human figures in the video stream and improving the accuracy and smoothness of human pose estimation.

[0016] 2. This invention directly processes the original speech waveform in the time domain and adopts an end-to-end architecture of encoder-separation module-decoder. This avoids the information loss and performance bottleneck caused by phase and amplitude decoupling and the suboptimal nature of time-frequency representation in traditional frequency domain methods. It can more completely preserve and utilize all the information of the speech signal, thereby directly improving the quality and clarity of the separated speech.

[0017] 3. In this invention, the core of the separation module adopts a lightweight network design. By using depthwise separable convolutions, the number of model parameters and computational complexity are significantly reduced, making the model easier to deploy on edge or mobile devices with limited computing power. Simultaneously, by utilizing one-dimensional dilated convolutional blocks stacked at different dilation rates, the receptive field of the network can be effectively expanded without a significant increase in parameters, accurately capturing long-range temporal dependencies in speech signals. This is crucial for understanding continuous speech content and separating overlapping speech.

[0018] 4. This invention integrates residual connections and skip connections. Residual connections effectively alleviate the vanishing gradient problem commonly encountered in deep network training, ensuring stable training and rapid convergence. Skip connections, on the other hand, enable the effective fusion of features from different network layers, allowing shallow detail features to be organically combined with deep semantic features. This enhances the model's ability to model global and local speech features, further optimizing separation accuracy.

[0019] 5. The method of this invention is compact and efficient, minimizing processing latency. The encoder processes short segments, and combined with the parallelization potential and lightweight structure of the separation module, this invention is not only suitable for offline voice file processing, but also meets the needs of real-time voice communication, conferencing systems, intelligent assistants and other application scenarios with strict low latency requirements, thus having broad application prospects. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention.

[0021] Figure 2 This is a schematic diagram of the principle of the audio-oriented one-dimensional convolution module in this invention.

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] This invention provides a deep neural network-based speaker separation method for audio, and the method flow is as follows: Figure 1 As shown, it includes: S1, the encoder maps a segment of the mixed waveform to a high-dimensional representation, using a one-dimensional convolution operation to segment the input signal into overlapping segments and transform them into representations in a high-dimensional feature space. These feature representations are optimized to facilitate the separation of different speakers.

[0025] Furthermore, individual source signals are estimated by analyzing the discrete waveforms of the mixed signal in the time domain through speech separation. By estimating individual source signals from the mixed audio signal, the aim is to improve the quality and clarity of separated speech signals for various applications in audio processing and communication systems.

[0026] In this embodiment, the convolutional encoder-decoder architecture is as follows: the encoder converts short segments of the mixed speech waveform into a representation in an intermediate feature space; the separation module estimates a multiplication function (mask) for each source at each time step; and the decoder reconstructs the source waveform from the features encoded by the mask. The convolutional separation module is built on a temporal convolutional network and utilizes a series of one-dimensional dilated convolutions to capture long-term dependencies in the speech signal. Specifically, the convolutional separation module also includes a one-dimensional dilated convolution block, which allows the network to increase its receptive field without increasing the number of parameters, thereby capturing longer-term dependencies.

[0027] A depthwise separable convolution is employed, which decomposes standard convolution into two consecutive operations: depthwise convolution and pointwise convolution. The depthwise convolution independently applies a filter to each input channel, while the pointwise convolution blends the feature channels based on the depthwise convolution.

[0028] This paper employs a residual connection technique for neural network neuron connections. The core idea of ​​residual connections is to allow signals in the network to bypass one or more layers and propagate directly. This means that the inputs of some layers can directly "skip" certain processing steps and combine with the outputs of subsequent layers. This helps gradients flow more efficiently during backpropagation in the network. Residual connections allow the network to learn the residuals between inputs and outputs, which helps solve the gradient vanishing problem in deep networks.

[0029] A skip connection is employed in the neural network neuron connection. The core idea of ​​skip connections is to allow signals in the network to skip one or more processing layers. This means that the output of some layers can be directly connected to subsequent layers, bypassing some intermediate layers. This design helps to: 1) Improve gradient flow: Improve gradient flow in deep networks and reduce the problems of vanishing or exploding gradients. 2) Fuse features: Fuse features at different stages of the network to enhance the model's ability to learn features. Skip connections sum the outputs of all convolutional blocks and then use the result as the final output of the temporal convolutional neural network.

[0030] A point-convolution-based mask estimation method is adopted, using a 1x1 convolutional layer at the end of the temporal convolutional neural network to estimate the mask of each target speaker.

[0031] S2 utilizes a separation module to compute a multiplication function (i.e., a mask) for each target source, employing a temporal convolutional network consisting of multiple one-dimensional dilated convolutional blocks, each with a different dilation rate to capture information at different time scales. Subsequently, depthwise separable convolutions are used to reduce the number of parameters and computational cost, while residual connections and skip connections are employed to enhance the network's learning ability and feature fusion. Finally, nonlinear activation functions and normalization operations are applied to stabilize the training process.

[0032] The specific steps of the depthwise separable convolution method are as follows: Depthwise Convolution Stage: In the depthwise convolution stage, each input channel undergoes a convolution operation through its own independent filter. Assuming the input X has G channels, depthwise convolution convolves each channel of X with an independent filter K. g Perform convolution, where g is the channel index, from 1 to G.

[0033] The formula is expressed as: D g =X K g Among them, D g This is the output of the depthwise convolution operation, corresponding to the depth feature map of the g-th channel. X is the input feature map with dimensions [H]. in W in [,G],H in and W in These are the height and width of the input feature map, respectively. K g It is the filter corresponding to the g-th channel, with a size of [D]. filter D filter [1], D filter It is the size of the filter.

[0034] Pointwise convolution stage: The pointwise convolution stage is a 1×1 convolution operation that blends the outputs of the depthwise convolution along the channel dimension to generate the final output feature map. Assuming the depthwise convolution produces G feature maps, the pointwise convolution uses a G×H filter L to blend these feature maps, where H is the number of channels in the output feature map.

[0035] The formula is expressed as: Where S is the output feature map after the point convolution operation, with dimensions [H]. out W out ,H].

[0036] D g It is the output feature map of depthwise convolution.

[0037] L is a point convolution filter, which can be decomposed into G [1,1,H] filters.

[0038] concat This indicates a connection operation along the channel dimension.

[0039] The main components and process of the one-dimensional dilated convolutional block are as follows: Input preparation: The input data X is a one-dimensional sequence, which can be a speech signal or any other type of one-dimensional signal. The shape of the input sequence is usually (N,L), where N is the number of sequences (e.g., multiple time steps or multiple samples), and L is the length of each sequence.

[0040] Define a convolution kernel: Define a one-dimensional convolution kernel K with shape (D K ,1), where D K It is the length of the convolution kernel. The size of the convolution kernel in the depth direction is the same as the number of channels in the input sequence.

[0041] Dilation Rate Setting: Set the dilation rate d. The dilation rate is an integer that determines the spacing between elements in the convolution kernel. For dilated convolution, the spacing between non-zero elements in the kernel is d. 1.

[0042] Dilated convolution operation: Applying a dilated convolution to the input sequence X. The output Y of the dilated convolution at position n is calculated using the following formula:

[0043] Padding: If it is necessary to keep the sequence length constant, zero padding can be added at the beginning and / or end of the sequence. The length of the padding depends on the size of the convolution kernel and the dilation rate.

[0044] Activation function: After dilated convolution, a non-linear activation function, such as ReLU, is usually applied to introduce non-linear characteristics and enhance the expressive power of the model. Y activated =ReLU( Y ) Residual connections (optional): If residual connections exist, the input X and output Y of the dilated convolution are added together, and the result is then passed to the next convolutional block or subsequent layer. Y residual = X + Y activated Skip connections (optional): In some architectures, the output of the current block may be added to the output of the subsequent block to form skip connections, in order to further enhance feature fusion.

[0045] Stacking multiple dilated convolutional blocks: Multiple dilated convolutional blocks can be stacked, each using a different dilation rate, such as d=1,2,4,…, to capture features at different scales.

[0046] Output: Finally, the output of the one-dimensional dilated convolution block. Y residual It will be used as input for the next stage of processing, such as being passed to the separation module for mask estimation.

[0047] In step S3, the decoder reconstructs the source waveform based on the masked features and uses a one-dimensional transposed convolution operation to convert the masked features back to the time-domain signal. Overlapping reconstructed segments are merged through a summation operation to form the final separated speech signal.

[0048] Figure 2 For example, the one-dimensional dilated convolution block in S2 of this invention Figure 2 As shown, these convolutional blocks are the core of the TCN, consisting of one-dimensional convolutional layers, where the filters in each convolutional layer have different dilation rates. Dilated convolutions allow the network to increase its receptive field without increasing the number of parameters, thereby capturing dependencies over longer time periods.

[0049] Furthermore, to reduce the number of parameters and computational cost, convolution operations are replaced by depthwise separable convolution. This type of convolution decomposes standard convolution into two consecutive operations: depthwise convolution and pointwise convolution. Depthwise convolution applies a filter independently to each input channel, while pointwise convolution blends the feature channels based on the depthwise convolution.

[0050] The output of each convolutional block is passed to the input of the next convolutional block via residual connections. This helps avoid the vanishing gradient problem in deep network training and improves the model's learning ability. The outputs of all convolutional blocks are summed via skip connections as the final output of the TCN. This allows the network to combine features at different levels, enhancing the representational power of features. A non-linear activation function, typically parametric ReLU (PReLU), is used after each convolutional block to introduce non-linearity and enhance the model's expressive power. Normalization methods are used to stabilize the training process and improve the model's generalization ability. The final output of the TCN is passed through a 1×1 convolutional layer to estimate mask vectors for C target sources. These mask vectors are then applied to the encoder output via element-wise multiplication.

[0051] This embodiment provides a lightweight human pose estimation algorithm for video streams, including a lightweight human pose estimation algorithm and a smoothing method for human pose keypoints in video streams. This invention can be deployed on low-computing-power devices, providing processing speeds no less than typical video frame rates while ensuring the accuracy of pose estimation. Furthermore, it proposes a smoothing method to address the jitter error problem that easily occurs in human pose estimation of video streams, effectively reducing jitter errors in keypoints of human figures in the video stream and improving the accuracy and smoothness of human pose estimation.

[0052] This application also provides an electronic device. Figure 3The structure of an electronic device provided in an embodiment of the present invention is illustrated. For example, the electronic device 30 may include a processor 31, a memory 32, and a transmission device 33. The processor 31 is used to execute the audio-oriented deep neural network-based speaker separation method mentioned in the above embodiments. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The transmission device can be connected to the processor and the memory via wired or wireless means. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the audio-oriented deep neural network-based speaker separation method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the audio-oriented deep neural network-based speaker separation method in the above method embodiments. The memory may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. In addition, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and this remote memory may be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The one or more modules stored in the memory, when executed by the processor, perform the audio-oriented deep neural network-based speaker separation method of the embodiments.

[0053] In another aspect, this application also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into the device. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the audio-oriented speaker separation method based on deep neural networks described in this application.

[0054] This invention includes, but is not limited to, the above embodiments. Any equivalent substitutions or partial improvements made under the spirit and principles of this invention shall be considered within the scope of protection of this invention.

Claims

1. A deep neural network based audio oriented speaker separation method, characterized in that, The method comprises: an encoder processing step of dividing an input mixed speech waveform into overlapping short segments by a one-dimensional convolution operation and mapping the short segments to a high-dimensional feature space to obtain a high-dimensional feature representation; a separation module processing step of receiving the high-dimensional feature representation and processing the high-dimensional feature representation by a convolution separation module constructed based on a time convolution network; the convolution separation module comprises a plurality of stacked one-dimensional dilated convolution blocks, each of the one-dimensional dilated convolution blocks has a different dilation rate; in the convolution separation module, a depth separable convolution is used to replace a standard convolution, and a residual connection and a skip connection are introduced; finally, a point convolution layer is used to estimate a multiplication mask for each target speech source; a decoder processing step of performing element-level multiplication operation on the multiplication mask and the high-dimensional feature representation output by the encoder to obtain a masked feature, reconstructing the masked feature back to a time-domain waveform by a one-dimensional transpose convolution operation, and summing overlapping reconstructed segments to output a separated single speech source signal.

2. The audio-oriented deep neural network-based speaker separation method of claim 1, wherein, The specific operation of the depth separable convolution comprises: a depth convolution stage of applying a convolution kernel to each channel of an input feature map to generate a depth feature map; a point convolution stage of using a 1x1 convolution kernel to linearly combine all channels of the depth feature map to mix channel information and generate a final output feature map after the depth convolution stage.

3. The audio-oriented deep neural network-based speaker separation method of claim 1, wherein, The specific operation of the one-dimensional dilated convolution block comprises: performing a dilated convolution operation on an input one-dimensional sequence by using a one-dimensional convolution kernel with a predetermined dilation rate; applying a nonlinear activation function after the dilated convolution operation; optionally adding the input of the dilated convolution operation to the output after the activation function to form a residual connection; stacking a plurality of one-dimensional dilated convolution blocks with different dilation rates.

4. The audio-oriented deep neural network-based speaker separation method of claim 1, wherein, The residual connection is used to directly pass the input of the one-dimensional dilated convolution block to the output end for addition.

5. The audio-oriented deep neural network-based speaker separation method of claim 1, wherein, The skip connection is used to sum all or part of the outputs of the one-dimensional dilated convolution blocks as the final feature output of the convolution separation module.

6. The audio-oriented deep neural network-based speaker separation method according to any one of claims 1-5, characterized in that, After each layer or a specific layer of the convolution separation module, a step of applying a normalization operation is further included to stabilize the training process.

7. An electronic device, comprising: The method comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the audio-oriented deep neural network-based speaker separation method according to any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the audio-oriented deep neural network-based speaker separation method according to any one of claims 1 to 6.

Images