A speech extraction method and system based on a sandglass structure and a self-attention mechanism

Abstract

The application discloses a voice extraction method and system based on a self-attention mechanism of a sandglass structure, and the method comprises the following steps: S1, an encoder encodes an input auxiliary audio signal and a mixed audio signal to obtain an auxiliary audio high-dimensional embedding and a mixed audio high-dimensional embedding; S2, a voiceprint extractor performs extraction processing on the auxiliary audio high-dimensional embedding to obtain a voiceprint feature; S3, a speaker extractor performs aggregation processing on the voiceprint feature and the mixed audio high-dimensional embedding to obtain an aggregated feature; the aggregated feature is further processed to obtain a voice mask feature; S4, a decoder estimates a mask from the voice mask feature; the mask is multiplied with the mixed audio high-dimensional embedding to obtain a target speaker audio embedding; and the target speaker audio embedding is reconstructed into a time-domain signal to obtain audio of the target speaker. Through the above improvement, the response speed of an intelligent voice customer service voice recognition system is improved.

Description

A speech extraction method and system based on hourglass structure and self-attention mechanism Technical Field

[0001] This invention relates to the field of speech processing technology, and more specifically, to a speech extraction method and system based on an hourglass structure and a self-attention mechanism. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, various fields are adopting different AI technologies to assist in their work and improve efficiency. In the customer service industry, human customer service is generally unable to provide 24 / 7 service, so companies typically use intelligent voice customer service. Intelligent voice customer service is a voice robot that integrates multiple intelligent interaction technologies such as speech recognition, semantic understanding, knowledge graphs, and deep learning. It can accurately identify the user's intent or question and provide satisfactory answers to customers, thereby reducing manual operations and lowering costs. However, the environment in which customers make business inquiries is complex and diverse, often containing various types of noise, especially voices that are not the customer's own. Such noise can affect the recognition results of intelligent voice customer service, leading to incorrect answers. When customers do not receive satisfactory answers, they may abandon their business, or even become dissatisfied and file complaints, causing the company to lose customers and transactions. Therefore, accurate speech recognition is a prerequisite for the normal operation of an intelligent voice customer service system.

[0003] Currently, the speech separation techniques used in the industry to extract the speech of a target speaker include speech separation methods based on speech amplitude spectrum and speech separation methods based on time domain signals.

[0004] Early speech separation techniques were based on speech amplitude spectrum, which is separated from the time-frequency features of speech after short-time Fourier transform processing. In addition to the amplitude spectrum, the phase spectrum was also separated simultaneously. However, in the speech separation process, usually only the amplitude spectrum is processed and enhanced. The final speech signal needs to be reconstructed by combining the amplitude spectrum and phase spectrum of the target speaker using inverse short-time Fourier transform. This leads to difficulties in phase reconstruction during reconstruction, thus affecting signal quality. Furthermore, the speaker classification network is pre-trained and not trained together with the separation network, so the extracted voiceprint features may not be the most suitable for speaker extraction.

[0005] In subsequent improvements to the SpEx+ model based on time-domain signals, an additional Conformer module was added directly after each TCN in the speaker extractor. This increases the computational load of the model, resulting in very high GPU resource requirements during training, long training times, and slower inference speed. Since the speaker extraction model is a front-end module of the speech recognition system, it should not consume too many resources and space, thus affecting the response speed of the speech recognition system.

[0006] Chinese invention patent application number 202010002838.X discloses a "Target Speech Extraction Method, Apparatus, Device, Medium, and Joint Training Method." Its technical solution involves: extracting features from training reference sample speech data using a speech feature extraction model to obtain a reference speech feature vector, wherein the training reference sample speech data is clean speech data corresponding to the target object; fusing features from the training speech data and the reference speech feature vector using the speech extraction model to obtain a fused feature vector, wherein the training speech data is noisy speech data from which the target object's speech data is extracted; extracting speech data from the training speech data and the fused feature vector using the speech extraction model to obtain the target speech data; and jointly training the speech feature extraction model and the speech extraction model based on the reference speech feature vector and the target speech data. Compared to this solution, the technical solution provided by this invention, through a combination of upsampling and downsampling and a self-attention mechanism, achieves multi-scale context sequence modeling, better combining local and global sequence features, and achieving superior performance with fewer model parameters. It can extract target customer speech with a high signal-to-noise ratio, providing clean speech for speech recognition systems and improving their recognition accuracy. Summary of the Invention

[0007] To address the problems of low efficiency and low accuracy in existing speech recognition technologies, this invention proposes a speech extraction method and system based on an hourglass structure and a self-attention mechanism. The technical solution adopted by this invention is as follows:

[0008] The first aspect of this invention provides a speech extraction method based on an hourglass structure and a self-attention mechanism, comprising the following steps:

[0009] S1, the encoder encodes the input auxiliary audio signal and the mixed audio signal to obtain the high-dimensional embedding of the auxiliary audio and the high-dimensional embedding of the mixed audio corresponding to the input;

[0010] S2, the voiceprint extractor extracts the high-dimensional embedding of the auxiliary audio to obtain voiceprint features of fixed dimensions;

[0011] S3, the speaker extractor aggregates the voiceprint features and the high-dimensional embedding of the mixed audio to obtain aggregated features; the aggregated features are further processed to obtain speech mask features;

[0012] S4, the decoder estimates a mask from the speech mask features; performs a dot product calculation between the mask and the mixed audio high-dimensional embedding to obtain the target speaker audio embedding; and reconstructs the target speaker audio embedding into a time-domain signal to obtain the target speaker's audio.

[0013] Compared to existing technologies, this method proposes a speaker extraction model based on an hourglass structure combining upsampling and downsampling and a self-attention mechanism. This model achieves multi-scale context sequence modeling by combining upsampling and downsampling and the self-attention mechanism, better combining local and global features of the sequence, achieving superior performance with fewer model parameters, and extracting target customer speech with a high signal-to-noise ratio, providing clean speech for speech recognition systems and improving their recognition accuracy.

[0014] As a preferred embodiment, the encoder includes a small-scale encoder, a medium-scale encoder, and a large-scale encoder; the small-scale encoder includes a convolutional neural network with small-sized convolutional kernels; the medium-scale encoder includes a convolutional neural network with medium-sized convolutional kernels; the large-scale encoder includes a convolutional neural network with large-sized convolutional kernels; and the small-scale encoder, medium-scale encoder, and large-scale encoder share the same linear correction unit.

[0015] As a preferred embodiment, in step S1, the auxiliary audio signal and the mixed audio signal share the same encoder weight, such that the temporal domain signals of the auxiliary audio signal and the mixed audio signal are processed by the encoder to obtain corresponding overlapping frame sequences, as specifically:

[0016]

[0017] Where x(t) represents the time-domain signal; Represents an overlapping frame sequence; Conv1D l,s,n (·) represents a one-dimensional convolutional network, l represents the size of the convolutional kernel, i.e. the encoding window length; s represents the stride of the convolutional network, n represents the number of filters in the convolutional neural network; ReLU(·) represents a linear correction unit.

[0018] As a preferred embodiment, the voiceprint extractor includes a first convolutional neural network, a residual block layer, a second convolutional neural network, a global average pooling layer, a fully connected layer, a softmax layer, and a cross-entropy loss layer connected in sequence.

[0019] Step S2 specifically includes the following processes:

[0020] The auxiliary audio high-dimensional embedding first performs feature fusion through the first convolutional neural network, then extracts features through the residual block layer, and then maps them to a fixed dimension through the second convolutional neural network to obtain a fixed-dimensional embedding. The fixed-dimensional embedding is then subjected to average pooling through the global average pooling layer to obtain fixed-dimensional voiceprint features. The voiceprint features are then subjected to voiceprint classification through the fully connected layer and the softmax layer, and finally the voiceprint classification is optimized through the cross-entropy loss layer.

[0021] As a preferred embodiment, the speaker extractor includes a sequentially connected convolutional neural network and several hourglass blocks;

[0022] Step S3 specifically includes the following processes:

[0023] The hybrid audio high-dimensional embedding first aggregates the channel features of multi-scale embedding through the convolutional neural network and reduces the channel dimension; then, the voiceprint features are concatenated with the dimensionality-reduced hybrid audio embedding and then processed through several hourglass blocks for feature extraction, finally obtaining the speech mask features.

[0024] As a preferred embodiment, the hourglass block includes a channel splicing layer, a stacked TCN layer, a downsampling layer, a Conformer block, and an upsampling layer;

[0025] The channel splicing layer is used to splice the voiceprint features and the output of the previous stage to obtain mixed features; the stacked TCN layer is used to incorporate sequence dependency information into the mixed features and prevent gradient vanishing to obtain local sequence features mixed with voiceprint information; the downsampling layer is used to reduce the length of the local sequence features; the Conformer block is used to transform the local sequence features processed by the downsampling layer into global sequence features; the upsampling layer is used to restore the length of the global sequence features to be consistent with the length of the local sequence features.

[0026] As a preferred embodiment, the Conformer block includes a first feedforward network, a multi-head attention module, a convolutional module, a second feedforward network, and a layer normalization module; the specific processing procedure of the Conformer block for the local sequence features is as follows:

[0027] The local sequence features processed by the downsampling layer are first processed by the first feedforward network, then global feature extraction is performed by the multi-head attention module, followed by local correlation learning by the convolution module, and finally input into the second feedforward network. Layer normalization is then performed by the layer normalization module to obtain the global sequence features. The specific formulas involved in the processing of the Conformer block are as follows:

[0028]

[0029] f″′=′+MHSA(f ′ );

[0030] f″′=f″+Conv(f″);

[0031]

[0032] Where FFN(·) represents the first and second feedforward networks, Conv(·) represents the convolutional module, LN(·) represents layer normalization, MHSA(·) represents the multi-head attention function containing h heads, and f global This represents global sequence features.

[0033] As a preferred embodiment, the decoder includes a sequentially connected convolutional neural network, a linear correction unit, and a one-dimensional transposed convolutional network;

[0034] The convolutional neural network and linear correction unit are used to estimate a mask from the speech mask features, and the one-dimensional transposed convolutional network is used to embed and reconstruct the target speaker's audio into a time-domain signal; the calculation formula for the time-domain signal is as follows:

[0035] y(t)=ConvTrans1D(M⊙f mix ) = ConvTrans1D(f target );

[0036] Where y(t) represents the time-domain signal, ConvTrans1D(·) represents the one-dimensional transposed convolution, M represents the mask, and f mix f represents a high-dimensional embedding of mixed audio. target This is represented as the target speaker's audio embedding.

[0037] A second aspect of the present invention also provides a speech extraction system based on a self-attention mechanism of an hourglass structure, including an audio coding unit, a voiceprint feature extraction unit, an aggregation extraction unit, a target audio generation unit, and a system optimization unit;

[0038] The audio encoding unit is used to encode the input auxiliary audio signal and the mixed audio signal to obtain the high-dimensional embedding of the auxiliary audio and the high-dimensional embedding of the mixed audio corresponding to the input.

[0039] The voiceprint feature extraction unit is used to extract the auxiliary audio high-dimensional embedding to obtain voiceprint features of fixed dimensions.

[0040] The aggregation extraction unit is used to aggregate the voiceprint features and the high-dimensional embedding of the mixed audio to obtain aggregated features; the aggregated features are further processed to obtain speech mask features;

[0041] The target audio generation unit is used to estimate a mask from the speech mask features; perform a dot product calculation between the mask and the mixed audio high-dimensional embedding to obtain the target speaker audio embedding; and reconstruct the target speaker audio embedding into a time-domain signal to obtain the target speaker's audio.

[0042] The system optimization unit uses an objective function to optimize all the above units.

[0043] As a preferred embodiment, the objective function is specifically:

[0044]

[0045] Where, x mix Indicates a mixed audio signal, x aux Indicates auxiliary audio signal, y target The audio represents the target speaker, I represents the category label of the auxiliary audio signal, and γ represents the weighting parameter. This represents the signal-to-noise ratio loss of a scale-invariant signal. This represents the cross-entropy loss.

[0046] The beneficial effects of this invention are:

[0047] This invention employs a main framework composed of TCN and self-attention modules, and introduces upsampling and downsampling operations of an hourglass structure to achieve multi-scale context sequence modeling. This better combines local and global features of the sequence, achieving superior performance with a smaller amount of parameter data and improving the response speed of intelligent voice customer service speech recognition systems. Attached Figure Description

[0048] Figure 1 is a flowchart of a speech extraction method based on an hourglass structure and self-attention mechanism provided by an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of the overall framework provided in an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of the encoder frame provided in an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of the residual block framework provided in an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of the hourglass block frame provided in an embodiment of the present invention;

[0053] Figure 6 is a schematic diagram of the TCN residual block framework provided in an embodiment of the present invention;

[0054] Figure 7 is a schematic diagram of the Conformer block framework provided in an embodiment of the present invention;

[0055] Figure 8 is a schematic diagram of the decoder framework provided in an embodiment of the present invention. Detailed Implementation

[0056] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0057] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0058] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0059] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0060] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0062] Example 1

[0063] Please refer to Figures 1 to 8. A speech extraction method based on a self-attention mechanism of an hourglass structure includes the following steps:

[0064] S1, the encoder encodes the input auxiliary audio signal and the mixed audio signal to obtain the high-dimensional embedding of the auxiliary audio and the high-dimensional embedding of the mixed audio corresponding to the input;

[0065] It should be noted that the auxiliary audio signal is a pre-collected segment of the target speaker's voice information; the mixed audio signal is the voice information of the target speaker and other speakers collected during speech recognition.

[0066] In a specific embodiment, as shown in Figures 2 and 3, the encoder includes a small-scale encoder, a medium-scale encoder, and a large-scale encoder; the small-scale encoder includes a convolutional neural network with small-sized convolutional kernels; the medium-scale encoder includes a convolutional neural network with medium-sized convolutional kernels; and the large-scale encoder includes a convolutional neural network with large-sized convolutional kernels; the small-scale encoder, the medium-scale encoder, and the large-scale encoder share the same linear correction unit.

[0067] Specifically, the encoder is a structure composed of convolutional neural networks with different kernel sizes, corresponding to different time scales. This structure extracts both local details and global features of the audio signal by performing convolution operations on different kernels. Smaller kernels extract fine-grained local features, capturing detailed information of the audio signal; while larger kernels extract broader contextual information, capturing the overall features of the audio signal. The encoder fuses these features at different scales to comprehensively consider both local details and global information of the audio signal. This structural design enables the model to better understand the features of the audio signal and perform effective representation learning, thereby providing more accurate predictions and better performance for speaker extraction tasks.

[0068] Specifically, the auxiliary audio signal and the mixed audio signal are mapped to high-dimensional embeddings through convolutional neural networks with different kernel sizes, and then activated by linear correction units to obtain high-dimensional embeddings at different time scales. These high-dimensional embeddings at different time scales are then concatenated by channels, and finally channel normalization is performed to form an auxiliary audio high-dimensional embedding and a mixed audio high-dimensional embedding containing global and detailed information.

[0069] It should be noted that in step S1, the auxiliary audio signal and the mixed audio signal share the same encoder weight, so that the time-domain signals of the auxiliary audio signal and the mixed audio signal are processed by the encoder to obtain the corresponding overlapping frame sequence, as shown in the following formula:

[0070]

[0071] Where x(t) represents the time-domain signal; Represents an overlapping frame sequence; Conv1D l,s,n (·) represents a one-dimensional convolutional network, l represents the size of the convolutional kernel, i.e. the encoding window length; s represents the stride of the convolutional network, n represents the number of filters in the convolutional neural network; ReLU(·) represents a linear correction unit.

[0072] Specifically, the overlapping frame sequence is a sequence of 50% overlapping frames; the step size s is usually set to half the window length, indicating that each sequence segment has 50% overlap, so as to prevent abrupt changes in the audio sequence features.

[0073] S2, the voiceprint extractor extracts the high-dimensional embedding of the auxiliary audio to obtain voiceprint features of fixed dimensions;

[0074] In one specific embodiment, as shown in FIG2, the voiceprint extractor includes a first convolutional neural network, a residual block layer, a second convolutional neural network, a global average pooling layer, a fully connected layer, a softmax layer, and a cross-entropy loss layer connected in sequence.

[0075] Step S2 specifically includes the following processes:

[0076] The auxiliary audio high-dimensional embedding first performs feature fusion through the first convolutional neural network, then extracts features through the residual block layer, and then maps them to a fixed dimension through the second convolutional neural network to obtain a fixed-dimensional embedding. The fixed-dimensional embedding is then subjected to average pooling through the global average pooling layer to obtain fixed-dimensional voiceprint features. The voiceprint features are then subjected to voiceprint classification through the fully connected layer and the softmax layer, and finally the voiceprint classification is optimized through the cross-entropy loss layer.

[0077] Specifically, as shown in Figure 4, the residual block layer includes N r Each residual block comprises two 1×1 convolutional neural networks (CNNs), and the outputs of each CNN are batch-normalized and nonlinearly transformed using channel normalization layers and parameterized linear correction units (PLCs). To prevent network degradation and gradient vanishing, the output of the second normalized CNN is residually concatenated with the original input, and then activated by a PLC. Finally, a 1×3 max-pooling layer is used to preserve key audio features and reduce sequence length.

[0078] S3, the speaker extractor aggregates the voiceprint features and the high-dimensional embedding of the mixed audio to obtain aggregated features; the aggregated features are further processed to obtain speech mask features;

[0079] In one specific embodiment, as shown in FIG2, the speaker extractor includes a sequentially connected convolutional neural network and several hourglass blocks;

[0080] Step S3 specifically includes the following processes:

[0081] The hybrid audio high-dimensional embedding first aggregates the channel features of multi-scale embedding through the convolutional neural network and reduces the channel dimension; then, the voiceprint features are concatenated with the dimensionality-reduced hybrid audio embedding and then processed through several hourglass blocks for feature extraction, finally obtaining the speech mask features.

[0082] Specifically, the main structure of the speaker extractor employs a combination of downsampling and upsampling, which causes the auxiliary audio high-dimensional embedding and the hybrid audio high-dimensional embedding to exhibit a shape change that first becomes smaller and then larger when modeling contextual features. This shape change process is similar to the change in the volume of sand in the upper and lower containers of an hourglass, hence this processing module is called the hourglass block.

[0083] It should be noted that, as shown in Figure 5, the hourglass block includes a channel splicing layer, a stacked TCN layer, a downsampling layer, a Conformer block, and an upsampling layer;

[0084] The channel splicing layer is used to splice the voiceprint features and the output of the previous stage to obtain mixed features; the stacked TCN layer is used to incorporate sequence dependency information into the mixed features and prevent gradient vanishing to obtain local sequence features mixed with voiceprint information; the downsampling layer is used to reduce the length of the local sequence features; the Conformer block is used to transform the local sequence features processed by the downsampling layer into global sequence features; the upsampling layer is used to restore the length of the global sequence features to be consistent with the length of the local sequence features.

[0085] Specifically, as shown in Figure 6, the stacked TCN layer includes B TCN residual blocks, each of which consists of two convolutional neural networks, two PReLU activation function layers, two channel normalization layers, and one dilated convolutional network layer.

[0086] The first convolutional neural network is used to aggregate data channel features. Typically, the output channel feature dimension is set to twice the input channel size to achieve dimensionality increase. Then, the aggregated features are activated by an activation function layer and subjected to channel normalization, mapping the data to the same scale and reducing data dispersion. Finally, the features are input into a circuit with a dilation factor of 2. B-1In a dilated convolutional network (TCN), the dependency information between sequential data is extracted. The dilation factor of the dilated convolutional network for different residual blocks in a TCN is exponentially related to the order of the residual blocks. Then, the dependency features are activated and channel normalized before being input into a second convolutional neural network. After feature aggregation, the dimensionality of the output channels is reduced to match the input. Finally, a residual connection is made between the output and the original input to incorporate sequence dependency information into the input and prevent gradient vanishing.

[0087] Taking into account both the performance and efficiency of the model, each stacked TCN in this method contains 8 TCN residual blocks, with an inflation factor d∈{1,2,4,8,16,32,64,128}.

[0088] Specifically, in step S3, the hourglass block first performs channel concatenation on the voiceprint features and the output from the previous stage, fusing the voiceprint information of the target customer into the mixed audio embedding to obtain mixed features. Then, the mixed features are input into the stacked TCN. Since the voiceprint features are concatenated with the mixed audio embedding channels, the number of channels of the resulting mixed features increases. Therefore, the first TCN residual block in each stacked TCN will fuse the mixed features to reduce the number of channels. That is, the number of input channels of the first TCN residual block in the stacked TCN is the sum of the number of voiceprint features and the number of mixed audio embedding channels, and the number of output channels is equal to the number of mixed audio embedding channels. The number of input and output channels of subsequent TCN residual blocks is consistent with the number of mixed audio embedding channels.

[0089] Specifically, this method first downsamples the sequence features containing voiceprint information at the downsampling layer before performing global contextual sequence modeling. Downsampling reduces the amount of modeling data, and different downsampling granularities can model sequences of different granularities, leading to better extraction of contextual features. The global contextual feature calculation process is as follows:

[0090]

[0091] in, Represents global context features, Represents the local contextual features modeled by stacked TCN. Conformer(·) represents the Conformer block for convolution enhancement, and its specific structure will be introduced below. b (l) and Down b (l) represents the operations corresponding to the upsampling layer and the downsampling layer in the b-th hourglass block, respectively, and l represents the filter length in the upsampling layer and downsampling layer operations.

[0092] Specifically, the downsampling layer and the upsampling layer use convolution and deconvolution to perform downsampling and upsampling operations, respectively. Deconvolution is the inverse operation of convolution, which can restore the convolved sequence to the original sequence length. The specific formula is as follows:

[0093]

[0094]

[0095] Specifically, the downsampling layer operation results in information loss. Therefore, this method uses upsampling to restore the global context feature length to match the local context feature length. Then, it performs a residual concatenation between the global and local context features to ensure no information loss, while simultaneously fusing local and global features. Thanks to the effectiveness of multi-scale context modeling, this method achieves only half the high-dimensional embedding encoding dimension of SpEx+ for audio, significantly reducing model parameters while maintaining performance comparable to existing methods.

[0096] In a specific embodiment, as shown in Figure 7, the Conformer block includes a first feedforward network, a multi-head attention module, a convolutional module, a second feedforward network, and a layer normalization module; the specific processing procedure of the Conformer block for the local sequence features is as follows:

[0097] The local sequence features processed by the downsampling layer are first processed by the first feedforward network, then global feature extraction is performed by the multi-head attention module, followed by local correlation learning by the convolution module, and finally input into the second feedforward network. Layer normalization is then performed by the layer normalization module to obtain the global sequence features. The specific formulas involved in the processing of the Conformer block are as follows:

[0098]

[0099] f″′=f′+MHSA(f′);

[0100] f″′=f″+CoNV(F″);

[0101]

[0102] Where fFN(·) represents the first and second feedforward networks, Conv(·) represents the convolutional module, LN(·) represents layer normalization, MHSA(·) represents the multi-head attention function containing h heads, and f global This represents global sequence features.

[0103] Specifically, the multi-head attention module includes h multi-head attention functions, which map the input into a query matrix, a key matrix, and a value matrix, forming multiple subspaces. Through the query matrix, it calculates the correlation coefficient between the input data and different key matrices. Then, it calculates the weighted dot product attention of different heads and combines the outputs together, multiplying the weights to obtain the result. The specific formula is as follows:

[0104]

[0105]

[0106] MHSA=W·Concat(Attention1,…,Attention h );

[0107] Where x represents the input, j (j∈{1,…,h}) represents different headers, and q j Represents the query matrix, k j Represents the key matrix, v j Let W represent the value matrix. and This represents the parameters of a multi-head attention network.

[0108] S4, the decoder estimates a mask from the speech mask features; performs a dot product calculation between the mask and the mixed audio high-dimensional embedding to obtain the target speaker audio embedding; and reconstructs the target speaker audio embedding into a time-domain signal to obtain the target speaker's audio.

[0109] In a specific embodiment, as shown in Figure 8, the decoder includes a convolutional neural network, a linear correction unit, and a one-dimensional transposed convolutional network connected in sequence; the convolutional neural network and the linear correction unit are used to estimate a mask from the speech mask features, and the one-dimensional transposed convolutional network is used to embed and reconstruct the target speaker's audio into a time-domain signal; the calculation formula for the time-domain signal is as follows:

[0110] y(t)=ConvTrans1D(M⊙f mix =ConvTrans1D(ft) target );

[0111] Where y(t) represents the time-domain signal, ConvTrans1D(·) represents the one-dimensional transposed convolution, M represents the mask, and f mix f represents a high-dimensional embedding of mixed audio. target This is represented as the target speaker's audio embedding.

[0112] The method of this invention adopts a main framework composed of TCN and self-attention module, and introduces upsampling and downsampling operations of hourglass structure to realize multi-scale context sequence modeling, better combine local and global features of sequence, and achieve superior performance with less parameter data, thereby improving the response speed of intelligent voice customer service speech recognition system.

[0113] Example 2

[0114] A speech extraction system based on a self-attention mechanism of an hourglass structure includes an audio coding unit, a speaker feature extraction unit, an aggregation extraction unit, a target audio generation unit, and a system optimization unit.

[0115] The audio encoding unit is used to encode the input auxiliary audio signal and the mixed audio signal to obtain the high-dimensional embedding of the auxiliary audio and the high-dimensional embedding of the mixed audio corresponding to the input.

[0116] The voiceprint feature extraction unit is used to extract the auxiliary audio high-dimensional embedding to obtain voiceprint features of fixed dimensions.

[0117] The aggregation extraction unit is used to aggregate the voiceprint features and the high-dimensional embedding of the mixed audio to obtain aggregated features; the aggregated features are further processed to obtain speech mask features;

[0118] The target audio generation unit is used to estimate a mask from the speech mask features; perform a dot product calculation between the mask and the mixed audio high-dimensional embedding to obtain the target speaker audio embedding; and reconstruct the target speaker audio embedding into a time-domain signal to obtain the target speaker's audio.

[0119] The system optimization unit uses an objective function to optimize all the above units.

[0120] In a specific embodiment, the objective function is:

[0121]

[0122] Where, x mix Indicates a mixed audio signal, x aux Indicates auxiliary audio signal, y target The audio represents the target speaker, I represents the category label of the auxiliary audio signal, and γ represents the weighting parameter. This represents the signal-to-noise ratio loss of a scale-invariant signal. This represents the cross-entropy loss.

[0123] Specifically, for high-quality audio from the target speaker, a scale-invariant signal-to-noise ratio (SI-SNR) loss L is used. SI-SNROptimization is performed; for voiceprint classification, cross-entropy (CE) loss is used. Optimization is performed by weighting two losses and using gradient propagation to optimize all modules simultaneously. Through joint training, the speaker extractor can help the voiceprint extractor extract voiceprint features that are more suitable for speaker extraction tasks, while the voiceprint extractor can help the speaker extractor extract customer speech with a higher signal-to-noise ratio.

[0124] Specifically, the aforementioned The goal is to minimize the audio signal reconstruction error, specifically defined as:

[0125]

[0126] Where <·> indicates multiplying the elements digit by digit and then summing them; x est represents the target clean audio value estimated by the model; ||·| represents the L2 norm. The definition is as follows:

[0127]

[0128] Where n represents the number of auxiliary audio frequencies, I i p represents the true label of the i-th audio. i This represents the predicted probability that the i-th audio belongs to each category.

[0129] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A speech extraction method based on a self-attention mechanism of an hourglass structure, characterized in that, Includes the following steps: S1, the encoder encodes the input auxiliary audio signal and the mixed audio signal to obtain the high-dimensional embeddings of the auxiliary audio and the mixed audio corresponding to the input; S2, the voiceprint extractor extracts the high-dimensional embeddings of the auxiliary audio to obtain fixed-dimensional voiceprint features; S3, the speaker extractor aggregates the voiceprint features and the high-dimensional embeddings of the mixed audio to obtain aggregated features; the aggregated features are further processed to obtain speech mask features; S4, the decoder estimates a mask from the speech mask features; the mask is multiplied by the high-dimensional embeddings of the mixed audio to obtain the target speaker audio embedding; the target speaker audio embedding is reconstructed into a time-domain signal to obtain the audio of the target speaker; the speaker extractor includes a sequentially connected convolutional neural network and several hourglass blocks; the hourglass block includes a channel splicing layer, a stacked TCN layer, a downsampling layer, a Conformer block, and an upsampling layer; the Conformer block is used to transform the local sequence features processed by the downsampling layer into global sequence features.

2. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 1, characterized in that, The encoder includes a small-scale encoder, a medium-scale encoder, and a large-scale encoder; the small-scale encoder includes a convolutional neural network with small-sized convolutional kernels; the medium-scale encoder includes a convolutional neural network with medium-sized convolutional kernels; and the large-scale encoder includes a convolutional neural network with large-sized convolutional kernels; the small-scale encoder, medium-scale encoder, and large-scale encoder share the same linear correction unit.

3. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 2, characterized in that, In step S1, the auxiliary audio signal and the mixed audio signal share the same encoder weight, so that the time-domain signals of the auxiliary audio signal and the mixed audio signal are processed by the encoder to obtain the corresponding overlapping frame sequence, as shown in the following formula: ;in, Represents a time-domain signal; This represents an overlapping frame sequence; This represents a one-dimensional convolutional network. This indicates the size of the convolutional kernel in the convolutional network, i.e., the length of the encoding window; This represents the stride of the convolutional network. Indicates the number of filters in a convolutional neural network; This represents a linear correction unit.

4. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 1, characterized in that, The voiceprint extractor includes a first convolutional neural network, a residual block layer, a second convolutional neural network, a global average pooling layer, a fully connected layer, a softmax layer, and a cross-entropy loss layer connected in sequence; step S2 further specifically... The process includes the following steps: the auxiliary audio high-dimensional embedding first performs feature fusion through the first convolutional neural network, then extracts features through the residual block layer, and then maps to a fixed dimension through the second convolutional neural network to obtain a fixed-dimensional embedding; the fixed-dimensional embedding is processed by average pooling through the global average pooling layer to obtain fixed-dimensional voiceprint features; the voiceprint features are processed by voiceprint classification through the fully connected layer and the softmax layer, and finally the voiceprint classification is optimized through the cross-entropy loss layer.

5. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 1, characterized in that, The specific steps S3 also include The process includes the following steps: First, the hybrid audio high-dimensional embedding aggregates the channel features of the multi-scale embedding through the convolutional neural network and reduces the channel dimension; then, the voiceprint features are concatenated with the dimensionality-reduced hybrid audio embedding and then processed through several hourglass blocks for feature extraction, finally obtaining the speech mask features.

6. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 5, characterized in that, The channel splicing layer is used to splice the voiceprint features and the output of the previous stage to obtain mixed features; the stacked TCN layer is used to incorporate sequence dependency information into the mixed features and prevent gradient vanishing to obtain local sequence features mixed with voiceprint information; the downsampling layer is used to reduce the length of the local sequence features; the upsampling layer is used to restore the length of the global sequence features to be consistent with the length of the local sequence features.

7. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 6, characterized in that, The Conformer block includes a first feedforward network, a multi-head attention module, a convolutional module, a second feedforward network, and a layer normalization module. The specific processing procedure of the local sequence features by the Conformer block is as follows: the local sequence features processed by the downsampling layer are first processed by the first feedforward network, then global feature extraction is performed by the multi-head attention module, followed by learning local correlations through the convolutional module, and finally input into the second feedforward network. Layer normalization is then performed by the layer normalization module to obtain the global sequence features. The specific formulas involved in the processing procedure of the Conformer block are as follows: ； ； ； ;in, This represents the first feedforward network and the second feedforward network. Represents a convolutional module. Representation layer normalization, This represents a multi-head attention function containing h heads. This represents global sequence features.

8. The speech extraction method based on a self-attention mechanism of an hourglass structure according to claim 1, characterized in that, The decoder includes a convolutional neural network, a linear correction unit, and a one-dimensional transposed convolutional network connected in sequence; the convolutional neural network and the linear correction unit are used to estimate a mask from the speech mask features, and the one-dimensional transposed convolutional network is used to embed and reconstruct the target speaker's audio into a time-domain signal; the calculation formula for the time-domain signal is as follows: ;in, Represents a time-domain signal. This represents a one-dimensional transposed convolution, where M represents the mask. This indicates high-dimensional embedding of mixed audio. This is represented as the target speaker's audio embedding.

9. A speech extraction system based on a self-attention mechanism of an hourglass structure, characterized in that, The system includes an audio encoding unit, a voiceprint feature extraction unit, an aggregation extraction unit, a target audio generation unit, and a system optimization unit. The audio encoding unit encodes the input auxiliary audio signal and the mixed audio signal to obtain high-dimensional embeddings of the auxiliary audio and the mixed audio. The voiceprint feature extraction unit extracts the high-dimensional embeddings of the auxiliary audio to obtain fixed-dimensional voiceprint features. The aggregation extraction unit aggregates the voiceprint features and the high-dimensional embeddings of the mixed audio to obtain aggregated features. These aggregated features are further processed to obtain speech mask features. The target audio generation unit is used to estimate a mask from the speech mask features; and to perform a dot product calculation between the mask and the mixed audio high-dimensional embedding to obtain the target speaker audio embedding. The target speaker's audio is embedded and reconstructed into a time-domain signal to obtain the target speaker's audio; The system optimization unit uses an objective function to optimize all the above units.

10. A speech extraction system based on a self-attention mechanism of an hourglass structure according to claim 9, characterized in that, The objective function is specifically: ;in, Indicates a mixed audio signal. Indicates auxiliary audio signal, The audio representing the target speaker. The category label indicates the auxiliary audio signal. Represents the weight parameters. This represents the signal-to-noise ratio loss of a scale-invariant signal. This represents the cross-entropy loss.

Images