Multi-feature adaptive selection fusion method suitable for voice sequence

Through the combination of the speech domain alignment module, adaptive selection module and multi-feature fusion module, the problem of insufficient feature fusion in speech separation is solved, and the adaptive selection fusion of different domain features is realized, and the performance of the separation model is improved.

CN119068901APending Publication Date: 2024-12-03SHANTOU VOCATIONAL & TECH COLLEGE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411055289.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to adaptively select and fuse features from different domains in speech separation, resulting in feature fusion being too simple and rough and unable to effectively improve model performance.

Method used

The voice range alignment module, adaptive selection module and multi-feature fusion module are adopted to realize adaptive selection fusion of multiple features through feature alignment, refinement selection and correlation calculation, and enhance feature representation capabilities.

Benefits of technology

Effectively fuse the feature vectors with inconsistent time dimensions, reduce feature redundancy, and improve the separation accuracy and separation quality of the separation model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068901A_ABST
    Figure CN119068901A_ABST
Patent Text Reader

Abstract

The invention provides a multi-feature adaptive selection fusion method suitable for a voice sequence, and relates to the technical field of voice separation. The method relates to a voice domain alignment module, an adaptive selection module ASM and a multi-feature fusion module FFM. The voice domain alignment module aligns the extracted input feature X1 by using the input feature X2 in the time dimension to obtain a feature F0; a self-adaptive selection module ASM carries out refining processing on the features of each channel in the input features through a self-adaptive Map block; selecting each piece of time sequence information in the refined features by selecting a Map block; and the multi-feature fusion module carries out refined selection on the F0 obtained by the voice domain alignment module and the input feature X2 through an adaptive selection module ASM and then carries out fusion so as to enrich the representation capability of the original feature. According to the method, two feature vectors with inconsistent time dimensions can be effectively fused, and the obtained fused features can further improve the separation precision and the separation quality of the separation model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of speech separation, and in particular to a multi-feature adaptive selection and fusion method suitable for speech sequences. Background Art

[0002] There are two common feature fusion methods based on deep learning. One is point-wise addition. Point-wise addition is to directly add the corresponding elements of two feature vectors. The premise of addition must be that all dimensions of the two feature vectors are the same. The other is vector concatenation, which is also a more general feature fusion method. Vector concatenation is to concatenate the corresponding dimensions of two feature vectors. The concatenation dimensions of the two feature vectors may be different, but the other dimensions must be the same. It can be seen that the above two feature fusion methods have limitations in their use scenarios. It cannot adaptively select and fuse features from different domains. In addition, the features after bit-wise addition fusion only obtain the average value of the features. Vector concatenation improves the perception of features by concatenating features of different scales, but these two fusion features are still too simple and rough.

[0003] In order to obtain richer features that are more conducive to improving model performance, it is necessary to design a module that adaptively selects and fuses two different domain features. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-feature adaptive selection fusion method suitable for speech sequences in view of the deficiencies of the above-mentioned prior art, so as to realize the adaptive selection fusion of multiple features.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-feature adaptive selection fusion method suitable for speech sequences, involving a speech domain alignment module, an adaptive selection module ASM and a multi-feature fusion module FFM;

[0006] The speech domain alignment module uses the input feature X2 to align the extracted input feature X1 in the time dimension to obtain the feature F0, which is expressed by the following formula:

[0007] F0=Grid_sample(X1.unsqueeze(-1),

[0008] conv3×3(Concat(Upsample(conv1×1(X1)),conv1×1(X2))).unsqueeze(2)).squeeze(-1)

[0009] Among them, unsqueeze(n) means adding a dimension of size 1 to the n-th dimension of the data; squeeze(n) means reducing a dimension of size 1 to the n-th dimension of the data; Upsample(·) is used to restore the time dimension of X1 so that the time dimension size of X1 is consistent with that of X2; conv1×1(·) and conv3×3(·) are used to extract semantic flows of different granularities in X1 and X2; Grid_sample(·) is a method to extract the expanded semantic flows. The bilinear interpolation function fills the feature values ​​at the corresponding positions in the semantic stream;

[0010] The adaptive selection module ASM includes an adaptive Map block and a selection Map block to achieve feature refinement and selection of any input feature;

[0011] The adaptive Map block refines the features of each channel in the input features. The specific process is as follows:

[0012] Input any feature x and first go through two different pooling operations to obtain features that contain both salient information and global information. Then the pooled results are multiplied by the learnable parameters α and β to obtain the feature F avg and F max , then multiply each by 1 / 2 and add them together to get the feature F middle , and then use one-dimensional convolution and activation function to realize the information interaction between different feature channels, so as to obtain the channel refinement feature F refined-map ;

[0013] The selected Map block is used to refine the feature F refined-map Select each timing information in, specifically:

[0014] The refined feature f refined-map The channel axis is divided into two groups. One group is to select a certain proportion of channels from the N-dimensional channels and assign them to 1, and the remaining channels are assigned to 0, forming an important feature map F import ; Another group selects a certain proportion of channels from the N-dimensional channels and assigns them to 0, and the remaining channels are assigned to 1, as the less important feature map F sub-import ; Then the original input feature x is element-wise multiplied with the two sets of features to obtain important features and less important features; then these two features are concatenated after average pooling and maximum pooling respectively, and input into a shared convolutional layer, normalization layer and activation layer to obtain the final refined selection feature F refined ;

[0015] The multi-feature fusion module refines and selects the F0 obtained by the speech domain alignment module and the input feature X2 through the adaptive selection module ASM and then fuses them to enrich the representation ability of the original features; the specific process is as follows:

[0016] The multi-feature fusion module takes feature F0 and feature X2 as input data. First, feature F0 is passed through the adaptive selection module ASM to obtain the refined selected feature F refined , refine the selected feature F refined The features are linearly projected through the fully connected layer to obtain the F1 feature. Similarly, the input feature X2 is refined and selected through the adaptive selection module ASM and linearly projected to obtain the F2 feature. The linear projection operation makes the features in different spaces in the same feature space to better integrate the features from different domains.

[0017] Secondly, it is necessary to fuse the input feature X2 and feature F1 to obtain a scale-invariant interactive feature; use a 1D convolution with a convolution kernel of 1 and a step size of 1 to extract information between different channels of feature X2 and feature F1, and then cascade the features of the two branches and input them into a 1D convolution with a convolution kernel of 3 and a step size of 1 to extract information from different channels between different features; then perform feature refinement selection and linear projection operations through the adaptive selection module ASM to obtain a refined and selected interactive feature F3;

[0018] Next, two-stage correlation calculation is used to update the F1 and F3 features in series to obtain new features F′1 and F′3; the first stage of correlation calculation extracts the highly relevant information of the F1 feature based on features F3 and F2, reduces the redundant features in F1, and obtains feature F′1; the second stage of correlation calculation extracts the highly relevant information of the interactive feature F3 based on F′1 and F2, reduces the redundant features, and obtains feature F′3; at the same time, the input feature X2 is refined and selected through the adaptive selection module ASM and used as feature F′2, as shown in the following formula:

[0019] F′1=SoftMax(Matmul(F3, F2))*F1

[0020] F′3=SoftMax(Matmul(F′1, F2))*F3

[0021] F′2=ASM(F2)

[0022] Among them, Matmul() represents the dot product operation;

[0023] Finally, by setting the learnable parameters w1 and w2, the features F′3 and F′2 are adaptively learned respectively, and the two features are concatenated to obtain the final fusion feature F FusedInput to the separation network to improve the performance of the separation model, as shown in the following formula:

[0024] F Fused =w1*F′3+w2*F′2.

[0025] The beneficial effect of adopting the above technical solution is that the multi-feature adaptive selection fusion method suitable for speech sequences provided by the present invention can effectively fuse feature vectors that are inconsistent in two time dimensions. Among them, the speech domain alignment module can align the two input features at the sampling point level, and at the same time can also capture the semantic information of the features to strengthen the semantic information of the features, so that the aligned features are more accurate. The adaptive selection module can not only refine each channel feature of the input feature, but also refine each time series information, so as to select important feature information from the rough original features to reduce feature redundancy. The multi-feature fusion module can use the correlation information between features to adaptively learn a variety of different features, and the obtained fusion features can further improve the separation accuracy and separation quality of the separation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the structure of a speech alignment module provided in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the structure of an adaptive selection module ASM provided in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the structure of the multi-feature fusion module FFM provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0030] In this embodiment, a multi-feature adaptive selection and fusion method applicable to a speech sequence involves a speech domain alignment module, an adaptive selection module ASM and a multi-feature fusion module FFM:

[0031] The speech domain alignment module is as follows Figure 1 As shown, the input features are used in the time dimension Aligning extracted input features Among them, B represents the number of speech sequence samples, N represents the dimension of the speech sequence sample features, L1 and L2 represent the length of the speech sequence respectively, and the feature F0 is obtained, which is expressed by the following formula:

[0032] F0=Grid_sample(X1.unsqueeze(-1),

[0033] conv3×3(Concat(Upsample(conv1×1(X1)),conv1×1(X2))).unsqueeze(2)).squeeze(-1)

[0034] Among them, unsqueeze(n) means adding a dimension of size 1 to the n-th dimension of the data; squeeze(n) means reducing a dimension of size 1 to the n-th dimension of the data; Upsample(·) is used to restore the time dimension of X1 so that the time dimension size of X1 is consistent with that of X2; conv1×1(·) and conv3×3(·) are used to extract semantic flows of different granularities in X1 and X2; Grid_sample(·) is a method to extract the expanded semantic flows. The bilinear interpolation function fills the feature values ​​at the corresponding positions in the semantic stream;

[0035] The adaptive selection module ASM is as follows Figure 2 As shown, it includes an adaptive Map block and a selection Map block to realize feature refinement and selection of any input feature, for example, refinement and selection of feature F0;

[0036] The adaptive Map block refines the features of each channel in the input features (N-dimensional channel features), and the specific process is as follows:

[0037] Input any feature x and first go through two different pooling operations to obtain features that contain both salient information and global information. Then the pooled results are multiplied by the learnable parameters α and β to obtain the feature F avg and F max , then multiply each by 1 / 2 and add them together to get the feature F middle , and then use one-dimensional convolution and activation function to realize the information interaction between different feature channels (N-dimensional feature channels), so as to obtain the feature F of channel refinement processing refined-map ; The final feature F after channel refinement refined-map Not only can the original features be retained, but the module can also adaptively adjust the features to enrich the feature F0; this process can be expressed by the following formula:

[0038] F avg =α*AvgPooling(x)

[0039] F max =β*MaxPooling(x)

[0040] F middle =0.5*F avg +0.5*Fmax

[0041] F refined-map =Sigmoid(Conv1d(F avg +F middle +F max ))

[0042] In the above formula, AvgPooling(·) represents the average pooling function, MaxPooling(·) represents the maximum pooling function; Conv1d(·) represents the 1D convolution operation, and Sigmoid(·) is the activation function.

[0043] The selected Map block is used to refine the feature F refined-map Each time series information in (a total of L1-dimensional time series features) is selected, specifically:

[0044] The refined feature F refined-map The channel axis is divided into two groups. One group is to select a certain proportion of channels from the N-dimensional channels and assign them to 1, and the remaining channels are assigned to 0, forming an important feature map F import ; Another group selects a certain proportion of channels from the N-dimensional channels and assigns them to 0, and the remaining channels are assigned to 1, as the less important feature map F sub-import ; Then the original input feature x is element-wise multiplied with the two sets of features to obtain important features and less important features; then the two features are concatenated (Concat) after average pooling (Avg) and maximum pooling (Max) respectively, and then input into a shared convolutional layer (Conv1d), normalization layer (BatchNorm) and activation layer (ReLU, Sigmoid) to obtain the final refined selection feature F refined ; The process can be expressed by the following formula:

[0045] F import =[1, 1, 1, ..., 0, 0, 0]

[0046] F sub-import =[0, 0, 0, ..., 1, 1, 1]

[0047] F main =F import *x

[0048] F sub =F sub-import *x

[0049] F′ main =Sigmoid(ReLU(BatchNorm(Conv1d(concat(Avg(F main ), Max(F main )))))))*Fmain

[0050] F′ sub =Sigmoid(ReLU(BatchNorm(Conv1d(Concat(Avg(F sub ), Max(F sub )))))))*F sub

[0051] F refined =F′ main *F′ sub

[0052] The above process of refining and selecting features (ASM module) is carried out inside the multi-feature fusion module. Figure 3 As shown in the figure, the F0 obtained by the speech domain alignment module and the input feature X2 are refined and selected by the adaptive selection module ASM and then fused to enrich the representation ability of the original features; the specific process is as follows:

[0053] The multi-feature fusion module takes feature F0 and feature X2 as input data. First, feature F0 is passed through the adaptive selection module ASM to obtain the refined selected feature F refined , refine the selected feature F refined The features are linearly projected through the fully connected layer to obtain the F1 feature. Similarly, the input feature X2 is selected through the adaptive selection module ASM and linearly projected to obtain the F2 feature. The linear projection operation makes the features in different spaces in the same feature space to better integrate the features from different domains, as shown in the following formula:

[0054] F1=FC(ASM(F0))

[0055] F2=FC(ASM(X2))

[0056] Secondly, the input feature X2 and feature F1 need to be fused to obtain a scale-invariant interactive feature; the main implementation method is to use a 1D convolution (Conv1x1) with a convolution kernel of 1 and a step size of 1 to extract the information between different channels of feature X2 and feature F1, and then concatenate the features of the two branches and input them into a 1D convolution (Conv3x3) with a convolution kernel of 3 and a step size of 1 to extract different channel information between different features; then perform feature refinement selection (ASM) and linear projection (FC) operations through the adaptive selection module ASM to obtain a refined selected interactive feature F3, as shown in the following formula:

[0057] F3=FC(ASM(Conv3x3(Concat(Conv1x1(F1),Conv1x1(x2))))

[0058] Next, two-stage correlation calculation is used to update the F1 and F3 features in series to obtain new features F′1 and F′3; the first stage of correlation calculation extracts the highly relevant information of the F1 feature based on features F3 and F2, reduces the redundant features in F1, and obtains feature F′1; the second stage of correlation calculation extracts the highly relevant information of the interactive feature F3 based on F′1 and F2, reduces the redundant features, and obtains feature F′3; at the same time, the input feature X2 is refined and selected through the adaptive selection module ASM and used as feature F′2, as shown in the following formula:

[0059] F′1=SoftMax(Matmul(F3, F2))*F1

[0060] F′3=SoftMax(Matmul(F′1,F2))*F3

[0061] F′2=ASM(F2)

[0062] Among them, Matmul() represents the dot product operation;

[0063] Finally, by setting the learnable parameters w1 and w2, the features F′3 and F′2 are adaptively learned respectively, and the two features are concatenated to obtain the final fusion feature F Fused Input to the separation network to improve the performance of the separation model, as shown in the following formula:

[0064] F Fused =w1*F′3+w2*F′2.

[0065] In this embodiment, the original input features and The intermediate features obtained Output the final fused features

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-feature adaptive selection fusion method suitable for speech sequences, characterized by: Involving the speech domain alignment module, adaptive selection module ASM and multi-feature fusion module FFM; The speech domain alignment module aligns the extracted input feature X1 with the input feature X2 in the time dimension to obtain feature F0; The adaptive selection module ASM includes an adaptive Map block and a selection Map block to achieve feature refinement and selection of any input feature; The adaptive Map block refines the features of each channel in the input features; The selection Map block selects each time series information in the refined features; The multi-feature fusion module refines and selects the F0 obtained by the speech domain alignment module and the input feature X2 through the adaptive selection module ASM and then fuses them to enrich the representation capability of the original features.

2. The multi-feature adaptive selection fusion method applicable to speech sequences according to claim 1, characterized in that: The speech domain alignment module obtains the feature F0 through the following formula: F0=Grid_sample(X1.unsqueeze(-1), conv3×3(Concat(Upsample(conv1×1(X1)),conv1×1(X2))).unsqueeze(2)).squeeze(-1) Among them, unsqueeze(n) means adding a dimension with a size of 1 to the n-th dimension of the data; squeeze(n) means reducing a dimension with a size of 1 to the n-th dimension of the data; Upsample(·) is used to restore the time dimension of X1 so that the time dimension size of X1 is consistent with that of X2; conv1×1(·) and conv3×3(·) are used to extract semantic flows of different granularities in X1 and X2; Grid_sample(·) is a method to extract the expanded semantic flows. The feature values ​​at the corresponding positions in are filled into the bilinear interpolation function in the semantic stream.

3. The multi-feature adaptive selection fusion method applicable to speech sequences according to claim 2, characterized in that: The adaptive Map block refines the features of each channel in the input features. The specific process is as follows: Input any feature x and first go through two different pooling operations to obtain features that contain both salient information and global information. Then the pooled results are multiplied by the learnable parameters α and β to obtain the feature F avg and F max , then multiply each by 1 / 2 and add them together to get the feature F middle , and then use one-dimensional convolution and activation function to realize the information interaction between different feature channels, so as to obtain the channel refinement feature F refined-map .

4. The multi-feature adaptive selection fusion method applicable to speech sequences according to claim 3, characterized in that: The selection Map block selects each time series information in the refined features, specifically: The refined feature F refined-map The channel axis is divided into two groups. One group is to select a certain proportion of channels from the N-dimensional channels and assign them to 1, and the remaining channels are assigned to 0, forming an important feature map F import ; Another group selects a certain proportion of channels from the N-dimensional channels and assigns them to 0, and the remaining channels are assigned to 1, as the less important feature map F sub-import ; Then the original input feature x is element-wise multiplied with the two sets of features to obtain important features and less important features; The two features are then concatenated after average pooling and maximum pooling respectively, and input into a shared convolutional layer, normalization layer, and activation layer to obtain the final refined selection feature F. refined .

5. The multi-feature adaptive selection fusion method applicable to speech sequences according to claim 4, characterized in that: The specific process of the multi-feature fusion module fusing the F0 obtained by the speech domain alignment module and the input feature X2 after refinement and selection by the adaptive selection module ASM is as follows: The multi-feature fusion module takes feature F0 and feature X2 as input data. First, feature F0 is passed through the adaptive selection module ASM to obtain the refined selected feature F refined , refine the selected feature F refined The features are linearly projected through the fully connected layer to obtain the F1 feature. Similarly, the input feature X2 is refined and selected through the adaptive selection module ASM and linearly projected to obtain the F2 feature. The linear projection operation makes the features in different spaces in the same feature space to better integrate the features from different domains. Secondly, it is necessary to fuse the input feature X2 and feature F1 to obtain a scale-invariant interactive feature; use a 1D convolution with a convolution kernel of 1 and a step size of 1 to extract information between different channels of feature X2 and feature F1, and then cascade the features of the two branches and input them into a 1D convolution with a convolution kernel of 3 and a step size of 1 to extract information from different channels between different features; then perform feature refinement selection and linear projection operations through the adaptive selection module ASM to obtain a refined and selected interactive feature F3; Next, two-stage correlation calculation is used to update the F1 and F3 features in series to obtain new features F′1 and F′3; the first stage of correlation calculation extracts the highly relevant information of the F1 feature based on features F3 and F2, reduces the redundant features in F1, and obtains feature F′1; the second stage of correlation calculation extracts the highly relevant information of the interactive feature F3 based on Fv1 and F2, reduces the redundant features, and obtains feature F′3; at the same time, the input feature X2 is refined and selected through the adaptive selection module ASM and used as feature F′2, as shown in the following formula: F′1=SoftMax(Matmul(F3,F2))*F1 F′3=SoftMax(Matmul(F′1,F2))*F3 F′2=ASM(F2) Among them, Matmul() represents the dot product operation; Finally, by setting the learnable parameters w1 and w2, the features F′3 and F′2 are adaptively learned respectively, and the two features are concatenated to obtain the final fusion feature F Fused Input to the separation network to improve the performance of the separation model, as shown in the following formula: F Fused =w1*F′3+w2*F′2.