Target Speaker Speech Extraction Method, System, Device and Medium Based on Contrastive Learning

Abstract

The present invention discloses a method, system, device and medium for extracting target speaker speech based on contrastive learning. The method comprises the following steps: in the pre-training stage, data enhancement is performed on unlabeled data after the same speech is cut into one long and one short; the one long and one short speech are formed into positive example pairs in contrastive learning; contrastive learning is used on the auxiliary network of the target speaker speech extraction network to extract features of the long and short speech, and the features between the long and short speech are automatically learned to be closer; the auxiliary network after pre-training is obtained, and the features of the main network during the separation of mixed speech in the target speaker speech extraction network and the features of the reference speech obtained by the auxiliary network are perceived by the attention mechanism, and adaptive fusion of the features is performed; the target speaker speech is extracted according to the adaptively fused features. The present invention captures the difference in features of long and short speech through contrastive learning, and closes its features through automatic learning, thereby effectively improving the performance of target speaker speech extraction when the input reference speech is too short.

Description

Technical Field

[0001] The present invention belongs to the technical field of target speaker voice extraction in speech, and in particular relates to a target speaker voice extraction method, system, equipment and medium based on contrastive learning. Background Art

[0002] During online conferences, we often encounter interference from noise or non-target speaker voices, which reduces the call quality of the conference. Target speaker voice extraction is an effective processing technology for the problem of background third-party human voice elimination. Its purpose is to improve the overall call clarity and improve the quality of voice signals by eliminating background noise and non-target speaker voice signals.

[0003] At present, in non-realistic scenarios, that is, closed scenarios, target speaker speech extraction tasks have achieved good results, but they still have some shortcomings, such as ignoring scenarios such as short reference speech or speakers that have not been encountered in reality. Due to the inability to better identify the characteristics of the reference speech, the performance of extracting the target speaker's speech is reduced. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method, system, device and medium for extracting the target speaker's speech based on contrastive learning. By using contrastive learning to capture the differences in long and short speech features and by automatically learning to bring their features closer, the performance of extracting the target speaker's speech is effectively improved when the input reference speech is too short.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] On one hand, the present invention provides a method for extracting target speaker speech based on contrastive learning, comprising the following steps:

[0007] In the pre-training stage, the unlabeled data is truncated, and the same speech is cut into a long and a short speech, and data enhancement is used for processing;

[0008] Cutting the same speech into a long speech and a short speech as positive example pairs in contrastive learning, and the other speech as negative example pairs;

[0009] Using contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learning to bring the features between long and short speech closer and alienate the features between negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers;

[0010] Obtain a pre-trained auxiliary network, input the target speaker's reference speech into the auxiliary network to obtain the feature representation of the target speaker's speech, use the attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features;

[0011] A mask is generated for the input mixed speech according to the adaptive fusion of the features to cover up the non-target speech and complete the extraction of the target speaker's speech.

[0012] Preferably, the data enhancement has four aspects: 1. fade-in / fade-out, i.e., the audio intensity is randomly and gradually increased / decreased at the beginning / end of the audio signal; 2. time masking, i.e., randomly selecting a small segment of the audio signal and setting its value to normal noise or a constant value; 3. pitch shifting, i.e., randomly raising or lowering the pitch of the audio signal; 4. time shifting, i.e., randomly moving the audio sample forward or backward.

[0013] Preferably, the positive example pair is: The negative example pair is: in, For long voice, For short voice, x k For other enhanced unlabeled speech.

[0014] Preferably, the method uses contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learns to bring the features between long and short speech closer and to distance the features between the long and short speech pairs from the negative example pairs, specifically:

[0015] First, the left and right sides of the input speech x are padded with 0 to different lengths, then convolved, and finally the encoder results of multiple different scales are concatenated as the output e after speech encoding;

[0016] Secondly, the encoded output e is subjected to convolution and multiple ResBlocks, and then convolution again to complete the feature extraction z of the speech. The calculation formula of the ResBlock is:

[0017] ResBlock(e)=MaxPool(PReLu(e+Conv1d(PReLu(Conv1d(e)))))

[0018] MaxPool refers to the maximum pooling operation; PReLu refers to the nonlinear activation function;

[0019] Will x kRepeat the steps of obtaining the output of the speech encoder and completing the feature extraction of the speech to obtain the corresponding z k ; Then calculate the loss l i :

[0020]

[0021] Among them, 1 [k≠i] ∈0,1 is an indicator function that evaluates to 1 when k≠i; τ represents the temperature parameter, x k For other enhanced unlabeled speech, is the feature of the extracted long speech. is the feature of the extracted short speech, z k are the features of other enhanced unlabeled speech extracted;

[0022] Calculate positive pairs and negative pairs Finally, the auxiliary network is automatically optimized through the gradient descent method, and automatically learns to bring the features between long and short speech closer and the features between negative examples farther away.

[0023] Preferably, the pre-trained auxiliary network is obtained, the reference speech of the target speaker is input into the auxiliary network to obtain the feature representation of the target speaker's speech, and the attention mechanism is used to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and adaptively fuse the features. The specific process is as follows:

[0024] First, the same operation is performed on the input mixed speech Y and the output e after speech encoding to obtain the intermediate state e after mixed speech Y encoding. Y , and then e Y Perform convolution to obtain the intermediate state feature representation Y of the mixed speech Y in ;

[0025] The reference speech X of the target speaker is subjected to the speech feature extraction operation to obtain the feature representation z of the reference speech. X , and then after the feature dimension is replicated and combined with Y in The feature dimension is aligned, and this feature is called E; then the attention mechanism is used to perceive Y in The feature between E and Y, where the attention mechanism calculates the intermediate state feature representation Y of the mixed speech Y frame by frame in The attention weight between the speaker's speech embedding E is calculated and applied to the final output adaptively fused feature E' to achieve adaptive fusion. The specific process is:

[0026] Get E,Y inThe output after downsampling, and the results of the downsampled output are respectively passed through the first transformation matrix to obtain the output of the first linear transformation, and at the same time, E and Y are calculated. in The result of the attention mechanism; the specific calculation formula is:

[0027] E ds , Y ds = DS4(E, Y in )

[0028]

[0029] Among them, is the first transformation matrix; DS4 represents a 4-fold downsampling operation on the input; E ds , Y ds are the outputs after downsampling of E and Y in ; is the output of the first linear transformation; d is the scaling factor; C Y,E is the result of applying the attention mechanism to E and Y in ;

[0030] Subsequently, the sum of the results of applying the attention mechanism to E and Y in and the downsampled output Y ds is regularized to obtain the result of the first-layer normalization; at the same time, the results of the first-layer normalization are respectively passed through the second transformation matrix to obtain the output of the second linear transformation; and the attention mechanism is applied to calculate the results of the first-layer normalization; the specific calculation formula is:

[0031] C (Y,E)′ = LayerNorm(C Y,E + Y ds )

[0032]

[0033] LayerNorm is a regularization operation; C (Y,E)′ is the result of the first-layer normalization; is the second transformation matrix; is the output of the second linear transformation; S is the result of applying the attention mechanism to C (Y,E)′ ;

[0034] Finally, calculate the result after the second-layer normalization and the residual connection, and upsample the result after the residual connection to obtain the adaptively fused feature; the specific calculation formula is:

[0035] S′ = LayerNorm(S + C (Y,E)′ )

[0036] E′ds = E ds + S'

[0037] E' = US4(E' ds )

[0038] where US4 represents upsampling the input by a factor of 4; S' is the result of the second layer normalization; E' ds is the result after residual connection; E' is the feature after adaptive fusion.

[0039] Preferably, generating a mask for the input mixed speech according to the adaptively fused features is specifically as follows:

[0040]

[0041] S'(x) = Conv1d(PReLU(DeConv1d(PReLu(Conv1d(x)))))

[0042] OUTPUT StackedTCNs = S(Y in , E') x4

[0043] OUTPUT Mask = ReLu(Conv1d(OUTPUT StackedTCNs ))

[0044] where OUTPUT StackedTCNs is composed of 4 nested S(), and among them, 6 S'() are nested. DeConv1d refers to dilated depthwise separable convolution; S(Y in , E') is an intermediate variable calculated according to the formula for the input Y in and the adaptively fused feature E'; S'(x) is a specific explanation of the process of the S'() formula; OUTPUT StackedTCNs is the result calculated after four stacks of S(Y in , E'); OUTPUT Mask is the mask generated according to Y in and E'.

[0045] Preferably, masking the non-target speech and completing the extraction of the target speaker's speech is to perform deconvolution on the mask OUTPUT in generated by Y Mask and E' with the intermediate state e Y after encoding the mixed speech Y to obtain the reconstructed output speech, and the reconstructed output speech only contains the speech of the target speaker.

[0046] In another aspect, the present invention provides a target speaker speech extraction system based on contrastive learning, which is applied to the target speaker speech extraction method based on contrastive learning, and includes an auxiliary network preprocessing module, an auxiliary network feature extraction module, a contrastive learning module, a feature fusion module, a main network feature extraction module, and a main network speech separation module;

[0047] The auxiliary network preprocessing module is used to truncate the unlabeled data input to the auxiliary network into a long and a short speech in the pre-training stage, perform data enhancement, and construct positive example pairs and negative example pairs;

[0048] The auxiliary network feature extraction module is used to extract features from the input speech;

[0049] The contrastive learning module is used to extract features of long and short speech using contrastive learning on the auxiliary network of the target speaker speech extraction network, and automatically learn to bring the features between the long and short speech closer and alienate the features between the negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers;

[0050] The feature fusion module is used to obtain a pre-trained auxiliary network, input the reference speech of the target speaker into the auxiliary network to obtain a feature representation of the target speaker's speech, use an attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features;

[0051] The main network feature extraction module is used to extract features during mixed speech separation;

[0052] The main network speech separation module is used to generate a mask for the input mixed speech according to the features of the adaptive fusion, to cover up the speech of the non-target speaker, and to complete the extraction of the speech of the target speaker.

[0053] Another aspect of the present invention provides an electronic device, characterized in that the electronic device comprises:

[0054] at least one processor; and,

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the target speaker speech extraction method based on contrastive learning.

[0057] In another aspect, the present invention provides a computer-readable storage medium storing a program, wherein when the program is executed by a processor, the target speaker speech extraction method based on contrastive learning is implemented.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] 1. In order to fully obtain the feature information of short reference speech, the present invention uses contrastive learning to bring the features between long and short reference speech closer. Specifically, the present invention uses a large amount of unlabeled speech data for truncation and enhancement, and uses the enhanced long and short speech as the input of the auxiliary network in the target speaker speech extraction network, learns to distinguish the features of different speakers in a large amount of data, and optimizes the auxiliary network through the loss function of contrastive learning to bring the features between long and short reference speech closer, so that the features extracted from the long speech can be captured on the short speech, thereby effectively obtaining the feature information of the short reference speech.

[0060] 2. The present invention uses an attention-based feature fusion method to adaptively fuse reference speech features with the features of mixed speech input, thereby improving the performance of generating masks, better masking noise and third-party voices, and improving the performance of target speaker speech extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 Flow chart of a target speaker speech extraction method based on contrastive learning in an embodiment of the present invention;

[0063] Figure 2 It is a framework diagram of a target speaker speech extraction method based on contrastive learning in an embodiment of the present invention;

[0064] Figure 3 It is a structural diagram of a target speaker speech extraction system based on contrastive learning in an embodiment of the present invention;

[0065] Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention;

[0066] Figure 5 This is the structural diagram of the computer-readable storage medium in the embodiment of the present invention. Detailed implementation manners

[0067] In order to enable those skilled in the art of this technology to better understand the solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

[0068] The mention of "embodiment" in this application means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0069] Please refer to Figure 1 、 Figure 2 In one embodiment of this application, a method for extracting the target speaker's voice based on contrastive learning is provided, including the following steps:

[0070] S1. First, in the pre-training stage, a large amount of unlabeled data is truncated. The same piece of voice is truncated into a long and a short voice, and data augmentation is used for processing. The processing process is expressed as:

[0071]

[0072] where x unlabel is the unlabeled voice data provided by the unlabeled voice data set; cut() means truncating the input voice data into a long and a short one; Aug() means performing augmentation processing on the input voice data; is the processed long voice, and is the processed short voice, where 0 ≤ i ≤ N, and N is the input batch size.

[0073] The data augmentation has four aspects: 1. Fade-in / fade-out, that is, the audio intensity randomly increases / decreases gradually at the beginning / end of the audio signal; 2. Time masking, that is, randomly selecting a small segment of the audio signal and setting its value to normal noise or a constant value; 3. Pitch shift, that is, randomly increasing or decreasing the pitch of the audio signal; 4. Time shift, that is, randomly moving the audio samples forward or backward.

[0074] S2, obtaining positive example pairs and negative example pairs, specifically, cutting the same speech into a long speech and a short speech as positive example pairs in comparative learning, and the other speech as negative example pairs;

[0075] Positive example:

[0076] Negative pair:

[0077] S3. Using contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learning to bring the features between long and short speech closer and alienate the features between negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers.

[0078] S31, using contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learning to bring the features between long and short speech closer and alienate the features between negative example pairs, specifically: first, pad the left and right sides of the input speech x with 0 to different lengths, then perform convolution, and finally concat multiple encoder results of different scales as speech encoder input, the calculation process is:

[0079] f i (x) = Covd1d(Pad(x))

[0080] e=Concat(f1(x),f2(x),f3(x))

[0081] Among them, f i () represents encoders of different scales, i.e., f1(), f2(), f3(), Concat refers to the concatenation operation of vectors, Pad is the padding operation, Covd1d represents one-dimensional convolution, x is the input speech, and e is the encoded output;

[0082] S32. Secondly, the encoded output is subjected to one-dimensional convolution and multiple ResBlocks, and then subjected to one-dimensional convolution to complete the feature extraction of the speech; the formula for completing the feature extraction of the speech is:

[0083] z=Convd1d(ResBlock(Covd1d(e))x3)

[0084] ResBlock(e)=MaxPool(PReLu(e+Conv1d(PReLu(Conv1d(e)))))

[0085] Among them, PReLu refers to the nonlinear activation function; z refers to the extracted features corresponding to the input speech x;

[0086] Will x k Repeat the steps of obtaining the output of the speech encoder and completing the feature extraction of the speech to obtain the corresponding z k ; Then calculate the loss l i :

[0087]

[0088] Among them, 1 [k≠i] ∈0,1 is an indicator function that evaluates to 1 when k≠i; τ represents the temperature parameter; x k For other enhanced unlabeled speech, is the feature of the extracted long speech. is the feature of the extracted short speech, z k are the features of other enhanced unlabeled speech extracted;

[0089] Calculate positive pairs and negative pairs The cosine similarity of the positive example The cosine similarity formula is:

[0090]

[0091] The negative examples The formula for calculating cosine similarity is:

[0092]

[0093] S33. Finally, the auxiliary network is automatically optimized through the gradient descent method, and automatically learns how to bring the features between long and short speech closer and distance the features between negative examples.

[0094] S4. Obtain the pre-trained auxiliary network, input the reference speech of the target speaker into the auxiliary network to obtain the feature representation of the target speaker's speech, use the attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features. The specific process is as follows:

[0095] First, the input mixed speech Y and the encoded output e are operated in the same way to obtain the intermediate state e after the mixed speech Y is encoded. Y , and then e Y Perform one-dimensional convolution to obtain the intermediate state feature representation Y of the mixed speech Y in , the specific calculation process is:

[0096] Y in = Covd1d(e Y )

[0097] e Y = Concat(f1(Y), f2(Y), f3(Y))

[0098] f i (Y) = Covd1d(Pad(Y))

[0099] For the reference speech X of the target speaker, through the above-mentioned speech feature extraction operation, the feature representation z of the reference speech can be obtained X , and after simple replication of the feature dimension, it is aligned with the feature dimension of Y in , and this feature is called E; then the attention mechanism is used to perceive the features between Y in and E. Among them, the attention mechanism calculates the attention weights between the intermediate state feature representation Y in of the mixed speech Y frame by frame and the speaker speech embedding E, and applies this attention weight to the finally output adaptively fused feature E', to achieve adaptive fusion. The specific process is as follows:

[0100] First, obtain the downsampled outputs of E, Y in , and pass the downsampled output results through the first transformation matrix respectively to obtain the output of the first linear transformation. At the same time, calculate the results after applying the attention mechanism to E, Y in ; The specific formula is:

[0101] E ds , Y ds = DS4(E, Y in )

[0102]

[0103] Among them, is the first transformation matrix; DS4 represents a 4-fold downsampling operation on the input; E ds , Y ds are the downsampled outputs of E, Y in ; is the output of the first linear transformation; d is the scale factor; C Y,E is the result after applying the attention mechanism to E, Y in ;

[0104] Subsequently, the result after applying the attention mechanism to E, Y in is combined with the downsampled output Y dsNormalize the sum to obtain the result of the first layer normalization; at the same time, pass the result of the first layer normalization through the second transformation matrix respectively to obtain the output of the second linear transformation; and apply the attention mechanism to the result of the first layer normalization for calculation; the specific formula is:

[0105] C (Y,E)′ = LayerNorm(C Y,E + Y ds )

[0106]

[0107] LayerNorm is a normalization operation; C (Y,E)′ is the result of the first layer normalization; is the second transformation matrix; is the output of the second linear transformation; S is the result after applying the attention mechanism to C (Y,E)′ for calculation;

[0108] Finally, calculate the result after connecting the second layer normalization result and the residual connection, and upsample the result after the residual connection to obtain the adaptively fused feature; the specific calculation formula is::

[0109] S′ = LayerNorm(S + C (Y,E)′ )

[0110] E′ ds = E ds + S′

[0111] E′ = US4(E′ ds )

[0112] where, US4 represents upsampling the input by 4 times; S′ is the result of the second layer normalization; E′ ds is the result after the residual connection; E′ is the adaptively fused feature.

[0113] S5. Generate a mask for the input mixed speech according to the adaptively fused feature to mask the non-target speech and complete the extraction of the target speaker's speech. The specific process is as follows:

[0114]

[0115] S′(x) = Conv1d(PReLu(DeConv1d(PReLu(Conv1d(x)))))

[0116] OUTPUT StackedTCNs = S(Y in , E′)x4

[0117] OUTPUT Mask= ReLu(Conv1d(OUTPUT StackedTCNs ))

[0118] where OUTPUT StackedTCNs is composed of 4 nested S(), and among them, 6 S'() are nested. DeConv1d refers to dilated depthwise separable convolution; S(Y in , E') is an intermediate variable calculated according to the formula for the input Y in and the feature E' after adaptive fusion; S'(x) is a specific explanation of the process of the S'() formula; OUTPUT StackedTCNs is the result calculated after stacking S(Y in , E') four times; OUTPUT Mask is a mask generated according to Y in and E'; subsequently, the mask OUTPUT in generated by Y Mask and E' is deconvolved one-dimensionally with the intermediate state e Y after encoding the mixed speech Y to obtain the reconstructed output speech. The calculation formula for the reconstructed output speech is:

[0119] OUTPUT = ConvTrans1D((OUTPUT Mask * e Y ))

[0120] where ConvTrans1D is one-dimensional deconvolution; OUTPUT is the reconstructed output speech, which only contains the speech of the target speaker, and finally the extraction of the target speaker's speech is completed.

[0121] To verify the effectiveness of the present invention, the method for extracting the target speaker's speech based on contrastive learning in this embodiment is compared with the existing technology SpEx+ on the public dataset LibriMix to test the performance of extracting the target speaker's speech. The SDRi and SI-SDRi scores are used in the test metrics. The higher the score, the better the extraction performance. The method of the present invention can obtain 15.11 dB and 14.10 dB under the input of the reference speech with normal length, while SpEx+ only reaches 14.06 dB and 13.17 dB. Therefore, the present invention has better performance than the existing technology.

[0122] To further verify the effectiveness of the present invention, the performance results under the input of short reference speech are tested. The present invention can obtain 13.87 dB and 12.43 dB, while SpEx+ only reaches 11.58 dB and 10.56 dB. Therefore, it further shows that the present invention has better performance than the existing technology.

[0123] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0124] Based on the same idea as the target speaker speech extraction method based on contrastive learning in the above embodiment, the present invention also provides a target speaker speech extraction system based on contrastive learning, which can be used to execute the above target speaker speech extraction method based on contrastive learning. For ease of explanation, the structural diagram of the target speaker speech extraction system based on contrastive learning only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0125] See also Figure 3 In another embodiment of the present application, a target speaker speech extraction system 100 based on contrastive learning is provided, the system comprising an auxiliary network preprocessing module 101, an auxiliary network feature extraction module 102, a contrastive learning module 103, a feature fusion module 104, a main network feature extraction module 105 and a main network speech separation module 106;

[0126] The auxiliary network preprocessing module 101 is used to truncate the unlabeled data input to the auxiliary network into a long and a short speech in the pre-training stage, perform data enhancement, and construct positive example pairs and negative example pairs;

[0127] The auxiliary network feature extraction module 102 is used to extract features from the input speech;

[0128] The contrastive learning module 103 is used to extract features of long and short speech using contrastive learning on an auxiliary network of the target speaker speech extraction network, and automatically learn to bring the features between long and short speech closer and alienate the features between negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers;

[0129] The feature fusion module 104 is used to obtain a pre-trained auxiliary network, input the reference speech of the target speaker into the auxiliary network to obtain a feature representation of the target speaker's speech, use an attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features;

[0130] The main network feature extraction module 105 is used for feature extraction during hybrid speech separation;

[0131] The main network speech separation module 106 is used to generate a mask for the input hybrid speech according to the adaptively fused features, cover the speech of non-target speakers, and complete the extraction of the speech of the target speaker.

[0132] It should be noted that the target speaker speech extraction system based on contrastive learning of the present invention corresponds one-to-one with the target speaker speech extraction method based on contrastive learning of the present invention. The technical features and their beneficial effects described in the embodiments of the above-mentioned target speaker speech extraction method based on contrastive learning are applicable to the embodiments of the target speaker speech extraction system based on contrastive learning. For specific content, reference can be made to the description in the method embodiments of the present invention, which will not be repeated here. This is hereby declared.

[0133] In addition, in the implementation manner of the target speaker speech extraction system based on contrastive learning in the above embodiments, the logical division of each program module is only an example. In practical applications, according to needs, for example, considering the configuration requirements of corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the target speaker speech extraction system based on contrastive learning is divided into different program modules to complete all or part of the functions described above.

[0134] Please refer to Figure 4 , in an embodiment, an electronic device for implementing a target speaker speech extraction method based on contrastive learning is provided. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may further include a computer program stored in the first memory 202 and executable on the first processor 201, such as a target speaker speech extraction program 203 based on contrastive learning.

[0135] Please refer to Figure 5, wherein the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disc, etc. The first memory 202 can be an internal storage unit of the electronic device 200 in some embodiments, such as the mobile hard disk of the electronic device 200. The first memory 202 can also be an external storage device of the electronic device 200 in some other embodiments, such as a plug-in mobile hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the electronic device 200. Further, the first memory 202 can also include both the internal storage unit and the external storage device of the electronic device 200. The first memory 202 can be used not only to store application software installed in the electronic device 200 and various types of data, such as the code of the target speaker voice extraction program 203 for contrastive learning, etc., but also to temporarily store data that has been output or will be output.

[0136] The first processor 201 can be composed of integrated circuits in some embodiments. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions packaged, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and circuits, and by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202, to execute various functions of the electronic device 200 and process data.

[0137] Figure 4 Only the electronic device with components is shown. Those skilled in the art can understand that Figure 4 the shown structure does not constitute a limitation on the electronic device 200, and it can include fewer or more components than shown, or combine some components, or have different component arrangements.

[0138] The target speaker voice extraction program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can implement:

[0139] In the pre-training stage, the unlabeled data is truncated, and the same speech is cut into a long and a short speech, and data enhancement is used for processing;

[0140] Cutting the same speech into a long speech and a short speech as positive example pairs in contrastive learning, and the other speech as negative examples;

[0141] Using contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learning to bring the features between long and short speech closer and alienate the features between negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers;

[0142] Obtain a pre-trained auxiliary network, input the target speaker's reference speech into the auxiliary network to obtain the feature representation of the target speaker's speech, use the attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features;

[0143] A mask is generated for the input mixed speech according to the adaptive fusion of the features to cover up the non-target speech and complete the extraction of the target speaker's speech.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0145] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0146] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for extracting the target speaker's speech based on contrastive learning, characterized in that The steps include: In the pre-training stage, the unlabeled data is truncated, and the same speech is cut into a long and a short speech, and data enhancement is used for processing; Cutting the same speech into a long speech and a short speech as positive example pairs in contrastive learning, and the long speech and other enhanced unlabeled speech as negative example pairs; Using contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learning to bring the features between long and short speech closer and alienate the features between negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers; Obtain a pre-trained auxiliary network, input the target speaker's reference speech into the auxiliary network to obtain the feature representation of the target speaker's speech, use the attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features; A mask is generated for the input mixed speech according to the adaptive fusion of the features to cover up the non-target speech and complete the extraction of the target speaker's speech.

2. The method for extracting the target speaker's speech based on contrastive learning according to claim 1, wherein There are four aspects of data enhancement:

1. Fade-in / fade-out, that is, the audio intensity is randomly increased / decreased gradually at the beginning / end of the audio signal; 2. Time masking, that is, randomly selecting a small segment of the audio signal and setting its value to normal noise or a constant value; 3. Pitch shift, that is, randomly raising or lowering the pitch of the audio signal; 4. Time shift, that is, randomly moving the audio sample forward or backward.

3. The method for extracting the target speaker's voice based on contrastive learning according to claim 1, wherein, The positive example pairs are as follows: The negative example pairs are as follows: Among them, is a long speech, is a short speech, x k is other enhanced unlabeled speech.

4. The method for extracting the target speaker's speech based on contrastive learning according to claim 1, wherein, The method uses contrastive learning on the auxiliary network of the target speaker speech extraction network to extract features from long and short speech, and automatically learns to bring the features between long and short speech closer and to distance the features between the long and short speech pairs from the negative example pairs, specifically: First, the left and right sides of the input speech x are padded with 0 to different lengths, then convolved, and finally the encoder results of multiple different scales are concatenated as the output e after speech encoding; Secondly, the encoded output e is subjected to convolution and multiple ResBlocks, and then convolution again to complete the feature extraction z of the speech. The calculation formula of the ResBlock is: ResBlock(e)=MaxPool(PReLu(e+Conv1d(PReLu(Conv1d(e))))) MaxPool refers to the maximum pooling operation; PReLu refers to the nonlinear activation function; After x k repeatedly obtaining the output after the speech encoder and completing the step of speech feature extraction, corresponding z k is obtained; then the loss l i is calculated as follows: where 1 [k≠i] ∈ {0, 1} is an indicator function that evaluates to 1 when k ≠ i; τ represents the temperature parameter, x k is other enhanced unlabeled speech, is the feature of the extracted long speech, is the feature of the extracted short speech, z k is the feature of other enhanced unlabeled speech extracted; Calculate positive pairs and negative pairs Finally, the auxiliary network is automatically optimized through the gradient descent method, and automatically learns to bring the features between long and short speech closer and the features between negative example pairs farther away.

5. The method for extracting the target speaker's speech based on contrastive learning according to claim 1, wherein The pre-trained auxiliary network is obtained, the reference speech of the target speaker is input into the auxiliary network to obtain the feature representation of the target speaker's speech, and the attention mechanism is used to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and the features are adaptively fused. The specific process is as follows: First, perform the same operations on the input mixed speech Y and the output e after speech coding to obtain the intermediate state e after coding the mixed speech Y Y , and then perform convolution on e Y to obtain the intermediate state feature representation Y of the mixed speech Y in ; The reference speech X of the target speaker is subjected to the above-mentioned speech feature extraction operation to obtain the feature representation z of the reference speech X , and after replication of the feature dimension, it is aligned with the feature dimension of Y in . This feature is called E; Subsequently, the attention mechanism is used to perceive Y in and the features between E, where the attention mechanism calculates the intermediate state feature representation Y of the mixed speech Y frame by frame in and the attention weights between the speaker voice embedding E, and applies the attention weights to the finally output adaptively fused feature E', to achieve adaptive fusion. The specific process is as follows: Obtain E, Y in The output after downsampling, and the output of the downsampling result is respectively passed through the first transformation matrix to obtain the output of the first linear transformation, and at the same time calculate E, Y in The result of the attention mechanism; the specific calculation formula is: E ds ,Y ds = DS4(E, Y in ) Among them, is the first transformation matrix; DS4 represents a 4-fold downsampling operation on the input; E ds , Y ds is the output after downsampling E, Y in ; is the output of the first linear transformation; d is the scaling factor; C Y,E is the result after applying the attention mechanism to E, Y in ; Subsequently, E and Y in The result after applying the attention mechanism is regularized with the sum of the downsampled output Y ds to obtain the result of the first layer normalization; meanwhile, the result of the first layer normalization is respectively passed through the second transformation matrix to obtain the output of the second linear transformation; and the attention mechanism is applied to the result of the first layer normalization for calculation; the specific calculation formula is as follows: C (Y,E)′ = LayerNorm(C Y,E + Y ds ) LayerNorm is a regularization operation; C (Y,E)′ is the result of the first layer normalization; is the second transformation matrix; is the output of the second linear transformation; S is the result after applying the attention mechanism to C (Y,E)′ ; Finally, the second layer normalization result and the result after residual connection are calculated, and the result after residual connection is upsampled to obtain the adaptive fusion feature; the specific calculation formula is: S′ = LayerNorm(S + C (Y,E)′ ) E′ ds = E ds + S′ E′ = US4(E′ ds ) Among them, US4 represents upsampling the input by 4 times; S′ is the result of the second layer normalization; E′ ds is the result after residual connection; E′ is the feature after adaptive fusion.

6. The method for extracting the target speaker's voice based on contrastive learning according to claim 5, wherein The masking of non-target speech and the extraction of the target speaker's speech are to use Y in and the mask OUTPUT generated by E' Mask to perform deconvolution with the intermediate state e after encoding the mixed speech Y Y to obtain the reconstructed output speech, which only contains the speech of the target speaker.

7. The target speaker voice extraction system based on contrastive learning is characterized in that It includes auxiliary network preprocessing module, auxiliary network feature extraction module, contrast learning module, feature fusion module, main network feature extraction module and main network speech separation module; The auxiliary network preprocessing module is used to truncate the unlabeled data input to the auxiliary network into a long and a short speech in the pre-training stage, perform data enhancement, and construct positive example pairs and negative example pairs; The auxiliary network feature extraction module is used to extract features from the input speech; The contrastive learning module is used to extract features of long and short speech using contrastive learning on the auxiliary network of the target speaker speech extraction network, and automatically learn to bring the features between the long and short speech closer and alienate the features between the negative example pairs; the target speaker speech extraction network includes an auxiliary network and a main network; the auxiliary network learns good target speaker speech features with high representation ability, and the main network uses the target speaker speech features to learn available masks to shield speech signals from other speakers; The feature fusion module is used to obtain a pre-trained auxiliary network, input the reference speech of the target speaker into the auxiliary network to obtain a feature representation of the target speaker's speech, use an attention mechanism to perceive the features of the main network during the mixed speech separation in the target speaker's speech extraction network and the features of the reference speech obtained by the auxiliary network, and perform adaptive fusion of the features; The main network feature extraction module is used to extract features during mixed speech separation; The main network speech separation module is used to generate a mask for the input mixed speech according to the features of the adaptive fusion, to cover up the speech of the non-target speaker, and to complete the extraction of the speech of the target speaker.

8. An electronic device, characterized in that, The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the target speaker speech extraction method based on contrastive learning as described in any one of claims 1-6.

9. A computer-readable storage medium storing a program, characterized in that, When the program is executed by a processor, the target speaker speech extraction method based on contrastive learning described in any one of claims 1 to 6 is implemented.

Images