Single-channel multi-speaker factor extraction model training method, extraction method, speaker separation method and related device

Abstract

The invention provides a single-channel multi-speaker factor extraction model training method, a single-channel multi-speaker factor extraction method, a speaker separation method and a related device, and the training method comprises the steps: employing the expression content text features of first single-channel voice segment data and second single-channel voice segment data sent by each speaker in a multi-speaker alternate speaking scene, and executing a model training step based on expression content text distillation for an encoder and a decoder in the variational auto-encoder so as to train the encoder into a single-channel multi-speaker factor extraction model used for extracting speaker speech features corresponding to the single-channel speech segment data. According to the invention, the expression content text corresponding to the single-channel speech of the speaker can be distilled in the model training process, so that the redundant information interference of the expression content text on the extraction effectiveness of the speech features of the speaker can be effectively reduced; the accuracy of speaker speech features extracted by the model and the effectiveness and accuracy of subsequent application for single-channel multi-speaker separation can be improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent speech processing technology, and in particular to a single-channel multi-speaker factor extraction model training method, extraction method, speaker separation method and related devices. Background Art

[0002] In multi-person alternating speaking scenarios where there are more than two speakers, such as multi-person voice connections, online or offline, etc., based on the need for automatic recording of speech content, it is usually necessary to use electronic equipment to automatically convert the voice data emitted by each speaker in the multi-person alternating speaking scenario into text data, and then annotate the unique identifier of the speaker corresponding to each text data in the multi-person alternating speaking scenario, that is, the role information. Although the technology of speaker role annotation has been widely used in alternating speaking scenarios of two-person conversations such as telephone calls, due to the large number of people who usually speak alternately in multi-person alternating speaking scenarios and uneven length and frequency of speakers' speeches, such as sequential reports and question-and-answer scenarios in large meetings, the blind source speaker separation effect in multi-person alternating speaking scenarios is poor, which cannot meet the practical requirements of large-scale meeting records such as enterprises.

[0003] At present, in the existing multi-speaking separation method in the multi-person alternating speaking scene, the multi-person alternating speaking scene audio data is usually first divided into single-channel speech segments, and then the speaker factors corresponding to each single-channel speech segment for characterizing the speech characteristics of the speaker are extracted, and then the single-channel speech segments are clustered for the same speaker according to the speaker factor, and then the single-channel speech segments of each speaker are separated in the multi-person alternating speaking scene audio data.

[0004] However, although the extraction of speaker factors is not easily affected by the channel, speaker status and environmental noise because multiple people speaking alternately usually take place indoors and the speakers speak through microphones, the speaker's expressions are rarely repeated. Therefore, the different expressions corresponding to each single-channel speech segment can easily interfere with the accuracy and effectiveness of speaker factor extraction, thereby affecting the effectiveness and accuracy of subsequent single-channel multi-speaker separation.

[0005] Therefore, there is an urgent need to design a solution that can solve the interference effect of the expression content text corresponding to the single-channel speech on the speaker voice feature extraction of the single-channel speech in the scene where multiple people are speaking alternately. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a single-channel multi-speaker factor extraction model training method, extraction method, speaker separation method and related devices to eliminate or improve one or more defects existing in the prior art.

[0007] The first aspect of the present application provides a single-channel multi-speaker factor extraction model training method, comprising:

[0008] Obtaining the expression content text features corresponding to the first single-channel voice segment data emitted by each speaker in the multi-person alternating speaking scenario;

[0009] constructing the expression content text features of each speaker and the second single-channel voice segment data into a training sample, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data is different;

[0010] Using each of the training samples, preset model training steps based on text distillation of expression content are performed for the encoder and decoder in the variational autoencoder respectively, so as to train the encoder as a single-channel multi-speaker factor extraction model for extracting speaker speech features corresponding to single-channel speech fragment data.

[0011] In some embodiments of the present application, the model training step based on the expression content text distillation includes:

[0012] Inputting the second single-channel speech segment data in the current training sample into the encoder in the variational autoencoder, so that the encoder outputs the speaker speech features corresponding to the second single-channel speech segment data;

[0013] The speaker speech features and the expression content text features corresponding to the same training sample are input into the decoder in the variational autoencoder, and the encoder and the decoder are trained with the goal of maximizing the similarity between the reconstructed speech data output by the decoder and the second single-channel speech segment data.

[0014] In some embodiments of the present application, the step of obtaining the expression content text features corresponding to the first single-channel voice segment data respectively emitted by each speaker in the multi-person alternating speaking scenario includes:

[0015] The first single-channel speech segment data emitted by each speaker in the multi-person alternating speaking scene are respectively input into the speech recognition system of the factor, so that the speech recognition system respectively extracts the expression content text features corresponding to each of the first single-channel speech segment data.

[0016] In some embodiments of the present application, before obtaining the expression content text features corresponding to the first single-channel voice segment data respectively emitted by each speaker in the multi-person alternating speaking scene, the method further includes:

[0017] Remove noise and silence from audio data in scenes where multiple people are speaking alternately;

[0018] Based on a preset speaker segmentation algorithm, speaker change point detection is performed on the audio data of the multiple-person alternating speaking scene after the noise and silence removal process to obtain each speaker switching point in the audio data of the multiple-person alternating speaking scene;

[0019] Based on each of the speaker switching points, the audio data of the multiple-person alternating speaking scene after the noise and silence removal process is performed to perform speech segmentation to obtain a plurality of single-channel speech segment data;

[0020] According to the unique identifier of the speaker corresponding to each of the pre-acquired single-channel voice segment data, the first single-channel voice segment data and the second single-channel voice segment data corresponding to each of the speakers are respectively obtained in each of the single-channel voice segment data, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data are different.

[0021] A second aspect of the present application provides a single-channel multi-speaker factor extraction method, comprising:

[0022] Acquire multiple target single-channel speech segment data corresponding to the target multi-person alternating speaking scene;

[0023] Each of the target single-channel speech segment data is respectively input into a single-channel multi-speaker factor extraction model, so that the single-channel multi-speaker factor extraction model respectively outputs the speaker speech features corresponding to each of the target single-channel speech segment data, wherein the single-channel multi-speaker factor extraction model is pre-trained based on the single-channel multi-speaker factor extraction model training method provided in the first aspect of the present application.

[0024] In some embodiments of the present application, the step of obtaining a plurality of target single-channel voice segment data corresponding to a target multi-person alternating speaking scene includes:

[0025] Perform noise and silence removal on the audio data of the target multi-person alternating speaking scene;

[0026] Based on a preset speaker segmentation algorithm, speaker change point detection is performed on the target multi-person alternating speaking scene audio data after the noise and silence removal processing to obtain each speaker switching point in the target multi-person alternating speaking scene audio data;

[0027] Based on each of the speaker switching points in the target multi-person alternating speaking scene audio data, speech segmentation is performed on the target multi-person alternating speaking scene audio data after the noise and silence removal processing to obtain a plurality of target single-channel speech segment data.

[0028] The third aspect of the present application provides a single-channel multi-speaker separation method, comprising:

[0029] Using a preset target clustering method and a target threshold value corresponding to the target clustering method, clustering speaker voice features corresponding to each target single-channel voice segment data corresponding to a target multi-person alternating speaking scene, so as to obtain a target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene, wherein the target speaker cluster contains all the target single-channel voice segment data belonging to the same speaker;

[0030] Among them, the speaker speech features corresponding to each of the target single-channel speech segment data are obtained in advance based on the single-channel multi-speaker factor extraction method provided in the second aspect of the present application.

[0031] In some embodiments of the present application, the target clustering method includes: a first clustering method and a second clustering method, and the target threshold includes: a first threshold corresponding to the first clustering method and a second threshold corresponding to the second clustering method;

[0032] Correspondingly, the preset target clustering method and the target threshold value corresponding to the target clustering method are used to cluster the speaker voice features corresponding to each target single-channel voice segment data corresponding to the target multi-person alternating speaking scene, so as to obtain the target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene, including:

[0033] Using a first clustering method and a first threshold value corresponding to the first clustering method, clustering the speaker voice features corresponding to each target single-channel voice segment data corresponding to the target multi-person alternating speaking scene to obtain a first speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene; the first speaker cluster contains at least one target single-channel voice segment data;

[0034] And, clustering the speaker voice features corresponding to each target single-channel voice segment data corresponding to the target multi-person alternating speaking scene using a second clustering method and a second threshold value corresponding to the second clustering method, so as to obtain a second speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene; the second speaker cluster contains at least one target single-channel voice segment data;

[0035] Based on a preset consensus clustering scoring strategy, the target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene is determined according to the first speaker cluster and the second speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene.

[0036] In some embodiments of the present application, the first clustering method includes spectral clustering, and / or the second clustering method includes Kmeans clustering.

[0037] A fourth aspect of the present application provides a single-channel multi-speaker factor extraction model training device, comprising:

[0038] A text factor acquisition module, used to acquire the text features of the expression content corresponding to the first single-channel voice segment data emitted by each speaker in a multi-person alternating speaking scenario;

[0039] A training sample construction module, used to construct the expression content text features of each speaker and the second single-channel voice segment data into a training sample, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data are different;

[0040] A model training module is used to use each of the training samples to perform preset model training steps based on text distillation of expression content for the encoder and decoder in the variational autoencoder, so as to train the encoder as a single-channel multi-speaker factor extraction model for extracting speaker speech features corresponding to single-channel speech fragment data.

[0041] A fifth aspect of the present application provides a single-channel multi-speaker factor extraction device, comprising:

[0042] A data acquisition module, used to acquire multiple target single-channel speech segment data corresponding to a target multi-person alternating speaking scene;

[0043] The speaker speech feature extraction module is used to input each of the target single-channel speech segment data into a single-channel multi-speaker factor extraction model, so that the single-channel multi-speaker factor extraction model outputs the speaker speech features corresponding to each of the target single-channel speech segment data, wherein the single-channel multi-speaker factor extraction model is pre-trained based on the single-channel multi-speaker factor extraction model training method provided in the first aspect of the present application.

[0044] A sixth aspect of the present application provides a single-channel multi-speaker separation device, comprising:

[0045] A clustering module, configured to cluster the speaker speech features corresponding to each target single-channel speech segment data corresponding to the target multi-person alternating speaking scene using a preset target clustering method and a target threshold value corresponding to the target clustering method, so as to obtain a target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene, wherein the target speaker cluster contains all the target single-channel speech segment data belonging to the same speaker;

[0046] Among them, the speaker speech features corresponding to each of the target single-channel speech segment data are obtained in advance based on the single-channel multi-speaker factor extraction method provided in the second aspect of the present application.

[0047] The seventh aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, at least one of the single-channel multi-speaker factor extraction model training method provided in the first aspect of the present application, the single-channel multi-speaker factor extraction method provided in the second aspect of the present application, and the single-channel multi-speaker separation method provided in the third aspect of the present application is implemented.

[0048] The eighth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements at least one of the single-channel multi-speaker factor extraction model training method provided in the first aspect of the present application, the single-channel multi-speaker factor extraction method provided in the second aspect of the present application, and the single-channel multi-speaker separation method provided in the third aspect of the present application.

[0049] The ninth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements at least one of the single-channel multi-speaker factor extraction model training method provided in the first aspect of the present application, the single-channel multi-speaker factor extraction method provided in the second aspect of the present application, and the single-channel multi-speaker separation method provided in the third aspect of the present application.

[0050] The present application provides a single-channel multi-speaker factor extraction model training method, which obtains the expression content text features corresponding to the first single-channel speech segment data emitted by each speaker in a multi-person alternating speaking scenario; constructs the expression content text features of each speaker and the second single-channel speech segment data into a training sample, wherein the expression content corresponding to the second single-channel speech segment data is different from that of the first single-channel speech segment data; uses each of the training samples to perform preset model training steps based on expression content text distillation for the encoder and decoder in the variational autoencoder, so as to train the encoder as a single-channel multi-speaker factor extraction model for extracting speaker speech features corresponding to the single-channel speech segment data, which can distill the expression content text corresponding to the speaker's single-channel speech during the model training process, and can effectively reduce the redundant information interference of the expression content text on the effectiveness of speaker speech feature extraction, thereby effectively improving the accuracy of speaker speech features extracted by using the model and the subsequent application effectiveness and accuracy for single-channel multi-speaker separation.

[0051] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0052] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0054] Figure 1 Schematic diagram of a first flow chart of a single-channel multi-speaker factor extraction model training method in an embodiment of the present application.

[0055] Figure 2 The flowchart is a schematic diagram of the model training step based on expression content text distillation in the single-channel multi-speaker factor extraction model training method in one embodiment of the present application.

[0056] Figure 3 Schematic diagram of a second flow chart of a single-channel multi-speaker factor extraction model training method in an embodiment of the present application.

[0057] Figure 4 This is a schematic diagram of the application architecture of a speech recognition system and a variational autoencoder in an example of the present application.

[0058] Figure 5 1 is a schematic diagram of a first flow chart of a single-channel multi-speaker factor extraction method in an embodiment of the present application.

[0059] Figure 6 Schematic diagram of a second flow chart of a single-channel multi-speaker factor extraction method in an embodiment of the present application.

[0060] Figure 7 Schematic diagram of the flow of a single-channel multi-speaker separation method in an embodiment of the present application.

[0061] Figure 8Schematic diagram of the structure of a single-channel multi-speaker factor extraction model training device in one embodiment of the present application.

[0062] Fig. 9 Schematic diagram of the structure of a single-channel multi-speaker factor extraction device in one embodiment of the present application.

[0063] Fig.10 Schematic diagram of the structure of a single-channel multi-speaker separation device in one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0065] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0066] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0067] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0068] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0069] The existing blind source speaker separation technology is mainly used in the front end of the audio transcription system to perform blind source classification of roles so that the subsequent transcription content can be automatically organized into a natural dialogue form that is easy for humans to understand. The accuracy of the separation affects the readability of the text content. This technology is mainly implemented in two stages: speech segmentation and clustering. The detailed implementation steps are as follows:

[0070] S1: Input audio data containing M speakers and obtain valid voice data through voice detection algorithm;

[0071] S2: For the human voice data obtained in (1), speaker change point detection is performed through an audio segmentation algorithm, such as the BIC criterion, and the audio is segmented into N audio segments, with the goal of making each audio segment contain only one speaker's voice data.

[0072] S3: Perform speaker representation on the N short audio clips segmented in (2), then perform clustering and merge two by two until convergence and N is greater than M.

[0073] Through the above process analysis, speaker separation of multiple speakers in a conference scene is achieved. There are two key points in the whole process:

[0074] Do the audio segments in S2 satisfy the single speaker assumption? Currently, both the BIC change point detection technology and the strong segmentation scheme based on the statistics of the duration of a single speaker have met the separation requirements and there is no bottleneck.

[0075] The robustness of speaker representation technology and clustering methods in S3. Due to the interference of text, channel, and speaker state information, the robustness of speaker information extraction and representation in acoustic signals is greatly affected, especially text information interference is more serious, because in the conference scene, the channel and speaker state are relatively stable, while the text content expressed by the speaker is constantly changing. At the same time, due to multi-person conferences, there are many speaker categories and the speaker audio data is unbalanced, which further aggravates the stability of speaker information representation.

[0076] In order to solve the problems existing in the existing single-channel multi-speaker factor extraction method, such as the interference effect of the expression content text corresponding to the single-channel speech in the multi-person alternating speaking scene on the speaker speech feature extraction of the single-channel speech, the embodiments of the present application respectively provide a single-channel multi-speaker factor extraction model training method, a single-channel multi-speaker factor extraction model training device for executing the single-channel multi-speaker factor extraction model training method, a single-channel multi-speaker factor extraction method, a single-channel multi-speaker factor extraction device for executing the single-channel multi-speaker factor extraction method, a single-channel multi-speaker separation method, a single-channel multi-speaker separation transposition for executing the single-channel multi-speaker separation method, related electronic equipment, a computer-readable storage medium and a computer program product, which can effectively reduce the redundant information interference of the expression content text on the effectiveness of the speaker speech feature extraction.

[0077] The details are described in detail through the following examples.

[0078] Based on this, the embodiment of the present application provides a single-channel multi-speaker factor extraction model training method that can be implemented by a single-channel multi-speaker factor extraction model training device, see Figure 1 The single-channel multi-speaker factor extraction model training method specifically includes the following contents:

[0079] Step 100: Obtain the text features of the expression content corresponding to the first single-channel voice segment data respectively emitted by each speaker in a multi-person alternating speaking scenario.

[0080] In one or more embodiments of the present application, the expression content text feature may also be referred to as an expression content text feature vector, an expression content text encoding feature or an expression content text factor, or may be simply referred to as a text factor. Specifically, C i Represents the i-th single-channel speech segment data U i (i.e., the first single-channel voice segment data) corresponding to the expression content text features. It should be noted that the expression content text features refer to the text feature vectors corresponding to the expression content extracted from the first single-channel voice segment data, rather than the feature vectors extracted from the text data.

[0081] It can be understood that a multi-person alternating speaking scene refers to a scene where only one speaker speaks at the same time and multiple people speak alternately, which is a blind source speaker separation task. That is to say, each single-channel voice segment data in the multi-person alternating speaking scene mentioned in the embodiment of the present application has only one speaker's voice, and there is no overlap of the voices of multiple speakers.

[0082] In other words, the specific manifestations of multiple people speaking alternately are:

[0083] 1) Audio data in a multi-person alternating speaking scene contains multiple speakers, and these speakers are unknown speakers, which belongs to the blind source separation task;

[0084] 2) In scenarios where multiple people are speaking alternately, the channels, speaker states, and environmental noise interference involved are relatively stable. Only the content of the speaker's speech is changing, and the duration of the speaker's audio data is unevenly distributed.

[0085] 3) Single-channel speaker separation, not multi-channel.

[0086] In one or more embodiments of the present application, the speaker voice feature refers to a unique voice feature that can characterize the speaker, such as a voiceprint feature, so as to identify the unique role information or unique identity information of the speaker in a multi-person alternating speaking scene based on the voice feature. The speaker voice feature can be written as a speaker voice coding feature, a speaker voice feature vector or a speaker factor. Specifically, E(x) can be used to represent the x-th single-channel voice segment data U x (ie, the second single-channel speech segment data) corresponds to the speaker voice feature, and x≠i. That is, in this application, the speaker factor refers to the speaker voice feature.

[0087] Step 200: constructing the expression content text features of each speaker and the second single-channel voice segment data into a training sample, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data are different.

[0088] Among them, the first single-channel voice segment data and the second single-channel voice segment data of the same speaker are selected from the single-channel voice segment data emitted by the same speaker in the same multi-person alternating speaking scene, and their corresponding expression contents are different. That is to say, if the expression content text data are extracted from the first single-channel voice segment data and the second single-channel voice segment data respectively, different expression content text data will be extracted.

[0089] That is to say, in the training process of the single-channel multi-speaker factor extraction model, each selected training sample contains the expression content text features and the second single-channel voice segment data corresponding to the first single-channel voice segment data of a speaker. And the training set used to train the single-channel multi-speaker factor extraction model is composed of training samples corresponding to different multiple speakers.

[0090] Step 300: Using each of the training samples, respectively execute a preset model training step based on text distillation of the expression content for the encoder and decoder in the variational autoencoder, so as to train the encoder into a single-channel multi-speaker factor extraction model for extracting speaker speech features corresponding to single-channel speech segment data.

[0091] It can be understood that the variational autoencoder is a self-supervised variational autoencoder architecture, and the variational autoencoder (VAE) is a typical generative model. In step 300, the variational autoencoder is composed of an encoder and a decoder connected to each other. The encoder is finally trained as a single-channel multi-speaker factor extraction model for extracting speaker voice features corresponding to single-channel voice segment data. The single-channel multi-speaker factor extraction model can also be called a speaker factor extractor or a speaker voice feature extraction model, etc.

[0092] In this application, the single channel is used to distinguish it from the speaker separation of the microphone array model, and specifically refers to the speaker separation of the speech that has been recorded and archived by a single microphone. When performing speaker separation in this scenario, the text content is a strong interference information to the speaker information. This application is based on the factor decoupling theory, and regards speech and text as special factors in the audio file. Through innovative structure, the encoder is driven to extract speaker factor information more, and the speech recognizer is used to extract text information. The two information are combined to allow the decoder to restore the audio file as much as possible. When the model converges, the encoder can only extract speaker information. As a subsequent feature extractor, the interference of text factors is effectively eliminated.

[0093] From the above description, it can be seen that the single-channel multi-speaker factor extraction model training method provided in the embodiment of the present application can distill the expression content text corresponding to the speaker's single-channel speech during the model training process, and can effectively reduce the redundant information interference of the expression content text on the effectiveness of speaker factor extraction, thereby effectively improving the accuracy of the speaker factors extracted using the model and the subsequent application effectiveness and accuracy for single-channel multi-speaker separation.

[0094] In order to further realize the distillation of the expression content text and improve the effectiveness and reliability of the distillation of the expression content text, in a single-channel multi-speaker factor extraction model training method provided in an embodiment of the present application, see Figure 2 The model training step based on expression content text distillation in step 300 of the single-channel multi-speaker factor extraction model training method specifically includes the following contents:

[0095] Step 310: input the second single-channel speech segment data in the current training sample into the encoder in the variational autoencoder, so that the encoder outputs a speaker factor corresponding to the second single-channel speech segment data.

[0096] Step 320: Input the speaker factor and the expression content text features corresponding to the same training sample into the decoder in the variational autoencoder, and train the encoder and the decoder with the goal of maximizing the similarity between the reconstructed speech data output by the decoder and the second single-channel speech segment data.

[0097] In the model training stage, the model training architecture is based on the reconstruction idea. It is hoped that the speaker factor information extracted by the encoder will be integrated with the text factor information extracted by the recognizer, and then the original audio file will be reconstructed through the decoder. Therefore, it is sufficient to calculate the similarity between the original audio and the restored audio. Assume that the original audio feature extraction is X = (x1, x2, ..., x n ); The restored audio Y = (y1, y2, ..., y n ); then we only need to maximize the co-distance between the two vectors:

[0098]

[0099] Specifically, the decoder inputs the i-th single-channel speech segment data U i (i.e. the first single-channel voice segment data) i And the x-th single-channel voice segment data U x (i.e. the second single-channel speech segment data) corresponds to the speaker's speech feature E(x), and outputs the reconstructed and restored reconstructed speech data U i R .

[0100] K speakers can be used, and each speaker corresponds to T single-channel voice segment data, K is greater than 2, and T is greater than K, and steps 310 and 320 are repeated until the variational autoencoder training converges to obtain a stable encoder and decoder. Then the encoder is a speaker factor extractor that can distill out text information factors and can only extract speaker factors. Of course, the extracted factors also include channel, environment and other factor information, but since these are relatively fixed in the conference scene, their impact can be ignored.

[0101] That is to say, by executing step 310 and step 320 respectively for each of the different training samples until the variational autoencoder converges, it is determined that the current variational autoencoder training is completed. At this time, the encoder in the variational autoencoder can be used as a single-channel multi-speaker factor extraction model to online extract the speaker speech features corresponding to the single-channel speech segment data.

[0102] In order to further improve the accuracy, effectiveness and intelligence of the text feature acquisition of the expression content, in a single-channel multi-speaker factor extraction model training method provided in an embodiment of the present application, see Figure 3 The step 100 of the single-channel multi-speaker factor extraction model training method specifically includes the following contents:

[0103] Step 110: inputting the first single-channel speech segment data emitted by each speaker in the multi-person alternating speaking scene into a speech recognition system of the factor, so that the speech recognition system extracts the expression content text features corresponding to each of the first single-channel speech segment data for representing the text features.

[0104] In step 110, the speech recognition system is an existing artificial intelligence model for real-time speech-to-text conversion and extracting text feature vectors for extracting text features of the expression content in audio data.

[0105] In one example, see Figure 4 First, the i-th single-channel speech segment data U of speaker A is i (i.e., the first single-channel speech segment data) is input into the speech recognition system so that the speech recognition system outputs the i-th single-channel speech segment data U i The corresponding expression content text feature C i , and then the expression content text feature C i Send it to the input end of the decoder in the variational self-encoder; at the same time, the x-th single-channel speech segment data U of the speaker A is sent ... x(i.e., the second single-channel speech segment data) is input into the encoder in the variational self-encoder, so that the encoder outputs the x-th single-channel speech segment data U x (i.e., the second single-channel speech segment data) corresponds to the speaker's speech feature E(x); then the i-th single-channel speech segment data U i (i.e. the first single-channel voice segment data) i And the x-th single-channel speech segment data U x The speaker speech feature E(x) corresponding to the second single-channel speech segment data is input into the decoder, so that the decoder outputs the reconstructed speech data U correspondingly reconstructed and restored i R .

[0106] That is to say, the embodiment of the present application fully considers the characteristics of the multi-speaker separation scenario in the single-channel multi-person conference scenario, and obtains a relatively pure speaker representation factor based on the speaker representation technology of text distillation to remove content factors and the clustering strategy of consensus clustering.

[0107] In order to further improve the accuracy, effectiveness and intelligence of the text feature acquisition of the expression content, in a single-channel multi-speaker factor extraction model training method provided in an embodiment of the present application, see Figure 3 The single-channel multi-speaker factor extraction model training method further specifically includes the following contents before step 100:

[0108] Step 010: Perform noise and silence removal processing on the audio data of the scene where multiple people speak alternately.

[0109] Step 020: Based on a preset speaker segmentation algorithm, speaker change point detection is performed on the audio data of the multiple-person alternating speaking scene after the noise and silence removal processing to obtain each speaker switching point in the audio data of the multiple-person alternating speaking scene.

[0110] Step 030: Based on each of the speaker switching points, the audio data of the multi-person alternating speaking scene after the noise and silence removal processing is subjected to speech segmentation to obtain a plurality of single-channel speech segment data.

[0111] Step 040: According to the unique identifier of the speaker corresponding to each of the pre-acquired single-channel voice segment data, respectively obtain the first single-channel voice segment data and the second single-channel voice segment data corresponding to each of the speakers in each of the single-channel voice segment data, wherein the second single-channel voice segment data and the first single-channel voice segment data have different expressions.

[0112] The speaker segmentation algorithm can adopt a Bayesian Information Criterion (BIC) speaker segmentation algorithm, which is a technology for segmenting a speech stream into processes with the same speaker, and implements speaker segmentation based on Bayesian Information Criterion, so as to achieve correct segmentation of speech segments and ensure that each minimum speech segment can only contain speech data of one speaker.

[0113] Furthermore, the speaker segmentation algorithm in step 020 may also adopt a strong segmentation scheme based on data statistics of the duration of a single speaker.

[0114] Based on the single-channel multi-speaker factor extraction model training method provided in the above embodiment, the present application embodiment also provides an embodiment of a single-channel multi-speaker factor extraction method based on a single-channel multi-speaker factor extraction device, see Figure 5 The single-channel multi-speaker factor extraction method specifically includes the following contents:

[0115] Step 400: Acquire multiple target single-channel voice segment data corresponding to a target multi-person alternating speaking scene;

[0116] Step 500: Input each of the target single-channel speech segment data into a single-channel multi-speaker factor extraction model, so that the single-channel multi-speaker factor extraction model outputs speaker factors corresponding to each of the target single-channel speech segment data, wherein the single-channel multi-speaker factor extraction model is pre-trained based on the single-channel multi-speaker factor extraction model training method.

[0117] The single-channel multi-speaker factor extraction model training method in step 500 of the embodiment of the single-channel multi-speaker factor extraction method provided in the present application can be specifically used for the processing flow of the embodiment of the single-channel multi-speaker factor extraction model training method provided by the above embodiment. Its functions will not be described in detail here, and reference can be made to the detailed description of the above embodiment of the single-channel multi-speaker factor extraction model training method.

[0118] From the above description, it can be seen that the single-channel multi-speaker factor extraction method provided in the embodiment of the present application can effectively reduce the redundant information interference of the expression content text on the effectiveness of speaker factor extraction, and thus can effectively extract the accuracy of speaker factors and the subsequent application effectiveness and accuracy for single-channel multi-speaker separation.

[0119] In order to further provide a more effective and reliable data basis for single-channel multi-speaker factor extraction, the present application embodiment also provides an embodiment of a single-channel multi-speaker factor extraction method based on a single-channel multi-speaker factor extraction device, see Figure 6The single-channel multi-speaker factor extraction method specifically includes the following contents:

[0120] Step 410: Perform noise and silence removal processing on the target multi-person alternating speaking scene audio data.

[0121] Step 420: Based on a preset speaker segmentation algorithm, speaker change point detection is performed on the target multi-person alternating speaking scene audio data after the noise and silence removal processing to obtain each speaker switching point in the target multi-person alternating speaking scene audio data.

[0122] Step 430: Based on each of the speaker switching points in the target multi-person alternating speaking scene audio data, voice segmentation is performed on the target multi-person alternating speaking scene audio data after the noise and silence removal processing to obtain a plurality of target single-channel voice segment data.

[0123] Specifically, voice detection is performed on the target audio data of the multi-person alternating speaking scene to obtain valid audio data (noise and silence segments are removed, and then speaker change point detection based on BIC is performed on the valid audio data. According to the detected speaker switching points, audio segmentation is performed to obtain N audio segments containing only a single speaker, where N is greater than 2.

[0124] It has important applications in intelligent speech processing in telephone and conference scenarios. After the audio is converted into text content, the role information provided by speaker separation can form a dialogue format that conforms to the understanding of natural people. In the telephone scenario of two-person dialogue, it has been very maturely applied. However, due to the large number of speakers involved in the speaker separation task in the conference scenario, and the data imbalance caused by uneven speeches of the meeting participants, the current single-channel blind source speaker separation in the conference scenario is not effective and cannot reach the practical level. The main reasons include:

[0125] In the multi-speaker separation scenario of a conference, the interference effect of redundant information such as text and channel is significantly amplified. Speech is a one-dimensional acoustic signal, and the acoustic features are convoluted with a large amount of redundant features of non-speaker identity information such as text content, channel, environmental noise, and speaker status. These non-speaker identity information are redundant information for the speaker clustering process, especially in the multi-speaker separation clustering scenario of a conference scene. The mutual interference of multiple speakers seriously affects the accuracy of speaker clustering.

[0126] That is to say, in the existing technical solutions, in the multi-speaker audio representation, due to the interference of the text channel, the threshold of the speaker clustering and merging process varies greatly, and it is difficult to obtain a relatively stable absolute threshold. The relative threshold method is adopted. On the one hand, the clustering converges slowly. On the other hand, if the number of speakers involved is not known in advance, the relative threshold leads to failure to converge and automatic blind source separation. Based on the single-channel multi-speaker factor extraction model training method and / or single-channel multi-speaker factor extraction method provided in the above embodiments of the present application, since the audio channel, speaker status, and noise environment are relatively stable, the main interference factor is the interference of the text content factor on the speaker information. When extracting and representing the speaker information, a speaker representation technology based on text distillation to remove the content factor is designed to improve the robustness of the speaker information representation, so as to obtain a relatively stable speech clustering threshold and solve the problems of slow convergence and failure to converge when the number of speakers is unknown.

[0127] In other words, this application can also implement a single-channel multi-speaker blind source speaker separation method in a conference scenario based on the speaker characterization technology of text distillation to remove content factors and the clustering strategy of consensus clustering. By removing the strong interference of content factors on speaker information through text distillation, a stable speaker factor representation is obtained, thereby avoiding the traditional solution that can only use a relative threshold speaker merging solution due to text interference. At the same time, the consensus clustering strategy effectively improves the clustering robustness under unbalanced data conditions.

[0128] Specifically, in order to solve the instability of the threshold in the clustering method in the existing multi-speaker recognition solution due to the change in the duration of the speaker's speech after merging, and thus the problem of transmission errors caused by the traditional hierarchical clustering and the problem that the clustering cannot converge under the condition of an unknown number of speakers, based on the single-channel multi-speaker factor extraction model training method and / or the single-channel multi-speaker factor extraction method provided in the above embodiments, the embodiment of the present application also provides an embodiment of implementing a single-channel multi-speaker separation method based on a single-channel multi-speaker separation device, and the single-channel multi-speaker separation method specifically includes the following contents:

[0129] Step 600: clustering the speaker factors corresponding to each target single-channel speech segment data corresponding to the target multi-person alternating speaking scene using a preset target clustering method and a target threshold value corresponding to the target clustering method, so as to obtain target speaker clusters corresponding to each speaker in the target multi-person alternating speaking scene, wherein the target speaker clusters contain all the target single-channel speech segment data belonging to the same speaker; wherein the speaker factors corresponding to each target single-channel speech segment data are obtained in advance by the single-channel multi-speaker factor extraction method.

[0130] The single-channel multi-speaker factor extraction method in step 600 of the embodiment of the single-channel multi-speaker separation method provided in the present application can be specifically used in the processing flow of the embodiment of the single-channel multi-speaker factor extraction method provided by the above embodiment. Its function will not be repeated here, and reference can be made to the detailed description of the above embodiment of the single-channel multi-speaker factor extraction method.

[0131] From the above description, it can be seen that the single-channel multi-speaker separation method provided in the embodiment of the present application can effectively weaken the influence of text factors and obtain a stable decision threshold by distilling the speaker factor extractor that removes the text factor; because of the stable decision threshold, the single-channel multi-speaker separation process only needs to compare two by two, and make decisions and merge according to a fixed threshold, which can effectively avoid the transmission errors caused by traditional hierarchical clustering, and at the same time solve the problem that clustering cannot converge under the condition of unknown number of speakers.

[0132] In addition, speaker clustering based on traditional algorithms is difficult to achieve global optimization. Current speech clustering technology generally adopts hierarchical clustering, which uses a greedy algorithm and only obtains local optimization. Especially under the condition of imbalanced speaker data, the weak information of minority classes will be ignored, resulting in separation failure.

[0133] In order to further improve the robustness of the clustering effect, in an embodiment of the single-channel multi-speaker separation method provided in the present application, the target clustering method includes: a first clustering method and a second clustering method, and the target threshold includes: a first threshold corresponding to the first clustering method and a second threshold corresponding to the second clustering method;

[0134] For the corresponding Figure 7 , step 600 of the single-channel multi-speaker separation method specifically includes the following contents:

[0135] Step 610: clustering speaker factors corresponding to each target single-channel speech segment data corresponding to the target multi-person alternating speaking scene using a first clustering method and a first threshold value corresponding to the first clustering method to obtain first speaker clusters corresponding to each speaker in the target multi-person alternating speaking scene; the first speaker clusters contain at least one target single-channel speech segment data;

[0136] And, step 620: clustering the speaker factors corresponding to each target single-channel speech segment data corresponding to the target multi-person alternating speaking scene using a second clustering method and a second threshold value corresponding to the second clustering method, so as to obtain a second speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene; the second speaker cluster contains at least one target single-channel speech segment data;

[0137] Step 630: Based on a preset consensus clustering scoring strategy, determine the target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene according to the first speaker cluster and the second speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene.

[0138] In step 610 to step 630, the speaker data is merged through a stable threshold, which solves the problem of slow clustering convergence, especially the problem of being unable to cluster when the number of speakers is unknown; at the same time, the consensus clustering strategy effectively improves the clustering robustness problem under the condition of uneven distribution of speaker audio data duration. The implementation process will be described in detail below.

[0139] In order to further effectively improve the robustness of the clustering effect based on the consensus clustering strategy, the first clustering method includes spectral clustering, and / or the second clustering method includes Kmeans clustering.

[0140] Specifically, spectral clustering is performed on each target single-channel speech segment data corresponding to the target multi-person alternating speaking scene. Since the single-channel speech segment data has been distilled out of the influence of text factors, an absolute first threshold th1 can be used. If it exceeds the first threshold th1, the data is merged and classified into the same speaker cluster to obtain M1 first speaker clusters, where M1 is greater than 2. Then, Kmeans clustering is performed on each target single-channel speech segment data. Since the text factors have been distilled out of each target single-channel speech segment data, an absolute second threshold th2 can be used. If it exceeds h2, the data is merged and classified into the same speaker cluster to obtain M2 second speaker clusters, where M2 is greater than 2. Then, consensus scoring is performed based on the clustering results of spectral clustering and Kmeans clustering, and the same results of the two clustering algorithms are selected as the final clustering results, to obtain M target speaker clusters, where M is greater than 2.

[0141] At this point, through the above steps, the influence of text factors is distilled to obtain a stable speaker judgment threshold. The threshold is no longer affected by the change in the length of the merge, and the merge classification can be completed in one go to form a speaker cluster, which can improve the clustering speed and solve the problem of unknown speakers being unable to cluster. At the same time, the voting consensus result of consensus clustering can effectively improve the robustness of clustering. This solution innovatively solves the problem of multiple speaker separation in conference scenarios.

[0142] That is to say, the single-channel multi-speaker separation method provided in the above embodiment of the present application is designed to solve the problem of uneven distribution of speaker audio data duration, and is based on identification clustering. It improves the robustness of clustering from the consensus of clustering results of multi-angle information and solves the problem of clustering failure caused by data imbalance.

[0143] The single-channel multi-speaker separation method provided in the above embodiment of the present application can focus on the blind source speaker separation goal of the audio of the conference scene, and for the interference of redundant information such as text and the data imbalance of multiple speakers, the interference suppression technology of speaker factor encoding of text distillation is used to improve the speaker information representation ability, and based on the consensus clustering idea, the clustering robustness under unbalanced conditions is improved, thereby effectively improving the multi-speaker separation accuracy of the conference scene. And then it can solve the problem of "who speaks in what time period". The multi-speaker separation of audio data mainly includes two stages, audio segmentation and clustering.

[0144] In an example of the present application, the target multi-person alternating speaking scene and the multi-person alternating speaking scene can both adopt conference scenes, which specifically involve intelligent speech processing technology in the field of perceptual artificial intelligence. According to the input audio data, the speaker data involved in the audio is automatically segmented and clustered in the form of a blind source. The goal of speaker separation is to divide the speech data into several subsets according to the speaker similarity between speech samples through an unsupervised learning method, so that the pronunciation segments in each subset come from the same speaker. It clusters the speech segments belonging to the same speaker in a speech data into one category and generates sequence labels to mark them. It can be mainly summarized as the problem of "who, when, speaks". Different from the supervised classification algorithm, speaker clustering does not require any prior information, which is called blind source speaker separation. This application takes the blind source separation of multiple speakers in the conference scene as an example, and aims at the strong interference problem of text content on speaker information. The interference factors of text information are removed based on text distillation technology to obtain a stable speaker category judgment threshold. At the same time, in order to solve the problem of poor engine clustering robustness due to uneven duration distribution of speaker data, a consensus clustering strategy is introduced to perform clustering from multiple dimensions such as density and distance. Through a voting mechanism, the clustering result with the most votes is taken as the clustering result. This application obtains stable clustering results based on the consensus judgment of two clustering algorithms, which can effectively solve the problem of multi-speaker separation in conference scenes, and solve the problem that the effect of multi-speaker separation in conference scenes does not meet the practical threshold from the perspective of representation accuracy and clustering robustness.

[0145] The present application also provides a single-channel multi-speaker factor extraction model training device for executing all or part of the single-channel multi-speaker factor extraction model training method, see Figure 8 The single-channel multi-speaker factor extraction model training device specifically includes the following contents:

[0146] The text factor acquisition module 10 is used to acquire the expression content text features for representing text features corresponding to the first single-channel voice segment data respectively emitted by each speaker in a multi-person alternating speaking scenario.

[0147] The training sample construction module 20 is used to construct the expression content text features of each speaker and the second single-channel voice segment data into a training sample, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data are different.

[0148] The model training module 30 is used to use each of the training samples to perform preset model training steps based on text distillation of expression content for the encoder and decoder in the variational autoencoder, so as to train the encoder into a single-channel multi-speaker factor extraction model for extracting speaker factors corresponding to single-channel speech fragment data.

[0149] The embodiment of the single-channel multi-speaker factor extraction model training device provided in the present application can be specifically used to execute the processing flow of the embodiment of the single-channel multi-speaker factor extraction model training method in the above embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above embodiment of the single-channel multi-speaker factor extraction model training method.

[0150] The part of the single-channel multi-speaker factor extraction model training device that performs the single-channel multi-speaker factor extraction model training can be executed in the server or completed in the client device. The specific selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the single-channel multi-speaker factor extraction model training.

[0151] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0152] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0153] From the above description, it can be seen that the single-channel multi-speaker factor extraction model training device provided in the embodiment of the present application can distill the expression content text corresponding to the speaker's single-channel speech during the model training process, and can effectively reduce the redundant information interference of the expression content text on the effectiveness of speaker factor extraction, thereby effectively improving the accuracy of the speaker factors extracted using the model and the subsequent application effectiveness and accuracy for single-channel multi-speaker separation.

[0154] The present application also provides a single-channel multi-speaker factor extraction device for executing all or part of the single-channel multi-speaker factor extraction method, see Fig. 9 The single-channel multi-speaker factor extraction device specifically includes the following contents:

[0155] The data acquisition module 40 is used to acquire a plurality of target single-channel speech segment data corresponding to a target multi-person alternating speaking scene.

[0156] The speaker factor extraction module 50 is used to input each of the target single-channel speech segment data into a single-channel multi-speaker factor extraction model, so that the single-channel multi-speaker factor extraction model outputs the speaker factors corresponding to each of the target single-channel speech segment data, wherein the single-channel multi-speaker factor extraction model is pre-trained based on the single-channel multi-speaker factor extraction model training method.

[0157] The embodiment of the single-channel multi-speaker factor extraction device provided in the present application can be specifically used to execute the processing flow of the embodiment of the single-channel multi-speaker factor extraction method in the above embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above embodiment of the single-channel multi-speaker factor extraction method.

[0158] The part of the single-channel multi-speaker factor extraction device that performs single-channel multi-speaker factor extraction can be executed in the server or in the client device. The specific selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of the single-channel multi-speaker factor extraction.

[0159] From the above description, it can be seen that the single-channel multi-speaker factor extraction device provided in the embodiment of the present application can effectively reduce the redundant information interference of the expression content text on the effectiveness of speaker factor extraction, and thus can effectively extract the accuracy of speaker factors and the subsequent application effectiveness and accuracy for single-channel multi-speaker separation.

[0160] The present application also provides a single-channel multi-speaker separation device for performing all or part of the single-channel multi-speaker separation method, see Fig.10 The single-channel multi-speaker separation device specifically includes the following contents:

[0161] The clustering module 60 is used to cluster the speaker factors corresponding to each target single-channel speech segment data corresponding to the target multi-person alternating speaking scene by using a preset target clustering method and a target threshold value corresponding to the target clustering method, so as to obtain a target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene, wherein the target speaker cluster contains all the target single-channel speech segment data belonging to the same speaker; wherein the speaker factors corresponding to each target single-channel speech segment data are pre-obtained based on the single-channel multi-speaker factor extraction method.

[0162] The embodiment of the single-channel multi-speaker separation device provided in the present application can be specifically used to execute the processing flow of the embodiment of the single-channel multi-speaker separation method in the above embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the above embodiment of the single-channel multi-speaker separation method.

[0163] The part of the single-channel multi-speaker separation device that performs single-channel multi-speaker separation can be executed in the server or in the client device. The selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of single-channel multi-speaker separation.

[0164] From the above description, it can be seen that the single-channel multi-speaker separation device provided in the embodiment of the present application can effectively weaken the influence of text factors and obtain a stable decision threshold by distilling the speaker factor extractor of the text factor; because of the stable decision threshold, the single-channel multi-speaker separation process only needs to compare two by two, and make decisions and merge according to a fixed threshold, which can effectively avoid the transmission errors caused by traditional hierarchical clustering, and at the same time solve the problem that clustering cannot converge under the condition of unknown number of speakers.

[0165] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute at least one of the single-channel multi-speaker factor extraction model training method, the single-channel multi-speaker factor extraction method and the single-channel multi-speaker separation method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, taking the bus connection as an example. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0166] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0167] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to at least one of the single-channel multi-speaker factor extraction model training method, the single-channel multi-speaker factor extraction method and the single-channel multi-speaker separation method in the embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing at least one of the single-channel multi-speaker factor extraction model training method, the single-channel multi-speaker factor extraction method and the single-channel multi-speaker separation method in the above method embodiments.

[0168] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The one or more modules are stored in the memory, and when executed by the processor, perform at least one of the single-channel multi-speaker factor extraction model training method, the single-channel multi-speaker factor extraction method and the single-channel multi-speaker separation method in the embodiment.

[0170] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0171] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0172] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0173] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, at least one of the steps of the single-channel multi-speaker factor extraction model training method, the single-channel multi-speaker factor extraction method, and the single-channel multi-speaker separation method is implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0174] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements at least one of the steps of the aforementioned single-channel multi-speaker factor extraction model training method, single-channel multi-speaker factor extraction method, and single-channel multi-speaker separation method.

[0175] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0176] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0177] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0178] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A single-channel multi-speaker factor extraction model training method, characterized in that: include: Obtaining the text features of the expression content corresponding to the first single-channel voice segment data emitted by each speaker in the multi-person alternating speaking scenario; constructing the expression content text features of each speaker and the second single-channel voice segment data into a training sample, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data is different; Using each of the training samples, preset model training steps based on text distillation of the expression content are performed for the encoder and decoder in the variational autoencoder respectively, so as to train the encoder into a single-channel multi-speaker factor extraction model for extracting speaker speech features corresponding to single-channel speech segment data.

2. The single-channel multi-speaker factor extraction model training method according to claim 1, characterized in that: The model training steps based on the text distillation of the expression content include: Inputting the second single-channel speech segment data in the current training sample into the encoder in the variational autoencoder, so that the encoder outputs the speaker speech features corresponding to the second single-channel speech segment data; The speaker speech features and the expression content text features corresponding to the same training sample are input into the decoder in the variational autoencoder, and the encoder and the decoder are trained with the goal of maximizing the similarity between the reconstructed speech data output by the decoder and the second single-channel speech segment data.

3. The single-channel multi-speaker factor extraction model training method according to claim 1, characterized in that: The obtaining of the expression content text features corresponding to the first single-channel voice segment data respectively emitted by each speaker in the multiple-person alternating speaking scenario includes: The first single-channel speech segment data emitted by each speaker in the multi-person alternating speaking scene are respectively input into the speech recognition system of the factor, so that the speech recognition system respectively extracts the expression content text features corresponding to each of the first single-channel speech segment data.

4. The single-channel multi-speaker factor extraction model training method according to claim 1, characterized in that: Before obtaining the expression content text features corresponding to the first single-channel voice segment data respectively emitted by each speaker in the multiple-person alternating speaking scenario, the method further includes: Remove noise and silence from audio data in scenes where multiple people are speaking alternately; Based on a preset speaker segmentation algorithm, speaker change point detection is performed on the audio data of the multiple-person alternating speaking scene after the noise and silence removal processing to obtain each speaker switching point in the audio data of the multiple-person alternating speaking scene; performing speech segmentation on the audio data of the multi-person alternating speaking scene after the noise and silence removal processing based on each speaker switching point to obtain a plurality of single-channel speech segment data; According to the unique identifier of the speaker corresponding to each of the pre-acquired single-channel voice segment data, the first single-channel voice segment data and the second single-channel voice segment data corresponding to each of the speakers are respectively obtained in each of the single-channel voice segment data, wherein the expression content corresponding to the second single-channel voice segment data and the first single-channel voice segment data are different.

5. A single-channel multi-speaker factor extraction method, characterized in that: include: Obtain multiple target single-channel speech segment data corresponding to the target multi-person alternating speaking scene; Each of the target single-channel speech segment data is respectively input into a single-channel multi-speaker factor extraction model, so that the single-channel multi-speaker factor extraction model respectively outputs the speaker speech features corresponding to each of the target single-channel speech segment data, wherein the single-channel multi-speaker factor extraction model is pre-trained based on the single-channel multi-speaker factor extraction model training method according to any one of claims 1 to 4.

6. The single-channel multi-speaker factor extraction method according to claim 5, characterized in that: The step of obtaining a plurality of target single-channel voice segment data corresponding to a target multi-person alternating speaking scene includes: Perform noise and silence removal on the audio data of the target multi-person alternating speaking scene; Based on a preset speaker segmentation algorithm, speaker change point detection is performed on the target multi-person alternating speaking scene audio data after the noise and silence removal processing to obtain each speaker switching point in the target multi-person alternating speaking scene audio data; Based on each of the speaker switching points in the target multi-person alternating speaking scene audio data, speech segmentation is performed on the target multi-person alternating speaking scene audio data after the noise and silence removal processing to obtain a plurality of target single-channel speech segment data.

7. A single-channel multi-speaker separation method, characterized in that: include: Using a preset target clustering method and a target threshold value corresponding to the target clustering method, clustering the speaker voice features corresponding to each target single-channel speech segment data corresponding to the target multiple-person alternating speaking scene to obtain a target speaker cluster corresponding to each speaker in the target multiple-person alternating speaking scene, wherein the target speaker cluster includes all the target single-channel speech segment data belonging to the same speaker; The speaker speech features corresponding to each target single-channel speech segment data are obtained in advance based on the single-channel multi-speaker factor extraction method according to claim 5 or 6.

8. The single-channel multi-speaker separation method according to claim 7, characterized in that: The target clustering method includes: a first clustering method and a second clustering method, and the target threshold includes: a first threshold corresponding to the first clustering method and a second threshold corresponding to the second clustering method; Correspondingly, the preset target clustering method and the target threshold corresponding to the target clustering method are used to cluster the speaker voice features corresponding to each target single-channel voice segment data corresponding to the target multiple people alternating speaking scene, so as to obtain the target speaker cluster corresponding to each speaker in the target multiple people alternating speaking scene, including: Clustering the speaker voice features corresponding to each target single-channel speech segment data corresponding to a target multiple-person alternating speaking scene using a first clustering method and a first threshold value corresponding to the first clustering method to obtain a first speaker cluster corresponding to each speaker in the target multiple-person alternating speaking scene; the first speaker cluster includes at least one target single-channel speech segment data; Furthermore, clustering the speaker voice features corresponding to each target single-channel speech segment data corresponding to the target multiple-person alternating speaking scene using a second clustering method and a second threshold value corresponding to the second clustering method to obtain a second speaker cluster corresponding to each speaker in the target multiple-person alternating speaking scene; the second speaker cluster includes at least one target single-channel speech segment data; Based on a preset consensus clustering scoring strategy, the target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene is determined according to the first speaker cluster and the second speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene.

9. The single-channel multi-speaker separation method according to claim 8, characterized in that: The first clustering method includes spectral clustering, and / or the second clustering method includes Kmeans clustering.

10. A single-channel multi-speaker factor extraction model training device, characterized in that: include: A text factor acquisition module is used to obtain the text features of the expression content corresponding to the first single-channel voice segment data emitted by each speaker in a multi-person alternating speaking scenario; a training sample construction module, configured to construct a training sample from the expression content text features of each speaker and second single-channel speech segment data, wherein the expression content corresponding to the second single-channel speech segment data and the first single-channel speech segment data are different; A model training module is used to use each of the training samples to perform preset model training steps based on text distillation of the expression content for the encoder and decoder in the variational autoencoder, so as to train the encoder into a single-channel multi-speaker factor extraction model for extracting speaker speech features corresponding to single-channel speech segment data.

11. A single-channel multi-speaker factor extraction device, characterized in that: include: A data acquisition module is used to acquire multiple target single-channel speech segment data corresponding to a target multi-person alternating speaking scene; A speaker speech feature extraction module is used to input each of the target single-channel speech segment data into a single-channel multi-speaker factor extraction model, so that the single-channel multi-speaker factor extraction model outputs the speaker speech features corresponding to each of the target single-channel speech segment data, wherein the single-channel multi-speaker factor extraction model is pre-trained based on the single-channel multi-speaker factor extraction model training method according to any one of claims 1 to 4.

12. A single-channel multi-speaker separation device, characterized in that: include: a clustering module, configured to cluster the speaker voice features corresponding to each target single-channel speech segment data corresponding to a target multi-person alternating speaking scene using a preset target clustering method and a target threshold value corresponding to the target clustering method, so as to obtain a target speaker cluster corresponding to each speaker in the target multi-person alternating speaking scene, wherein the target speaker cluster includes all the target single-channel speech segment data belonging to the same speaker; The speaker speech features corresponding to each target single-channel speech segment data are obtained in advance based on the single-channel multi-speaker factor extraction method according to claim 5 or 6.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements at least one of the single-channel multi-speaker factor extraction model training method according to any one of claims 1 to 4, the single-channel multi-speaker factor extraction method according to claim 5 or 6, and the single-channel multi-speaker separation method according to any one of claims 7 to 9.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements at least one of the single-channel multi-speaker factor extraction model training method according to any one of claims 1 to 4, the single-channel multi-speaker factor extraction method according to claim 5 or 6, and the single-channel multi-speaker separation method according to any one of claims 7 to 9.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements at least one of the single-channel multi-speaker factor extraction model training method according to any one of claims 1 to 4, the single-channel multi-speaker factor extraction method according to claim 5 or 6, and the single-channel multi-speaker separation method according to any one of claims 7 to 9.

Images