Voice separation method, device and equipment, storage medium and voice transfer method
The method improves speech separation and transcription accuracy in complex environments by using neural networks for speaker segmentation and clustering, addressing issues of echo interference and multiple signal sources in far-field speech recognition.
Patent Information
- Application Number
- CN202310649621.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
Existing far-field speech recognition technologies face challenges in accurately separating and transcribing speech in complex acoustic environments due to issues like echo interference, indoor reverberation, and interference from multiple signal sources, leading to inaccurate speech separation and transcription.
A method involving multi-channel voice data processing, including speaker segmentation and clustering, using neural networks and spatial filtering to separate and classify speech segments, followed by speech recognition and transcription to improve accuracy.
Enhances the accuracy and precision of speech separation and transcription, even in complex scenarios with changing speakers and environments, by effectively segmenting and clustering speech data, and integrating recognition results in chronological order.
Smart Images

Figure CN119068900A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of voice interaction technology, and in particular to a voice separation method and device, a voice transcription method and device, a computer device and a storage medium. Background Art
[0002] With the development of voice interaction technology, far-field speech recognition technology has emerged. Far-field speech recognition is an important technology in the field of voice interaction. It mainly faces problems such as echo interference, indoor reverberation, and interference from multiple signal sources. This technology has important application prospects in scenarios such as smart homes (smart speakers, smart TVs, etc.), conference rooms, and car navigation. For example, conference scenes contain a variety of speech styles and complex acoustic conditions, and it is necessary to consider overlapping speech, an unknown number of speakers, far-field signals in large conference rooms, noise, and reverberation.
[0003] In traditional technology, the method of recognizing speech from far-field scenes such as conference scenes is to obtain the conference speech signal through a microphone array, then obtain the corresponding multi-speaker separated output speech signal through a spatial speech role separation module, and finally transcribe the output speech signal through a speech recognition module to generate speech recognition text. However, the traditional method has the problem of inaccurate speech separation, which will also lead to inaccurate speech transcription text. Summary of the invention
[0004] Based on this, it is necessary to provide a speech separation method, device, computer equipment and storage medium that can improve the accuracy of speech separation, as well as a speech transcription method, device, computer equipment and storage medium that can improve the accuracy of speech transcription in response to the above-mentioned technical problems.
[0005] A speech separation method, the method comprising:
[0006] Obtain multi-channel mixed voice data;
[0007] Separating single-channel voice data from multi-channel mixed voice data;
[0008] Determine the time point of identity change of different speakers from the single-channel voice data, and divide the single-channel voice data into multiple short voice segments of different speakers according to the time point;
[0009] Classify short speech segments of multiple different speakers to obtain short speech segments belonging to the same speaker;
[0010] Speaker clustering is performed on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker.
[0011] In some embodiments, separating single-channel voice data from multi-channel mixed voice data includes:
[0012] Acquire the amplitude spectrum of the observation signal of the target channel of the multi-channel mixed speech data;
[0013] Extracting phase difference features between channels of multi-channel mixed speech data;
[0014] Input the amplitude spectrum and phase difference features into the masking network based on the recurrent neural network to obtain the masking value output by the masking network;
[0015] Calculate the covariance matrix of the observed signal based on the masked value;
[0016] Derivation of spatial filter coefficients based on minimum mean square distortion-free response according to the covariance matrix;
[0017] The single-channel voice data corresponding to the target channel is separated from the multi-channel mixed voice data according to the spatial filtering coefficient.
[0018] In some embodiments, the speech segments of multiple different speakers are classified to obtain the speech segments belonging to the same speaker, including:
[0019] Inputting speech clips of multiple different speakers into a speaker segmentation network to obtain the prediction probability of each speech clip predicted by the speaker segmentation network;
[0020] The speech clips with the same prediction probability are regarded as the speech clips of the same speaker.
[0021] In some embodiments, speaker clustering is performed on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker, including:
[0022] Extract embedded coding features from short speech segments of the same speaker;
[0023] The restricted incremental clustering algorithm is used to cluster the embedded coding features to obtain the separated speech data belonging to the same speaker.
[0024] In some embodiments, extracting embedded coding features from short speech segments of the same speaker includes:
[0025] Input the short speech segment of the same speaker into the convolutional filter feature extraction layer to obtain the first feature;
[0026] Input the first feature into the time-delay neural network to obtain the second feature;
[0027] Input the second feature into the deep neural network to obtain the third feature;
[0028] The third feature is input into the embedding coding network to obtain the embedding coding feature.
[0029] A speech separation device, the device comprising:
[0030] A mixed voice acquisition module is used to acquire multi-channel mixed voice data;
[0031] A single-channel voice separation module is used to separate single-channel voice data from multi-channel mixed voice data;
[0032] A speaker segmentation module is used to determine the time point when the identities of different speakers change from single-channel voice data, segment the single-channel voice data into multiple short voice segments of different speakers according to the time point, and classify the short voice segments of multiple different speakers using a speaker segmentation network to obtain short voice segments belonging to the same speaker;
[0033] The speaker clustering module is used to perform speaker clustering processing on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker.
[0034] A speech transcription method, the method comprising:
[0035] Obtaining separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained according to any of the above-mentioned speech separation methods;
[0036] Performing speech recognition operations on each separated speech data respectively to obtain speech recognition texts corresponding to each separated speech data;
[0037] The speech recognition texts of multiple channels are merged, and the speech recognition texts corresponding to the separated speech data are output in the order of the time when the separated speech data are generated.
[0038] A speech transcription device, comprising:
[0039] A mixed speech separation module is used to obtain separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained by any of the above-mentioned speech separation methods;
[0040] A speech recognition module is used to perform speech recognition operations on each separated speech data to obtain speech recognition results corresponding to each separated speech data;
[0041] The transcription output module is used to merge the speech recognition results of multiple channels and convert the speech recognition results corresponding to each separated speech data into text for output in the order of the time when the separated speech data was generated.
[0042] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the steps of any one of the above-mentioned speech separation methods or the steps of any one of the above-mentioned speech transcription methods.
[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned speech separation methods or the steps of any one of the above-mentioned speech transcription methods.
[0044] The above-mentioned speech separation method, device, computer equipment and storage medium separate single-channel speech data from multi-channel mixed speech data, and perform speaker segmentation and speaker clustering on the separated single-channel speech data, so as to obtain separated speech data belonging to the same speaker, thereby improving the accuracy of speech separation. Even in complex voice interaction scenarios such as meetings, personnel changes, speaker direction changes, etc., they can adapt well to changes, thereby improving the accuracy and precision of speech separation. Furthermore, the separated speech data is transcribed into text through speech recognition, and the separated speech data corresponding to multiple channels are integrated, and the speech recognition text is output in chronological order, thereby improving the integrity, logic and accuracy of speech transcription. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A flowchart of a speech separation method in some embodiments;
[0046] Figure 2 A schematic flow chart of the steps of separating single-channel voice data from multi-channel mixed voice data in some embodiments;
[0047] Figure 3 A schematic diagram of the structure of a speaker segmentation network in some embodiments;
[0048] Figure 4 A schematic diagram of a neural network structure for extracting embedded coding features in some embodiments;
[0049] Figure 5 A flowchart of a speech transcription method in some embodiments;
[0050] Figure 6 is a structural block diagram of a speech separation device in some embodiments;
[0051] Figure 7 It is a structural block diagram of a speech transcription device in some embodiments;
[0052] Figure 8 It is a structural block diagram of a speech transcription device in some application examples;
[0053] Fig. 9 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0055] In some embodiments, the speech separation method and speech transcription method provided in the present application can be applied to terminal devices, and the terminal devices may be but are not limited to various personal computers, laptops, smart phones, tablet computers, portable wearable devices, smart audio equipment, smart microphones, smart conference transcription tools, smart voice recorders, vehicle-mounted terminal devices, etc.
[0056] In some embodiments, the speech separation method and speech transcription method provided in the present application can be applied to a server, and the server can be implemented as an independent server or a server cluster composed of multiple servers.
[0057] In some embodiments, Figure 1 As shown, a speech separation method is provided, which is described by taking the application of the method to a terminal device as an example, and includes the following steps:
[0058] Step S102: Acquire multi-channel mixed voice data.
[0059] Among them, multi-channel mixed voice data can be obtained through a multi-channel microphone array. Mixed voice data refers to audio data that may contain overlapping voices of an unknown number of speakers, for example, it can be conference voice data collected in large conference scenes, event scenes, public transportation scenes, etc.
[0060] Exemplarily, in a scenario where conference speech separation or transcription is set, an 8-channel microphone array can be used to pick up or collect multi-channel mixed speech data of the conference speech.
[0061] Step S104: Separate the multi-channel mixed voice data to obtain single-channel voice data.
[0062] The single-channel voice data may be understood as single-channel voice data from a single-channel sound source.
[0063] Specifically, there is no limitation on the method for separating single-channel voice data from multi-channel mixed voice data. It can be a method of spatial voice role separation, or separation based on the minimum variance distortion-free response method, or the separation method proposed in the following embodiments of the present application, etc.
[0064] Step S106: determining the time point when the identities of different speakers change from the single-channel voice data, and dividing the single-channel voice data into a plurality of short voice segments of different speakers according to the time point.
[0065] Among them, due to the complex personnel changes and voice interactions in scenarios such as meetings, even in the separated single-channel voice data, due to reasons such as personnel position, conversation direction, the addition or change of new participants, etc., a certain sound source may not have only the same speaker, but there may be voice alternation or changes between speakers of different styles or identities.
[0066] In this step, the time points of identity changes of different speakers can be extracted by processing each separated single-channel voice data separately. For example, a speaker segmentation model can be used to find the time points of identity changes of different speakers. By identifying and extracting feature information that can reflect the speaker identity change in each single-channel voice data, the time points of identity changes of different speakers can be determined, and then the single-channel voice data can be segmented into several short voice segments of different speakers based on these time points.
[0067] Step S108: classifying the speech segments of multiple different speakers to obtain speech segments belonging to the same speaker.
[0068] In this step, the multiple short speech segments segmented by identifying the time points when different speakers change their identities may not belong to the same speaker, and there may be a phenomenon that short speech segments of different speakers appear alternately. Therefore, the speaker segmentation network can be used to further classify the short speech segments of multiple different speakers. According to the probability output by the speaker segmentation network, the short speech segments belonging to the same speaker are classified into the same category, thereby obtaining short speech segments belonging to the same speaker.
[0069] Step S110: Perform speaker clustering processing on the short speech segments of the same speaker to obtain separated speech data belonging to the same speaker.
[0070] In this step, embedded coding features can be extracted from each short speech segment of the same speaker, and then the embedded coding features are clustered based on the restricted incremental clustering algorithm to obtain complete and non-overlapping single-person speech data belonging to the same speaker.
[0071] The above-mentioned speech separation method separates single-channel speech data from multi-channel mixed speech data, and performs speaker segmentation and speaker clustering on the separated single-channel speech data, thereby obtaining separated speech data belonging to the same speaker, thereby improving the accuracy of speech separation. Even in complex voice interaction scenarios such as meetings, personnel changes, speaker direction changes, etc., it can adapt well to changes, thereby improving the accuracy and precision of speech separation.
[0072] In some embodiments, reference Figure 2 As shown, Figure 2 A flow chart of steps for separating single-channel voice data from multi-channel mixed voice data in some embodiments is shown. Separating single-channel voice data from multi-channel mixed voice data may specifically include the following steps:
[0073] S202: Acquire the amplitude spectrum of the observation signal of the target channel of the multi-channel mixed speech data.
[0074] S204: Extracting phase difference features between channels of multi-channel mixed speech data.
[0075] Exemplarily, in the above two steps, the amplitude spectrum of the observed signal of a target channel and the phase difference features between the channels can be combined as the input of the masking network of the recurrent neural network (RNN), wherein, exemplarily, the phase difference features between the channels can be calculated as follows:
[0076]
[0077] Where i,j are the channel indices of the microphone or microphone array, and A vector of frequency domain observation signals with the specified channel index.
[0078] S206: Input the amplitude spectrum and phase difference features into a masking network based on a recurrent neural network to obtain a masking value output by the masking network.
[0079] For example, the masking network can use the short-time Fourier transform mask (STFT mask) as a training target, and its loss function uses a permutation-invariant training criterion based on the minimum mean square error. The mask value (masking value) estimated by the network is k is the sound source index. For example, here we need to estimate at least two sound sources k∈{0, 1}.
[0080] S208: Calculate the covariance matrix of the observation signal according to the masking value.
[0081] Exemplarily, based on the estimated masking value, the covariance matrix of the observed signal is The following calculation formula can be used:
[0082]
[0083] The masking value is k is the sound source index.
[0084] S208: Deriving spatial filter coefficients based on minimum mean square distortion-free response according to the covariance matrix.
[0085] For example, the spatial filter coefficient based on MVDR (Minimum Variance Distortionless Response) can refer to the following calculation formula:
[0086]
[0087] Among them, tr is the trace operation of the matrix, u r is the one-hot vector representing the reference microphone.
[0088] S210: Separating the single-channel voice data corresponding to the target channel from the multi-channel mixed voice data according to the spatial filtering coefficient.
[0089] Among them, the speaker signal of the separated single-channel sound source That is, single-channel voice data can be expressed as:
[0090]
[0091] In this embodiment, at least two channels of voice data from different sound sources can be separated from the multi-channel mixed voice data collected by the microphone array, thereby achieving preliminary separation of the mixed voice data of multiple speakers.
[0092] In some embodiments, speech segments of multiple different speakers are classified to obtain speech segments belonging to the same speaker, including: inputting the speech segments of multiple different speakers into a speaker segmentation network to obtain the prediction probability of each speech segment predicted by the speaker segmentation network; and treating the speech segments with the same prediction probability as the speech segments of the same speaker.
[0093] For example, the speaker segmentation network can be based on the PyanNet network structure. The specific structure can be found in Figure 3 As shown, Figure 3The schematic diagram of the structure of the speaker segmentation network in some embodiments is shown, which may specifically include: Sinc convolution layer, CNN (convolution neural network) layer, recursive loop layer, pooling layer and fully connected layer.
[0094] For example, the loss function of the speaker segmentation network can be referred to as follows:
[0095]
[0096] in, is the binary cross entropy loss function for multi-label classification, z = {z1,…,z T}, K max is the maximum number of speakers. Output probabilities of the speaker segmentation model.
[0097] In the above-mentioned embodiment, the single-channel voice data is segmented by a speaker segmentation network, and then the voice activity probabilities of the segmented short voice segments of different speakers are predicted, so that the short voice segments of different speakers in the single-channel voice data can be segmented more accurately.
[0098] In some embodiments, speaker clustering is performed on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker, including: extracting embedded coding features from each short speech segment of the same speaker; clustering each embedded coding feature using a restricted incremental clustering algorithm to obtain separated speech data belonging to the same speaker.
[0099] In some embodiments, embedded coding features are extracted from short speech segments of the same speaker, including: inputting the short speech segments of the same speaker into a convolutional filter feature extraction layer to obtain a first feature; inputting the first feature into a time-delay neural network to obtain a second feature; inputting the second feature into a deep neural network to obtain a third feature; and inputting the third feature into an embedded coding network to obtain an embedded coding feature.
[0100] For example, reference may be made to Figure 4 As shown, Figure 4A schematic diagram of a neural network structure for extracting embedded coding features in some embodiments is shown. Specifically, short speech segments of the same speaker can be input into the neural network, and after passing through a convolutional filter feature extraction layer, that is, a SincNet feature extraction layer, each input speech short segment is subjected to SincNet feature extraction, that is, first feature extraction, and then the first feature is sequentially passed through three layers of TDNN (Time Delay Neural Network) layers to obtain a second feature, and then the second feature is passed through two layers of DNN (Deep Neural Networks) to obtain a third feature, and then the third feature is input into an embedded coding network layer after statistical pooling processing by a statistical pooling layer, and embedded coding features are extracted according to the third feature in the embedded coding network layer, and then the extracted embedded coding features are output after passing through a DNN layer, and the output result is further input into a restricted incremental clustering algorithm module, and based on the restricted incremental clustering algorithm, speaker clustering processing is completed, thereby obtaining non-overlapping complete speech data of the same speaker.
[0101] In another aspect of the present application, a speech transcription method is also provided. Figure 5 As shown, Figure 5 A flow chart of a speech transcription method in some embodiments is shown. Taking the method using a terminal device as an example, the method includes the following steps:
[0102] Step S502: obtaining separated speech data corresponding to each channel and belonging to the same speaker.
[0103] In this step, the separated speech data corresponding to each channel and belonging to the same speaker can be obtained according to the speech separation method described in any of the above embodiments, which will not be described in detail here.
[0104] Step S504: performing speech recognition operations on each separated speech data respectively to obtain speech recognition texts corresponding to each separated speech data.
[0105] In this step, the separated speech data corresponding to each channel and belonging to the same speaker can be input into the speech recognition engine corresponding to each channel, for example, it can be an end-to-end speech recognition engine based on a conformer, so as to obtain the speech recognition text output by each speech recognition engine. Of course, the separated speech data of different channels can also be input into the same speech recognition engine for recognition.
[0106] Step S506: merging the speech recognition texts of multiple channels, and outputting the speech recognition texts corresponding to the separated speech data in the order of the time when the separated speech data were generated.
[0107] The above-mentioned speech transcription method separates single-channel speech data from multi-channel mixed speech data, and performs speaker segmentation and speaker clustering processing on the separated single-channel speech data, thereby obtaining separated speech data belonging to the same speaker, thereby improving the accuracy of speech separation. Even in complex voice interaction scenarios such as meetings, personnel changes, speaker direction changes, etc., it can adapt well to changes and improve the accuracy and precision of speech separation. Furthermore, the separated speech data is transcribed into text through speech recognition, and the separated speech data corresponding to multiple channels are integrated, and the speech recognition text is output in chronological order, thereby improving the integrity, logic and accuracy of speech transcription.
[0108] It should be understood that although Figure 1 , 2 The steps in the flowcharts of and 5 are shown in sequence as indicated by arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , 2 At least part of the steps in 5 may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequentially, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0109] In some embodiments, Figure 6 As shown, a speech separation device is provided, including: a mixed speech acquisition module 602, a single-channel speech separation module 604, a speaker segmentation module 606 and a speaker clustering module 608, wherein:
[0110] A mixed voice acquisition module 602 is used to acquire multi-channel mixed voice data;
[0111] A single-channel voice separation module 604 is used to separate single-channel voice data from multi-channel mixed voice data;
[0112] A speaker segmentation module 606 is used to determine the time point when the identities of different speakers change from the single-channel voice data, segment the single-channel voice data into multiple short voice segments of different speakers according to the time point, and classify the short voice segments of multiple different speakers using a speaker segmentation network to obtain short voice segments belonging to the same speaker;
[0113] The speaker clustering module 608 is used to perform speaker clustering processing on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker.
[0114] In some embodiments, the single-channel speech separation module 604 obtains the amplitude spectrum of the observation signal of the target channel of the multi-channel mixed speech data; extracts the phase difference characteristics between the channels of the multi-channel mixed speech data; inputs the amplitude spectrum and the phase difference characteristics into a masking network based on a recurrent neural network to obtain a masking value output by the masking network; calculates the covariance matrix of the observation signal according to the masking value; derives the spatial filtering coefficient based on the minimum mean square distortion-free response according to the covariance matrix; and separates the single-channel speech data corresponding to the target channel from the multi-channel mixed speech data according to the spatial filtering coefficient.
[0115] In some embodiments, the speaker segmentation module 606 inputs speech segments of multiple different speakers into a speaker segmentation network to obtain the prediction probability of each speech segment predicted by the speaker segmentation network; and regards speech segments with the same prediction probability as speech segments of the same speaker.
[0116] In some embodiments, the speaker clustering module 608 extracts embedded coding features from each short speech segment of the same speaker; and clusters each embedded coding feature using a restricted incremental clustering algorithm to obtain separated speech data belonging to the same speaker.
[0117] In some embodiments, the speaker clustering module 608 inputs short speech segments of the same speaker into the convolutional filter feature extraction layer to obtain a first feature; inputs the first feature into the time-delay neural network to obtain a second feature; inputs the second feature into the deep neural network to obtain a third feature; and inputs the third feature into the embedded coding network to obtain an embedded coding feature.
[0118] For the specific definition of the speech separation device, please refer to the definition of the speech separation method above, which will not be repeated here. Each module in the above-mentioned speech separation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0119] In some embodiments, Figure 7 As shown, a speech transcription device is provided, comprising: a mixed speech separation module 702, a speech recognition module 704 and a transcription output module 706, wherein:
[0120] The mixed speech separation module 702 is used to obtain separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained by any of the above-mentioned speech separation methods;
[0121] The speech recognition module 704 is used to perform speech recognition operations on each separated speech data to obtain speech recognition results corresponding to each separated speech data;
[0122] The transcription output module 706 is used to merge the speech recognition results of multiple channels, and convert the speech recognition results corresponding to each separated speech data into text for output in the order of the time when the separated speech data was generated.
[0123] Below, in conjunction with an application embodiment, reference Figure 8 As shown, Figure 8 The following is a schematic diagram showing the structure of a speech transcription device in some application examples.
[0124] from Figure 8 It can be seen that based on Figure 8 The voice transcription transposition shown in the figure performs voice transcription, which may include the following steps:
[0125] Step 1: Set up the conference speech transcription scenario and use an 8-channel microphone array to pick up multi-channel mixed speech data of the conference speech;
[0126] Step 2: The multi-channel mixed voice data that has been picked up is processed by a speaker voice separation module based on MVDR to obtain single-channel voice data (speaker voice signal of a single sound source);
[0127] Step 3: Perform speaker log analysis on the single-channel voice data corresponding to one channel (first channel) obtained. The speaker log analysis may include speaker segmentation processing, finding the time points when the identities of different speakers change from the audio of multi-person conversations, and then segmenting the voice into several short voice segments according to the time points of these changes;
[0128] Step 4: Perform speaker clustering processing in the analysis of speaker logs on the obtained short speech segments to obtain speech data of the same speaker without overlap;
[0129] Step 5: Perform speech recognition processing on the speech segments of the same speaker to obtain corresponding speech recognition text results;
[0130] Step 6: Perform speaker segmentation, speaker clustering, and speech recognition on the single-channel speech data of another channel (second channel) obtained in the same manner as steps 3-5 described above, and obtain corresponding speech recognition text results;
[0131] Step 7: Finally, the speech recognition text results of the two channels are merged and the speech transcription separation results are output in chronological order.
[0132] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech separation method or a speech transcription method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0133] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0134] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: acquiring multi-channel mixed voice data; separating single-channel voice data from the multi-channel mixed voice data; determining time points of identity changes of different speakers from the single-channel voice data, and dividing the single-channel voice data into multiple short voice segments of different speakers according to the time points; classifying the multiple short voice segments of different speakers to obtain short voice segments belonging to the same speaker; and performing speaker clustering processing on the short voice segments of the same speaker to obtain separated voice data belonging to the same speaker.
[0135] In some embodiments, when the processor executes a computer program to separate single-channel voice data from multi-channel mixed voice data, the following steps are specifically implemented: obtaining the amplitude spectrum of the observation signal of the target channel of the multi-channel mixed voice data; extracting the phase difference characteristics between the channels of the multi-channel mixed voice data; inputting the amplitude spectrum and the phase difference characteristics into a masking network based on a recurrent neural network to obtain a masking value output by the masking network; calculating the covariance matrix of the observation signal according to the masking value; deriving the spatial filtering coefficient based on the minimum mean square distortion-free response according to the covariance matrix; and separating the single-channel voice data corresponding to the target channel from the multi-channel mixed voice data according to the spatial filtering coefficient.
[0136] In some embodiments, the processor executes a computer program to classify speech clips of multiple different speakers, and when obtaining speech clips belonging to the same speaker, the following steps are specifically implemented: the speech clips of multiple different speakers are input into a speaker segmentation network to obtain the predicted probability of each speech clip predicted by the speaker segmentation network; and the speech clips with the same predicted probability are regarded as speech clips of the same speaker.
[0137] In some embodiments, when the processor executes a computer program to implement speaker clustering processing on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker, the following steps are specifically implemented: extracting embedded coding features from each short speech segment of the same speaker; clustering each embedded coding feature using a restricted incremental clustering algorithm to obtain separated speech data belonging to the same speaker.
[0138] In some embodiments, the processor executes a computer program to extract embedded coding features from short speech segments of the same speaker, input the short speech segments of the same speaker into a convolutional filter feature extraction layer to obtain a first feature; input the first feature into a time-delay neural network to obtain a second feature; input the second feature into a deep neural network to obtain a third feature; input the third feature into an embedded coding network to obtain an embedded coding feature.
[0139] In some embodiments, the processor executes the computer program to obtain separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained according to any of the above-mentioned speech separation methods; each separated speech data is subjected to speech recognition operation respectively to obtain speech recognition text corresponding to each separated speech data; the speech recognition texts of multiple channels are merged, and the speech recognition texts corresponding to each separated speech data are output in the order of the time when the separated speech data is generated.
[0140] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: acquiring multi-channel mixed voice data; separating single-channel voice data from the multi-channel mixed voice data; determining the time point of identity change of different speakers from the single-channel voice data, and dividing the single-channel voice data into multiple short voice segments of different speakers according to the time point; classifying the multiple short voice segments of different speakers to obtain short voice segments belonging to the same speaker; and performing speaker clustering processing on the short voice segments of the same speaker to obtain separated voice data belonging to the same speaker.
[0141] In some embodiments, when a computer program is executed by a processor to separate single-channel voice data from multi-channel mixed voice data, the following steps are specifically implemented: obtaining the amplitude spectrum of the observation signal of the target channel of the multi-channel mixed voice data; extracting the phase difference characteristics between the channels of the multi-channel mixed voice data; inputting the amplitude spectrum and the phase difference characteristics into a masking network based on a recurrent neural network to obtain a masking value output by the masking network; calculating the covariance matrix of the observation signal according to the masking value; deriving the spatial filtering coefficient based on the minimum mean square distortion-free response according to the covariance matrix; and separating the single-channel voice data corresponding to the target channel from the multi-channel mixed voice data according to the spatial filtering coefficient.
[0142] In some embodiments, the computer program is executed by a processor to classify speech clips of multiple different speakers. When speech clips belonging to the same speaker are obtained, the following steps are specifically implemented: the speech clips of multiple different speakers are input into a speaker segmentation network to obtain the predicted probability of each speech clip predicted by the speaker segmentation network; and the speech clips with the same predicted probability are regarded as speech clips of the same speaker.
[0143] In some embodiments, when a computer program is executed by a processor to perform speaker clustering processing on short speech segments of the same speaker and obtain separated speech data belonging to the same speaker, the following steps are specifically implemented: extracting embedded coding features from each short speech segment of the same speaker; clustering each embedded coding feature using a restricted incremental clustering algorithm to obtain separated speech data belonging to the same speaker.
[0144] In some embodiments, a computer program is executed by a processor to extract embedded coding features from short speech segments of the same speaker, input the short speech segments of the same speaker into a convolutional filter feature extraction layer to obtain a first feature; input the first feature into a time-delay neural network to obtain a second feature; input the second feature into a deep neural network to obtain a third feature; and input the third feature into an embedded coding network to obtain an embedded coding feature.
[0145] In some embodiments, the computer program is executed by the processor to obtain separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained according to any of the above-mentioned speech separation methods; each separated speech data is subjected to speech recognition operation respectively to obtain speech recognition text corresponding to each separated speech data; the speech recognition texts of multiple channels are merged, and the speech recognition texts corresponding to each separated speech data are output in the order of the time when the separated speech data is generated.
[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0148] In addition, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the characters in this article generally indicate that the associated objects before and after are in an "or" relationship.
[0149] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A speech separation method, the method comprising: Obtain multi-channel mixed voice data; Separating the multi-channel mixed voice data to obtain single-channel voice data; Determining the time points at which the identities of different speakers change from the single-channel voice data, and dividing the single-channel voice data into a plurality of short voice segments of different speakers according to the time points; Classify the short speech segments of multiple different speakers to obtain the short speech segments belonging to the same speaker; Speaker clustering is performed on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker.
2. The method according to claim 1, characterized in that The step of separating the single-channel voice data from the multi-channel mixed voice data comprises: Acquire the amplitude spectrum of the observation signal of the target channel of the multi-channel mixed speech data; Extracting phase difference features between channels of multi-channel mixed speech data; Inputting the amplitude spectrum and the phase difference feature into a masking network based on a recurrent neural network to obtain a masking value output by the masking network; Calculating a covariance matrix of the observation signal according to the masking value; Derivation of spatial filter coefficients based on minimum mean square distortion-free response according to the covariance matrix; The single-channel voice data corresponding to the target channel is separated from the multi-channel mixed voice data according to the spatial filter coefficient.
3. The method according to claim 1, characterized in that: The method of classifying the short speech segments of multiple different speakers to obtain the short speech segments belonging to the same speaker includes: Inputting the speech short segments of the multiple different speakers into a speaker segmentation network to obtain a prediction probability of each speech short segment predicted by the speaker segmentation network; The speech segments with the same prediction probability are regarded as the speech segments of the same speaker.
4. The method according to claim 1, characterized in that: The speaker clustering process is performed on the short speech segments of the same speaker to obtain separated speech data belonging to the same speaker, including: Extract embedded coding features from short speech segments of the same speaker; The embedded coding features are clustered using a restricted incremental clustering algorithm to obtain separated speech data belonging to the same speaker.
5. The method according to claim 1, characterized in that The step of extracting embedded coding features from short speech segments of the same speaker includes: Input the short speech segment of the same speaker into the convolutional filter feature extraction layer to obtain the first feature; Inputting the first feature into a time-delay neural network to obtain a second feature; Inputting the second feature into a deep neural network to obtain a third feature; The third feature is input into the embedding coding network to obtain the embedding coding feature.
6. A speech separation device, characterized in that: The device comprises: A mixed voice acquisition module is used to acquire multi-channel mixed voice data; A single-channel voice separation module, used to separate the single-channel voice data from the multi-channel mixed voice data; A speaker segmentation module, used to determine the time point when the identity of different speakers changes from the single-channel voice data, segment the single-channel voice data into a plurality of short voice segments of different speakers according to the time point, and further used to classify the short voice segments of the plurality of different speakers using a speaker segmentation network to obtain short voice segments belonging to the same speaker; The speaker clustering module is used to perform speaker clustering processing on short speech segments of the same speaker to obtain separated speech data belonging to the same speaker.
7. A speech transcription method, the method comprising: Acquire separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained according to the speech separation method according to any one of claims 1 to 5; Performing speech recognition operations on each of the separated speech data to obtain speech recognition texts corresponding to each of the separated speech data; The speech recognition texts of multiple channels are merged, and the speech recognition texts corresponding to the separated speech data are output in the order of the generation time of the separated speech data.
8. A speech transcription device, characterized in that: The device comprises: A mixed speech separation module, used for obtaining separated speech data corresponding to each channel and belonging to the same speaker; wherein the separated speech data is obtained according to the speech separation method according to any one of claims 1 to 5; A speech recognition module, used to perform speech recognition operations on each of the separated speech data respectively to obtain speech recognition results corresponding to each of the separated speech data; The transcription output module is used to merge the speech recognition results of multiple channels, and convert the speech recognition results corresponding to each separated speech data into text for output in the order of the time when the separated speech data was generated.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the speech separation method described in any one of claims 1 to 5, or implements the steps of the speech transcription method described in claim 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech separation method described in any one of claims 1 to 5 are implemented, or the steps of the speech transcription method described in claim 7 are implemented.
Citation Information
Cited By
Speaker recognition method in medical real-time speech recognition scene
CN121938378A
A medical real-time speech recognition scene speaker recognition method
CN121938378B
