Speech recognition method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a voice recognition method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the voice activity detection of to-be-processed voice data, and obtaining a plurality of voice segments; determining the number of speakers in each voice segment; under the condition that the number of speakers in the voice segment is multiple, performing voice separation on the voice segment to obtain multiple audio segments; performing voice recognition on the plurality of audio segments to obtain a voice recognition result of each audio segment; performing feature extraction on the plurality of audio segments to obtain a voiceprint feature vector of each audio segment; the speakers are clustered based on the voiceprint feature vectors of the multiple audio segments, a clustering result is obtained, and the clustering result comprises the audio segment corresponding to each speaker; and obtaining a voice recognition result of each speaker in the to-be-processed voice data based on the clustering result and the voice recognition result of each audio segment. According to the embodiment of the invention, multi-channel multi-speaker speech recognition can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a speech recognition method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the application scenarios of multi-speaker multi-channel speech recognition, such as conferences, multi-person customer service, etc., it is necessary to identify not only the speakers, but also the corresponding voices and texts of each speaker. In the communication and dialogue phase of the conference, the voices of different speakers are prone to overlap, and the current single-channel speech recognition method cannot recognize the voices of multiple speakers, especially the overlapping parts of the voices. Therefore, it is necessary to develop a multi-channel multi-speaker speech recognition method. Summary of the invention

[0003] The present disclosure provides a speech recognition method and device, an electronic device, and a computer-readable storage medium, which can realize multi-channel and multi-speaker speech recognition.

[0004] In a first aspect, the present disclosure provides a speech recognition method, the speech recognition method comprising:

[0005] Perform voice activity detection on the voice data to be processed to obtain multiple voice segments;

[0006] determining the number of speakers in each of the speech segments;

[0007] When there are multiple speakers in the speech segment, performing speech separation on the speech segment to obtain multiple audio segments;

[0008] Performing speech recognition on the multiple audio segments to obtain a speech recognition result for each of the audio segments;

[0009] Performing feature extraction on the multiple audio segments to obtain a voiceprint feature vector of each audio segment;

[0010] Clustering the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result, wherein the clustering result includes the audio segment corresponding to each of the speakers;

[0011] Based on the clustering result and the speech recognition result of each of the audio segments, a speech recognition result of each of the speakers in the speech data to be processed is obtained.

[0012] In a second aspect, the present disclosure provides a speech recognition device, the speech recognition device comprising:

[0013] A detection module, used for performing voice activity detection on the voice data to be processed to obtain multiple voice segments;

[0014] A determination module, used to determine the number of speakers in each of the speech segments;

[0015] A separation module, configured to perform speech separation on the speech segment to obtain multiple audio segments when there are multiple speakers in the speech segment;

[0016] A first recognition module, configured to perform speech recognition on the plurality of audio segments to obtain a speech recognition result for each of the audio segments;

[0017] An extraction module, configured to extract features from the plurality of audio segments to obtain a voiceprint feature vector of each of the audio segments;

[0018] A clustering module, configured to cluster the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result, wherein the clustering result includes the audio segment corresponding to each speaker;

[0019] An obtaining module is used to obtain a speech recognition result of each speaker in the speech data to be processed based on the clustering result and the speech recognition result of each audio segment.

[0020] In a third aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned speech recognition method.

[0021] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned speech recognition method when executed by a processor / processing core.

[0022] The speech recognition method provided by the embodiment of the present disclosure processes the speech data to be processed to obtain multiple speech segments. When there are multiple speakers, the speech separation is performed on each of the speech segments, and the overlapping speech parts of multiple speakers are separated to obtain multiple audio segments, so that each of the audio segments corresponds to one speaker. Then, speech recognition is performed on each of the audio segments to obtain a speech recognition result of each of the audio segments. Furthermore, feature extraction is performed on the multiple audio segments to obtain a voiceprint feature vector of each of the audio segments. The speakers are clustered based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result. Then, based on the clustering result and each of the speech recognition results, a speech recognition result of each speaker in the speech data to be processed is obtained. Since both speech recognition and clustering are performed on the basis of the separated audio segments, compared with directly recognizing the speech segments, separating the overlapping parts of multiple speakers into different audio segments before performing speech recognition can improve the accuracy of speech recognition. Clustering based on the separated audio segments is more accurate, so that the audio segments of different speakers can be accurately classified. Therefore, even in a multi-channel multi-speaker scenario, the same speaker and the corresponding speech can be inaccurately recognized.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0025] Figure 1 The following schematically shows an application scenario diagram of the speech recognition method and device provided by the embodiment of the present disclosure;

[0026] Figure 2 A flow chart of a speech recognition method provided by an embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of the structure of a speaker segmentation model provided in an embodiment of the present disclosure;

[0028] Figure 4 A flow chart for determining the number of speakers for an embodiment of the present disclosure;

[0029] Figure 5 A structural diagram of a speech separation model provided in an embodiment of the present disclosure;

[0030] Figure 6 A flow chart of speech separation provided by an embodiment of the present disclosure;

[0031] Figure 7 A block diagram of a speech recognition device provided by an embodiment of the present disclosure;

[0032] Figure 8 A flowchart of a method for generating meeting minutes provided in an embodiment of the present disclosure;

[0033] Fig. 9 A block diagram of a device for generating meeting minutes provided in an embodiment of the present disclosure;

[0034] Fig.10 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0036] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0037] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0038] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0039] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0040] In scenarios where multiple people are talking (such as conferences and customer service), the voice data obtained by the transcription device is multi-channel voice data. In the voice recognition process, it is necessary not only to identify the speaker, but also to identify the voice corresponding to each speaker. During the conversation, the voices of different speakers are prone to overlap, and the existing voice recognition methods are mainly for single-channel multi-speaker voice data, and cannot recognize multi-channel multi-speaker voice data.

[0041] Figure 1 The application scenario diagram of the speech recognition method and device provided by the embodiments of the present disclosure is schematically shown.

[0042] like Figure 1 As shown, the application scenario of the embodiment of the present disclosure may include a terminal device 101, a network 103 and a server 102. The network 103 is used to provide a medium for a communication link between the terminal device 101 and the server 102. The network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] The user can use the terminal device 101 to interact with the server 102 through the network 103 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0044] The terminal device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0045] The server 102 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal device 101. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0046] It should be noted that the speech recognition method and apparatus provided in the embodiments of the present disclosure may be executed by the server 102. Accordingly, the speech recognition method and apparatus provided in the embodiments of the present disclosure may be arranged in the server 102. The speech recognition method and apparatus provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 102 and that can communicate with the terminal device 101 and / or the server 102. Accordingly, the speech recognition method and apparatus provided in the embodiments of the present disclosure may also be arranged in a server or server cluster that is different from the server 102 and that can communicate with the terminal device 101 and / or the server 102.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0048] Figure 2 A flow chart of a speech recognition method provided by an embodiment of the present disclosure. Figure 2 , the method comprising:

[0049] Step S201: Perform voice activity detection on the voice data to be processed to obtain multiple voice segments.

[0050] The voice data to be processed may be voice data in a conference scenario held by multiple microphones and multiple speakers. The voice activity detection may be online detection or offline detection.

[0051] Voice activity detection can determine whether there is voice activity (speaker's voice) in the voice data to be processed, remove the voice segments without voice activity, and only retain the voice segments with voice. After removing the voice segments without voice activity, the voice data to be processed is divided into multiple voice segments. The embodiments of the present disclosure do not limit the method of voice activity detection.

[0052] Step S202, determining the number of speakers in each speech segment.

[0053] Each speech segment is processed to determine the number of speakers in each speech segment. In each speech segment, there may be one speaker or multiple speakers.

[0054] In some embodiments, a speaker segmentation model is used to determine the number of speakers corresponding to each speech segment. Figure 3 A schematic diagram of the structure of the speaker segmentation model provided in the embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the speaker segmentation model includes a feature extraction module 31, an encoding module 32, a decoding module 33, a merging module 34 and a prediction module 35.

[0055] In some embodiments, the feature extraction module 31 is used to extract the voiceprint features and Mel-cepstral coefficient features of the speech segment, wherein the feature extraction module 31 includes a voiceprint feature network 311 and a Mel-cepstral coefficient network 312, and the voiceprint feature network 311 is used to extract the voiceprint features of the speech segment. Each speech segment is input into the voiceprint feature network, and the voiceprint feature network outputs the voiceprint feature vector (x-vector) corresponding to each speech segment. The present disclosure embodiment does not limit the voiceprint feature network. For example, the voiceprint feature network includes any one of the cam++ network and the resnet network.

[0056] The Mel-scale Frequency Cepstral Coefficient network 312 is used to extract Mel-scale Frequency Cepstral Coefficients (MFCC) features. Each speech segment is input into the Mel-scale Frequency Cepstral Coefficient network to obtain MFCC features.

[0057] In some embodiments, the encoding module 32 is used to encode the Mel cepstral coefficient feature to obtain a feature encoding vector. The encoding module 32 includes a convolution enhanced self-attention neural (Conformer) network. The Conformer network can combine the Transformer network and the CNN network. The Transformer network is good at capturing content-based global interaction information, while the CNN network can effectively capture local features. Therefore, the Conformer network can model both long-term global interaction information and local features better. The encoding module 32 in the disclosed embodiment can use a multi-layer Conformer network, such as a 4-layer Conformer network.

[0058] In some embodiments, the decoding module 33 is used to decode the feature encoding vector to obtain a decoded feature vector. The decoding module 33 includes a self-attention neural network. The embodiment of the present disclosure does not limit the self-attention neural network. For example, the self-attention neural network can use a Transformer network. The Transformer network can effectively encode time series information, and its processing ability for time series information is better than that of a long short-term memory network, and the processing speed is fast.

[0059] In some embodiments, the merging module 34 is used to merge the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector. The merging module 34 includes a merge sort algorithm, which merges the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector. The feature prediction vector is used to represent the prediction vector of the speaker's characteristics, and the number of speakers can be determined by the prediction vector.

[0060] In some embodiments, the prediction module 35 is used to predict the number of speakers of each speech segment based on the feature prediction vector to obtain the number of speakers of each speech segment. The prediction module 35 includes a Long Short-Term Memory (LSTM) network.

[0061] Figure 4 FIG. 1 is a flow chart for determining the number of speakers in an embodiment of the present disclosure. Figure 4 As shown, the method for determining the number of speakers includes:

[0062] Step S401, extracting voiceprint features and Mel-frequency cepstral coefficient features of a speech segment.

[0063] The voiceprint feature network is used to extract the voiceprint features of the speech segment, and the Mel-cepstral coefficient network is used to extract the Mel-cepstral coefficient features.

[0064] Step S402, encode the Mel-frequency cepstral coefficient feature to obtain a feature encoding vector.

[0065] The Mel cepstral coefficient features are encoded through the encoding module to obtain the feature encoding vector. For the specific structure and function of the encoding module, see Figure 3 The corresponding text part will not be repeated here.

[0066] Step S403: decode the feature encoding vector to obtain a decoded feature vector.

[0067] By decoding the feature encoding vector, the decoded feature vector is obtained. For the specific structure and function of the decoding module, see Figure 3 The corresponding text part will not be repeated here.

[0068] Step S404: merge the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector.

[0069] The merging module is used to merge the decoded feature vector and the voiceprint feature vector to obtain the feature prediction vector. The specific structure and function of the merging module can be found in Figure 3 The corresponding text part will not be repeated here.

[0070] Step S405 , predicting the number of speakers based on the feature prediction vector to obtain the number of speakers in the speech segment.

[0071] The prediction module is used to predict the number of speakers based on the feature prediction vector to obtain the number of speakers in the speech segment. For the specific structure and function of the prediction module, see Figure 3 The corresponding text part will not be repeated here.

[0072] The speaker segmentation model is trained by the following steps:

[0073] Step S41, obtaining a voiceprint feature vector of a first speech segment sample, where the first speech segment sample is a speech segment marked with the number of speakers.

[0074] There may be one or more speakers in the first speech segment sample. Since each speaker has different voiceprint features, different speakers can be distinguished based on the voiceprint feature vector.

[0075] The disclosed embodiment does not limit the method for obtaining the voiceprint feature vector of the first speech segment sample. For example, the voiceprint feature vector of the first speech segment sample is obtained through the cam++ model, that is, the first speech segment sample is input into the cam++ model, and the cam++ model outputs the voiceprint feature vector (x-vector) of the first speech segment sample. The cam++ model can be obtained through a large number of sample training, or a trained model can be used.

[0076] For example, the first speech segment sample obtains four voiceprint vectors, namely, v1 vector, v2 vector, v3 vector and v4 vector.

[0077] Step S42: extracting Mel-frequency cepstral coefficient features from the first speech segment sample.

[0078] In the disclosed embodiment, a Mel-scale Frequency Cepstral Coefficient network is used to extract Mel-scale Frequency Cepstral Coefficient (MFCC) features from the first speech segment sample.

[0079] Step S43: input the Mel-frequency cepstral coefficient feature into the encoding module for encoding to obtain a feature encoding vector.

[0080] The encoding module includes a 4-layer Conformer network. The Mel-frequency cepstral coefficient feature is input into the 4-layer Conformer network to obtain a feature encoding vector.

[0081] Step S44, input the feature encoding vector into the temporal neural network to obtain a decoded feature vector.

[0082] In the decoding module, the feature encoding vector passes through a two-layer Transformer network to obtain the decoded feature vector.

[0083] Step S45, merging the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector.

[0084] The disclosed embodiment can combine the decoded feature vector and the voiceprint feature vector respectively through matrix multiplication operation to obtain a feature prediction vector.

[0085] The decoded feature vector obtained in step S44 is merged with the voiceprint feature vectors v1 vector, v2 vector, v3 vector and v4 vector obtained in step S41 through a merge sort algorithm, such as merging the decoded feature vector with the v1 vector to obtain a feature prediction vector SD1, merging the decoded feature vector with the v2 vector to obtain a feature prediction vector SD2, merging the decoded feature vector with the ,3 vector to obtain a feature prediction vector SD3, and merging the decoded feature vector with the ,4 vector to obtain a feature prediction vector SD4.

[0086] Step S46, inputting the feature prediction vector into the long short-term memory network to obtain the predicted number of speakers in the speech segment.

[0087] The feature prediction vectors SD1, SD2, SD3 and SD4 are input into the long short-term memory network, and the feature prediction vectors SD1, SD2, SD3 and SD4 are merged and mapped using the long short-term memory network to output the predicted quantities output1, output2, output3 and output4.

[0088] When the difference between the predicted number of speakers and the labeled number is within a preset difference threshold range, a speaker segmentation model is obtained.

[0089] Step S203: When there are multiple speakers in the speech segment, perform speech separation on the speech segment to obtain multiple audio segments.

[0090] When there is only one speaker in a speech segment, the speech segment with only one speaker is used as an audio segment, and each audio segment corresponds to one speaker.

[0091] Each speech segment may include one or more speakers. When it is determined that there is only one speaker in the speech segment, the speech segment is used as an audio segment. When it is determined that there are multiple speakers in the speech segment, speech separation is performed on each speech segment to obtain multiple audio segments, so that each audio segment corresponds to one speaker.

[0092] In some embodiments, a speech separation model is used to perform speech separation on speech segments with multiple speakers to obtain multiple audio segments.

[0093] Figure 5 FIG. 1 is a structural diagram of a speech separation model provided in an embodiment of the present disclosure. Figure 5As shown, the speech separation model 500 includes a Fourier transform module 51, a compression module 52, a network encoder 53, a mapping encoder 54, a first decoder 55, a second decoder 56, an amplification module 57, an inverse Fourier transform module 58 and a loss module 59, wherein the Fourier transform module 51 is used to input the second speech segment sample into the Fourier transform module for short-time Fourier transform to obtain a complex spectrum. The compression module 52 is used to input the complex spectrum into the compression module for power-law compression to obtain a spectrum map. The network encoder 53 is used to input the spectrum map into the network encoder for encoding to obtain a network code. The mapping encoder 54 is used to input the network code into the mapping encoder for mapping coding to obtain a mapping code. The first decoder 55 is used to input the mapping code into the first decoder to obtain the predicted phase information of the second speech segment sample. The first decoder is a prediction mask, and a decoder that predicts the speech segment phase information based on the predicted code and the amplitude of the second speech segment sample. The second decoder 56 is used to input the mapping code into the second decoder to obtain the predicted amplitude information of the second speech segment sample. The amplification module 57 is used to input the predicted phase information and the predicted amplitude information into the amplification module to obtain an enhanced complex spectrum with amplitude enhancement. The inverse Fourier transform module 58 is used to input the enhanced complex spectrum into the inverse Fourier transform module for inverse short-time Fourier transform to obtain the predicted time domain signal of the second speech segment sample. The loss module 59 is used to input the predicted time domain signal of the second speech segment sample into the judgment module and compare it with the standard time domain signal. When the difference between the predicted time domain signal of the second speech segment sample and the standard time domain signal is within a preset difference range, a speech separation model is obtained.

[0094] Figure 6 A flow chart of speech separation provided by an embodiment of the present disclosure. Figure 6 As shown, the speech separation model is used to separate each speech segment, and multiple audio segments are obtained including:

[0095] Step S601, performing short-time Fourier transform on the speech segment to obtain a complex spectrum.

[0096] The Fourier transform module is used to perform short-time Fourier transform on the speech segment to obtain a complex spectrum. The structure and function of the Fourier transform module can be found in Figure 5 And the corresponding text part will not be repeated here.

[0097] Step S602, performing power law compression on the complex spectrum to obtain a frequency spectrum.

[0098] The compressed spectrum is obtained by power law compression using the compression module. The structure and function of the compression module can be found in Figure 5 And the corresponding text part will not be repeated here.

[0099] Step S603: encode the spectrum graph to obtain network coding.

[0100] The spectrum graph is encoded using a network encoder to obtain network coding. The structure and function of the network encoder refer to Figure 5 And the corresponding text part will not be repeated here.

[0101] Step S604, mapping coding is performed on the network coding to obtain a mapping code.

[0102] Use the mapping encoder to map the network code to obtain the mapping code. The structure and function of the mapping encoder refer to Figure 5 And the corresponding text part will not be repeated here.

[0103] Step S605, processing the mapping code to obtain the predicted phase information and predicted amplitude information of the speech segment.

[0104] The mapping code is processed by the first decoder to obtain the predicted phase information and predicted amplitude information of the speech segment. The structure and function of the first decoder refer to Figure 5 And the corresponding text part will not be repeated here.

[0105] Step S606, processing the mapping code to obtain the real part and the imaginary part of the speech segment.

[0106] The mapping code is processed by the second decoder to obtain the real part and imaginary part of the speech segment. The structure and function of the second decoder refer to Figure 5 And the corresponding text part will not be repeated here.

[0107] Step S607: Process the predicted phase information, the predicted amplitude information, and the real part and the imaginary part of the speech segment to obtain an enhanced complex spectrum with amplitude enhancement.

[0108] Step S608: perform inverse short-time Fourier transform on the enhanced complex spectrum, process the enhanced complex spectrum, and obtain multiple audio segments of the speech segment.

[0109] In some embodiments, the speech separation model is trained by the following steps:

[0110] Step S61: Input the second speech segment sample into the Fourier transform module for short-time Fourier transform (STFT) to obtain a complex spectrum.

[0111] The speech waveform in the second speech segment sample is converted into a complex spectrum by short-time Fourier transform.

[0112] Step S62, input the complex spectrum into a compression module for power law compression to obtain a spectrum diagram.

[0113] The compressed spectrum is obtained by power law compression, so that the importance of quiet sounds is equal to that of loud sounds.

[0114] Step S63, inputting the spectrum graph into the network encoder for encoding to obtain the network code.

[0115] The network encoder includes two convolution blocks, each of which includes a convolution layer, an instance normalization, and a PReLU activation. The network coding can be performed using the ffmpeg converter.

[0116] Step S64, input the network code into a mapping encoder for mapping coding to obtain a mapping code.

[0117] The mapping encoder may use two layers of TS-Conformer encoders, each layer having input_channel=32 and conv_kernel=7. The TS-Conformer encoder may capture time dependency and frequency dependency.

[0118] Step S65, inputting the mapping code into the first decoder to obtain the predicted phase information of the second speech segment sample.

[0119] The first decoder is used to predict a mask, that is, to obtain a predicted mask, and then predict the speech segment phase information based on the predicted code and the amplitude of the second speech segment sample. The first decoder of the embodiment of the present disclosure may be a mask decoder.

[0120] Step S66, input the mapping code into the second decoder to obtain the real part and the imaginary part of the second speech segment sample.

[0121] The second decoder may adopt a complex decoder, and the complex decoder may directly predict the real part and the imaginary part of the second speech segment sample.

[0122] Step S67, inputting the predicted phase information, the predicted amplitude information, and the real part and the imaginary part of the second speech segment sample into an amplification module to obtain an enhanced complex spectrum with amplitude enhancement.

[0123] The predicted phase information Y P and forecast amplitude information and then combined with the output of the second decoder Add the elements to get the enhanced complex spectrum with enhanced amplitude

[0124]

[0125]

[0126] Step S68: input the enhanced complex spectrum into an inverse Fourier transform module for inverse short-time Fourier transform to obtain a time domain signal of the second speech segment sample.

[0127] Step S69, determining a loss value based on the time domain signal and a preset loss function, and obtaining a speech separation model when the loss value is within a preset loss range.

[0128] The loss function in the embodiment of the present disclosure may adopt the time-domain MSEloss function. In order to solve the speaker prediction order problem, the speech separation model may be trained in a PIT manner, and the distance between the separated speech sounds is added as the separation loss.

[0129] In some embodiments, the second speech segment sample may be raw data generated by a customer service system, or near-field data obtained by a conference cloud. In order to make the trained speech separation model more accurate, the raw data is processed for noise mixing and augmentation to obtain more second speech segment samples.

[0130] The noise-addition and noise-addition processing includes noise reduction, pitch shifting, speech mixing, reverberation and noise addition.

[0131] Among them, noise reduction is to remove background noise and noise such as microphone spray. The noise reduction model can be the 16k FRCRN on ModelScope.

[0132] Tone shifting is to increase the disturbance of speech. After the disturbance, the speech is regarded as different speakers. You can use sox to perform the disturbance of [-200,200].

[0133] When mixing voices, 1 to 3 different speakers can be randomly selected, and the audio of each speaker can be randomly cropped and spliced ​​(including overlap) into 10s of audio, with SIR = -3 to 3dB. Among them, these audios contain 140,000 data of 1 speaker, 140,000 data of 2 speakers, and 49,000 data of 3 speakers, with a total audio duration of 914 hours.

[0134] Adding reverberation can make the sound more natural. The disclosed embodiment can randomly generate 20,000 rooms with 8 channels RIR, RT60 = 0.03-0.2, the room length and width range is 3-10 meters, and the height is 2.5-4 meters, and reverberation is added to speech and noise. The open source tool FRAM_RIR is used to generate RIR.

[0135] The DNS noise data set can be used to add noise, with a total of 65,000 voices. Each voice is randomly added with 1 to 3 types of noise and 1 type of white noise, with the signal-to-noise ratio (SNR) of the noise being 10 to 20 dB and the signal-to-noise ratio (SNR) of the white noise being 15 to 35 dB.

[0136] By using the augmented second speech segment samples to train the speech separation model, the number of second speech segment samples can be increased, and more importantly, the accuracy of the speech separation model can be improved.

[0137] It should be noted that when training the speech separation model, the second speech segment samples can be classified according to the number of speakers, and the speech separation model can be trained using the second speech segment samples with the same number of speakers, and the speech separation model can be trained in order from the number of speakers to the number of speakers. For example, when the number of speakers is 3, 2 channels can be randomly selected, and the model can be trained based on the two channels, and then 3 channels can be randomly selected and the model can be trained based on the three channels.

[0138] In order to increase the predictability of the speaker order, the permutation invariance (PIT) method can be used for training. The distance between the separated voices can also be added as the separation loss to determine the cutoff condition of the voice separation model.

[0139] In the embodiment of the present disclosure, the speech separation model can also adopt a CMGAN model or other models and be trained using the second speech segment sample. The training method is the same as the speech separation model training method provided in the embodiment of the present disclosure, and will not be repeated here.

[0140] Step S204: performing speech recognition on the multiple audio segments to obtain a speech recognition result for each audio segment.

[0141] The embodiment of the present disclosure can perform speech recognition on multiple audio segments through a speech recognition model to obtain a speech recognition result for each audio segment.

[0142] In some embodiments, the speech recognition model is trained by the following steps:

[0143] Perform noise augmentation processing on speech recognition samples to obtain speech recognition training samples, which include speech recognition samples and augmented samples after noise augmentation processing; input the speech recognition training samples into the speech recognition model to be trained, and the speech recognition model outputs the speech recognition results corresponding to the speech recognition samples; determine the speech recognition loss value based on the speech recognition results corresponding to the speech recognition samples and a preset loss function, and obtain the speech recognition model when the speech recognition loss value is less than or equal to a preset loss threshold.

[0144] The speech recognition sample is an audio segment after speech separation, or an audio segment with only one speaker. In other words, there is only one speaker in each speech recognition sample.

[0145] The embodiments of the present disclosure can be used for noise augmentation processing in an online manner or in an offline manner. When performing noise augmentation processing in an online manner, noise can be added, speed can be changed, and spectrum can be enhanced. The signal-to-noise ratio of the noise addition can be randomly selected, such as 15:20:25. The speed change can be an increase in speed and a decrease in speed, such as a speed range of 0.9-1.1 times the original speed. The maximum mask time during spectrum enhancement is 20h, and the maximum mask frequency is 10, which increases the number of speech recognition samples, that is, the total time of the speech recognition samples is increased by dozens of times, thereby enhancing the robustness of the speech recognition model.

[0146] When the noise-mixing augmentation processing is performed in an offline manner, the noise-mixing augmentation processing method is the same as the noise-mixing augmentation processing method when training the speech separation model, and will not be described in detail here.

[0147] In some embodiments, the speech recognition samples for training the speech recognition model are audio segments output by the speech separation model.

[0148] The disclosed embodiment uses the audio segment output by the speech separation model as a speech recognition sample for training the speech recognition model, with the purpose of jointly training the speech separation model and the speech recognition model. The disclosed embodiment can use the separated audio segment output by the speech separation model as a speech recognition sample for training the speech recognition model, and can also separate the test set used when training the speech separation model to obtain the separated audio segment, and use the audio segment as a speech recognition sample for training the speech recognition model.

[0149] When the speech separation model and the speech recognition model are jointly trained, the weights of the speech separation loss and the speech recognition loss can be set. The embodiments of the present disclosure do not limit the specific values ​​of the weights of the speech separation loss and the speech recognition loss. For example, the weight of the speech separation loss is set to 0.1, and the weight of the speech recognition loss is set to 0.9. Alternatively, the weight of the speech separation loss can be set to 0.2, and the weight of the speech recognition loss can be set to 0.8.

[0150] When the speech separation model and the speech recognition model are jointly trained, the learning rate can also be set. The learning rate can be set arbitrarily as needed, such as setting the learning rate to 0.0005. It should be pointed out that setting the learning rate to a smaller value can reduce the amplitude of the oscillation of the model learning direction, thereby reducing learning in the wrong direction.

[0151] In some embodiments, the original speech data can be directly used as a training sample, and the original speech data can be subjected to voice activity detection to obtain multiple speech segments, and the number of speakers in each speech segment can be determined. If there is only one speaker in the speech segment, it can be directly used as a training sample for the speech recognition model. If there are multiple speakers in the speech segment, the speech segment is input into the speech separation model, and the audio segment output by the speech separation model is used as a training sample for the speech recognition model, and the overall gradient is returned. The total loss value is determined based on the speech separation loss value and the speech recognition loss value. When the total loss value is less than or equal to the preset total loss threshold, the speech separation model and the speech recognition model are trained.

[0152] Step S205: extract features from multiple audio segments to obtain a voiceprint feature vector for each audio segment.

[0153] In the disclosed embodiment, feature extraction may be performed on the audio segment using feature extraction tools such as kaldi and python_speech.

[0154] Step S206: clustering the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain clustering results.

[0155] Clustering is to cluster the audio segments of the same speaker together, that is, the clustering result includes the audio segment corresponding to each speaker.

[0156] Since a conversation (a conference recording is referred to as a conversation in this disclosure embodiment) usually includes multiple audio segments, which are generated by multiple speakers, it is necessary to cluster the audio segments of each speaker together, that is, to determine the speaker of each audio segment.

[0157] In some embodiments, clustering of audio segments is performed by:

[0158] The voiceprint feature vectors of multiple audio segments are normalized to obtain the normalized feature vector corresponding to each audio segment; each normalized feature vector is transposed to obtain the corresponding transposed feature vector; each normalized feature vector and the corresponding transposed feature vector are inner-product processed to obtain the cross-similarity matrix of the multiple audio segments, where each row represents a class; a clustering operation is performed based on the cross-similarity matrix until the new cross-similarity matrix meets the clustering termination condition to obtain the clustering result.

[0159] In the embodiment of the present disclosure, the purpose of normalization is to eliminate the size difference between different audio segments, and the amplitude of different audio segments can be limited to between [-1, 1]. The embodiment of the present disclosure does not limit the specific method of normalization, for example, the normalization can be performed by ASV-Subtools.

[0160] In the disclosed embodiment, the transposition process is to obtain the audio similarity matrix. In the similarity matrix, the value of each element represents the similarity between the voiceprint information of the audio segment in the row and the voiceprint information of the audio frequency band in the column. The disclosed embodiment does not limit the specific method of the transposition process. For example, the normalized feature vector of N*M dimension is transposed to the transposed feature vector of M*N dimension. Wherein, N represents the number of chunks, and M represents the length of each voiceprint vector, which can be 192, 256, 512, etc.

[0161] In the disclosed embodiment, each normalized feature vector and the corresponding transposed feature vector are inner-producted to obtain a cross-similarity matrix of multiple audio segments. For example, an N*M-dimensional normalized feature vector and an M*N-dimensional transposed feature vector are inner-producted to obtain an N*N-dimensional cross-similarity matrix, which is a scoring matrix, and each row represents a class.

[0162] In some embodiments, the clustering operation includes: calculating the distance between any two rows in the cross-similarity matrix; merging the two rows with the shortest distance to obtain a new cross-similarity matrix.

[0163] The distance between any two rows can be Consine similarity or Euclidean distance. Calculate the distance between any two rows in the cross-similarity matrix and get Distance, merge the two rows with the closest distance into one class. After merging the classes with the closest distance, a new cross-similarity matrix is ​​obtained.

[0164] In the disclosed embodiment, the distance between any two rows is calculated again based on the new cross-similarity matrix until the new cross-similarity matrix meets the termination condition to obtain a clustering result. In the clustering result, each class corresponds to an audio segment of a speaker, that is, the audio segments of the same speaker are clustered into one class.

[0165] For example, if the input speech is 6 times, the result after clustering is [0 1 1 0 1 1], then the speaker categories of the first to sixth times are 0, 1, 1, 0, 1, 1, where 0 and 1 represent different speakers respectively.

[0166] In some embodiments, the clustering termination condition includes only two rows remaining in the cross-similarity matrix after clustering.

[0167] In the disclosed embodiment, the rows in the cross-similarity matrix are merged, and clustering is completed when only two rows are left in the entire cross-similarity matrix.

[0168] Step S207: based on the clustering result and the speech recognition result of each audio segment, obtain the speech recognition result corresponding to each speaker in the speech data to be processed.

[0169] The clustering results of each audio segment in the speech data to be processed are associated with the speech recognition results, and the audio segments belonging to the same category are associated with the speech recognition results to obtain the speech recognition results corresponding to each speaker in the speech data to be processed.

[0170] In some embodiments, based on the clustering results and the speech recognition results corresponding to each audio segment, the speech recognition results corresponding to each speaker in the speech data to be processed are obtained, including: based on the clustering results, the target audio segment of each speaker in the speech data to be processed is associated with the speaker, and the speech recognition results corresponding to each speaker in the speech data to be processed are obtained.

[0171] In the clustering results, each class corresponds to a speaker. Therefore, the speaker corresponding to each audio segment can be determined based on the clustering results. After the audio segment is matched with the speaker, the speech recognition result corresponding to the audio segment is associated with the speaker to obtain the speech recognition result corresponding to each speaker in the speech data to be processed.

[0172] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0173] The speech recognition method provided by the embodiment of the present disclosure processes the speech data to be processed through speech activity detection to obtain multiple speech segments. When there are multiple speakers, speech separation is performed on each speech segment, and the overlapping speech parts of multiple speakers are separated to obtain multiple audio segments, so that each audio segment corresponds to one speaker. When there is only one speaker, the speech segment is used as an audio segment, and speech recognition is performed on each audio segment to obtain a speech recognition result of each audio segment. Feature extraction is performed on the multiple audio segments to obtain a voiceprint feature vector of each audio segment, and the speakers are clustered based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result. Then, based on the clustering result and the speech recognition result, a speech recognition result of each speaker in the speech data to be processed is obtained. Since speech recognition and clustering are both performed on the basis of the separated audio segments, compared with directly recognizing the speech segments, separating the overlapping parts of multiple speakers into different audio segments can improve the accuracy of speech recognition, clustering based on the separated audio segments is more accurate, and the judgment of the speaker is more accurate. Therefore, even in a multi-channel multi-speaker scenario, speech recognition can be accurately performed.

[0174] Figure 7 A block diagram of a speech recognition device provided in an embodiment of the present disclosure.

[0175] Reference Figure 7 , the embodiment of the present disclosure provides a speech recognition device, the speech recognition device 700 includes:

[0176] The detection module 701 is used to perform voice activity detection on the voice data to be processed to obtain multiple voice segments.

[0177] The determination module 702 is used to determine the number of speakers in each speech segment.

[0178] In some embodiments, the determination module 702 uses a speaker segmentation model to determine the number of speakers corresponding to each speech segment.

[0179] In some embodiments, the speaker segmentation model includes a feature extraction module, an encoding module, a decoding module, a merging module and a prediction module; wherein the feature extraction module is used to extract voiceprint features and Mel-cepstral coefficient features of a speech segment; the encoding module is used to encode the Mel-cepstral coefficient features to obtain a feature encoding vector; the decoding module is used to decode the feature encoding vector to obtain a decoded feature vector; the merging module is used to merge the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector; the prediction module is used to predict the number of speakers based on the feature prediction vector to obtain the number of speakers.

[0180] In some embodiments, a speaker segmentation model is used to determine the number of speakers corresponding to each speech segment; voiceprint features and Mel-cepstral coefficient features of the speech segment are extracted; the Mel-cepstral coefficient features are encoded to obtain a feature encoding vector; the feature encoding vector is decoded to obtain a decoded feature vector; the decoded feature vector and the voiceprint feature vector are merged to obtain a feature prediction vector; and the number of speakers is predicted based on the feature prediction vector to obtain the number of speakers in the speech segment.

[0181] In some embodiments, the feature extraction module includes a voiceprint feature network and a Mel cepstral coefficient network, wherein the voiceprint feature network is used to extract voiceprint features of a speech segment, and the Mel cepstral coefficient network is used to extract Mel cepstral coefficient features. The encoding module includes a convolution enhanced self-attention neural network, the decoding module includes a self-attention neural network, the merging module merges the decoded feature vector and the voiceprint feature vector through a merge sort algorithm, and the prediction module includes a long short-term memory network.

[0182] The separation module 703 is used to perform speech separation on the speech segment to obtain multiple audio segments when there are multiple speakers in the speech segment.

[0183] When the number of speakers is one, the speech segments of the speaker are taken as audio segments, and each audio segment corresponds to one speaker.

[0184] In some embodiments, the separation module 703 performs speech separation on each speech segment using a speech separation model to obtain multiple audio segments.

[0185] In some embodiments, the speech separation model includes a Fourier transform module, a compression module, a network encoder, a mapping encoder, a first decoder, a second decoder, an amplification module, an inverse Fourier transform module and a loss module.

[0186] The Fourier transform module is used to perform short-time Fourier transform on the second speech segment sample to obtain a complex spectrum;

[0187] The compression module is used to perform power-law compression on the complex spectrum to obtain a spectrum diagram;

[0188] The network encoder is used to encode the spectrum graph to obtain the network code;

[0189] The mapping encoder is used to perform mapping encoding on the network coding to obtain the mapping coding;

[0190] The first decoder is used to process the mapping code to obtain the predicted phase information and predicted amplitude information of the second speech segment sample;

[0191] The second decoder is used to process the mapping code to obtain the real part and the imaginary part of the second speech segment sample;

[0192] The amplification module is used to process the predicted phase information, the predicted amplitude information, and the real part and the imaginary part of the second speech segment sample to obtain an enhanced complex spectrum with amplitude enhancement;

[0193] The inverse Fourier transform module performs inverse short-time Fourier transform to process the enhanced complex spectrum to obtain a time domain signal of the second speech segment sample;

[0194] The loss module is used to determine the loss value based on the time domain signal and the preset loss function, and obtain the speech separation model when the loss value is within the preset loss range.

[0195] In some embodiments, a speech separation model is used to perform speech separation on each speech segment to obtain multiple audio segments; the speech segment is short-time Fourier transformed to obtain a complex spectrum; the complex spectrum is power-law compressed to obtain a spectrogram; the spectrum is encoded to obtain a network code; the network code is mapped to obtain a mapping code; the mapping code is processed to obtain predicted phase information and predicted amplitude information of the speech segment; the mapping code is processed to obtain the real and imaginary parts of the speech segment; the predicted phase information, predicted amplitude information, and the real and imaginary parts of the speech segment are processed to obtain an enhanced complex spectrum with amplitude enhancement; the enhanced complex spectrum is inversely short-time Fourier transformed, and the enhanced complex spectrum is processed to obtain multiple audio segments of the speech segment.

[0196] The first recognition module 704 is used to perform speech recognition on multiple audio segments to obtain a speech recognition result corresponding to each audio segment.

[0197] In some embodiments, the first recognition module 704 performs speech recognition on each audio segment using a speech recognition model to obtain a speech recognition result corresponding to each audio segment.

[0198] In some embodiments, a speech recognition model is obtained through the following training steps: performing noise augmentation processing on a speech recognition sample to obtain a speech recognition training sample, the speech recognition training sample including a speech recognition sample and an augmented sample after the noise augmentation processing; inputting the speech recognition training sample into the speech recognition model to be trained, and the speech recognition model outputs a speech recognition result corresponding to the speech recognition sample; determining a speech recognition loss value based on the speech recognition result corresponding to the speech recognition sample and a preset loss function, and obtaining a speech recognition model when the speech recognition loss value is less than or equal to a preset loss threshold.

[0199] In some embodiments, the speech recognition samples used to train the speech recognition model are audio segments output by the speech separation model, thereby achieving joint training of the speech recognition model and the speech separation model.

[0200] The extraction module 705 is used to extract features from multiple audio segments and obtain a voiceprint feature vector corresponding to each audio segment.

[0201] The clustering module 706 is used to cluster the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result, where the clustering result includes the audio segment corresponding to each speaker.

[0202] In some embodiments, the clustering module 706 normalizes the voiceprint feature vectors of multiple audio segments to obtain a normalized feature vector for each audio segment; transposes each normalized feature vector to obtain a corresponding transposed feature vector; performs inner product processing on each normalized feature vector and the corresponding transposed feature vector to obtain a cross-similarity matrix for the multiple audio segments, wherein each row in the cross-similarity matrix represents a class; and performs clustering operation steps based on the cross-similarity matrix until the new cross-similarity matrix meets the clustering termination condition to obtain a clustering result.

[0203] In some embodiments, the clustering operation step includes: calculating the distance between any two rows in the cross-similarity matrix; merging the two rows with the closest distance to obtain a new cross-similarity matrix.

[0204] In some embodiments, the clustering termination condition includes only two rows remaining in the cross-similarity matrix after clustering.

[0205] The obtaining module 707 is used to obtain the speech recognition result corresponding to each speaker in the speech data to be processed based on the clustering result and the speech recognition result corresponding to each audio segment.

[0206] In some embodiments, the acquisition module 707 determines the target audio segment of each speaker in the speech data to be processed based on the clustering results; associates the speech recognition result of each target audio segment with the speaker to obtain the speech recognition result corresponding to each speaker in the speech data to be processed.

[0207] The speech recognition device provided by the embodiment of the present disclosure comprises a detection module that processes the speech data to be processed through speech activity detection to obtain a plurality of speech segments, a determination module that determines the number of speakers in each speech segment, and when there are multiple speakers, a separation module that performs speech separation on each speech segment, separates the overlapping speech parts of multiple speakers, and obtains a plurality of audio segments, so that each audio segment corresponds to one speaker, and when there is only one speaker, the separation module uses the speech segment of the speaker as the audio segment, a first recognition module that performs speech recognition on each audio segment, and obtains a speech recognition result corresponding to each audio segment, and an extraction module that performs feature extraction on the plurality of audio segments, and obtains a speech recognition result corresponding to each audio segment. The clustering module clusters the speakers based on the voiceprint feature vectors of multiple audio segments to obtain clustering results. The acquisition module obtains the speech recognition results corresponding to each speaker in the speech data to be processed based on the clustering results and the speech recognition results. Since speech recognition and clustering are both performed on the basis of the separated audio segments, compared with directly identifying the speech segments, separating the overlapping parts of multiple speakers into different audio segments and then performing speech recognition can improve the accuracy of speech recognition, and clustering based on the separated audio segments is more accurate, so that the audio segments of different speakers can be accurately classified. Therefore, even in a multi-channel and multi-speaker scenario, the same speaker and the corresponding speech can be inaccurately identified.

[0208] An embodiment provides a method for generating meeting minutes in a multi-person meeting scenario. Figure 8 A flowchart of a method for generating meeting minutes provided in an embodiment of the present disclosure. Figure 8 As shown, the method for generating meeting minutes includes:

[0209] Step S801, obtaining conference audio.

[0210] Among them, conference audio is the audio recorded during a conference with multiple microphones and multiple speakers.

[0211] Step S802: Perform speech recognition on the conference speech to obtain speech recognition results corresponding to each speaker.

[0212] In step S802, the conference speech is recognized by the embodiment of the present disclosure to obtain speech recognition results corresponding to each speaker. The speech recognition method adopts the speech recognition method provided by the embodiment of the present disclosure, which will not be described in detail here.

[0213] Step S803: Generate meeting minutes based on the speech recognition results corresponding to each speaker.

[0214] The speech recognition results corresponding to each speaker are sorted according to the speaking order of each speaker to generate the meeting minutes.

[0215] In some embodiments, after step S801, the process also includes saving the conference voice, which may be saved locally or in a designated server.

[0216] The method for generating conference minutes provided by the embodiment of the present disclosure adopts the speech recognition method provided by the embodiment of the present disclosure, so as to recognize the speech of multiple channels and multiple speakers, thereby improving the accuracy of conference minutes.

[0217] The present disclosure also provides a device for generating meeting minutes. Fig. 9 A block diagram of a conference minutes generating device provided by an embodiment of the present disclosure. Fig. 9 As shown, the conference minutes generating device 900 includes:

[0218] The acquisition module 901 is used to acquire conference voice, which is the voice recorded in a multi-channel multi-speaker conference.

[0219] The second recognition module 902 is used to perform speech recognition on the conference speech by using the speech recognition method provided by the embodiment of the present disclosure to obtain speech recognition results corresponding to each speaker.

[0220] The generating module 903 is used to generate the meeting minutes based on the speech recognition results corresponding to each speaker.

[0221] In some embodiments, the conference minutes generating device 900 further includes a storage module for storing conference voices. The storage module may be a local storage module or a storage module in a designated server.

[0222] The conference minutes generating device provided by the embodiment of the present disclosure can recognize the voices of multiple speakers in multiple channels and improve the accuracy of the conference minutes by adopting the voice recognition method provided by the embodiment of the present disclosure.

[0223] Fig.10 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0224] Reference Fig.10An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 1001; at least one memory 1002, and one or more I / O interfaces 1003, connected between the processor 1001 and the memory 1002; wherein the memory 1002 stores one or more computer programs that can be executed by the at least one processor 1001, and the one or more computer programs are executed by the at least one processor 1001, so that the at least one processor 1001 can perform the above-mentioned speech recognition method.

[0225] In some embodiments, processor 1001 is further used to perform voice activity detection on the voice data to be processed to obtain multiple voice segments; determine the number of speakers in each voice segment; when there are multiple speakers in the voice segment, perform voice separation on the voice segment to obtain multiple audio segments; perform voice recognition on the multiple audio segments to obtain a voice recognition result for each audio segment; perform feature extraction on the multiple audio segments to obtain a voiceprint feature vector for each audio segment; cluster the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain clustering results, the clustering results including the audio segment corresponding to each speaker; based on the clustering results and the voice recognition result of each audio segment, obtain a voice recognition result for each speaker in the voice data to be processed.

[0226] In some embodiments, when the number of speakers in a speech segment is one, the speech segment is taken as an audio segment, and each audio segment corresponds to one speaker.

[0227] In some embodiments, processor 1001 is also used to determine the number of speakers in each speech segment, including: using a speaker segmentation model to determine the number of speakers corresponding to each speech segment; extracting voiceprint features and Mel-cepstral coefficient features of the speech segment; encoding the Mel-cepstral coefficient features to obtain a feature encoding vector; decoding the feature encoding vector to obtain a decoded feature vector; merging the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector; and predicting the number of speakers based on the feature prediction vector to obtain the number of speakers in the speech segment.

[0228] In some embodiments, the processor 1001 is also used to perform speech recognition on multiple audio segments, including: using a speech recognition model to perform speech recognition on multiple audio segments; the speech recognition model is obtained through the following training steps: performing noise augmentation processing on the speech recognition samples to obtain speech recognition training samples, the speech recognition training samples include speech recognition samples and augmented samples after the noise augmentation processing; inputting the speech recognition training samples into the speech recognition model to be trained, and the speech recognition model outputs the speech recognition results corresponding to the speech recognition samples; determining the speech recognition loss value based on the speech recognition results and a preset loss function, and obtaining the speech recognition model when the speech recognition loss value is less than or equal to a preset loss threshold.

[0229] In some embodiments, the processor 1001 is also used to use a speech separation model to perform speech separation on each speech segment to obtain multiple audio segments; perform short-time Fourier transform on the speech segment to obtain a complex spectrum; perform power-law compression on the complex spectrum to obtain a spectrogram; encode the spectrum to obtain a network code; perform mapping coding on the network code to obtain a mapping code; process the mapping code to obtain predicted phase information and predicted amplitude information of the speech segment; process the mapping code to obtain the real and imaginary parts of the speech segment; process the predicted phase information, predicted amplitude information, and the real and imaginary parts of the speech segment to obtain an enhanced complex spectrum with amplitude enhancement; perform an inverse short-time Fourier transform on the enhanced complex spectrum, process the enhanced complex spectrum, and obtain multiple audio segments of the speech segment.

[0230] In some embodiments, processor 1001 is also used to normalize the voiceprint feature vectors of multiple audio segments to obtain a normalized feature vector for each audio segment; transpose each normalized feature vector to obtain a corresponding transposed feature vector; perform inner product processing on each normalized feature vector and the corresponding transposed feature vector to obtain a cross-similarity matrix for multiple audio segments, wherein each row in the cross-similarity matrix represents a class; perform clustering operation based on the cross-similarity matrix to obtain a new cross-similarity matrix, and obtain a clustering result when the new cross-similarity matrix satisfies the clustering termination condition; the steps of the clustering operation include: calculating the distance between any two rows in the cross-similarity matrix; merging the two rows with the closest distance to obtain a new cross-similarity matrix.

[0231] In some embodiments, the processor 1001 is further configured to associate the audio segment of each speaker in the speech data to be processed with each speaker based on the clustering result, and obtain a speech recognition result corresponding to each speaker in the speech data to be processed.

[0232] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned speech recognition method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0233] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned speech recognition method.

[0234] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0235] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0236] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0237] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0238] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0239] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0240] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0241] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0242] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0243] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A speech recognition method, characterized in that: include: Perform voice activity detection on the voice data to be processed to obtain multiple voice segments; determining the number of speakers in each of the speech segments; When there are multiple speakers in the speech segment, performing speech separation on the speech segment to obtain multiple audio segments; Performing speech recognition on the multiple audio segments to obtain a speech recognition result for each of the audio segments; Performing feature extraction on the multiple audio segments to obtain a voiceprint feature vector of each audio segment; Clustering the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result, wherein the clustering result includes the audio segment corresponding to each of the speakers; Based on the clustering result and the speech recognition result of each of the audio segments, a speech recognition result of each of the speakers in the speech data to be processed is obtained.

2. The method according to claim 1, characterized in that The method further includes: when the number of speakers in the speech segment is one, using the speech segment as an audio segment, each of the audio segments corresponding to one speaker.

3. The method according to claim 1, characterized in that The determining the number of speakers in each of the speech segments comprises: Determining the number of speakers corresponding to each of the speech segments using a speaker segmentation model; Extracting voiceprint features and Mel-cepstral coefficient features of the speech segment; Encoding the Mel-frequency cepstral coefficient feature to obtain a feature encoding vector; Decoding the feature encoding vector to obtain a decoded feature vector; Merging the decoded feature vector and the voiceprint feature vector to obtain a feature prediction vector; and The number of speakers is predicted based on the feature prediction vector to obtain the number of speakers of the speech segment.

4. The method according to claim 3, characterized in that: The extracting of the voiceprint features and Mel-cepstral coefficient features of the speech segment includes: The voiceprint feature network extracts the voiceprint features of the speech segment, and the Mel-cepstral coefficient network extracts the Mel-cepstral coefficient features.

5. The method according to claim 1, characterized in that The performing speech recognition on the multiple audio segments includes: performing speech recognition on the multiple audio segments using a speech recognition model; the speech recognition model is obtained through the following training steps: Performing noise-mixing and augmentation processing on the speech recognition sample to obtain a speech recognition training sample, wherein the speech recognition training sample includes the speech recognition sample and the augmented sample after the noise-mixing and augmentation processing; Inputting the speech recognition training sample into the speech recognition model to be trained, and the speech recognition model outputs the speech recognition result corresponding to the speech recognition sample; A speech recognition loss value is determined based on the speech recognition result and a preset loss function, and when the speech recognition loss value is less than or equal to a preset loss threshold, the speech recognition model is obtained.

6. The method according to claim 5, characterized in that When there are multiple speakers in the speech segment, performing speech separation on the speech segment to obtain multiple audio segments includes: Performing speech separation on each of the speech segments using a speech separation model to obtain the multiple audio segments; Perform short-time Fourier transform on the speech segment to obtain a complex spectrum; Performing power law compression on the complex number spectrum to obtain a frequency spectrum; Encoding the spectrum graph to obtain network coding; Performing mapping coding on the network coding to obtain mapping coding; Processing the mapping code to obtain predicted phase information and predicted amplitude information of the speech segment; Processing the mapping code to obtain the real part and the imaginary part of the speech segment; Processing the predicted phase information, the predicted amplitude information, and the real part and the imaginary part of the speech segment to obtain an enhanced complex spectrum with amplitude enhancement; An inverse short-time Fourier transform is performed on the enhanced complex spectrum, and the enhanced complex spectrum is processed to obtain a plurality of audio segments of the speech segment.

7. The method according to claim 4, characterized in that The speech recognition samples for training the speech recognition model are audio segments output by the speech separation model.

8. The method according to claim 1, characterized in that The clustering of the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain clustering results includes: Normalizing the voiceprint feature vectors of the plurality of audio segments to obtain a normalized feature vector of each audio segment; Transpose each of the normalized feature vectors to obtain a corresponding transposed feature vector; Performing inner product processing on each of the normalized feature vectors and the corresponding transposed feature vector to obtain a cross-similarity matrix of the multiple audio segments, wherein each row in the cross-similarity matrix represents a class; Performing a clustering operation based on the cross-similarity matrix to obtain a new cross-similarity matrix, and obtaining the clustering result when the new cross-similarity matrix satisfies a clustering termination condition; The steps of the clustering operation include: Calculate the distance between any two rows in the cross-similarity matrix; Merge the two rows with the closest distance to obtain a new cross-similarity matrix.

9. The method according to claim 8, characterized in that The clustering termination condition includes that only two rows remain in the cross-similarity matrix after clustering.

10. The method according to claim 1, characterized in that The obtaining, based on the clustering result and the speech recognition result of each of the audio segments, the speech recognition result of each of the speakers in the speech data to be processed comprises: Based on the clustering result, the audio segment of each speaker in the speech data to be processed is associated with each speaker to obtain a speech recognition result of each speaker in the speech data to be processed.

11. A speech recognition device, characterized in that: include: A detection module, used for performing voice activity detection on the voice data to be processed to obtain multiple voice segments; A determination module, used to determine the number of speakers in each of the speech segments; A separation module, configured to perform speech separation on the speech segment to obtain multiple audio segments when there are multiple speakers in the speech segment; A first recognition module, configured to perform speech recognition on the plurality of audio segments to obtain a speech recognition result for each of the audio segments; An extraction module, configured to extract features from the plurality of audio segments to obtain a voiceprint feature vector of each of the audio segments; A clustering module, configured to cluster the speakers based on the voiceprint feature vectors of the multiple audio segments to obtain a clustering result, wherein the clustering result includes the audio segment corresponding to each speaker; An obtaining module is used to obtain a speech recognition result of each speaker in the speech data to be processed based on the clustering result and the speech recognition result of each audio segment.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the speech recognition method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the speech recognition method according to any one of claims 1 to 10.

Images