Speech recognition device, training method, electronic equipment and storage medium

Abstract

The invention relates to a voice recognition device, a training method, electronic equipment and a storage medium, and the device comprises a voice text recognition module which receives an input audio signal, carries out the text recognition of the audio signal, and obtains a text in the audio signal and a corresponding timestamp; the speaker identity information recognition module is used for recognizing speaker identity information in the audio signals based on acoustic coding features in the audio signals to obtain a speaker identity information sequence in sequential arrangement; the speaker conversion time point identification module is used for identifying a time point at which a speaker changes in the audio signal based on semantic information of a text in the audio signal to obtain a speaker conversion time point; and the integration module is used for determining a text corresponding to the speaker identity information based on the recognized text, the corresponding timestamp, the speaker identity information sequence and the speaker conversion time point. According to the embodiment of the invention, the recognition accuracy of the text corresponding to the speaker can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech recognition device and method, an electronic device, a storage medium, and a training method for a speech recognition device. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition technology, as an important part of human-computer interaction, has been widely used in smartphones, smart homes, car systems and various smart devices. Traditional speech recognition systems mainly focus on converting audio signals into text, but in practical applications, especially in scenarios involving multi-person conversations, simply recognizing text content is often difficult to meet the needs.

[0003] Therefore, a target speaker-oriented speech recognition system came into being, which can further identify the speaker's identity information and divide the speech paragraphs of different speakers based on the recognized text. In the existing speech recognition technology, although the text recognition of audio signals can be realized, the recognition of speaker identity information and the detection of speaker switching time points are still not accurate enough.

[0004] Therefore, in the related art, the accuracy of identifying the speaker's identity information is low. Summary of the invention

[0005] The present disclosure proposes a speech recognition technology solution.

[0006] According to one aspect of the present disclosure, there is provided a speech recognition device, comprising:

[0007] A speech-to-text recognition module receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and a corresponding timestamp;

[0008] A speaker identity information recognition module, based on acoustic coding features in the audio signal, identifies the speaker identity information in the audio signal to obtain a speaker identity information sequence arranged in time sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0009] A speaker switching time point recognition module, which identifies the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal, and obtains the speaker switching time point;

[0010] The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, wherein the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0011] In a possible implementation, the speaker switching time point identification module identifies the time point at which the speaker in the audio signal changes based on the semantic integrity of a sentence composed of multiple characters in the audio signal to obtain the speaker switching time point.

[0012] In a possible implementation, the speech-to-text recognition module includes: a streaming encoder that receives an input audio signal, encodes the acoustic features of each audio frame in the audio signal, and obtains acoustic coding features and corresponding probabilities corresponding to each audio frame;

[0013] The speaker identity information recognition module includes:

[0014] The non-streaming encoder concatenates the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to the sentences;

[0015] The speaker identity detector determines a speaker identity information sequence arranged in time sequence according to the acoustic coding features and corresponding probabilities corresponding to the sentences.

[0016] In a possible implementation, the speech-to-text recognition module includes: a predictor, for predicting the next character feature and the corresponding probability of the historical character feature based on the predicted historical character feature, and obtaining the current character feature and the corresponding probability;

[0017] The speaker switching time point recognition module determines whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent.

[0018] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point;

[0019] The integration module determines the text corresponding to each speaker identity information according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0020] In a possible implementation manner, the device further includes:

[0021] A re-scoring module, which adds speaker labels to the multiple candidate text sequences obtained by the speech text recognition module according to the speaker identity information sequence and the speaker switching time point, to obtain multiple candidate text sequences with speaker identity information;

[0022] The re-scoring module re-scores the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence;

[0023] The re-scoring module determines a target probability for each candidate text sequence according to the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module;

[0024] The re-scoring module determines a target text sequence with speaker identity information based on the target probability.

[0025] In a possible implementation, the re-scoring module determines the probability of each character and label in the candidate text sequence with speaker identity information, and adds the probability of each character and label in the same candidate text sequence to obtain a first probability of the candidate text sequence.

[0026] According to one aspect of the present disclosure, a method for training a speech recognition device is provided, for training any of the above-mentioned devices, the method comprising:

[0027] Using a speech data set containing speaker labels, a speaker identity information recognition module is trained, and the trained speaker recognition module is used to construct a multi-speaker speech recognition data set;

[0028] Use the speech dataset containing text labels to train the speech-to-text recognition module;

[0029] Keeping the parameters of the speech-to-text recognition module unchanged, using a multi-speaker speech recognition data set to train a speaker identity information recognition module and a speaker transition time point recognition module in the speech recognition device;

[0030] The speech recognition device is trained as a whole using a speech data set including speaker labels and text labels to obtain a trained speech recognition device.

[0031] In a possible implementation, the method further includes:

[0032] Add speaker identity information and speaker switching time points to the text obtained by speech recognition in the speech-to-text recognition module to construct a training set for the re-scoring module;

[0033] Using the speaker identity information and the speaker switching time point, construct a hybrid dictionary including the speaker identity information and the speaker switching time point as a dictionary of the re-scoring module;

[0034] The re-scoring module training set is input into the re-scoring module, and the re-scoring module is trained using the hybrid dictionary.

[0035] According to one aspect of the present disclosure, there is provided a speech recognition method, comprising:

[0036] Receive an input audio signal, and perform text recognition on the audio signal to obtain text in the audio signal and a corresponding timestamp;

[0037] Based on acoustic coding features in an audio signal, identifying speaker identity information in the audio signal to obtain a speaker identity information sequence arranged in time sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0038] Based on the semantic information of the text in the audio signal, identifying the time point at which the speaker in the audio signal changes, and obtaining the speaker switching time point;

[0039] Based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, the text corresponding to the speaker identity information is determined, and the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0040] In a possible implementation, identifying the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point includes:

[0041] Based on the semantic integrity of the sentence composed of multiple characters in the audio signal, the time point when the speaker in the audio signal changes is identified to obtain the speaker switching time point.

[0042] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0043] Receive an input audio signal, encode the acoustic features of each audio frame in the audio signal, and obtain the acoustic coding features and corresponding probabilities corresponding to each audio frame;

[0044] The method of identifying the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a speaker identity information sequence arranged in time sequence includes:

[0045] Concatenating the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to the sentences;

[0046] According to the acoustic coding features and corresponding probabilities corresponding to the sentences, a speaker identity information sequence arranged in time sequence is determined.

[0047] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0048] Based on the predicted historical character features, predict the next character feature of the historical character features and the corresponding probability, and obtain the current character feature and the corresponding probability;

[0049] The step of identifying the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point includes:

[0050] Based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent, it is determined whether the current character feature corresponds to the speaker switching time point.

[0051] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point;

[0052] The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, including:

[0053] The text corresponding to each speaker identity information is determined according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0054] In a possible implementation, the method further includes:

[0055] According to the speaker identity information sequence and the speaker switching time point, speaker labels are added to the multiple candidate text sequences obtained by the speech text recognition module to obtain multiple candidate text sequences with speaker identity information;

[0056] Re-scoring the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence;

[0057] Determine a target probability for each candidate text sequence according to the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module;

[0058] Based on the target probability, a target text sequence with speaker identity information is determined.

[0059] In a possible implementation, the rescoring operation on the candidate text sequence with speaker identity information to obtain a first probability of each candidate text sequence includes:

[0060] The probability of each character and label in the candidate text sequence with speaker identity information is determined, and the probability of each character and label in the same candidate text sequence is added to obtain a first probability of the candidate text sequence.

[0061] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0062] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0063] In the disclosed embodiment, the speech text recognition module receives the input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp; the speaker identity information recognition module recognizes the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a speaker identity information sequence arranged in time sequence, and the acoustic coding features are extracted based on the acoustic features in the audio signal; the speaker switching time point recognition module recognizes the time point when the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point; the integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point. Thus, the speaker identity information recognition module can recognize and distinguish different speakers according to the acoustic features in the audio signal, such as pitch, timbre, speech speed, etc.; at the same time, the speaker switching time point recognition module can use the semantic content changes of the text, such as topic switching, tone difference, etc., as auxiliary information to identify the speaker switching time point, thereby improving the detection accuracy of the speaker switching point. By fusing multi-source information, the speech recognition device can understand the audio content more comprehensively and accurately, thereby improving the accuracy of the text corresponding to the recognized speaker identity information.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0066] Figure 1A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown.

[0067] Figure 2 A schematic diagram of the structure of a speaker identity information recognition module provided by an embodiment of the present disclosure is shown.

[0068] Figure 3 A structural schematic diagram of a speech-to-text recognition module provided by an embodiment of the present disclosure is shown.

[0069] Figure 4 A structural diagram of a speaker identity information recognition module and a speaker transition time point recognition module provided by an embodiment of the present disclosure is shown.

[0070] Figure 5 A schematic diagram of the structure of an end-to-end speech recognition device provided by an embodiment of the present disclosure is shown.

[0071] Figure 6 A schematic diagram of the structure of a training architecture of a re-scoring module provided in an embodiment of the present disclosure is shown.

[0072] Figure 7 A flow chart of a speech recognition method according to an embodiment of the present disclosure is shown.

[0073] Figure 8 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0074] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0075] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0076] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0077] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0078] In related technologies, the identification of speaker identity information and the detection of speaker switching time points are still not accurate enough. Specifically, due to the differences in speech characteristics of different speakers, including pitch, speaking speed, pronunciation habits, etc., these differences need to be accurately captured and distinguished during the recognition process; on the other hand, the switching between speakers is often accompanied by changes in semantic content such as topic and tone, which requires a comprehensive judgment based on the characteristics of text information and audio signals.

[0079] In the disclosed embodiment, the speech text recognition module receives the input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp; the speaker identity information recognition module recognizes the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a speaker identity information sequence arranged in time sequence, and the acoustic coding features are extracted based on the acoustic features in the audio signal; the speaker switching time point recognition module recognizes the time point when the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point; the integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point. Thus, the speaker identity information recognition module can recognize and distinguish different speakers according to the acoustic features in the audio signal, such as pitch, timbre, speech speed, etc.; at the same time, the speaker switching time point recognition module can use the semantic content changes of the text, such as topic switching, tone difference, etc., as auxiliary information to identify the speaker switching time point, thereby improving the detection accuracy of the speaker switching point. By fusing multi-source information, the speech recognition device can understand the audio content more comprehensively and accurately, thereby improving the accuracy of the text corresponding to the recognized speaker identity information.

[0080] Figure 1 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the device comprises:

[0081] The speech-to-text recognition module 11 receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp;

[0082] A speaker identity information recognition module 12, based on acoustic coding features in the audio signal, identifies the speaker identity information in the audio signal to obtain a speaker identity information sequence arranged in time sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0083] A speaker switching time point identification module 13, based on the semantic information of the text in the audio signal, identifies the time point at which the speaker in the audio signal changes, and obtains the speaker switching time point;

[0084] The integration module 14 determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, where the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0085] The speech-to-text recognition module is used to convert the input audio signal into readable text information and mark the timestamp of each text segment. Exemplarily, the speech-to-text recognition module can be implemented using a recurrent neural network transducer (RNNT) structure, which combines the sequence modeling capability of a recurrent neural network (RNN) and the decoding efficiency of connection temporal classification (CTC), and can efficiently and accurately recognize text content from a continuous audio stream, and assign a timestamp to each recognized text segment, indicating the time position of the text in the audio.

[0086] For example, in an audio clip containing a conversation between multiple people, the speech-to-text recognition module can recognize text such as "Good morning, how's the weather today?" and mark which time periods in the audio these words correspond to (such as 00:00:00-00:00:04).

[0087] The speech-to-text recognition module recognizes the text in the audio signal and marks the timestamp, providing the required text and time information for the subsequent speaker identification and speaker switching time point identification. The audio signal can be streaming audio or non-streaming audio, which is not limited in this disclosure.

[0088] The speaker identity information recognition module is used to identify the speaker identity information from the audio signal based on the coded acoustic features in the audio signal, and form a speaker identity information sequence arranged in time sequence. The acoustic coding features are extracted based on the acoustic features in the audio signal, and the acoustic coding features can be obtained by encoding the acoustic features of each audio frame in the audio signal by the streaming encoder in the speech-to-text recognition module.

[0089] Acoustic features are physical properties in audio signals that can reflect the content of text and its pronunciation characteristics. Such features may include pitch, timbre, volume, duration, etc. Acoustic features are closely related to the pronunciation of the text, the timbre, intonation, and speaking speed of the speaker, and are an important basis for identifying the identity information of the speaker. For example, when identifying the word "hello", the pronunciation of different speakers may have different pitch and timbre characteristics, which will be used to distinguish the identity information of the speaker.

[0090] Speaker identity information is information used to identify an individual speaker. It can be a unique identifier or a set of multiple feature vectors. For example, in a conversation, it can be recognized that "this is speaker A speaking" or "this is speaker B's voice", and these identity information are arranged into a sequence in chronological order.

[0091] The time-sequential speaker identity information sequence arranges the identified speaker identity information into a sequence according to the order in which the speakers appear in the audio signal. This sequence not only contains the speaker's identity information, but also retains the time order of this information in the audio signal, which helps to accurately locate the speaker's speaking time point in subsequent processing. For example, in an audio clip containing a conversation between three people, the system can identify the identity information of speakers A, B, and C, and arrange them into a time sequence of "ABACB" according to the order in which they spoke.

[0092] The speaker change time point recognition module identifies the time point when the speaker in the audio changes based on the semantic information of the text in the audio signal. This module uses the semantic content difference detection method to determine when the speaker changes by analyzing the changes in the text content (such as topic changes, tone changes, etc.). This method not only takes into account the physical characteristics of the audio, but also incorporates semantic level analysis to improve the accuracy of recognition.

[0093] For example, in a conversation, when A says "I think the weather is nice today", and B then says "but I feel a little cold", the speaker transition time point recognition module can recognize the semantic change and use the time point of transition from A to B as the speaker transition time point.

[0094] The integration module integrates the recognized text, the corresponding timestamp, the speaker identity information sequence and the speaker switching time point to determine the text content corresponding to each speaker identity information.

[0095] Specifically, the audio is first segmented into different speaker segments according to the speaker switching time point, and then the text of each segment is assigned to the corresponding speaker in chronological order based on the speaker identity information sequence. At the same time, the timestamp corresponding to the text is used to ensure the accuracy of the assignment.

[0096] For example, in the previous example, the integration module would assign "Good morning, how's the weather today?" to A and "But I feel a little cold" to B, and keep the timestamp information of each text segment.

[0097] In a possible implementation, the speaker switching time point identification module identifies the time point at which the speaker in the audio signal changes based on the semantic integrity of a sentence composed of multiple characters in the audio signal to obtain the speaker switching time point.

[0098] The speaker switching time point refers to the specific moment when one speaker changes to another in a continuous audio signal. This time point marks the change in the identity of the speaker in the audio and is used to distinguish the speech segments of different speakers. For example, in an audio segment containing two people talking, when speaker A finishes speaking and speaker B starts speaking, the moment of handover is the speaker switching time point.

[0099] Semantic integrity refers to the completeness and coherence of a sentence or text in expressing its meaning. In speech recognition, the judgment of semantic integrity can usually be based on whether the information conveyed by the sentence is complete and clear, and whether the logical relationship between the parts of the sentence is tight. A semantically complete sentence can clearly express a complete thought or information.

[0100] For example, the sentence "The weather is nice today, let's go for a walk in the park" is a semantically complete sentence, which contains complete information such as time (today), weather (nice), action (go for a walk in the park), etc., and the logic between these information is clear and coherent.

[0101] Therefore, the speaker switch can be inferred by analyzing the semantic integrity at the sentence level. When it is determined that the semantics of a sentence is no longer continuous with the semantics of the previous sentence, it is determined that the speaker has switched.

[0102] In the disclosed embodiment, by introducing the concept of semantic integrity, the recognition accuracy of the speaker switching time point is improved. Compared with the method that only relies on acoustic features, this method can more comprehensively consider the context information and semantic coherence of the conversation, thereby more accurately judging the speaker's switching.

[0103] In a possible implementation, the speech-to-text recognition module includes: a streaming encoder that receives an input audio signal, encodes the acoustic features of each audio frame in the audio signal, and obtains the acoustic coding features and corresponding probabilities corresponding to each audio frame; the speaker identity information recognition module includes: a non-streaming encoder that concatenates the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to each sentence; and a speaker identity detector that determines a speaker identity information sequence arranged in time sequence according to the acoustic coding features and corresponding probabilities corresponding to the sentences.

[0104] A streaming encoder can be an encoder that processes audio signals in real time and can encode continuously input audio signals. A streaming encoder can divide continuous audio signals into multiple shorter frames through frame processing technology, and encode the acoustic features of each frame independently to obtain acoustic coding features and corresponding probabilities. This can achieve instant encoding of audio signals, thereby quickly obtaining recognition results.

[0105] The streaming encoder in the embodiment of the present disclosure can be an encoder in the RNNT model. The streaming encoder receives the audio signal input by the user and converts it into a series of feature vectors (such as Mel-frequency cepstral coefficient MFCC features), and assigns one or more possible characters and their corresponding probabilities to each feature vector.

[0106] The non-streaming encoder does not process the audio signal in real time, but waits for a certain amount of audio signal (such as a complete sentence) to be input before encoding the whole thing. For example, when a period of silence is detected, the whole thing can be encoded. It can capture the long-term dependencies in the audio signal, thereby more accurately extracting the semantic information of the sentence. In the speech recognition device, the non-streaming encoder can generate sentence-level feature representations.

[0107] The non-streaming encoder receives the sequence of acoustic coding features output by the streaming encoder, and performs operations such as splicing and fusion on these features to form the acoustic coding features corresponding to the complete sentence. The acoustic coding features corresponding to the sentence not only contain the recognition information of a single character, but also the contextual relationship between characters and the semantic information of the entire sentence.

[0108] The speaker identity detector receives the acoustic coding features and corresponding probabilities corresponding to the sentences output by the non-streaming encoder to determine the identity information of the current speaker and generate a time-series sequence of speaker identity information.

[0109] For example, in the process of speech recognition, the streaming encoder receives the audio signal of the word "hello" said by the user and divides it into multiple frames. Then, MFCC and other features are extracted for each frame, and the most likely acoustic coding features and their corresponding probabilities are obtained by comparing with the preset character template; then the non-streaming encoder splices the acoustic coding features to obtain the acoustic coding features and corresponding probabilities corresponding to multiple sentences, and the non-streaming encoder receives the acoustic coding features and their corresponding probabilities output by the streaming encoder. Then, these character features are spliced ​​and fused to form a complete sentence feature vector, and the corresponding probability value is generated based on the probability output by the streaming encoder to represent the probability of the entire sentence feature vector. Finally, the speaker identity detector determines the speaker identity information sequence arranged in time sequence according to the multiple sentence features and the corresponding probabilities.

[0110] For a single audio frame, the streaming encoder may output multiple different acoustic coding features and corresponding probabilities, that is, obtain the probability distribution of multiple acoustic coding features; then, when splicing and fusing the acoustic coding features of multiple audio frames, the multiple acoustic coding features of different audio frames can be arranged and combined, and each combination is the acoustic coding feature of a sentence. By multiplying or adding the probabilities in each combination, the probability of the acoustic coding feature corresponding to each sentence can be obtained.

[0111] In the disclosed embodiment, the acoustic features of each audio frame in the audio signal are encoded in real time by a streaming encoder to obtain acoustic coding features and corresponding probabilities corresponding to each audio frame, and then the acoustic coding features are spliced ​​by a non-streaming encoder to obtain acoustic coding features and corresponding probabilities corresponding to multiple sentences to improve recognition accuracy, and combined with a speaker identity detector to determine a time-series speaker identity information sequence based on the acoustic coding features and corresponding probabilities corresponding to each sentence, the speaker identity information in the audio can be efficiently and accurately identified, and a time-series sequence can be generated, which significantly improves the real-time and accuracy of speaker identity recognition.

[0112] In one possible implementation, the speech-to-text recognition module includes: a predictor for predicting the next character feature of the historical character feature and the corresponding probability based on the predicted historical character feature, and obtaining the current character feature and the corresponding probability; and a speaker switching time point recognition module for determining whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence composed of the historical character feature and the current character feature with the highest probability are coherent.

[0113] The predictor can predict the future (i.e., the next) character feature and its corresponding probability based on the predicted historical character features. By leveraging the context understanding ability of the deep learning model, the predictor analyzes the previously recognized character sequence (i.e., the historical character features) to predict the next possible characters and the probabilities of their occurrences. This helps to more accurately understand the semantic content represented by the current audio signal and improve the recognition accuracy of the speaker transition time point.

[0114] For example, assume the user is saying "What's the weather like tomorrow". After the predictor recognizes "tomorrow", based on the semantics and context information of these two characters, it predicts that the next character is likely to be a word related to "weather", such as "天", "气", or "的", etc., and gives the probabilities of these predicted characters respectively. In this way, in subsequent processing, the system can search and match possible character sequences more targeted.

[0115] The speaker change detection (SCD detect) module can be a binary classifier that can determine whether a speaker change has occurred at the current moment based on the historical character features, the current character features, and their corresponding probabilities. Specifically, this module determines whether the current character features correspond to the speaker transition time point based on whether the semantics of the sentence formed by the historical character features and the current character features with the highest probability is coherent. If the semantics is coherent, it is determined that the current character features do not correspond to the speaker transition time point; if the semantics is not coherent, it is determined that the current character features correspond to the speaker transition time point. For example, the start time point of the current character is taken as a speaker transition time point.

[0116] For example, assume the user is saying "What's the weather like tomorrow". After the predictor recognizes "明天天", the predictor continues to output the current character with the highest probability as "气". Since the semantics of "明天天" + "气" is coherent, it does not correspond to the speaker transition time point. While assume the historical character features are "What's the weather like tomorrow" and the recognized current character is "明". Since the semantics of the sentence formed by "What's the weather like tomorrow" and "明" is not coherent, it is determined that the current character "明" corresponds to the speaker transition time point, and the start time point of the current character "明" can be taken as a speaker transition time point.

[0117] The speaker change detection module can be implemented using machine learning or deep learning algorithms, which is not limited in this disclosure.

[0118] In the speech-to-text recognition module provided by the present disclosure, the input audio signal is processed by a streaming encoder to generate a sequence of acoustic coding features; then the predictor and the jointer work together to gradually generate an output character sequence. The predictor is responsible for predicting the next possible character based on the historical output token i of the jointer, while the jointer determines the final output character token i+1 based on the current acoustic coding features output by the streaming encoder and the output of the predictor.

[0119] In the disclosed embodiment, the predictor predicts the next character feature of the historical character feature and the corresponding probability based on the predicted historical character feature, and obtains the current character feature and the corresponding probability; the speaker switching time point recognition module determines whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence composed of the historical character feature and the current character feature with the highest probability are coherent. Thus, the predictor can enhance the system's adaptability to complex contexts by predicting future character features and their probabilities, thereby combining the semantic information of the text; further, the speaker switching time point recognition module can more accurately recognize speaker switching in multi-person conversations based on whether the semantics of the sentence composed of the historical character feature and the current character feature with the highest probability are coherent.

[0120] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point; the integration module determines the text corresponding to each speaker identity information according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0121] The integration module first analyzes the speaker identity information sequence and speaker transition time points to identify the specific time points when each speaker starts and ends speaking. In this way, the audio signal can be accurately divided into multiple independent speaker speaking time periods, each of which corresponds to a specific speaker.

[0122] Then, the integration module uses the text output by the text recognition module and its corresponding timestamp to match the recognized text content with the previously determined speaker speech time period. Specifically, the corresponding text content can be intercepted at the timestamp of the text segment according to the speech time period of each speaker, and the text segment can be assigned to the speaker.

[0123] In this way, a structured conversation record can be constructed, which includes the identity information of each speaker, the speaking time and the corresponding text content.

[0124] Suppose there is an audio file that records the conversation between three people (speaker A, speaker B, and speaker C) in a meeting. The content of the audio file is roughly as follows:

[0125] Speaker A spoke first: "Hello everyone, let's discuss the progress of the project today."

[0126] Then, speaker B took over the topic: "Yes, I have completed my part, but I encountered some problems."

[0127] Then, speaker A spoke again: "Oh, what are the specific questions?"

[0128] Then, speaker C joins the discussion: "I've noticed this problem, too. Maybe we can solve it this way..."

[0129] The specific workflow of the integration module is as follows:

[0130] First, analyze the speaker identity information and transition time points: receive the identity information sequence output by the speaker identity information recognition module, such as "[A,A,B,A,C]", which means that the speeches of speakers A, A, B, A, and C appear in the audio in sequence.

[0131] At the same time, the output of the speaker transition time point recognition module is also received, which marks the moment of transition between speakers. For example, time point T1 indicates the transition from speaker A to speaker B, time point T2 indicates the transition from speaker B back to speaker A, and time point T3 indicates that speaker C joins the discussion.

[0132] With this information, the integration module can identify the specific time points when each speaker starts speaking (such as A at the beginning of the audio, B after T1, and C after T3) and ends speaking (such as A before T1 and after T2, B ends before T2, and C ends at the end of the audio or before the next transition point).

[0133] The text recognition module has processed the audio and output text snippets with timestamps, such as "[00:00-00:05,'Hi everyone, let's discuss the progress of the project today.'], [00:06-00:12,'Yes, I have completed my part, but encountered some problems.']" and so on.

[0134] The integration module intercepts and matches the corresponding text according to the speech time period of each speaker and the timestamp of the text segment. For example, the timestamp of the text segment "[00:00-00:05,...]" falls completely within the first speech time period of speaker A, so the text can be assigned to speaker A.

[0135] Similarly, "[00:06-00:12,...]" is assigned to speaker B, and subsequent text segments matching the speaking time periods of speakers A and C are assigned to the two speakers respectively.

[0136] Constructing structured conversation records:

[0137] After the above steps, the integration module constructs a structured conversation record as follows:

[0138] Speaker A: 00:00-00:05,"Hello everyone, today we are going to discuss the progress of the project."

[0139] Speaker B: 00:06-00:12,"Yes, I have completed my part but I have encountered some problems."

[0140] Speaker A: 00:13-00:15,"Oh, what are the specific problems?"

[0141] Speaker C: 00:16-...,"I've noticed this problem too, maybe we can solve it this way..."

[0142] This record clearly shows the identity information, speaking time and corresponding text content of each speaker, which is convenient for subsequent analysis and processing.

[0143] In addition, content from a specific speaker can be identified by pre-registering the target speaker.

[0144] Specifically, speaker registration can be performed in advance by collecting multiple audio signals of the target speaker, and then using a pre-trained speaker identity information recognition module to extract the speaker embeddings (Speaker Embedding) of these audios, and calculating the average or weighted average of these embeddings as the representative vector (Target Speaker Embedding) of the target speaker.

[0145] Then, the audio to be recognized is subjected to speech recognition to obtain the text content, and at the same time, the speaker identity information recognition module is used to extract speaker embeddings for each speaking segment in the audio. These embeddings are compared with the representative vector of the target speaker and the similarity (such as cosine similarity) is calculated.

[0146] Based on the similarity score, the speech segment that is closest to the target speaker is selected. If the similarity exceeds the preset threshold, the speech segment is considered to belong to the target speaker and its corresponding text content is extracted. The identified target speaker content is then displayed to the user or sent to downstream tasks (such as sentiment analysis, intent recognition, etc.) for processing.

[0147] In the disclosed embodiment, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point; the integration module determines the text corresponding to each speaker identity information according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp. In this way, the speaker's speech paragraphs can be accurately divided and the corresponding text content can be matched.

[0148] In a possible implementation, the device also includes: a re-scoring module, which adds speaker labels to multiple candidate text sequences obtained by the speech-to-text recognition module according to the speaker identity information sequence and the speaker switching time point, so as to obtain multiple candidate text sequences with speaker identity information; the re-scoring module re-scores the candidate text sequences with speaker identity information to obtain a first probability for each candidate text sequence; the re-scoring module determines a target probability for each candidate text sequence according to the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module; and the re-scoring module determines a target text sequence with speaker identity information based on the target probability.

[0149] The re-scoring module can add speaker labels to multiple candidate text sequences output by the speech text recognition module based on the speaker identity information sequence output by the speaker identity information recognition module and the speaker switching time point output by the speaker switching time point recognition module, and re-score based on these candidate text sequences with speaker identity information, and finally determine the target text sequence with speaker identity information. For the specific process of adding labels, please refer to the relevant process of the integration module "determining the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point" in the embodiment of the present disclosure, which will not be repeated here.

[0150] The re-scoring module can receive multiple candidate text sequences output by the speech-to-text recognition module, as well as corresponding speaker identity information and speaker switching time points, and then construct a candidate text sequence with speaker identity information.

[0151] Specifically, among the multiple candidate text sequences output by the speech-to-text recognition module, a corresponding speaker label (such as <spk1> , <spk2>etc.), generate multiple candidate text sequences with speaker identity information. For example, if the original candidate text sequence is "XXX XXXXXX", and the speaker transition points are identified between "XXX" and "XXXX", and the identified speakers are SPK1 and SPK2 respectively, then the processed sequence is " <spk1>XXX <spk2>XXXX <spk1>XX". In addition, if this sentence is the beginning of a sentence, you can also add <sos>Tags, get " <sos> <spk1>XXX <spk2>XXXX <spk1>XX". According to this method, multiple candidate text sequences can be obtained for multiple candidate text sequences output by the speech-to-text recognition module.

[0152] Then, the multiple candidate text sequences are re-scored to obtain the first probability of each candidate text sequence. The re-scoring module has learned to score by combining the speaker identity information, the speaker transition point information and the semantic information of the text sequence during the training phase. The specific training process can refer to the training method provided in the present disclosure, which will not be described in detail here.

[0153] At the same time, combined with the second probability originally calculated by the speech-to-text recognition module for each candidate text sequence, the target probability of each candidate text sequence is calculated by weighting and other methods, and then based on the calculated target probability, the candidate text sequence with the highest probability is selected as the final target text sequence with speaker identity information, and finally the target text sequence with speaker identity information is output for subsequent processing or display.

[0154] For example, the re-scoring module can calculate the target probability of each candidate text sequence according to a certain weight ratio, and select the candidate text sequence with the highest target probability as the final output result. The calculation process can be expressed as formula (1).

[0155] Target probability = k1*second probability + (1-k1)*first probability (1)

[0156] Among them, k1 is the weight coefficient.

[0157] In the disclosed embodiment, among the multiple candidate text sequences output by the speech text recognition module, the re-scoring module first adds a speaker label at an appropriate position of each candidate text sequence according to the speaker identity information and the speaker switching time point. Then, the re-scoring module re-scores the candidate text sequence with the speaker label added, and evaluates its rationality and accuracy by calculating the sequence probability (i.e., the first probability) of the candidate text sequence. Since the re-scoring module has learned the speaker identity information, the semantic information of the text sequence, and the characteristics of the speaker switching point during the training process, it can more accurately evaluate the quality of the candidate text sequence. After obtaining the first probability of each candidate text sequence, the re-scoring module also considers the second probability of each candidate text sequence directly output by the speech text recognition module, and combines the first probability and the second probability in a weighted manner to obtain the final target probability of each candidate text sequence. According to the target probability, the target text sequence with the speaker identity information is determined and output. Since the target probability comprehensively considers the direct output of the speech text recognition module and the evaluation result of the re-scoring module, it can more accurately reflect the actual quality of the candidate text sequence, thereby improving the accuracy of the final recognition result.

[0158] In a possible implementation, the re-scoring module determines the probability of each character and label in the candidate text sequence with speaker identity information, and adds the probability of each character and label in the same candidate text sequence to obtain a first probability of the candidate text sequence.

[0159] For candidate text sequences with speaker identity information, the probability of each word and label in the sequence can be calculated and accumulated to obtain the first probability of the entire candidate text sequence. By comprehensively considering the probability information of all elements in the sequence (including words and labels), the overall reliability of the candidate text sequence can be more accurately evaluated, thereby improving the accuracy and robustness of the speech recognition system.

[0160] During the rescoring process, the rescoring module can traverse each word and label in the candidate text sequence and calculate their probabilities respectively. Then, these probability values ​​are accumulated to obtain the first probability of the entire candidate text sequence. The uncertainty of all elements in the sequence is comprehensively considered by accumulation, so as to more comprehensively evaluate the reliability of the candidate sequence.

[0161] Since the candidate text sequence contains speaker identity information (such as speaker tags), the rescoring module can consider acoustic features, semantic features, and speaker features simultaneously when calculating the probability. The fusion of multi-source information can more accurately identify the target speaker and what he or she said, improving the personalized experience and security of the system.

[0162] For example, when a candidate text sequence obtained by the speech-to-text recognition module is " <spk1>Good morning", during the rescoring phase, the sequence will be processed into a format that matches the format used when the rescoring module was trained. <sos> <spk1>Good morning <eos>”. Then, the rescoring module receives this processed sequence as input and calculates <sos> 、 <spk1>, "Good morning" and <eos>Specifically:

[0163] <sos>The probability of: as the start mark of the sequence, <sos>The probability reflects the reliability of the initial part of the sequence.

[0164] <spk1>Probability: Speaker label <spk1>The probability of represents the recognition confidence of the speaker's identity.

[0165] The probability of "good morning" is the probability of the actual spoken content, which represents the recognition accuracy of this content.

[0166] <eos>The probability of: as the end mark of the sequence, used to evaluate the integrity of the entire sequence.

[0167] After obtaining these probabilities, the rescoring module will add them together to obtain the entire candidate text sequence" <sos> <spk1>Good morning <eos>” first probability.

[0168] In addition, the present disclosure also provides a method for training a speech recognition device, which is used to train any speech recognition device provided by the present disclosure, and the method comprises:

[0169] In step S21, a speaker identity information recognition module is trained using a speech data set containing speaker labels, and a multi-speaker speech recognition data set is constructed using the trained speaker recognition module;

[0170] In step S22, the speech-to-text recognition module is trained using the speech data set containing the text labels;

[0171] In step S23, the parameters of the speech text recognition module are kept unchanged, and the speaker identity information recognition module and the speaker transition time point recognition module in the speech recognition device are trained using the multi-speaker speech recognition data set;

[0172] In step S24, the speech recognition device is trained as a whole using the speech data set including the speaker label and the text label to obtain a trained speech recognition device.

[0173] The speaker identification module is responsible for identifying the speaker from the speech data. By training with a speech data set labeled with speaker labels, the speaker identification module can learn the speech characteristics of different speakers and accurately distinguish different speakers. After training, the speaker identification module can be used to construct a data set containing the speech of multiple speakers.

[0174] Figure 2 A schematic diagram of the structure of a speaker identity information recognition module provided by an embodiment of the present disclosure is shown. The module can be trained using a triplet loss method. Afterwards, the trained speaker identity information recognition module is used to perform data expansion on a normal speech recognition data set, and each voice data is labeled with a corresponding speaker label, thereby forming a multi-speaker speech recognition data set. For example, the data set U = {(x1, y1), (x2, y2) ...} (where x is audio and y is a text label) is expanded to U + s = {(x1, y1, s1), (x2, y2, s2) ...}, where s is the corresponding speaker label. Then, long audios of multiple speakers are obtained by random splicing. For example, after the audios (x1, y1, s1) and (x2, y2, s2) are randomly selected, they can be spliced ​​to obtain audios (Xlong1, Ylong1, Slong1), where Xlong is the spliced ​​audio and Ylong is in the form of y1 <scd>y2, where <scd>Indicates the speaker switching point, Slong = {(s1, start1, end1), (s2, start2, end2)} represents the starting point (start1, start2) and end point (end1, end2) of the audio corresponding to different speakers.

[0175] Then, the speech-to-text recognition module is trained using the speech data set containing text labels. The speech-to-text recognition module can convert the input audio signal into the corresponding text information. By training with the speech data set annotated with text labels, the speech-to-text recognition module can learn the mapping relationship between speech and text, thereby improving the accuracy and efficiency of recognition.

[0176] Figure 3 A structural schematic diagram of a speech text recognition module provided by an embodiment of the present disclosure is shown. The module includes components such as a streaming encoder, a predictor, and a joiner, wherein the streaming encoder processes the input audio signal and generates a sequence of acoustic coding features; then the predictor is responsible for predicting the next possible character based on the historical output token i of the joiner, and the joiner determines the final output character token i+1 based on the current acoustic coding features output by the streaming encoder and the output of the predictor.

[0177] In order to further improve the adaptability of the speech recognition device to multi-speaker scenarios, the speaker identity information recognition module and the speaker transition time point recognition module may be further optimized. Figure 4 The structure diagram of a speaker identity information recognition module and a speaker switching time point recognition module provided by the embodiment of the present disclosure is shown. When training the speaker identity information recognition module and the speaker switching time point recognition module, the parameters of the speech text recognition module can be fixed unchanged, and then joint training can be performed through a speech data set containing multiple speaker labels, so that the switching mode between different speakers and the time point of speaker switching can be learned, thereby improving the overall performance of the system.

[0178] Specifically, a multi-speaker speech recognition dataset or a dataset constructed by a pre-trained speaker recognition module is used to generate long audio signals (Xlong1, Ylong1, Slong1) containing multiple speakers. Xlong1 is the concatenated long audio, Ylong1 is the corresponding text label, and <scd>To indicate the speaker switching point (such as y1 <scd>y2), Slong1 contains the start and end information of the audio corresponding to different speakers, such as {(s1,start1,end1),(s2,start2,end2)}.

[0179] At the beginning of joint training, the parameters of the streaming encoder, predictor, and aggregator are fixed. These modules only perform forward reasoning during training and do not participate in parameter updating.

[0180] Input the long audio Xlong into the streaming encoder, and the acoustic coding features of each audio frame output by it will accumulate over time. When the accumulation reaches a certain length or reaches a certain preset condition, the accumulated acoustic coding features are sent to the non-streaming encoder for further processing.

[0181] The output of the non-streaming encoder is input to the speaker identification module for speaker identification training. The speaker label is obtained through the timestamp information in Slong (start1, end1, start2, end2, etc.). For example, if the current time frame t falls within the time range of a certain speaker (start2 <= t <= end2), the label of the time frame is s2.

[0182] At the same time, the predictor predicts the speaker transition point for the input token (predicted by the previous token output by the fusion). <scd>appears to determine whether a speaker switch has occurred. For example, in the text "Hello <scd>Good morning", the predictor will predict the corresponding SCD label "blank" based on the input "hello good morning". <scd>blank blank blank", thereby determining the location of the speaker's switching point.

[0183] During the joint training process, the total loss function is the weighted sum of the speaker recognition loss (loss_spk_identify) and the speaker transition point prediction loss (loss_spk_scd), that is, the total loss total_loss = w1*loss_spk_identify+w2*loss_spk_scd, where w1 and w2 are weights.

[0184] At the beginning of training, the parameters of the streaming encoder, predictor, and amalgamator are fixed, and only the speaker identity information recognition module and the speaker transition point prediction module are trained. After these two modules are trained to a certain extent, all modules are released for overall joint training. At this time, the total loss function also includes the loss of the speech text recognition module itself (loss_rnnt). That is, the total loss loss = w1*loss_spk_identify+w2*loss_spk_scd+loss_rnnt.

[0185] Through the above joint training process, the tasks of streaming speech recognition, speaker recognition and speaker transition point prediction can be completed simultaneously, and an end-to-end target speaker speech recognition device can be realized. This method can improve the stability and efficiency of training by fixing the parameters of the speech text recognition module and then training other modules. The training process makes full use of acoustic information and semantic information, thereby improving the accuracy of the speech recognition device.

[0186] After the above training, we can construct Figure 5 The end-to-end speech recognition device shown, Figure 5 The schematic diagram of the structure of an end-to-end speech recognition device provided by the embodiment of the present disclosure is shown. The functions and training process of each module can be found in the relevant description above, which will not be repeated here.

[0187] In a possible implementation, the method further includes: adding speaker identity information and speaker switching time points to the text obtained by speech recognition by the speech-to-text recognition module to construct a re-scoring module training set; using the speaker identity information and speaker switching time points to construct a hybrid dictionary containing the speaker identity information and speaker switching time points as a dictionary for the re-scoring module; inputting the re-scoring module training set into the re-scoring module, and using the hybrid dictionary to train the re-scoring module.

[0188] In this implementation, the re-scoring module will be trained. During the recognition process of the audio signal by the speech-to-text recognition module, a series of recognition results will be obtained, which are generated in real time in units of characters or words. After the entire speech input is completed, these results will be merged into a complete sentence. After the complete sentence is merged, multiple alternative paths (or candidate sentences) are usually retained. These paths are possible recognition results. These alternative paths can provide more options for the re-scoring process, so as to find the optimal recognition result.

[0189] In order to train the re-scoring module, clips can be randomly extracted from long audio, and the long audio can be used to construct a training set containing "speaker identity information, speaker transition point information, and speech content".

[0190] In the process of constructing the training set, special labels (such as <spk1> , <spk2>etc.) to indicate the speaker identity and the start and end of a sentence, and insert these tags into the text decoded by the speech-to-text recognition module. For example, if a random audio clip is taken from a long audio clip, and the conversation in the audio clip contains two speakers, the decoded text after inserting the tags may be as follows: <sos> <spk1>Good morning <spk2>Do you eat well? <spk1>have eaten <eos>.here, <sos>and <eos>Indicates the beginning and end of a sentence, respectively. <spkj>(where j is the speaker number) is used to mark speaker transitions. <sos> <spk1>Good morning <spk2>Do you eat well? <spk1>have eaten <eos>" is the expected output of the corresponding audio clip.

[0191] In order to train the rescoring module, these special tags and the speech recognition decoding vocab (i.e., dictionary, which contains all possible recognized characters or words) need to be combined into a mixture-vocab. This mixture-vocab contains not only language vocabulary but also special tags indicating the identity of the speaker, so that the rescoring module can recognize both the text content and the speaker identity information during decoding.

[0192] After preparing the training set and the mixed dictionary, you can start training the re-scoring module. Figure 6 A schematic diagram of the structure of a training architecture of a re-scoring module provided in an embodiment of the present disclosure is shown. The re-scoring module is trained based on the training architecture. It should be noted that the training architecture may also include other parts not shown, such as Figure 5 Related modules in .

[0193] During the training process, the streaming encoder and non-streaming encoder of the previously trained speech-to-text recognition module are used as feature extractors, and the parameters of these parts are fixed. Only the re-scoring module (including encoder and decoder) is trained. This allows the good feature representation that has been learned to be utilized while focusing on optimizing the performance of the re-scoring module. Input the audio clip into Figure 6 In the streaming encoder shown, feature encoding is performed through the streaming encoder and the non-streaming encoder to obtain sentence features and corresponding probabilities, which are then input into the re-scoring module. The re-scoring module obtains the predicted output, and the loss is calculated using the expected output and the predicted output to update the parameters of the re-scoring module.

[0194] In one possible implementation, the speech recognition device can be implemented by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The device can be implemented by a processor calling computer-readable instructions stored in a memory.

[0195] In addition, the present disclosure also provides a speech recognition method, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any speech recognition device provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0196] Figure 7 A flow chart of a speech recognition method according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the method includes:

[0197] In step S21, an input audio signal is received, and text recognition is performed on the audio signal to obtain text in the audio signal and a corresponding timestamp;

[0198] In step S22, based on acoustic coding features in the audio signal, speaker identity information in the audio signal is identified to obtain a speaker identity information sequence arranged in time sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0199] In step S23, based on the semantic information of the text in the audio signal, the time point at which the speaker in the audio signal changes is identified to obtain the speaker switching time point;

[0200] In step S24, based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, the text corresponding to the speaker identity information is determined, and the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0201] In a possible implementation, identifying the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point includes:

[0202] Based on the semantic integrity of the sentence composed of multiple characters in the audio signal, the time point when the speaker in the audio signal changes is identified to obtain the speaker switching time point.

[0203] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0204] Receive an input audio signal, encode the acoustic features of each audio frame in the audio signal, and obtain the acoustic coding features and corresponding probabilities corresponding to each audio frame;

[0205] The method of identifying the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a speaker identity information sequence arranged in time sequence includes:

[0206] Concatenating the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to the sentences;

[0207] According to the acoustic coding features and corresponding probabilities corresponding to the sentences, a speaker identity information sequence arranged in time sequence is determined.

[0208] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0209] Based on the predicted historical character features, predict the next character feature of the historical character features and the corresponding probability, and obtain the current character feature and the corresponding probability;

[0210] The step of identifying the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point includes:

[0211] Based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent, it is determined whether the current character feature corresponds to the speaker switching time point.

[0212] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point;

[0213] The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, including:

[0214] The text corresponding to each speaker identity information is determined according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0215] In a possible implementation, the method further includes:

[0216] According to the speaker identity information sequence and the speaker switching time point, speaker labels are added to the multiple candidate text sequences obtained by the speech text recognition module to obtain multiple candidate text sequences with speaker identity information;

[0217] Re-scoring the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence;

[0218] Determine a target probability for each candidate text sequence according to the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module;

[0219] Based on the target probability, a target text sequence with speaker identity information is determined.

[0220] In a possible implementation, the rescoring operation on the candidate text sequence with speaker identity information to obtain a first probability of each candidate text sequence includes:

[0221] The probability of each character and label in the candidate text sequence with speaker identity information is determined, and the probability of each character and label in the same candidate text sequence is added to obtain a first probability of the candidate text sequence.

[0222] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0223] In some embodiments, the functions or modules included in the method provided by the embodiments of the present disclosure can be used to execute the device described in the above method embodiments. The specific implementation can refer to the description of the above device embodiments, and for the sake of brevity, it will not be repeated here.

[0224] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0225] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0226] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0227] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0228] Figure 8 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0229] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0230] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0231] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0232] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0233] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0234] The computer program instructions for performing the disclosed operation may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0235] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0236] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0237] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0238] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0239] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0240] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0241] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0242] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0243] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.< / eos> < / sos> < / spkj> < / eos> < / sos> < / eos> < / sos> < / spk1> < / scd> < / scd> < / scd> < / scd> < / scd> < / scd> < / scd> < / eos> < / sos> < / eos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / sos> < / spk1>

Claims

1. A speech recognition device, characterized in that: include: A speech-to-text recognition module receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and a corresponding timestamp; A speaker identity information recognition module, based on acoustic coding features in the audio signal, identifies the speaker identity information in the audio signal to obtain a speaker identity information sequence arranged in time sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal; A speaker switching time point recognition module, which identifies the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal, and obtains the speaker switching time point; The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, wherein the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

2. The device according to claim 1, characterized in that The speaker switching time point identification module identifies the time point at which the speaker in the audio signal changes based on the semantic integrity of the sentence composed of multiple characters in the audio signal to obtain the speaker switching time point.

3. The device according to claim 1, characterized in that The speech-to-text recognition module comprises: a streaming encoder, which receives an input audio signal, encodes the acoustic features of each audio frame in the audio signal, and obtains the acoustic coding features and corresponding probabilities corresponding to each audio frame; The speaker identity information recognition module includes: The non-streaming encoder concatenates the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to the sentences; The speaker identity detector determines a speaker identity information sequence arranged in time sequence according to the acoustic coding features and corresponding probabilities corresponding to the sentences.

4. The device according to claim 1, characterized in that The speech-to-text recognition module includes: a predictor for predicting the next character feature and the corresponding probability of the historical character feature based on the predicted historical character feature, and obtaining the current character feature and the corresponding probability; The speaker switching time point recognition module determines whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent.

5. The device according to claim 1, characterized in that The integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point; The integration module determines the text corresponding to each speaker identity information according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

6. The device according to claim 1, characterized in that The device also includes: A re-scoring module, which adds speaker labels to the multiple candidate text sequences obtained by the speech text recognition module according to the speaker identity information sequence and the speaker switching time point, to obtain multiple candidate text sequences with speaker identity information; The re-scoring module re-scores the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence; The re-scoring module determines a target probability for each candidate text sequence according to the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module; The re-scoring module determines a target text sequence with speaker identity information based on the target probability.

7. The device according to claim 6, characterized in that The re-scoring module determines the probability of each character and label in the candidate text sequence with speaker identity information, and adds the probability of each character and label in the same candidate text sequence to obtain a first probability of the candidate text sequence.

8. A method for training a speech recognition device, characterized in that: For training the device according to any one of claims 1 to 7, the method comprising: Using a speech data set containing speaker labels, a speaker identity information recognition module is trained, and the trained speaker recognition module is used to construct a multi-speaker speech recognition data set; Use the speech dataset containing text labels to train the speech-to-text recognition module; Keeping the parameters of the speech-to-text recognition module unchanged, using a multi-speaker speech recognition data set to train a speaker identity information recognition module and a speaker transition time point recognition module in the speech recognition device; The speech recognition device is trained as a whole using a speech data set including speaker labels and text labels to obtain a trained speech recognition device.

9. The method according to claim 8, characterized in that The method further comprises: Add speaker identity information and speaker switching time points to the text obtained by speech recognition in the speech-to-text recognition module to construct a training set for the re-scoring module; Using the speaker identity information and the speaker switching time point, construct a hybrid dictionary including the speaker identity information and the speaker switching time point as a dictionary of the re-scoring module; The re-scoring module training set is input into the re-scoring module, and the re-scoring module is trained using the hybrid dictionary.

10. A speech recognition method, characterized in that: include: Receive an input audio signal, and perform text recognition on the audio signal to obtain text in the audio signal and a corresponding timestamp; Based on acoustic coding features in an audio signal, identifying speaker identity information in the audio signal to obtain a speaker identity information sequence arranged in time sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal; Based on the semantic information of the text in the audio signal, identifying the time point at which the speaker in the audio signal changes, and obtaining the speaker switching time point; Based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, the text corresponding to the speaker identity information is determined, and the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to implement the device described in any one of claims 1 to 7.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the apparatus according to any one of claims 1 to 7 is implemented.

Images