Speech recognition device and training method, electronic device and storage medium

Abstract

The present disclosure relates to a speech recognition device and training method, electronic device, and storage medium. The device includes: a speech-to-text recognition module that receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp; a speaker identity information recognition module that identifies the speaker identity information in the audio signal based on acoustic coding features in the audio signal to obtain a time-sequential speaker identity information sequence; a speaker transition time point recognition module that identifies the time point when the speaker in the audio signal changes based on semantic information of the text in the audio signal to obtain the speaker transition time point; and an integration module that determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker transition time point. Embodiments of the present disclosure can improve the accuracy of speaker-to-text recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech recognition device and method, an electronic device, a storage medium, and a training method for a speech recognition device. Background Art

[0002] With the rapid development of artificial intelligence (AI), speech recognition technology, as a crucial component of human-computer interaction, has been widely adopted in smartphones, smart homes, in-vehicle systems, and various other smart devices. Traditional speech recognition systems primarily focus on converting audio signals into text. However, in practical applications, especially in scenarios involving multi-person conversations, simply recognizing text content often falls short.

[0003] Therefore, target-speaker speech recognition systems have emerged. These systems can identify the speaker's identity and segment speech into segments based on text recognition. Existing speech recognition technologies, while capable of text recognition of audio signals, are still inaccurate for identifying the speaker's identity and detecting the timing of speaker transitions.

[0004] Therefore, in related technologies, the accuracy of speaker identity recognition is low. Summary of the Invention

[0005] The present disclosure proposes a speech recognition technology solution.

[0006] According to one aspect of the present disclosure, there is provided a speech recognition device, comprising:

[0007] A speech-to-text recognition module receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp;

[0008] a speaker identity information recognition module that recognizes speaker identity information in the audio signal based on acoustic coding features in the audio signal, thereby obtaining a time-sequentially arranged speaker identity information sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0009] A speaker change time point recognition module, which identifies the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal, and obtains the speaker change time point;

[0010] The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, where the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0011] In a possible implementation, the speaker switching time point identification module identifies the time point at which the speaker in the audio signal changes based on the semantic integrity of a sentence composed of multiple characters in the audio signal to obtain the speaker switching time point.

[0012] In one possible implementation, the speech-to-text recognition module includes: a streaming encoder that receives an input audio signal, encodes the acoustic features of each audio frame in the audio signal, and obtains acoustic coding features and corresponding probabilities corresponding to each audio frame;

[0013] The speaker identity information recognition module includes:

[0014] The non-streaming encoder concatenates the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to each sentence;

[0015] The speaker identity detector determines a time-sequential sequence of speaker identity information according to the acoustic coding features and corresponding probabilities corresponding to the sentences.

[0016] In one possible implementation, the speech-to-text recognition module includes: a predictor configured to predict, based on the predicted historical character features, the next character feature of the historical character features and the corresponding probability, and obtain the current character feature and the corresponding probability;

[0017] The speaker switching time point recognition module determines whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent.

[0018] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point;

[0019] The integration module determines the text corresponding to each speaker identity information according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0020] In a possible implementation, the apparatus further includes:

[0021] A rescoring module, which adds speaker labels to the multiple candidate text sequences obtained by the speech-to-text recognition module according to the speaker identity information sequence and the speaker switching time point, to obtain multiple candidate text sequences with speaker identity information;

[0022] The re-scoring module re-scores the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence;

[0023] The re-scoring module determines a target probability for each candidate text sequence based on the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module;

[0024] The re-scoring module determines a target text sequence with speaker identity information based on the target probability.

[0025] In one possible implementation, the re-scoring module determines the probability of each character and label in the candidate text sequence with speaker identity information, and adds the probabilities of each character and label in the same candidate text sequence to obtain a first probability of the candidate text sequence.

[0026] According to one aspect of the present disclosure, a method for training a speech recognition device is provided, for training any of the above-mentioned devices, the method comprising:

[0027] Using a speech dataset containing speaker labels, the speaker identity information recognition module is trained, and the trained speaker recognition module is used to construct a multi-speaker speech recognition dataset;

[0028] Use the speech dataset containing text labels to train the speech-to-text recognition module;

[0029] Keeping the parameters of the speech-to-text recognition module unchanged, the speaker identity information recognition module and the speaker transition time point recognition module in the speech recognition device are trained using a multi-speaker speech recognition dataset;

[0030] The speech recognition device is trained as a whole using a speech data set containing speaker labels and text labels to obtain a trained speech recognition device.

[0031] In a possible implementation, the method further includes:

[0032] Add speaker identity information and speaker switching time points to the text obtained by the speech recognition module to construct a training set for the re-scoring module;

[0033] Using the speaker identity information and the speaker switching time points, constructing a hybrid dictionary containing the speaker identity information and the speaker switching time points as a dictionary of the re-scoring module;

[0034] The re-scoring module training set is input into the re-scoring module, and the re-scoring module is trained using the hybrid dictionary.

[0035] According to one aspect of the present disclosure, there is provided a speech recognition method, comprising:

[0036] Receive an input audio signal and perform text recognition on the audio signal to obtain text in the audio signal and a corresponding timestamp;

[0037] Based on acoustic coding features in an audio signal, identifying speaker identity information in the audio signal to obtain a time-sequentially arranged speaker identity information sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0038] Based on semantic information of text in the audio signal, identifying a time point at which a speaker in the audio signal changes, to obtain a speaker switching time point;

[0039] Based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, the text corresponding to the speaker identity information is determined, where the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0040] In a possible implementation, identifying the time point at which the speaker in the audio signal changes based on semantic information of text in the audio signal to obtain the speaker transition time point includes:

[0041] Based on the semantic integrity of a sentence composed of multiple characters in the audio signal, a time point at which a speaker in the audio signal changes is identified to obtain a speaker switching time point.

[0042] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0043] Receive an input audio signal, encode the acoustic features of each audio frame in the audio signal, and obtain an acoustic coding feature and a corresponding probability corresponding to each audio frame;

[0044] The identifying of the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a time-sequentially arranged speaker identity information sequence includes:

[0045] Concatenating the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to each sentence;

[0046] A time-sequential sequence of speaker identity information is determined based on the acoustic coding features and corresponding probabilities corresponding to the sentences.

[0047] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0048] Based on the predicted historical character features, predict the next character feature and the corresponding probability of the historical character feature to obtain the current character feature and the corresponding probability;

[0049] The step of identifying the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point includes:

[0050] Based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent, it is determined whether the current character feature corresponds to a speaker switching time point.

[0051] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point;

[0052] The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, including:

[0053] The text corresponding to each speaker identity information is determined according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0054] In a possible implementation, the method further includes:

[0055] According to the speaker identity information sequence and the speaker switching time point, speaker labels are added to the multiple candidate text sequences obtained by the speech-to-text recognition module to obtain multiple candidate text sequences with speaker identity information;

[0056] Rescoring the candidate text sequences with speaker identity information to obtain a first probability for each candidate text sequence;

[0057] Determining a target probability for each candidate text sequence based on the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module;

[0058] Based on the target probability, a target text sequence with speaker identity information is determined.

[0059] In a possible implementation, the rescoring operation on the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence includes:

[0060] The probability of each character and label in the candidate text sequence with speaker identity information is determined, and the probability of each character and label in the same candidate text sequence is added together to obtain a first probability of the candidate text sequence.

[0061] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0062] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0063] In the disclosed embodiment, a speech-to-text recognition module receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp. A speaker identification module identifies the speaker identity information in the audio signal based on acoustic coding features in the audio signal to obtain a time-sequential speaker identity information sequence, wherein the acoustic coding features are extracted based on the acoustic features in the audio signal. A speaker transition time point recognition module identifies the time point when the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker transition time point. An integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker transition time point. Thus, the speaker identification module can identify and distinguish different speakers based on the acoustic features in the audio signal, such as pitch, timbre, and speaking speed. Furthermore, the speaker transition time point recognition module can use the semantic content changes in the text, such as topic changes and differences in tone, as auxiliary information to identify speaker transition time points, thereby improving the detection accuracy of speaker transition points. By fusing multi-source information, the speech recognition device can understand the audio content more comprehensively and accurately, thereby improving the accuracy of the text corresponding to the recognized speaker identity information.

[0064] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0066] Figure 1A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown.

[0067] Figure 2 A schematic structural diagram of a speaker identity information recognition module provided by an embodiment of the present disclosure is shown.

[0068] Figure 3 A structural diagram of a speech-to-text recognition module provided by an embodiment of the present disclosure is shown.

[0069] Figure 4 A structural diagram of a speaker identity information recognition module and a speaker transition time point recognition module provided by an embodiment of the present disclosure is shown.

[0070] Figure 5 A schematic structural diagram of an end-to-end speech recognition device provided by an embodiment of the present disclosure is shown.

[0071] Figure 6 A schematic diagram illustrating the structure of a training architecture of a re-scoring module provided in an embodiment of the present disclosure is shown.

[0072] Figure 7 A flowchart of a speech recognition method according to an embodiment of the present disclosure is shown.

[0073] Figure 8 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0074] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0075] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0076] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0077] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0078] In related technologies, speaker identification and speaker transition detection remain inaccurate. Specifically, different speakers have varying speech characteristics, including pitch, speaking speed, and pronunciation habits. These differences need to be accurately captured and distinguished during the recognition process. Furthermore, speaker transitions are often accompanied by changes in semantic content, such as topic and tone, requiring a comprehensive assessment based on both textual information and audio signal characteristics.

[0079] In the disclosed embodiment, a speech-to-text recognition module receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp. A speaker identification module identifies the speaker identity information in the audio signal based on acoustic coding features in the audio signal to obtain a time-sequential speaker identity information sequence, wherein the acoustic coding features are extracted based on the acoustic features in the audio signal. A speaker transition time point recognition module identifies the time point when the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker transition time point. An integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker transition time point. Thus, the speaker identification module can identify and distinguish different speakers based on the acoustic features in the audio signal, such as pitch, timbre, and speaking speed. Furthermore, the speaker transition time point recognition module can use the semantic content changes in the text, such as topic changes and differences in tone, as auxiliary information to identify speaker transition time points, thereby improving the detection accuracy of speaker transition points. By fusing multi-source information, the speech recognition device can understand the audio content more comprehensively and accurately, thereby improving the accuracy of the text corresponding to the recognized speaker identity information.

[0080] Figure 1 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown. Figure 1 As shown, the device includes:

[0081] The speech-to-text recognition module 11 receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp;

[0082] a speaker identity information recognition module 12 for recognizing speaker identity information in the audio signal based on acoustic coding features in the audio signal, and obtaining a time-sequentially arranged speaker identity information sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal;

[0083] A speaker change time point identification module 13 identifies the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal, thereby obtaining the speaker change time point;

[0084] The integration module 14 determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, where the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0085] The speech-to-text recognition module is used to convert the input audio signal into readable text information and mark the timestamp of each text segment. Exemplarily, the speech-to-text recognition module can be implemented using a recurrent neural network transformer (RNNT) structure. This structure combines the sequence modeling capabilities of a recurrent neural network (RNN) with the decoding efficiency of connectionist temporal classification (CTC). It can efficiently and accurately recognize text content from a continuous audio stream and assign a timestamp to each recognized text segment, indicating the time position of the text in the audio.

[0086] For example, in an audio clip containing a conversation between multiple people, the speech-to-text recognition module can recognize text such as "Good morning, how's the weather today?" and mark which time periods in the audio these words correspond to (such as 00:00:00-00:00:04).

[0087] The speech-to-text recognition module identifies the text in the audio signal and timestamps it, providing the necessary text and time information for subsequent speaker identification and speaker transition time identification. The audio signal can be streaming or non-streaming, which is not limited in this disclosure.

[0088] The speaker identification module is used to identify the speaker's identity information from the audio signal based on the encoded acoustic features in the audio signal, generating a time-sequential sequence of speaker identity information. The acoustic encoding features are extracted from the audio signal and can be obtained by encoding the acoustic features of each audio frame in the audio signal using the streaming encoder in the speech-to-text recognition module.

[0089] Acoustic features are physical properties of audio signals that reflect the content and pronunciation of text. These features can include pitch, timbre, intensity, and duration. They are closely related to the pronunciation of the text, the speaker's timbre, intonation, and speaking speed, and are crucial for identifying the speaker. For example, when recognizing the word "hello," different speakers may have different pitch and timbre characteristics, and these characteristics are used to distinguish the speaker's identity.

[0090] Speaker identity information is used to identify individual speakers. It can be a unique identifier or a set of feature vectors. For example, in a conversation, it can be identified as "This is speaker A speaking" or "This is speaker B speaking," and this identity information can be arranged into a chronological sequence.

[0091] The time-sequential speaker identity information sequence arranges the identified speaker identity information into a sequence based on the order in which the speakers appear in the audio signal. This sequence not only contains the speaker identity information but also preserves the temporal order of this information in the audio signal, which helps accurately locate the speaker's speaking time in subsequent processing. For example, in an audio clip containing a three-person conversation, the system can identify the identity information of speakers A, B, and C and arrange them into a time-sequential sequence of "ABACB" based on the order in which they spoke.

[0092] The speaker change timing recognition module identifies the time points when the speaker in an audio file changes based on the semantic information of the text in the audio signal. This module uses semantic content difference detection to analyze changes in text content (such as topic changes and changes in tone) to determine when a speaker change occurs. This approach not only considers the physical characteristics of the audio but also incorporates semantic analysis, improving recognition accuracy.

[0093] For example, in a conversation, when A says "I think the weather is nice today", and B then says "But I feel a little cold", the speaker transition time point recognition module can recognize the semantic change and use the time point of the transition from A to B as the speaker transition time point.

[0094] The integration module integrates the recognized text, the corresponding timestamp, the speaker identity information sequence, and the speaker transition time point to determine the text content corresponding to each speaker identity information.

[0095] Specifically, the audio is first segmented into different speaker segments based on the speaker transition time points. Then, combined with the speaker identity information sequence, the text of each segment is assigned to the corresponding speaker in chronological order. At the same time, the timestamps corresponding to the text are used to ensure the accuracy of the assignment.

[0096] For example, in the previous example, the integration module will assign "Good morning, how is the weather today?" to A and "But I feel a little cold" to B, and preserve the timestamp information of each text segment.

[0097] In a possible implementation, the speaker switching time point identification module identifies the time point at which the speaker in the audio signal changes based on the semantic integrity of a sentence composed of multiple characters in the audio signal to obtain the speaker switching time point.

[0098] A speaker transition is the moment in a continuous audio signal when one speaker changes to another. This moment marks a change in speaker identity and is used to distinguish between segments of speech from different speakers. For example, in an audio segment containing two people, when speaker A stops speaking and speaker B begins speaking, the transition is the speaker transition.

[0099] Semantic integrity refers to the completeness and coherence of a sentence or text in expressing meaning. In speech recognition, semantic integrity is typically determined by whether the information conveyed by the sentence is complete and clear, and whether the logical relationships between the various parts of the sentence are tight. A semantically complete sentence clearly expresses a complete thought or message.

[0100] For example, the sentence "The weather is so nice today, let's go for a walk in the park" is a semantically complete sentence, which contains complete information such as time (today), weather (so nice), action (go for a walk in the park), etc., and the logic between these information is clear and coherent.

[0101] Therefore, we can infer speaker switching by analyzing the semantic integrity of sentences. When it is determined that the semantics of a sentence is no longer continuous with the semantics of the previous sentence, it is determined that the speaker has switched.

[0102] In this disclosed embodiment, the concept of semantic integrity is introduced to improve the accuracy of identifying speaker transitions. Compared to methods that rely solely on acoustic features, this method more comprehensively considers the contextual information and semantic coherence of the conversation, resulting in more accurate judgment of speaker transitions.

[0103] In one possible implementation, the speech-to-text recognition module includes: a streaming encoder that receives an input audio signal and encodes the acoustic features of each audio frame in the audio signal to obtain acoustic coding features and corresponding probabilities corresponding to each audio frame; the speaker identity information recognition module includes: a non-streaming encoder that concatenates the acoustic coding features corresponding to the audio frames to obtain acoustic coding features and corresponding probabilities corresponding to each sentence; and a speaker identity detector that determines a time-series sequence of speaker identity information based on the acoustic coding features and corresponding probabilities corresponding to the sentences.

[0104] A streaming encoder can process audio signals in real time, encoding continuously input audio signals. Using frame processing technology, a streaming encoder can segment continuous audio signals into multiple shorter frames and independently encode the acoustic features of each frame to obtain acoustic coding features and corresponding probabilities. This allows for instant encoding of audio signals, resulting in rapid recognition results.

[0105] The streaming encoder in the embodiment of the present disclosure can be an encoder in the RNNT model. The streaming encoder receives the audio signal input by the user and converts it into a series of feature vectors (such as Mel-frequency cepstral coefficient MFCC features), while assigning one or more possible characters and their corresponding probabilities to each feature vector.

[0106] Non-streaming encoders do not process audio signals in real time. Instead, they wait for a certain amount of audio data (such as a complete sentence) to be input before encoding the entire sentence. For example, they can start encoding the entire sentence after detecting a period of silence. They can capture long-term dependencies in the audio signal, thereby more accurately extracting the semantic information of the sentence. In speech recognition devices, non-streaming encoders can generate sentence-level feature representations.

[0107] The non-streaming encoder receives the sequence of acoustic coding features output by the streaming encoder and performs operations such as concatenation and fusion on these features to form the acoustic coding features corresponding to the complete sentence. The acoustic coding features corresponding to the sentence not only contain the recognition information of individual characters, but also the contextual relationships between characters and the semantic information of the entire sentence.

[0108] The speaker identity detector receives the acoustic coding features and corresponding probabilities corresponding to the sentences output by the non-streaming encoder to determine the identity information of the current speaker and generate a time-series sequence of speaker identity information.

[0109] For example, during speech recognition, the streaming encoder receives the audio signal of the word "hello" spoken by the user and divides it into multiple frames. It then extracts features such as MFCC from each frame and compares them with a preset character template to obtain the most likely acoustic coding features and their corresponding probabilities for each frame. The non-streaming encoder then concatenates the acoustic coding features to obtain acoustic coding features and their corresponding probabilities corresponding to multiple sentences. The non-streaming encoder receives the acoustic coding features and their corresponding probabilities output by the streaming encoder. These character features are then concatenated and fused to form a complete sentence feature vector, and a corresponding probability value is generated based on the probability output by the streaming encoder to represent the probability of the entire sentence feature vector. Finally, the speaker identity detector determines a time-series sequence of speaker identity information based on the multiple sentence features and their corresponding probabilities.

[0110] For a single audio frame, the streaming encoder may output multiple different acoustic coding features and their corresponding probabilities, thus obtaining the probability distribution of multiple acoustic coding features. Then, when concatenating and fusing the acoustic coding features of multiple audio frames, the multiple acoustic coding features of different audio frames can be arranged and combined. Each combination is the acoustic coding feature of a sentence. By multiplying or adding the probabilities in each combination, the probability of the acoustic coding feature corresponding to each sentence can be obtained.

[0111] In an embodiment of the present disclosure, the acoustic features of each audio frame in the audio signal are encoded in real time by a streaming encoder to obtain acoustic coding features and corresponding probabilities corresponding to each audio frame. Then, the acoustic coding features are spliced by a non-streaming encoder to obtain acoustic coding features and corresponding probabilities corresponding to multiple sentences to improve recognition accuracy. In combination with a speaker identity detector, a time-series sequence of speaker identity information is determined based on the acoustic coding features and corresponding probabilities corresponding to each sentence. This can efficiently and accurately identify speaker identity information in the audio and generate a time-series sequence, significantly improving the real-time and accuracy of speaker identity recognition.

[0112] In one possible implementation, the speech-to-text recognition module includes: a predictor for predicting the next character feature and the corresponding probability of the historical character feature based on the predicted historical character feature, and obtaining the current character feature and the corresponding probability; and a speaker switching time point recognition module for determining whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence composed of the historical character feature and the current character feature with the highest probability are coherent.

[0113] The predictor can predict the future (i.e., the next) character feature and its corresponding probability based on the predicted historical character features. By leveraging the context understanding ability of the deep learning model, the predictor analyzes the previously recognized character sequence (i.e., historical character features) to predict the next possible characters and the probabilities of these characters' occurrences. This helps to more accurately understand the semantic content represented by the current audio signal and improve the recognition accuracy of the speaker transition time point.

[0114] For example, assume the user is saying "What's the weather like tomorrow". After the predictor recognizes "tomorrow", based on the semantics and context information of these two characters, it predicts that the next character is likely to be a word related to "weather", such as "天 (sky)", "气 (weather)", or "的 (of)", etc., and gives the respective probabilities of these predicted characters. In this way, in subsequent processing, the system can more specifically search for and match possible character sequences.

[0115] The speaker change detection module (SCD detect) can be a binary classifier that can determine whether a speaker change has occurred at the current moment based on historical character features, current character features, and their corresponding probabilities. Specifically, this module determines whether the current character feature corresponds to the speaker transition time point based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability is coherent. If the semantics is coherent, it is determined that the current character feature does not correspond to the speaker transition time point; if the semantics is incoherent, it is determined that the current character feature corresponds to the speaker transition time point. For example, the start time point of the current character is used as a speaker transition time point.

[0116] For example, assume the user is saying "What's the weather like tomorrow". After the predictor recognizes "tomorrow 天 (tomorrow sky)", the predictor continues to output the current character with the highest probability as "气 (weather)". Since the semantics of "tomorrow 天 (tomorrow sky)" + "气 (weather)" is coherent, it does not correspond to the speaker transition time point. However, assume the historical character features are "What's the weather like tomorrow", and the recognized current character is "明 (tomorrow)". Since the semantics of the sentence formed by "What's the weather like tomorrow" and "明 (tomorrow)" is incoherent, it is determined that the current character "明 (tomorrow)" corresponds to the speaker transition time point, and the start time point of the current character "明 (tomorrow)" can be used as a speaker transition time point.

[0117] The speaker change detection module can be implemented using machine learning or deep learning algorithms, and this disclosure does not limit this.

[0118] In the speech-to-text recognition module provided by this disclosure, a streaming encoder processes the input audio signal to generate a sequence of acoustic coding features. A predictor and a jointer then work together to gradually generate an output character sequence. The predictor predicts the next possible character based on the jointer's historical output token i, while the jointer determines the final output character, token i+1, based on the current acoustic coding features output by the streaming encoder and the predictor's output.

[0119] In the disclosed embodiment, a predictor predicts the next character feature and corresponding probability of the historical character feature based on the previously predicted character features, thereby obtaining the current character feature and corresponding probability. A speaker transition time point identification module determines whether the current character feature corresponds to a speaker transition time point based on the semantic coherence of the sentence formed by the historical character features and the current character feature with the highest probability. Thus, by predicting future character features and their probabilities, the predictor can incorporate semantic information from the text, enhancing the system's adaptability to complex contexts. Furthermore, the speaker transition time point identification module can more accurately identify speaker transitions in multi-person conversations based on the semantic coherence of the sentence formed by the historical character features and the current character feature with the highest probability.

[0120] In one possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal based on the speaker identity information sequence and the speaker transition time point; the integration module determines the text corresponding to each speaker identity information based on the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0121] The integration module first analyzes the speaker identity sequence and speaker transition times to identify the specific times each speaker begins and ends speaking. This allows the audio signal to be accurately divided into multiple independent speaker segments, each corresponding to a specific speaker.

[0122] The integration module then uses the text output from the text recognition module and its corresponding timestamps to match the recognized text content with the previously determined speaker speaking time period. Specifically, the integration module can extract the corresponding text content based on the timestamp of each speaker's speaking time period and assign the text segment to the speaker.

[0123] In this way, a structured conversation record can be constructed, which includes the identity information of each speaker, the speaking time and the corresponding text content.

[0124] Suppose there is an audio file that records the conversation between three people (Speaker A, Speaker B, and Speaker C) in a meeting. The content of the audio file is roughly as follows:

[0125] Speaker A speaks first: "Hello everyone, let's discuss the progress of the project today."

[0126] Then, speaker B took over the topic: "Yes, I have completed my part, but I encountered some problems."

[0127] Then, speaker A spoke again: "Oh, what are the specific problems?"

[0128] Then, speaker C joins the discussion: "I've noticed this problem too. Perhaps we can solve it this way..."

[0129] The specific workflow of the integration module is as follows:

[0130] First, analyze the speaker identity information and transition time points: receive the identity information sequence output by the speaker identity information recognition module, such as "[A,A,B,A,C]", which means that the speeches of speakers A, A, B, A, and C appear in the audio in sequence.

[0131] At the same time, the system also receives the output of the speaker transition time point recognition module, which marks the moment when the speaker changes. For example, time point T1 indicates the transition from speaker A to speaker B, time point T2 indicates the transition from speaker B back to speaker A, and time point T3 indicates that speaker C joins the discussion.

[0132] With this information, the integration module can identify the specific time points when each speaker starts speaking (such as A at the beginning of the audio, B after T1, and C after T3) and ends speaking (such as A before T1 and after T2, B ends before T2, and C ends at the end of the audio or before the next transition point).

[0133] The text recognition module has processed the audio and output text fragments with timestamps, such as "[00:00-00:05,'Hi everyone, let's discuss the progress of the project today.'], [00:06-00:12,'Yes, I have completed my part, but encountered some problems.']" and so on.

[0134] The integration module extracts and matches the corresponding text based on the timestamps of each speaker's speech time. For example, the timestamp of the text segment "[00:00-00:05,...]" falls entirely within the first speech time of speaker A, so the text can be assigned to speaker A.

[0135] Similarly, “[00:06-00:12,...]” is assigned to speaker B, and subsequent text segments matching the speaking time periods of speakers A and C are assigned to the two speakers respectively.

[0136] Constructing structured conversation records:

[0137] After the above steps, the integration module constructs a structured conversation record as shown below:

[0138] Speaker A: 00:00-00:05, "Hello everyone, today we are going to discuss the progress of the project."

[0139] Speaker B: 00:06-00:12,"Yes, I have completed my part, but I have encountered some problems."

[0140] Speaker A: 00:13-00:15, "Oh, what are the specific issues?"

[0141] Speaker C: 00:16-..., "I've noticed this problem too, maybe we can solve it this way..."

[0142] This record clearly shows the identity information, speaking time and corresponding text content of each speaker, which facilitates subsequent analysis and processing.

[0143] In addition, content from a specific speaker can be identified by pre-registering the target speaker.

[0144] Specifically, speaker registration can be performed in advance by collecting multiple audio signals of the target speaker, and then using a pre-trained speaker identity information recognition module to extract the speaker embeddings (Speaker Embedding) of these audios, and calculating the average or weighted average of these embeddings as the representative vector (Target Speaker Embedding) of the target speaker.

[0145] Then, the audio to be recognized is subjected to speech recognition to obtain the text content. At the same time, the speaker identity information recognition module is used to extract speaker embeddings for each speech segment in the audio. These embeddings are compared with the representative vector of the target speaker and the similarity (such as cosine similarity) is calculated.

[0146] Based on the similarity score, the speech segments that are most similar to the target speaker are selected. If the similarity exceeds a preset threshold, the speech segment is considered to belong to the target speaker and the corresponding text content is extracted. The identified target speaker content is then displayed to the user or sent to downstream tasks (such as sentiment analysis and intent recognition) for processing.

[0147] In the disclosed embodiment, the integration module determines the start and end times corresponding to each speaker identity in the audio signal based on the speaker identity sequence and speaker transition time points. Furthermore, the integration module determines the text corresponding to each speaker identity based on the start and end times corresponding to each speaker identity, the recognized text, and the corresponding timestamp. This allows for precise segmentation of speaker speech segments and matching of corresponding text content.

[0148] In one possible implementation, the device further includes: a rescoring module, which adds speaker labels to multiple candidate text sequences obtained by the speech-to-text recognition module based on the speaker identity information sequence and the speaker switching time point, to obtain multiple candidate text sequences with speaker identity information; the rescoring module rescores the candidate text sequences with speaker identity information to obtain a first probability for each candidate text sequence; the rescoring module determines a target probability for each candidate text sequence based on the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module; and the rescoring module determines a target text sequence with speaker identity information based on the target probability.

[0149] The rescoring module can add speaker labels to the multiple candidate text sequences output by the speech-to-text recognition module based on the speaker identity information sequence output by the speaker identity information recognition module and the speaker transition time points output by the speaker transition time point recognition module. The module then rescores the candidate text sequences with speaker identity information, ultimately determining a target text sequence with speaker identity information. The specific labeling process can be found in the integration module's "determining the text corresponding to the speaker identity information based on the recognized text and corresponding timestamps, the speaker identity information sequence, and the speaker transition time points" process in the disclosed embodiments, and will not be elaborated upon here.

[0150] The rescoring module can receive multiple candidate text sequences output by the speech-to-text recognition module, as well as the corresponding speaker identity information and speaker switching time points, and then construct a candidate text sequence with speaker identity information.

[0151] Specifically, among the multiple candidate text sequences output by the speech-to-text recognition module, a corresponding speaker label (such as <spk1> , <spk2>etc.), generate multiple candidate text sequences with speaker identity information. For example, if the original candidate text sequence is "XXX XXXXXX", and the speaker transition points are between "XXX" and "XXXX", and the identified speakers are SPK1 and SPK2 respectively, then the processed sequence is " <spk1>XXX <spk2>XXXX <spk1>XX". In addition, if the sentence is the beginning of a sentence, you can also add <sos>Tags, get " <sos> <spk1>XXX <spk2>XXXX <spk1>XX". According to this method, multiple candidate text sequences can be obtained for the multiple candidate text sequences output by the speech-to-text recognition module.

[0152] These multiple candidate text sequences are then rescored to obtain a first probability for each candidate text sequence. The rescoring module has already learned to combine speaker identity information, speaker transition point information, and semantic information of the text sequence to perform scoring during the training phase. The specific training process can be found in the training method provided in this disclosure and will not be detailed here.

[0153] At the same time, combined with the second probability originally calculated by the speech-to-text recognition module for each candidate text sequence, the target probability of each candidate text sequence is calculated through weighting and other methods. Then, based on the calculated target probability, the candidate text sequence with the highest probability is selected as the final target text sequence with speaker identity information. Finally, the target text sequence with speaker identity information is output for subsequent processing or display.

[0154] For example, the re-scoring module can calculate the target probability of each candidate text sequence according to a certain weight ratio, and select the candidate text sequence with the highest target probability as the final output result. The calculation process can be expressed as formula (1).

[0155] Target probability = k1*second probability + (1-k1)*first probability (1)

[0156] Among them, k1 is the weight coefficient.

[0157] In the disclosed embodiment, among the multiple candidate text sequences output by the speech-to-text recognition module, the rescoring module first adds a speaker label to the appropriate position of each candidate text sequence based on the speaker identity information and the speaker transition time point. Then, the rescoring module rescores the candidate text sequences with the speaker label added, and evaluates their rationality and accuracy by calculating the sequence probability (i.e., the first probability) of the candidate text sequence. Since the rescoring module has learned the speaker identity information, the semantic information of the text sequence, and the characteristics of the speaker transition point during the training process, it can more accurately evaluate the quality of the candidate text sequence. After obtaining the first probability of each candidate text sequence, the rescoring module also considers the second probability of each candidate text sequence directly output by the speech-to-text recognition module, and combines the first and second probabilities in a weighted manner to obtain the final target probability of each candidate text sequence. Based on the target probability, the target text sequence with speaker identity information is determined and output. Since the target probability comprehensively considers the direct output of the speech-to-text recognition module and the evaluation results of the rescoring module, it can more accurately reflect the actual quality of the candidate text sequence, thereby improving the accuracy of the final recognition result.

[0158] In one possible implementation, the re-scoring module determines the probability of each character and label in the candidate text sequence with speaker identity information, and adds the probabilities of each character and label in the same candidate text sequence to obtain a first probability of the candidate text sequence.

[0159] For candidate text sequences with speaker identity information, the probability of each word and label in the sequence can be calculated and accumulated to obtain the first probability of the entire candidate text sequence. By comprehensively considering the probability information of all elements in the sequence (including words and labels), the overall reliability of the candidate text sequence can be more accurately assessed, thereby improving the accuracy and robustness of the speech recognition system.

[0160] During the rescoring process, the rescoring module iterates over each word and label in the candidate text sequence, calculating their respective probabilities. These probabilities are then accumulated to obtain a first probability for the entire candidate text sequence. This cumulative approach comprehensively considers the uncertainty of all elements in the sequence, providing a more comprehensive assessment of the reliability of the candidate sequence.

[0161] Because candidate text sequences contain speaker identity information (such as speaker tags), the rescoring module can simultaneously consider acoustic and semantic features, as well as speaker characteristics, when calculating probabilities. By integrating multi-source information, the target speaker and their content can be more accurately identified, improving the personalized experience and security of the system.

[0162] For example, when a candidate text sequence obtained by the speech-to-text recognition module is " <spk1>Good morning", during the rescoring phase, the sequence will be processed into a format that matches the format used when the rescoring module was trained. <sos> <spk1>Good morning <eos>”. Then, the rescoring module receives this processed sequence as input and calculates <sos> 、 <spk1>, "Good morning" and <eos>Specifically:

[0163] <sos>The probability of: as the start mark of the sequence, <sos>The probability reflects the reliability of the initial part of the sequence.

[0164] <spk1>Probability of: speaker label <spk1>The probability of represents the confidence in identifying the speaker's identity.

[0165] The probability of "good morning" is the probability of the actual spoken content, which represents the recognition accuracy of this content.

[0166] <eos>The probability of: as the end mark of the sequence, used to evaluate the integrity of the entire sequence.

[0167] After obtaining these probabilities, the rescoring module will add them up to get the entire candidate text sequence" <sos> <spk1>Good morning <eos>”The first probability.

[0168] In addition, the present disclosure also provides a method for training a speech recognition device, which is used to train any speech recognition device provided by the present disclosure, and the method includes:

[0169] In step S21, a speaker identity information recognition module is trained using a speech dataset containing speaker labels, and a multi-speaker speech recognition dataset is constructed using the trained speaker recognition module;

[0170] In step S22, the speech-to-text recognition module is trained using the speech data set containing the text labels;

[0171] In step S23, the parameters of the speech-to-text recognition module are kept unchanged, and the speaker identity information recognition module and the speaker transition time point recognition module in the speech recognition device are trained using the multi-speaker speech recognition dataset;

[0172] In step S24, the speech recognition device is trained as a whole using the speech data set containing speaker labels and text labels to obtain a trained speech recognition device.

[0173] The Speaker Identification module is responsible for identifying the speaker from speech data. Trained on speaker-labeled speech datasets, the Speaker Identification module learns the speech characteristics of different speakers, enabling accurate distinction between them. Once trained, the Speaker Identification module can be used to construct datasets containing speech from multiple speakers.

[0174] Figure 2 A structural diagram of a speaker identity information recognition module provided by an embodiment of the present disclosure is shown. The module can be trained using a triplet loss method. Afterwards, the trained speaker identity information recognition module is used to perform data expansion on a normal speech recognition dataset, and each piece of speech data is labeled with a corresponding speaker label, thereby forming a multi-speaker speech recognition dataset. For example, the dataset U = {(x1, y1), (x2, y2) ...} (where x is audio and y is a text label) is expanded to U + s = {(x1, y1, s1), (x2, y2, s2) ...}, where s is the corresponding speaker label. Then, long audios of multiple speakers are obtained by random splicing. For example, after the audios (x1, y1, s1) and (x2, y2, s2) are randomly selected, they can be spliced to obtain audios (Xlong1, Ylong1, Slong1), where Xlong is the spliced audio and Ylong is in the form of y1 <scd>y2, where <scd>Indicates the speaker switching point. Slong = {(s1, start1, end1), (s2, start2, end2)} represents the starting point (start1, start2) and end point (end1, end2) of the audio corresponding to different speakers.

[0175] The speech-to-text recognition module is then trained using a speech dataset containing text labels. The speech-to-text recognition module converts input audio signals into corresponding text information. By training on a speech dataset annotated with text labels, the speech-to-text recognition module learns the mapping between speech and text, thereby improving recognition accuracy and efficiency.

[0176] Figure 3 The following is a schematic diagram of the structure of a speech-to-text recognition module provided by an embodiment of the present disclosure. The module includes components such as a streaming encoder, a predictor, and a jointer. The streaming encoder processes the input audio signal and generates a sequence of acoustic coding features. The predictor is then responsible for predicting the next possible character based on the historical output token i of the jointer, while the jointer determines the final output character token i+1 based on the current acoustic coding features output by the streaming encoder and the output of the predictor.

[0177] In order to further improve the adaptability of the speech recognition device to multi-speaker scenarios, the speaker identity information recognition module and the speaker transition time point recognition module may be further optimized. Figure 4 A structural diagram of a speaker identity information recognition module and a speaker transition timing recognition module provided by an embodiment of the present disclosure is shown. When training these modules, the parameters of the speech-to-text recognition module can be fixed. Joint training can then be performed using a speech dataset containing multiple speaker labels. This allows learning the switching patterns between different speakers and the timing of speaker transitions, thereby improving the overall performance of the system.

[0178] Specifically, a multi-speaker speech recognition dataset or a dataset constructed by a pre-trained speaker recognition module is used to generate long audio signals (Xlong1, Ylong1, Slong1) containing multiple speakers. Among them, Xlong1 is the concatenated long audio, Ylong1 is the corresponding text label, and <scd>To indicate the speaker switching point (such as y1 <scd>y2), Slong1 contains the start and end point information of the audio corresponding to different speakers, such as {(s1,start1,end1),(s2,start2,end2)}.

[0179] At the beginning of joint training, the parameters of the streaming encoder, predictor, and fuser are fixed. These modules only perform forward inference during training and do not participate in parameter updates.

[0180] Input long audio Xlong to the streaming encoder, and the acoustic coding features of each audio frame are accumulated over time. When the accumulated acoustic coding features reach a certain length or meet a certain preset condition, they are sent to the non-streaming encoder for further processing.

[0181] The output of the non-streaming encoder is fed into the speaker identification module for speaker recognition training. Speaker labels are obtained using the timestamp information (start1, end1, start2, end2, etc.) in Slong. For example, if the current time frame t falls within the time range of a certain speaker (start2 <= t <= end2), the label for that time frame is s2.

[0182] At the same time, the predictor will predict the speaker transition point of the input token (predicted by the previous token output by the fusion unit). <scd>appears to determine whether a speaker switch occurs. For example, in the text "Hello <scd>Good morning", the predictor will predict the corresponding SCD label "blank" based on the input "hello good morning". <scd>blank blank blank", thereby determining the location of the speaker's switching point.

[0183] During the joint training process, the total loss function is the weighted sum of the speaker recognition loss (loss_spk_identify) and the speaker transition point prediction loss (loss_spk_scd), that is, total loss total_loss = w1*loss_spk_identify + w2*loss_spk_scd, where w1 and w2 are weights.

[0184] At the beginning of training, the parameters of the streaming encoder, predictor, and fuser are fixed, and only the speaker identification module and the speaker transition prediction module are trained. Once these two modules have been trained to a certain level, all modules are released for joint training. The total loss function at this point also includes the loss of the speech-to-text recognition module itself (loss_rnnt). That is, the total loss = w1*loss_spk_identify + w2*loss_spk_scd + loss_rnnt.

[0185] This joint training process enables simultaneous completion of streaming speech recognition, speaker identification, and speaker transition prediction, resulting in an end-to-end target speaker speech recognition system. By fixing the parameters of the speech-to-text recognition module and then training the other modules, this method improves training stability and efficiency. Furthermore, this training process fully utilizes both acoustic and semantic information, enhancing the accuracy of the speech recognition system.

[0186] After the above training, we can construct Figure 5 The end-to-end speech recognition device shown, Figure 5 The structure diagram of an end-to-end speech recognition device provided by an embodiment of the present disclosure is shown. The functions and training process of each module can be found in the relevant description above and will not be repeated here.

[0187] In one possible implementation, the method further includes: adding speaker identity information and speaker switching time points to the text obtained by speech recognition by the speech-to-text recognition module to construct a re-scoring module training set; using the speaker identity information and speaker switching time points to construct a hybrid dictionary containing the speaker identity information and speaker switching time points as a dictionary of the re-scoring module; inputting the re-scoring module training set into the re-scoring module, and using the hybrid dictionary to train the re-scoring module.

[0188] In this implementation, the rescoring module is trained. During the recognition process of the audio signal by the speech-to-text recognition module, a series of recognition results are obtained. These results are generated in real time in units of characters or words. After the entire speech input is completed, these results are fused into a complete sentence. After the complete sentence is fused, multiple alternative paths (or candidate sentences) are usually retained. These paths are possible recognition results. These alternative paths can provide more options for the rescoring process, thereby finding the optimal recognition result.

[0189] In order to train the re-scoring module, you can randomly extract clips from long audio and use the long audio to construct a training set that contains "speaker identity information, speaker transition point information, and speech content."

[0190] In the process of constructing the training set, special labels (such as <spk1> , <spk2>The tags are used to indicate the speaker identity and the start and end of sentences, and these tags are inserted into the text decoded by the speech-to-text recognition module. For example, if a random audio clip is taken from a long audio clip, and the conversation in this audio clip contains two speakers, the decoded text after inserting the tags may look like this: <sos> <spk1>Good morning <spk2>Did you eat well? <spk1>have eaten <eos>.here, <sos>and <eos>Indicates the beginning and end of a sentence, <spkj>(where j is the speaker number) is used to mark the transition of speakers. <sos> <spk1>Good morning <spk2>Did you eat well? <spk1>have eaten <eos>" is the expected output of the corresponding audio clip.

[0191] To train the rescoring module, these special tags are combined with the speech recognition decoding vocab (a dictionary containing all possible recognized characters or words) into a mixture-vocab. This mixture-vocab contains not only the language vocabulary but also special tags indicating the speaker's identity, allowing the rescoring module to simultaneously identify the text content and speaker identity during decoding.

[0192] After preparing the training set and mixed dictionary, you can start training the re-scoring module. Figure 6 A schematic diagram of the structure of a training architecture of a re-scoring module provided in an embodiment of the present disclosure is shown. The re-scoring module is trained based on the training architecture. It should be noted that the training architecture may also include other parts not shown, such as Figure 5 Related modules in .

[0193] During training, we use the previously trained streaming encoder and non-streaming encoder of the speech-to-text recognition module as feature extractors, and fix the parameters of these parts. We only train the re-scoring module (including the encoder and decoder). This allows us to take advantage of the good feature representations we have learned while focusing on optimizing the performance of the re-scoring module. Figure 6 In the streaming encoder shown, feature encoding is performed through the streaming encoder and the non-streaming encoder to obtain sentence features and corresponding probabilities, which are then input into the re-scoring module. The re-scoring module obtains the predicted output, and the loss is calculated using the expected output and the predicted output to update the parameters of the re-scoring module.

[0194] In one possible implementation, the speech recognition device can be implemented by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The device can be implemented by a processor calling computer-readable instructions stored in a memory.

[0195] In addition, the present disclosure also provides a speech recognition method, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any speech recognition device provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0196] Figure 7 A flow chart of a speech recognition method according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the method includes:

[0197] In step S21, an input audio signal is received, and text recognition is performed on the audio signal to obtain text in the audio signal and a corresponding timestamp;

[0198] In step S22, based on acoustic coding features in the audio signal, speaker identity information in the audio signal is identified to obtain a time-sequentially arranged speaker identity information sequence, wherein the acoustic coding features are extracted based on the acoustic features in the audio signal;

[0199] In step S23, based on the semantic information of the text in the audio signal, the time point at which the speaker in the audio signal changes is identified to obtain the speaker switching time point;

[0200] In step S24, based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, the text corresponding to the speaker identity information is determined, and the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal.

[0201] In a possible implementation, identifying the time point at which the speaker in the audio signal changes based on semantic information of text in the audio signal to obtain the speaker transition time point includes:

[0202] Based on the semantic integrity of a sentence composed of multiple characters in the audio signal, a time point at which a speaker in the audio signal changes is identified to obtain a speaker switching time point.

[0203] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0204] Receive an input audio signal, encode the acoustic features of each audio frame in the audio signal, and obtain an acoustic coding feature and a corresponding probability corresponding to each audio frame;

[0205] The identifying of the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a time-sequentially arranged speaker identity information sequence includes:

[0206] Concatenating the acoustic coding features corresponding to the audio frames to obtain the acoustic coding features and corresponding probabilities corresponding to each sentence;

[0207] A time-sequential sequence of speaker identity information is determined based on the acoustic coding features and corresponding probabilities corresponding to the sentences.

[0208] In a possible implementation, the receiving an input audio signal and performing text recognition on the audio signal includes:

[0209] Based on the predicted historical character features, predict the next character feature and the corresponding probability of the historical character feature to obtain the current character feature and the corresponding probability;

[0210] The step of identifying the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal to obtain the speaker switching time point includes:

[0211] Based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent, it is determined whether the current character feature corresponds to a speaker switching time point.

[0212] In a possible implementation, the integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point;

[0213] The integration module determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker switching time point, including:

[0214] The text corresponding to each speaker identity information is determined according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

[0215] In a possible implementation, the method further includes:

[0216] According to the speaker identity information sequence and the speaker switching time point, speaker labels are added to the multiple candidate text sequences obtained by the speech-to-text recognition module to obtain multiple candidate text sequences with speaker identity information;

[0217] Rescoring the candidate text sequences with speaker identity information to obtain a first probability for each candidate text sequence;

[0218] Determining a target probability for each candidate text sequence based on the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module;

[0219] Based on the target probability, a target text sequence with speaker identity information is determined.

[0220] In a possible implementation, the rescoring operation on the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence includes:

[0221] The probability of each character and label in the candidate text sequence with speaker identity information is determined, and the probability of each character and label in the same candidate text sequence is added together to obtain a first probability of the candidate text sequence.

[0222] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0223] In some embodiments, the functions or modules included in the method provided in the embodiments of the present disclosure can be used to execute the device described in the above method embodiment. Its specific implementation can refer to the description of the above device embodiment. For the sake of brevity, it will not be repeated here.

[0224] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0225] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0226] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0227] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0228] Figure 8 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0229] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0230] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0231] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0232] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0233] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0234] The computer program instructions for performing the disclosed operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as, for example, utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0235] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0236] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0237] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0238] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0239] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0240] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0241] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0242] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the individual of the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information. Among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0243] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.< / eos> < / sos> < / spkj> < / eos> < / sos> < / eos> < / sos> < / spk1> < / scd> < / scd> < / scd> < / scd> < / scd> < / scd> < / scd> < / eos> < / sos> < / eos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos> < / sos> < / sos> < / spk1>

Claims

1. A speech recognition device, characterized in that: include: A speech-to-text recognition module receives an input audio signal and performs text recognition on the audio signal to obtain the text in the audio signal and the corresponding timestamp; a speaker identity information recognition module that recognizes speaker identity information in the audio signal based on acoustic coding features in the audio signal, thereby obtaining a time-sequentially arranged speaker identity information sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal; A speaker change time point recognition module, which identifies the time point at which the speaker in the audio signal changes based on the semantic information of the text in the audio signal, and obtains the speaker change time point; an integration module, which determines the text corresponding to the speaker identity information based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker transition time point, wherein the text corresponding to the speaker identity information is the words spoken by the speaker in the audio signal; The speaker identity information recognition module includes: The non-streaming encoder concatenates the acoustic coding features corresponding to each audio frame in the audio signal to obtain the acoustic coding features and corresponding probabilities corresponding to each sentence; The speaker identity detector determines a time-sequential sequence of speaker identity information according to the acoustic coding features and corresponding probabilities corresponding to the sentences.

2. The device according to claim 1, characterized in that The speaker switching time point identification module identifies the time point at which the speaker in the audio signal changes based on the semantic integrity of the sentence composed of multiple characters in the audio signal to obtain the speaker switching time point.

3. The device according to claim 1, characterized in that The speech-to-text recognition module includes: a streaming encoder that receives an input audio signal, encodes the acoustic features of each audio frame in the audio signal, and obtains the acoustic coding features and corresponding probabilities corresponding to each audio frame.

4. The device according to claim 1, characterized in that The speech-to-text recognition module includes: a predictor for predicting the next character feature and the corresponding probability of the historical character feature based on the predicted historical character feature, and obtaining the current character feature and the corresponding probability; The speaker switching time point recognition module determines whether the current character feature corresponds to the speaker switching time point based on whether the semantics of the sentence formed by the historical character features and the current character feature with the highest probability are coherent.

5. The device according to claim 1, characterized in that The integration module determines the start time and end time corresponding to each speaker identity information in the audio signal according to the speaker identity information sequence and the speaker switching time point; The integration module determines the text corresponding to each speaker identity information according to the start time and end time corresponding to each speaker identity information, the recognized text and the corresponding timestamp.

6. The device according to claim 1, characterized in that The device further comprises: A rescoring module, which adds speaker labels to the multiple candidate text sequences obtained by the speech-to-text recognition module according to the speaker identity information sequence and the speaker switching time point, to obtain multiple candidate text sequences with speaker identity information; The re-scoring module re-scores the candidate text sequences with speaker identity information to obtain a first probability of each candidate text sequence; The re-scoring module determines a target probability for each candidate text sequence based on the first probability and the second probability of each candidate text sequence obtained by the speech-to-text recognition module; The re-scoring module determines a target text sequence with speaker identity information based on the target probability.

7. The device according to claim 6, characterized in that The re-scoring module determines the probability of each character and label in the candidate text sequence with speaker identity information, and adds the probabilities of each character and label in the same candidate text sequence to obtain a first probability of the candidate text sequence.

8. A method for training a speech recognition device, characterized in that: For training the apparatus according to any one of claims 1 to 7, the method comprising: Using a speech dataset containing speaker labels, the speaker identity information recognition module is trained, and the trained speaker recognition module is used to construct a multi-speaker speech recognition dataset; Use the speech dataset containing text labels to train the speech-to-text recognition module; Keeping the parameters of the speech-to-text recognition module unchanged, the speaker identity information recognition module and the speaker transition time point recognition module in the speech recognition device are trained using a multi-speaker speech recognition dataset; The speech recognition device is trained as a whole using a speech data set containing speaker labels and text labels to obtain a trained speech recognition device.

9. The method according to claim 8, characterized in that The method further comprises: Add speaker identity information and speaker switching time points to the text obtained by the speech recognition module to construct a training set for the re-scoring module; Using the speaker identity information and the speaker switching time points, constructing a hybrid dictionary containing the speaker identity information and the speaker switching time points as a dictionary of the re-scoring module; The re-scoring module training set is input into the re-scoring module, and the re-scoring module is trained using the hybrid dictionary.

10. A speech recognition method, characterized in that: include: Receive an input audio signal and perform text recognition on the audio signal to obtain text in the audio signal and a corresponding timestamp; Based on acoustic coding features in an audio signal, identifying speaker identity information in the audio signal to obtain a time-sequentially arranged speaker identity information sequence, wherein the acoustic coding features are extracted based on acoustic features in the audio signal; Based on semantic information of text in the audio signal, identifying a time point at which a speaker in the audio signal changes, to obtain a speaker switching time point; Determining, based on the recognized text and the corresponding timestamp, the speaker identity information sequence, and the speaker transition time point, the text corresponding to the speaker identity information being the words spoken by the speaker in the audio signal; The identifying of the speaker identity information in the audio signal based on the acoustic coding features in the audio signal to obtain a time-sequentially arranged speaker identity information sequence includes: The acoustic coding features corresponding to each audio frame in the audio signal are concatenated to obtain the acoustic coding features and corresponding probabilities corresponding to each sentence; A time-sequential sequence of speaker identity information is determined based on the acoustic coding features and corresponding probabilities corresponding to the sentences.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to implement the method according to any one of claims 8 to 10.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 8 to 10 is implemented.

Images