Speech recognition method, speech recognition device and speech recognition system

Abstract

The invention discloses a speech recognition method, a speech recognition device and a speech recognition system. The speech recognition method provided by the embodiment of the invention comprises the following steps: acquiring audio stream data; in the process of acquiring the audio stream data, acquiring a multi-modal endpoint signal corresponding to the audio stream data; segmenting the audio stream data based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment; performing streaming speech recognition on the at least one audio segment based on an offline speech recognition model to obtain a recognition text; the recognized text is streamed. Therefore, the delay of outputting the recognition text is reduced, the occupation of hardware resources is reduced, and the method can be suitable for a real-time interaction scene and a scene with a relatively high recognition precision requirement.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and particularly to a speech recognition method, a speech recognition device, a speech recognition system, and a computer-readable storage medium. Background Art

[0002] As one of the key technologies in the field of artificial intelligence, speech recognition technology has made remarkable progress in recent years. In practical applications, speech recognition technology still faces many challenges. Especially when processing long audio inputs, current speech recognition systems often need to wait for the user to finish speaking an entire sentence or paragraph before starting the recognition and outputting the recognition result, resulting in significant latency issues. Summary of the Invention

[0003] Embodiments of this application provide a speech recognition method, a speech recognition device, a speech recognition system, and a computer-readable storage medium to solve at least one of the above technical problems.

[0004] The speech recognition method according to the embodiments of this application includes:

[0005] Obtain audio stream data;

[0006] During the process of obtaining the audio stream data, obtain the multi-modal endpoint signal corresponding to the audio stream data;

[0007] Segment the audio stream data based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment;

[0008] Perform streaming speech recognition on the at least one audio segment based on an offline speech recognition model to obtain recognition text;

[0009] Stream and output the recognition text.

[0010] In some embodiments, the speech recognition method further includes:

[0011] Obtain video stream data and physical button data, where the video stream data includes any one or more of face data and lip movement data;

[0012] The step of obtaining the multi-modal endpoint signal corresponding to the audio stream data during the process of obtaining the audio stream data includes:

[0013] Generate the multi-modal endpoint signal based on the audio stream data and the video stream data and / or the physical button data.

[0014] In some embodiments, the audio stream data includes multiple audio streams, and the speech recognition method further includes:

[0015] Obtain the first audio length of each of the audio streams, and determine whether the first audio length meets a first preset condition;

[0016] Determine whether the multimodal endpoint signal meets a second preset condition;

[0017] When the first audio length meets the first preset condition and the multimodal endpoint signal meets the second preset condition, remove the silent audio streams from the audio stream data according to the multimodal endpoint signal, and cache at least one of the remaining audio streams to obtain an audio to be recognized;

[0018] The segmenting the audio stream data based on the multimodal endpoint signal and a voice activity detection model to obtain at least one audio segment includes:

[0019] Segment the audio to be recognized based on the multimodal endpoint signal and the voice activity detection model to obtain the at least one audio segment.

[0020] In some embodiments, during the process of caching at least one of the remaining audio streams to obtain an audio to be recognized, the speech recognition method further includes:

[0021] Obtain the total audio length of the cached audio streams to obtain a second audio length, and obtain the average energy value of the cached audio streams;

[0022] The segmenting the audio to be recognized based on the multimodal endpoint signal and the voice activity detection model to obtain the at least one audio segment includes:

[0023] Determine a dynamic segmentation threshold of the voice activity detection model based on the second audio length and / or the average energy value;

[0024] Segment the audio to be recognized based on the multimodal endpoint signal, the voice activity detection model and the dynamic segmentation threshold to obtain the at least one audio segment.

[0025] In some embodiments, when the dynamic segmentation threshold of the voice activity detection model determined based on the second audio length and / or the average energy value is determined based on the second audio length, the determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length includes:

[0026] When the second audio length is less than or equal to a first length threshold, determine the dynamic segmentation threshold as a first preset threshold;

[0027] When the length of the second audio is greater than the first length threshold and less than the second length threshold, the dynamic segmentation threshold is negatively correlated with the length of the second audio;

[0028] When the length of the second audio is greater than or equal to the second length threshold, determine the dynamic segmentation threshold as the second preset threshold;

[0029] Wherein, the first preset threshold is greater than the second preset threshold.

[0030] In some embodiments, the voice activity detection model includes any one of a detection model based on a threshold discrimination method, a detection model based on a model matching method, and a detection model based on deep learning.

[0031] In some embodiments, the streaming speech recognition of the at least one audio segment based on the offline speech recognition model to obtain the recognition text includes:

[0032] Determine the end signal of each audio segment;

[0033] When the length of the received audio segment is an even multiple of the sampling rate, or when the end signal of the received audio segment is received, take the current part of the audio segment as a sub-audio segment and input it into the offline speech recognition model;

[0034] Based on the offline speech recognition model, perform streaming speech recognition on the sub-audio segment corresponding to each audio segment to obtain the recognition text.

[0035] In some embodiments, the multi-modal endpoint signal includes an end signal, the recognition text includes an intermediate recognition text and a result recognition text, and the streaming speech recognition of the sub-audio segment corresponding to each audio segment based on the offline speech recognition model to obtain the recognition text includes:

[0036] Based on the offline speech recognition model, perform streaming speech recognition on each sub-audio segment to obtain a corresponding sub-recognition text;

[0037] When the sub-audio segment does not include the end signal, update the recognition text according to the corresponding sub-recognition text, mark a first recognition character after the recognition text, and mark the recognition text as the first recognition state;

[0038] When the sub-audio segment includes the end signal, update the recognition text according to the corresponding sub-recognition text, mark a second recognition character after the recognition text, and mark the recognition text as the second recognition state;

[0039] The streaming output of the recognition text includes:

[0040] When the sub-audio segment does not include the end signal, the recognition text marked as the first recognition state is output as the intermediate recognition text;

[0041] When the sub-audio segment includes the end signal, the recognition text marked as the second recognition state is output as the result recognition text.

[0042] The voice recognition device according to the embodiment of the present application includes:

[0043] A data processing module, configured to obtain audio stream data;

[0044] The data processing module is further configured to obtain the multi-modal endpoint signal corresponding to the audio stream data during the process of obtaining the audio stream data;

[0045] A voice segmentation module, configured to segment the audio stream data based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment;

[0046] A streaming recognition module, configured to perform streaming voice recognition on the at least one audio segment based on an offline voice recognition model to obtain recognition text;

[0047] A streaming output module, configured to stream output the recognition text.

[0048] The voice recognition system according to the embodiment of the present application, the voice recognition system includes one or more processors and a memory, the memory stores a computer program, and when the computer program is executed by the processor, the voice recognition method according to any of the above embodiments is implemented.

[0049] The computer-readable storage medium according to the embodiment of the present application, on which a computer program is stored, and when the program is executed by a processor, the voice recognition method according to any of the above embodiments is implemented.

[0050] In the voice recognition method, voice recognition device, voice recognition system and computer-readable storage medium according to the embodiment of the present application, during the process of obtaining audio stream data, the multi-modal endpoint signal corresponding to the audio stream data is obtained, the audio stream data is segmented based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment, and then streaming voice recognition is performed on the segmented audio segment based on an offline voice recognition model to obtain recognition text, and the recognition text is streamed output. In this way, the delay of outputting the recognition text is reduced, the occupation of hardware resources is reduced, and it can be applied to real-time interaction scenarios and scenarios with high requirements for recognition accuracy.

[0051] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the embodiments of the present application. Description of the Drawings

[0052] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0053] Figure 1 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0054] Figure 2 is a schematic flowchart of obtaining a recognized text based on audio stream data in a speech recognition method according to some embodiments of the present application;

[0055] Figure 3 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0056] Figure 4 is a schematic flowchart of preprocessing audio stream data according to some embodiments of the present application;

[0057] Figure 5 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0058] Figure 6 is a schematic flowchart of obtaining an audio to be recognized based on audio stream data and multimodal endpoint signals according to some embodiments of the present application;

[0059] Figure 7 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0060] Figure 8 is a schematic diagram of an audio segment obtained without segmentation inside the audio to be recognized according to some embodiments of the present application;

[0061] Figure 9 is a schematic diagram of an audio segment obtained by one segmentation inside the audio to be recognized according to some embodiments of the present application;

[0062] Figure 10 is a schematic diagram of an audio segment obtained by two segmentations inside the audio to be recognized according to some embodiments of the present application;

[0063] Figure 11 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0064] Figure 12 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0065] Figure 13 is a schematic flowchart of a speech recognition method according to some embodiments of the present application;

[0066] Figure 14 is a schematic flowchart of streaming speech recognition and output according to some embodiments of the present application;

[0067] Figure 15 is a schematic diagram of modules of a speech recognition device according to some embodiments of the present application;

[0068] Figure 16 is a schematic diagram of modules of a speech recognition system according to some embodiments of the present application;

[0069] Figure 17 is a schematic diagram of the connection state between a computer-readable storage medium and a processor according to some embodiments of the present application. Specific Embodiments

[0070] The following further describes the embodiments of the present application with reference to the accompanying drawings. The same or similar reference numerals in the drawings denote the same or similar elements or elements having the same or similar functions throughout. Additionally, the embodiments of the present application described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application and should not be construed as a limitation of the present application.

[0071] Please refer to Figure 1 and Figure 2 , embodiments of the present application provide a speech recognition method. The speech recognition method includes:

[0072] 010: Obtain audio stream data;

[0073] 020: During the process of obtaining the audio stream data, obtain multi-modal endpoint signals corresponding to the audio stream data;

[0074] 030: Segment the audio stream data based on the multi-modal endpoint signals and a voice activity detection model to obtain at least one audio segment;

[0075] 040: Perform streaming speech recognition on at least one audio segment based on an offline speech recognition model to obtain recognition text;

[0076] 050: Stream and output the recognition text.

[0077] In the speech recognition method according to the embodiments of the present application, during the process of obtaining audio stream data, multi-modal endpoint signals corresponding to the audio stream data are obtained, and the audio stream data is segmented based on the multi-modal endpoint signals and a speech activity detection model to obtain at least one audio segment. Then, an offline speech recognition model is used to perform streaming speech recognition on the segmented audio segments to obtain recognition text, and the recognition text is output in a streaming manner. In this way, the latency of outputting the recognition text is reduced, the hardware resource occupancy is decreased, and it can be applied to real-time interaction scenarios and scenarios with high requirements for recognition accuracy.

[0078] Specifically, during the process of a user speaking, the audio stream data of the user is obtained. During the process of obtaining the audio stream data, multi-modal endpoint signals corresponding to the audio stream data are obtained. The multi-modal endpoint signals include 0, 1, 2, and 3, which are used to represent four states of the audio stream data. Among them, 0 is a silence signal, representing a silent state; 1 is a start signal, representing the starting state of speaking; 2 is an intermediate signal, representing the intermediate state of speaking; 3 is an end signal, representing the ending state of speaking.

[0079] Based on the multi-modal endpoint signals and the speech activity detection model, the audio stream data can be segmented. For relatively short audio stream data, segmentation may not occur based on the multi-modal endpoint signals and the speech activity detection model, that is, one audio segment is obtained. For relatively long audio stream data, it can be segmented into smaller audio segments based on the multi-modal endpoint signals and the speech activity detection model, and at least two audio segments can be obtained from one audio stream data.

[0080] An offline speech recognition model is used to perform streaming speech recognition on the segmented audio segments to obtain recognition text. It should be noted that the offline speech recognition model is a non-streaming recognition model, and streaming speech recognition can be achieved by sequentially inputting the segmented audio segments into the offline speech recognition model. During the streaming speech recognition process, the obtained recognition text can be output in a streaming manner, so as to achieve recognition while the user is speaking and output the recognition text in real time. In this way, the latency of outputting the recognition text is reduced, the subjective waiting time of the user is decreased, and the application of real-time interaction scenarios can be supported.

[0081] In the related art, a non-streaming speech recognition technology is adopted, and recognition is performed only after obtaining a complete sentence or paragraph, resulting in a relatively high occupancy of hardware resources. In addition, as the audio length increases, the computational amount and memory occupancy will also increase accordingly. In some solutions, the offline speech recognition model is modified so that the modified model can directly obtain audio stream data for streaming recognition, but the implementation cost of the modification algorithm is relatively large, and the recognition accuracy of the modified model has decreased to varying degrees.

[0082] In the embodiments of the present application, the offline speech recognition model is not modified. Thus, while maintaining a high recognition accuracy of the offline speech recognition model, streaming speech recognition is achieved, the occupancy of hardware resources is reduced, and it can be applied to application scenarios with high requirements for recognition accuracy.

[0083] Please refer to Figure 3 , in some embodiments, the speech recognition method further includes:

[0084] 060: Obtain video stream data and physical button data, where the video stream data includes any one or more of face data and lip movement data;

[0085] At this time, during the process of obtaining the audio stream data, obtain the multi-modal endpoint signal corresponding to the audio stream data (i.e., 020), including:

[0086] 021: Generate a multi-modal endpoint signal based on the audio stream data and the video stream data and / or the physical button data.

[0087] Specifically, as Figure 4 shown, the multi-modal endpoint signal can be directly obtained during the process of obtaining the audio stream data. For example, obtain an externally input multi-modal endpoint signal. The multi-modal endpoint signal can also be generated based on the obtained audio stream data. The following details the specific process of generating the multi-modal endpoint signal based on the obtained audio stream data.

[0088] During the process of obtaining the audio stream data, the video stream data and the physical button data can also be obtained simultaneously. The video stream data includes any one or more of the user's face data and lip movement data. For example, it can include only face data; or only lip movement data; or both face data and lip movement data. The physical button data can include the status data of the physical button. For example, it includes button press status data and button release status data.

[0089] Based on the audio stream data, the multi-modal endpoint signal can be generated in combination with the video stream data. Taking the video stream data including face data and lip movement data as an example, perform feature extraction and detection on the face data and the lip movement data. When no face is detected or the lip movement is in the closed state, set the multi-modal endpoint signal at this time to the silent signal 0. When a face is detected and the lip movement is in the speaking state, set the multi-modal endpoint signal at this time to the start signal 1. After setting the start signal 1, if a face is continuously detected and the lip movement is in the speaking state, set the multi-modal endpoint signal at this time to the intermediate signal 2. After setting the intermediate signal 2, when no face is detected or the lip movement is in the closed state, set the multi-modal endpoint signal at this time to the end signal 3. Of course, other arbitrary methods can also be used to generate the multi-modal endpoint signal by combining the audio stream data and the video stream data.

[0090] Based on the audio stream data, multimodal endpoint signals can be generated in combination with physical button data. For example, when no physical button data is received, the multimodal endpoint signal at this time is set to the silent signal 0. When the button press state data is received, the multimodal endpoint signal at this time is set to the start signal 1. After setting the start signal, or when continuously receiving the button press state data, the multimodal endpoint signal at this time is set to the intermediate signal 2. After setting the intermediate signal 2, if the button release state data is received, the multimodal endpoint signal at this time is set to the end signal 3. Of course, other arbitrary methods can also be used to generate multimodal endpoint signals by combining the audio stream data with the physical button data.

[0091] Based on the audio stream data, multimodal endpoint signals can also be generated in combination with video stream data and physical button data, that is, the current state of the audio stream is determined by comprehensively considering the judgment conditions of the video stream data and the physical button data. For example, when a face is detected, the mouth shape is in the speaking state, and the button press state data is received, the multimodal endpoint signal at this time is set to the start signal 1.

[0092] In the embodiments of the present application, multimodal endpoint signals are generated based on the audio stream data in combination with the video stream data and / or physical button data. Compared with determining the endpoint signal only using the audio signal, more accurate judgment of the start signal and the end signal can be performed, and thus the accuracy and stability of subsequent speech recognition can be improved.

[0093] Please refer to Figure 5 and Figure 6 , in some embodiments, the audio stream data includes multiple audio streams. The speech recognition method further includes:

[0094] 070: Obtain the first audio length of each audio stream, and determine whether the first audio length meets the first preset condition;

[0095] 080: Determine whether the multimodal endpoint signal meets the second preset condition;

[0096] 090: When the first audio length meets the first preset condition and the multimodal endpoint signal meets the second preset condition, remove the silent audio streams from the audio stream data according to the multimodal endpoint signal, and cache the remaining at least one audio stream to obtain the audio to be recognized;

[0097] At this time, the audio stream data is segmented based on the multimodal endpoint signal and the voice activity detection model to obtain at least one audio segment (i.e., 030), including:

[0098] 031: Segment the audio to be recognized based on the multimodal endpoint signal and the voice activity detection model to obtain at least one audio segment.

[0099] Specifically, as Figure 4As shown, after obtaining the audio stream data and the multi-modal endpoint signal, it is necessary to verify the audio stream data and the multi-modal endpoint signal. The verification process is as follows: The audio stream data includes multiple audio streams. Obtain the first audio length of each audio stream and determine whether the first audio length meets the first preset condition. The first audio length can be the binary length, and the first preset condition is set according to the actual application situation.

[0100] In an example, the length of the audio stream is set to a fixed 80 milliseconds, the sampling rate is 16000 Hz, and the corresponding binary length is 2560. If the first audio length of the obtained audio stream is not equal to 2560, it indicates that the audio stream has been damaged, lost, or tampered with during transmission or storage. It is determined that the audio stream is non-compliant and cannot proceed with subsequent steps such as voice segmentation and recognition, thus avoiding subsequent mis-segmentation and mis-recognition situations.

[0101] The verification of the multi-modal endpoint signal is to determine whether the multi-modal endpoint signal meets the second preset condition. It can be understood that the arrangement pattern of the multi-modal endpoint signal is 01230. If the arrangement pattern of the multi-modal endpoint signal is not 01230, it is determined that the multi-modal endpoint signal is non-compliant and cannot proceed with subsequent steps such as voice segmentation and recognition, thus avoiding subsequent mis-segmentation and mis-recognition situations.

[0102] It can be understood that if the multi-modal endpoint signal is directly obtained, for example, directly obtaining the multi-modal endpoint signal from an external input. In this case, the multi-modal endpoint signal is unstable and has a greater possibility of being non-compliant. Therefore, it is necessary to verify the multi-modal endpoint signal. If the multi-modal endpoint signal is generated based on the obtained audio stream data, the stability of the multi-modal endpoint signal is stronger. In this case, the multi-modal endpoint signal does not need to be verified.

[0103] When the first audio length meets the first preset condition and the multi-modal endpoint signal meets the second preset condition, it indicates that both the audio stream data and the multi-modal endpoint signal are compliant. According to the multi-modal endpoint signal, the silent audio stream in the audio stream data can be removed. The silent audio stream is also the audio stream when the multi-modal endpoint signal is the silent signal 0. Cache the remaining audio stream to obtain the audio to be recognized.

[0104] In a real-time voice interaction scenario, to improve the accuracy of speech recognition and reduce the hardware power consumption, the audio to be recognized can be segmented by Voice Activity Detection (VAD) technology first. Based on the voice activity detection model, the audio to be recognized can be segmented in combination with multi-modal endpoint signals. For a relatively short audio to be recognized, segmentation may not occur based on the multi-modal endpoint signals and the voice activity detection model, that is, an audio segment is obtained. For a relatively long audio to be recognized, it can be segmented into smaller audio segments based on the multi-modal endpoint signals and the voice activity detection model, and at least two audio segments can be obtained from one audio to be recognized.

[0105] The segmentation process of the audio to be recognized is described in detail below.

[0106] Please refer to Figure 7 , in some embodiments, during the process of caching at least one remaining audio stream to obtain the audio to be recognized (i.e., 090), the speech recognition method further includes:

[0107] 0100: Obtain the total audio length of the cached audio stream to obtain the second audio length, and obtain the average energy value of the cached audio stream;

[0108] At this time, segmenting the audio to be recognized based on the multi-modal endpoint signals and the voice activity detection model to obtain at least one audio segment (i.e., 031) includes:

[0109] 0311: Determine the dynamic segmentation threshold of the voice activity detection model based on the second audio length and / or the average energy value;

[0110] 0312: Segment the audio to be recognized based on the multi-modal endpoint signals, the voice activity detection model, and the dynamic segmentation threshold to obtain at least one audio segment.

[0111] Specifically, the basic principle of the voice activity detection model is to analyze and calculate the speech coding parameters and their characteristic values, and then use a certain logical judgment criterion to determine whether there is a speech signal in the current audio.

[0112] In some embodiments, the voice activity detection model includes any one of a detection model based on a threshold discrimination method, a detection model based on a model matching method, and a detection model based on deep learning.

[0113] Among them, the detection model based on the threshold discrimination method refers to distinguishing speech and non-speech in the audio segment by setting the threshold value of energy or spectral features. The detection model based on the model matching method refers to using a statistical model to model speech and non-speech in the audio segment and making a discrimination by matching the model. The statistical model is, for example, the Hidden Markov Model (HMM). The detection model based on deep learning refers to using a deep neural network to automatically learn the feature representation of speech and non-speech in the audio segment. In the embodiments of the present application, the detection model based on deep learning is taken as an example for expansion and explanation.

[0114] During the process of caching the audio stream, at least one audio stream is cached in sequence, and the total audio length and average energy value of the cached audio stream are obtained. The total audio length of the cached audio stream is also the second audio length. Based on the second audio length and / or the average energy value, the dynamic segmentation threshold of the voice activity detection model can be determined. The dynamic segmentation threshold can be the energy threshold of the audio.

[0115] The dynamic segmentation threshold of the voice activity detection model can be determined only based on the second audio length; or, the dynamic segmentation threshold of the semantic activity detection model can also be determined only based on the average energy value (for example, when using the detection model based on the threshold discrimination method); or, the dynamic segmentation threshold of the voice activity detection model can also be determined based on the second audio length and the average energy value together.

[0116] After determining the dynamic segmentation threshold, the audio to be recognized is segmented based on the multi-modal endpoint signal, the voice activity detection model, and the dynamic segmentation threshold, and at least one audio segment can be obtained. When the multi-modal endpoint signal is the start signal 1, the corresponding position of the audio to be recognized can be segmented. When the multi-modal endpoint signal is the end signal 3, the corresponding position of the audio to be recognized can be segmented.

[0117] The dynamic segmentation threshold can also be combined with a preset time threshold to segment the audio to be recognized. For example, when the time that the energy of the audio to be recognized continuously remains lower than the dynamic segmentation threshold is greater than or equal to the preset time threshold, the audio to be recognized can be segmented once. In this way, the problems of too long inference time and too high memory occupancy rate caused by too long audio length can be prevented.

[0118] It should be noted that after starting to segment the audio to be recognized based on the multi-modal endpoint signal, the voice activity detection model, and the dynamic segmentation threshold, the audio length of the audio stream in the segmented audio segment does not belong to the total audio length of the cached audio stream. That is to say, after each segmentation, the total audio length of the cached audio stream needs to be re-obtained for the next segmentation.

[0119] Such as Figure 8As shown, no segmentation occurs inside the audio to be recognized, and 1 audio segment is obtained, which is audio segment a1. As Figure 9 shown, 1 segmentation occurs inside the audio to be recognized, and 2 audio segments are obtained, which are audio segment a1 and audio segment a2 respectively. As Figure 10 shown, 2 segmentations occur inside the audio to be recognized, and 3 audio segments are obtained, which are audio segment a1, audio segment a2, and audio segment a3 respectively. Currently, 3, 4, 5, or more segmentations can also occur inside the audio to be recognized, and no further examples will be given here.

[0120] The following details the specific process of determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length.

[0121] Please refer to Figure 11 , in some embodiments, when the dynamic segmentation threshold (i.e., 0311) of the voice activity detection model determined based on the second audio length and / or the average energy value is the dynamic segmentation threshold determined based on the second audio length, determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length includes:

[0122] 03111: When the second audio length is less than or equal to the first length threshold, determine the dynamic segmentation threshold as the first preset threshold;

[0123] 03112: When the second audio length is greater than the first length threshold and less than the second length threshold, the dynamic segmentation threshold is negatively correlated with the second audio length;

[0124] 03113: When the second audio length is greater than or equal to the second length threshold, determine the dynamic segmentation threshold as the second preset threshold;

[0125] wherein, the first preset threshold is greater than the second preset threshold.

[0126] Specifically, when the second audio length is less than or equal to the first length threshold, it indicates that the total length of the cached audio stream is short, and the dynamic segmentation threshold can be determined as the first preset threshold. The first preset threshold is greater than the second preset threshold, that is, a relatively large dynamic segmentation threshold is taken, so that the probability of the audio to be recognized being segmented is small.

[0127] When the second audio length is greater than the first length threshold and less than the second length threshold, the dynamic segmentation threshold is negatively correlated with the second audio length, that is, the longer the second audio length, the smaller the dynamic segmentation threshold, and the greater the probability of being segmented.

[0128] When the second audio length is greater than or equal to the second length threshold, it indicates that the total length of the cached audio stream is long, and the dynamic segmentation threshold can be determined as the second segmentation threshold, that is, a relatively small dynamic segmentation threshold is taken, so that the probability of the audio to be recognized being segmented is large.

[0129] An audio segment with an audio length exceeding the second length threshold is an overly long audio segment, and an overly long audio segment will cause long subsequent speech recognition inference time and high memory occupancy. An audio segment with an audio length lower than the first length threshold is an overly short audio segment, and an overly short audio segment may be noise data that the voice activity detection model cannot filter out, which will cause misrecognition in subsequent speech recognition.

[0130] In one example, according to the second audio length, a dynamic segmentation threshold can be determined according to a piecewise function, and the piecewise function is shown as follows:

[0131]

[0132] Among them, x is the second audio length, with the unit of second. y is the dynamic segmentation threshold. The first length threshold is 5. The second length threshold is 20. The first preset threshold is 1. The second preset threshold is 0. It can be understood that the first preset threshold is 1, that is to say, when the second audio length is less than 5s, the audio to be recognized will not be segmented. The second preset threshold is 0, that is to say, when the second audio length is 20s, the audio to be recognized is forcibly segmented, so that the audio length of the audio segment can be controlled within 5 - 20s.

[0133] In this way, the best segmentation point in the audio to be recognized can be dynamically and accurately identified and located, the audio length of the audio segment can be controlled within a reasonable range, avoiding the audio segment being too long or too short, reducing the inference time and memory occupancy rate, and can also avoid misrecognition caused by the overly short audio segment.

[0134] Please refer to Figure 12 , in some embodiments, performing streaming speech recognition on at least one audio segment based on an offline speech recognition model to obtain a recognition text (i.e., 040), including:

[0135] 041: Determine the end signal of each audio segment;

[0136] 042: When the length of the received audio segment is an even multiple of the sampling rate, or when the end signal of the received audio segment is received, take the current part of the audio segment as a sub-audio segment and input it into the offline speech recognition model;

[0137] 043: Perform streaming speech recognition on the sub-audio segment corresponding to each audio segment based on the offline speech recognition model to obtain a recognition text.

[0138] Specifically, after obtaining at least one audio segment, determine the end signal of each audio segment. According to different segmentation situations of the audio to be recognized, there are three situations for the cache queue where the audio segment is located. In the three different situations, the end signal of each audio segment can be determined respectively.

[0139] In the first case, N - 1 segmentations occur inside the audio to be recognized, resulting in N audio segments (N > 1), including three types of cache queues:

[0140] In the first type of cache queue, it is used to cache audio segment a1. The start signal of audio segment a1 is start signal 1 in the multi-modal endpoint signal, and the end signal is segmentation point b1.

[0141] In the second type of cache queue, it is used to cache audio segments a2 to a(N - 1). The start signal of audio segment a2 is segmentation point b1, and the end signal is segmentation point b2. The start signal of audio segment a3 is segmentation point b2, and the end signal is segmentation point b3.... The start signal of audio segment a(N - 1) is segmentation point b(N - 2), and the end signal is segmentation point b(N - 1).

[0142] In the third type of cache queue, it is used to cache audio segments a(N - 1) and aN. The start signal of audio segment aN is segmentation point b(N - 1), and the end signal is end signal 3 in the multi-modal endpoint signal.

[0143] As Figure 10 shown, the audio to be recognized belongs to the first case.

[0144] In the second case, one segmentation occurs inside the audio to be recognized, resulting in 2 audio segments, including two types of cache queues:

[0145] The first type of cache queue is used to cache audio segment a1. The start signal of audio segment a1 is start signal 1 in the multi-modal endpoint signal, and the end signal is segmentation point b1.

[0146] In the second type of cache queue, it is used to cache audio segment a2. The start signal of audio segment a2 is segmentation point b1, and the end signal is end signal 3 in the multi-modal endpoint signal.

[0147] As Figure 9 shown, the audio to be recognized belongs to the second case.

[0148] In the third case, no segmentation occurs inside the audio to be recognized, resulting in 1 audio segment, including one type of cache queue:

[0149] In the first type of cache queue, it is used to cache audio segment a1. The start signal of audio segment a1 is start signal 1 in the multi-modal endpoint signal, and the end signal is end signal 3 in the multi-modal endpoint signal.

[0150] As Figure 8 shown, the audio to be recognized belongs to the third case.

[0151] It should be noted that in the first case, since multiple segmentations occur within the audio to be recognized, the last audio segment, i.e., audio segment aN, may be blank audio or noise audio. If the third type of cache queue only includes audio segment aN, misrecognition may occur in subsequent speech recognition. Therefore, the third type of cache queue is set to include audio segment a(N-1) and audio segment aN, that is, there is a duplicate audio segment a(N-1) in the second type of cache queue. In this way, the probability of misrecognition in speech recognition is reduced.

[0152] In the three cases, the cache queue is used to cache audio segments in sequence. When the length of the received audio segment is an even multiple of the sampling rate, or when the end signal of the audio segment is received, the current part of the audio segment is taken as the sub-audio segment and input into the offline speech recognition model. The audio segments in multiple cache queues are input into the offline speech recognition model in sequence according to the above method.

[0153] In an example, the sampling rate is 16,000 Hz, that is, 16,000 samples are taken per second. The maximum length of the audio segment is 20 s. When the length of the received audio segment is 2 s, 4 s, 6 s,..., 18 s, 20 s, the current part of the audio segment is taken as the sub-audio segment and input into the offline speech recognition model.

[0154] For example, if the length of an audio segment is 5 s, when the length of the received audio segment is 2 s, the 0-2 s of the audio segment is taken as the first sub-audio segment and input into the offline speech recognition model. When the length of the received audio segment is 4 s, the 0-4 s of the audio segment is taken as the second sub-audio segment and input into the offline speech recognition model. When the length of the received audio segment is 5 s and the end signal is received, the 0-5 s of the audio segment is taken as the third sub-audio segment and input into the offline speech recognition model.

[0155] The sub-audio segments of each audio segment are input into the offline speech recognition model in sequence. The offline speech recognition model performs speech recognition on each input sub-audio segment, so as to realize continuous and real-time streaming speech recognition and obtain the recognition text. Compared with complex transformation of the streaming model, the offline speech recognition model has higher flexibility and adaptability, can quickly adapt to a variety of non-streaming models, not only greatly reduces the workload of engineering development, but also can effectively avoid the accuracy loss caused by transforming the streaming model, and is more efficient and reliable in practical applications.

[0156] Please refer to Figure 13 and Figure 14 , in some embodiments, the multi-modal endpoint signal includes an end signal, and the recognition text includes an intermediate recognition text and a result recognition text. Based on the offline speech recognition model, streaming speech recognition is performed on the sub-audio segments corresponding to each audio segment to obtain the recognition text (i.e., 043), including:

[0157] 0431: Perform streaming speech recognition on each sub-audio segment based on an offline speech recognition model to obtain a corresponding sub-recognition text;

[0158] 0432: When the sub-audio segment does not include an end signal, update the recognition text according to the corresponding sub-recognition text, mark a first recognition character after the recognition text, and mark the recognition text as the first recognition state;

[0159] 0433: When the sub-audio segment includes an end signal, update the recognition text according to the corresponding sub-recognition text, mark a second recognition character after the recognition text, and mark the recognition text as the second recognition state;

[0160] At this time, stream output the recognition text (i.e., 050), including:

[0161] 051: When the sub-audio segment does not include an end signal, output the recognition text marked as the first recognition state as the intermediate recognition text;

[0162] 052: When the sub-audio segment includes an end signal, output the recognition text marked as the second recognition state as the result recognition text.

[0163] Specifically, the sub-audio segments of each audio segment are sequentially input into the offline speech recognition model, and the offline speech recognition model performs speech recognition on each input sub-audio segment to obtain a corresponding sub-recognition text.

[0164] Taking the audio to be recognized including 3 audio segments as an example, in the implementation manner of the foregoing step "041", the 3 audio segments belong to the first case, and a total of three types of cache queues are formed. The audio segment a1 belongs to the first type of cache queue. For the first sub-audio segment of the audio segment a1, the corresponding sub-recognition text is the initial recognition text. For each sub-audio segment in the audio segment a1, when the sub-audio segment does not include an end signal, cover the previously obtained recognition text according to the corresponding sub-recognition text, mark a first recognition character after the recognition text, and mark the recognition text as the first recognition state. The first recognition character can be set according to the actual situation. In one example, the first recognition character is "|1".

[0165] It can be understood that when the sub-recognition text covers the previously obtained recognition text, the first recognition character will also be covered. When receiving the segmentation point b1 when recognizing the last audio segment of the audio segment a1, after the audio segment a1 is recognized, cache the recognition text corresponding to the audio segment a1.

[0166] The audio segment a2 belongs to the second type of cache queue. For the first sub-audio segment in the audio segment a2, the corresponding sub-recognition text is added after the cached recognition text (the recognition text corresponding to the audio segment a1). For each sub-audio segment in the audio segment a2, when the sub-audio segment does not include an end signal, the previously obtained recognition text is overwritten according to the corresponding sub-recognition text, and a first recognition character is marked after the recognition text, and the recognition text is marked as the first recognition state.

[0167] It can be understood that the recognition text corresponding to the cached audio segment a1 will not be overwritten at this time. When the last audio segment of the audio segment a2 is recognized and the segmentation point b2 is received, after the audio segment a2 is recognized, the recognition text corresponding to the audio segment a2 is added after the recognition text corresponding to the cached audio segment a1.

[0168] In addition, the audio segment a2 also belongs to the third type of cache queue, and the third type of cache queue includes the audio segment a2 and the audio segment a3. When performing speech recognition on the audio segments in the third type of cache queue, the audio segment a2 will be recognized again and then the audio segment a3 will be recognized. The last sub-audio segment of the audio segment a3 includes an end signal 3. To avoid duplicate recognition text, the recognition text corresponding to the audio segment a2 is deleted from the cached recognition text, and the recognition texts corresponding to the audio segment a2 and the audio segment a3 in the third type of cache queue are added. A second recognition character is marked after the cached recognition text, and the recognition text is marked as the second recognition state. The second recognition character can be set according to the actual situation. In one example, the second recognition character is "|2".

[0169] During the process of speech recognition by the offline speech recognition model, the obtained sub-recognition texts can be output sequentially. The output of the sub-recognition texts includes three types:

[0170] The first output type includes outputting the sub-recognition text c1 corresponding to the sub-audio segment when the output audio length is a multiple of 2s, outputting the sub-recognition text c2 corresponding to the sub-audio segment including the segmentation point, and outputting the sub-recognition text c3 corresponding to the sub-audio segment including the end signal.

[0171] The second output type includes outputting the sub-recognition text c1 corresponding to the sub-audio segment including the segmentation point, and outputting the sub-recognition text c2 corresponding to the sub-audio segment including the end signal.

[0172] The third output type includes outputting the sub-recognition text c1 corresponding to the sub-audio segment including the end signal.

[0173] The recognized text output includes two states, namely the above-mentioned first recognition state and the second recognition state. The first recognition state indicates that the audio to be recognized is in the process of recognition, and the second recognition state indicates that the recognition of the audio to be recognized is completed. When the sub-audio segment does not include an end signal, the recognized text marked with the first recognition state is sequentially output as the intermediate recognized text. In this way, the speech recognition text can be instantaneously fed back, enabling the user to obtain the intermediate result of speech recognition while speaking, thereby improving the user experience. When the sub-audio segment includes an end signal, the recognized text marked with the second recognition state is output as the result recognized text.

[0174] In a real-time interaction scenario, the marking of the second recognition state enables the triggering of a call signal for the downstream question-and-answer module to achieve interaction.

[0175] Please refer to Figure 15 , an embodiment of the present application further provides a speech recognition device 100. The speech recognition device 100 includes a data processing module 10, a speech segmentation module 20, a streaming recognition module 30, and a streaming output module 40. The data processing module 10 is used to obtain audio stream data. The data processing module 10 is further used to obtain a multi-modal endpoint signal corresponding to the audio stream data during the process of obtaining the audio stream data. The speech segmentation module 20 is used to segment the audio stream data based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment. The streaming recognition module 30 is used to perform streaming speech recognition on at least one audio segment based on an offline speech recognition model to obtain recognized text. The streaming output module 40 is used to stream and output the recognized text.

[0176] In some embodiments, the data processing module 10 is further used to obtain video stream data and physical button data, where the video stream data includes any one or more of face data and lip movement data. At this time, the obtaining module is specifically used to generate a multi-modal endpoint signal based on the audio stream data and the video stream data and / or the physical button data.

[0177] In some embodiments, the audio stream data includes multiple audio streams. The data processing module 10 is further used to obtain the first audio length of each audio stream and determine whether the first audio length meets a first preset condition; determine whether the multi-modal endpoint signal meets a second preset condition; when the first audio length meets the first preset condition and the multi-modal endpoint signal meets the second preset condition, remove the silent audio stream from the audio stream data according to the multi-modal endpoint signal and cache the remaining at least one audio stream to obtain the audio to be recognized. The speech segmentation module 20 is specifically used to segment the audio to be recognized based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment.

[0178] In some embodiments, during the process of caching at least one remaining audio stream to obtain the audio to be recognized, the speech segmentation module 20 is further configured to obtain the total audio length of the cached audio stream to obtain a second audio length, and obtain the average energy value of the cached audio stream. At this time, the speech segmentation module 20 is specifically configured to determine a dynamic segmentation threshold of the voice activity detection model based on the second audio length and / or the average energy value; segment the audio to be recognized based on the multi-modal endpoint signal, the voice activity detection model, and the dynamic segmentation threshold to obtain at least one audio segment.

[0179] In some embodiments, when determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length and / or the average energy value is to determine the dynamic segmentation threshold of the voice activity detection model based on the second audio length, the speech segmentation module 20 is specifically configured to determine that the dynamic segmentation threshold is a first preset threshold when the second audio length is less than or equal to a first length threshold; when the second audio length is greater than the first length threshold and less than a second length threshold, the dynamic segmentation threshold is negatively correlated with the second audio length; when the second audio length is greater than or equal to the second length threshold, determine that the dynamic segmentation threshold is a second preset threshold; wherein, the first preset threshold is greater than the second preset threshold.

[0180] In some embodiments, the voice activity detection model includes any one of a detection model based on a threshold discrimination method, a detection model based on a model matching method, and a detection model based on deep learning.

[0181] In some embodiments, the streaming recognition module 30 is specifically configured to determine the end signal of each audio segment; when the length of the received audio segment is an even multiple of the sampling rate or the end signal of the received audio segment is received, take the current part of the audio segment as a sub-audio segment and input it into the offline speech recognition model; perform streaming speech recognition on the sub-audio segment corresponding to each audio segment based on the offline speech recognition model to obtain the recognition text.

[0182] In some embodiments, the multi-modal endpoint signal includes an end signal, and the recognized text includes intermediate recognized text and result recognized text. The streaming recognition module 30 is specifically configured to perform streaming speech recognition on each sub-audio segment based on an offline speech recognition model to obtain a corresponding sub-recognized text; when the sub-audio segment does not include an end signal, update the recognized text according to the corresponding sub-recognized text, mark a first recognition character after the recognized text, and mark the recognized text as a first recognition state; when the sub-audio segment includes an end signal, update the recognized text according to the corresponding sub-recognized text, mark a second recognition character after the recognized text, and mark the recognized text as a second recognition state. At this time, the streaming output module 40 is specifically configured to, when the sub-audio segment does not include an end signal, output the recognized text marked as the first recognition state as the intermediate recognized text; when the sub-audio segment includes an end signal, output the recognized text marked as the second recognition state as the result recognized text.

[0183] It should be noted that the explanations of the speech recognition method in the foregoing embodiments also apply to the speech recognition device 100 of the embodiments of the present application, and will not be elaborated here.

[0184] Please refer to Figure 16 , the embodiments of the present application also provide a speech recognition system 200. The speech recognition system 200 includes one or more processors 210 and a memory 220. The memory 220 stores a computer program. When the computer program is executed by the processor 210, the speech recognition method of any of the foregoing embodiments is implemented.

[0185] For example, when the computer program is executed by the processor 210, the following speech recognition method is implemented:

[0186] 010: Obtain audio stream data;

[0187] 020: During the process of obtaining the audio stream data, obtain the multi-modal endpoint signal corresponding to the audio stream data;

[0188] 030: Segment the audio stream data based on the multi-modal endpoint signal and a voice activity detection model to obtain at least one audio segment;

[0189] 040: Perform streaming speech recognition on at least one audio segment based on an offline speech recognition model to obtain recognized text;

[0190] 050: Stream and output the recognized text.

[0191] For another example, when the computer program is executed by the processor 210, the following speech recognition method is implemented:

[0192] 060: Obtain video stream data and physical button data, where the video stream data includes any one or more of face data and lip movement data;

[0193] At this time, in the process of obtaining the audio stream data, obtain the multi-modal endpoint signal corresponding to the audio stream data (i.e., 020), including:

[0194] 021: Generate a multi-modal endpoint signal based on the audio stream data and the video stream data and / or physical button data.

[0195] It should be noted that the explanations of the speech recognition method and the speech recognition device 100 in the foregoing embodiments are equally applicable to the speech recognition system 200 of the embodiments of the present application, and will not be elaborated here.

[0196] Please refer to Figure 17 , the embodiments of the present application further provide a computer-readable storage medium 300, on which a computer program 310 is stored. When the program is executed by a processor 320, the speech recognition method of any of the foregoing embodiments is implemented.

[0197] For example, when the computer program 310 is executed by the processor 320, the following speech recognition method is implemented:

[0198] 010: Obtain audio stream data;

[0199] 020: In the process of obtaining the audio stream data, obtain the multi-modal endpoint signal corresponding to the audio stream data;

[0200] 030: Segment the audio stream data based on the multi-modal endpoint signal and the voice activity detection model to obtain at least one audio segment;

[0201] 040: Perform streaming speech recognition on at least one audio segment based on an offline speech recognition model to obtain a recognized text;

[0202] 050: Stream output the recognized text.

[0203] For another example, when the computer program 310 is executed by the processor 320, the following speech recognition method is implemented:

[0204] 060: Obtain video stream data and physical button data, where the video stream data includes any one or more of face data and lip movement data;

[0205] At this time, in the process of obtaining the audio stream data, obtain the multi-modal endpoint signal corresponding to the audio stream data (i.e., 020), including:

[0206] 021: Generate a multi-modal endpoint signal based on the audio stream data and the video stream data and / or physical button data.

[0207] It should be noted that the explanations of the speech recognition method and the speech recognition device 100 in the foregoing embodiments are equally applicable to the computer-readable storage medium 300 of the embodiments of the present application, and will not be elaborated herein.

[0208] In summary, in the speech recognition method, the speech recognition device 100, the speech recognition system 200, and the computer-readable storage medium 300 of the embodiments of the present application, in the process of acquiring the audio stream data, the multi-modal endpoint signal corresponding to the audio stream data is acquired, and the audio stream data is segmented based on the multi-modal endpoint signal and the voice activity detection model to obtain at least one audio segment. Then, the segmented audio segment is subjected to streaming speech recognition based on the offline speech recognition model to obtain the recognition text, and the recognition text is output in a streaming manner. In this way, the delay of outputting the recognition text is reduced, the occupation of hardware resources is reduced, and it can be applied to real-time interaction scenarios and scenarios with high requirements for recognition accuracy.

[0209] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0210] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0211] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered as a definable sequence of executable instructions for implementing logical functions, and can be embodied specifically in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection part having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage medium can even be paper or other suitable media on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then storing it in a computer memory.

[0212] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0213] Those of ordinary skill in the art can understand that all or part of the steps carried out in the method of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments. In addition, in each of the embodiments of the present application, each functional unit can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk, an optical disc, etc.

[0214] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A speech recognition method, characterized in that: include: Get audio stream data; In the process of acquiring the audio stream data, acquiring a multimodal endpoint signal corresponding to the audio stream data; Segmenting the audio stream data based on the multimodal endpoint signal and the voice activity detection model to obtain at least one audio segment; Performing streaming speech recognition on the at least one audio segment based on an offline speech recognition model to obtain a recognized text; The recognized text is output in a streaming manner.

2. The speech recognition method according to claim 1, characterized in that: The speech recognition method further comprises: Acquire video stream data and physical button data, wherein the video stream data includes any one or more of face data and lip shape data; In the process of acquiring the audio stream data, acquiring the multimodal endpoint signal corresponding to the audio stream data includes: The multimodal endpoint signal is generated based on the audio stream data and the video stream data and / or the physical button data.

3. The speech recognition method according to claim 1, characterized in that: The audio stream data includes a plurality of audio streams, and the speech recognition method further includes: Obtaining a first audio length of each of the audio streams, and determining whether the first audio length meets a first preset condition; Determining whether the multimodal endpoint signal satisfies a second preset condition; When the first audio length meets the first preset condition and the multimodal endpoint signal meets the second preset condition, removing the silent audio stream in the audio stream data according to the multimodal endpoint signal, and caching the remaining at least one audio stream to obtain the audio to be recognized; The step of segmenting the audio stream data based on the multimodal endpoint signal and the voice activity detection model to obtain at least one audio segment includes: The audio to be recognized is segmented based on the multimodal endpoint signal and the voice activity detection model to obtain the at least one audio segment.

4. The speech recognition method according to claim 3, characterized in that: In the process of obtaining the audio to be recognized by caching at least one of the remaining audio streams, the speech recognition method further includes: Obtaining a total audio length of the cached audio stream, obtaining a second audio length, and obtaining an average energy value of the cached audio stream; The step of segmenting the audio to be recognized based on the multimodal endpoint signal and the voice activity detection model to obtain the at least one audio segment includes: Determine a dynamic segmentation threshold of the voice activity detection model based on the second audio length and / or the average energy value; The audio to be recognized is segmented based on the multimodal endpoint signal, the voice activity detection model and the dynamic segmentation threshold to obtain the at least one audio segment.

5. The speech recognition method according to claim 4, characterized in that: When the determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length and / or the average energy value is determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length, the determining the dynamic segmentation threshold of the voice activity detection model based on the second audio length includes: When the second audio length is less than or equal to a first length threshold, determining the dynamic segmentation threshold to be a first preset threshold; When the second audio length is greater than the first length threshold and less than the second length threshold, the dynamic segmentation threshold is negatively correlated with the second audio length; When the second audio length is greater than or equal to the second length threshold, determining the dynamic segmentation threshold to be a second preset threshold; Wherein, the first preset threshold is greater than the second preset threshold.

6. The speech recognition method according to any one of claims 1 to 5, characterized in that: The voice activity detection model includes any one of a detection model based on a threshold discrimination method, a detection model based on a model matching method, and a detection model based on deep learning.

7. The speech recognition method according to claim 1, characterized in that: The step of performing streaming speech recognition on the at least one audio segment based on an offline speech recognition model to obtain a recognized text includes: Determining an end point signal of each of the audio segments; When the length of the received audio segment is an even multiple of the sampling rate, or when an end signal of the audio segment is received, taking a current portion of the audio segment as a sub-audio segment and inputting it into the offline speech recognition model; Based on the offline speech recognition model, streaming speech recognition is performed on the sub-audio segment corresponding to each of the audio segments to obtain the recognition text.

8. The speech recognition method according to claim 7, characterized in that: The multimodal endpoint signal includes an end signal, the recognition text includes an intermediate recognition text and a result recognition text, and the step of performing streaming speech recognition on the sub-audio segment corresponding to each audio segment based on the offline speech recognition model to obtain the recognition text includes: Performing streaming speech recognition on each of the sub-audio segments based on the offline speech recognition model to obtain a corresponding sub-recognition text; When the sub-audio segment does not include the end signal, updating the recognition text according to the corresponding sub-recognition text, marking a first recognition character after the recognition text, and marking the recognition text as a first recognition state; When the sub-audio segment includes the end signal, updating the recognition text according to the corresponding sub-recognition text, marking a second recognition character after the recognition text, and marking the recognition text as a second recognition state; The streaming output of the recognized text includes: When the sub-audio segment does not include the end signal, outputting the recognized text marked as the first recognition state as an intermediate recognized text; When the sub-audio segment includes the end signal, the recognized text marked as the second recognition state is output as the result recognition text.

9. A speech recognition device, characterized in that: include: A data processing module, used for acquiring audio stream data; The data processing module is further used to obtain a multimodal endpoint signal corresponding to the audio stream data during the process of obtaining the audio stream data; A voice segmentation module, configured to segment the audio stream data based on the multimodal endpoint signal and the voice activity detection model to obtain at least one audio segment; A streaming recognition module, configured to perform streaming speech recognition on the at least one audio segment based on an offline speech recognition model to obtain a recognized text; A streaming output module is used to stream output the recognized text.

10. A speech recognition system, characterized in that: The speech recognition system includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the speech recognition method according to any one of claims 1 to 8 is implemented.

Images