Speech recognition method and apparatus, server, and storage medium

Abstract

The embodiment of the application discloses a speech recognition method and device, a server and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: in the process of a speech dialogue, storing audio frames in a received speech data stream in a data stream queue; in the process of real-time detection on the audio frames in the data stream queue, in response to the detection of a first audio frame in the data stream queue, filtering the audio frames before the first audio frame in the data stream queue, the first audio frame being the first audio frame containing sound in the data stream queue; and performing speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue, to obtain a speech recognition result. The scheme provided by the embodiment of the application can complete speech recognition on the audio frames in the speech data stream as soon as the audio frames are received, realize synchronous recognition of a speech stream, and further ensure the speech recognition efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech recognition method, device, server, and storage medium. Background Technology

[0002] With the development of computer technology, voice dialogue is becoming increasingly common in people's daily lives. In many scenarios, speech recognition is needed to convert speech into text, thereby enabling other functions, such as recording text for real-time recording of voice dialogue, or generating control commands via voice. Currently, the usual approach is to record the entire voice data first, and then perform speech recognition on the recorded data to obtain the speech recognition result. However, this method of speech recognition is inefficient. Summary of the Invention

[0003] This application provides a speech recognition method, apparatus, server, and storage medium, which can improve speech recognition efficiency. The technical solution is as follows:

[0004] On the one hand, a speech recognition method is provided, the method comprising:

[0005] During a voice conversation, the audio frames in the received voice data stream are stored in a data stream queue.

[0006] During the real-time detection of audio frames in the data stream queue, in response to the detection of the first audio frame in the data stream queue, the audio frames in the data stream queue preceding the first audio frame are filtered, where the first audio frame is the first audio frame containing sound in the data stream queue.

[0007] Speech recognition is performed on the audio frames in the filtered data stream queue according to their order to obtain the speech recognition result.

[0008] On the other hand, a voice recognition device is provided, the device comprising:

[0009] The storage module is used to store audio frames from the received voice data stream in a data stream queue during a voice conversation.

[0010] The filtering module is used to filter audio frames in the data stream queue before the first audio frame in the data stream queue in response to the detection of the first audio frame in the data stream queue during the real-time detection of audio frames in the data stream queue. The first audio frame is the first audio frame containing sound in the data stream queue.

[0011] The recognition module is used to perform speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue, and obtain the speech recognition result.

[0012] In one possible implementation, the filtering module is configured to, in response to detecting the first audio frame, filter out audio frames in the data stream queue that precede the first audio frame and whose interval duration with respect to the first audio frame is greater than a first duration.

[0013] In another possible implementation, the storage module is configured to, when the duration of an audio frame in the voice data stream is less than a second duration, determine the ratio of the second duration to the duration of the audio frame as the synthesis quantity; synthesize multiple consecutive audio frames in the voice data stream, and store the synthesized audio frames in the data stream queue, wherein the number of the multiple consecutive audio frames is the synthesis quantity.

[0014] In another possible implementation, the device further includes:

[0015] The acquisition module is configured to acquire a second duration configured for the first speech recognition service when a first speech recognition service is assigned among multiple speech recognition services for the speech dialogue, wherein the first speech recognition service is used to recognize audio frames of the second duration.

[0016] The recognition module is used to perform speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue through the first speech recognition service, and obtain the speech recognition result.

[0017] In another possible implementation, the device further includes:

[0018] The deletion module is used to, in the process of real-time detection of audio frames in the data stream queue, delete the second audio frame in response to the detection of the second audio frame in the data stream queue, and not continue to store audio frames in the voice data stream in the data stream queue, wherein the second audio frame is the first audio frame without sound after the first audio frame in the voice data stream.

[0019] In another possible implementation, the voice dialogue is a human-computer voice dialogue, and the storage module is used to receive the voice data stream input by the terminal participating in the human-computer voice dialogue in the current dialogue round during the human-computer voice dialogue, and store the audio frames in the voice data stream of the current dialogue round in the data stream queue of the current dialogue round.

[0020] In another possible implementation, the recognition module is configured to sequentially perform speech recognition on each audio frame in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue, and obtain the recognition result of each audio frame; when the recognition result of the third audio frame is obtained, the recognition results of the first audio frame, the third audio frame, and the audio frames between the first audio frame and the third audio frame are concatenated to obtain the speech recognition result, wherein the third audio frame is the last audio frame containing sound in the data stream queue after the first audio frame;

[0021] The device further includes:

[0022] The sending module is used to send voice data for the next dialogue round to the terminal based on the speech recognition result.

[0023] In another possible implementation, the storage module is configured to, during the human-machine voice dialogue, in response to the start of the current dialogue turn, create a data stream queue for the current dialogue turn; receive the voice data stream input by the terminal in the current dialogue turn in real time, and store the audio frames in the voice data stream of the current dialogue turn in the data stream queue of the current dialogue turn.

[0024] On the other hand, a server is provided, the server including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the speech recognition method as described above.

[0025] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to perform the operations performed by the speech recognition method as described above.

[0026] In another aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, performs the operations performed by the speech recognition method described above.

[0027] The solution provided in this application stores the voice data stream of the voice dialogue in a data stream queue during the voice dialogue process. Audio frames in the data stream queue are detected to determine if they contain sound. Once the first audio frame containing sound is detected, the audio frames preceding the first audio frame in the data stream queue are filtered to remove audio frames that do not contain sound. Simultaneously, speech recognition is performed on the filtered audio frames in the data stream queue. This eliminates the need for speech recognition on audio frames that do not contain sound. Furthermore, speech recognition can be performed on audio frames in the data stream queue simultaneously while storing the audio frames of the voice data stream, without waiting for all audio frames in the voice data stream to be received. This allows for simultaneous speech recognition of audio frames as soon as they are received, thus ensuring high speech recognition efficiency. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the structure of an implementation environment provided in an embodiment of this application;

[0030] Figure 2 This is a flowchart of a speech recognition method provided in an embodiment of this application;

[0031] Figure 3 This is a flowchart of another speech recognition method provided in the embodiments of this application;

[0032] Figure 4 This is a flowchart of another speech recognition method provided in the embodiments of this application;

[0033] Figure 5 This is a flowchart of another speech recognition method provided in the embodiments of this application;

[0034] Figure 6 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of another speech recognition device provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0038] The terms "first," "second," etc., used in this application may be used to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first audio frame may be referred to as a second audio frame, and similarly, a second audio frame may be referred to as a first audio frame.

[0039] As used in this application, the terms "at least one," "multiple," "each," and "any" mean that at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any means refers to any one of the multiple. For example, multiple audio frames include three audio frames, where each refers to each of the three audio frames, and any means refers to any one of the three audio frames, which could be the first audio frame, the second audio frame, or the third audio frame.

[0040] It should be noted that all data involved in this application (including but not limited to voice data streams, audio frames, and speech recognition results) has been authorized by the user or by all parties, and the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the voice data streams involved in this application were obtained with full authorization.

[0041] The speech recognition method provided in this application can be executed by a server. Optionally, the server is a single server, a server cluster consisting of several servers, or a cloud computing service center.

[0042] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes a first terminal 101 and a server 102. The first terminal 101 and the server 102 are connected via a wireless or wired network.

[0043] The first terminal 101 is used to participate in voice dialogue. The server 102 is used to receive the voice data stream of the voice dialogue during the voice dialogue in which the first terminal 101 participates, and to perform speech recognition on the audio frames in the voice data stream.

[0044] Optionally, if the first terminal 101 is used to participate in human-computer voice dialogue, then the server 102 is used to provide human-computer voice dialogue services. The server 102 interacts with the first terminal 101 to achieve human-computer voice dialogue with the first terminal 101. During the human-computer voice dialogue, the server 102 sends the voice data of the current dialogue round to the first terminal 101, which then plays the voice data. The first terminal 101 then obtains the voice data stream input by the user in the current dialogue round and sends it to the server 102. The server 102 performs speech recognition on the audio frames in the voice data stream input by the user in the current dialogue round to obtain the speech recognition result. Based on the speech recognition result, the server 102 can then send the voice data of the next dialogue round to the first terminal 101, thus realizing intelligent multi-turn human-computer dialogue.

[0045] Optionally, the first terminal 101 may install an application provided by the server 102. This application can be any application, such as a calling application or an instant messaging application. The first terminal 102 interacts with the server 102 through this application to achieve human-computer voice dialogue.

[0046] Optionally, the implementation environment also includes a second terminal 103, which is connected to the server 102 via a wireless or wired network. The first terminal 101 and the second terminal 103 engage in voice conversation, that is, the user of the first terminal 101 and the user of the second terminal 103 conduct a voice conversation. During the voice conversation between the first terminal 101 and the second terminal 103, the server 102 can receive the voice data stream of the voice conversation and perform speech recognition on the voice data stream.

[0047] Optionally, both the first terminal 101 and the second terminal 103 have an application provided by the server 102 installed. This application can be any application, such as a calling application or an instant messaging application. The first terminal 102 and the second terminal 103 conduct voice conversations through this application. During the voice conversation between the first terminal 102 and the second terminal 103 through this application, the server 102 can receive the voice data stream of the voice conversation and perform voice recognition on the voice data stream.

[0048] Figure 2 This is a flowchart of a speech recognition method provided in an embodiment of this application, executed by a server, such as... Figure 2 As shown, the method includes:

[0049] 201. During a voice conversation, the server stores the audio frames in the received voice data stream in a data stream queue.

[0050] In this embodiment of the application, during a voice conversation, the server receives the voice data stream of the voice conversation and stores the audio frames in the voice data stream in a data stream queue to achieve caching of the voice data stream so that real-time speech recognition can be performed on the voice data stream of the voice conversation in the future.

[0051] The voice dialogue can be of any type, such as a multi-party dialogue, a dialogue involving multiple users, or a human-computer dialogue involving at least one user and an intelligent chatbot. The voice data stream exists as a sequence of audio frames; that is, during the voice dialogue, the voice data stream received by the server is a sequence of audio frames. A data stream queue is used to store the voice data stream of the voice dialogue.

[0052] 202. During the real-time detection of audio frames in the data stream queue, in response to the detection of the first audio frame in the data stream queue, the server filters the audio frames in the data stream queue before the first audio frame. The first audio frame is the first audio frame in the data stream queue that contains sound.

[0053] In this embodiment of the application, during the voice dialogue, audio frames in the voice data stream are stored into the data stream queue frame by frame. During the process of storing the audio frames in the voice data stream into the data stream queue frame by frame, the audio frames in the data stream queue are also detected in real time according to the order of the audio frames in the data stream queue to detect whether the audio frames contain sound. When the first audio frame containing sound is detected in the data stream queue, it is called the first audio frame. Then, the audio frames in the data stream queue before the first audio frame do not contain sound. The audio frames between the first audio frames in the data stream queue are filtered to filter out the audio frames in the data stream queue that do not contain sound.

[0054] 203. The server performs speech recognition on the audio frames in the filtered data stream queue according to their order, and obtains the speech recognition results.

[0055] In this embodiment, when filtering the data stream queue, the server also stores audio frames from the voice data stream into the data stream queue. After filtering the data stream queue, the server immediately performs speech recognition on the audio frames in the filtered data stream queue in the same order. This allows for simultaneous speech recognition of the audio frames in the data stream queue while storing the audio frames of the voice data stream, without waiting for all audio frames in the voice data stream to be received. This enables speech recognition of audio frames as soon as they are received, achieving synchronous speech stream recognition.

[0056] The solution provided in this application stores the voice data stream of the voice dialogue in a data stream queue during the voice dialogue process. Audio frames in the data stream queue are detected to determine if they contain sound. Once the first audio frame containing sound is detected, the audio frames preceding the first audio frame in the data stream queue are filtered to remove audio frames that do not contain sound. Simultaneously, speech recognition is performed on the filtered audio frames in the data stream queue. This eliminates the need for speech recognition on audio frames that do not contain sound. Furthermore, speech recognition can be performed on audio frames in the data stream queue simultaneously while storing the audio frames of the voice data stream, without waiting for all audio frames in the voice data stream to be received. This allows for simultaneous speech recognition of audio frames as soon as they are received, thus ensuring high speech recognition efficiency.

[0057] In the above Figure 2 Based on the illustrated embodiment, in this embodiment of the application, the server synthesizes the audio frames in the voice data stream and stores them in the data stream queue to ensure that the duration of the audio frames in the data stream queue is large enough to ensure the efficiency of subsequent speech recognition. For details of the process, please refer to the following embodiment.

[0058] Figure 3 This is a flowchart of a speech recognition method provided in an embodiment of this application. The method is executed by a server and a first terminal, such as... Figure 3 As shown, the method includes:

[0059] 301. During a voice conversation, the server receives the voice data stream of the voice conversation. If the duration of the audio frame in the voice data stream is less than the second duration, the ratio of the second duration to the duration of the audio frame is determined as the number of synthesized data.

[0060] The second duration can be any duration, for example, 40 milliseconds. The duration of the audio frames in the voice data stream can be any duration, for example, 10 milliseconds. In this embodiment, during a voice dialogue, the server receives the voice data stream, which exists in the form of an audio frame sequence. If the duration of an audio frame in the voice data stream is less than the second duration, the ratio of the second duration to the duration of the audio frame is determined as the number of frames to be synthesized, so that the audio frames in the voice data stream can be synthesized into audio frames of the second duration and stored in the data stream queue.

[0061] In one possible implementation, the process of determining the number of synthesized components includes: rounding down the ratio of the second duration to the duration of the audio frame to obtain the number of synthesized components; or, rounding up the ratio of the second duration to the duration of the audio frame to obtain the number of synthesized components; or, rounding the ratio of the second duration to the duration of the audio frame to obtain the number of synthesized components.

[0062] In the embodiments of this application, the number of synthesized components is determined according to any of the above methods to ensure that the number of synthesized components is an integer, so that the audio frames can be accurately synthesized into audio frames with a longer duration in the future, thereby ensuring the feasibility of audio frame synthesis.

[0063] In one possible implementation, the process of determining the second duration includes: in the case of assigning a first speech recognition service among multiple speech recognition services for a voice dialogue, obtaining a second duration configured for the first speech recognition service, which is used to recognize audio frames of the second duration.

[0064] In this embodiment, the server is configured with multiple speech recognition services. Each speech recognition service is used to perform speech recognition on audio frames in the speech data stream of a speech dialogue. During the speech dialogue, a speech recognition service is assigned to each dialogue so that the audio frames in the speech data stream of the assigned speech dialogue can be recognized. Different speech recognition services are used to recognize audio frames of different durations. Therefore, when it is determined that a first speech recognition service is assigned to the speech dialogue, a second duration configured for the first speech recognition service is obtained so that the audio frames in the speech data stream can be synthesized into audio frames of the second duration, so that the synthesized audio frames can be recognized by the first speech recognition service.

[0065] In this embodiment of the application, when a first speech recognition service is assigned to a voice dialogue, the second duration configured by the first speech recognition service is obtained so that audio frames in the voice data stream can be synthesized into audio frames of the second duration, so that the first speech recognition service can recognize the synthesized audio frames and ensure the accuracy of subsequent speech recognition.

[0066] 302. The server synthesizes multiple consecutive audio frames in the voice data stream and stores the synthesized audio frames in the data stream queue. The number of consecutive audio frames is the number of synthesized frames.

[0067] In this embodiment of the application, the voice data stream is a sequence of audio frames. When the voice data stream is received, starting from the first audio frame in the voice data stream, a number of consecutively synthesized audio frames are synthesized into one audio frame, and the synthesized audio frame is stored in the data stream queue so that the duration of the audio frame stored in the data stream queue matches the second duration.

[0068] For example, if the number of synthesized frames is 3, then starting from the first audio frame in the speech data stream, the first 3 audio frames are synthesized into one audio frame, and the synthesized audio frame is stored in the data stream queue; the 4th to 6th audio frames are synthesized into one audio frame, and the synthesized audio frame is stored in the data stream queue; the 7th to 9th audio frames are synthesized into one audio frame, and the synthesized audio frame is stored in the data stream queue, and so on.

[0069] In this embodiment, when the duration of an audio frame in the voice data stream is less than the second duration, consecutive audio frames in the voice data stream are combined into an audio frame of the second duration to ensure that the duration of the audio frames stored in the data stream queue is large enough so that when performing voice recognition on the audio frames in the data stream queue, the number of video frames to be recognized is reduced, thereby improving the voice recognition efficiency.

[0070] It should be noted that the embodiments of this application are illustrated using the example that the duration of the audio frame in the voice data stream is less than the second duration. In another embodiment, it is not necessary to perform the above steps 301-302. Instead, other methods are adopted to store the audio frame in the received voice data stream in the data stream queue.

[0071] 303. During the real-time detection of audio frames in the data stream queue, in response to the detection of the first audio frame, the server filters out audio frames in the data stream queue that are earlier than the first audio frame and whose interval length between them is longer than the first duration. The first audio frame is the first audio frame containing sound in the data stream queue.

[0072] In this embodiment, during the real-time detection of audio frames in the data stream queue, once the first audio frame containing sound is detected in the data stream queue, audio frames in the data stream queue that precede the first audio frame and whose interval length with the first audio frame is greater than a first duration are filtered out. This is to filter out audio frames in the data stream queue that do not contain sound. Considering that audio frames before the first audio frame may contain sound but have not been detected, audio frames that precede the first audio frame and whose interval length with the first audio frame is not greater than the first duration are retained. This is to filter out audio frames that do not contain sound as much as possible, while retaining audio frames that may contain sound, so as to ensure the integrity of the audio frames in the filtered data stream queue and ensure that the effective range of voice data is not lost, thereby ensuring the accuracy of subsequent speech recognition.

[0073] The first duration can be any duration, for example, the first duration is 0.5 seconds.

[0074] Optionally, the process of obtaining the first duration includes: in the case of assigning a first speech recognition service among multiple speech recognition services for a speech dialogue, obtaining the first duration configured for the first speech recognition service.

[0075] In this embodiment, the first duration can reflect the error of the first speech recognition service. That is, when performing speech recognition through the first speech recognition service, a sufficiently long audio frame needs to be retained before the first audio frame in order to accurately recognize the speech content in the subsequent speech recognition process. This is to avoid the situation where the speech content in the first audio frame is misrecognized when the first audio frame is used as the first audio frame for speech recognition. Therefore, based on the first speech recognition service allocated for the voice dialogue, the first duration configured for the first speech recognition service is determined so that the audio frames retained in the subsequently filtered data stream queue are applicable to the first speech recognition service, thereby ensuring the accuracy of subsequent speech recognition.

[0076] Optionally, the first duration can be adjusted based on the error of the first speech recognition service in order to ensure the accuracy of subsequent speech recognition through the first speech recognition service as much as possible.

[0077] It should be noted that the embodiments of this application are illustrated by filtering out audio frames in the data stream queue that precede the first audio frame and whose interval length with the first audio frame is longer than the first duration. In another embodiment, it is not necessary to perform the above step 303. Instead, other methods are adopted to filter audio frames in the data stream queue that precede the first audio frame in response to the detection of the first audio frame in the data stream queue.

[0078] 304. The server performs speech recognition on the audio frames in the filtered data stream queue according to their order, and obtains the speech recognition results.

[0079] In one possible implementation, step 304 includes: the server performs speech recognition on each audio frame in the filtered data stream queue in the order of the audio frames, and obtains the recognition result of each audio frame. When the recognition result of the third audio frame is obtained, the recognition results of the first audio frame, the third audio frame, and the audio frames between the first and third audio frames are concatenated to obtain the speech recognition result. The third audio frame is the last audio frame containing sound in the data stream queue after the first audio frame.

[0080] In this embodiment of the application, in the filtered data stream queue, the audio frames after the third audio frame do not contain sound, so there is no need to perform speech recognition on the audio frames after the third audio frame. Moreover, the first to third audio frames in the filtered data stream queue can already reflect the speech content. Therefore, the recognition results of the first audio frame, the third audio frame, and the audio frames between the first and third audio frames are spliced ​​together to obtain the speech recognition result, so as to ensure the accuracy of the obtained speech recognition result.

[0081] In one possible implementation, step 304 includes: when a first speech recognition service is assigned among multiple speech recognition services for a voice dialogue, the first speech recognition service performs speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue to obtain a speech recognition result.

[0082] In this embodiment, the server is configured with multiple speech recognition services, each of which performs speech recognition on audio frames in the speech data stream of a voice dialogue. When a first speech recognition service is assigned to the voice dialogue, this service performs speech recognition on the audio frames in the filtered data stream queue according to their order, ensuring real-time speech recognition of the voice data stream and guaranteeing the accuracy of the speech recognition.

[0083] The solution provided in this application stores the voice data stream of the voice dialogue in a data stream queue during the voice dialogue process. Audio frames in the data stream queue are detected to determine if they contain sound. Once the first audio frame containing sound is detected, the audio frames preceding the first audio frame in the data stream queue are filtered to remove audio frames that do not contain sound. Simultaneously, speech recognition is performed on the filtered audio frames in the data stream queue. This eliminates the need for speech recognition on audio frames that do not contain sound. Furthermore, speech recognition can be performed on audio frames in the data stream queue simultaneously while storing the audio frames of the voice data stream, without waiting for all audio frames in the voice data stream to be received. This allows for simultaneous speech recognition of audio frames as soon as they are received, thus ensuring high speech recognition efficiency.

[0084] Furthermore, the embodiments of this application can change the duration of audio frames stored in the data stream queue for audio frames in the original voice data stream, so as to reduce the number of stored audio frames and thus improve the efficiency of subsequent speech recognition. In addition, during the speech recognition process, it is no longer necessary to package the entire voice data stream into a voice file before speech recognition, thus avoiding the process of reading and writing files and further improving the efficiency of speech recognition.

[0085] It should be noted that, in the above Figure 3 Based on the illustrated embodiment, during the process of the server storing audio frames in the voice data stream of the voice dialogue in real time in the data stream queue, if certain conditions are met, the server will no longer store audio frames in the voice data stream into the data stream queue. That is, the method further includes: during the real-time detection of audio frames in the data stream queue, in response to the detection of a second audio frame in the data stream queue, deleting the second audio frame and no longer storing audio frames in the voice data stream into the data stream queue. The second audio frame is the first audio frame without sound after the first audio frame in the voice data stream.

[0086] In this embodiment, the data stream queue is used to store audio frames from the voice data stream of a voice dialogue. The audio frames in the data stream queue are used for subsequent speech recognition. Therefore, once it is detected that a stored audio frame does not contain sound, the audio frame without sound is called the second audio frame. At this time, it reflects that there are no audio frames containing sound in the voice data stream, and it also reflects that the user has finished speaking. Therefore, the audio frames after the second audio frame in the voice data stream will not contain sound either. Therefore, the second audio frame already stored in the data stream queue is deleted, and the audio frames after the second audio frame in the data stream queue are stored in the data stream queue, so that the data stream queue only stores audio frames containing sound, thereby saving storage space. Furthermore, there is no need to perform speech recognition on the audio frames without sound, thus ensuring the efficiency of subsequent speech recognition.

[0087] For example, during a voice dialogue, the voice data stream is received in real time, and the audio frames in the received voice data stream are stored in a data stream queue. The audio frames stored in the data stream queue are detected in real time to determine if they contain sound. Once the first audio frame containing sound is detected in the data stream queue, the audio frames in the data stream queue are filtered according to step 303 above. At this time, speech recognition is simultaneously performed on the filtered audio frames in the data stream queue. Furthermore, during this process, the audio frames in the received voice data stream are also stored in the data stream queue in real time. The system detects newly stored audio frames in the data stream queue. If the first second audio frame without sound is detected after the first audio frame, the second audio frame is deleted. Furthermore, no subsequent audio frames from the voice data stream are stored in the data stream queue. This ensures that only audio frames containing sound are stored in the data stream queue, and speech recognition is performed on the audio frames in the data stream queue simultaneously. When storing audio frames containing sound in the voice data stream into the data stream queue is completed, speech recognition of these audio frames is also completed simultaneously, without waiting for storage to be completed before starting speech recognition, thus improving speech recognition efficiency.

[0088] In the above Figure 2 Based on the embodiments shown, this application uses human-computer voice dialogue as an example to illustrate the specific process, as detailed in the following embodiments.

[0089] Figure 4 This is a flowchart of a speech recognition method provided in an embodiment of this application. The method is executed by a server, such as... Figure 4 As shown, the method includes:

[0090] 401. During the human-computer voice dialogue process, the server receives the voice data stream input by the terminal participating in the human-computer voice dialogue in the current dialogue round, and stores the audio frames in the voice data stream of the current dialogue round in the data stream queue of the current dialogue round.

[0091] In this human-computer voice dialogue, the participants include the user and the intelligent dialogue robot. That is, during the human-computer voice dialogue, the intelligent dialogue robot simulates the user and converses with a user. The terminal participating in the human-computer voice dialogue is the terminal used by the user participating in the human-computer voice dialogue; that is, the user conducts the human-computer voice dialogue with the intelligent dialogue robot through the terminal.

[0092] In this embodiment, during human-computer voice dialogue, the intelligent dialogue robot engages in multiple dialogue rounds with the terminal. In each round, the intelligent dialogue robot outputs voice data once, and the user inputs voice data once through the terminal. The voice data output by the intelligent dialogue robot in a single dialogue round is controlled by the server; that is, the server is aware of the voice data output by the intelligent dialogue robot in that round. Therefore, the server only receives the voice data stream input by the terminal participating in the human-computer voice dialogue in the current round. This allows the server to perform speech recognition on the audio frames in the input voice data stream to understand its meaning and, consequently, the user's intent, enabling further human-computer voice dialogue. Furthermore, each dialogue round corresponds to a data stream queue for storing the voice data stream of that round.

[0093] In one possible implementation, step 401 includes: during the human-computer voice dialogue, in response to the start of the current dialogue turn, creating a data stream queue for the current dialogue turn; receiving the voice data stream input by the terminal in the current dialogue turn in real time, and storing the audio frames in the voice data stream of the current dialogue turn in the data stream queue for the current dialogue turn.

[0094] In this embodiment, the server creates a data stream queue corresponding to each dialogue round only at the beginning of each dialogue round, so that the received voice data stream is stored in the data stream queue in that dialogue round. This not only distinguishes the voice data streams of each dialogue round to ensure the accuracy of voice data stream storage, but also eliminates the need to create multiple data stream queues for each dialogue round in advance, thus saving storage space.

[0095] In one possible implementation, each dialogue turn corresponds to a dialogue duration, and the voice data stream received by the server within the dialogue duration of any dialogue turn is used as the voice data stream of that dialogue turn for corresponding storage.

[0096] The dialogue duration is arbitrary and may be the same or different for different dialogue rounds. In this embodiment, during the process of the server storing the voice data stream of the current dialogue round in the data stream queue of the current dialogue round, regardless of whether the audio frames in the voice data stream queue contain sound, as long as the audio frame is received within the dialogue duration of the current dialogue round, it is stored in the data stream queue of the current dialogue round.

[0097] It should be noted that the embodiments of this application use human-computer voice dialogue as an example for illustration. In another embodiment, it is not necessary to perform the above step 401. Instead, other methods are adopted to store the audio frames in the received voice data stream in the data stream queue during the voice dialogue process.

[0098] 402. During the real-time detection of audio frames in the data stream queue, in response to the detection of the first audio frame in the data stream queue, the server filters the audio frames in the data stream queue before the first audio frame. The first audio frame is the first audio frame in the data stream queue that contains sound.

[0099] Step 402 is the same as step 202 above, and will not be repeated here.

[0100] 403. The server performs speech recognition on each audio frame in the filtered data stream queue in sequence according to the order of the audio frames, and obtains the recognition result for each audio frame.

[0101] In this embodiment, when the server detects the first audio frame containing sound in the data stream queue, it immediately filters the audio frames in the data stream queue and immediately starts frame-by-frame speech recognition for the filtered audio frames in the data stream queue, so as to obtain the recognition result of each audio frame, thereby ensuring that the speech recognition process and the process of receiving the speech data stream of human-machine voice dialogue are executed synchronously.

[0102] The recognition result of each audio frame is used to represent the meaning of the sound contained in the audio frame. The recognition result can be represented in any form, for example, the recognition result can be represented in the form of text.

[0103] 404. After obtaining the recognition result of the third audio frame, the server concatenates the recognition results of the first audio frame, the third audio frame, and the audio frames between the first and third audio frames to obtain the speech recognition result. The third audio frame is the last audio frame containing sound in the data stream queue after the first audio frame.

[0104] In this embodiment of the application, in the filtered data stream queue, the audio frames after the third audio frame do not contain sound, so there is no need to perform speech recognition on the audio frames after the third audio frame. Moreover, the first to third audio frames in the filtered data stream queue can already reflect the speech content. Therefore, the recognition results of the first audio frame, the third audio frame, and the audio frames between the first and third audio frames are spliced ​​together to obtain the speech recognition result, so as to ensure the accuracy of the obtained speech recognition result.

[0105] In one possible implementation, the process of obtaining speech recognition results includes: the server concatenating the recognized audio frames in the order they appear in the filtered data stream queue to obtain the speech recognition result.

[0106] In this embodiment of the application, after obtaining the recognition results of the first audio frame, the third audio frame, and the audio frame between the first and third audio frames, the recognition results are spliced ​​together in the order of the first audio frame, the third audio frame, and the audio frame between the first and third audio frames to ensure the accuracy of the obtained speech recognition results.

[0107] It should be noted that the embodiments of this application are illustrated by first obtaining the recognition results of each audio frame and then splicing them together. In another embodiment, it is not necessary to perform the above steps 403-404. Instead, other methods are adopted to perform speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue, and obtain the speech recognition results.

[0108] 405. Based on the speech recognition results, the server sends the speech data for the next round of dialogue to the terminal.

[0109] In this embodiment, when the server has obtained the speech recognition result of the user's voice in the current dialogue round after multiple dialogue rounds between the intelligent chatbot and the user corresponding to the terminal, it can send the voice data of the next dialogue round to the terminal based on the speech recognition result, thereby realizing the intelligent chatbot sending the voice data of the next dialogue round to the user's terminal.

[0110] The solution provided in this application stores the voice data stream of the voice dialogue in a data stream queue during the voice dialogue process. Audio frames in the data stream queue are detected to determine if they contain sound. Once the first audio frame containing sound is detected, the audio frames preceding the first audio frame in the data stream queue are filtered to remove audio frames that do not contain sound. Simultaneously, speech recognition is performed on the filtered audio frames in the data stream queue. This eliminates the need for speech recognition on audio frames that do not contain sound. Furthermore, speech recognition can be performed on audio frames in the data stream queue simultaneously while storing the audio frames of the voice data stream, without waiting for all audio frames in the voice data stream to be received. This allows for simultaneous speech recognition of audio frames as soon as they are received, thus ensuring high speech recognition efficiency.

[0111] Furthermore, in this embodiment, in the filtered data stream queue, the audio frames after the third audio frame do not contain sound, so there is no need to perform speech recognition on the audio frames after the third audio frame. Moreover, the first to third audio frames in the filtered data stream queue can already reflect the speech content. Therefore, the recognition results of the first audio frame, the third audio frame, and the audio frames between the first and third audio frames are spliced ​​together to obtain the speech recognition result, so as to ensure the accuracy of the obtained speech recognition result.

[0112] It should be noted that, in the above Figure 4 Based on the illustrated embodiment, each dialogue turn corresponds to a dialogue duration. Considering that users may pause during the speech when inputting voice data stream through the terminal, during the speech recognition process of the audio frames in the voice data stream of the current dialogue turn, only the first audio frame in the data stream queue is detected. Audio frames before the first audio frame in the data stream queue are filtered, and audio frames after the first audio frame in the data stream queue are not deleted or filtered out, so as to ensure the integrity of the voice data and ensure that the effective range of voice data is not lost.

[0113] It should be noted that the above Figures 2 to 4 The illustrated embodiment uses a server to perform speech recognition on audio frames as an example. In another embodiment, the speech recognition method provided in this application can be executed by an outbound calling system, which includes a first server and a recognition server. The recognition server is used for speech recognition of audio frames. The first server is equivalent to the one described above. Figures 2 to 3The server in the illustrated embodiment, and the first server further configured to provide call center services, is equipped with a Free Switch (a self-interactive voice communication scheme) component and MRCP (Media Resource Control Protocol). Figure 5 As shown, speech recognition methods include:

[0114] The first server is used to provide call center services and can initiate human-machine voice dialogue to the user's terminal. When the user's terminal confirms participation in the human-machine voice dialogue, the first server is used to maintain the human-machine voice dialogue through Free Switch and can obtain the voice data stream of the human-machine voice dialogue in real time.

[0115] The first server, through MRCP, can capture real-time voice data streams for each dialogue turn in a human-computer voice conversation from the Free Switch component, and can perform the above-mentioned... Figures 2 to 4 The illustrated embodiment stores the audio frames in the voice data stream of the current dialogue turn in the data stream queue of the current dialogue turn. A VAD (Voice Activity Detection) detection method is used to sequentially detect the audio frames in the data stream queue of the current dialogue turn to determine if each audio frame contains sound. In response to the detection of the first audio frame containing sound in the data stream queue, i.e., in response to the detection of the first audio frame, audio frames in the data stream queue that precede the first audio frame and whose interval length with respect to the first audio frame is greater than a first duration are filtered out. This ensures the integrity of the voice data and guarantees that no valid range of voice data is missing. Furthermore, in response to detecting the first audio frame containing sound in the data stream queue, which is equivalent to detecting a dequeue operation, the first server immediately uses a WebSocket (a communication protocol) or UDP (User Datagram Protocol) channel to interact with the recognition server after filtering the audio frames in the data stream queue. The server then sequentially sends the filtered audio frames in the data stream queue to the recognition server so that the recognition server can synchronously perform speech recognition on the audio frames in the filtered data stream queue. The recognition server performs speech recognition on each received audio frame and obtains the recognition result for each audio frame.

[0116] When the first server finishes its current dialogue turn, it has already sent each audio frame in the filtered data stream queue to the recognition server. After completing speech recognition for each received audio frame, the recognition server concatenates the recognition results of multiple audio frames in the order they were received to obtain the speech recognition result. Alternatively, during real-time detection of audio frames in the data stream queue for the current dialogue turn, if the first server detects the first audio frame without sound (i.e., the second audio frame), it deletes the second audio frame from the data stream queue for the current dialogue turn and does not continue to store audio frames from the speech data stream of the current dialogue turn in the data stream queue. In this case, the last audio frame containing sound in the data stream queue for the current dialogue turn is the third audio frame. Upon receiving the third audio frame and completing speech recognition for it, the recognition server concatenates the recognition results of the first audio frame, the audio frames between the first and third audio frames, and the third audio frame to obtain the speech recognition result.

[0117] The recognition server sends the speech recognition result to the first server. The first server, based on the speech recognition server's ability to send the speech data of the next dialogue round to the terminal participating in the human-computer speech dialogue, enables the terminal to play the speech data. Then, it takes the speech data input by the user in the next dialogue round to realize the human-computer speech dialogue in the next dialogue round.

[0118] It should be noted that the above-mentioned various optional embodiments can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0119] Figure 6 This application provides a speech recognition device, such as... Figure 6 As shown, the device includes:

[0120] Storage module 601 is used to store audio frames from the received voice data stream in a data stream queue during a voice conversation.

[0121] The filtering module 602 is used to filter audio frames in the data stream queue before the first audio frame in the data stream queue in response to the detection of the first audio frame in the data stream queue during the real-time detection of audio frames in the data stream queue. The first audio frame is the first audio frame containing sound in the data stream queue.

[0122] The recognition module 603 is used to perform speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue, and obtain the speech recognition result.

[0123] In one possible implementation, the filtering module 602 is configured to filter out audio frames in the data stream queue that precede the first audio frame and whose interval duration with respect to the first audio frame is greater than a first duration in response to the detection of the first audio frame.

[0124] In another possible implementation, the storage module 601 is used to determine the ratio of the second duration to the duration of the audio frame as the synthesis quantity when the duration of the audio frame in the voice data stream is less than the second duration; to synthesize multiple consecutive audio frames in the voice data stream, and to store the synthesized audio frames in the data stream queue, wherein the number of multiple consecutive audio frames is the synthesis quantity.

[0125] In another possible implementation, such as Figure 7 As shown, the device also includes:

[0126] The acquisition module 604 is used to acquire a second duration configured for the first speech recognition service when a first speech recognition service is assigned among multiple speech recognition services for a voice dialogue, and the first speech recognition service is used to recognize audio frames of the second duration.

[0127] The recognition module 603 is used to perform speech recognition on the audio frames in the filtered data stream queue according to the order of the audio frames in the filtered data stream queue through the first speech recognition service, and obtain the speech recognition result.

[0128] In another possible implementation, such as Figure 7 As shown, the device also includes:

[0129] The deletion module 605 is used to delete the second audio frame in response to the detection of the second audio frame in the data stream queue during the real-time detection of audio frames in the data stream queue, and not to continue storing audio frames in the voice data stream in the data stream queue. The second audio frame is the first audio frame without sound after the first audio frame in the voice data stream.

[0130] In another possible implementation, the voice dialogue is a human-computer voice dialogue. The storage module 601 is used to receive the voice data stream input by the terminal participating in the human-computer voice dialogue in the current dialogue round during the human-computer voice dialogue process, and store the audio frames in the voice data stream of the current dialogue round in the data stream queue of the current dialogue round.

[0131] In another possible implementation, such as Figure 7As shown, the recognition module 603 is used to perform speech recognition on each audio frame in the filtered data stream queue in sequence according to the order of the audio frames in the filtered data stream queue, and obtain the recognition result of each audio frame; when the recognition result of the third audio frame is obtained, the recognition results of the first audio frame, the third audio frame and the audio frames between the first audio frame and the third audio frame are concatenated to obtain the speech recognition result, wherein the third audio frame is the last audio frame containing sound after the first audio frame in the data stream queue;

[0132] The device also includes:

[0133] The sending module 606 is used to send voice data for the next round of dialogue to the terminal based on the speech recognition results.

[0134] In another possible implementation, the storage module 601 is used to create a data stream queue for the current dialogue turn in response to the start of the current dialogue turn during the human-computer voice dialogue process; receive the voice data stream input by the terminal in the current dialogue turn in real time, and store the audio frames in the voice data stream of the current dialogue turn in the data stream queue of the current dialogue turn.

[0135] It should be noted that the speech recognition device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server can be divided into different functional modules to complete all or part of the functions described above. In addition, the speech recognition device and the speech recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0136] This application also provides a server, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed by the speech recognition method of the above embodiments.

[0137] Figure 8This is a schematic diagram of a server structure provided in an embodiment of this application. The server 800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 801 and one or more memories 802. The memories 802 store at least one computer program, which is loaded and executed by the processor 801 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0138] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the speech recognition method of the above embodiments.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the operations performed by the speech recognition method of the above embodiments.

[0140] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0141] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A speech recognition method, characterized in that, The method includes: In response to the start of a voice dialogue, a data stream queue is created. During the voice dialogue, the input voice data stream is received in real time, and the audio frames in the received voice data stream are stored frame by frame in the data stream queue. The duration of the audio frames stored in the data stream queue matches the second duration corresponding to the speech recognition service allocated for the voice dialogue. When the duration of the audio frames in the voice data stream is less than the second duration, multiple consecutive audio frames in the voice data stream are first synthesized, and then the synthesized audio frames are stored in the data stream queue. During the process of storing audio frames from the voice data stream into the data stream queue frame by frame, the audio frames in the data stream queue are detected in real time according to the order of the audio frames in the data stream queue to detect whether the audio frames contain sound. During the real-time detection of audio frames in the data stream queue, in response to the detection of the first audio frame in the data stream queue, the audio frames preceding the first audio frame in the data stream queue are immediately filtered, and speech recognition is immediately started frame by frame in the filtered audio frames in the data stream queue according to the order of the audio frames, so as to ensure that the speech recognition process and the process of receiving the voice data stream are executed synchronously; while the data stream queue is being filtered, the process of storing the audio frames in the voice data stream into the data stream queue continues to run; the first audio frame is the first audio frame containing sound in the data stream queue; Based on the recognition results of each audio frame, the speech recognition result is obtained.

2. The method according to claim 1, characterized in that, The step of immediately filtering audio frames preceding the first audio frame in the data stream queue in response to detecting the first audio frame in the data stream queue includes: In response to detecting the first audio frame, audio frames in the data stream queue that precede the first audio frame and whose interval length with the first audio frame is greater than a first duration are filtered out.

3. The method according to claim 1, characterized in that, The step of storing the audio frames from the received voice data stream frame by frame in the data stream queue includes: If the duration of an audio frame in the voice data stream is less than the second duration, the ratio of the second duration to the duration of the audio frame is determined as the number of synthesized components. Multiple consecutive audio frames in the speech data stream are synthesized, and the synthesized audio frames are stored in the data stream queue. The number of the multiple consecutive audio frames is the number of synthesized frames.

4. The method according to claim 3, characterized in that, Before determining the ratio of the second duration to the duration of the audio frame as the number of synthesized components when the duration of the audio frame in the speech data stream is less than the second duration, the method further includes: In the case of assigning a first speech recognition service among multiple speech recognition services to the voice dialogue, the second duration configured for the first speech recognition service is obtained, and the first speech recognition service is used to recognize audio frames of the second duration. The step of immediately starting frame-by-frame speech recognition according to the order of audio frames in the filtered data stream queue includes: Using the first speech recognition service, speech recognition is performed on the audio frames in the filtered data stream queue according to their order.

5. The method according to claim 1, characterized in that, During the voice dialogue, after receiving the input voice data stream in real time and storing the audio frames of the received voice data stream frame by frame in the data stream queue, the method further includes: During the real-time detection of audio frames in the data stream queue, in response to the detection of a second audio frame in the data stream queue, the second audio frame is deleted, and audio frames in the voice data stream are no longer stored in the data stream queue. The second audio frame is the first audio frame without sound in the voice data stream after the first audio frame.

6. The method according to any one of claims 1-5, characterized in that, The voice dialogue is a human-computer voice dialogue. During the voice dialogue, the process of receiving the input voice data stream in real time and storing the audio frames in the received voice data stream frame by frame in the data stream queue includes: During the human-computer voice dialogue, the system receives the voice data stream input by the terminal participating in the human-computer voice dialogue in the current dialogue round, and stores the audio frames in the voice data stream of the current dialogue round in the data stream queue of the current dialogue round.

7. The method according to claim 6, characterized in that, Following the order of the audio frames in the filtered data stream queue, speech recognition is immediately performed frame by frame on the audio frames in the filtered data stream queue to obtain the speech recognition result, including: According to the order of the audio frames in the filtered data stream queue, speech recognition is performed on each audio frame in the filtered data stream queue in turn to obtain the recognition result of each audio frame. Given the recognition result of the third audio frame, the recognition results of the first audio frame, the third audio frame, and the audio frames between the first audio frame and the third audio frame are concatenated to obtain the speech recognition result. The third audio frame is the last audio frame containing sound in the data stream queue after the first audio frame. Following the order of audio frames in the filtered data stream queue, speech recognition is immediately performed frame by frame on the audio frames in the filtered data stream queue. After obtaining the speech recognition result, the method further includes: Based on the speech recognition results, the speech data for the next round of dialogue is sent to the terminal.

8. The method according to claim 6, characterized in that, During the human-computer voice dialogue, receiving the voice data stream input by the terminal participating in the human-computer voice dialogue in the current dialogue round, and storing the audio frames in the voice data stream of the current dialogue round in the data stream queue of the current dialogue round, includes: During the human-computer voice dialogue, in response to the start of the current dialogue round, a data stream queue for the current dialogue round is created; The system receives the voice data stream input by the terminal in the current dialogue round in real time, and stores the audio frames in the voice data stream of the current dialogue round in the data stream queue of the current dialogue round.

9. A voice recognition device, characterized in that, The device includes: The module used to perform the following steps: In response to the start of a voice conversation, a data stream queue is created; The storage module is used to receive the input voice data stream in real time during the voice dialogue and store the audio frames in the received voice data stream frame by frame in the data stream queue. The duration of the audio frames stored in the data stream queue is consistent with the second duration corresponding to the voice recognition service allocated for the voice dialogue. When the duration of the audio frames in the voice data stream is less than the second duration, multiple consecutive audio frames in the voice data stream are first synthesized, and then the synthesized audio frames are stored in the data stream queue. The module is used to perform the following steps: during the process of storing audio frames in the voice data stream frame by frame into the data stream queue, the audio frames in the data stream queue are detected in real time frame by frame according to the order of the audio frames in the data stream queue, so as to detect whether the audio frame contains sound. The filtering module is used to, in the process of real-time detection of audio frames in the data stream queue, immediately filter the audio frames in the data stream queue before the first audio frame in the data stream queue in response to the detection of the first audio frame in the data stream queue, wherein the first audio frame is the first audio frame containing sound in the data stream queue. The recognition module is configured to, after the filtering module performs the step of immediately filtering the audio frames preceding the first audio frame in the data stream queue, immediately begin frame-by-frame speech recognition of the audio frames in the filtered data stream queue in the order of the audio frames, so as to ensure that the speech recognition process and the process of receiving the speech data stream are executed synchronously; while filtering the data stream queue, the process of storing the audio frames in the speech data stream into the data stream queue continues to run; and based on the recognition results of each audio frame, a speech recognition result is obtained.

10. A server, characterized in that, The server includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations of the speech recognition method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the speech recognition method as described in any one of claims 1 to 8.

Images