Speech recognition method, electronic device, storage medium and program product

Abstract

The invention provides a voice recognition method, electronic equipment, a readable storage medium and a computer program product. The speech recognition method comprises the following steps: acquiring a speech segment to be recognized and an associated speech segment of the speech segment to be recognized from speech data; inputting the to-be-recognized voice segment and the associated voice segment into a voice recognition model to obtain a current voice recognition result and current time information of the current voice recognition result; acquiring a historical voice recognition result and historical time information of the historical voice recognition result; according to the current time information and the historical time information, combining the current voice recognition result and the historical voice recognition result to obtain a target voice recognition result; wherein the historical voice recognition result is a voice recognition result corresponding to at least one voice segment before the to-be-recognized voice segment in the voice data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computers and the like, and in particular to a speech recognition method, an electronic device, a storage medium and a program product. Background Art

[0002] Speech recognition technology is an important branch of artificial intelligence. It realizes the function of human-computer interaction by converting human speech signals into corresponding text information. In recent years, with the rapid development of deep learning technology, speech recognition technology has made significant progress, and the accuracy and application scenarios of real-time speech recognition have been significantly improved.

[0003] In the prior art, the process of real-time speech recognition is as follows: receiving the user's input speech through a microphone, segmenting the input speech to obtain speech segments, and then inputting each speech segment into a speech recognition model in sequence to obtain a recognition result.

[0004] However, since each speech segment is processed independently, the speech recognition model cannot understand the speech context and the recognition accuracy is low. Summary of the invention

[0005] The present disclosure provides a speech recognition method, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of the present disclosure, there is provided a speech recognition method, comprising: Acquire a speech segment to be recognized and a speech segment associated with the speech segment to be recognized from the speech data; Inputting the to-be-recognized speech segment and the associated speech segment into a speech recognition model to obtain a current speech recognition result and current time information of the current speech recognition result; Acquire historical speech recognition results and historical time information of the historical speech recognition results; According to the current time information and the historical time information, the current speech recognition result and the historical speech recognition result are merged to obtain a target speech recognition result; The historical speech recognition result is a speech recognition result corresponding to at least one speech segment before the speech segment to be recognized in the speech data.

[0007] According to the speech recognition method of at least one embodiment of the present disclosure, each step of the speech recognition method is executed using a first thread.

[0008] According to at least one embodiment of the speech recognition method of the present disclosure, before acquiring the speech segment to be recognized and the associated speech segment of the speech segment to be recognized from the speech data, the method further includes: The voice data is received using a second thread.

[0009] According to the speech recognition method of at least one embodiment of the present disclosure, the receiving the speech data by using the second thread includes: Using the second thread to receive a communication request sent by a user terminal; Using the second thread to establish a communication connection with the user terminal according to the communication request; The voice data sent by the user terminal is received through the communication connection using the second thread.

[0010] According to the speech recognition method of at least one embodiment of the present disclosure, the using the second thread to establish a communication connection with the user terminal according to the communication request includes: Using the second thread to parse the communication request to obtain user identity information; Using the second thread to perform identity authentication according to the user identity information to obtain a verification result; and When the verification result is successful, a communication connection is established with the user terminal using the second thread.

[0011] According to the speech recognition method of at least one embodiment of the present disclosure, the using the second thread to establish a communication connection with the user terminal includes: Using the second thread to determine whether the voice service balance is less than a balance threshold according to the user identity information; and If it is not less than, using the second thread to establish a communication connection with the user terminal.

[0012] According to the speech recognition method of at least one embodiment of the present disclosure, the using the second thread to receive the speech data includes: using the second thread to receive a speech recognition request sent by a user terminal; using the second thread to determine whether the total number of speech recognition requests currently to be processed is greater than a request threshold; if not, using the second thread to obtain speech data according to the speech recognition request; otherwise, using the second thread to reject the speech recognition request; Optionally, the acquiring the voice segment to be recognized and the associated voice segment of the voice segment to be recognized from the voice data comprises: using a third thread to cache the voice data through a ring buffer; using the first thread to acquire the voice segment to be recognized and the associated voice segment of the voice segment to be recognized from the ring buffer; Optionally, the speech recognition model tracks and manages the speech recognition process through sentence start events, transcription result change events, and sentence end events; Optionally, the merging the current speech recognition result and the historical speech recognition result according to the current time information and the historical time information includes: performing time alignment on the time information of the speech segment to be recognized and the current time information according to the current speech recognition result to obtain the aligned time information of the current speech recognition result; merging the current speech recognition result and the historical speech recognition result according to the aligned time information and the historical time information; Optionally, after obtaining the target speech recognition result, it also includes: using the first thread to put the target speech recognition result into a text queue; using a fourth thread to obtain the target speech recognition result from the text queue; using the fourth thread to split the target speech recognition result to obtain multiple split results; using the fourth thread to send the multiple split results to the user terminal corresponding to the voice data.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the speech recognition method of any embodiment of the present disclosure.

[0014] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the speech recognition method of any embodiment of the present disclosure is implemented.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the speech recognition method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 The process of the speech recognition method of one embodiment of the present disclosure is Figure 1 .

[0018] Figure 2 The process of the speech recognition method of one embodiment of the present disclosure is Figure 2 .

[0019] Figure 3 yes Figure 2 The process of the speech receiving method in the speech recognition method shown Figure 1 .

[0020] Figure 4 yes Figure 3 The process of the connection establishment method in the voice receiving method shown Figure 1 .

[0021] Figure 5 yes Figure 3 The process of the connection establishment method in the voice receiving method shown Figure 2 .

[0022] Figure 6 yes Figure 2 The process of the speech receiving method in the speech recognition method shown Figure 2 .

[0023] Figure 7 yes Figure 1 The flowchart of the segment acquisition method in the speech recognition method shown is shown.

[0024] Figure 8 yes Figure 1 A flowchart of a method for merging results in a speech recognition method is shown.

[0025] Fig. 9 The process of the speech recognition method of one embodiment of the present disclosure is Figure 3 .

[0026] Fig.10 It is a schematic flow chart of a speech recognition method according to an embodiment of the present disclosure.

[0027] Fig.11 It is a schematic block diagram of the structure of a speech recognition device according to another embodiment of the present disclosure.

[0028] Fig.12 It is a schematic block diagram of the structure of an electronic device equipped with a speech recognition device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0030] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] Real-time Speech Recognition (RTSR) is a technology that converts speech data into text by analyzing audio streams, and can output speech recognition results while listening. For example, in a customer service system, real-time speech recognition can help the system automatically divide the voice data input by the user into speech segments and then convert each speech segment into text through a speech recognition model, thereby quickly generating customer service records to support dialogue management, data analysis, and report generation in the customer service system. However, since each speech segment is processed independently, the speech recognition model cannot understand the context of the speech, making it impossible for the speech recognition model to distinguish between speech with similar pronunciations or even the same pronunciation but different meanings, and the accuracy of speech recognition is low.

[0032] To this end, the present disclosure proposes a speech recognition method, electronic device, storage medium and program product, which uses the associated speech segments of the speech segment to be recognized as associated information, so that the speech recognition model can improve the semantic understanding ability when performing speech recognition, and merge the current speech recognition result with the historical speech recognition result to obtain the target speech recognition result, thereby realizing speech recognition. When a user needs to perform speech recognition, he can input voice data on a terminal device (such as a mobile phone, a computer, etc.). After the terminal device receives the voice data, it can directly perform the speech recognition operation locally, or send the voice data to an electronic device such as a server, so that the electronic device such as the server performs the corresponding speech recognition operation.

[0033] For the convenience of description and to make the technical solutions of the specific implementation methods of the present disclosure easier to understand, before describing the speech recognition method implemented in the present disclosure, the technical terms involved in the specific implementation methods of the present disclosure are explained as follows: A speech segment refers to a continuous segment of audio data in speech data. It usually represents a meaningful speech content, which may be a word, a sentence, or speech of a specific duration.

[0034] The sentence start event (sentenceBegin event) indicates the beginning of a new speech segment.

[0035] The transcriptionResultChanged event indicates that the recognition result has changed. It is triggered whenever the speech recognition model updates a partial or final recognition result.

[0036] The sentenceEnd event indicates the end of a speech segment.

[0037] Figure 1FIG. 2 shows a schematic diagram of the overall flow of a speech recognition method M100 according to an embodiment of the present disclosure. Figure 1 The method shown includes steps S110 to S140. The method can be executed by electronic devices such as mobile phones, tablet computers, and servers.

[0038] Specifically, Figure 1 The methods shown include: Step S110, obtaining a speech segment to be recognized and a speech segment associated with the speech segment to be recognized from the speech data.

[0039] In some embodiments of the present disclosure, step S110 can obtain the voice segment to be recognized based on the timestamp of the voice data and the last recognized voice segment; if there is no last recognized voice segment in the voice data, that is, when the voice data is first subjected to voice recognition operation, the first voice segment can be used as the voice segment to be recognized. The size of the voice segment to be recognized can be controlled by a fixed duration, such as 500 milliseconds; the size of the voice segment to be recognized can also be controlled by a fixed byte, such as 32KB; the fixed duration and fixed byte can be adjusted according to specific application scenarios, etc. The associated voice segment in step S110 can be the previous voice segment and / or the following voice segment of the voice segment to be recognized.

[0040] When the associated voice segment only includes the above voice segment, step S110 can first obtain the voice segment to be recognized from the voice data; then determine whether the voice segment to be recognized is the first voice segment of the voice data; if so, directly input the voice segment to be recognized into the voice recognition model to obtain the target voice recognition result; otherwise, obtain the above voice segment of the voice segment to be recognized. Among them, the size of the above voice segment can be controlled by presetting the above duration, such as setting the preset above duration to 10 seconds; when the total duration of all above voice segments of the voice segment to be recognized in the voice data is not greater than the preset above duration, all above voice segments can be directly used as the final above voice segment; when the total duration of all above voice segments of the voice segment to be recognized in the voice data is greater than the preset above duration, the above voice segment of the voice segment to be recognized with the nearest preset above duration can be cut from all above voice segments. For example, if the starting point of the speech segment to be recognized is the 6th second, if the preset context length is 5 seconds, since 5 seconds < 6 seconds, the speech between the 1st second and the 6th second can be cut from the speech data as the above context speech segment; if the preset context length is 10 seconds, since 10 seconds > 6 seconds, all the speech before the 6th second in the speech data can be used as the above context speech segment.

[0041] When the associated voice segment only includes the following voice segment, step S110 can first obtain the voice segment to be recognized from the voice data; then determine whether the duration of all the following voice segments of the voice segment to be recognized in the currently received voice data is greater than the preset following duration; if not, all the following voice segments can be directly used as the final following voice segment; if greater, the following voice segment of the voice segment to be recognized with the nearest preset following duration can be cut from all the following voice segments. For example, the total duration of the currently received voice data is 60 seconds, the starting point of the voice segment to be recognized is the 53rd second, and the duration of the voice segment to be recognized is 0.5 seconds. If the preset following duration is 10 seconds, since 10 seconds>6.5 seconds (60-53-0.5), all the voices after the 53.5th second in the current voice data can be used as the following voice segment; if the preset following duration is 5 seconds, since 5 seconds<6.5 seconds, the voices between the 53.5th second and the 58.5th second in the currently received voice data can be used as the following voice segment.

[0042] When the associated voice segment includes both the preceding voice segment and the following voice segment, the preceding voice segment duration and the following voice segment duration can be set respectively, and then the preceding voice segment and the following voice segment are obtained according to the corresponding durations respectively, and finally the two voice segments are merged into the associated voice segment. The process of obtaining the preceding voice segment and the following voice segment is similar to the above process.

[0043] Step S120, inputting the speech segment to be recognized and the associated speech segment into the speech recognition model to obtain the current speech recognition result and the current time information of the current speech recognition result.

[0044] In some embodiments of the present disclosure, the speech recognition model in step S120 can be an end-to-end speech recognition model based on deep learning, a self-supervised speech recognition model based on a Transformer architecture, or a speech recognition model based on an attention mechanism, etc. The current time information of the current speech recognition result in step S120 is the start time and end time of the current speech recognition result in the entire speech data. The current speech recognition result obtained by step S120 can be composed of multiple words, in which case the current time information is the start time and end time of each word in the entire speech data. The speech recognition model can be set locally or on other electronic devices such as servers.

[0045] In some embodiments of the present disclosure, the speech segment to be recognized and the associated speech segment can be directly input into the speech recognition model through step S120; in particular, in order to improve the accuracy and efficiency of speech recognition, step S120 can also first perform denoising, enhancement and other processing on the speech segment to be recognized and the associated speech segment, and then input the processed speech segment into the speech recognition model.

[0046] Step S130, obtaining historical speech recognition results and historical time information of the historical speech recognition results.

[0047] In some embodiments of the present disclosure, the historical speech recognition result in step S130 is the speech recognition result corresponding to at least one speech segment before the speech segment to be recognized in the speech data; the historical speech recognition result may be the speech recognition result corresponding to all speech segments recognized before the current speech recognition result; the historical speech recognition result in step S130 may also be the speech recognition result corresponding to the most recent preset number of speech segments recognized before the current speech recognition result, etc. The speech recognition result corresponding to at least one speech segment is the final speech recognition result obtained by combining the speech recognition results corresponding to each speech segment, and the speech recognition result corresponding to each speech segment is the speech recognition result obtained by performing speech recognition on the speech segment and its associated speech segments.

[0048] For example, a voice message is "Hello, my name is Li Ming. Today's weather is very good. What are your plans? We can go for a walk in the park together." When the current voice recognition result is "what", its historical voice recognition result can be "Hello, my name is Li Ming. Today's weather is very good. You have", or "Today's weather is very good. You have".

[0049] Step S140, merging the current speech recognition result and the historical speech recognition result according to the current time information and the historical time information to obtain the target speech recognition result.

[0050] In some embodiments of the present disclosure, step S140 can perform deduplication processing on the recognition results by merging to obtain the target speech recognition result. The merging can be a simple merging, or it can include processes such as text alignment, error correction, and fusion decision; wherein, text alignment can be implemented based on time information; error correction can compare two speech recognition results (i.e., the current speech recognition result and the result corresponding to the same speech segment in the historical speech recognition result), identify possible misrecognition parts and then correct them; fusion decision can select one of the parts that appear in both speech recognition results but may be different as the final output (such as selecting to retain the part with a later timestamp and deleting the part with an earlier timestamp; or retaining a relatively complete and correct part based on confidence or language rules), or combine the advantages of both to generate a new and more accurate result.

[0051] In some embodiments of the present disclosure, after obtaining the target speech recognition result through step S140, if the above-mentioned speech recognition process is executed on the user side, the target speech recognition result can be directly displayed, or the target speech recognition result can be analyzed and the analysis result can be displayed; if the above-mentioned speech recognition process is executed on the server side, the target speech recognition result can be directly sent to the user terminal, or the analysis result can be sent to the user terminal after content analysis is performed based on the target speech recognition result.

[0052] The speech recognition method provided by the present disclosure is that when the speech recognition model performs speech recognition, the input data includes not only the speech segment to be recognized, but also the associated speech segments of the speech segment to be recognized. By associating the speech segments, the speech recognition model can better understand the background of the speech segment to be recognized, thereby reducing misunderstandings such as homophonic but different characters, and thus improving the recognition accuracy. The method solves the problem in the prior art that the speech recognition model cannot understand the speech background and has low recognition accuracy because each speech segment is processed independently.

[0053] In some embodiments of the present disclosure, step S110 to step S140 may be executed using a first thread, and the first thread may be a speech transcription thread (Transcriber thread).

[0054] Furthermore, if Figure 2 As shown, before step S110, the following steps may also be included: Step S150: using the second thread to receive voice data.

[0055] In some embodiments of the present disclosure, the second thread used in step S150 may be the same as the first thread used in steps S110 to S140. In particular, in order to achieve significant improvements in performance, real-time performance, and user experience, the second thread used in step S150 may also be different from the first thread used in steps S110 to S140.

[0056] When steps S110 to S150 are executed on the user side, the second thread may be specifically a recording thread, responsible for continuously capturing voice data from the microphone; when steps S110 to S150 are executed on the server side, the second thread may be specifically a receiving thread, responsible for continuously receiving voice data.

[0057] In particular, when steps S110 to S150 are performed on the server side, step S150, in some embodiments of the present disclosure, may include: Figure 3 Steps S151 to S153 are shown.

[0058] Step S151: Use the second thread to receive a communication request sent by a user terminal.

[0059] In some embodiments of the present disclosure, the user terminal may request to exchange information by sending a "communication request", thereby receiving the communication request through step S151; this "communication request" may include information such as request type, request header, and request body. The communication request may be a communication request such as HTTP / HTTPS (Hypertext Transfer Protocol), SIP (Session Initiation Protocol), RTP (Real-time Transport Protocol), etc. In particular, in order to improve communication efficiency, the communication request may also be a Websocket communication request, which is used to request the server side to upgrade the current HTTP connection to the Websocket protocol; the Websocket request may carry some specific header information to identify that this is a Websocket handshake request; wherein the header information may include: "Upgrade: websocket: indicates a request to upgrade the connection to the WebSocket protocol; Connection: Upgrade: indicates that the connection type is upgraded; Sec-WebSocket-Key: a randomly generated Base64 encoded string for handshake verification; Sec-WebSocket-Version: specifies the WebSocket protocol version to be used; Origin: indicates the source of the request, which is used for security verification".

[0060] Among them, the user terminal can send a "Websocket" request through the gateway. The specific process may include: the user terminal requests to obtain a token, and the server side returns the result to the user terminal after receiving the request; when the returned result indicates that the token is successfully obtained, the user terminal sends a Websocket request with the token to the gateway; the gateway sends a request to the server side based on the token.

[0061] Step S152: Use the second thread to establish a communication connection with the user terminal according to the communication request.

[0062] In some embodiments of the present disclosure, after receiving the communication request in step S151, a series of verifications (such as identity verification, authority verification, etc.) can be performed to ensure the legitimacy and security of the communication request. Taking a Websocket request as an example, if it is legitimate, an HTTP 101 status code will be sent to the user terminal, indicating that the connection has been successfully upgraded to the Websocket protocol; after the user terminal receives the 101 response, it indicates that the handshake is successful and the Websocket connection is formally established.

[0063] Step S153: Use the second thread to receive voice data sent by the user terminal through the communication connection.

[0064] In some embodiments of the present disclosure, after the communication connection is established in step S152, data can be sent and received through the communication connection at any time.

[0065] Data exchange between the user side and the server side can be achieved through steps S151 to S153. In particular, when the communication connection is a Websocket connection, two-way communication can be achieved, and the Websocket connection does not need to re-establish and close the connection every time communication occurs, which reduces delays and improves real-time performance, thereby achieving efficient data transmission.

[0066] Regarding step S152, in some embodiments of the present disclosure, it may include the following: Figure 4 Steps S1521 to S1523 are shown.

[0067] Step S1521: Use the second thread to parse the communication request to obtain user identity information.

[0068] In some implementations of the present disclosure, when the request header of the communication request includes Authorization, step S1521 may specifically be to extract user identity information from Authorization.

[0069] Step S1522: Use the second thread to perform identity verification according to the user identity information to obtain a verification result.

[0070] In some embodiments of the present disclosure, step S1522 can match the user identity information with a token pre-stored in a database or cache to implement identity authentication. When the verification result obtained by step S1522 is a successful verification, step S1523 is executed; when the verification result obtained by step S1522 is a failed verification, a verification failure message can be sent to the user terminal.

[0071] Step S1523: When the verification result is successful, a communication connection is established with the user terminal using the second thread.

[0072] By establishing a communication connection after successful identity authentication through steps S1521 to S1523, the security, resource protection capability and user interaction experience of the system can be improved.

[0073] Regarding step S1523, in some embodiments of the present disclosure, if Figure 5 As shown, it can be replaced by: Using the second thread to determine whether the voice service balance is less than the balance threshold according to the user identity information; If it is not less than, using the second thread to establish a communication connection with the user terminal.

[0074] In some embodiments of the present disclosure, a voice service quota of a specific period (such as daily, hourly, etc.) can be allocated to the user for free, or the user can pay to purchase the voice service quota. Before each voice recognition, first determine whether the voice service balance is sufficient by judging whether the voice service balance is less than the balance; if insufficient, send a reminder of insufficient balance to the user terminal, prompting the user to use it after a certain time point or prompt the user to recharge; if sufficient, establish a communication connection to provide voice service.

[0075] The above steps ensure that only users with sufficient balance can use the speech recognition service, thereby enhancing resource management and security and preventing abuse and excessive use.

[0076] Regarding step S150, in some embodiments of the present disclosure, it may also include the following Figure 6 Steps S154 to S157 are shown.

[0077] Step S154: Use the second thread to receive a speech recognition request sent by the user terminal.

[0078] Step S155: Use the second thread to determine whether the total number of speech recognition requests currently to be processed is greater than the request threshold.

[0079] In some embodiments of the present disclosure, when it is determined by step S155 that the total number of voice recognition requests currently to be processed is not greater than the request threshold, step S156 is executed; when it is determined by step S155 that the total number of voice recognition requests currently to be processed is greater than the request threshold, step S157 is executed.

[0080] In some embodiments of the present disclosure, step S155 may also be replaced by using a second thread to determine whether the total number of speech recognition requests in the current cycle is greater than a request threshold, etc. The current cycle is the cycle to which the currently processed speech recognition request belongs.

[0081] Step S156: Use the second thread to obtain voice data according to the voice recognition request.

[0082] In some embodiments of the present disclosure, the process of acquiring voice data through step S156 may include: reading the encoded voice data and configuration information from the voice recognition request; decoding the encoded voice data according to the configuration information to obtain the original voice data.

[0083] Step S157: Use the second thread to reject the speech recognition request.

[0084] In some embodiments of the present disclosure, when it is determined in step S155 that the total number of voice recognition requests currently to be processed is greater than the request threshold, the voice recognition request can be directly rejected in step S157, or the voice recognition request can be first placed in a request queue, and then the requests in the request queue are processed in a first-in-first-out or priority order. The method of rejecting the voice recognition request in step S157 can be to send an error code and an error message to the user terminal.

[0085] By controlling the amount of speech recognition requests through steps S154 to S157, system overload and resource abuse can be prevented, thereby improving system stability and fair allocation of resources.

[0086] Regarding step S110, in some embodiments of the present disclosure, it may also include the following Figure 7 Steps S111 to S112 are shown.

[0087] Step S111: Use a third thread to cache voice data through a ring buffer.

[0088] In some embodiments of the present disclosure, the third thread used in step S111 may be the same as the first thread used in steps S110 to S140 or the second thread used in step S150. In particular, in order to achieve significant improvements in performance, real-time performance, and user experience, the third thread used in step S111 may be different from the first thread used in steps S110 to S140 or the second thread used in step S150, and the third thread may be an AsrSessionManager thread, through which voice data is managed. The circular buffer in step S111, also known as a circular buffer or a circular queue, is a fixed-size data structure in which voice data is stored and read in a circular manner. When the circular buffer reaches its upper capacity limit, new voice data will overwrite the oldest voice data.

[0089] Specifically, step S111 may be to use the AsrSessionManager thread to store the voice data into the ring buffer through the add_audio_frame_to_audio_ringbuffer interface.

[0090] Step S112: Use the first thread to obtain the speech segment to be recognized and the speech segment associated with the speech segment to be recognized from the ring buffer.

[0091] In some implementations of the present disclosure, the first thread may read data from the ring buffer through the read_lastest_audio_ringbuffer function.

[0092] Step S111 and step S112 use the third thread to cache the voice data through the circular buffer and then use the first thread to read it, so as to realize asynchronous processing, improve the reading efficiency, smooth the voice data flow, reduce the load, improve the reliability and optimize the memory usage, and ensure that the real-time voice service can still run stably under high load conditions.

[0093] In some embodiments of the present disclosure, in order to improve the efficiency, accuracy and user experience of the speech recognition model, the speech recognition model can track and manage the speech recognition process through sentence start events, transcription result change events and sentence end events. For example: when a new voice segment needs to be processed, the start command of the speech recognition is triggered, and a handshake is performed through the start command and a confirmation message is returned; when the speech recognition model detects a new voice segment input, a sentence start event is triggered; during the speech recognition process of the speech recognition model, a transcription result change event is triggered, and each time the result changes, the transcription result change event is triggered to ensure that the recognition result is continuously updated; when the speech recognition model detects the end of a new voice segment, a sentence end event is triggered, and the speech recognition of this new voice segment is completed. The speech recognition model can only send the final result at the end of the recognition; in particular, in order to improve the user experience, during the speech recognition process, the speech recognition model can send the transcription intermediate results in real time.

[0094] Regarding step S140, in some embodiments of the present disclosure, it may also include the following Figure 8 Steps S141 to S142 are shown.

[0095] Step S141, time-aligning the time information of the speech segment to be recognized and the current time information according to the current speech recognition result to obtain the aligned time information of the current speech recognition result.

[0096] In some embodiments of the present disclosure, step S141 may time-align the time information of the speech segment to be recognized and the current time information through a factor recognition model.

[0097] Step S142: merging the current speech recognition result and the historical speech recognition result according to the alignment time information and the historical time information.

[0098] By merging step S141 and step S142 after alignment, it is possible to reduce fluctuations in recognition results caused by unstable output of the speech recognition model, ensure natural connection between the content of each speech segment, avoid sentence breaks and repetitions, and improve the coherence and readability of the target speech recognition results.

[0099] Furthermore, if Fig. 9 As shown, the speech recognition method provided by the present disclosure may further include, after step S140: Step S160: Use the first thread to put the target speech recognition result into a text queue.

[0100] In some embodiments of the present disclosure, the first thread may call merge_segments to transcribe the speech segments into target speech recognition results through the speech recognition model and then store them in a text queue.

[0101] Step S170: Use the fourth thread to obtain the target speech recognition result from the text queue.

[0102] In some embodiments of the present disclosure, the fourth thread used in step S170 may be the same as the first thread used in steps S110 to S140, the second thread used in step S150, or the third thread used in step S111. In particular, in order to achieve significant improvements in performance, real-time, and user experience, the fourth thread used in step S170 may be different from the first thread used in steps S110 to S140, the second thread used in step S150, or the third thread used in step S111, and the fourth thread may be a Sender thread responsible for reading and sending the target speech recognition result. Specifically, step S170 may obtain the target speech recognition result through the get_segments_after interface.

[0103] Step S180: Use the fourth thread to split the target speech recognition result to obtain multiple split results.

[0104] In some embodiments of the present disclosure, step S140 may perform semantic segmentation on the target speech recognition result to obtain multiple segmentation results, i.e., multiple sentences. The semantic segmentation may be performed using a rule-based method, a statistics-based method, a machine learning or deep learning-based method, etc.

[0105] Step S190: Use the fourth thread to send multiple splitting results to the user terminal corresponding to the voice data.

[0106] In some embodiments of the present disclosure, the process of sending multiple split results in step S190 may include: after converting the multiple split results into a format required by the protocol, sending the converted data to the user terminal. The split results in step S190 may specifically be text sentences.

[0107] Steps S160 to S190 transmit the target speech recognition results between the first thread and the fourth thread through the text queue, which can ensure the orderly transmission of the target speech recognition results and synchronization between threads, and improve concurrency and stability.

[0108] In some embodiments of the present disclosure, the method of synchronizing results through steps S160 to S190 can also be replaced by the following process: using the first thread to put the target speech recognition result into a text queue; using the fourth thread to obtain the target speech recognition result from the text queue and then execute the corresponding processing process, which can be a direct sending process of the result, a result display process, a result analysis process, etc.

[0109] In some embodiments of the present disclosure, after the current voice recognition is completely completed, the voice service used in the current voice recognition can be charged and the balance can be modified. The charging method can be calculated based on the total duration of the voice recognized this time.

[0110] The speech recognition method provided by the present disclosure can uniformly manage the speech recognition process through the Timeline class, coordinate the processes of receiving, processing and sending the recognition results of speech data, simplify the design and maintenance of the speech recognition system, improve the scalability and stability of the speech recognition system, and enable the method to better cope with complex speech data. In addition, when the threads used in the processes of receiving, processing and sending text data in the method are different, the delay of speech recognition can be further reduced and the scalability can be improved. In actual application, a speech system can include multiple first threads, and the speech recognition process can be executed in parallel by multiple first threads to further improve the efficiency of speech recognition.

[0111] The voice recognition method provided by the present disclosure can be used for contract signing services. Through this method, users can have natural language conversations with the customer service system, so that the customer service system can instantly understand and identify user needs, improve service efficiency and record the contract signing process; it can also be used for meeting records, converting the voice content in the meeting into text in real time, helping participants to quickly obtain meeting records and key information; it can also be used for live broadcast review, providing real-time voice recognition capabilities for the live broadcast process, so that it can more accurately and timely identify inappropriate remarks, sensitive words and illegal content, ensure the compliance and security of live broadcast content, and reduce the workload of manual review.

[0112] Fig.10 A schematic flow chart of the speech recognition method according to the present invention is shown.

[0113] exist Fig.10 In the schematic flow chart shown, the speech recognition process may include: 1. Use a second thread (such as a receiving thread) to receive voice data.

[0114] Second, use a third thread (such as AsrSessionManager thread) to store the received voice data into a ring buffer.

[0115] 3. Use the first thread (such as the Transcriber thread) to obtain the 500-millisecond voice segment to be recognized and the associated voice segment 10 seconds before the voice segment to be recognized from the ring buffer.

[0116] In some embodiments of the present disclosure, in order to facilitate the positioning of the current speech recognition segment to be recognized, when the speech segment to be recognized and its associated speech segments are obtained through this step, the time position of the speech segment can be synchronously recorded.

[0117] 4. Use the first thread to input the speech segment to be recognized and the associated speech segment into the speech recognition model to obtain the current speech recognition result and the current time information of the current speech recognition result.

[0118] In some embodiments of the present disclosure, the current speech recognition result obtained through this step can be composed of words; such as "The weather is very good today", the current speech recognition result can include "today", "weather", "very good", and the current time information of a word is the start time and end time of the word in the entire speech data.

[0119] 5. Use the first thread to obtain historical speech recognition results and historical time information of the historical speech recognition results.

[0120] In some embodiments of the present disclosure, the historical speech recognition result in this step is a speech recognition result corresponding to at least one speech segment before the speech segment to be recognized in the speech data.

[0121] 6. Use the first thread to merge the current speech recognition result and the historical speech recognition result according to the current time information and the historical time information to obtain the target speech recognition result.

[0122] 7. Use the first thread to put the target speech recognition result into the text queue.

[0123] 8. Use the fourth thread (such as the Sender thread) to obtain the target speech recognition result from the text queue.

[0124] 9. Use the fourth thread to split the target speech recognition result to obtain multiple split results.

[0125] 10. Use the fourth thread to send multiple split results to the user terminal according to the format required by the protocol.

[0126] The word accuracy rate of speech recognition through the above-mentioned speech recognition method can reach 92%, that is, the method can achieve high-accuracy speech recognition; and the first frame delay of the speech recognition method is about 500ms, which can achieve speech recognition with low first frame delay.

[0127] Based on any of the above implementations, the present disclosure also provides a speech recognition device (corresponding to a speech recognition method).

[0128] Fig.11 It is a schematic block diagram of the structure of a speech recognition device according to an embodiment of the present disclosure.

[0129] like Fig.11 As shown, the speech recognition device comprises: The segment acquisition module 110 is used to acquire the speech segment to be recognized and the speech segment associated with the speech segment to be recognized from the speech data.

[0130] The speech recognition module 120 is used to input the speech segment to be recognized and the associated speech segment into the speech recognition model to obtain the current speech recognition result and the current time information of the current speech recognition result.

[0131] The history acquisition module 130 is used to acquire historical speech recognition results and historical time information of the historical speech recognition results.

[0132] The result merging module 140 is used to merge the current speech recognition result and the historical speech recognition result according to the current time information and the historical time information to obtain the target speech recognition result.

[0133] The historical speech recognition result is a speech recognition result corresponding to at least one speech segment before the speech segment to be recognized in the speech data.

[0134] The above modules can be implemented by computer program modules.

[0135] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0136] The execution subject of the speech recognition method in the specific implementation of the present disclosure may be an electronic device such as a server.

[0137] Therefore, based on any of the above embodiments, the present disclosure further provides an electronic device, which can execute the speech recognition method of any of the embodiments described above in the present disclosure, and the speech recognition device of any of the embodiments described above can be configured on the electronic device.

[0138] Fig.12 1 is a schematic block diagram of the structure of an electronic device 1000 equipped with a speech recognition device according to an embodiment of the present disclosure.

[0139] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0140] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0141] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0142] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0143] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0144] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0149] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0150] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A speech recognition method, characterized in that: include: Acquire a speech segment to be recognized and a speech segment associated with the speech segment to be recognized from the speech data; Inputting the to-be-recognized speech segment and the associated speech segment into a speech recognition model to obtain a current speech recognition result and current time information of the current speech recognition result; Acquire historical speech recognition results and historical time information of the historical speech recognition results; as well as According to the current time information and the historical time information, the current speech recognition result and the historical speech recognition result are merged to obtain a target speech recognition result; The historical speech recognition result is a speech recognition result corresponding to at least one speech segment before the speech segment to be recognized in the speech data.

2. The speech recognition method according to claim 1, characterized in that: Each step of the speech recognition method is executed using a first thread.

3. The speech recognition method according to claim 2, characterized in that: Before acquiring the voice segment to be recognized and the associated voice segment of the voice segment to be recognized from the voice data, the method further includes: The voice data is received using a second thread.

4. The speech recognition method according to claim 3, characterized in that: The using the second thread to receive the voice data includes: Using the second thread to receive a communication request sent by a user terminal; Using the second thread to establish a communication connection with the user terminal according to the communication request; and The voice data sent by the user terminal is received through the communication connection using the second thread.

5. The speech recognition method according to claim 4, characterized in that: The using the second thread to establish a communication connection with the user terminal according to the communication request includes: Using the second thread to parse the communication request to obtain user identity information; Using the second thread to perform identity authentication according to the user identity information to obtain a verification result; and When the verification result is successful, a communication connection is established with the user terminal using the second thread.

6. The speech recognition method according to claim 5, characterized in that: The using the second thread to establish a communication connection with the user terminal includes: Using the second thread to determine whether the voice service balance is less than a balance threshold according to the user identity information; and If it is not less than, using the second thread to establish a communication connection with the user terminal.

7. The speech recognition method according to any one of claims 3 to 6, characterized in that: The using the second thread to receive the voice data includes: using the second thread to receive a voice recognition request sent by a user terminal; using the second thread to determine whether the total number of voice recognition requests currently to be processed is greater than a request threshold; if not, using the second thread to obtain voice data according to the voice recognition request; otherwise, using the second thread to reject the voice recognition request; Optionally, the acquiring the voice segment to be recognized and the associated voice segment of the voice segment to be recognized from the voice data comprises: using a third thread to cache the voice data through a ring buffer; using the first thread to acquire the voice segment to be recognized and the associated voice segment of the voice segment to be recognized from the ring buffer; Optionally, the speech recognition model tracks and manages the speech recognition process through sentence start events, transcription result change events, and sentence end events; Optionally, the merging the current speech recognition result and the historical speech recognition result according to the current time information and the historical time information includes: performing time alignment on the time information of the speech segment to be recognized and the current time information according to the current speech recognition result to obtain the aligned time information of the current speech recognition result; merging the current speech recognition result and the historical speech recognition result according to the aligned time information and the historical time information; Optionally, after obtaining the target speech recognition result, it also includes: using the first thread to put the target speech recognition result into a text queue; using a fourth thread to obtain the target speech recognition result from the text queue; using the fourth thread to split the target speech recognition result to obtain multiple split results; using the fourth thread to send the multiple split results to the user terminal corresponding to the voice data.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the speech recognition method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the speech recognition method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech recognition method according to any one of claims 1 to 7 is implemented.

Images