Speech recognition method, speech recognition device and system

Abstract

The present application provides a speech recognition method, speech recognition device and system, which are related to the field of artificial intelligence. The method includes the steps of obtaining audio data including a plurality of audio frames, extracting meaning and a phonetic category of the plurality of audio frames, and obtaining a speech end point of the audio data according to the meaning and the phonetic category. According to the solution, the speech end point of the audio data is obtained by extracting and combining the meaning and the phonetic category of the audio data, so that the speech end point of the audio data can be determined more accurately, and the subsequent voice-based operation can be responded to more accurately, and the user experience can be improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of artificial intelligence, and more specifically, to a speech recognition method, speech recognition device, and system.

Background Art

[0002] Currently, voice interaction products, such as intelligent terminal devices, smart home devices, and intelligent in-vehicle devices, are widely used in people's daily lives. In mainstream voice interaction products, voice interaction is actively initiated by the user, but the speech end state (i.e., the end point of speech in one round of interaction) is usually automatically determined by automatic speech recognition. Currently, there are two main problems in speech end state detection: a late speech end state caused by background noise and an early speech end state caused by speech interruption.

[0003] Therefore, a method for more accurately determining the speech end state is an urgent technical problem to be solved.

Summary of the Invention

[0004] Embodiments of the present application provide a speech recognition method, speech recognition device, and system, thereby more accurately determining the speech end state and accurately responding by subsequent operations based on the speech.

[0005] According to a first aspect, a speech recognition method is provided. The method includes the steps of obtaining audio data including a plurality of audio frames, extracting the meaning and the speech category of the plurality of audio frames, and obtaining the speech end point of the audio data based on the meaning and the speech category of the plurality of audio frames.

[0006] In the present invention's technical solution, the speech end point of the audio data is obtained by extracting and combining the meaning and the voice category of the audio data. This allows for a more accurate determination of the speech end point of the audio data, thereby enabling a more accurate response to subsequent voice-based operations and improving the user experience. Furthermore, when the solution of this embodiment is used for speed recognition, the waiting time until the speech end point is not constant but changes along with the actual speech recognition process. Compared to conventional methods that pre-set a fixed waiting time and respond after the waiting time ends, this solution can more accurately obtain the speech end point, improving the timeliness and accuracy of responses to user voice commands and reducing user waiting time.

[0007] For example, the audio category for multiple audio frames may include the audio category of each of the multiple audio frames. Alternatively, the audio category for multiple audio frames may include some of the audio categories of the multiple audio frames.

[0008] Note that while the speech category may be extracted from each audio frame, the meaning does not necessarily have to be extracted from each audio frame. Specifically, an audio frame that does not contain human speech has no meaning. Therefore, it is not possible to extract meaning from an audio frame that does not contain speech. In this case, the meaning of an audio frame that does not contain speech can be considered empty or null.

[0009] Referring to the first embodiment, in some implementations of the first embodiment, the method further includes the step of obtaining the speech end point and then responding to an instruction corresponding to audio data prior to the speech end point of the audio data.

[0010] After obtaining the end point of speech, the system may respond to commands immediately or after a certain period of time. That is, actions corresponding to audio data prior to the end point of speech in the audio data may be executed immediately after obtaining the end point of speech, or actions corresponding to audio data prior to the end point of speech in the audio data may be executed after a certain period of time has passed since obtaining the end point of speech. This period may be redundancy time, error time, etc. According to the solution of this application, the end point of speech in the audio data can be determined more accurately, and therefore, commands corresponding to audio data prior to the end point of speech can be responded to more accurately. This improves the timeliness and accuracy of responses to user voice commands, reduces user waiting time, and improves the user experience.

[0011] In the first embodiment, and in some embodiments of the first embodiment, the audio category of the plurality of audio frames can be obtained based on the relationship between the energy of the plurality of audio frames and a preset energy threshold.

[0012] In the first embodiment, in some embodiments of the first embodiment, the audio categories include "speech," "neutral," and "silence," and the preset energy thresholds include a first energy threshold and a second energy threshold, where the first energy threshold is greater than the second energy threshold. Among a plurality of audio frames, the audio category of audio frames whose energy is greater than or equal to the first energy threshold can be determined as "speech." Among multiple audio frames, the audio category of an audio frame whose energy is below the first energy threshold and greater than the second energy threshold can be determined as "neutral," or, Among multiple audio frames, the audio category of audio frames whose energy is below the second energy threshold can be determined as "silence".

[0013] In relation to the first embodiment, in some implementations of the first embodiment, the first energy threshold and the second energy threshold can be determined based on the energy of the background sound of the first audio data. In different background environments, the silence energy curve will differ. For example, in a relatively quiet environment, the silence energy (i.e., the energy of the background sound) will be relatively low, and in a relatively noisy environment, the silence energy (i.e., the energy of the background sound) will be relatively high. Therefore, obtaining the first and second energy thresholds based on the silence energy can be adapted to the requirements of different environments.

[0014] Referring to the first embodiment, in some embodiments of the first embodiment, the plurality of audio frames includes a first audio frame and a second audio frame, wherein the first audio frame is an audio frame having the above-mentioned meaning, and the second audio frame is an audio frame that follows the first audio frame among the plurality of audio frames. The step of obtaining the speech end point of the audio data based on the aforementioned meaning and the aforementioned audio category is: The step includes obtaining the speech end point based on the aforementioned meaning and the audio category of the second audio frame.

[0015] The first audio frame is a plurality of audio frames having the aforementioned meaning. The second audio frame is one or more audio frames following the first audio frame.

[0016] Note that the "multiple meaningful audio frames" included in the audio data and the "multiple audio frames" are different concepts. The number of audio frames included in the first audio frame is less than the number of audio frames included in the audio data.

[0017] In relation to the first embodiment, in some implementations of the first embodiment, the meaning and the speech category of the second audio frame can be integrated to obtain integrated features of the plurality of audio frames, and then the speech end point can be obtained based on the integrated features. Processing using integrated features can improve processing efficiency and accuracy.

[0018] In relation to the first embodiment, in some implementations of the first embodiment, the speech end category includes "speak", "think", and "end". The speech end category of the audio data may be determined based on the meaning and the speech category of the second audio frame, and the speech end may be obtained if the speech end category of the audio data is "end".

[0019] Furthermore, if the combined features of multiple audio frames are obtained, the speech endpoint category of the audio data may be determined based on the combined features of the multiple audio frames.

[0020] In relation to the first embodiment, in some implementations of the first embodiment, the speech end point classification model is used to process the semantics and the speech categories of the second audio frame to obtain the speech end point category, the speech end point classification model is obtained using a speech sample and the end point category label of the speech sample, the format of the speech sample corresponds to the semantics and the format of the second audio frame, and the end point category included in the end point category label corresponds to the speech end point category.

[0021] Furthermore, if integrated features are obtained from multiple audio frames, these integrated features may be processed using a speech endpoint classification model to obtain speech endpoint categories. The speech endpoint classification model is obtained using speech samples and the endpoint category labels of the speech samples, where the format of the speech samples corresponds to the format of the integrated features, and the endpoint categories included in the endpoint category labels correspond to the speech endpoint categories.

[0022] According to the second aspect, a speech recognition method, Steps include acquiring the first audio data, The steps include determining the first speech end point of the first audio data, The steps include obtaining the first speech end point and then responding to an instruction corresponding to the audio data in the first audio data prior to the first speech end point, Steps to obtain the second audio data, The steps include determining the second speech end point of the second audio data, After obtaining the second speech end point, the process includes responding to a command corresponding to the audio data between the first speech end point in the first audio data and the second speech end point in the second audio data, A method including this is provided.

[0023] According to the solution of the present embodiment of the present application, the speech end point can be obtained more accurately, and an overly long response delay due to the delay in detecting the speech end point can be avoided. Therefore, the speech end point can be obtained more quickly, and subsequent responses can be made in a timely manner, shortening the user's waiting time and improving the user experience. Specifically, in the solution of the present embodiment of the present application, audio data is acquired in real time, and by identifying the speech end point in the audio data, the speech end points of different commands can be identified in real time, and responses can be made to each command after obtaining the speech end point of the command. In particular, when the intervals between multiple commands sent by the user are relatively short, using the solution of the present application can help identify the speech end point of each command after the command is sent and respond to each command in a timely manner, rather than responding to all commands after all multiple commands are sent.

[0024] In relation to the second aspect, in some embodiments of the second aspect, the first audio data includes a plurality of audio frames, and the step of determining the first speech end point of the first audio data includes: extracting the meaning and the voice category of the plurality of audio frames; and obtaining the first speech end point of the first audio data based on the meaning and the voice category of the plurality of audio frames. It includes.

[0025] In relation to the second aspect, in some embodiments of the second aspect, the step of extracting the meaning and the voice category of the plurality of audio frames includes: obtaining the voice category of the plurality of audio frames based on the relationship between the energy of the plurality of audio frames and a preset energy threshold.

[0026] In a second embodiment, in some embodiments of the second embodiment, the speech categories include "speech," "neutral," and "silence," and the preset energy thresholds include a first energy threshold and a second energy threshold, the first energy threshold being greater than the second energy threshold. Among a plurality of audio frames, the speech category of the audio frame whose energy is greater than or equal to the first energy threshold is "speech." Among multiple audio frames, the audio category of an audio frame whose energy is below the first energy threshold and greater than the second energy threshold is "neutral," or, Among multiple audio frames, the audio category of audio frames with energy below the second energy threshold is "Silence".

[0027] In relation to the second embodiment, in some implementations of the second embodiment, the first energy threshold and the second energy threshold are determined based on the energy of the background sound of the first audio data.

[0028] Referring to the second aspect, in some embodiments of the second aspect, the plurality of audio frames includes a first audio frame and a second audio frame, wherein the first audio frame is an audio frame having the aforementioned meaning, and the second audio frame is an audio frame that follows the first audio frame among the plurality of audio frames. The step of obtaining the first speech endpoint of the first audio data based on the aforementioned meaning and the aforementioned audio category is: The step includes obtaining the first speech termination point based on the aforementioned meaning and the audio category of the second audio frame.

[0029] In relation to the second aspect, in some embodiments of the second aspect, the speech end point categories include “speak,” “think,” and “end,” and the step of obtaining the first speech end point based on the meaning and the audio category of the second audio frame is: The process includes determining the speech end category of the first audio data based on the aforementioned meaning and the audio category of the second audio frame, and obtaining the first speech end if the speech end category of the first audio data is "end".

[0030] In relation to the second aspect, in some embodiments of the second aspect, the step of determining the speech end point category of the first audio data based on the meaning and the speech category of the second audio frame is: A step of obtaining a speech end category by processing the semantics and the voice categories of the second audio frame using the speech end category classification model, wherein the speech end category classification model is obtained using a speech sample and the end category label of the speech sample, the format of the speech sample corresponds to the format of the semantics and the voice categories of the second audio frame, and the end category included in the end category label corresponds to the speech end category.

[0031] According to the third aspect, a speech endpoint classification model training method is provided. The training method is: A step of acquiring training data, wherein the training data includes a speech sample and an end-point category label of the speech sample, the format of the speech sample corresponds to the semantic format of multiple audio frames of audio data and the audio category of a second audio frame, the multiple audio frames include a first audio frame and a second audio frame, the first audio frame is a semantic audio frame, the second audio frame is an audio frame following the first audio frame, and the end-point category included in the end-point category label corresponds to a speech end-point category, The steps include training a speech endpoint classification model using the aforementioned training data and obtaining a target speech endpoint classification model, Includes.

[0032] A target speech endpoint classification model obtained using the method of the third embodiment can be used to perform the operation of the first embodiment, which involves processing semantics and speech categories of the second audio frame using the speech endpoint classification model and obtaining speech endpoint categories.

[0033] In relation to the third aspect, in some embodiments of the third aspect, the speech sample may be in the format of "starter + meaning + phonetic category + ender", or the speech sample may be in the format of "starter + phonetic category + meaning + ender".

[0034] Optionally, several text corpora may be obtained, a dictionary tree may be established from these corpora, and each node in the dictionary tree (each node corresponding to one word) may contain the following information: whether the node is an endpoint and its prefix frequency. Speech samples may then be generated based on the node information. An endpoint is the end of a sentence. Prefix frequency is used to represent the number of words between a word and an endpoint. A higher prefix frequency indicates that the word is less likely to be an endpoint. For example, verbs like "give" and "take," and other words like prepositions, are relatively less likely to be used as the end of a sentence and have a relatively high prefix frequency in the dictionary tree. On the other hand, words like "right?" and "correct?" are relatively more likely to be used as the end of a sentence and usually have a relatively low prefix frequency in the dictionary tree.

[0035] According to a fourth aspect, a speech recognition device is provided. The device includes a unit configured to perform the method in any implementation of the first aspect.

[0036] According to a fifth aspect, a speech recognition device is provided. The device includes a unit configured to perform a method in any implementation of the second aspect.

[0037] According to the sixth aspect, a speech endpoint classification model training device is provided. The training device includes a unit configured to perform the method in any implementation of the third aspect.

[0038] According to the seventh aspect, a speech recognition device is provided. The device includes a memory configured to store a program, and a processor configured to execute the program stored in the memory. Once the program stored in the memory is executed, the processor is configured to execute any implementation of the first or second aspect. The device can be placed in a device or system that needs to determine the end point of a speech, such as a speech recognition device, voice assistant, or smart speaker. For example, the device may be a terminal device such as a mobile phone terminal, an in-vehicle terminal, or a wearable device, or a device having computing capabilities such as a computer, host, or server. Alternatively, the device may be a chip.

[0039] According to the eighth aspect, a speech endpoint classification model training device is provided. The training device includes a memory configured to store a program, and a processor configured to execute the program stored in the memory. Once the program stored in the memory is executed, the processor is configured to execute any implementation method of the third aspect. The training device may be any device having computing capabilities, such as a computer, host, or server. Alternatively, the training device may be a chip.

[0040] According to the ninth aspect, a computer-readable medium is provided. The computer-readable medium stores program code used for execution by a device. The program code is used to execute any implementation method of the first, second, or third aspect.

[0041] According to the tenth aspect, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer is made executable any implementation method of the first, second, or third aspect.

[0042] According to the eleventh aspect, an in-vehicle system is provided. The system includes equipment in any implementation of the fourth, fifth, or sixth aspect.

[0043] For example, the in-vehicle system may include a cloud service device and a terminal device. The terminal device may be a vehicle, an in-vehicle chip, or in-vehicle equipment (e.g., an in-vehicle infotainment system or an in-vehicle computer).

[0044] According to the twelfth aspect, an electronic device is provided. The electronic device includes equipment in any implementation of the fourth, fifth, or sixth aspect.

[0045] For example, the electronic device may specifically include one or more devices such as a computer, smartphone, tablet computer, personal digital assistant (PDA), wearable device, smart speaker, television, unmanned aerial vehicle, vehicle, in-vehicle chip, in-vehicle equipment (e.g., in-vehicle infotainment or in-vehicle computer), and robot.

[0046] In this application, the speech end point of the audio data is obtained by extracting and combining the meaning and the voice category of the audio data, thereby more accurately determining the speech end point of the audio data. This allows for more accurate responses to subsequent voice-based operations and improves the user experience. Specifically, the solution of this application avoids excessively long response delays due to delays in speech end point detection, allowing for faster acquisition of the speech end point, timely responses, reduced user waiting time, and improved user experience. Furthermore, the solution of this application allows for accurate acquisition of the speech end point, preventing the user's voice commands from being interrupted due to early detection of the speech end point. This allows for the acquisition of audio data with complete meaning, enabling accurate understanding of the user's intent, accurate responses, and improved user experience. By obtaining an energy threshold based on the energy of background sound, it is possible to respond to the requirements of different environments, thereby further improving the accuracy of speech end point determination. First, by integrating the voice category and meaning, and then acquiring the speech end point based on the integrated features, processing efficiency can be improved, and the accuracy of speech end point determination can be further enhanced. [Brief explanation of the drawing]

[0047] [Figure 1] This is a schematic diagram of the structure of a speech recognition device according to an embodiment of the present invention.

[0048] [Figure 2] This is a schematic diagram illustrating the classification into audio categories according to the embodiment of the present invention.

[0049] [Figure 3] This is a schematic diagram of the processing of integrated features by the speech endpoint classification model according to the embodiment of the present invention.

[0050] [Figure 4] This is a schematic flowchart of the speech recognition method according to the embodiment of the present invention.

[0051] [Figure 5] This is a schematic flowchart of the speech recognition method according to the embodiment of the present invention.

[0052] [Figure 6] This is a schematic flowchart of the speech endpoint classification model training method according to the embodiment of the present invention.

[0053] [Figure 7] This is a schematic diagram of the dictionary tree according to the embodiment of the present invention.

[0054] [Figure 8] This is a schematic block diagram of a speech recognition device according to an embodiment of the present invention.

[0055] [Figure 9] This is a schematic diagram of the hardware structure of a speech recognition device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0056] Referring to the attached drawings, the technical solution of the embodiment of the present application is described below.

[0057] The solution of this application can be applied to various voice interaction scenarios. For example, the solution of this application can be applied to voice interaction scenarios for electronic devices and voice interaction scenarios for electronic systems. Specifically, an electronic device can include one or more devices such as a computer, smartphone, tablet computer, personal digital assistant (PDA), wearable device, smart speaker, television, unmanned aerial vehicle, vehicle, in-vehicle chip, in-vehicle equipment (e.g., in-vehicle infotainment or in-vehicle computer), and robot. An electronic system can include a cloud service device and a terminal device. For example, an electronic system may be an in-vehicle system or a smart home system. The terminal device of an in-vehicle system may include any of the devices such as a vehicle, an in-vehicle chip, or in-vehicle equipment (e.g., in-vehicle infotainment or in-vehicle computer). A cloud service device includes a physical server and a virtual server. The server receives data uploaded from the terminal side (e.g., in-vehicle infotainment), processes the data, and then transmits the processed data to the terminal side.

[0058] Next, we will briefly describe two relatively common application scenarios.

[0059] Application Scenario 1: Smartphone Voice Interaction

[0060] Smartphones can enable voice interaction using voice assistants. For example, users can operate their smartphones or converse with voice assistants through voice interaction. Specifically, a voice assistant can acquire audio data using a microphone, then use a processing unit to determine the speech end point of the audio data, and after acquiring the speech end point, it can trigger a subsequent response. For example, the voice assistant can report the user's intent in the audio data to the operating system for the response.

[0061] Voice interaction enables functions such as making calls, sending information, obtaining directions, playing music, and responding to conversations, significantly improving the technological sophistication and convenience of smartphones.

[0062] According to the present invention, the speech termination point of audio data can be accurately identified, thereby improving the accuracy and timeliness of subsequent responses and enhancing the user experience.

[0063] Application Scenario 2: Voice Interaction in Vehicle Systems

[0064] In-vehicle systems allow for vehicle control through voice interaction. Specifically, the system uses a microphone to acquire audio data, then a processing unit determines the speech end point of the audio data, and triggers a subsequent response after the speech end point is obtained. For example, the user's intent within the audio data is reported to the in-vehicle system for the response.

[0065] Voice interaction can enable functions such as route acquisition, music playback, and control of in-vehicle hardware (e.g., windows and air conditioning), thereby improving the interactive experience of in-vehicle systems.

[0066] According to the present invention, the speech termination point of audio data can be accurately identified, thereby improving the accuracy and timeliness of subsequent responses and enhancing the user experience.

[0067] Figure 1 is a schematic diagram of the structure of a speech recognition device according to an embodiment of the present invention. As shown in Figure 1, the speech recognition device 100 is configured to process audio data and obtain the speech end point of the audio data, that is, the speech stopping point in the audio data. For example, the input audio data includes the speech segment, "I want to make a call." In this case, the speech recognition device 100 can obtain the speech end point of the segment of the audio data through processing. Here, the speech end point may be the last audio frame corresponding to the last word in the speech segment. That is, the speech end point of the segment of the audio data is the last audio frame corresponding to the word "call."

[0068] Note that the end point of audio data and the end point of speech within audio data are different concepts. The end point of audio data refers to the end of the audio. For example, the end point of 5 seconds of audio data is the last audio frame of the audio segment. The end point of speech within audio data is the end of the speech within the audio data segment. Here again, we will use 5 seconds of audio data as an example. The first Assume that the 4-second audio contains speech, and the 5-second section contains no speech. In this case, the speech end point of the audio data corresponds to the audio frame corresponding to the end of the 4-second section. Also, assume that the 5-second audio data contains no speech, and the pre-set time interval for speech recognition is 3 seconds. In this case, if no speech is recognized for 3 consecutive seconds, speech recognition will be terminated, and the speech end point of the 5-second audio data corresponds to the audio frame corresponding to the end of the 3-second section.

[0069] Audio data may contain multiple audio frames. Furthermore, input audio data may or may not contain speech. For example, suppose the speech capture function is activated, but no one speaks for several seconds. In this case, the captured audio data will not contain any speech.

[0070] Multiple audio frames in audio data may be consecutive or discontinuous.

[0071] The speech recognition device 100 includes an acquisition module 110, a processing module 120, and a decision module 130. Alternatively, the decision module 130 may be integrated with the processing module 120.

[0072] The acquisition module 110 is configured to acquire audio data, which may include multiple audio frames. The acquisition module 110 may include a speech capture device, such as a microphone, configured to acquire speech audio in real time. Alternatively, the acquisition module 110 may include a communication interface. A transceiver device, such as a transceiver, may be used for the communication interface to communicate with other devices or communication networks and acquire audio data from those devices or communication networks.

[0073] The processing module 120 processes multiple audio frames in the audio data to obtain meaning and the speech categories of the multiple audio frames. This can be understood as follows: The processing module 120 is configured to extract the speech categories and meaning of multiple audio frames in the audio data.

[0074] Meaning is used to represent the language contained in audio data and may also be called text meaning, word meaning, language meaning, etc. Meaning may also be conveyed through the audio stream.

[0075] For example, the audio category for multiple audio frames may include the audio category for each of the multiple audio frames.

[0076] Alternatively, the audio category of multiple audio frames may include some of the audio categories from those multiple audio frames.

[0077] In other words, the processing module 120 may extract all audio categories from multiple audio frames, or it may extract audio categories from only some of the multiple audio frames.

[0078] Optionally, an audio stream conveying meaning may be obtained using equipment such as an automatic speech recognition (ASR) device. Each segment of the audio stream can be represented by corresponding text. Each segment of the audio stream may contain one or more audio frames.

[0079] The audio categories may include "speech (SPE)", "neutral (NEU)", and "silence (SIL)". "Speech" is the portion of the audio that is clearly spoken by a human (or can be understood as the portion of the audio that is human). "Neutral" is the portion of the audio that is relatively fuzzy (or can be understood as the portion of the audio that is fuzzy) and cannot be clearly determined as speech. "Silence" is the portion of the audio that clearly does not contain human voice (or can be understood as the portion of the audio that does not contain human voice). In this embodiment of the present application, it should be understood that "silence" may mean the absence of speech, the absence of voice, or only background noise, rather than a decibel value of 0 in a physical sense or the complete absence of sound.

[0080] It should be understood that there are other possible ways of classifying voice categories. For example, voice categories may include only "speech" and "silence," or "silence" and "non-silence," or "speech" and "non-speech." "Speech" and "non-speech" may also be called "human voice" and "non-human voice," respectively. Furthermore, "speech," "neutral," and "silence" may also be called "human voice," "potentially human voice," and "not human voice," respectively. This is merely an example, and it should be understood that the method of classifying voice categories is not limited in this embodiment of the application.

[0081] Furthermore, the audio category corresponds to determining and classifying audio data from an acoustic perspective and is used to distinguish audio frame categories. Meaning is obtained by extracting linguistic components from audio data and is used to infer from a linguistic perspective whether speech has been completed. It should be understood that an audio category may be extracted from each audio frame. However, since audio frames that do not contain a human voice have no meaning, the meaning of audio frames that do not contain speech can be considered empty or null.

[0082] Generally, audio frames belonging to different speech categories have different energies. For example, "speech" audio frames have relatively high energy, "silence" audio frames have relatively low energy, and "neutral" audio frames have lower energy than "speech" but higher energy than "silence."

[0083] The energy of an audio frame is also called the strength of an audio frame.

[0084] Optionally, audio frames can be classified based on their energy, and the audio category of each audio frame can be obtained. Specifically, the audio category of a corresponding audio frame is obtained based on the energy of the audio frame in the audio data.

[0085] One implementation can obtain the speech category of an audio frame based on the relationship between the energy of the audio frame and a preset energy threshold. For example, the preset energy threshold includes a first energy threshold and a second energy threshold, where the first energy threshold is greater than the second energy threshold. Audio frames with energy greater than or equal to the first energy threshold are determined to be "speech". Audio frames with energy less than the first energy threshold and greater than the second energy threshold are determined to be "neutral". Audio frames with energy less than or equal to the second energy threshold are determined to be "silence". In other words, among multiple audio frames, the speech category of audio frames with energy greater than or equal to the first energy threshold is determined to be "speech", among multiple audio frames, the speech category of audio frames with energy less than the first energy threshold and greater than the second energy threshold is determined to be "neutral", and among multiple audio frames, the speech category of audio frames with energy less than or equal to the second energy threshold is determined to be "silence". Alternatively, audio whose energy is equal to the first energy threshold or the second energy threshold may be determined as "neutral." The above description is used as an example in the present embodiment of the present application.

[0086] Figure 2 is a schematic diagram of the classification into audio categories according to an embodiment of the present invention. As shown in Figure 2, the horizontal coordinate represents an audio frame sequence, and the vertical coordinate represents the energy value corresponding to the audio frame sequence. The energy curve represents the energy change curve of multiple audio frames in the audio data. The first energy threshold curve represents the lower energy limit curve for "speech" and the upper energy limit curve for "neutral". The second energy threshold curve represents the lower energy limit curve for "neutral" and the upper energy limit curve for "silence". The silence energy curve represents the energy curve of the background sound of the audio segment. Both the first and second energy threshold curves can be obtained based on the silence energy curve. That is, both the first and second energy thresholds can be obtained based on the energy of the background sound.

[0087] Furthermore, the silence energy curve differs depending on the background environment. For example, in a relatively quiet environment, the silence energy (i.e., the energy of the background sound) is relatively low, while in a relatively noisy environment, the silence energy (i.e., the energy of the background sound) is relatively high. Therefore, obtaining the first and second energy threshold curves based on the silence energy curve can be adapted to the requirements of different environments.

[0088] As shown in Figure 2, by comparing the energy of audio frames in the audio data with two thresholds, the audio frames are classified into three speech categories: "Speech" (SPE in the figure), "Neutral" (NEU in the figure), and "Silence" (SIL in the figure). As shown in Figure 2, the speech category sequence of the audio frame sequence is "SPE, NEU, SIL, SIL, SPE, NEU, SIL, SPE, NEU, SPE, NEU, SPE, NEU, SIL" from left to right.

[0089] The processing module 120 may be a processor capable of performing data processing, such as a central processing unit or a microprocessor, or it may be another device, chip, integrated circuit, etc., capable of calculation.

[0090] The decision module 130 is configured to obtain the speech end point of the audio data based on the speech category and meaning from the processing module 120. The speech end point is the result of detecting the speech end state.

[0091] Optionally, if the acquisition module 110 acquires audio data in real time, it may terminate the acquisition of audio data after acquiring the detection results.

[0092] For example, the decision module 130 may obtain the speech end point based on all the speech categories and meanings of multiple audio frames.

[0093] For example, one might predetermine text endpoints that could be speech endpoints based on their meaning, and then obtain speech endpoints from these potential text endpoints based on the audio category of each text endpoint. In another example, one might predetermine candidate audio frames that could be speech endpoints based on their audio category, and then determine speech endpoints based on the meaning preceding these candidate audio frames.

[0094] For example, multiple audio frames include a first audio frame and a second audio frame, where the first audio frame is a meaningful audio frame, and the second audio frame is the audio frame that follows the first audio frame among the multiple audio frames. The decision module 130 can obtain the speech end point based on the meaning and the speech category of the second audio frame.

[0095] The first audio frame consists of multiple meaningful audio frames. The second audio frame consists of one or more audio frames that follow the first audio frame.

[0096] Note that "multiple meaningful audio frames" and "multiple audio frames" contained in audio data are different concepts. The number of audio frames contained in the first audio frame is less than the number of audio frames contained in the audio data.

[0097] Alternatively, the decision module 130 may integrate the meaning and the speech category of the second audio frame to obtain an integrated feature, and obtain the speech end point based on the integrated feature. The integrated feature can be understood as being obtained by superimposing one or more subsequent audio frames, whose speech categories have been determined, onto the audio stream that has meaning. For example, suppose a segment of audio data contains five audio frames following "I want to watch television" and "television". By extracting the speech category and meaning, the meaning "I want to watch television" and the speech categories of the five subsequent audio frames can be obtained. In this case, the integrated feature is obtained by superimposing five audio frames with speech categories onto multiple audio frames that have the meaning "I want to watch television". Processing using the integrated feature can improve processing efficiency and accuracy compared to the direct processing described above.

[0098] If necessary, the speech end category may include "speak," "think," and "end." The decision module 130 determines the speech end category of the audio data based on the meaning and the speech category of the second audio frame, and may obtain the speech end if the speech end category of the audio data is "end."

[0099] The fact that the speech end category of audio data is "end" can be understood as the audio data containing a text end point whose speech end category is "end". The audio frame corresponding to a text end point whose speech end category is "end" may also be used as the speech end point.

[0100] Furthermore, if an integrated feature is obtained, the decision module 130 determines the speech end point category of the audio data based on the integrated feature, and obtains the speech end point if the speech end point category of the audio data is "end".

[0101] In some implementations, the decision module 130 may further process the meaning and speech categories of the second audio frame using a speech end classification model to obtain speech end categories, thereby obtaining the speech end points of the audio data.

[0102] Alternatively, if integrated features are obtained, the decision module 130 may process the integrated features using a speech end-point classification model to obtain speech end-point categories, thereby obtaining the speech end points of the audio data.

[0103] In other words, by inputting the integrated features as a single input feature into the speech end point classification model and processing it, the speech end point category is obtained, and thereby the speech end points of the audio data are obtained.

[0104] It should be understood that this is merely one example. For instance, alternatively, the decision module 130 may process the meaning and the speech category of the second audio frame directly as two input features in the speech endpoint classification model, without processing the meaning and the speech category of the second audio frame.

[0105] The speech recognition device 100 is implemented in the form of a functional module, and the term "module" here may refer to a module implemented in the form of software and / or hardware. This is not limited to this embodiment of the present application. The above-mentioned division into modules is merely a logical functional division, and other division methods may be used in actual implementations. For example, multiple modules may be integrated into one module. That is, the acquisition module 110, the processing module 120, and the decision module 130 may be integrated into one module. Alternatively, multiple modules may exist independently of each other. Or, two of the multiple modules may be integrated into one module. For example, the decision module 130 may be integrated with the processing module 120.

[0106] Multiple modules may be located on the same hardware or on different hardware. That is, the functions that multiple modules need to perform may be performed on the same hardware or on different hardware. This is not limited to the present embodiment of the application.

[0107] Speech ending categories can include "speaking," "thinking," and "ending." "Speaking" can be understood as speech in progress; that is, such an ending is neither an ending of an ending nor an ending of a stop. "Thinking" can be understood as something under consideration or a pause; that is, such an ending is simply an ending of a pause, after which speech may continue. "Ending" can be understood as a stop or an ending; that is, such an ending is an ending of a speech ending.

[0108] In some implementations, speech endpoint classification models can be obtained by using language class models, such as the bidirectional encoder representations from transformers (BERT) model. While the BERT model is used as an example below for illustrative purposes, it should be understood that any other language class model capable of performing the aforementioned classification can be used as an alternative.

[0109] Figure 3 is a schematic diagram of the processing of integrated features by a classification model according to an embodiment of the present invention. As shown in Figure 3, the classification model may be a BERT model. The BERT model includes an embedding layer and a fully connected layer (shown as white frame C in Figure 3). The embedding layer includes token embeddings, segment embeddings, and position embeddings. The embedding layer outputs the results obtained by processing the input data (i.e., integrated features) to the fully connected layer. The fully connected layer further obtains the speech endpoint category.

[0110] As shown in Figure 3, the input data provides examples of five integrated features represented by In1 to In5 in Figure 3. In1 is "[CLS]open the car window[SPE][SIL][SEP]". In2 is "[CLS]open the car window[SIL][SPE][SIL][SIL][SEP]". In3 is "[CLS]please adjust the temperature to 20[SIL][NEU][SIL][SIL][SEP]". In4 is "[CLS]please adjust the temperature to 26[SIL][SIL][SIL][SEP]". In5 is "[CLS]please adjust the temperature to 26[SIL][SIL][SEP]". Each circle represents one element. [CLS] is the start data of the input data for the BERT model, and is sometimes called the start marker. [SEP] is the end data of the input data for the BERT model, and is sometimes called the termination marker. In other words, in this example, the format of the integrated feature is "[CLS] + semantic + speech category + [SEP]".

[0111] As shown in Figure 3, EX represents the energy value of X, for example, ECLS represents the energy corresponding to [CLS]. Input data processing performed by the embedding layer and fully connected layer of the BERT model allows us to obtain speech endpoint categories.

[0112] Figure 3 is merely a concrete example of a speech endpoint classification model, and is not limited to this example. The training method for the speech endpoint classification model will be described later, so it will not be explained here.

[0113] Figure 4 is a schematic flowchart of a speech recognition method according to an embodiment of the present invention. The steps in Figure 4 will be described below. The method shown in Figure 4 may be performed by an electronic device. Specifically, the electronic device may include one or more devices such as a computer, smartphone, tablet computer, personal digital assistant (PDA), wearable device, smart speaker, television, unmanned aerial vehicle, vehicle, in-vehicle chip, in-vehicle equipment (e.g., in-vehicle infotainment or in-vehicle computer), or robot. Alternatively, the method 400 shown in Figure 4 may be performed by a cloud service device. Alternatively, the method 400 shown in Figure 4 can be implemented by a system including a cloud service device and terminal devices, such as an in-vehicle system or a smart home system.

[0114] For example, the method shown in Figure 4 can be implemented using the speech recognition device 100 shown in Figure 1.

[0115] 401: Retrieve audio data. The audio data contains multiple audio frames.

[0116] Audio data may be acquired using speech capture equipment such as a microphone, or it may be acquired from a storage device or network. Audio data may be acquired in real time or it may already be stored. Step 401 may be performed using the acquisition module 110 described above. For a description of the audio data and how to acquire it, please refer to the description above. Further details will not be provided here.

[0117] Audio frames may be obtained by performing a framing operation on the audio data. For example, the duration of a single audio frame may be several tens of milliseconds or even tens of milliseconds.

[0118] 402: Extract the audio category and meaning from multiple audio frames.

[0119] Step 402 may be performed using the processing module 120 described above. See above for explanations of speech categories and meanings. Details are not described again here. Speech categories may be extracted from each audio frame, but meanings do not have to be extracted from each audio frame. Specifically, audio frames that do not contain human speech have no meaning. Therefore, it is not possible to extract meaning from audio frames that do not contain speech. In this case, the meaning of an audio frame that does not contain speech can be considered empty or null.

[0120] Optionally, the speech category can be obtained based on the relationship between the energy of an audio frame and a preset energy threshold. For example, the preset energy threshold includes a first energy threshold and a second energy threshold, where the first energy threshold is greater than the second energy threshold. Among multiple audio frames, the speech category of an audio frame whose energy is greater than or equal to the first energy threshold can be determined as "Speech". Among multiple audio frames, the speech category of an audio frame whose energy is less than the first energy threshold but greater than the second energy threshold can be determined as "Neutral". Here, the first energy threshold is greater than the second energy threshold. Among multiple audio frames, the speech category of an audio frame whose energy is less than or equal to the second energy threshold can be determined as "Silence".

[0121] Optionally, an audio stream conveying meaning may be obtained using equipment such as an automatic speech recognition device. Each segment of the audio stream can be represented by corresponding text. Each segment of the audio stream may contain one or more audio frames.

[0122] For example, the audio category for multiple audio frames may include the audio category for each of the multiple audio frames.

[0123] Alternatively, the audio category of multiple audio frames may include some of the audio categories from those multiple audio frames.

[0124] In other words, in step 402, you may extract all audio categories from multiple audio frames, or you may extract audio categories from only some of the multiple audio frames.

[0125] Furthermore, audio categories may be extracted from each audio frame. In other words, an audio category corresponding to each audio frame may be obtained. Typically, multiple audio frames correspond to one word. That is, multiple audio frames represent one word. Figure 3 is used as an example. In Figure 3, each word in "open the car window" corresponds to multiple audio frames, and each audio category corresponds to one audio frame.

[0126] 403: Retrieve the speech end point of audio data based on speech category and meaning.

[0127] For example, step 403 may be performed using the decision module 130 described above.

[0128] Optionally, method 400 further includes step 404 (not shown).

[0129] 404: After obtaining the speech end point, respond to the command corresponding to the audio data prior to the speech end point of the audio data.

[0130] In other words, the actions corresponding to audio data prior to the end of the speech in the audio data may be performed after the end of the speech has been obtained. Furthermore, the actions corresponding to audio signals prior to the end of the speech in the audio data can also be understood as actions corresponding to the end of the speech.

[0131] Furthermore, the operation corresponding to the audio data prior to the end of the speech may be performed immediately after the end of the speech is obtained, or it may be performed a certain period of time after the end of the speech is obtained. This period may be redundancy time, error time, etc.

[0132] The action corresponding to the end of a speech may also be an action within the service processing function.

[0133] For example, speech recognition may be stopped after obtaining the end point of the speech. Alternatively, the speech recognition result may be returned to the user after obtaining the end point of the speech. Alternatively, the speech recognition result may be sent to a subsequent module after obtaining the end point of the speech, so that the subsequent module can perform the corresponding action. For example, the audio data may contain control instructions, so that the subsequent module can perform the control action corresponding to those instructions. For example, the audio data may contain query instructions, so that the subsequent module can return a response sentence corresponding to the query instructions to the user.

[0134] The action corresponding to the end of the speech, i.e., step 404, may be performed by the execution device of method 400 or by another device. This is not limited to the present embodiment of the application.

[0135] The following explanation uses the example of including user commands in an audio signal. User commands are used to implement various functions, such as obtaining a path, playing music, and controlling hardware (e.g., lighting or air conditioning). We will use air conditioning control as an example. In this case, the user command could be "turn on the air conditioner." Note that the subsequent module may be one module or multiple modules.

[0136] For example, after obtaining the end point of a speech, instruction information indicating the end point of the speech is sent to a subsequent module. The subsequent module then retrieves the audio data prior to the end point of the speech, obtains semantic text (e.g., "turn on the air conditioner") based on the audio data, parses the user command based on the semantic text, and controls the corresponding module to perform the action indicated by the voice command.

[0137] For example, after obtaining the end point of the speech, the ASR can be instructed to stop speech recognition, and the ASR can send the speech recognition result (e.g., the semantic text of "turn on the air conditioner") to the semantic analysis module. The semantic analysis module can then analyze the user command and control the air conditioner by sending a control signal to turn it on.

[0138] For example, after obtaining the end point of a speech, by sending the audio data prior to the end point to a subsequent module, the subsequent module can obtain semantic text (e.g., "turn on the air conditioner") based on the audio data, then parse the user command based on the semantic text, and control the corresponding module to perform the action indicated by the voice command.

[0139] For example, after obtaining the end point of a speech, semantic text (e.g., "turn on the air conditioner") can be obtained based on the audio data, and then the user command can be analyzed based on the semantic text, and the corresponding module can be controlled to perform the action indicated by the voice command.

[0140] For example, the speech end point may be obtained based on all audio categories and meanings of multiple audio frames.

[0141] For example, one might predetermine text endpoints that could be speech endpoints based on their meaning, and then obtain speech endpoints from these potential text endpoints based on the audio category of each text endpoint. In another example, one might predetermine candidate audio frames that could be speech endpoints based on their audio category, and then determine speech endpoints based on the meaning preceding these candidate audio frames.

[0142] For example, a set of audio frames may include a first audio frame and a second audio frame, where the first audio frame is a meaningful audio frame, and the second audio frame is the audio frame that follows the first audio frame among the set of audio frames. Based on the meaning and the audio category of the second audio frame, the speech end point can be obtained.

[0143] The first audio frame consists of multiple meaningful audio frames. The second audio frame consists of one or more audio frames that follow the first audio frame.

[0144] Note that "multiple meaningful audio frames" and "multiple audio frames" contained in audio data are different concepts. The number of audio frames contained in the first audio frame is less than the number of audio frames contained in the audio data.

[0145] Furthermore, by integrating the meaning with the speech category of the second audio frame, an integrated feature can be obtained, and then the speech end point can be obtained based on the integrated feature. By processing using the integrated feature, processing efficiency and accuracy can be improved.

[0146] In some implementations, the speech end category may include "speak," "think," and "end." The speech end category of the audio data may be determined based on its meaning and the speech category of the second audio frame, and the speech end may be obtained if the speech end category of the audio data is "end."

[0147] The fact that the speech end category of audio data is "end" can be understood as the audio data containing a text end point whose speech end category is "end". The audio frame corresponding to a text end point whose speech end category is "end" may also be used as the speech end point.

[0148] Furthermore, if integrated features are obtained, the speech end point category of the audio data may be determined based on the integrated features, and the speech end point may be obtained if the speech end point category of the audio data is "end".

[0149] Optionally, a speech end classification model may be used to process the meaning and speech categories of the second audio frame to obtain speech end categories, thereby obtaining the speech end points of the audio data.

[0150] Alternatively, if integrated features are obtained, the integrated features may be processed using a speech end-point classification model to obtain the speech end-point category of the integrated features, thereby obtaining the speech end points of the audio data.

[0151] The speech endpoint classification model may be obtained using speech samples and the endpoint category labels of the speech samples. Furthermore, the format of the speech samples corresponds to the format of the integrated features, and the endpoint categories included in the endpoint category labels correspond to the speech endpoint categories.

[0152] In the solution shown in Figure 4, the speech end point of the audio data is obtained by extracting and combining meaning and speech categories from the audio data, thus enabling more accurate determination of the speech end point of the audio data.

[0153] The audio data may be real-time audio data or stored audio data. These two cases can be understood as online speech recognition and offline speech recognition, respectively. After obtaining the speech end point, the speech end point can be used to perform subsequent actions such as speech-based control. For example, the speech end point can be used to control switches on electronic devices, execute information queries, or control audio / video playback. In the case of online speech recognition, after obtaining the speech end point, the acquisition of audio data may be terminated, i.e., speech recognition may be stopped. Alternatively, in the case of online speech recognition, after obtaining the speech end point, the system may execute commands corresponding to the audio data prior to the speech end point and continue acquiring audio data.

[0154] According to the solution of this embodiment of the present application, the end point of a speech can be acquired more accurately, thereby enabling a more accurate response to subsequent speech-based actions and improving the user experience. Specifically, the solution of the present application avoids excessively long response delays caused by delays in speech end point detection, allowing for faster acquisition of the speech end point and timely subsequent responses, thereby reducing user waiting time and improving the user experience. Furthermore, because the solution of the present application can acquire accurate speech end points, the user's voice commands are not interrupted midway through early detection of the speech end point, and complete, meaningful audio data can be acquired. This allows for accurate understanding of the user's intent, accurate responses, and an improved user experience.

[0155] For example, in a scenario where an electronic device is controlled using voice, suppose the user generates a total of 7 seconds of audio data. Here, The first The audio consists of three seconds of speech saying "please turn on the air conditioner," a one-second pause at the fourth second, and a cough from five to seven seconds. Using the solution of this embodiment of the present application, the word "conditioner" can be accurately identified as the end of the speech, allowing speech acquisition to be terminated after the third or fourth second. Conventional activity detection-based methods require setting a certain waiting time, and speech acquisition is not terminated until the actual waiting time exceeds this waiting time. Here, an activity detection-based method is used, with a waiting time of two seconds. In this case, the one-second pause is considered a temporary pause, and speech recognition continues. Therefore, the subsequent three seconds of coughing must be recognized, and another certain waiting time must be waited after the cough. Compared to the solution of this embodiment of the present application, the activity detection-based method delays the termination of speech acquisition by at least a total of five seconds.

[0156] In a scenario where an electronic device is controlled using voice, suppose a user generates a total of 6 seconds of audio data saying "please call Qian Yi'er". Here, there is a 1.5-second pause after the word "Qian", and a fixed waiting time of 1.5 seconds in the activity detection-based method. Using the solution of this embodiment of the present application, the word "er" can be accurately acquired as the speech end point. According to the semantic information, the word "Qian" cannot be used as the "end" point unless there are no further audio frames following the word "Qian" whose voice category is "silence". Therefore, speech acquisition will end at the 6th second or later, rather than after the 1.5-second pause. However, when using the activity detection-based method, speech acquisition ends when the 1.5-second pause after the word "Qian" ends. As a result, the speech end point is incorrectly determined and cannot be addressed in accordance with the subsequent control policy.

[0157] The solution of this embodiment of the present application is fast blood When used for recognition, the waiting time until the end of a speech is not constant but changes along with the actual speech recognition process. Compared to conventional methods that pre-set a fixed waiting time and acquire the end of a speech after the waiting time has ended, this solution can acquire the end of a speech more accurately, thereby improving the timeliness and accuracy of subsequent responses, reducing user waiting time, and enhancing the user experience.

[0158] Embodiments of the present invention further provide a speech recognition method 500. Method 500 includes steps 501 to 506. The following describes each step of Method 500. Method 500 can be understood as an example of an online speech recognition method.

[0159] 501: Retrieve the first audio data.

[0160] The first audio data can be acquired in real time. The first audio data may contain multiple audio frames.

[0161] 502: Determine the end point of the first speech in the first audio data.

[0162] Optionally, the first audio data includes multiple audio frames. Step 502 may include steps of extracting the meaning and audio categories of the multiple audio frames, and obtaining the first speech end point of the first audio data based on the meaning and audio categories of the multiple audio frames.

[0163] For specific methods on determining the end point of the first speech, please refer to Method 400 mentioned above. Simply replace "audio data" with "first audio data" in Method 400. Further details will not be explained again here.

[0164] 503: After obtaining the end point of the first speech, respond to the command corresponding to the audio data prior to the end point of the first speech in the first audio data.

[0165] In other words, the operation corresponding to the audio data prior to the end of the first speech may be performed after the end of the first speech has been obtained.

[0166] To simplify the explanation, the instruction corresponding to the audio data prior to the end of the first speech in the first audio data will be referred to as the first user instruction.

[0167] The first speech end point is the speech end point of the first user instruction. The first audio data may contain only one user instruction, and the speech end point of the instruction can be identified by performing step 502.

[0168] 504: Retrieve the second audio data.

[0169] Note that the sequence numbers for each step in Method 500 are used only to facilitate explanation and do not restrict the execution order of each step. In Method 500, the audio data processing and the audio data acquisition processes may be made independent. That is, while step 502 is being executed, if the second audio data is audio data acquired after the first audio data, the audio data acquisition may continue. In other words, step 504 may be executed.

[0170] For example, the first audio data and the second audio data may be audio data captured consecutively.

[0171] 505: Determine the end point of the second speech in the second audio data.

[0172] Optionally, the second audio data may contain multiple audio frames. Step 50 5 This may include the steps of extracting meaning and audio categories of multiple audio frames, and obtaining a second speech endpoint of second audio data based on the meaning and audio categories of multiple audio frames.

[0173] Alternatively, the audio data, which includes the second audio data and the audio data that follows the end of the first speech and is included in the first audio data, includes multiple audio frames, step 50 5 This may include the steps of extracting the speech categories and meanings of multiple audio frames, and obtaining the second speech end point of the second audio data based on the speech categories and meanings of the multiple audio frames.

[0174] For example, the audio data included in the first audio data and following the first speech end point contains 4 audio frames, and the second audio data contains 10 audio frames. In this case, the audio data containing 4 audio frames and 10 audio frames contains 14 audio frames. The audio categories and meanings of the 14 audio frames are extracted, and the second speech end point of the second audio data is obtained based on the audio categories and meanings of the 14 audio frames.

[0175] For specific methods on determining the end point of the second speech, please refer to Method 400 mentioned above. Further details will not be explained again here.

[0176] 506: After obtaining the second speech end point, respond to the command corresponding to the audio data between the first speech end point in the first audio data and the second speech end point in the second audio data.

[0177] For example, the audio data included in the first audio data and following the first speech end point contains four audio frames, and the second audio data contains ten audio frames. In this case, the audio data containing four audio frames and ten audio frames contains fourteen audio frames, and the second speech end point of the second audio data is located at the twelfth audio frame. After obtaining the second speech end point, the system responds to commands corresponding to audio data prior to the twelfth audio frame among the fourteen audio frames.

[0178] To simplify the explanation, the instruction corresponding to the audio data between the first speech end point of the first audio data and the second speech end point of the second audio data will be called the second user instruction.

[0179] The second speech end point is the speech end point of the second user instruction. The audio data, which includes the second audio data and the audio data after the first speech end point of the first audio data, may contain only one user instruction, and the speech end point of that instruction can be identified by performing step 505. The first and second user instructions may be two user instructions sent consecutively by the user, i.e., the time interval between the two user instructions is relatively small. For example, if the first and second audio data are consecutively captured audio data, the time interval between the first and second user instructions is relatively small. The solution of the present invention helps to distinguish between the speech end point of the first user instruction and the speech end point of the second user instruction.

[0180] According to the solution of this embodiment of the present application, speech end points can be acquired more accurately, and excessively long response delays due to delays in speech end point detection can be avoided. As a result, speech end points can be acquired more quickly, subsequent responses can be made in a timely manner, reducing user waiting time and improving the user experience. Specifically, the solution of this embodiment of the present application can acquire audio data in real time and identify speech end points in the audio data, thereby identifying the speech end points of different commands in real time and responding to each command after acquiring the speech end point of the command. In particular, when the intervals between multiple commands sent by the user are relatively short, using the solution of the present application helps to identify the speech end point of each command after it has been sent and respond to each command in a timely manner, rather than responding to all commands after all of them have been sent.

[0181] For example, in a scenario where an electronic device is controlled using voice, suppose a user generates a total of 8 seconds of audio data, which includes two user commands: "please close the car window" and "please turn on the air conditioner." The interval between the two user commands is relatively short, for example, 1 second, and the user sends the command "please turn on the air conditioner" 1 second after sending the command "please close the car window." Using the solution of this embodiment of the present invention, the speech end point corresponding to the first user command can be obtained by acquiring and processing the audio data in real time. Based on semantic information, when the word "window" is followed by several audio frames whose speech category is "silence," the word "window" can be used as the "end" point. That is, after multiple audio frames following the word "window," it can be determined that the speech corresponding to the user command has ended, and the action of closing the car window in response to the user command can be executed in a timely manner. Furthermore, according to the solution of this embodiment of the present invention, audio data may be acquired and processed to obtain the speech end point corresponding to the second user command. For example, after several audio frames following the word "conditioner," it can be determined that the speech corresponding to the user command has ended, and the action of turning on the air conditioner in response to the user command can be executed in a timely manner. If the waiting time in the activity detection-based method described above is 1.5 seconds, the aforementioned 1-second pause is considered a temporary pause and speech recognition continues. Therefore, it is not possible to obtain the speech end point corresponding to the first user command. In this case, the speech is considered to have ended for the first time 1.5 seconds after the user sends the second user command, and the actions corresponding to each of the two user commands are performed.

[0182] In other words, the solution of this embodiment of the present application can accurately acquire the speech end point corresponding to multiple user commands and respond to each user command in a timely manner. In particular, when the interval between multiple user commands is relatively short, the solution of this embodiment of the present application can acquire the speech end point of each user command more accurately and respond in a timely manner after each user command has been sent, rather than waiting until the user has sent all user commands before responding.

[0183] Figure 5 is a schematic flowchart of the speech recognition method according to an embodiment of the present invention. Figure 5 can be considered a specific example of the method shown in Figure 4. In this example, audio data acquisition and speech recognition are performed in real time.

[0184] 601: Acquire audio data in real time.

[0185] Audio data may be acquired using a speech capture device such as a microphone.

[0186] Alternatively, step 601 may be performed using the acquisition module 110 described above.

[0187] 602: Use ASR to perform speech recognition on audio data and obtain a meaningful audio stream.

[0188] Step 602 is a concrete example of how to obtain meaning, specifically, an example of obtaining meaning using ASR.

[0189] Optionally, if the ASR identifies that the audio stream has ended, steps 603 through 606 may be omitted, and step 607 may be performed directly. This corresponds to a situation where no speech is recognized for a relatively long time interval in the acquired audio data, and it is no longer necessary to determine the end of the speech.

[0190] Step 602 may also be performed using the processing module 120 described above.

[0191] 603: Obtain the audio category of the audio frame in the audio data based on the relationship between the energy of the audio data and a preset energy threshold.

[0192] Step 603 is a specific example of how to obtain the audio category.

[0193] Steps 602 and 603 may be executed simultaneously or at different times; the order of execution is not limited. Steps 602 and 603 can be considered as specific examples of step 402.

[0194] Step 603 may also be performed using the processing module 120 described above.

[0195] 604: Integrate semantic and phonetic categories to obtain integrated features.

[0196] Integrated features may be obtained by overlaying semantic and phonetic categories.

[0197] For example, step 602 may be performed in real time. That is, speech recognition may be performed in real time on the acquired audio data. In step 604, each time a word is recognized, the current meaning and audio category of one or more audio frames following that word are superimposed to obtain a combined feature, and this combined feature is input into the subsequent speech endpoint classification model for processing (i.e., step 605). For example, suppose the complete command sent by the user is "I want to watch television," and the instruction currently being sent is "I want to watch." In step 602, speech recognition is performed on the currently acquired audio data. After the word "watch" is recognized, a combined feature can be obtained based on the meaning of "I want to watch" and the audio categories of one or more audio frames following "watch." Then, in step 605, the combined feature is input into the speech endpoint classification model for processing. After the user then sends "television," speech recognition is performed on the currently acquired audio data via step 602. After the word "television" is recognized, a combined feature can be obtained based on the meaning "I want to watch television" and the audio category of one or more audio frames that follow "television". Then, in step 605, the combined feature is input into the speech endpoint classification model for processing.

[0198] Processing using integrated features can improve processing efficiency and accuracy compared to the direct processing described above. For specific details, please refer to the explanation of the input data in Figure 3. Further details will not be explained again here.

[0199] Step 604 may also be performed using the decision module 130 described above.

[0200] 605: Process integrated features using a speech end point classification model to obtain speech end point categories.

[0201] The speech end point category can include "Speak," "Think," and "End." If the speech end point category is "End," the audio frame corresponding to that end point becomes the speech end point.

[0202] Steps 604 and 605 can be considered as specific examples of step 403.

[0203] Step 605 may also be performed using the decision module 130 described above.

[0204] 606: Determine whether the speech end point category is "end" or not. If the result is "Yes", execute step 607; if the result is "No", execute step 601.

[0205] Step 606 may also be performed using the decision module 130 described above.

[0206] 607: Outputs the end point of the speech.

[0207] Step 607 may also be performed using the decision module 130 described above.

[0208] Figure 6 is a schematic flowchart of the speech endpoint classification model training method according to the embodiment of the present invention. The steps in Figure 6 will be explained below.

[0209] 701: Obtain training data. The training data includes speech samples and endpoint category labels for the speech samples.

[0210] Furthermore, the format of the speech sample may correspond to the format of the data input to the speech endpoint classification model, and the endpoint categories included in the endpoint category labels may correspond to the speech endpoint categories described above. That is, the speech sample includes the audio category and meaning of the audio data, and the endpoint category labels include "speak," "think," and "end."

[0211] The data input to the speech endpoint classification model is the input data for the inference phase of the speech endpoint classification model, for example, the semantic and speech categories of the second audio frame as described above. That is, the format of the speech sample may correspond to the format of the semantic and speech categories of the second audio frame as described above. As another example, the input data for the inference phase of the speech endpoint classification model may be the integrated features described above. That is, the format of the speech sample may correspond to the format of the integrated features described above.

[0212] In some embodiments, the speech sample may be in the format of "starter + meaning + phonetic category + ender", or the speech sample may be in the format of "starter + phonetic category + meaning + ender".

[0213] Optionally, several text corpora may be obtained, a dictionary tree may be established from these corpora, and each node in the dictionary tree (each node corresponding to one word) may contain the following information: whether the node is an endpoint and its prefix frequency. Speech samples may then be generated based on the node information. An endpoint is the end of a sentence. Prefix frequency is used to represent the number of words between a word and an endpoint. A higher prefix frequency indicates that the word is less likely to be an endpoint. For example, verbs like "give" and "take," and other words like prepositions, are relatively less likely to be used as the end of a sentence and have a relatively high prefix frequency in the dictionary tree. On the other hand, words like "right?" and "correct?" are relatively more likely to be used as the end of a sentence and usually have a relatively low prefix frequency in the dictionary tree.

[0214] If a node is not an endpoint, the endpoint category label for its plaintext (i.e., semantics) is "speaking," and the endpoint category label for its signal classification information (i.e., speech category) is "thinking." If a node is an endpoint, speech samples with different endpoint category labels are generated based on the prefix frequency and the signal classification information (i.e., speech category). A higher prefix frequency indicates that more audio frames with the speech category "silence" are needed for such a node to be marked as an "end." The following provides an explanation with reference to Figure 7.

[0215] Figure 7 is a schematic diagram of a dictionary tree according to an embodiment of the present invention. In the dictionary tree shown in Figure 7, each node contains one word, a gray circle indicates that the node is not an endpoint, a white circle indicates that the node is an endpoint, and prefix frequencies are represented by numbers or letters (e.g., 0, 1, 2, x, y, z in the figure). For example, "five, 0" enclosed in a white circle indicates that the node is an endpoint, and its prefix frequency is 0. "ten, z" enclosed in a white circle indicates that the node is an endpoint, and its prefix frequency is z = x + 5 + y + 2 + 1 + 0 + 0, which is the sum of the prefix frequencies of all nodes from that node to the end of the dictionary tree. Note that the letters in Figure 7 are used to illustrate the method of marking the prefix frequencies of nodes in the dictionary tree. In practice, it is not necessary to introduce letters, and after the dictionary tree is constructed, the prefix frequencies of the nodes in the dictionary tree are fixed.

[0216] Step 701 may be performed by a training device. The training device may be, for example, a cloud service device, a terminal device such as a computer, server, mobile phone, smart speaker, vehicle, unmanned aerial vehicle, robot, or a system including a cloud service device and a terminal device. This is not limited to the present embodiment of the application.

[0217] 702: Train a speech endpoint classification model using training data to obtain a target speech endpoint classification model.

[0218] For the speech end point classification model, please refer to the explanation above. Details will not be listed again. The target speech end point classification model can be used to obtain the speech end points of audio data based on speech category and semantics. The target speech end point classification model can be used to perform step 403, or it can be used by the decision module 130 to obtain the speech end points.

[0219] Step 702 may be performed by a training device.

[0220] Figure 8 is a schematic block diagram of a speech recognition device according to an embodiment of the present invention. The device 2000 shown in Figure 8 includes an acquisition unit 2001 and a processing unit 2002.

[0221] The acquisition unit 2001 and the processing unit 2002 can be configured to perform the speech recognition method according to the embodiments of this application. For example, the acquisition unit 2001 may perform step 401 and the processing unit 2002 may perform steps 402 and 403. As another example, the acquisition unit 2001 may perform steps 501 and 504 and the processing unit 2002 may perform steps 502 and 503, 505 and 506. As yet another example, the acquisition unit 2001 may perform step 601 and the processing unit 2002 may perform steps 602 to 606.

[0222] The acquisition unit 2001 may include an acquisition module 110, and the processing unit 2002 may include a processing module 120 and a decision module 130.

[0223] Please understand that the processing unit 2002 of device 2000 may be equivalent to the processor 3002 of device 3000, which is described below.

[0224] It should be noted that device 2000 is implemented in the form of a functional unit. The term "unit" here may be implemented in the form of software and / or hardware, but is not specifically limited to that form.

[0225] For example, “unit” could be a software program, hardware circuitry, or a combination thereof for performing the functions described above. Hardware circuitry may include application-specific integrated circuitry (ASIC), electronic circuits, a processor (e.g., a shared processor, a dedicated processor, or a group processor) configured to run one or more software or firmware programs and memory, combinational logic circuits, and / or other suitable components that support the functions described.

[0226] Accordingly, the units in the examples described in the embodiments of this application can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art may use different methods to implement the described functions for each specific application, but the implementation should not be considered to be beyond the scope of this application.

[0227] Figure 9 is a schematic diagram of the hardware structure of a speech recognition device according to an embodiment of the present invention. The speech recognition device 3000 shown in Figure 9 (device 3000 may specifically be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, the processor 3002, and the communication interface 3003 are connected to each other so as to be able to communicate with each other using the bus 3004.

[0228] The memory 3001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are configured to perform the steps of the speech recognition method in the embodiment of this application.

[0229] The processor 3002 may be a central processing unit (CPU), a microprocessor, an ASIC, a graphics processing unit (GPU), or one or more integrated circuits, and is configured to execute a related program, implement a function that needs to be performed by the processing unit 2002 of the speech recognition device in the embodiment of this application, or execute the speech recognition method in the embodiment of the method of this application.

[0230] Alternatively, the processor 3002 may be an integrated circuit chip having signal processing capabilities. In the implementation process, the steps of the conversation recognition method of the present application may be implemented using hardware integrated logic circuits or instructions in the form of software within the processor 3002. Alternatively, the processor 3002 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA), or another programmable logic element, discrete gate or transistor logic element, or discrete hardware component that can implement or execute the methods, steps, and logic block diagrams disclosed in embodiments of the present application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the methods disclosed with reference to embodiments of the present application may be implemented directly by a hardware decoding processor, or by using a combination of hardware and software modules within the decoding processor. The software modules may reside in conventional mature storage media such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory 3001. The processor 3002 reads the information in memory 3001 and, in combination with the processor hardware, performs functions that need to be performed by units included in the speech recognition device of the embodiment of the present invention, or performs the speech recognition method of the embodiment of the method of the present invention. For example, the processor 3002 may perform steps 402 and 403. As another example, the processor 3002 may perform steps 502 and 503, and steps 505 and 506. For example, the processor 3002 may perform steps 602 to 606.

[0231] As an example, and not limited to this, transceiver equipment such as a transceiver may be used as the communication interface 3003 to enable communication between the device 3000 and other devices or a communication network. The communication interface 3003 may be configured to perform functions that need to be performed by the acquisition unit 2001 shown in Figure 8. For example, the communication interface 3003 may perform step 401. As another example, the communication interface 3003 may perform steps 501 and 504. As yet another example, the communication interface 3003 may perform step 601. That is, the above audio data may be acquired using the communication interface 3003.

[0232] The bus 3004 may include paths for transferring information between components of the device 3000 (e.g., memory 3001, processor 3002, and communication interface 3003).

[0233] The speech recognition device 3000 may also be installed in the in-vehicle system. Specifically, the speech recognition device 3000 may be installed in an in-vehicle terminal. Alternatively, the speech recognition device may be installed in a server.

[0234] For illustrative purposes, this explanation will only describe an example where the speech recognition device 3000 is installed in a vehicle. The speech recognition device 3000 may be installed in other devices as an alternative. For example, the speech recognition device 3000 may be applied to devices such as computers, servers, mobile phones, smart speakers, wearable devices, unmanned aerial vehicles, and robots.

[0235] Although only memory, a processor, and a communication interface are shown in Figure 9 of the device 3000, a person skilled in the art should understand that in certain implementation processes, the device 3000 may include other components necessary for normal operation. Furthermore, a person skilled in the art should understand that, based on specific requirements, the device 3000 may include additional hardware components to implement other additional functions. In addition, a person skilled in the art should understand that the device 3000 may include only the components necessary to carry out this embodiment of the application, and may not necessarily include all the components shown in Figure 9.

[0236] Those skilled in the art will readily understand that the units and algorithmic steps described as examples with reference to the embodiments disclosed in this specification can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different equipment to perform the described functions for each specific application, but the implementation should not be considered to be beyond the scope of this application.

[0237] For the sake of ease and conciseness of explanation, it will be readily apparent to those skilled in the art that, for specific operating processes of the aforementioned systems, apparatus, and units, one can refer to the corresponding processes in the embodiments of the methods described above. Further details will not be described again here.

[0238] It should be understood that in some embodiments provided herein, the disclosed systems, methods, and devices can be implemented in other ways. For example, the embodiments of the described devices are merely examples. For example, the division into units is merely a logical functional division, and there may be other divisions during actual implementation. For example, multiple units or components may be coupled or integrated into another system, or some features may be ignored or not performed. Furthermore, the mutual coupling, direct coupling, or communication connection shown or discussed may be implemented using some interfaces. Indirect coupling or communication connection between devices or units may be implemented electronically, mechanically, or in other forms.

[0239] Units described as separate components may or may not be physically separated. Components shown as units may or may not be physical units, may be located in one place, or may be distributed among multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solution of the embodiment.

[0240] Furthermore, the functional units in the embodiments of the present invention may be integrated into a single processing unit, or each unit may exist physically independently, or two or more units may be integrated into a single unit.

[0241] When a function is implemented in the form of a software function unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The storage medium includes any medium capable of storing program code, such as a Universal Serial Bus flash disk (USB flash disk, UFD), which may also be called a USB flash drive, removable hard disk, ROM, RAM, magnetic disk, or optical disk.

[0242] The foregoing description is merely a specific implementation of the present application and is not intended to limit the scope of protection of the present application. Any modifications or substitutions that can be readily conceived by a person skilled in the art, within the scope of the technical scope disclosed herein, should be included within the scope of protection of the present application. Accordingly, the scope of protection of the present application should be subject to the scope of protection of the claims.

Claims

1. A speech recognition method, A step of acquiring audio data containing multiple audio frames, wherein the multiple audio frames include a first audio frame and a second audio frame, the first audio frame is an audio frame having meaning in the language contained in the multiple audio frames, and the second audio frame is an audio frame that follows the first audio frame among the multiple audio frames and does not have the aforementioned meaning; A step of extracting the meaning from the first audio frame and the speech category of the second audio frame from the second audio frame using an automatic speech recognition (ASR) method, wherein the step of extracting the meaning and the speech category includes a step of obtaining the speech category of the second audio frame based on the relationship between the energy of the second audio frame and a preset energy threshold, A step of obtaining the speech end point of the audio data based on the aforementioned meaning and the aforementioned audio category, A method that includes this.

2. The method according to claim 1, further comprising the step of obtaining the speech end point and then responding to an instruction corresponding to audio data prior to the speech end point of the audio data.

3. The aforementioned voice categories include "speech," "neutral," and "silence," and the aforementioned preset energy thresholds include a first energy threshold and a second energy threshold, wherein the first energy threshold is greater than the second energy threshold. Among the second audio frames, the audio category of the audio frame whose energy is equal to or greater than the first energy threshold is "speech". Among the second audio frames, the audio category of an audio frame whose energy is less than the first energy threshold and greater than the second energy threshold is "neutral," or The method according to claim 2, wherein the audio category of the audio frame among the second audio frames whose energy is less than or equal to the second energy threshold is "silence".

4. The method according to claim 3, wherein the first energy threshold and the second energy threshold are determined based on the energy of the background sound of the audio data.

5. The speech end point categories include "speak," "think," and "end," and the step of obtaining the speech end point based on the aforementioned meaning and the audio category of the second audio frame is: The method according to claim 1, comprising the steps of determining the speech end point category of the audio data based on the meaning and the audio category of the second audio frame, and obtaining the speech end point if the speech end point category of the audio data is "end".

6. A speech recognition device, An acquisition unit configured to acquire audio data containing multiple audio frames, wherein the multiple audio frames include a first audio frame and a second audio frame, the first audio frame is an audio frame having meaning in the language contained in the multiple audio frames, and the second audio frame is an audio frame that follows the first audio frame among the multiple audio frames and does not have the aforementioned meaning, A processing unit configured to extract the meaning from the first audio frame and the speech category of the second audio frame from the second audio frame using an automatic speech recognition (ASR) method, Includes, The aforementioned processing unit is Based on the relationship between the energy of the second audio frame and a preset energy threshold, the audio category of the second audio frame is obtained. A device further configured to acquire the speech endpoint of the audio data based on the aforementioned meaning and the aforementioned audio category.

7. The apparatus according to claim 6, wherein the processing unit is further configured to respond to commands corresponding to audio data prior to the speech end point of the audio data after obtaining the speech end point.

8. The aforementioned voice categories include "speech," "neutral," and "silence," and the aforementioned preset energy thresholds include a first energy threshold and a second energy threshold, wherein the first energy threshold is greater than the second energy threshold. Among the second audio frames, the audio category of the audio frame whose energy is equal to or greater than the first energy threshold is "speech". Among the second audio frames, the audio category of an audio frame whose energy is less than the first energy threshold and greater than the second energy threshold is "neutral," or The apparatus according to claim 7, wherein the audio category of the audio frame among the second audio frames whose energy is less than or equal to the second energy threshold is "silence".

9. The apparatus according to claim 8, wherein the first energy threshold and the second energy threshold are determined based on the energy of the background sound of the audio data.

10. The device according to claim 6, wherein the speech end category includes “speak,” “think,” and “end,” and the processing unit is configured to specifically determine the speech end category of the audio data based on the meaning and the voice category of the second audio frame, and to acquire the speech end if the speech end category of the audio data is “end.”

11. A computer-readable storage medium, wherein the computer-readable storage medium stores program code used for execution by a device, and the program code includes instructions used to perform the method according to any one of claims 1 to 5.

Images