Speech recognition method, speech recognition device and system

By combining meaning and speech category extraction with adaptive energy thresholds, the method accurately determines speech endpoints, enhancing voice interaction systems' responsiveness and user experience.

JP2024543155AActive Publication Date: 2024-11-19YINWANG INTELLIGENT TECHNOLOGIES CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024531298
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2021-11-25
Publication Date
2024-11-19
Estimated Expiration
2041-11-25

AI Technical Summary

Technical Problem

Existing voice interaction systems face challenges in accurately determining the end-of-speech state due to late or premature detections, often caused by background noise or speech interruptions.

Method used

A method that combines the extraction of meaning and speech category from audio frames to accurately determine the end-of-speech point, adapting energy thresholds based on background noise levels, and using integrated features for improved processing efficiency and accuracy.

Benefits of technology

This approach enhances the timeliness and accuracy of voice command responses, reducing user waiting time and improving the overall user experience by accurately identifying speech endpoints.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543155000001_ABST
    Figure 2024543155000001_ABST
Patent Text Reader

Abstract

The present application provides a speech recognition method, speech recognition device and system, which are related to the field of artificial intelligence. The method includes the steps of obtaining audio data including a plurality of audio frames, extracting meaning and a phonetic category of the plurality of audio frames, and obtaining a speech end point of the audio data according to the meaning and the phonetic category. According to the solution, the speech end point of the audio data is obtained by extracting and combining the meaning and the phonetic category of the audio data, so that the speech end point of the audio data can be determined more accurately, and the subsequent voice-based operation can be responded to more accurately, and the user experience can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present application relate to the field of artificial intelligence, and more particularly to a speech recognition method, a speech recognition device and a system. [Background technology]

[0002] At present, voice interaction products, such as intelligent terminal devices, smart household devices, and intelligent vehicle-mounted devices, are widely used in people's daily life. In mainstream voice interaction products, voice interaction is actively initiated by users, but the end-of-speech state (i.e., the end point of speech in one round of interaction) is usually automatically determined by automatic speech recognition. At present, there are two main problems in end-of-speech state detection: late end-of-speech state caused by background noise and early end-of-speech state caused by speech interruption.

[0003] Therefore, a method for determining the end of speech more accurately is an urgent technical problem to be solved. Summary of the Invention

[0004] SUMMARY OF THE DISCLOSURE The embodiments of the present application provide a speech recognition method, speech recognition apparatus and system, which can more accurately determine the speech end state and respond to subsequent actions based on the speech more accurately.

[0005] According to a first aspect, there is provided a speech recognition method, comprising the steps of obtaining audio data comprising a plurality of audio frames, extracting meaning and a phonetic category of the plurality of audio frames, and obtaining speech end points of the audio data based on the meaning and the phonetic category of the plurality of audio frames.

[0006] In the technical solution of the present application, the speech end point of the audio data is obtained by extracting and combining the meaning and the voice category of the audio data, so that the speech end point of the audio data can be determined more accurately, so that the subsequent voice-based operation can be responded to more accurately and the user experience can be improved. In addition, when the solution of the present embodiment of the present application is used to perform speed recognition, the waiting time to the speech end point is not constant, but changes with the actual speech recognition process. Compared with the conventional method of setting a certain waiting time in advance and responding after the waiting time ends, the solution can obtain the speech end point more accurately, improve the timeliness and accuracy of the response to the user's voice instruction, and shorten the user's waiting time.

[0007] For example, the speech category of the multiple audio frames may include the speech category of each of the multiple audio frames. Alternatively, the speech category of the multiple audio frames may include the speech category of some of the multiple audio frames.

[0008] It should be noted that although a phonetic category may be extracted from each audio frame, meaning may not be extracted from each audio frame. Specifically, audio frames that do not contain human speech have no meaning. Thus, meaning cannot be extracted from audio frames that do not contain speech. In this case, the meaning of audio frames that do not contain speech can be considered to be empty or null.

[0009] Referring to the first aspect, in some implementations of the first aspect, the method further includes, after obtaining the speech end point, responding to an instruction corresponding to audio data prior to the speech end point of the audio data.

[0010] After obtaining the speech end point, the command may be responded to immediately, or after a certain time period. That is, the operation corresponding to the audio data before the speech end point in the audio data may be performed immediately after obtaining the speech end point, or the operation corresponding to the audio data before the speech end point in the audio data may be performed after a certain time period after obtaining the speech end point. This time period may be a redundant time, an error time, etc. According to the solution of the present application, the speech end point of the audio data can be determined more accurately, so that the command corresponding to the audio data before the speech end point can be responded to more accurately. This can improve the timeliness and accuracy of the response to the user's voice command, reduce the user's waiting time, and improve the user experience.

[0011] In a first aspect, in some embodiments of the first aspect, the voice category of the plurality of audio frames may be obtained based on a relationship between the energy of the plurality of audio frames and a preset energy threshold.

[0012] In some embodiments of the first aspect, the audio categories include "speech", "neutral" and "silence", the preset energy thresholds include a first energy threshold and a second energy threshold, and the first energy threshold is greater than the second energy threshold. Among the plurality of audio frames, the audio category of an audio frame whose energy is equal to or greater than the first energy threshold can be determined as "speech", A speech category of the audio frames of the plurality of audio frames having an energy less than the first energy threshold and greater than the second energy threshold may be determined as “neutral”; or The speech category of the audio frames of the plurality of audio frames whose energy is less than or equal to a second energy threshold may be determined as "silence."

[0013] In relation to the first aspect, in some implementations of the first aspect, the first energy threshold and the second energy threshold can be determined based on the energy of background sound of the first audio data. In different background environments, silence energy curves are different. For example, in a relatively quiet environment, the silence energy (i.e., the energy of the background sound) is relatively low, and in a relatively noisy environment, the silence energy (i.e., the energy of the background sound) is relatively high. Therefore, obtaining the first energy threshold and the second energy threshold based on the silence energy can adapt to the requirements in different environments.

[0014] Referring to the first aspect, in some embodiments of the first aspect, the plurality of audio frames include a first audio frame and a second audio frame, the first audio frame being an audio frame having the meaning, and the second audio frame being an audio frame following the first audio frame in the plurality of audio frames; obtaining speech end points of the audio data based on the meaning and the phonetic category, Obtaining the speech end point based on the meaning and a phonetic category of the second audio frame.

[0015] The first audio frames are a plurality of audio frames having the meaning, and the second audio frames are one or more audio frames following the first audio frames.

[0016] Note that the "plurality of meaningful audio frames" included in the audio data are different concepts from the "plurality of audio frames." The number of the audio frames included in the first audio frame is smaller than the number of the audio frames included in the audio data.

[0017] In relation to the first aspect, in some implementations of the first aspect, the meaning and the phonetic category of the second audio frame can be integrated to obtain an integrated feature of the plurality of audio frames, and then the speech end point can be obtained based on the integrated feature. By processing using the integrated feature, processing efficiency can be improved and accuracy can also be improved.

[0018] In relation to the first aspect, in some implementations of the first aspect, the speech end point category includes “speak”, “think”, and “end”. Then, the speech end point category of the audio data may be determined based on the meaning and the voice category of the second audio frame, and the speech end point may be obtained when the speech end point category of the audio data is “end”.

[0019] In addition, when the integrated features of the multiple audio frames are obtained, the speech end point category of the audio data may be determined based on the integrated features of the multiple audio frames.

[0020] In relation to the first aspect, in some implementations of the first aspect, the speech end point classification model is used to process the meaning and the phonetic category of the second audio frame to obtain the speech end point category, and the speech end point classification model is obtained using a speech sample and an end point category label of the speech sample, wherein the format of the speech sample corresponds to the meaning and the format of the second audio frame, and the end point category included in the end point category label corresponds to the speech end point category.

[0021] Furthermore, when the integrated features of the multiple audio frames are obtained, the integrated features may be processed using a speech end point classification model to obtain speech end point categories. The speech end point classification model is obtained using speech samples and end point category labels of the speech samples, where a format of the speech samples corresponds to a format of the integrated features, and the end point categories included in the end point category labels correspond to the speech end point categories.

[0022] According to a second aspect, there is provided a speech recognition method comprising the steps of: obtaining first audio data; determining a first speech end point of the first audio data; after obtaining the first speech end point, responding to a command corresponding to audio data in the first audio data prior to the first speech end point; obtaining second audio data; determining a second speech end point of the second audio data; after obtaining the second speech end point, responding to an instruction corresponding to audio data between the first speech end point in the first audio data and the second speech end point in the second audio data; A method is provided that includes:

[0023] According to the solution of the present embodiment of the present application, the speech end point can be obtained more accurately, and excessively long response delays caused by delays in detecting the speech end point can be avoided, so that the speech end point can be obtained more quickly, and subsequent responses can be made in a timely manner, thereby shortening the user's waiting time and improving the user experience. Specifically, the solution of the present embodiment of the present application can obtain audio data in real time, identify the speech end points of different commands in real time by identifying the speech end points in the audio data, and respond to each command after obtaining the speech end points of the commands. In particular, when the intervals between multiple commands sent by a user are relatively short, the solution of the present application can help identify the speech end point of each command after the command is sent, and respond to each command in a timely manner, instead of responding to all commands after all the multiple commands are sent.

[0024] In relation to the second aspect, in some embodiments of the second aspect, the step of determining a first speech end point of the first audio data includes a plurality of audio frames comprising the first audio data, the step of: extracting meaning and phonetic categories of the plurality of audio frames; obtaining the first speech end point of the first audio data based on the meaning and the phonetic category of the plurality of audio frames; Includes.

[0025] In relation to the second aspect, in some embodiments of the second aspect, the step of extracting meaning and phonetic categories of the plurality of audio frames comprises: Obtaining the speech category of the plurality of audio frames based on a relationship between the energy of the plurality of audio frames and a preset energy threshold.

[0026] In some embodiments of the second aspect, the audio categories include "speech", "neutral" and "silence", the preset energy thresholds include a first energy threshold and a second energy threshold, and the first energy threshold is greater than the second energy threshold. The audio category of the audio frames having an energy equal to or greater than the first energy threshold among the plurality of audio frames is "speech", The audio category of the audio frames having an energy less than the first energy threshold and greater than the second energy threshold is “neutral”; or The audio category of the audio frames among the plurality of audio frames whose energy is less than or equal to the second energy threshold is "silence."

[0027] In relation to the second aspect, in some implementations of the second aspect, the first energy threshold and the second energy threshold are determined based on an energy of a background sound of the first audio data.

[0028] Referring to the second aspect, in some embodiments of the second aspect, the plurality of audio frames include a first audio frame and a second audio frame, the first audio frame being an audio frame having the meaning, and the second audio frame being an audio frame following the first audio frame in the plurality of audio frames; obtaining a first speech end point of the first audio data based on the meaning and the sound category, Obtaining the first speech end point based on the meaning and a phonetic category of the second audio frame.

[0029] In relation to the second aspect, in some embodiments of the second aspect, speech end point categories include "speak", "think" and "end", and obtaining the first speech end point based on the meaning and the phonetic category of the second audio frame includes: The step includes determining a speech end point category of the first audio data based on the meaning and the voice category of the second audio frame, and obtaining the first speech end point if the speech end point category of the first audio data is "end".

[0030] In relation to the second aspect, in some embodiments of the second aspect, determining a speech end point category of the first audio data based on the meaning and the phonetic category of the second audio frame comprises: The speech end point classification model includes a step of processing the meaning and the phonetic category of the second audio frame using the speech end point classification model to obtain the speech end point category, the speech end point classification model being obtained using a speech sample and an end point category label of the speech sample, a format of the speech sample corresponding to a format of the meaning and the phonetic category of the second audio frame, and an end point category included in the end point category label corresponding to the speech end point category.

[0031] According to a third aspect, there is provided a speech end point classification model training method, the training method comprising: obtaining training data, the training data including speech samples and end point category labels of the speech samples, a format of the speech samples corresponding to a semantic format of a plurality of audio frames of audio data and a speech category of a second audio frame, the plurality of audio frames including a first audio frame and a second audio frame, the first audio frame being a semantic audio frame and the second audio frame being an audio frame following the first audio frame, and an end point category included in the end point category label corresponds to a speech end point category; training a speech end point classification model using the training data to obtain a target speech end point classification model; Includes.

[0032] The target speech end point classification model obtained using the method of the third aspect can be used to perform the operation of "processing the meaning and phonetic category of the second audio frame using the speech end point classification model to obtain a speech end point category" of the first aspect.

[0033] In relation to the third aspect, in some embodiments of the third aspect, the speech sample may be in the format of "start mark + meaning + phonetic category + end mark", or the speech sample may be in the format of "start mark + phonetic category + meaning + end mark".

[0034] Optionally, some text corpora are obtained, and dictionary trees of these corpora are established, and each node of the dictionary tree (each node corresponds to one word) includes the following information: whether the node is an end point or not, and the prefix frequency. Then, speech samples may be generated based on the node information. An end point is the end point of one sentence. The prefix frequency is used to represent the amount of words between a word and an end point. A higher prefix frequency indicates that the word is less likely to be an end point. For example, other words such as verbs and prepositions, such as "give" and "take", are relatively less likely to be used as end points of sentences, and have a relatively high prefix frequency in the dictionary tree. On the other hand, words such as "right?" and "correct?" are relatively more likely to be used as end points of sentences, and usually have a relatively low prefix frequency in the dictionary tree.

[0035] According to a fourth aspect, there is provided a speech recognition apparatus, said apparatus comprising: a unit configured to perform the method of any implementation of the first aspect.

[0036] According to a fifth aspect, there is provided a speech recognition apparatus, said apparatus comprising: a unit configured to perform the method of any implementation of the second aspect.

[0037] According to a sixth aspect, there is provided a speech end point classification model training apparatus, said training apparatus comprising a unit configured to perform the method of any implementation of the third aspect.

[0038] According to a seventh aspect, a speech recognition device is provided. The device includes a memory configured to store a program and a processor configured to execute the program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute the method of any implementation of the first or second aspect. The device can be located in devices or systems that need to determine speech end points, such as various speech recognition devices, voice assistants, or smart speakers. For example, the device can be various terminal devices such as mobile phone terminals, in-vehicle terminals, wearable devices, etc., or various devices with computing capabilities such as computers, hosts, servers, etc. Alternatively, the device can be a chip.

[0039] According to an eighth aspect, there is provided a speech end point classification model training device, the training device comprising a memory configured to store a program and a processor configured to execute the program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute the method of any implementation of the third aspect. The training device may be any device having computing capabilities, such as a computer, a host, or a server. Alternatively, the training device may be a chip.

[0040] According to a ninth aspect, there is provided a computer readable medium storing program code for execution by an apparatus to perform a method of any implementation of the first, second or third aspect.

[0041] According to a tenth aspect, there is provided a computer program product comprising instructions which, when executed on a computer, enable the computer to perform a method of any implementation of the first, second or third aspect.

[0042] According to an eleventh aspect, there is provided an in-vehicle system, the system comprising an apparatus according to any implementation of the fourth, fifth or sixth aspects.

[0043] For example, the in-vehicle system may include a cloud service device and a terminal device, and the terminal device may be any of a vehicle, an in-vehicle chip, an in-vehicle device (such as an in-vehicle infotainment device or an in-vehicle computer), etc.

[0044] According to a twelfth aspect, there is provided an electronic device, the electronic device comprising a device according to any implementation of the fourth, fifth or sixth aspects.

[0045] For example, the electronic device may specifically include one or more devices such as a computer, a smartphone, a tablet computer, a personal digital assistant (PDA), a wearable device, a smart speaker, a television, an unmanned aerial vehicle, a vehicle, an in-vehicle chip, an in-vehicle device (e.g., in-vehicle infotainment or in-vehicle computer), a robot, etc.

[0046] In the present application, the speech end point of the audio data is obtained by extracting and combining the meaning and the audio category of the audio data, so that the speech end point of the audio data can be determined more accurately, so that the subsequent voice-based operation can be responded to more accurately and the user experience can be improved. Specifically, the solution of the present application can avoid excessively long response delays due to delays in speech end point detection, so that the speech end point can be obtained more quickly, so that the subsequent response can be made timely, so that the user's waiting time can be shortened and the user experience can be improved. In addition, the solution of the present application can obtain an accurate speech end point, so that the user's voice command is not cut off due to early detection of the speech end point, and audio data with complete meaning can be obtained, so that the user's intention can be accurately grasped, an accurate response can be made, and the user experience can be improved. Obtaining an energy threshold based on the energy of the background sound can meet the requirements in different environments, so that the accuracy of determining the speech end point can be further improved. First, integrating the audio category and the meaning, and then obtaining the speech end point based on the integrated features can improve the processing efficiency and further improve the accuracy of determining the speech end point. [Brief description of the drawings]

[0047] [Figure 1] 1 is a schematic diagram of the structure of a speech recognition device according to an embodiment of the present application;

[0048] [Diagram 2] FIG. 2 is a schematic diagram of classification into audio categories according to an embodiment of the present application;

[0049] [Diagram 3] FIG. 2 is a schematic diagram of processing of integrated features by a speech end point classification model according to an embodiment of the present application;

[0050] [Figure 4] 1 is a schematic flowchart of a speech recognition method according to an embodiment of the present application;

[0051] [Diagram 5] 1 is a schematic flowchart of a speech recognition method according to an embodiment of the present application;

[0052] [Figure 6] 1 is a schematic flowchart of a speech end point classification model training method according to an embodiment of the present application;

[0053] [Figure 7] FIG. 2 is a schematic diagram of a dictionary tree according to an embodiment of the present application;

[0054] [Figure 8] 1 is a schematic block diagram of a speech recognition device according to an embodiment of the present application;

[0055] [Figure 9] 1 is a schematic diagram of a hardware structure of a speech recognition device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0056] With reference to the accompanying drawings, the technical solutions of the embodiments of the present application are described below.

[0057] The solution of the present application can be applied to various voice interaction scenarios. For example, the solution of the present application can be applied to voice interaction scenarios of electronic devices and voice interaction scenarios of electronic systems. The electronic devices can specifically include one or more devices such as computers, smartphones, tablet computers, personal digital assistants (PDAs), wearable devices, smart speakers, televisions, unmanned aerial vehicles, vehicles, on-board chips, on-board equipment (e.g., on-board infotainment and on-board computers), robots, etc. The electronic systems can include a cloud service device and a terminal device. For example, the electronic system may be an on-board system or a smart home system. The terminal side device of the on-board system may include any of the devices such as vehicles, on-board chips, and on-board equipment (e.g., on-board infotainment and on-board computers). The cloud service device includes a physical server and a virtual server. The server receives data uploaded from the terminal side (e.g., on-board infotainment), processes the data, and then transmits the processed data to the terminal side.

[0058] Next, we briefly describe two relatively common application scenarios.

[0059] Application scenario 1: Smartphone voice interaction

[0060] In a smartphone, a voice assistant can be used to realize a voice interaction. For example, a smartphone can be operated by a voice interaction with a voice assistant, or a user can have a conversation with the voice assistant. Specifically, the voice assistant can obtain audio data using a microphone, then determine a speech end point of the audio data using a processing unit, and trigger a subsequent response after obtaining the speech end point of the audio data. For example, the voice assistant reports the user's intention in the audio data to an operating system for a response.

[0061] Voice interaction can realize functions such as making calls, sending information, getting routes, playing music, and responding to conversations, greatly improving the technical feel and convenience of interaction with smartphones.

[0062] The present solution allows for accurate identification of speech end points in audio data, thereby improving the accuracy and timeliness of subsequent responses and enhancing the user experience.

[0063] Application scenario 2: Voice interaction in in-vehicle systems

[0064] In the in-vehicle system, a vehicle can be controlled by voice interaction. Specifically, in the in-vehicle system, a microphone is used to obtain audio data, and then a processing unit is used to determine a speech end point of the audio data, and a subsequent response is triggered after obtaining the speech end point of the audio data. For example, the user's intention in the audio data is reported to the in-vehicle system for response.

[0065] Voice interaction can enable functions such as getting routes, playing music, and controlling in-vehicle hardware (e.g., windows and air conditioning), improving the interaction experience of in-vehicle systems.

[0066] The present solution allows for accurate identification of speech end points in audio data, thereby improving the accuracy and timeliness of subsequent responses and enhancing the user experience.

[0067] FIG. 1 is a schematic diagram of the structure of a speech recognition device according to an embodiment of the present application. As shown in FIG. 1, the speech recognition device 100 is configured to process audio data to obtain a speech end point of the audio data, i.e., a speech stop point in the audio data. For example, the input audio data includes a speech segment of "I want to make a call". In this case, the speech end point of the segment of the audio data can be obtained through processing by the speech recognition device 100. Here, the speech end point may be the last audio frame corresponding to the last word in the speech segment. That is, the speech end point of the segment of the audio data is the last audio frame corresponding to the word "call".

[0068] Note that the end point of audio data and the speech end point of audio data are different concepts. The end point of audio data means the end of audio. For example, the end point of 5 seconds of audio data is the last audio frame of an audio segment. The speech end point of audio data is the end point of speech in an audio data segment. Again, take 5 seconds of audio data as an example. Assume that the 14th second of audio contains speech and the 5th second has no speech. In this case, the speech end point of the audio data is the audio frame corresponding to the end of the 4th second. Also assume that the 5 seconds of audio data does not contain speech and the preset time interval for speech recognition is 3 seconds. In this case, if no speech is recognized for 3 consecutive seconds, speech recognition is terminated and the speech end point of the 5 seconds of audio data is the audio frame corresponding to the end of the 3rd second.

[0069] The audio data may include multiple audio frames. Note that the input audio data may or may not include speech. For example, suppose a speech capture function is activated but no speech is spoken for several seconds. In this case, the captured audio data is audio data that does not include speech.

[0070] The multiple audio frames in the audio data may be consecutive audio frames or may be discontinuous audio frames.

[0071] The speech recognition device 100 includes an acquisition module 110, a processing module 120, and a decision module 130. Alternatively, the decision module 130 may be integrated with the processing module 120.

[0072] The acquisition module 110 is configured to acquire audio data, where the audio data may include a plurality of audio frames. The acquisition module 110 may include a speech capture device, such as a microphone, configured to acquire speech audio in real time. Alternatively, the acquisition module 110 may include a communication interface. A transceiver device, such as a walkie-talkie, may be used for the communication interface to communicate with and acquire audio data from other devices or communication networks.

[0073] The processing module 120 processes the audio frames in the audio data to obtain meaning and phonetic categories of the audio frames, which may be understood as follows: The processing module 120 is configured to extract phonetic categories and meanings of the audio frames in the audio data.

[0074] Semantics is used to describe the language contained in the audio data and may also be referred to as text semantics, word semantics, linguistic semantics, etc. Semantics may also be conveyed by the audio stream.

[0075] For example, the phonetic category of a plurality of audio frames may include a phonetic category of each of the plurality of audio frames.

[0076] Alternatively, the phonetic category of the plurality of audio frames may include the phonetic category of some of the plurality of audio frames.

[0077] In other words, the processing module 120 may extract the phonetic categories for all of the audio frames, or may extract the phonetic categories for only a portion of the audio frames.

[0078] Optionally, an automatic speech recognition (ASR) device or other device may be used to obtain an audio stream that conveys meaning. Each segment of the audio stream may be represented by a corresponding text. Each segment of the audio stream may include one or more audio frames.

[0079] The audio categories can include "speech (SPE)", "neutral (NEU)", and "silence (SIL)". "Speech" is a portion of audio that is definitely a human speaking (or can be understood as a human voice portion of the audio). "Neutral" is a relatively fuzzy portion of audio that cannot be clearly determined as speech (or can be understood as a fuzzy portion of the audio). "Silence" is a portion of audio that does not clearly contain human voice (or can be understood as a portion of audio without human voice). It should be understood that in this embodiment of the present application, "silence" does not mean a decibel value of 0 in the physical sense, or the absence of any voice at all, but may mean no speech voice, no voice, only background sound, etc.

[0080] It should be understood that there are other possible classification methods for the audio category. For example, the audio category may include only "speech" and "silence", or may include "silence" and "non-silence", or may include "speech" and "non-speech". "Speech" and "non-speech" may also be called "human voice" and "non-human voice", respectively. In addition, the above-mentioned "speech", "neutral", and "silence" may also be called "human voice", "possible human voice", and "not human voice", respectively. It should be understood that this is merely an example, and the classification method for the audio category is not limited in this embodiment of the present application.

[0081] It should be noted that the phonetic category corresponds to determining and classifying the audio data from an acoustic point of view and is used to distinguish the audio frame category. The meaning is obtained by extracting linguistic components from the audio data and is used to infer whether the speech is complete from a linguistic point of view. It should be understood that a phonetic category may be extracted from each audio frame. However, since an audio frame that does not contain a human voice has no meaning, the meaning of an audio frame that does not contain speech can be considered as empty or null.

[0082] In general, the energies of audio frames of different sound categories are different, for example, an audio frame of "speech" has a relatively high energy, an audio frame of "silence" has a relatively low energy, and an audio frame of "neutral" has a lower energy than "speech" and a higher energy than "silence".

[0083] The energy of an audio frame is also called the intensity of the audio frame.

[0084] Optionally, the audio frames can be classified based on the energy of the audio frames to obtain the speech category of the audio frames, specifically, based on the energy of the audio frames in the audio data, the speech category of the corresponding audio frames is obtained.

[0085] In one implementation, a speech category of a corresponding audio frame can be obtained based on a relationship between the energy of the audio frame and a preset energy threshold. For example, the preset energy threshold includes a first energy threshold and a second energy threshold, and the first energy threshold is greater than the second energy threshold. The audio frame whose energy is equal to or greater than the first energy threshold is determined as "speech". The audio frame whose energy is less than the first energy threshold and greater than the second energy threshold is determined as "neutral". The audio frame whose energy is equal to or less than the second energy threshold is determined as "silence". In other words, the speech category of the audio frame whose energy is equal to or greater than the first energy threshold among the plurality of audio frames is determined as "speech", the speech category of the audio frame whose energy is less than the first energy threshold and greater than the second energy threshold among the plurality of audio frames is determined as "neutral", and the speech category of the audio frame whose energy is equal to or less than the second energy threshold among the plurality of audio frames is determined as "silence". However, alternatively, audio whose energy is equal to the first energy threshold or equal to the second energy threshold may be determined to be “neutral.” The above description is used as an example in this embodiment of the present application.

[0086] FIG. 2 is a schematic diagram of classification into audio categories according to an embodiment of the present application. As shown in FIG. 2, the abscissa represents an audio frame sequence, and the ordinate represents an energy value corresponding to the audio frame sequence. The energy curves represent energy change curves of a plurality of audio frames in the audio data. The first energy threshold curve represents a lower energy limit curve of "speech" and an upper energy limit curve of "neutral". The second energy threshold curve represents a lower energy limit curve of "neutral" and an upper energy limit curve of "silence". The silence energy curve represents an energy curve of background sound of a segment of audio. Both the first energy threshold curve and the second energy threshold curve can be obtained based on the silence energy curve. That is, both the first energy threshold curve and the second energy threshold curve can be obtained based on the energy of the background sound.

[0087] It should be noted that in different background environments, the silence energy curve is different. For example, in a relatively quiet environment, the silence energy (i.e., the energy of background sound) is relatively low, and in a relatively noisy environment, the silence energy (i.e., the energy of background sound) is relatively high. Therefore, obtaining the first energy threshold curve and the second energy threshold curve based on the silence energy curve can adapt to the requirements in different environments.

[0088] As shown in Fig. 2, by comparing the energy of audio frames in audio data with two thresholds, audio frames are classified into three speech categories: "Speech" (SPE in the figure), "Neutral" (NEU in the figure), and "Silence" (SIL in the figure). As shown in Fig. 2, the speech category sequence of the audio frame sequence is "SPE, NEU, SIL, SIL, SPE, NEU, SIL, SPE, NEU, SPE, NEU, SPE, NEU, SIL" from left to right.

[0089] The processing module 120 may be a processor capable of performing data processing, such as a central processing unit or microprocessor, or may be any other device, chip, integrated circuit, etc. capable of performing computations.

[0090] The decision module 130 is configured to obtain a speech end point of the audio data based on the audio category and the meaning from the processing module 120. The speech end point is a detection result of an end-of-speech condition.

[0091] Optionally, if the acquiring module 110 acquires audio data in real time, it may terminate acquiring the audio data after acquiring the detection result.

[0092] For example, the decision module 130 may obtain the speech end point based on all the voice categories and the meanings of the multiple audio frames.

[0093] For example, potential text end points may be pre-determined based on meaning, and then speech end points may be obtained from the potential text end points based on the phonetic category of each text end point. In another example, potential candidate audio frames may be pre-determined based on phonetic category, and then speech end points may be determined based on meaning preceding these candidate audio frames.

[0094] For example, the plurality of audio frames includes a first audio frame and a second audio frame, where the first audio frame is an audio frame having a meaning, and the second audio frame is an audio frame following the first audio frame among the plurality of audio frames, and the determination module 130 can obtain the speech end point based on the meaning and the phonetic category of the second audio frame.

[0095] The first audio frames are multiple audio frames that have meaning, and the second audio frames are one or more audio frames that follow the first audio frames.

[0096] Note that the "plurality of meaningful audio frames" included in the audio data are different concepts from the "plurality of audio frames." The number of audio frames included in the first audio frame is smaller than the number of audio frames included in the audio data.

[0097] The determination module 130 may also integrate the meaning and the phonetic category of the second audio frame to obtain an integrated feature, and obtain a speech end point based on the integrated feature. The integrated feature can be understood to be obtained by superimposing one or more subsequent audio frames whose phonetic categories have been determined on an audio stream having a meaning. For example, a segment of audio data includes "I want to watch television" and five audio frames following "television". By extracting the phonetic category and the meaning, the meaning "I want to watch television" and the phonetic category of the five subsequent audio frames can be obtained. In this case, the integrated feature is obtained by superimposing five audio frames having the phonetic category on the multiple audio frames having the meaning "I want to watch television". The processing using the integrated feature can improve the processing efficiency and also improve the accuracy compared to the above direct processing.

[0098] Optionally, the speech end point category may include "speak", "think", and "end". The determination module 130 may determine a speech end point category of the audio data based on the meaning and the phonetic category of the second audio frame, and obtain the speech end point if the speech end point category of the audio data is "end".

[0099] The speech end point category of the audio data being "end" can be understood as the audio data including a text end point with the speech end point category being "end". An audio frame corresponding to a text end point with the speech end point category being "end" may be used as a speech end point.

[0100] In addition, when the integrated feature is obtained, the determination module 130 determines a speech end point category of the audio data based on the integrated feature, and obtains the speech end point if the speech end point category of the audio data is "end".

[0101] In some implementations, the decision module 130 may further process the meaning and phonetic category of the second audio frame using a speech end point classification model to obtain a speech end point category, thereby obtaining a speech end point of the audio data.

[0102] Also, if the integrated features are obtained, the decision module 130 may process the integrated features using a speech end point classification model to obtain a speech end point category, and thereby obtain a speech end point of the audio data.

[0103] That is, the integrated feature is input as one input feature into a speech end point classification model and processed to obtain a speech end point category, thereby obtaining the speech end point of the audio data.

[0104] It should be understood that this is just one example, for example, instead of processing the meaning and the phonetic category of the second audio frame, the decision module 130 may directly input and process the meaning and the phonetic category of the second audio frame as two input features into the speech end point classification model.

[0105] It should be noted that the speech recognition device 100 is realized in the form of a functional module, and the term module herein may be realized in the form of software and / or hardware. This is not limited in this embodiment of the present application. The division into modules described above is merely a logical division of functions, and may be a different division method in actual implementation. For example, multiple modules may be integrated into one module. That is, the acquisition module 110, the processing module 120, and the decision module 130 may be integrated into one module. Also, the multiple modules may exist independently of each other. Alternatively, two of the multiple modules may be integrated into one module. For example, the decision module 130 may be integrated with the processing module 120.

[0106] The modules may be located on the same hardware or on different hardware, i.e., the functions that the modules need to perform may be performed by the same hardware or by different hardware, which is not limited in this embodiment of the present application.

[0107] Speech endpoint categories can include "speaking," "thinking," and "ending." "Speaking" can be understood as ongoing speech, i.e., such an endpoint is neither an endpoint of an end nor an endpoint of a stop. "Thinking" can be understood as considering or pausing, i.e., such an endpoint is merely a pause endpoint, possibly followed by speech. "Ending" can be understood as a stop or an end, i.e., such an endpoint is an endpoint of an end of speech.

[0108] In some implementations, the speech end point classification model can be obtained by using a language class model, for example, by using a bidirectional encoder representations from transformers (BERT) model. In the following, the BERT model is used as an illustrative example, but it should be understood that any other language class model capable of performing the aforementioned classification can be used instead.

[0109] Fig. 3 is a schematic diagram of processing the integrated features by a classification model according to an embodiment of the present application. As shown in Fig. 3, the classification model may be a BERT model. The BERT model includes an embedding layer and a fully connected layer (depicted by a white box C in Fig. 3). The embedding layer includes a token embedding layer, a segment embedding layer, and a position embedding layer. The embedding layer outputs the result obtained by processing the input data (i.e., the integrated features) to the fully connected layer. The fully connected layer further obtains the speech end point category.

[0110] As shown in Figure 3, the input data provides five example integrated features, represented as In1 to In5 in Figure 3. In1 is "[CLS] open the car window [SPE] [SIL] [SEP]". In2 is "[CLS] open the car window [SIL] [SPE] [SIL] [SIL] [SEP]". In3 is "[CLS] please adjust the temperature to twenty degrees [SIL] [NEU] [SIL] [SIL] [SEP]". In4 is "[CLS] please adjust the temperature to twenty six [SIL] [SIL] [SIL] [SEP]". In5 is "[CLS] please adjust the temperature to twenty six [SIL] [SIL] [SEP]". Each circle represents one element. [CLS] is the start data of the input data of the BERT model, and is sometimes called the start symbol. [SEP] is the end of the input data for the BERT model, and is sometimes called the terminator. That is, in this example, the format of the integrated features is “[CLS] + semantic + phonetic category + [SEP]”.

[0111] As shown in Figure 3, EX represents the energy value of X, for example, ECLS represents the energy corresponding to [CLS]. The speech end point category can be obtained through the input data processing performed by the embedding layer and fully connected layer of the BERT model.

[0112] 3 is merely a specific example of the speech end point classification model, and is not limited thereto. The speech end point classification model training method will be described later, and therefore will not be described here.

[0113] FIG. 4 is a schematic flow chart of a speech recognition method according to an embodiment of the present application. Each step of FIG. 4 will be described below. The method shown in FIG. 4 may be performed by an electronic device. The electronic device may specifically include one or more devices such as a computer, a smartphone, a tablet computer, a personal digital assistant (PDA), a wearable device, a smart speaker, a television, an unmanned aerial vehicle, a vehicle, an in-vehicle chip, an in-vehicle device (e.g., an in-vehicle infotainment or an in-vehicle computer), a robot, etc. Alternatively, the method 400 shown in FIG. 4 may be performed by a cloud service device. Alternatively, the method 400 shown in FIG. 4 may be implemented by a system including a cloud service device and a terminal device, such as an in-vehicle system or a smart home system.

[0114] For example, the method illustrated in FIG. 4 may be implemented by the speech recognition device 100 of FIG.

[0115] 401: Obtain audio data. The audio data includes a number of audio frames.

[0116] The audio data may be acquired using a speech capture device such as a microphone, or the audio data may be acquired from a storage device or a network. The audio data may be acquired in real time or may be already stored. Step 401 may be performed using the above-mentioned acquisition module 110. For a related description of the audio data and the method of acquiring the audio data, please refer to the above description. The details will not be described again here.

[0117] The audio frames may be obtained by performing a framing operation on the audio data. For example, the duration of one audio frame may be a dozen milliseconds or several tens of milliseconds.

[0118] 402: Extract phonetic categories and meanings of multiple audio frames.

[0119] Step 402 may be performed using the processing module 120 mentioned above. See above for a description of the speech categories and meanings. The details will not be described again here. It should be noted that although a speech category may be extracted from each audio frame, a meaning may not be extracted from each audio frame. Specifically, audio frames that do not contain human speech have no meaning. Thus, no meaning can be extracted from audio frames that do not contain speech. In this case, the meaning of the audio frames that do not contain speech can be considered as empty or null.

[0120] Optionally, the speech category may be obtained based on a relationship between the energy of the audio frame and a preset energy threshold. For example, the preset energy threshold includes a first energy threshold and a second energy threshold, and the first energy threshold is greater than the second energy threshold. The speech category of the audio frames among the plurality of audio frames whose energy is equal to or greater than the first energy threshold may be determined as "speech". The speech category of the audio frames among the plurality of audio frames whose energy is less than the first energy threshold and greater than the second energy threshold may be determined as "neutral", where the first energy threshold is greater than the second energy threshold. The speech category of the audio frames among the plurality of audio frames whose energy is equal to or less than the second energy threshold may be determined as "silence".

[0121] Optionally, an automatic speech recognition device or other device may be used to obtain an audio stream that conveys meaning. Each segment of the audio stream may be represented by a corresponding text. Each segment of the audio stream may include one or more audio frames.

[0122] For example, the phonetic category of a plurality of audio frames may include a phonetic category of each of the plurality of audio frames.

[0123] Alternatively, the phonetic category of the plurality of audio frames may include the phonetic category of some of the plurality of audio frames.

[0124] In other words, in step 402, the phonetic categories of all of the plurality of audio frames may be extracted, or the phonetic categories of only a portion of the plurality of audio frames may be extracted.

[0125] It should be noted that a speech category may be extracted from each audio frame, i.e., a speech category corresponding to each audio frame may be obtained. Usually, multiple audio frames correspond to one word, i.e., multiple audio frames carry one word. Figure 3 is used as an example. Each word of "open the car window" in Figure 3 corresponds to multiple audio frames, and each speech category corresponds to one audio frame.

[0126] 403: Obtain speech end points of the audio data based on the audio category and meaning.

[0127] For example, step 403 may be performed using the decision module 130 described above.

[0128] Optionally, the method 400 further includes step 404 (not shown).

[0129] 404: After obtaining the speech end point, respond to the command corresponding to the audio data before the speech end point of the audio data.

[0130] That is, an operation corresponding to the audio data before the speech end point in the audio data may be executed after obtaining the speech end point. Note that an operation corresponding to the audio signal before the speech end point in the audio data may also be understood as an operation corresponding to the speech end.

[0131] In addition, the operation corresponding to the audio data before the speech end point in the audio data may be executed immediately after the speech end point is acquired, or the operation corresponding to the audio data before the speech end point in the audio data may be executed a certain time period after the speech end point is acquired. This time period may be a redundant time, an error time, etc.

[0132] The action corresponding to the end of speech may be an action in a service processing function.

[0133] For example, the speech recognition may be stopped after obtaining the speech end point. Alternatively, the speech recognition result may be returned to the user after obtaining the speech end point. Alternatively, the speech recognition result may be sent to a subsequent module, so that the subsequent module performs a corresponding operation after obtaining the speech end point. For example, the audio data may include a control instruction, so that the subsequent module performs a control operation corresponding to the instruction. For example, the audio data may include a query instruction, so that the subsequent module returns an answer sentence corresponding to the query instruction to the user.

[0134] The operation corresponding to the end of speech, i.e., step 404, may be performed by the device executing the method 400 or by other devices, which is not limited to this embodiment of the present application.

[0135] In the following, an example will be described in which an audio signal includes a user command. The user command is used to realize various functions such as route acquisition, music playback, and hardware (e.g., lighting and air conditioner) control. Air conditioner control is used as an example. In this case, the user command may be "turn on the air conditioner." Note that the subsequent module may be one module or multiple modules.

[0136] For example, after obtaining the speech end point, instruction information indicating the speech end point is sent to a subsequent module, and the subsequent module obtains audio data before the speech end point in the audio data, obtains meaning text (e.g., "turn on the air conditioner") based on the audio data, and then analyzes the user instruction based on the meaning text, and controls a corresponding module to perform the operation indicated by the voice instruction.

[0137] For example, after obtaining the speech end point, the ASR can be instructed to stop speech recognition, and the ASR can send the speech recognition result (e.g., the semantic text of "turn on the air conditioner") to the semantic analysis module, so that the semantic analysis module can analyze the user command and control the air conditioner to turn on by sending a control signal to the air conditioner.

[0138] For example, after obtaining the speech end point, the audio data before the speech end point in the audio data can be sent to a subsequent module, so that the subsequent module can obtain a semantic text (e.g., "turn on the air conditioner") based on the audio data, and then parse the user command based on the semantic text, and control a corresponding module to perform an action indicated by the voice command.

[0139] For example, after obtaining the speech end point, it can obtain a meaning text (e.g., "turn on the air conditioner") based on the audio data, and then analyze the user command based on the meaning text, and control a corresponding module to perform the action indicated by the voice command.

[0140] For example, the speech end point may be obtained based on all the phonetic categories and the meanings of multiple audio frames.

[0141] For example, potential text end points may be pre-determined based on meaning, and then speech end points may be obtained from the potential text end points based on the phonetic category of each text end point. In another example, potential candidate audio frames may be pre-determined based on phonetic category, and then speech end points may be determined based on meaning preceding these candidate audio frames.

[0142] For example, the plurality of audio frames includes a first audio frame and a second audio frame, where the first audio frame is an audio frame having a meaning, and the second audio frame is an audio frame following the first audio frame among the plurality of audio frames, and the speech end point can be obtained based on the meaning and the phonetic category of the second audio frame.

[0143] The first audio frames are multiple audio frames that have meaning, and the second audio frames are one or more audio frames that follow the first audio frames.

[0144] Note that the "plurality of meaningful audio frames" included in the audio data are different concepts from the "plurality of audio frames." The number of audio frames included in the first audio frame is smaller than the number of audio frames included in the audio data.

[0145] Furthermore, the meaning and the phonetic category of the second audio frame can be integrated to obtain an integrated feature, and then the speech end point is obtained based on the integrated feature. By processing using the integrated feature, the processing efficiency can be improved and the accuracy can be improved.

[0146] In some implementations, the speech end point category may include "speak", "think", and "end". Then, the speech end point category of the audio data may be determined based on the meaning and the voice category of the second audio frame, and the speech end point may be obtained when the speech end point category of the audio data is "end".

[0147] The speech end point category of the audio data being "end" can be understood as the audio data including a text end point with the speech end point category being "end". An audio frame corresponding to a text end point with the speech end point category being "end" may be used as a speech end point.

[0148] In addition, when the integrated feature is obtained, the speech end point category of the audio data may be determined based on the integrated feature, and the speech end point may be obtained when the speech end point category of the audio data is "end".

[0149] Optionally, the speech end point classification model may be used to process the meaning and phonetic category of the second audio frame to obtain a speech end point category, thereby obtaining a speech end point of the audio data.

[0150] Also, once the integrated features are obtained, the integrated features may be processed using a speech end point classification model to obtain a speech end point category for the integrated features, thereby obtaining the speech end points of the audio data.

[0151] The speech end-point classification model may be obtained using the speech samples and the end-point category labels of the speech samples, where the format of the speech samples corresponds to the format of the integrated features, and the end-point categories included in the end-point category labels correspond to the speech end-point categories.

[0152] In the solution shown in FIG. 4, the speech end points of the audio data are obtained by extracting and combining the meaning and the phonetic category of the audio data, so that the speech end points of the audio data can be determined more accurately.

[0153] It should be noted that the audio data may be audio data acquired in real time, or stored audio data may be read. These two cases can be understood as online speech recognition and offline speech recognition, respectively. After acquiring the speech end point, the speech end point can be used to perform subsequent operations, such as speech-based control. For example, the speech end point can be used to control a switch of an electronic device, perform an information query, control audio / video playback, etc. In the case of online speech recognition, after acquiring the speech end point, the acquisition of audio data may be terminated, i.e., speech recognition may be stopped. Also, in the case of online speech recognition, after acquiring the speech end point, an instruction corresponding to the audio data before the speech end point may be executed, and the acquisition of audio data may be continued.

[0154] According to the solution of the present embodiment of the present application, the speech end point can be obtained more accurately, so that the speech-based subsequent action can be responded to more accurately, and the user experience can be improved. Specifically, according to the solution of the present application, the excessively long response delay due to the delay in the speech end point detection can be avoided, so that the speech end point can be obtained more quickly, so that the subsequent response can be made in a timely manner, so that the user's waiting time can be shortened and the user experience can be improved. In addition, according to the solution of the present application, the speech end point can be obtained accurately, so that the user's voice command is not cut off due to the early detection of the speech end point, and audio data with complete meaning can be obtained, so that the user's intention can be accurately understood, and an accurate response can be made, and the user experience can be improved.

[0155] For example, in a scenario in which an electronic device is controlled using voice, a user generates audio data for a total of 7 seconds. Here, the voice at 13 seconds is "please turn on the air conditioner", the voice at 4 seconds is a 1-second pause, and the voice at 5-7 seconds is a cough sound. Using the solution of this embodiment of the present application, the word "conditioner" can be accurately captured as the speech end point, and in this case, speech capture can be terminated after 3 seconds or at 4 seconds. In the conventional activity detection-based method, a certain waiting time needs to be set, and speech capture is not terminated until the actual waiting time is equal to or greater than the certain waiting time. Here, the activity detection-based method is used, and the certain waiting time is set to 2 seconds. In this case, the 1-second pause is regarded as a temporary pause, and speech recognition is continued. Therefore, it is necessary to continue recognizing the cough sound for the following 3 seconds, and wait for another certain waiting time after the cough sound. Compared to the solution of this embodiment of the present application, the activity detection-based method delays the end of speech capture by at least 5 seconds in total.

[0156] In a scenario of controlling an electronic device using voice, suppose a user generates audio data of "please call Qian Yi'er" with a total of 6 seconds. Here, the word "Qian" is followed by a pause of 1.5 seconds, and the fixed waiting time in the activity detection-based method is 1.5 seconds. Using the solution of this embodiment of the present application, the word "er" can be accurately obtained as the speech end point. According to the semantic information, the word "Qian" cannot be used as the "end" point unless it is followed by an audio frame with a speech category of "silence". Therefore, the speech acquisition is not terminated after the 1.5-second pause, but at or after the 6th second. However, when using the activity detection-based method, the speech acquisition is terminated when the 1.5-second pause after the word "Qian" ends. As a result, the speech end point is determined incorrectly, and the subsequent control policy cannot be accommodated.

[0157] When the solution of the present embodiment of the present application is used to perform speed recognition, the waiting time to the speech end point is not constant, but changes with the actual speech recognition process. Compared with the traditional method of presetting a certain waiting time and obtaining the speech end point after the waiting time is over, the solution can obtain the speech end point more accurately, thereby improving the timeliness and accuracy of subsequent responses, shortening the user's waiting time, and improving the user experience.

[0158] The embodiment of the present application further provides a speech recognition method 500. The method 500 includes steps 501 to 506. The following describes each step of the method 500. The method 500 can be understood as an example of an online speech recognition method.

[0159] 501: First audio data is acquired.

[0160] The first audio data may be acquired in real time and may include a plurality of audio frames.

[0161] 502: Determine a first speech end point of the first audio data.

[0162] Optionally, the first audio data includes a plurality of audio frames. Step 502 may include: extracting a meaning and a phonetic category of the plurality of audio frames; and obtaining a first speech end point of the first audio data based on the meaning and the phonetic category of the plurality of audio frames.

[0163] For a specific method of determining the first speech end point, please refer to the above method 400. Just replace "audio data" in the method 400 with "first audio data". The details will not be described again here.

[0164] 503: After obtaining the first speech end point, respond to a command corresponding to the audio data before the first speech end point of the first audio data.

[0165] That is, an operation corresponding to the audio data before the first speech end point may be performed after obtaining the first speech end point.

[0166] For ease of explanation, the command corresponding to the audio data before the first speech end point in the first audio data is referred to as the first user command.

[0167] The first speech end point is a speech end point of a first user instruction. The first audio data may include only one user instruction, and by performing step 502, the speech end point of the instruction can be identified.

[0168] 504: Second audio data is obtained.

[0169] It should be noted that the sequence numbers of each step in the method 500 are used only for ease of explanation and do not limit the order of execution of each step. In the method 500, the process of processing audio data and the process of acquiring audio data may be independent. That is, while step 502 is being performed, if the second audio data is the audio data acquired after the first audio data, the audio data acquisition may be continued. That is, step 504 may be performed.

[0170] For example, the first audio data and the second audio data may be consecutively captured audio data.

[0171] 505: A second speech end point of the second audio data is determined.

[0172] Optionally, the second audio data includes a plurality of audio frames. Step 502 may include: extracting a meaning and a phonetic category of the plurality of audio frames; and obtaining a first speech end point of the first audio data based on the meaning and the phonetic category of the plurality of audio frames.

[0173] Alternatively, the audio data including the second audio data and the audio data following the first speech end point and included in the first audio data may include a plurality of audio frames, and step 502 may include extracting a phonetic category and a meaning of the plurality of audio frames, and obtaining a second speech end point of the second audio data based on the phonetic category and the meaning of the plurality of audio frames.

[0174] For example, the audio data included in the first audio data and following the first speech end point includes 4 audio frames, and the second audio data includes 10 audio frames, in this case, the audio data including 4 audio frames and 10 audio frames includes 14 audio frames, extracting the sound category and meaning of the 14 audio frames, and obtaining the second speech end point of the second audio data based on the sound category and meaning of the 14 audio frames.

[0175] For a specific method of determining the second speech end point, please refer to the above method 400, and the details will not be described again here.

[0176] 506: After obtaining the second speech end point, respond to the instruction corresponding to the audio data between the first speech end point in the first audio data and the second speech end point in the second audio data.

[0177] For example, the audio data included in the first audio data and following the first speech end point includes 4 audio frames, and the second audio data includes 10 audio frames. In this case, the audio data including 4 audio frames and 10 audio frames includes 14 audio frames, and the second speech end point of the second audio data is located at the 12th audio frame. After obtaining the second speech end point, respond to the command corresponding to the audio data before the 12th audio frame among the 14 audio frames.

[0178] For ease of explanation, the instruction corresponding to the audio data between the first speech end point of the first audio data and the second speech end point of the second audio data is referred to as the second user instruction.

[0179] The second speech end point is the speech end point of the second user instruction. The audio data including the second audio data and the audio data after the first speech end point of the first audio data may include only one user instruction, and the speech end point of the instruction can be identified by performing step 505. The first user instruction and the second user instruction may be two user instructions transmitted by a user in succession, that is, the time interval between the two user instructions is relatively small. For example, when the first audio data and the second audio data are audio data captured in succession, the time interval between the first user instruction and the second user instruction is relatively small. The solution of the present invention helps to distinguish the speech end point of the first user instruction from the speech end point of the second user instruction.

[0180] According to the solution of the present embodiment of the present application, the speech end point can be obtained more accurately, and excessively long response delays caused by delays in detecting the speech end point can be avoided, so that the speech end point can be obtained more quickly, and subsequent responses can be made in a timely manner, thereby shortening the user's waiting time and improving the user experience. Specifically, the solution of the present embodiment of the present application can obtain audio data in real time, identify the speech end points of different commands in real time by identifying the speech end points in the audio data, and respond to each command after obtaining the speech end points of the commands. In particular, when the intervals between multiple commands sent by a user are relatively short, the solution of the present application can help identify the speech end point of each command after the command is sent, and respond to each command in a timely manner, instead of responding to all commands after all the multiple commands are sent.

[0181] For example, in a scenario of controlling an electronic device using voice, a user generates a total of 8 seconds of audio data, and the 8 seconds of audio data includes two user commands, "please close the car window" and "please turn on the air conditioner". The interval between the two user commands is relatively short, e.g., 1 second, and the user sends the user command "please turn on the air conditioner" 1 second after sending the user command "please close the car window". Using the solution of this embodiment of the present invention, the speech end point corresponding to the first user command can be obtained by acquiring and processing the audio data in real time. Based on the semantic information, when the word "window" is followed by several audio frames with the audio category of "silence", the word "window" can be used as the end point of "end". That is, it can be determined that the speech corresponding to the user command has ended after several audio frames following the word "window", and the operation of closing the car window according to the user command can be performed in a timely manner. Also, according to the solution of the present embodiment of the present invention, the audio data may be continuously acquired and processed to obtain a speech end point corresponding to the second user command. For example, it may be determined that the speech corresponding to the user command ends after a number of audio frames following the word "conditioner", and an action of turning on the air conditioner in response to the user command may be performed in a timely manner. If the certain latency in the above-mentioned activity detection-based method is 1.5 seconds, the above-mentioned 1-second pause is regarded as a temporary pause and speech recognition continues. Therefore, the speech end point corresponding to the first user command cannot be obtained. In this case, it is only 1.5 seconds after the user transmits the second user command that the speech is deemed to end, and the actions corresponding to the two user commands are performed.

[0182] That is, the solution of this embodiment of the present application can accurately obtain the speech end points corresponding to the multiple user commands, and can respond to each user command in a timely manner. Especially, when the interval between the multiple user commands is relatively short, the solution of this embodiment of the present application can more accurately obtain the speech end points of each user command, and can respond in a timely manner after each user command is sent, instead of the user sending all the user commands and then responding.

[0183] Fig. 5 is a schematic flow chart of a speech recognition method according to an embodiment of the present application. Fig. 5 can be considered as a specific example of the method shown in Fig. 4. In this example, audio data acquisition and speech recognition are performed in real time.

[0184] 601: Acquires audio data in real time.

[0185] The audio data may be obtained using a speech capture device such as a microphone.

[0186] Alternatively, step 601 may be performed using the acquisition module 110 described above.

[0187] 602: Perform speech recognition on the audio data using ASR to obtain a meaningful audio stream.

[0188] Step 602 is an example of a method for obtaining meaning, that is, an example of obtaining meaning using ASR.

[0189] Optionally, steps 603 to 606 may not be performed and step 607 may be performed directly if the ASR identifies that the audio stream has ended, which corresponds to the case where no speech has been recognized in the captured audio data for a relatively long time interval, and it is no longer necessary to determine the speech end point.

[0190] Step 602 may be performed using the processing module 120 described above.

[0191] 603: Obtain a voice category of an audio frame in the audio data based on a relationship between the energy of the audio data and a preset energy threshold.

[0192] Step 603 is an example of a method for obtaining a sound category.

[0193] Step 602 and step 603 may be performed simultaneously or at different times, and the order of execution is not limited. Step 602 and step 603 can be considered as specific examples of step 402.

[0194] Step 603 may be performed using the processing module 120 described above.

[0195] 604: Integrate meaning and phonetic category to obtain integrated features.

[0196] The integrated features may be obtained by overlapping semantic and phonetic categories.

[0197] For example, step 602 may be performed in real time. That is, speech recognition may be performed on the obtained audio data in real time. In step 604, each time a word is recognized, the current meaning and phonetic category of one or more audio frames following the word are superimposed to obtain an integrated feature, and the integrated feature is input to the subsequent speech endpoint classification model for processing (i.e., step 605). For example, the complete command sent by the user is "I want to watch television", and the currently sent instruction is "I want to watch". In step 602, speech recognition is performed on the currently obtained audio data. After the word "watch" is recognized, an integrated feature can be obtained based on the meaning of "I want to watch" and the phonetic category of one or more audio frames following "watch". Then, in step 605, the integrated feature is input to the speech endpoint classification model for processing. After the user continues to send "television", speech recognition is performed on the currently obtained audio data via step 602. After the word "television" is recognized, the integrated features can be derived based on the meaning of "I want to watch television" and the phonetic categories of one or more audio frames following "television", and the integrated features are then input into a speech end-point classification model for processing in step 605.

[0198] The processing using the integrated features can improve the processing efficiency and accuracy compared to the above direct processing. For specific details, please refer to the explanation of the input data in Figure 3. The details will not be described again here.

[0199] Step 604 may be performed using the decision module 130 described above.

[0200] 605: Process the integrated features using a speech end-point classification model to obtain a speech end-point category.

[0201] The speech end point categories may include "speaking," "thinking," and "ending." If the speech end point category is "ending," then the audio frame corresponding to the ending point is the speech end point.

[0202] Steps 604 and 605 can be considered as specific examples of step 403.

[0203] Step 605 may be performed using the decision module 130 described above.

[0204] 606: Determine whether the speech end point category is “End”; if the result of the determination is “Yes”, execute step 607; if the result of the determination is “No”, execute step 601;

[0205] Step 606 may be performed using the decision module 130 described above.

[0206] 607: The speech end point is output.

[0207] Step 607 may be performed using the decision module 130 described above.

[0208] 6 is a schematic flow chart of a speech end point classification model training method according to an embodiment of the present application. Each step in FIG. 6 will be described below.

[0209] 701: Obtain training data. The training data includes speech samples and endpoint category labels of the speech samples.

[0210] Also, the format of the speech sample may correspond to the format of the data input to the speech end point classification model, and the end point categories included in the end point category labels may correspond to the speech end point categories, i.e., the speech sample includes the phonetic categories and meanings of the audio data, and the end point category labels include "speak", "think" and "end".

[0211] The data input to the speech end point classification model may be input data in the inference phase of the speech end point classification model, such as the above-mentioned meaning and phonetic category of the second audio frame, i.e. the format of the speech sample may correspond to the above-mentioned format of the meaning and phonetic category of the second audio frame. As another example, the input data in the inference phase of the speech end point classification model may be the above-mentioned integrated features, i.e. the format of the speech sample may correspond to the above-mentioned format of the integrated features.

[0212] In some embodiments, the speech sample may be in the format "start mark + meaning + phonetic category + end mark", or the speech sample may be in the format "start mark + phonetic category + meaning + end mark".

[0213] Optionally, some text corpora are obtained, and dictionary trees of these corpora are established, and each node of the dictionary tree (each node corresponds to one word) includes the following information: whether the node is an end point or not, and the prefix frequency. Then, speech samples may be generated based on the node information. An end point is the end point of one sentence. The prefix frequency is used to represent the amount of words between a word and an end point. A higher prefix frequency indicates that the word is less likely to be an end point. For example, other words such as verbs and prepositions, such as "give" and "take", are relatively less likely to be used as end points of sentences, and have a relatively high prefix frequency in the dictionary tree. On the other hand, words such as "right?" and "correct?" are relatively more likely to be used as end points of sentences, and usually have a relatively low prefix frequency in the dictionary tree.

[0214] If the node is not an endpoint, the endpoint category label of the plaintext (i.e., meaning) of the node is "speaking" and the endpoint category label of the signal classification information (i.e., speech category) is "thinking". If the node is an endpoint, generate speech samples with different endpoint category labels based on the prefix frequency and the signal classification information (i.e., speech category). A higher prefix frequency indicates that more audio frames with the speech category "silence" need to be added for such a node to be marked as "end". The following provides an explanation with reference to FIG. 7.

[0215] FIG. 7 is a schematic diagram of a dictionary tree according to an embodiment of the present application. In the dictionary tree shown in FIG. 7, each node contains one word, a grey circle indicates that the node is not an end point, a white circle indicates that the node is an end point, and the prefix frequency is represented by a number or letter (e.g., 0, 1, 2, x, y, z obtained in the figure). For example, "five, 0" in a white circle indicates that the node is an end point, and the prefix frequency is 0. "ten, z" in a white circle indicates that the node is an end point, and the prefix frequency is z=x+5+y+2+1+0+0, that is, the sum of the prefix frequencies of all nodes from the node to the end of the dictionary tree. Note that the letters in FIG. 7 are used to explain the method of marking the prefix frequency of a node in the dictionary tree. In practice, there is no need to introduce letters, and the prefix frequency of a node in the dictionary tree is fixed after the dictionary tree is constructed.

[0216] Step 701 may be performed by a training device. The training device may be, for example, a cloud service device, or a terminal device, such as a computer, a server, a mobile phone, a smart speaker, a vehicle, an unmanned aerial vehicle, a robot, or a system including a cloud service device and a terminal device. This is not limited to this embodiment of the present application.

[0217] 702: Train a speech end point classification model using the training data to obtain a target speech end point classification model.

[0218] For the speech end point classification model, please refer to the above description. Details will not be listed again. The target speech end point classification model can be used to obtain the speech end point of the audio data based on the voice category and the meaning. The target speech end point classification model can be used to perform step 403, or can be used by the decision module 130 to obtain the speech end point.

[0219] Step 702 may be performed by a training device.

[0220] 8 is a schematic block diagram of a speech recognition device according to an embodiment of the present application. The device 2000 shown in FIG. 8 includes: an acquisition unit 2001 and a processing unit 2002.

[0221] The acquisition unit 2001 and the processing unit 2002 can be configured to perform the speech recognition method in the embodiment of the present application. For example, the acquisition unit 2001 may perform step 401, and the processing unit 2002 may perform steps 402 and 403. As another example, the acquisition unit 2001 may perform step 501 and step 504, and the processing unit 2002 may perform steps 502 and 503, and steps 505 and 506. As another example, the acquisition unit 2001 may perform step 601, and the processing unit 2002 may perform steps 602 to 606.

[0222] The acquisition unit 2001 may include the acquisition module 110, and the processing unit 2002 may include the processing module 120 and the decision module 130.

[0223] It should be understood that the processing unit 2002 of the device 2000 may be equivalent to the processor 3002 of the device 3000 described below.

[0224] It should be noted that the device 2000 is implemented in the form of functional units, the term "unit" here may be implemented in the form of software and / or hardware, without being specifically limited thereto.

[0225] For example, a "unit" may be a software program, a hardware circuit, or a combination thereof for performing the functions described above. A hardware circuit may include an application-specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) configured to execute one or more software or firmware programs and a memory, a combinatorial logic circuit, and / or another suitable component that supports the described functions.

[0226] Therefore, the units in the examples described in the embodiments of this application can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether a function is implemented by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation should not be considered to go beyond the scope of this application.

[0227] Fig. 9 is a schematic diagram of a hardware structure of a speech recognition device according to an embodiment of the present application. The speech recognition device 3000 (the device 3000 may specifically be a computer device) shown in Fig. 9 includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, the processor 3002, and the communication interface 3003 are communicatively connected to each other using the bus 3004.

[0228] The memory 3001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are configured to execute steps of the speech recognition method in the embodiment of the present application.

[0229] The processor 3002 may be a general purpose central processing unit (CPU), a microprocessor, an ASIC, a graphics processing unit (GPU), or one or more integrated circuits and is configured to execute associated programs and implement the functions required to be performed by the processing unit 2002 of the speech recognition device in the embodiments of the present application or perform the speech recognition method in the method embodiments of the present application.

[0230] Alternatively, the processor 3002 may be an integrated circuit chip and has signal processing capabilities. In the implementation process, the steps of the speech recognition method of the present application may be implemented using instructions in the form of integrated logic circuits of hardware or software in the processor 3002. Alternatively, the processor 3002 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or another programmable logic element, a discrete gate or transistor logic element, or a discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the methods disclosed with reference to the embodiments of the present application may be directly performed by a hardware decoding processor, or may be performed using a combination of hardware and software modules in the decoding processor. The software modules may be located in conventional mature storage media, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is disposed in memory 3001. Processor 3002 reads information in memory 3001, and in combination with the processor's hardware, performs functions that need to be performed by units included in the speech recognition device of the embodiment of the present application, or executes the speech recognition method of the method embodiment of the present application. For example, processor 3002 may execute steps 402 and 403. As another example, processor 3002 may execute steps 502 and 503, and steps 505 and 506. For example, processor 3002 may execute steps 602 to 606.

[0231] As an example, but not limited thereto, a transceiver device, such as a transceiver, is used in the communication interface 3003 to realize communication between the device 3000 and other devices or communication networks. The communication interface 3003 may be configured to perform functions that need to be performed by the acquisition unit 2001 shown in Fig. 8. For example, the communication interface 3003 may perform step 401. As another example, the communication interface 3003 may perform step 501 and step 504. As another example, the communication interface 3003 may perform step 601. That is, the communication interface 3003 may be used to acquire the audio data.

[0232] The bus 3004 may include a path for transferring information between components of the device 3000 (eg, the memory 3001, the processor 3002, and the communication interface 3003).

[0233] The speech recognition device 3000 may be disposed in an in-vehicle system. Specifically, the speech recognition device 3000 may be disposed in an in-vehicle terminal. The speech recognition device may also be disposed in a server.

[0234] For the sake of explanation, only an example in which the speech recognition device 3000 is disposed in a vehicle will be described here. The speech recognition device 3000 may alternatively be disposed in other devices. For example, the speech recognition device 3000 may be applied to devices such as computers, servers, mobile phones, smart speakers, wearable devices, unmanned aerial vehicles, and robots.

[0235] It should be noted that, although only a memory, a processor, and a communication interface are shown in the device 3000 shown in FIG. 9, in a specific implementation process, a person skilled in the art should understand that the device 3000 further includes other components required for normal operation. Furthermore, based on specific requirements, a person skilled in the art should understand that the device 3000 may further include hardware components for implementing other additional functions. In addition, a person skilled in the art should understand that the device 3000 may alternatively include only the components required to implement this embodiment of the present application, and may not necessarily include all the components shown in FIG. 9.

[0236] Those skilled in the art can easily understand that the units and algorithm steps described as examples with reference to the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different devices to implement the described functions for each specific application, but the implementation should not be considered to go beyond the scope of the present application.

[0237] It can be clearly understood by those skilled in the art that for ease and conciseness of description, the specific operation processes of the above-mentioned systems, devices and units can be referred to the corresponding processes in the above-mentioned method embodiments, and the details will not be described again here.

[0238] In some embodiments provided herein, it should be understood that the disclosed system, method, and device can be implemented in other ways. For example, the described device embodiment is merely an example. For example, the division into units is merely a logical functional division, and may be divided differently during actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, and may be located in one place or distributed among multiple network units. Some or all of the units may be selected based on actual requirements to achieve the purpose of the solution of the embodiment.

[0240] Furthermore, the functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0241] When a function is implemented in the form of a software functional unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on this understanding, essentially the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes some instructions for instructing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the steps of the method of the embodiments of the present application. The above storage medium includes any medium that can store program code, such as a Universal Serial Bus Flash Disk (USB flash disk, UFD), and the UFD may also be called a USB flash drive, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0242] The above description is merely a specific implementation of the present application, and is not intended to limit the scope of protection of the present application. Any variations or replacements that are readily conceived by those skilled in the art within the technical scope disclosed in the present application should be encompassed within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the scope of protection of the claims.

Claims

1. 1. A speech recognition method comprising: obtaining audio data comprising a plurality of audio frames; extracting meaning and phonetic categories of the plurality of audio frames; obtaining speech end points of the audio data based on the meaning and phonetic categories of the plurality of audio frames; The method includes:

2. The method of claim 1 , further comprising the step of responding to an instruction corresponding to audio data prior to the speech end point in the audio data after obtaining the speech end point.

3. The step of extracting meaning and phonetic categories of the plurality of audio frames comprises: The method of claim 1 or 2, comprising obtaining the phonetic category of the plurality of audio frames based on a relationship between the energy of the plurality of audio frames and a preset energy threshold.

4. the audio categories include "speech", "neutral" and "silence", the preset energy thresholds include a first energy threshold and a second energy threshold, the first energy threshold being greater than the second energy threshold; A speech category of the audio frames among the plurality of audio frames whose energy is equal to or greater than the first energy threshold is “speech”; a speech category of the audio frames of the plurality of audio frames whose energy is less than the first energy threshold and greater than the second energy threshold is “neutral”; or The method according to any one of claims 1 to 3, wherein the audio category of the audio frames of the plurality of audio frames whose energy is less than or equal to the second energy threshold is "silence".

5. The method of claim 4 , wherein the first energy threshold and the second energy threshold are determined based on an energy of a background sound of the audio data.

6. the plurality of audio frames include a first audio frame and a second audio frame, the first audio frame being an audio frame having the meaning, and the second audio frame being an audio frame following the first audio frame among the plurality of audio frames; obtaining speech end points of the audio data based on the meaning and the phonetic categories of the plurality of audio frames, The method according to any one of claims 1 to 5, comprising obtaining the speech end point based on the meaning and a phonetic category of the second audio frame.

7. Speech end point categories include "speak", "think" and "end", and obtaining the speech end point based on the meaning and the phonetic category of the second audio frame includes:

7. The method of claim 6, further comprising the step of determining a speech end point category of the audio data based on the meaning and the phonetic category of the second audio frame, and obtaining the speech end point if the speech end point category of the audio data is "end".

8. determining a speech end point category of the audio data based on the meaning and the phonetic category of the second audio frame, 8. The method of claim 7, comprising: processing the meaning and the phonetic category of the second audio frame using a speech end point classification model to obtain the speech end point category, wherein the speech end point classification model is obtained using a speech sample and an end point category label of the speech sample, a format of the speech sample corresponds to a format of the meaning and the phonetic category of the second audio frame, and an end point category included in the end point category label corresponds to the speech end point category.

9. 1. A speech recognition method comprising: obtaining first audio data; determining a first speech end point of the first audio data; after obtaining the first speech end point, responding to a command corresponding to audio data in the first audio data prior to the first speech end point; obtaining second audio data; determining a second speech end point of the second audio data; after obtaining the second speech end point, responding to a command corresponding to audio data between the first speech end point in the first audio data and the second speech end point in the second audio data; The method includes:

10. The first audio data includes a plurality of audio frames, and determining a first speech end point of the first audio data includes: extracting meaning and phonetic categories of the plurality of audio frames; obtaining the first speech end point of the first audio data based on the meaning and the phonetic category of the plurality of audio frames; 10. The method of claim 9, comprising:

11. The step of extracting meaning and phonetic categories of the plurality of audio frames comprises: The method of claim 10 , comprising obtaining the phonetic category of the plurality of audio frames based on a relationship between an energy of the plurality of audio frames and a preset energy threshold.

12. the audio categories include "speech", "neutral" and "silence", the preset energy thresholds include a first energy threshold and a second energy threshold, the first energy threshold being greater than the second energy threshold; A speech category of the audio frames among the plurality of audio frames whose energy is equal to or greater than the first energy threshold is “speech”; a speech category of the audio frames of the plurality of audio frames whose energy is less than the first energy threshold and greater than the second energy threshold is “neutral”; or The method of claim 10 or 11, wherein the speech category of the audio frames of the plurality of audio frames whose energy is less than or equal to the second energy threshold is "silence".

13. The method of claim 12 , wherein the first energy threshold and the second energy threshold are determined based on an energy of a background sound of the first audio data.

14. the plurality of audio frames include a first audio frame and a second audio frame, the first audio frame being an audio frame having the meaning, and the second audio frame being an audio frame following the first audio frame among the plurality of audio frames; obtaining a first speech end point of the first audio data based on the meaning and the phonetic category of the plurality of audio frames, The method according to any one of claims 10 to 13, comprising obtaining the first speech end point based on the meaning and a phonetic category of the second audio frame.

15. Speech end point categories include "speak", "think" and "end", and obtaining the first speech end point based on the meaning and the phonetic category of the second audio frame includes:

15. The method of claim 14, further comprising: determining a speech end point category of the first audio data based on the meaning and the phonetic category of the second audio frame; and obtaining the first speech end point if the speech end point category of the first audio data is "end".

16. determining a speech end point category of the first audio data based on the meaning and the phonetic category of the second audio frame, 16. The method of claim 15, comprising: processing the meaning and the phonetic category of the second audio frame using a speech end point classification model to obtain the speech end point category, wherein the speech end point classification model is obtained using a speech sample and an end point category label of the speech sample, a format of the speech sample corresponds to a format of the meaning and the phonetic category of the second audio frame, and an end point category included in the end point category label corresponds to the speech end point category.

17. 1. A speech recognition device, comprising: a capture unit configured to capture audio data comprising a plurality of audio frames; a processing unit configured to extract meanings and phonetic categories of the plurality of audio frames; Including, The processing unit is further configured to obtain speech end points of the audio data based on the meaning and phonetic categories of the plurality of audio frames.

18. 20. The apparatus of claim 17, wherein the processing unit is further configured to, after obtaining the speech end point, respond to an instruction corresponding to audio data prior to the speech end point in the audio data.

19. The apparatus of claim 17 or 18, wherein the processing unit is specifically configured to obtain the phonetic category of the plurality of audio frames based on a relationship between an energy of the plurality of audio frames and a preset energy threshold.

20. the audio categories include "speech", "neutral" and "silence", the preset energy thresholds include a first energy threshold and a second energy threshold, the first energy threshold being greater than the second energy threshold; A speech category of the audio frames among the plurality of audio frames whose energy is equal to or greater than the first energy threshold is “speech”; a speech category of the audio frames of the plurality of audio frames whose energy is less than the first energy threshold and greater than the second energy threshold is “neutral”; or The apparatus of any one of claims 17 to 19, wherein a speech category of the audio frames of the plurality of audio frames whose energy is equal to or less than the second energy threshold is "silence".

21. The apparatus of claim 20 , wherein the first energy threshold and the second energy threshold are determined based on an energy of a background sound of the audio data.

22. the plurality of audio frames include a first audio frame and a second audio frame, the first audio frame being an audio frame having the meaning, and the second audio frame being an audio frame following the first audio frame among the plurality of audio frames; The apparatus according to any one of claims 17 to 21, wherein the processing unit is specifically configured to obtain the speech end point based on the meaning and a phonetic category of the second audio frame.

23. 23. The device of claim 22, wherein speech end point categories include "speaking", "thinking" and "end", and the processing unit is specifically configured to determine a speech end point category of the audio data based on the meaning and the phonetic category of the second audio frame, and obtain the speech end point if the speech end point category of the audio data is "end".

24. 24. The device of claim 23, wherein the processing unit is specifically configured to process the meaning and the phonetic category of the second audio frame using a speech end point classification model to obtain the speech end point category, the speech end point classification model being obtained using a speech sample and an end point category label of the speech sample, a format of the speech sample corresponding to a format of the meaning and the phonetic category of the second audio frame, and an end point category included in the end point category label corresponding to the speech end point category.

25. 1. A speech recognition device, comprising: an acquisition unit configured to acquire first audio data; A processing unit; Including, The processing unit includes: determining a first speech end point of the first audio data; configured to, after obtaining the first speech end point, respond to a command corresponding to audio data in the first audio data prior to the first speech end point; The acquiring unit is further configured to acquire second audio data; The processing unit includes: determining a second speech end point of the second audio data; after obtaining the second speech end point, responding to a command corresponding to audio data between the first speech end point in the first audio data and the second speech end point in the second audio data; The device further comprises:

26. The first audio data includes a plurality of audio frames, and the processing unit specifically: Extracting meaning and phonetic categories of the plurality of audio frames; obtaining the first speech end point of the first audio data based on the meaning and the phonetic category of the plurality of audio frames; 26. The apparatus of claim 25, configured to:

27. 27. The apparatus of claim 26, wherein the processing unit is specifically configured to obtain the phonetic category of the plurality of audio frames based on a relationship between an energy of the plurality of audio frames and a preset energy threshold.

28. the audio categories include "speech", "neutral" and "silence", the preset energy thresholds include a first energy threshold and a second energy threshold, the first energy threshold being greater than the second energy threshold; A speech category of the audio frames among the plurality of audio frames whose energy is equal to or greater than the first energy threshold is “speech”; a speech category of the audio frames of the plurality of audio frames whose energy is less than the first energy threshold and greater than the second energy threshold is “neutral”; or 28. The apparatus of claim 26 or 27, wherein a speech category of the audio frames of the plurality of audio frames whose energy is less than or equal to the second energy threshold is "silence".

29. 30. The apparatus of claim 28, wherein the first energy threshold and the second energy threshold are determined based on an energy of a background sound of the first audio data.

30. the plurality of audio frames include a first audio frame and a second audio frame, the first audio frame being an audio frame having the meaning, and the second audio frame being an audio frame following the first audio frame among the plurality of audio frames; The apparatus of any one of claims 26 to 29, wherein the processing unit is specifically configured to obtain the first speech end point based on the meaning and a phonetic category of the second audio frame.

31. 31. The device of claim 30, wherein speech end point categories include "speaking", "thinking" and "end", and the processing unit is specifically configured to determine a speech end point category of the first audio data based on the meaning and a phonetic category of the second audio frame, and obtain the first speech end point if the speech end point category of the first audio data is "end".

32. 32. The device of claim 31, wherein the processing unit is specifically configured to process the meaning and the phonetic category of the second audio frame using a speech end point classification model to obtain the speech end point category, the speech end point classification model being obtained using a speech sample and an end point category label of the speech sample, a format of the speech sample corresponding to a format of the meaning and the phonetic category of the second audio frame, and an end point category included in the end point category label corresponding to the speech end point category.

33. A computer readable storage medium, the computer readable storage medium storing program code for execution by an apparatus, the program code comprising instructions for use in carrying out a method according to any one of claims 1 to 8 or any one of claims 9 to 16.

34. A speech recognition device comprising a processor and a data interface, the processor reading instructions stored in a memory via the data interface to perform a method according to any one of claims 1 to 8 or any one of claims 9 to 16.

35. A computer program product, which when said computer program is run on a computer, causes said computer to carry out the method according to any one of claims 1 to 8 or any one of claims 9 to 16.

36. 1. An in-vehicle system, comprising: A device according to any one of claims 17 to 24, or A device according to any one of claims 25 to 32, A system including:

Citation Information

Patent Citations

  • Double-threshold limited place name speech endpoint detection method

    CN106847270A

  • Man-machine conversation detection method and device

    CN108257616A

  • Information control method and device and storage medium

    CN111583923A

  • Audio processing method and device and storage medium

    CN113192502A

  • Enhanced endpoint detection for speech recognition

    US9437186B1