An audio processing method, system and apparatus

Abstract

This invention discloses an audio processing method, system, and apparatus. Upon receiving an audio stream from a client, a two-layer speech activity detection is performed in parallel. When the duration of silence detected exceeds a relatively short first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding pre-recognition text. When the duration of silence detected exceeds a relatively long second time threshold, the audio stream input is determined to have ended. The recognition status of the pre-recognition text is then queried. If recognition is successful, the pre-recognition text is used as the final speech recognition text. This method introduces a shorter first time threshold to trigger pre-recognition speech recognition, providing a pre-trigger time closer to the end of the user's speech before the termination judgment (i.e., the second time threshold). Furthermore, it uses an asynchronous approach to start the pre-recognition speech recognition task to complete some key computational tasks in advance, reducing processing latency caused by the waiting time for the termination judgment.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and more specifically, to an audio processing method, system, and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence, audio processing-based smart devices have been widely used in daily life, such as smart speakers, smart glasses, and in-vehicle voice assistants. These devices typically rely on voice activity detection to determine whether the user has stopped speaking, and then sequentially perform subsequent processing such as speech recognition and domain classification. The audio processing chain is executed in a serial manner as described above, which leads to accumulated time consumption and makes it difficult to meet the experience requirements of high-speed smart devices. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an audio processing method, system, and apparatus to shorten the processing delay caused by the pause waiting time.

[0004] Firstly, an audio processing method is provided, the method comprising: Receive audio streams collected by the client; Perform voice activity detection on the audio stream; In response to the silence duration exceeding a first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding preceding recognition text; In response to the silence duration of the voice exceeding a second time threshold, the recognition status of the preceding text is determined, wherein the first time threshold is less than the second time threshold; In response to the recognition status being successful, the preceding recognition text is obtained as the speech recognition text of the audio stream.

[0005] Secondly, an audio processing system is provided, the system comprising: The client is used to send the captured audio stream; The server performs voice activity detection on the received audio stream. In response to the voice silence duration exceeding a first time threshold, it performs speech recognition on the received audio stream asynchronously to determine the corresponding preceding recognition text. In response to the voice silence duration exceeding a second time threshold, it determines the recognition status of the preceding recognition text, wherein the first time threshold is greater than the second time threshold. In response to the recognition status being successful, it obtains the preceding recognition text as the speech recognition text of the audio stream.

[0006] Thirdly, an audio processing apparatus is provided, the apparatus comprising: The receiving module is used to receive the audio stream collected by the client; The detection module is used to detect voice activity in the audio stream; The recognition module is used to asynchronously perform speech recognition on the received audio stream in response to the silence duration exceeding a first time threshold, so as to determine the corresponding preceding recognition text. A determination module is used to determine the recognition status of the preceding text in response to the voice silence duration exceeding a second time threshold, wherein the first time threshold is greater than the second time threshold; The acquisition module is used to acquire the preceding recognition text as the speech recognition text of the audio stream in response to the recognition status being successful.

[0007] Fourthly, an electronic device is provided, including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect above.

[0008] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the method described in the first aspect.

[0009] In a sixth aspect, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the method described in the first aspect above.

[0010] The technical solution of this invention, upon receiving the audio stream collected by the client, performs parallel two-layer speech activity detection. When the duration of silence detected exceeds a relatively short first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding pre-recognition text. When the duration of silence detected exceeds a relatively long second time threshold, the audio stream input is determined to have ended, and the recognition status of the pre-recognition text is queried. If the recognition is successful, the pre-recognition text is used as the final speech recognition text. This technical solution introduces a relatively short first time threshold to trigger pre-recognition speech recognition, providing a pre-trigger time point closer to the end of the user's speech before the termination judgment (i.e., the second time threshold), and uses an asynchronous method to start the pre-recognition speech recognition task to complete some key computational tasks in advance, shortening the processing latency caused by the termination waiting time. Attached Figure Description

[0011] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of the audio processing flow in existing technology; Figure 2This is a schematic diagram of the audio processing system according to an embodiment of the present invention; Figure 3 This is a flowchart of an audio processing method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the two-layer stop determination in an embodiment of the present invention; Figure 5 This is a schematic diagram of multiple pre-judgment stopping points in an embodiment of the present invention; Figure 6 A flowchart illustrating an example of an audio processing method according to an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the updating of the preprocessing results in an embodiment of the present invention; Figure 8 This is a flowchart of an audio processing method according to an embodiment of the present invention; Figure 9 This is a schematic diagram of an audio processing device according to an embodiment of the present invention; Figure 10 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0012] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0013] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0014] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0015] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0016] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0017] In the description of this invention, a "client" refers to an application running on a terminal device in a computer network, used to initiate requests to a server and receive responses from the server. It typically runs on a user's local computer, mobile device, or other smart terminal, communicating with the server via the network to obtain data, services, or perform specific operations. A "server" refers to an application in a network environment that provides data, resources, or services to a client. It typically runs on high-performance server hardware, receiving client requests via the network, processing the request content, and ultimately returning the processing result to the client. An "application" refers to a software program designed to complete a specific task or provide a specific function, and can run on various devices, such as personal computers, smartphones, and tablets. Applications include not only software running based on interfaces provided by the operating system, but also applets running within operating system or platform applications, such as shortcuts and applets on mobile phones and web applications in browsers. These applications interact with underlying hardware and system resources through APIs (Application Programming Interfaces) provided by the operating system to achieve their functions. A "server" refers to a computer hardware and software system that provides hardware resources for server-side applications. It can be a single computer device, a cluster of computer devices, or a virtual computer device deployed in the cloud.

[0018] In the field of speech processing technology, Voice Activity Detection (VAD) is a technique for determining the presence of valid speech (i.e., human voice) in an audio stream, used to distinguish between segments of human voice and silent / noise segments. VAD is the technological foundation for implementing pause detection. When a continuous period of silence in the audio stream is detected to reach a predetermined pause time threshold, it can be determined that the user has finished speaking (i.e., pause detection), thereby triggering the termination of pause detection, stopping the continuous acquisition of the audio stream, and initiating the subsequent speech processing flow. Simply put, VAD is used to determine whether the user has finished voice input.

[0019] Automatic Speech Recognition (ASR) is a technology that converts speech into corresponding text.

[0020] Domain classification is a text classification task that determines the domain or category to which a piece of text belongs. For example, domains can include technology, food and beverage, finance, and healthcare.

[0021] The aforementioned speech activity detection, speech recognition, and domain classification are all upstream tasks for determining response actions, providing an analytical basis for the determination and execution of response actions.

[0022] Figure 1 This is a schematic diagram of the existing audio processing flow. The existing audio processing flow uses a serial execution method to recognize user input speech and perform different response processing based on the recognition results. For example, it can perform question-and-answer based on user input speech, or translate user input speech into different languages, etc. Figure 1 As shown, the audio stream processing chain is: speech activity detection → speech recognition → domain classification → response generation. Specifically, after the user begins to speak, the smart device starts collecting the audio stream and performing speech activity detection. When the detected silence duration reaches a predetermined stopping threshold (e.g., 700ms in the figure), it determines "silence" and triggers the subsequent processing flow: first, the speech recognition task is started to convert the received audio stream into corresponding speech recognition text; then, the domain classification task is entered to determine the intent and classify the recognized text to obtain the classification result; finally, based on the classification result, the corresponding response execution logic is called, such as querying information, playing music, or translating. The entire processing chain exhibits a strict serial dependency relationship, with each stage waiting for the previous stage to complete before starting, resulting in high response latency. Especially in multimodal, complex, or long-tail scenarios, latency will affect the user experience.

[0023] Easy to understand Figure 1 The processing times shown for each stage are examples.

[0024] Figure 2 This is a schematic diagram of an audio processing system according to an embodiment of the present invention. Figure 2As shown, the audio processing system includes a terminal device 21, a network 22, and a server 23. The terminal device 21 can run a client 21a. The client 21a is a firmware application or a user-defined application used for audio processing. In the following description, the client can be a smart assistant application for interacting with the user via voice or other types of applications. The server 23 can run a server 23a. The server 23a is an application suitable for communicating with the client 21a and performing audio processing defined by the developer. The terminal device 21 can communicate with the server 23 through the network 22, thereby enabling data exchange between the client 21a and the server 23a. This allows the terminal device 21 to delegate audio processing tasks requiring significant computational resources and fast response times to the server 23a, and return relevant information about the corresponding response actions to the client 21a. This allows the terminal device 21 to perform functions requiring substantial computational resources while maintaining a small size and low power consumption. In this embodiment, the terminal device 21 can be a smart speaker, a smartwatch, or smart glasses, or other wearable smart devices. Network 22 can be a single network, such as a mobile data communication network based on a mobile communication base station, or it can be a composite network, including a local area network (LAN) and a wide area network (WAN). Terminal device 21 can directly access network 22, which requires terminal device 21 to be configured with the necessary hardware modules for network access, such as a mobile communication chip. In some alternative implementations, for power consumption or cost considerations, terminal device 21 (e.g., smart glasses) can also connect to a user's mobile phone or tablet computer 24 via Wi-Fi or Bluetooth, and access network 22 via a relay from the mobile phone or tablet computer 24.

[0025] In the audio processing system of this embodiment, the client 21a of the terminal device 21 is used to send the acquired audio stream, which is audio data acquired by the terminal device 21 through its microphone. This audio data may include ambient sound, as well as the voice of the user of the client 21a or other persons. The client 21a continuously sends the audio stream to the server 23a when a trigger condition is met. The server 23a, upon receiving the audio stream, performs parallel two-layer voice activity detection. One layer performs pre-voice activity detection based on a first time threshold. When the detected silence duration exceeds the relatively short first time threshold, it asynchronously performs speech recognition on the received audio stream to determine the corresponding pre-recognition text. The other layer performs end-voice activity detection based on a second time threshold. When the detected silence duration exceeds the relatively long second time threshold, it determines that the audio stream input has ended, queries the recognition status of the pre-recognition text, and if recognition is successful, uses the pre-recognition text as the final speech recognition text. Therefore, server 23a introduces a relatively short first time threshold to trigger pre-speech recognition, providing a pre-trigger time point closer to the end of the user's speech before the end of the stop judgment (i.e. the end of speech activity detection), and adopts an asynchronous method to start the pre-speech recognition task to complete some key calculation tasks in advance, shortening the processing latency caused by the stop judgment waiting time and improving the user experience.

[0026] Figure 3 This is a flowchart of an audio processing method according to an embodiment of the present invention. Figure 3 As shown, the method includes the following steps: Step S301: Receive the audio stream collected by the client.

[0027] The audio stream is a continuous audio data sequence continuously collected by the client. Its core feature is that it is collected, transmitted and processed simultaneously. That is, while the following steps S302-S305 are being executed, step S301 is also being executed continuously.

[0028] Step S302: Perform voice activity detection on the audio stream.

[0029] In one possible implementation, a first voice activity detection and a second voice activity detection are performed in parallel on the audio stream. The first voice activity detection is used to detect that the duration of silence in the audio stream exceeds a first time threshold, and the second voice activity detection is used to detect that the duration of silence in the audio stream exceeds a second time threshold. The first time threshold is less than the second time threshold. In this embodiment, the first time threshold is a predetermined pre-stop threshold, and the second time threshold is a predetermined end-stop threshold.

[0030] The second speech activity detection task is part of the main audio processing chain. The first speech activity detection task is an additional pre-processing task added to the main audio processing chain. The first speech activity detection triggers the pre-processing task in advance by setting a pre-stop threshold (i.e., the first time threshold) shorter than the termination threshold. The pre-processing task includes at least pre-processing speech recognition. After the first speech activity detection terminates the task in advance, the second speech activity detection continues until the duration of silence reaches the second time threshold, at which point the termination is terminated.

[0031] Figure 4 This is a schematic diagram of the two-layer stop determination method according to an embodiment of the present invention. Figure 4 As shown, when a non-human voice segment is detected, the duration of voice silence is counted. The first voice activity detection task detects that the duration of voice silence has reached the pre-stop point, which is the first time threshold (e.g., 300ms). If it is still silent, the second voice activity detection task, which is executed in parallel, can detect that the duration of voice silence has reached the end stop point, which is the second time threshold (e.g., 700ms).

[0032] In one possible implementation, in a complete audio stream, there may be multiple preceding stop points before reaching the end stop point. That is, the user often pauses while speaking, and the duration of each pause is between the first time threshold and the second time threshold. The end stop point is not reached and continues to be delayed.

[0033] Figure 5 This is a schematic diagram illustrating multiple pre-judgment stopping points according to an embodiment of the present invention. For example... Figure 5 As shown, when a non-human voice segment is detected, the voice silence duration timing begins. The first voice activity detection task detects that the voice silence duration has reached the first pre-stop point (2300ms). Before the second voice activity detection task detects that the voice silence duration has reached the end stop point (2700ms), if the user continues to speak, both the first and second voice activity detection tasks continue detection. When another non-human voice segment is detected, the voice silence duration timing restarts. The first voice activity detection task detects that the voice silence duration has reached the second pre-stop point (4300ms), and then the second voice activity detection task detects that the voice silence duration has reached the end stop point (4700ms), thus determining that the user has finished speaking. Therefore, there are two pre-stop points before the end stop point in the diagram.

[0034] In step S303, in response to the silence duration exceeding a first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding pre-recognition text.

[0035] The asynchronous method refers to starting subsequent speech recognition tasks and other processing flows in advance when a pre-judgment stop point is detected, without waiting for the user's speech to officially end. This subsequent processing flow starts in another flow and does not block the main audio processing link where the second speech activity detection is located. The main audio processing link continues to perform the second speech task detection until the end judgment.

[0036] In one possible implementation, the pre-processing task includes not only speech recognition but also domain classification. Specifically, after determining the pre-recognition text, the pre-recognition text is classified into different domains to determine the corresponding pre-recognition domain classification result.

[0037] In one possible implementation, the preprocessing task further includes a rejection task. Rejection refers to refusing to process invalid or unresponsive audio streams to avoid making erroneous responses. In this embodiment, rejection means refusing to determine the currently pending audio stream as a valid audio stream, avoiding erroneous responses to idle chatter, malicious statements, or noise, thereby improving the user experience. For example, if the semantics of the user's input speech are incomplete or the speech in the audio stream is determined to be noise, it will be rejected.

[0038] In one possible implementation, the method further includes maintaining the pre-processing results. Specifically, the pre-processing results are stored, including the task results of the pre-speech recognition task and the task results of the pre-domain classification task. The task results of the pre-speech recognition task include a recognition status, which includes recognition success and recognition failure. If recognition is successful, the task results also include the pre-recognized text. The task results of the pre-domain classification task include a classification status, which includes classification success and classification failure. If classification is successful, the task results also include the pre-domain classification result. In other words, the pre-recognized text obtained asynchronously and the corresponding pre-domain classification result are cached.

[0039] Optionally, the prior domain classification result uses a combination of device identifier and corresponding prior identification text as a cache key, whereby the device identifier is used to uniquely identify the terminal device to which the client belongs.

[0040] In one possible implementation, if there are multiple pre-processing stops during audio processing, the pre-processing result needs to be updated each time a new pre-processing stop occurs and the corresponding recognition status is determined to be successful.

[0041] Specifically, when determining the preceding recognition text corresponding to the current preceding stop point, it is necessary to query the cache to see if there is any historical preceding recognition text corresponding to the audio stream. If there is, the preceding recognition text is overwritten or appended to the cache corresponding to the audio stream, and the preceding domain classification result is appended to the domain classification result sequence in the cache.

[0042] In other words, there are two ways to update the preceding recognition text. One is that the audio recognized each time includes all audio streams received at the current preceding stop point. Therefore, the corresponding update method is to overwrite the previously stored historical preceding recognition text in the cache with the currently recognized preceding recognition text. For example, if the preceding recognition text corresponding to the first preceding stop point is "I want to listen to music", it is written to the cache. If the preceding recognition text corresponding to the second preceding stop point is "I want to listen to music, choose rock", then "I want to listen to music, choose rock" overwrites "I want to listen to music". The other is that the audio recognized each time includes the audio between the previous preceding stop point and the current preceding stop point. Therefore, the corresponding update method is to append the currently recognized preceding recognition text to the previously stored historical preceding recognition text in the cache. For example, if the preceding recognition text corresponding to the first preceding stop point is "I want to listen to music", it is written to the cache. If the preceding recognition text corresponding to the second preceding stop point is "choose rock", then "choose rock" is appended to "I want to listen to music". The domain classification results are stored as sequences, for example, the sequence is {[UUID+“I want to listen to music”]-music domain; [UUID+“I want to listen to music”]-music domain}.

[0043] Step S304: In response to the voice silence duration exceeding the second time threshold, determine the recognition status of the preceding text.

[0044] Specifically, if the duration of voice silence exceeds the second time threshold, a pre-processing result query task is initiated. The pre-processing result query task refers to querying the processing results of the pre-processing task, which includes at least the pre-recognized text and its recognition status.

[0045] In one possible implementation, if the preprocessing task includes a pre-processing domain classification task, the preprocessing result query task further includes querying the pre-processing classification result and its classification status. Specifically, in response to the speech silence duration exceeding a second time threshold and the pre-processing domain classification ending, the pre-processing domain classification result corresponding to the pre-recognition text is obtained as the domain classification result of the audio stream.

[0046] Step S305: In response to the recognition status being successful, the preceding recognition text is obtained as the speech recognition text of the audio stream.

[0047] In one possible implementation, in response to the classification status being successful, the preceding domain classification result corresponding to the preceding recognized text is obtained as the domain classification result of the audio stream.

[0048] In one possible implementation, a response action can be determined based on the speech recognition text of the audio stream and the domain classification result, and the client can be responded to based on the response action. For example, if the speech recognition text is "I want to listen to music, so I'll choose rock music," then the domain classification result is "music domain," and the corresponding response action should be to call the music player to play rock music. Or, if the speech recognition text is "What is one plus one?", then the domain classification result is "data calculation," and the corresponding response action is to calculate that one plus one equals two and return the calculation result "two" to the client for display.

[0049] Steps S304-S305 refer to the following: After the termination judgment, if there are already valid results generated by the previous speech recognition task and the previous domain classification task in the cache, that is, the previous recognition text and the previous classification result, they can be quickly queried and used as the speech recognition text and domain classification result of the audio stream. There is no need to continue to perform speech recognition and domain classification on the complete audio in the main audio processing link, thereby shortening the response delay.

[0050] In one possible implementation, a preset query time is set to constrain the processing time of the preprocessing result query task. If no valid pre-recognition text and pre-domain classification result are found within the preset query time, the process returns to the main audio processing link to continue serial processing, so as to avoid additional delays or request loss due to query task obstruction or failure, thereby improving robustness.

[0051] In one possible implementation, the preset query time can be the longest time used to query and / or obtain the preceding identification text or the preceding identification text and its corresponding preceding domain classification results.

[0052] Optionally, the preset query time can be further divided into a first query time and a second query time. The first query time is used to constrain the query time corresponding to the preceding recognized text, and the second query time is used to constrain the query time corresponding to the preceding domain classification result.

[0053] In one possible implementation, the preset query time is positively correlated with the overall duration of the audio stream. The longer the audio stream and the more user input, the longer the processing time for the preceding speech recognition and domain classification tasks typically becomes. Appropriately extending the query time can improve the query hit rate and avoid prematurely abandoning cached queries and repeatedly executing time-consuming speech recognition and domain classification tasks. This dynamic adaptation mechanism can maximize the utilization of asynchronous processing results while ensuring low latency, thereby optimizing overall processing efficiency.

[0054] In one possible implementation, in response to the failure to find the recognition status within a preset query time or the found recognition status being a recognition failure, speech recognition and domain classification are performed on the received audio stream to determine the corresponding speech recognition text and domain classification result. A response action is then determined based on the speech recognition text and the domain classification result, and a response is made to the client based on the response action. That is, if no valid preceding recognition text and preceding domain classification result are found within the preset query time, the process returns to the main audio processing link to continue performing speech recognition and domain classification serially.

[0055] In one possible implementation, the query process is as follows: first, query the preceding recognition text. If the preceding recognition text is successfully found, then use the device identifier + (the found) preceding recognition text as the key to query the corresponding preceding classification result. Since the same terminal device may trigger preceding speech recognition and preceding domain classification multiple times in a short period of time, and different preceding recognition texts may correspond to different preceding classification results, combining the device identifier and the preceding recognition text as the key can ensure that the corresponding preceding classification result corresponds to the preceding recognition text, avoiding mismatches. At the same time, after the formal termination of the judgment, the preceding processing result can be quickly and accurately reused. Compared with using the preceding recognition text encoding as the key, the encoding and decoding process can be saved, improving the response speed.

[0056] The above Figure 3 The method described can be flexibly deployed on different devices based on the computing power of the terminal devices and the actual application requirements. It can be executed entirely locally on terminal devices with strong computing capabilities, or it can be entirely implemented by the server. In addition, the above processing flow can be broken down, with some tasks completed on the terminal devices and the rest processed collaboratively by the server. The specific deployment should comprehensively consider the hardware resources of the terminal devices, the timeliness requirements of audio processing, network conditions, and the design goals of the overall system architecture to achieve a balance between performance, efficiency, and resource utilization.

[0057] The method of this invention, upon receiving the audio stream collected by the client, performs parallel two-layer speech activity detection. When the duration of silence detected exceeds a relatively short first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding pre-recognition text. When the duration of silence detected exceeds a relatively long second time threshold, the audio stream input is determined to have ended, and the recognition status of the pre-recognition text is queried. If the recognition is successful, the pre-recognition text is used as the final speech recognition text. This method introduces a relatively short first time threshold to trigger pre-recognition speech recognition, providing a pre-trigger time point closer to the end of the user's speech before the termination judgment (i.e., the second time threshold), and uses an asynchronous approach to start the pre-recognition speech recognition task to complete some key computational tasks in advance, thus shortening the processing latency caused by the termination waiting time.

[0058] Figure 6 This is a flowchart illustrating an example of an audio processing method according to an embodiment of the present invention. Figure 6 As shown, the processing flow for this example is as follows: Step S601: Start acquiring the audio stream.

[0059] Specifically, once the user begins to speak, the system enters the streaming voice acquisition phase to obtain the audio stream. For example, it might acquire the user's first sentence, "I want to listen to music."

[0060] Step S602: The first voice activity detection and the second voice activity detection are started in parallel to continuously detect the voice activity status.

[0061] Step S603: When a non-human voice segment is detected, start timing the duration of voice silence.

[0062] Step S604: When the duration of voice silence is detected to exceed the first time threshold, the preceding speech recognition task and the preceding domain classification task are executed asynchronously.

[0063] Specifically, if a user pauses briefly after saying "I want to listen to music," meeting the pre-stop requirement (i.e., the duration of the silence exceeds a first time threshold), then the first pre-stop point is determined. The pre-recognition task includes: performing speech recognition on the received audio stream to determine the corresponding pre-recognition text – "I want to listen to music," and caching this text after successful identification. The pre-domain classification task includes: classifying the pre-recognition text into a domain to determine the corresponding pre-domain classification result – "music domain." After successfully determining the pre-classification result, "I want to listen to music" is cached as a key-value pair, with the key being the device identifier + "I want to listen to music," to ensure uniqueness and prevent overwriting by subsequent pre-classification results.

[0064] At this point, the user has not actually finished speaking, so the second voice activity detection in the main audio processing chain is still waiting.

[0065] Step S605: Before the duration of voice silence reaches the second time threshold, a human voice segment is detected.

[0066] For example, the second sentence the user said was, "I'll choose rock music."

[0067] Step S606: If the duration of voice silence exceeds the first time threshold again, execute the preceding speech recognition task and the preceding domain classification task asynchronously.

[0068] Specifically, when the user pauses briefly again while speaking, meeting the pre-stop requirement, a second pre-stop point is identified, triggering a new round of pre-speech recognition and pre-domain classification tasks. The key for the pre-classification result in the second round is the device identifier plus "select rock music".

[0069] Step S607: Update the cached previous recognition text and previous classification results.

[0070] Specifically, the preceding recognized text is overwritten or appended to the cache corresponding to the audio stream, and the preceding domain classification results are appended to the domain classification result sequence in the cache.

[0071] Figure 7 This is a schematic diagram illustrating the updating of the preprocessing results in an embodiment of the present invention. Figure 7 As shown, the preceding recognition text determined based on the first preceding judgment point is "I want to listen to music", and the preceding domain classification result is music. This preceding domain classification result is stored in key-value pairs, specifically [UUID + "I want to listen to music"] - music domain, where the key is the device identifier + "I want to listen to music". At this time, the preceding recognition text stored in the cache is "I want to listen to music", and the domain classification result sequence is {[UUID + "I want to listen to music"] - music domain}. The preceding recognition text determined based on the second preceding judgment point is "I want to listen to music, so I'll choose rock music", and the preceding domain classification result is music. This preceding recognition text overwrites the recognition text corresponding to the first preceding judgment point, and the preceding domain classification result is appended to the domain classification result sequence. Therefore, the preceding recognition text updated in the cache at this time is "I want to listen to music, so I'll choose rock music", and the domain classification result sequence is {[UUID + "I want to listen to music"] - music domain; [UUID + "I want to listen to music"] - music domain}.

[0072] Since the same terminal device may trigger pre-processing speech recognition and pre-processing domain classification multiple times in a short period of time, and the pre-processing classification results may differ for different pre-processing texts, combining the device identifier and the pre-processing text as a key can ensure that the corresponding pre-processing classification result corresponds to the pre-processing text, avoiding mismatches. Furthermore, the pre-processing results can be quickly and accurately reused after completion. Compared to using the encoded pre-processing text as a key, this eliminates the encoding and decoding process, improving response speed.

[0073] Step S608: When the duration of voice silence is detected to exceed the second time threshold, the recognition status and classification status of the preceding text are determined.

[0074] Specifically, after the user stops speaking at the second pre-stop point, the stop-stop condition is met until the silence duration exceeds the second time threshold, at which point the stop-stop point is determined to have occurred. At this point, before starting the speech recognition and domain classification tasks in the main audio processing chain, it is first checked whether there are valid pre-recognition texts and pre-classification results.

[0075] In one possible implementation, a preset query time is set to constrain the query time of the preceding recognized text.

[0076] Step S609: In response to the recognition status being successful and the classification status being successful, the preceding recognition text is obtained as the speech recognition text of the audio stream, and the preceding domain classification result corresponding to the preceding recognition text is obtained as the domain classification result of the audio stream.

[0077] Specifically, during the pre-query process, if the pre-recognized text is found, the process of performing speech recognition on the entire audio stream in the main audio processing link is eliminated. If the pre-recognized domain classification result is found, the domain classification process in the main audio processing link is eliminated.

[0078] Step S610: Determine the response action based on the speech recognition text and domain classification results of the audio stream.

[0079] Step S611: In response to the failure to find the recognition status or the failure to find the recognition status, perform speech recognition on the received audio stream to determine the corresponding speech recognition text.

[0080] Step S612: In response to the failure to find the classification status or the failure to find the classification status, perform domain classification on the received audio stream and determine the corresponding domain classification result.

[0081] Specifically, if no preceding text is found, speech recognition is performed in the main audio processing link; if no preceding classification result is found, domain classification is performed in the main audio processing link.

[0082] Step S613: Determine the response action based on the speech recognition text and the domain classification result.

[0083] Step S614: Respond to the client according to the response action.

[0084] It is easy to understand that the audio stream is continuously collected and updated until the stop judgment ends.

[0085] The specific implementation methods of each of the above steps are detailed in the above embodiments, and will not be repeated here.

[0086] The method of this invention introduces a relatively short first time threshold to trigger pre-speech recognition, provides a pre-trigger time point closer to the end of the user's speech before the end of the stop judgment (i.e., the second time threshold), and adopts an asynchronous approach to start the pre-speech recognition task to complete some key computing tasks in advance, thereby shortening the processing delay caused by the stop judgment waiting time.

[0087] Figure 8 This is a flowchart of an audio processing method according to an embodiment of the present invention. Figure 8 The method shown describes the audio processing method of this embodiment from the perspective of a two-layer link. For example... Figure 8 As shown, the processing flow for this example is as follows: Step S801: Obtain the audio stream.

[0088] In step S802, the front-end audio processing link starts the first voice activity detection, and the main audio processing link starts the second voice activity detection.

[0089] In step S803, both the pre-audio processing link and the main audio processing link start timing the duration of voice silence after detecting non-human voice segments.

[0090] In step S804, the pre-audio processing link detects for the first time that the duration of speech silence exceeds the first time threshold, and executes the pre-speech recognition task and the pre-domain classification task asynchronously.

[0091] Furthermore, the preceding recognition text and the corresponding preceding domain classification result obtained from performing the preceding speech recognition task and the preceding domain classification task are cached, wherein the preceding domain classification result uses the combination of the device identifier and the corresponding preceding recognition text as the cache key.

[0092] In step S805, the main audio processing link detects a human voice segment before the voice silence duration reaches the second time threshold, and determines that the microphone is still not turned on.

[0093] In step S806, the pre-audio processing link detects for the second time that the duration of speech silence exceeds the first time threshold, and then executes the pre-speech recognition task and the pre-domain classification task asynchronously again.

[0094] Furthermore, update the cached previous recognized text and previous classification results.

[0095] In step S807, the main audio processing link detects that the duration of voice silence exceeds the second time threshold and queries the preprocessing results.

[0096] The pre-processing results include pre-recognition text and pre-classification results. The specific query process involves first querying the pre-recognition text. If the pre-recognition text is successfully found, the corresponding pre-classification result is retrieved using the device identifier plus the (found) pre-recognition text as the key. Since the same terminal device may trigger pre-recognition speech recognition and pre-domain classification multiple times within a short period, and different pre-recognition texts may correspond to different pre-classification results, combining the device identifier and the pre-recognition text as the key ensures that the corresponding pre-classification result matches the pre-recognition text, avoiding mismatches. Furthermore, the pre-processing results can be quickly and accurately reused after completion. Compared to using the encoded pre-recognition text as the key, this eliminates the encoding and decoding process, improving response speed.

[0097] In one possible implementation, a preset query time is set to constrain the query time of the preceding recognized text.

[0098] Step S808: If the main audio processing link successfully obtains the pre-recognition text and the pre-classification result, then skip the speech recognition process and the domain classification process in the main audio processing link, and directly use the pre-recognition text and its corresponding pre-classification result as the speech recognition text and domain classification result of the audio stream.

[0099] Step S809: If the main audio processing link fails to obtain the preceding recognition text and preceding classification results, then the main audio processing link performs speech recognition and domain classification to obtain the speech recognition text and domain classification results of the audio stream.

[0100] In step S810, the main audio processing link determines a response action based on the speech recognition text and the domain classification result, and responds to the client based on the response action.

[0101] The specific implementation methods of each of the above steps are detailed in the above embodiments, and will not be repeated here.

[0102] The method in this invention introduces a relatively short first time threshold to trigger pre-speech recognition, providing a pre-trigger time point closer to the end of the user's speech before the termination judgment (i.e., the second time threshold). It also employs an asynchronous approach to start the pre-speech recognition task, allowing for the early completion of some critical computational tasks and reducing processing latency caused by the termination waiting time. By adding a pre-audio processing link without altering the main audio processing link logic, and without relying on complex overwrites or logic reconstructions, this method exhibits good versatility and scalability.

[0103] Figure 9 This is a schematic diagram of an audio processing apparatus according to an embodiment of the present invention. Figure 9 As shown, the device includes: The receiving module 901 is used to receive the audio stream collected by the client; Detection module 902 is used to detect voice activity in the audio stream; The recognition module 903 is used to asynchronously perform speech recognition on the received audio stream in response to the silence duration exceeding a first time threshold, so as to determine the corresponding preceding recognition text. The determination module 904 is used to determine the recognition status of the preceding recognition text in response to the voice silence duration exceeding the second time threshold, wherein the first time threshold is greater than the second time threshold; The acquisition module 905 is used to acquire the preceding recognition text as the speech recognition text of the audio stream in response to the recognition status being successful.

[0104] The device in this embodiment of the invention introduces a relatively short first time threshold to trigger pre-speech recognition, provides a pre-trigger time point closer to the end of the user's speech before the termination judgment (i.e., the second time threshold), and starts the pre-speech recognition task asynchronously to complete some key computing tasks in advance, thereby shortening the processing delay caused by the termination waiting time.

[0105] Figure 10 This is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 1000 includes a server, a terminal, etc. Figure 10 As shown, the electronic device 1000 includes at least one processor 1001; a memory 1002 communicatively connected to at least one processor 1001; and a communication component 1003 communicatively connected to a scanning device, wherein the communication component 1003 receives and transmits data under the control of the processor 1001; wherein the memory 1002 stores instructions executable by at least one processor 1001, which are executed by at least one processor 1001 to implement the above-described audio processing method.

[0106] Specifically, the electronic device includes: one or more processors 1001 and a memory 1002. Figure 10 Taking a processor 1001 as an example, the processor 1001 and the memory 1002 can be connected via a bus or other means. Figure 10 Taking a bus connection as an example, memory 1002, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 1001 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 1002, thereby implementing the aforementioned audio processing method.

[0107] The memory 1002 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, the memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1002 may optionally include memory remotely located relative to the processor 1001, and these remote memories can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] One or more modules are stored in memory 1002 and, when executed by one or more processors 1001, perform the audio processing method in any of the above method embodiments.

[0109] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0110] The technical solution of this invention, upon receiving the audio stream collected by the client, performs parallel two-layer speech activity detection. When the duration of silence detected exceeds a relatively short first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding pre-recognition text. When the duration of silence detected exceeds a relatively long second time threshold, the audio stream input is determined to have ended, and the recognition status of the pre-recognition text is queried. If the recognition is successful, the pre-recognition text is used as the final speech recognition text. This technical solution introduces a relatively short first time threshold to trigger pre-recognition speech recognition, providing a pre-trigger time point closer to the end of the user's speech before the termination judgment (i.e., the second time threshold), and uses an asynchronous method to start the pre-recognition speech recognition task to complete some key computational tasks in advance, shortening the processing latency caused by the termination waiting time.

[0111] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements some or all of the above-described method embodiments.

[0112] Another embodiment of the present invention relates to a computer program product, including a computer program / instructions that, when executed by a processor, implement some or all of the above-described method embodiments.

[0113] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0114] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0115] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An audio processing method, characterized in that, The method includes: Receive audio streams collected by the client; Perform voice activity detection on the audio stream; In response to the silence duration exceeding a first time threshold, speech recognition is performed asynchronously on the received audio stream to determine the corresponding preceding recognition text; In response to the silence duration of the voice exceeding a second time threshold, the recognition status of the preceding text is determined, wherein the first time threshold is less than the second time threshold; In response to the recognition status being successful, the preceding recognition text is obtained as the speech recognition text of the audio stream.

2. The method according to claim 1, characterized in that, After determining the preceding recognition text, the method further includes: Perform domain classification on the pre-identified text to determine the corresponding pre-identified domain classification result; In response to the speech silence duration exceeding the second time threshold and the previous domain classification ending, the previous domain classification result corresponding to the previous recognized text is obtained as the domain classification result of the audio stream; The response action is determined based on the speech recognition text and domain classification results of the audio stream; The system responds to the client based on the stated response action.

3. The method according to claim 2, characterized in that, The method further includes: The preceding recognition text and the corresponding preceding domain classification result are cached, wherein the preceding domain classification result uses the combination of the device identifier and the corresponding preceding recognition text as the cache key.

4. The method according to claim 3, characterized in that, The step of caching the preceding recognized text and the corresponding preceding domain classification result includes: The preceding recognized text is overwritten or appended to the cache corresponding to the audio stream; The preceding domain classification results are appended to the domain classification result sequence in the cache.

5. The method according to claim 4, characterized in that, The step of overwriting or appending the pre-recognized text to the cache corresponding to the audio stream includes: In response to the existence of historical preceding recognition text in the cache corresponding to the audio stream, the preceding recognition text is overwritten by the historical preceding recognition text, or the preceding recognition text is appended to the historical preceding text.

6. The method according to claim 1, characterized in that, The process of detecting voice activity in the audio stream includes: The audio stream is subjected to a first speech activity detection and a second speech activity detection in parallel. The first speech activity detection is used to detect that the duration of speech silence in the audio stream exceeds a first time threshold, and the second speech activity detection is used to detect that the duration of speech silence in the audio stream exceeds a second time threshold.

7. The method according to claim 4, characterized in that, The step of responding to the recognition status as successful recognition and obtaining the preceding recognition text as the speech recognition text of the audio stream includes: In response to a successful recognition status found within a preset query time, the preceding recognition text is obtained as the speech recognition text of the audio stream.

8. The method according to claim 7, characterized in that, The method further includes: In response to the failure to find the recognition status within a preset query time or the recognition status found is recognition failure, the received audio stream is subjected to speech recognition and domain classification to determine the corresponding speech recognition text and domain classification result. The response action is determined based on the speech-recognized text and the domain classification result; Respond to the client according to the response action.

9. The method according to claim 7, characterized in that, The preset query time is positively correlated with the overall duration of the audio stream.

10. An audio processing system, characterized in that, The system includes: The client is used to send the captured audio stream; The server performs voice activity detection on the received audio stream. In response to the voice silence duration exceeding a first time threshold, it performs speech recognition on the received audio stream asynchronously to determine the corresponding preceding recognition text. In response to the voice silence duration exceeding a second time threshold, it determines the recognition status of the preceding recognition text, wherein the first time threshold is greater than the second time threshold. In response to the recognition status being successful, it obtains the preceding recognition text as the speech recognition text of the audio stream.

11. An audio processing apparatus, characterized in that, The device includes: The receiving module is used to receive the audio stream collected by the client; The detection module is used to detect voice activity in the audio stream; The recognition module is used to asynchronously perform speech recognition on the received audio stream in response to the silence duration exceeding a first time threshold, so as to determine the corresponding preceding recognition text. A determination module is used to determine the recognition status of the preceding text in response to the voice silence duration exceeding a second time threshold, wherein the first time threshold is greater than the second time threshold; The acquisition module is used to acquire the preceding recognition text as the speech recognition text of the audio stream in response to the recognition status being successful.

12. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-9.

Images