A low-latency speech input method and system based on a resident audio stream

Abstract

The application discloses a low-delay voice input method and system based on a resident audio stream. The method comprises the following steps: when the voice input system is started, an audio input stream is created and kept in an active state, and the life cycle of the audio input stream is independent of a recording flag; when the recording flag is closed, the audio stream keeps running, but the data is discarded in silence; when a trigger signal is received, the recording flag is opened, audio data is captured, and the audio data is written into a memory buffer and a disk temporary file at the same time; when a release signal or a retrigger signal is received, the capturing is stopped, the audio data is sent to a local offline voice recognition model for recognition, and the recognition result is transmitted to an application system for operation. The application reduces the recording starting delay from hundreds of milliseconds to sub-microsecond level through the resident audio stream, eliminates the first word swallowing problem, and balances the data safety, recognition accuracy and long voice processing capacity through the double-writing mechanism, two-time recognition and voice activity detection segmentation.

Description

Technical Field

[0001] This invention relates to the field of speech recognition and human-computer interaction technology, and more specifically, to a low-latency speech input method and system based on a resident audio stream. Background Technology

[0002] Voice input technology, as an important method of human-computer interaction, has been widely used in recent years in fields such as office automation, smart home control, in-vehicle interaction, and entertainment device control. However, existing voice input solutions suffer from several long-standing technical problems in terms of user experience, which are particularly prominent in certain application scenarios.

[0003] First, microphone cold start latency is a long-standing technical pain point in the voice input field. Traditional voice input follows a serial processing model: the user first triggers the recording operation, the system responds and begins initializing the audio device, including turning on the microphone hardware, establishing the audio input stream, and allocating buffer memory. Audio data is only collected after the audio device initialization is complete, and finally, the collected audio data is sent to the speech recognition engine for recognition. In this process, the audio device initialization step typically takes 200 to 500 milliseconds. More seriously, there is a time lag between user behavior and the system processing flow—users are accustomed to starting to speak at the same time as pressing the trigger button, but the system has not yet completed the audio device initialization within the first 200 to 500 milliseconds after triggering. This results in the first 0 to 0.5 seconds of the user's speech not being collected and recognized; this phenomenon is known in the industry as the first-word swallowing problem. The first-word swallowing problem is particularly severe in noisy environments such as in-car environments and KTV rooms, because users in these environments typically need to complete interactions with shorter commands. Losing the first word will directly cause the entire command to fail to be recognized, severely impacting user experience and interaction efficiency.

[0004] Secondly, existing patented technologies primarily target wake-word detection scenarios in smart speakers or button truncation prevention scenarios in medical dictation devices. Their technical solutions differ significantly from those for multi-terminal physical button triggering scenarios. Existing technologies, represented by US Patent No. 8781832B2 and Chinese Patent No. 106796497B, while employing a continuous recording scheme with a circular buffer, have several limitations. First, these solutions rely on wake-word triggers, making them unsuitable for physical button-triggered interaction scenarios. Physical button triggering offers irreplaceable advantages over wake-word triggers in scenarios such as desktop office work, in-vehicle control, and karaoke, including high trigger certainty, triggering without vocalization, and immunity to environmental noise. Second, these solutions are designed for single-function devices, failing to consider the unified architecture requirements of multi-terminal forms, leading to redundant development of underlying functions for different terminals. Third, these solutions rely on cloud-based automatic speech recognition services, resulting in network latency issues and data privacy risks, rendering them unusable in scenarios with poor network signal or no network coverage. Fourth, these solutions do not solve the problem of low-power resident audio stream management in continuous recording scenarios for embedded devices, making them difficult to implement in battery-powered devices such as karaoke microphones.

[0005] Third, the bottleneck problem of wake-up words, which is prevalent in the field of in-vehicle voice interaction, has not been effectively resolved. Existing in-vehicle voice interaction solutions generally adopt a two-stage interaction mode of wake-up word plus voice command. Users need to say the wake-up word and wait for the system response before giving the command. This mode suffers from multi-level latency accumulation: wake-up word detection itself requires a delay of 0.5 to 1 second; recording after successful wake-up requires an audio device initialization delay of 0.2 to 0.5 seconds; plus a 1 to 2 second network delay for cloud recognition, the overall interaction latency reaches more than two seconds. This latency not only affects the user experience in a driving environment but may also distract the driver, posing a safety hazard. Furthermore, wake-up word solutions have a high false wake-up rate in noisy in-vehicle environments; air conditioning noise, music playback, and passenger conversations can all accidentally trigger the voice assistant, further reducing user trust and willingness to use this interaction method.

[0006] Fourth, there is a lack of natural and smooth voice interaction solutions in KTV settings. Existing KTV song selection systems mainly rely on touchscreen search or remote control button operation. Voice song selection is limited by the traditional sequential process of pressing a button, waiting to speak, and waiting for recognition, resulting in a noticeable break in the user experience. It is worth noting that karaoke microphones, as devices that users hold in the hand throughout the KTV setting, naturally possess the potential to serve as a voice interaction entry point. However, existing microphone products lack low-latency button triggering solutions, and the sound pickup advantage of the microphone's built-in microphone is not fully utilized for voice command control.

[0007] Fifth, existing voice input tools generally lack crash protection mechanisms. When the program crashes unexpectedly during recording, all recorded audio data is usually lost, forcing the user to speak the entire audio content again. This problem is particularly serious when recording long audio content, potentially rendering several minutes or even longer of voice input work completely useless.

[0008] In summary, existing technologies have significant technical shortcomings in achieving low-latency, high-reliability, and offline voice input in multi-terminal environments, and there is an urgent need for a unified technical solution to address these issues. Summary of the Invention

[0009] This invention provides a low-latency voice input method and system based on a resident audio stream to solve the problems in the prior art, such as loss of the first character due to microphone cold start delay, high latency and false wake-up rate of wake-up word scheme, network latency and privacy risks due to reliance on cloud recognition, loss of audio data due to program crashes, and long voice input exceeding the model context window limit.

[0010] To achieve the above objectives, the present invention provides a low-latency voice input method based on a persistent audio stream. The method includes: S1. When the voice input system starts, an audio input stream is created and continuously kept active. The audio input stream continuously receives audio data from an audio acquisition device via a callback mode. The lifecycle of the audio input stream is independent of the recording flag and does not change with the start or stop of recording. S2. When the recording flag is off, the audio input stream remains running but does not capture or store the received audio data; the audio data is silently discarded. S3. When a trigger signal is received from a trigger source, the recording flag is set to on to start capturing audio data, and the captured audio data is simultaneously written to a memory buffer and a temporary disk file. S4. When a release signal or a re-trigger signal is received from the trigger source, audio data capture stops, and the buffered audio data is sent to a local offline speech recognition model for recognition to obtain an offline recognition result. S5. The recognition result is transmitted to the application system running on the current terminal, and the application system executes corresponding operation instructions based on the recognition result.

[0011] Optionally, writing the captured audio data to both the memory buffer and the temporary disk file simultaneously includes: at the beginning of recording, in response to receiving the trigger signal, clearing the historical data in the memory buffer and creating the temporary disk file in the operating system's temporary file directory; during the recording process, appending each piece of captured audio data to the end of the dynamic array of the memory buffer by reference, and simultaneously appending the same piece of audio data to the temporary disk file in a streaming manner and performing a disk flushing operation.

[0012] Optionally, it also includes: during the normal end of recording, in response to receiving a release signal or a re-trigger signal from the trigger source, closing the temporary disk file and completing the file header information, and moving the completed temporary disk file to a preset output path.

[0013] Optionally, it also includes: when the program terminates abnormally during recording, the audio data written to the temporary disk file is retained in the temporary file directory of the operating system for use after the voice input system restarts.

[0014] Optionally, the trigger source includes at least one of the following: a global hotkey on the keyboard, a physical button on the vehicle steering wheel, and a function button set on the audio acquisition device itself.

[0015] Optionally, the keyboard global hotkey includes two concurrent modes: a long-press mode and a switching mode. In the long-press mode, audio data capture begins when the user presses the hotkey, and the recording flag remains on while the user holds the hotkey down. When the user releases the hotkey, the recording flag is set to off, thus stopping capture. In the switching mode, audio data capture begins the first time the user presses the hotkey, and the recording flag is set to on and remains on, allowing the user to speak continuously without holding the key down. When the user presses the same hotkey again, the recording flag is set to off, thus stopping capture. Both modes are processed simultaneously through a single global keyboard listener instance. The listener contains event dispatch logic and mutual exclusion logic. The mutual exclusion logic ensures that when one mode is in an active recording state, the triggering event of the other mode is ignored, thereby avoiding recording state conflicts.

[0016] Optionally, a streaming recognition step is included between S3 and S4, and in S4, the offline recognition result is used to replace the streaming recognition result: the streaming recognition step includes: during the recording process, the captured audio data is resampled in real time to the sampling rate and channel number format required by the streaming speech recognition model, and then fed to the streaming speech recognition model in units of fixed-size audio blocks; the streaming speech recognition model achieves contextual continuity across audio blocks by maintaining a key-value cache in the attention mechanism, and whenever an audio block is fed, the streaming speech recognition model returns the current partial recognition result, which is displayed on the user interface in real time; S4 includes: when a release signal or a re-trigger signal is received from the trigger source, the capture of audio data is stopped, and the complete audio file of this recording is sent to the offline speech recognition model for re-recognition to obtain the offline recognition result; the offline recognition result is used to replace the partial recognition result previously displayed on the user interface, and the offline recognition result is output as the final recognition result.

[0017] Optionally, before sending the buffered audio data to the local offline speech recognition model for recognition, an automatic speech activity detection and segmentation step is included. The automatic speech activity detection and segmentation step includes: determining whether the total duration of the recorded audio exceeds a preset duration threshold; if it does not exceed the duration threshold, the complete audio is directly sent to the local offline speech recognition model; if it exceeds the duration threshold, the speech activity detection and segmentation process is initiated. The speech activity detection and segmentation process includes: loading a pre-trained speech activity detection model, and feeding the complete audio frame by frame to the speech activity detection model with a fixed window size. The system obtains the speech probability of each frame, and marks each frame as a speech frame or a silence frame according to a preset confidence threshold. When the duration of consecutive silence frames exceeds the minimum silence threshold, the speech segment is segmented at that position. At the same time, a maximum duration limit constraint is applied to a single speech segment. When no silence interval exceeding the minimum silence threshold appears in the speech segment within the maximum duration limit, the segment is forcibly segmented at the maximum duration limit. Each segmented speech segment is independently sent to the local offline speech recognition model for recognition, and the recognized text is sequentially spliced ​​together according to the original time order of each segment to form a complete final recognition result.

[0018] Optionally, in stage S2, the audio data received by the audio input stream is not discarded, but written to a fixed-capacity circular buffer in a first-in-first-out manner. When the circular buffer is full, the oldest data is overwritten. In stage S3, the existing audio data in the circular buffer is spliced ​​to the audio front end of the current recording.

[0019] On the other hand, the present invention provides a low-latency voice input system based on a resident audio stream. This system includes: a resident audio stream module, used to create and continuously maintain an active audio input stream when the voice input system starts. The audio input stream continuously receives audio data from an audio acquisition device through a callback mode, wherein the lifecycle of the audio input stream is independent of the recording flag and does not change with the start or stop of recording; a silent discard module, used to keep the audio input stream running when the recording flag is off, but not to capture or store the received audio data, which is then silently discarded; a dual-write buffer module, used to start capturing audio data by setting the recording flag to on when a trigger signal from a trigger source is received, and to simultaneously write the captured audio data to a memory buffer and a temporary disk file; a local terminal-side recognition module, used to stop capturing audio data when a release signal or a re-trigger signal from the trigger source is received, and to send the buffered audio data to a local terminal-side offline speech recognition model for recognition to obtain an offline recognition result; and an instruction execution module, used to transmit the recognition result to the application system currently running on the terminal, whereby the application system executes corresponding operation instructions based on the recognition result.

[0020] The beneficial effects of this invention are: This invention provides a low-latency voice input method and system based on a resident audio stream. The method creates and continuously keeps the audio input stream active, decoupling the audio input stream's lifecycle from the recording flag. This simplifies the recording initiation operation to a single Boolean assignment, reducing the initial recording latency from 200-500 milliseconds in existing technologies to less than one microsecond, completely eliminating the first-word swallowing problem in voice input. This invention uses physical buttons or global hotkeys as trigger sources, avoiding the 0.5-1 second wake-up detection latency of existing wake-up word schemes and the problem of false wake-ups in noisy environments. In in-vehicle scenarios, the interaction loop time can be shortened to less than 200 milliseconds. This invention simultaneously writes the captured audio data to a memory buffer and a temporary disk file. This invention employs a dual-write crash safety mechanism, ensuring that audio data already written to disk can be recovered when the program terminates abnormally, thus solving the problem of complete loss of recorded audio data due to program crashes in existing technologies. Furthermore, the invention sends the recognition results to a local offline speech recognition model for identification, eliminating reliance on cloud services, network latency, and privacy risks, and remaining usable even in scenarios without network coverage, such as tunnels and underground parking lots. Moreover, by maintaining a fixed-capacity circular buffer while the recording flag is off, the system already possesses audio data prior to the trigger moment when it receives the trigger signal, achieving a "negative latency" speech capture effect. This completely eliminates the time difference between the user's key press and the start of the voice, ensuring that the first word of the user's voice command is fully captured. Attached Figure Description

[0021] Figure 1 This is a flowchart of a low-latency voice input method based on a resident audio stream provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a low-latency voice input system based on a resident audio stream, provided by an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a flowchart of a low-latency voice input method based on a resident audio stream, provided by an embodiment of the present invention; as shown below. Figure 1 As shown, the method includes steps S1 to S5. The following detailed explanation of each step is provided in conjunction with specific embodiments.

[0024] S1. When the voice input system starts, an audio input stream is created and kept active. The audio input stream continuously receives audio data from the audio acquisition device through a callback mode. The lifecycle of the audio input stream is independent of the recording flag and does not change with the start or stop of the recording state. The core problem this step addresses is the "cold start delay" present in traditional voice input solutions. To understand the innovation of this invention, it is first necessary to analyze why this delay exists in traditional solutions.

[0025] In traditional voice input solutions, the audio acquisition device (i.e., the microphone) is in a switched-off or sleep state most of the time, only being woken up and activated when the user triggers a recording command. This activation process is not a simple switch operation, but involves a series of system calls and hardware initialization steps.

[0026] When a user presses the record button, the operating system needs to perform the following operations in sequence: First, it queries the audio driver for currently available audio input devices, which involves a context switch between user mode and kernel mode; second, it opens the selected audio device, including verifying the device's existence, checking if the device is being used by other processes, and initializing the device handle; third, it negotiates audio parameters with the audio driver, including the sampling rate, number of channels, sampling format, and buffer size; fourth, it allocates a memory buffer for the audio stream; and fifth, it starts the audio data transmission channel. Each of these steps takes time, typically accumulating to two hundred to five hundred milliseconds.

[0027] Even more serious is the time lag between user behavior and system processing. Users are accustomed to starting to speak as soon as they press a button, but the system is not yet ready to receive audio data within 200 to 500 milliseconds, resulting in the loss of the first 0 to 0.5 seconds of the user's speech, which is known in the industry as the "first-word swallowing problem".

[0028] This invention completely overturns the above design concept. The core insight of this invention is: instead of repeatedly performing device initialization operations every time recording is needed, it is better to complete this initialization operation once at system startup, and then keep the audio device in a ready state at all times.

[0029] Specifically, when the voice input system starts up, it immediately performs the initialization operations that traditional solutions only execute when recording is triggered: querying the audio device, opening the device, negotiating parameters, allocating buffers, and establishing a data transmission channel. Once these operations are completed in one go, the audio input stream is created and immediately started, entering a continuously active state. This means that from the moment the system starts up, the microphone is in working order, and audio data begins to flow from the microphone to the system.

[0030] The "callback mode" mentioned in this step is a key technical mechanism for implementing persistent audio streams. To understand the callback mode, it is first necessary to understand the transmission path of audio data from the physical microphone to the application.

[0031] A microphone is a hardware device that converts sound wave signals into digital signals at a fixed frequency (e.g., 44,100 times per second). These digital signals are temporarily stored in a hardware buffer managed by the audio driver. When the hardware buffer is full, the audio driver generates an interrupt signal to notify the operating system that new audio data has arrived.

[0032] In the traditional polling model, the application needs to actively and repeatedly ask the operating system, "Is there any new audio data?" Each query is a system call, and even if there is no new data, the application is idling and consuming CPU resources. If the query frequency is too low, it may lead to data loss; if the query frequency is too high, it will result in a large number of invalid system calls, wasting CPU resources. This model is like a person checking the door every few seconds to see if their takeout has arrived, and even if the takeout hasn't arrived yet, they still have to run over to check repeatedly.

[0033] In the callback mode adopted in this invention, the application's role changes from "actively asking" to "passively waiting." The application registers a callback function with the audio driver, which is like a "delivery address." The application tells the driver, "When you receive new audio data, please actively call this function to send the data." Then, the application can handle other tasks, and the central processing unit doesn't need to idle and wait. When the audio data actually arrives, the driver will actively call the callback function, and the application will process the data within the callback function. This mode is like telling the delivery driver, "Call me when it arrives," and then you can relax and do your own thing; you'll naturally receive a call when the food arrives.

[0034] Another key statement in this step is that "the lifecycle of the audio input stream is independent of the recording flag." In traditional solutions, the audio input stream is tied to the recording state: starting recording means creating and initiating the audio stream, and stopping recording means stopping and destroying the audio stream. This binding relationship means that each switch in recording state requires a complete device initialization or cleanup process.

[0035] This invention completely separates these two concepts. The audio input stream has its own lifecycle—it is created when the system starts and destroyed when the system shuts down, running continuously within this time window. The recording flag is an independent Boolean state variable that only controls whether the audio callback function saves or discards data when it receives it. This decoupling simplifies the overhead of switching recording states from a complex operation involving operating system kernel calls to a simple assignment operation of a user-space memory variable, reducing the time taken from hundreds of milliseconds to sub-microseconds.

[0036] Specific examples (1) Specific examples of vehicle-mounted scenarios Taking Mr. Zhang driving a certain brand of smart car as an example. When Mr. Zhang starts the car engine, the in-vehicle infotainment system is turned on. After the in-vehicle infotainment system is turned on, the voice input system of this invention automatically runs as a background service in the in-vehicle infotainment system.

[0037] When the voice input system of this invention is activated, it immediately detects the microphone array equipped in the vehicle through the audio device interface provided by the vehicle operating system. This microphone array is typically installed in the headliner or near the steering wheel to collect voice commands from the driver and passengers. The voice input system of this invention queries the audio parameters supported by the microphone array through the audio device interface of the vehicle operating system, including a sampling rate of 44,100 times per second and a two-channel stereo format. Subsequently, the voice input system of this invention creates an audio input stream in callback mode, registers a callback function with the audio driver of the vehicle operating system, and starts the audio stream. At this time, the audio driver of the vehicle operating system begins to continuously receive audio data from the microphone array, and whenever the hardware buffer is full, the driver calls the pre-registered callback function of the voice input system of this invention.

[0038] At this time, the recording flag inside the voice input system of this invention is in the off state. Therefore, each time the callback function is called, it checks that the recording flag is false and returns directly without saving the received audio data. The audio data is silently discarded, but the audio input stream itself remains active. Mr. Zhang is completely unaware that an audio stream is running in the background. He drives his car normally, listens to the radio, or plays music, and the resource usage of the in-vehicle infotainment system is extremely low.

[0039] This state will continue throughout the entire driving process. Regardless of whether Mr. Zhang uses the voice interaction function, the vehicle microphone array will always be in a ready state, waiting for the recording flag position to switch. When Mr. Zhang enters an underground parking garage or tunnel, the mobile phone signal will completely disappear, but the resident audio stream mechanism of the voice input system of this invention will not be affected, and the microphone array will still work normally.

[0040] (2) Specific examples of KTV scenarios Taking user Xiao Li singing in a KTV room as an example, when the KTV room's song selection system starts, the song selection operating system completes hardware initialization and system service startup. The voice input system of this invention runs automatically as a built-in module of the song selection system. Simultaneously, after the karaoke microphone system in the room is powered on, the microphone portion of the voice input system of this invention built into it also automatically starts.

[0041] When the karaoke microphone system starts up, its built-in microcontroller executes embedded firmware code and immediately detects the microphone in the microphone head. This microphone is typically only five to ten centimeters away from the user's mouth, offering a natural advantage in near-field sound pickup. The karaoke microphone system creates an audio input stream through its embedded real-time operating system, registers callback functions with the audio driver, and starts the audio stream. The microphone then begins continuously capturing sound.

[0042] At this time, the recording flag of the voice input system of this invention is in a closed state on the karaoke microphone system. Each time the callback function is called, it checks that the recording flag is false and returns directly without saving the audio data. Although the microphone is constantly collecting sound, it does not save any content, nor does it send audio data to the karaoke system via wireless communication. When Xiao Li picks up the microphone to test his voice, speak, and sing, the microphone continuously collects these sounds, but because the recording flag is closed, all sound data is silently discarded within the karaoke microphone system. Xiao Li is completely unaware that an audio stream is running in the background. He sings normally using the microphone, and the resident audio stream of the karaoke microphone system runs stably in a silent state with extremely low power consumption, not affecting the microphone's battery life.

[0043] S2. When the recording flag is off, the audio input stream remains running, but the received audio data is not captured or stored, and the audio data is silently discarded. This step further clarifies the specific manifestation of "lifecycle decoupling" in step S1. Once created, the audio input stream continues to run, unaffected by changes in the recording flag. However, continuous operation does not mean continuous storage. The recording flag acts as a "data valve": when the valve is closed, the flowing audio data is allowed to pass directly without any storage operation; when the valve is open, the audio data is intercepted and stored in a buffer.

[0044] The brilliance of this design lies in the fact that the system requires no preparation when recording begins, because all preparation (device initialization, stream creation, callback registration) is completed during system startup. Starting recording is simply opening a valve that is already in place, and stopping recording is simply closing that valve. The opening and closing of the valve itself incurs almost no time cost.

[0045] From a resource consumption perspective, audio processing overhead in silent mode is extremely low. The callback function simply checks a boolean variable and returns, which is a lightweight operation at the CPU level.

[0046] Specific examples (1) Specific examples of vehicle-mounted scenarios Mr. Zhang was driving on the highway with music playing in the car and talking to his passenger. At the same time, the car's microphone array continuously collected all sounds inside the vehicle, including music, conversation, engine noise, and wind noise. Each time the audio driver collected a piece of data, it called a callback function. The callback function checked the recording flag and, if it was false, returned immediately. All this audio data was silently discarded, without occupying the vehicle's memory or being written to storage devices.

[0047] Mr. Zhang was completely unaware that any audio processing was taking place in the background. He could drive, chat, and listen to music normally. The voice input system ran stably in silent mode without interfering with any other in-vehicle functions.

[0048] (2) Specific examples of KTV scenarios Xiao Li and his friends were singing in a karaoke room. The background music was very loud, and his friends were singing a fast-paced song together. Xiao Li was holding a microphone, which was continuously recording sound, including the background music, his own singing, and his friends' singing. However, because the microphone system's recording flag was off, all the recorded sound data was silently discarded.

[0049] At this time, the karaoke system's resident audio stream is also running, and the ambient microphone on the ceiling of the room continuously collects sound from the entire room. Again, because the recording flag is off, all data is discarded. The entire system runs in silent mode, waiting for the user's trigger command.

[0050] In an optional implementation, in the S2 stage, the audio data received by the audio input stream is not silently discarded, but written to a fixed-capacity circular buffer in a first-in-first-out manner. When the circular buffer is full, the newly arrived audio data overwrites the oldest data, so that the circular buffer always retains the audio content of the most recent period. The capacity of the circular buffer is set according to actual needs. In a specific embodiment of the present invention, the capacity of the circular buffer is set to store the audio data of the most recent two seconds. With a sampling rate of 44,100 times per second, dual channels, and four bytes per sampling point, two seconds of audio occupies about 705,600 bytes of memory space, which is negligible for the memory resources of modern terminal devices. When a trigger signal is received and the S3 stage is entered, the voice input system of the present invention first exports the existing audio data in the circular buffer and splices it to the audio front end of the current recording, and then executes the normal recording process described in S3. This mechanism enables the present invention to capture the audio content before the user trigger time, realizing the "negative delay" voice capture effect. Specific examples (1) Specific examples of vehicle scenarios Old Zhang wanted to listen to a song while driving, so he pressed the voice button on the steering wheel and said, "Let's have a song A." Due to the user's habits, Old Zhang's mouth started to make the sound "lai" the moment his finger touched the button. In a scheme without a circular buffer, the first tens of milliseconds of the sound "lai" may be lost due to the time difference between the button action and the system response. In this embodiment, the circular buffer continuously retains the audio data of the most recent two seconds. When Old Zhang presses the button, the system splices the existing audio in the circular buffer (including the beginning part of the sound "lai") to the recording front end to ensure that every word of "Let's have a song A" is captured completely. (2) Specific examples of KTV scenarios Xiao Li picked up the microphone, pressed the voice function button, and said, "Pick a song B." Xiao Li's mouth began to speak the moment he pressed the button. The microphone system's circular buffer continuously retained the near-field audio data from the most recent two seconds. When the button was triggered, the audio in the buffer (including the beginning of the word "point") was spliced ​​to the recording front end to ensure the instruction was complete.

[0051] S3. When a trigger signal is received from the trigger source, the recording flag is set to the on state to start capturing audio data, and the captured audio data is written to the memory buffer and the disk temporary file at the same time. This step is the core of the zero-latency recording achieved by this invention. When the user issues a command to start recording in some way, the voice input system of this invention responds to this trigger signal and sets the recording flag maintained internally by the voice input system from false to true. This setting operation only involves writing to an address in the memory of the terminal device where the voice input system is located, and takes much less than one microsecond. From this moment on, subsequent callback functions called by the audio driver, upon receiving audio data, will check that the recording flag is true and then perform a data saving operation.

[0052] Unlike the traditional "initialize first, then record" mode, this invention adopts a "initialized, just start" mode. Since the audio stream has already been initialized and is running continuously in step S1, the microphone is already collecting sound, and the callback function is already being called; therefore, there is no waiting time for triggering the response. The voice input system of this invention is already saving audio data the moment the user presses the button.

[0053] This step also introduces a "dual-write mechanism"—the voice input system of this invention simultaneously writes the captured audio data to two different storage locations. The first location is a memory buffer in the memory of the terminal device where the voice input system is located, used for subsequent fast access and real-time processing; the second location is a temporary disk file in the storage medium of the terminal device where the voice input system is located, used for persistent storage. This dual-write strategy lays the foundation for subsequent crash safety mechanisms.

[0054] In one optional implementation, the trigger source includes at least one of a global keyboard hotkey, a physical button on the vehicle steering wheel, and a function button on the audio acquisition device itself.

[0055] The purpose of this design is to enable the invention to adapt to various terminal forms. In a desktop scenario, the most convenient triggering method for users is keyboard hotkeys, as their hands are usually already on the keyboard. In a vehicle scenario, physical buttons on the steering wheel are the best choice, because the driver can operate the device while holding the steering wheel without taking their eyes off the road. In a karaoke microphone scenario, function buttons on the microphone itself are the most natural choice, as the user's thumb can easily reach the buttons when holding the microphone.

[0056] In one optional implementation, the keyboard global hotkeys include two concurrent modes: a long press mode and a switching mode. In the long-press mode, audio data capture begins when the user presses the hotkey. The recording flag remains on while the user holds the hotkey down. When the user releases the hotkey, the recording flag is set to off, thus stopping the capture. The characteristic of this mode is that the recording duration is entirely determined by the duration the user holds the button down, making it suitable for short voice command inputs, such as short commands like "next song," "copy," and "paste."

[0057] In the switching mode, audio data capture begins when the user presses the hotkey for the first time, and the recording flag is set to the on state and maintained. The user can continue speaking without holding down the button. When the user presses the same hotkey again, the recording flag is set to the off state, thus stopping the capture. The characteristic of this mode is that the recording duration is determined by the time interval between the first press and the second press, which is suitable for long voice content input, such as dictating an article or recording a meeting transcript.

[0058] Both modes are handled simultaneously through a single global keyboard listener instance. This listener contains event dispatch logic and mutual exclusion logic. The event dispatch logic identifies which hotkey a received keyboard event belongs to; the mutual exclusion logic ensures that when one mode is actively recording, the triggering event of the other mode is ignored, thus avoiding recording state conflicts. For example, if a user is recording in long-press mode (the button is not yet released), and the user accidentally presses the mode switching hotkey, the system will ignore this accidental press event and will not interrupt the ongoing recording.

[0059] In an optional implementation, writing the captured audio data simultaneously to a memory buffer and a temporary disk file includes: At the start of recording, in response to receiving the trigger signal, the historical data in the memory buffer is cleared, and the temporary disk file is created in the operating system's temporary file directory; During the recording phase, each captured audio data block is appended to the end of the dynamic array of the memory buffer by reference, and the same audio data block is appended to the temporary disk file by streaming and a disk flushing operation is performed.

[0060] Specifically, at the start of recording, in response to receiving a trigger signal, the voice input system of this invention first clears the historical data in the memory buffer of the terminal device where the voice input system is located. This is to ensure that the current recording starts from a blank state and does not mix in residual data from the previous recording. Subsequently, the voice input system of this invention creates a temporary disk file in the temporary file directory of the operating system of the terminal device, usually named temp.wav or a similar temporary filename. At this time, the file has been created, but the metadata in the file header (such as data length) has not yet been filled in, because the voice input system of this invention does not yet know the total duration of the current recording.

[0061] During the recording phase, the voice input system of this invention appends each captured audio data block to the end of a dynamic array in the memory buffer using a reference method. The "reference method" is used instead of a "data copy method" because the dynamic array stores pointers to audio data blocks, not complete copies of the data blocks themselves. This method avoids unnecessary data copying and improves the efficiency of memory operations. Simultaneously, the voice input system of this invention appends the same audio data block to a temporary disk file using a streaming write method and immediately performs a flush operation. The flush operation forces the data to be written from the file cache of the terminal device's operating system to the physical storage medium, ensuring that even if the voice input system of this invention or the terminal device's operating system crashes immediately, the written data will not be lost.

[0062] In an optional implementation, the method further includes: During the normal recording end phase, in response to receiving the release signal or re-trigger signal from the trigger source, the temporary disk file is closed and the file header information is completed, and the completed temporary disk file is moved to the preset output path.

[0063] Specifically, when the voice input system of this invention receives a release signal or a re-trigger signal from the trigger source, the voice input system of this invention first closes the temporary disk file. This closing operation triggers the file system of the operating system of the terminal device where the voice input system is located to complete all incomplete write operations. Subsequently, the voice input system of this invention needs to complete the file header metadata. The temporary file created at the start of recording lacks some crucial information, such as the actual length of the audio data. Now that recording is complete, the voice input system of this invention knows the total data length and can therefore correctly fill in the relevant fields in the file header. Finally, the voice input system of this invention moves the completed temporary disk file to a preset output path for subsequent recognition.

[0064] In an optional implementation, the method further includes: If the program terminates abnormally during recording, the audio data already written to the temporary file on the disk is retained in the temporary file directory of the operating system for use after the voice input system restarts.

[0065] Specifically, if the voice input system of this invention crashes unexpectedly during recording (e.g., an error occurs in the operating system of the terminal device where the voice input system is located, the terminal device has insufficient memory, or the terminal device experiences a power failure), the audio data already written to the temporary disk file will be retained in the temporary file directory of the terminal device's operating system. This is because the dual-write mechanism continuously flushes data to the physical storage medium during the recording phase, and this data will not be lost due to the crash of the voice input system of this invention.

[0066] When a user restarts the voice input system of this invention, the system detects that there are improperly closed temporary files in the temporary file directory of the terminal device's operating system. The system can then prompt the user that a previously incomplete recording has been found and ask if they wish to resume it. Upon user confirmation, the system can move the temporary file to a preset output path for subsequent recognition. Thus, even if the voice input system crashes, most of the audio data already recorded by the user can be retained, eliminating the need to repeat what has already been said.

[0067] Specific examples (1) Specific examples of vehicle-mounted scenarios Mr. Zhang was driving through the city streets when he wanted to listen to a song called "A" by a certain singer. Unlike traditional wake-up word schemes where you first say "Hello Xiao Ai" and wait for the in-car voice assistant system to respond, Mr. Zhang only needed to press the physical voice button on the steering wheel.

[0068] When Mr. Zhang presses the voice button on the steering wheel, the electrical signal generated by the button is transmitted to the processing unit running the voice input system of this invention via the vehicle's infotainment system's human-machine interface protocol or controller area network bus. The button event processing module of the vehicle's infotainment system forwards the event to the trigger monitoring interface of the voice input system of this invention. Upon receiving the button press event, the voice input system of this invention immediately invokes the start recording method of its internal audio recorder.

[0069] The voice input system of this invention first clears the historical data in the memory buffer of the in-vehicle infotainment system to ensure that the recording starts from a blank state. Then, the voice input system creates a temporary audio file named voice_cmd_20250115_143022.tmp in the temporary file directory of the in-vehicle operating system. Finally, the voice input system sets its internal recording flag from false to true. The total time for these three operations is less than one microsecond.

[0070] As Lao Zhang pressed the button, he said, "Next song by a certain singer." Throughout the process of Lao Zhang uttering the command, the microphone array of the in-vehicle infotainment system continuously collects sound, and the pre-registered audio callback function of the voice input system of this invention is continuously called. Each time a piece of audio data is received, the voice input system of this invention performs a deep copy operation, appending the data block to the end of the dynamic array of the memory buffer in the in-vehicle infotainment system's memory using a reference. Simultaneously, the voice input system of this invention appends the same piece of audio data to a temporary disk file in the in-vehicle infotainment system's storage medium using a streaming write method, and immediately performs a flush operation to ensure that the data is actually written to the in-vehicle storage medium.

[0071] Assuming Mr. Zhang releases the steering wheel button normally after giving the command, the voice input system of this invention will complete the normal termination process. The voice input system of this invention closes the temporary disk file, completes the header information required by the waveform audio file format, and then moves the file to the preset output path for use by the local offline recognition module of the voice input system of this invention.

[0072] In another scenario, while Mr. Zhang is giving a command, the vehicle suddenly experiences a large bump, causing the in-vehicle infotainment system to unexpectedly restart. Upon restarting, the voice input system of this invention also restarts, detecting any improperly closed temporary files in the vehicle's operating system's temporary file directory. The voice input system can then display a message on the in-vehicle infotainment system screen: "A previously incomplete voice command has been detected. Restore?" After Mr. Zhang confirms, the voice input system can restore the recorded file and move it to a preset output path for recognition.

[0073] (2) Specific examples of KTV scenarios Xiao Li was singing with his friends in a karaoke room, and the background music was very loud. Xiao Li wanted to sing a song he was good at. He picked up the microphone in front of him and pressed the dedicated voice function button on the karaoke microphone system.

[0074] The key detection module of the karaoke microphone system detects a key press and generates a trigger signal. This trigger signal is transmitted to the wireless communication receiving module of the karaoke system via the Bluetooth human-machine interface device protocol. The key event processing module of the karaoke system forwards this event to the trigger monitoring interface of the voice input system (karaoke system side) of this invention. After receiving the key press event, the voice input system (karaoke system side) of this invention immediately sends a start recording command to the karaoke microphone system via wireless communication.

[0075] After receiving the start recording command, the karaoke microphone system first clears the historical data in its memory buffer. Then, it creates a temporary audio file in its internal flash memory's temporary file directory. Finally, it sets its internal recording flag from false to true. The total time for these three operations is less than one microsecond.

[0076] As Xiao Li pressed the button, he said, "Play a song B." The built-in microphone of the karaoke microphone system is only about five centimeters away from Xiao Li's mouth, which is a typical near-field sound pickup. In the environment of high-decibel background music in the private room, the signal-to-noise ratio of near-field sound pickup is significantly better than that of the ambient microphone in the karaoke system. Xiao Li's voice signal strength is much higher than that of the background music, so the quality of the captured audio is very high. The audio callback function of the karaoke microphone system is continuously called, and each piece of audio data is simultaneously written to the memory buffer in the karaoke microphone system's memory and to a temporary disk file in the karaoke microphone system's flash memory.

[0077] Meanwhile, the karaoke microphone system transmits the collected audio data to the karaoke system in real time via Bluetooth wireless communication. After receiving the audio data, the karaoke system's voice input system (karaoke system end) simultaneously writes the received audio data to both the memory buffer in the karaoke system's memory and a temporary disk file in the karaoke system's storage medium, achieving dual-write storage on the karaoke system end.

[0078] After Xiao Li finished speaking and released the voice function button on the karaoke microphone system, the system detected the button release, stopped capturing audio data, and moved the temporary file in the system's flash memory to the output directory after completing the file header information. Simultaneously, the system sent a recording end command to the song selection system and completed the transmission of the last audio segment.

[0079] S4. When a release signal or a re-trigger signal is received from the trigger source, stop capturing audio data and send the buffered audio data to the local offline speech recognition model for recognition to obtain the offline recognition result. This step is responsible for converting the captured audio data into text or commands. When the user finishes recording, the system first sets the recording flag to false, stopping the capture of new audio data. Then, the system sends the complete audio data obtained from this recording to the local offline speech recognition model for recognition.

[0080] "Local device-side" means the speech recognition model runs on the user's local terminal device, not a cloud server. Audio data doesn't need to be uploaded to the internet; all computation is done locally. This brings two significant advantages: first, it eliminates network transmission latency, allowing for faster return of recognition results; second, it protects user privacy, ensuring sensitive speech content never leaves the user's device. "Offline" means the recognition process is completely independent of a network connection and can function normally even in environments without mobile network signals (such as underground parking garages or tunnels). This is particularly important for in-vehicle scenarios.

[0081] In an optional implementation, a streaming identification step is further included between S3 and S4, and in S4, the streaming identification result is replaced with the offline identification result: The streaming recognition steps include: during recording, resampling the captured audio data in real time to the sampling rate and channel number format required by the streaming speech recognition model, and then feeding it to the streaming speech recognition model in units of fixed-size audio blocks; the streaming speech recognition model achieves contextual continuity across audio blocks by maintaining a key-value cache in the attention mechanism; whenever an audio block is fed, the streaming speech recognition model returns the current partial recognition result, which is displayed on the user interface in real time; S4 includes: when a release signal or a re-trigger signal is received from the trigger source, stopping the capture of audio data, and sending the complete audio file of this recording to an offline speech recognition model for re-recognition to obtain an offline recognition result; replacing the part of the recognition result previously displayed on the user interface with the offline recognition result, and outputting the offline recognition result as the final recognition result.

[0082] The streaming recognition process is performed in real time while the user is speaking. As the user utters a voice command, the system does not wait for the user to finish speaking; instead, it captures and recognizes the information simultaneously, displaying some of the recognition results in real time so the user can see the system's response promptly.

[0083] Specifically, the streaming identification process includes the following sub-steps.

[0084] First, resampling processing. The voice input system of this invention resamples the captured audio data in real time to the sampling rate and channel number format required by streaming speech recognition models. Most microphones capture raw audio at 44,100 samples per second in a two-channel stereo format, while streaming speech recognition models typically require 16,000 samples per second in a mono format. The resampling process includes: mixing the two-channel data into mono by averaging, and then using a signal processing algorithm to reduce the sampling rate from 44,100 samples per second to 16,000 samples per second.

[0085] Second, segmented feeding. After resampling, the speech input system of this invention feeds audio blocks of a fixed size to the streaming speech recognition model. This fixed size is typically set to 9,600 sampling points, corresponding to 0.6 seconds of audio at a sampling rate of 16,000 samples per second. The reason for choosing 0.6 seconds is that this duration is sufficient for the model to extract meaningful speech features, while also being short enough to ensure smooth real-time feedback.

[0086] Third, contextual continuity maintenance. The streaming speech recognition model achieves contextual continuity across audio blocks by maintaining a key-value cache in the attention mechanism. Specifically, when processing the current audio block, the model saves the attention key-value pairs generated in that block. When processing the next audio block, the model loads the previously saved key-value pairs, thus "remembering" previously recognized content. This mechanism avoids recognition breaks caused by block boundary segmentation. For example, when a user says "The weather is nice today," the first audio block might only contain the speech signal of "today," while the second audio block contains the speech signal of "The weather is nice." Without contextual continuity, the model might independently recognize the two blocks as "today" and "The weather is nice," leading to "weather" being misrecognized as "climate." With contextual continuity, the model knows that "today" has already been recognized when processing the second block, thus correctly recognizing the subsequent content as "The weather is nice."

[0087] Fourth, real-time result display. Each time an audio block is fed in, the streaming speech recognition model returns a partial recognition result. This result may be incomplete—for example, when the user has just said "today," the model may only return the two characters "today" because the rest of the content has not yet been processed. The voice input system of this invention displays this partial recognition result on the user interface in real time, allowing the user to see preliminary text feedback while speaking.

[0088] After the user finishes recording, step S4 performs offline recognition and replaces the streaming results.

[0089] First, stop capturing. When the voice input system of the present invention receives a release signal from the trigger source (the user releases the button in long-press mode) or a re-trigger signal (the user presses the same button again in switch mode), the voice input system of the present invention sets the recording flag to false and stops capturing new audio data.

[0090] Second, offline recognition. The voice input system of this invention sends the complete audio file recorded this time (i.e., the temporary disk file saved by the S3 double-write mechanism) to an offline speech recognition model for re-recognition to obtain offline recognition results. Unlike streaming recognition models, offline recognition models can access the complete audio context at once, including the context of the entire sentence, complete intonation changes, and pause patterns. Therefore, offline recognition results usually have higher accuracy than streaming recognition results.

[0091] Third, result replacement. After obtaining the offline recognition result, the voice input system of the present invention replaces the portion of the recognition result previously displayed on the user interface with the offline recognition result. That is, the text that may have contained a few errors and was originally displayed in real time on the user interface is instantly replaced with the accurate offline recognition result.

[0092] Fourth, final output. The voice input system of the present invention outputs the offline recognition result as the final recognition result and passes it to the subsequent instruction execution module.

[0093] In an optional implementation, before sending the buffered audio data into the local on-device offline speech recognition model for recognition, an automatic segmentation step based on speech activity detection is also included. The automatic segmentation step for voice activity detection includes: Determine whether the total duration of the recorded audio exceeds a preset duration threshold. If it does not exceed the duration threshold, the complete audio is directly sent to the local offline speech recognition model. If it exceeds the duration threshold, the speech activity detection and segmentation process is started. The speech activity detection segmentation process includes: loading a pre-trained speech activity detection model; feeding the complete audio frame by frame into the speech activity detection model with a fixed window size to obtain the speech probability of each frame; marking each frame as a speech frame or a silence frame according to a preset confidence threshold; segmenting the speech into a segment at the position when the duration of consecutive silence frames exceeds the minimum silence threshold; applying a maximum duration limit constraint to a single speech segment; forcibly segmenting the speech segment at the maximum duration limit when no silence interval exceeding the minimum silence threshold occurs within the maximum duration limit; independently feeding each segmented speech segment into the local offline speech recognition model for recognition; and sequentially concatenating the recognized text according to the original time order of each segment to form a complete final recognition result.

[0094] Specifically, offline speech recognition models with large language model architectures on the device side typically have context window limitations. If the input audio is too long, the key-value cache inside the model may overflow, leading to recognition failure or program crash. Therefore, it is necessary to intelligently segment long audio files exceeding a certain duration into several shorter segments, recognize each segment separately, and then concatenate the results.

[0095] This step uses a speech activity detection model to identify natural pauses in human speech, segments audio at these locations, and sets a maximum duration limit as a protection mechanism.

[0096] The automatic segmentation process for voice activity detection first determines whether the total duration of the recorded audio exceeds a preset duration threshold.

[0097] The voice input system of this invention obtains the total duration of the recorded audio file and compares this duration with a preset duration threshold. In one specific embodiment of this invention, the duration threshold is set to fifteen seconds. This value is set based on the upper limit of the key-value cache capacity of the offline speech recognition model based on the edge-side large language model architecture. After extensive experimental testing, audio within fifteen seconds can be safely processed on most edge-side large language model architecture offline speech recognition models without causing cache overflow.

[0098] If the total duration of the recorded audio does not exceed fifteen seconds, the voice input system of this invention determines that the audio is short enough to be processed by the local offline speech recognition model in one go without exceeding the model's context window limit. In this case, the voice input system of this invention skips the segmentation step and directly sends the complete audio to the local offline speech recognition model for recognition.

[0099] If the total duration of the recorded audio exceeds fifteen seconds, the voice input system of this invention will initiate the voice activity detection and segmentation process.

[0100] The speech activity detection and segmentation process includes the following sub-steps: First sub-step: Load the pre-trained speech activity detection model; The voice input system of this invention loads a pre-trained voice activity detection model. In one specific embodiment of this invention, the voice input system loads the Silero voice activity detection model. This model is encapsulated in the Open Neural Network Exchange (ONE) format and executed on the local device via the ONE runtime inference engine. This model is characterized by its small size, fast inference speed, and real-time operation without the need for graphics processor acceleration, making it highly suitable for deployment on terminal devices such as in-vehicle infotainment systems, karaoke systems, and personal computers.

[0101] The second sub-step: Analyze the audio frame by frame to obtain the speech probability; The voice input system of this invention feeds complete audio to the voice activity detection model frame by frame with a fixed window size to obtain the voice probability of each frame. In a specific embodiment of this invention, the window size is set to 512 sampling points, corresponding to 32 milliseconds at a sampling rate of 16,000 samples per second.

[0102] The speech activity detection model processes each frame of audio and outputs a speech probability value between zero and one. This probability value represents the confidence level that human voices are present within that frame's time period. For example, a probability value of 0.95 means that there is a 95% probability that the frame contains human voices; a probability value of 0.05 means that there is a 95% probability that the frame is silent or contains background noise.

[0103] Third sub-step: Label speech frames and silence frames according to the confidence threshold; The voice input system of this invention performs binarization judgment on the speech probability of consecutive frames based on a preset confidence threshold. In a specific embodiment of this invention, the confidence threshold is set to 0.5. Frames with a probability higher than 0.5 are marked as "speech frames," indicating that human voice is present in the frame; frames with a probability lower than 0.5 are marked as "silent frames," indicating that no human voice is detected in the frame.

[0104] Fourth sub-step: Detect silence intervals and segment speech segments; The voice input system of the present invention detects silence intervals in a continuous sequence of voice frames. Specifically, the system analyzes a continuous sequence marked as silence frames and calculates the duration of each continuous silence frame sequence. When the duration of a continuous silence frame exceeds a minimum silence threshold, the voice input system of the present invention segments the voice at that silence position.

[0105] In one specific embodiment of the invention, the minimum silence threshold is set to 0.3 seconds. This value is based on the analysis of a large amount of speech data: at normal speaking speed, the natural pauses between words are usually shorter than 0.3 seconds, while the natural pauses between sentences are usually longer than 0.3 seconds. Therefore, the 0.3-second threshold can effectively distinguish between pauses between words and pauses between sentences, avoiding mis-segmentation between words. For example, when a user says "The weather is really nice today," the pause between "today" and "weather" is usually much shorter than 0.3 seconds and will not be segmented; while the pause at the end of the sentence after "The weather is really nice" is usually longer than 0.3 seconds and will be identified as the end of a paragraph.

[0106] Fifth sub-step: Apply a maximum duration limit constraint; The voice input system of the present invention simultaneously imposes a maximum duration constraint on a single voice segment. In one specific embodiment of the present invention, the maximum duration limit is set to fifteen seconds.

[0107] The purpose of this constraint is to force a segmentation at the 15-second mark if a speech segment does not have a sufficiently long silence interval (i.e., the user speaks continuously for more than 15 seconds without pausing). This forced segmentation mechanism is a protective measure to prevent excessively long single speech segments from causing key-value buffer overflows in the automatic speech recognition model of the large language model architecture on the device side. For example, if a user reads a 20-second passage continuously without pausing, the system will force a segmentation at the 15-second mark: the first segment is the first 15 seconds, and the second segment is the last 5 seconds.

[0108] The sixth sub-step: After each paragraph is identified independently, the results are spliced ​​together.

[0109] After segmentation, the voice input system of this invention independently sends each segmented voice segment to the local offline speech recognition model for recognition. Each segment generates its own recognized text. Since the length of each segment is within fifteen seconds, which is within the model's context window limit, each segment can be recognized normally.

[0110] After recognition is completed, the voice input system of the present invention splices the recognized text sequentially according to the original time sequence of each paragraph to form a complete final recognition result.

[0111] Specific examples illustrate: (1) Specific examples of vehicle-mounted scenarios After saying "Next song by [singer's name]", Lao Zhang released the steering wheel button. The system detected the button release event, immediately set the recording flag to false, and stopped capturing new audio data.

[0112] During the recording process, the system performed streaming recognition in real time. Every 0.6 seconds, the system sent a piece of audio to the streaming recognition model. When Mr. Zhang said "next song," the streaming model had already recognized "next song" and displayed it in the corner of the car's screen; when he continued to say "singer," the screen updated to "next song by singer"; when he finally said "song," the screen displayed "next song by singer." Although streaming recognition may have brief uncertainties, Mr. Zhang could see that the system was responding in real time.

[0113] After recording, the system sends the complete audio file to the offline speech recognition model for re-recognition. Using the complete audio context, the offline model accurately confirmed the recognition result as "the next song by a certain artist." The system replaces the streaming result with this offline result and then transmits the final instruction to the in-vehicle music system.

[0114] The entire interaction loop, from when Mr. Zhang releases the button to when the music starts playing, takes less than 200 milliseconds. In contrast, traditional wake-up word solutions require saying "Hello Xiao Ai," waiting for 0.5 to 1 second of wake-up detection, then saying the command, and then waiting for cloud recognition, resulting in a total delay of more than one to two seconds. The solution of this invention significantly improves the smoothness of in-vehicle voice interaction.

[0115] (2) Specific examples of KTV scenarios After saying "Play a song B," Xiao Li released the voice function button on the microphone. The system stopped capturing audio and sent the audio data to the recognition module.

[0116] The built-in microphone captured very high-quality audio because near-field pickup effectively suppressed background music. During recording, streaming recognition displayed partial results for "Play a song B" in real time. After recording, the offline recognition model re-identified the entire audio, confirming the accuracy of the results.

[0117] The system transmits the recognition result "Play a song B" to the song search module of the karaoke system. The karaoke system immediately searches its song library, finds song B, and adds it to the playback queue. The song information is displayed on the large screen, and the accompaniment music begins to play. Xiao Li picks up the microphone and begins to sing the song he requested.

[0118] Suppose Xiao Li gives a longer instruction, such as "Please select song B, then song C, and finally song D," which might take over fifteen seconds to record. In this case, the automatic segmentation step using voice activity detection comes into play. The system detects consecutive silence frames exceeding 0.3 seconds and divides the instruction into three independent segments: "Please select song B," "Then select song C," and "Finally select song D." Each segment is recognized separately and then concatenated to form a complete recognition result. Based on this, the song selection system adds the three songs to the playback queue sequentially.

[0119] S5. The recognition result is transmitted to the application system running on the current terminal, and the application system executes the corresponding operation instructions based on the recognition result.

[0120] This step is where the invention connects to specific application scenarios. The identified result is merely a command string; it needs to be passed to the upper-layer application system running on the current terminal to generate real value. The content of the operation command varies depending on the terminal type and the application scenario.

[0121] In an in-vehicle scenario, the voice input system of this invention transmits the recognition results to the in-vehicle application system currently running on the terminal through the application interface of the vehicle system, such as a music playback application system, a navigation application system, an air conditioning control application system, and a telephone application system. After receiving the recognition results, these in-vehicle application systems execute corresponding operation commands according to their internal functional modules (such as a song switching module, a route planning module, and a temperature adjustment module).

[0122] In a KTV microphone scenario, the voice input system (song selection system terminal) of this invention transmits the recognition result to the KTV application system running on the current terminal, such as a song selection application system. After receiving the recognition result, the song selection application system executes corresponding operation instructions based on the recognition result through its internal functional modules (such as song search module, playback control module, and volume adjustment module).

[0123] Specific application examples (1) Specific examples of vehicle-mounted scenarios After Mr. Zhang's voice command "Next song by [singer's name]" was recognized by the voice input system of this invention, the system transmitted the command to the in-vehicle application system running on the current terminal, specifically the music playback application system, through the application interface of the in-vehicle infotainment system. Upon receiving the command, the music playback application system's internal playback control module performed the following operations: stopped the currently playing song, searched for the song "A" sung by the specified artist using its internal song search module, and then began playing the song. The in-vehicle infotainment system screen simultaneously displayed the song title, artist information, and album art. The entire process was smooth and natural; Mr. Zhang's hands never left the steering wheel, and his gaze remained fixed on the road ahead.

[0124] If Mr. Zhang says "Navigate to the company," the voice input system of this invention will transmit the recognition result to the navigation application system. After receiving the instruction, the navigation application system will automatically plan the best route from the current location to the company through its internal route planning module, and the navigation guidance module will begin navigation.

[0125] If Mr. Zhang says "Set the air conditioner to 22 degrees," the voice input system of this invention will transmit the recognition result to the air conditioning control application system. After receiving the instruction, the air conditioning control application system will automatically adjust the interior temperature to 22 degrees Celsius using its internal temperature regulation module.

[0126] (2) Specific examples of KTV scenarios After Xiao Li's voice command "Play a song B" is recognized by the voice input system (song selection system terminal) of this invention, the voice input system (song selection system terminal) transmits the command to the KTV application system currently running on the terminal through the application interface of the song selection system, specifically the song selection application system. Upon receiving the command, the song selection application system uses its internal song search module to search for "song B" in its local song library. Once found, its internal playlist management module adds the song to the end of the playback queue. The song selection interface on the large screen of the song selection system displays "Added: Song B," and simultaneously, the voice prompt module of the song selection system issues a notification: "Song B has been added to the playlist."

[0127] If Xiao Li says "cut off", the voice input system (song selection system end) of this invention will transmit the recognition result to the song selection application system, and its internal playback control module will immediately stop the currently playing song and switch to the next song in the playback queue.

[0128] If Xiao Li says "louder", the voice input system (song selection system end) of this invention will transmit the recognition result to the song selection application system, and its internal volume control module will adjust the volume of the accompaniment track and the vocal track.

[0129] As a device that users always hold in a KTV setting, the karaoke microphone system, combined with the resident audio stream and button triggering mechanism of the voice input system of this invention, realizes a natural interaction method where "the microphone is the interaction entry point." Users do not need to put down the microphone to operate the touch screen or look for the remote control; they can simply press the buttons on the karaoke microphone system and speak commands to complete operations such as selecting songs, changing songs, and adjusting the volume.

[0130] Figure 2 This is a schematic diagram of a low-latency voice input system based on a resident audio stream, provided by an embodiment of the present invention; as shown below. Figure 2 As shown, the system includes: The resident audio stream module 201 is used to create and keep the audio input stream active when the voice input system starts. The audio input stream continuously receives audio data from the audio acquisition device through a callback mode. The lifecycle of the audio input stream is independent of the recording flag and does not change with the start or stop of the recording state. The silent discard module 202 is used to keep the audio input stream running when the recording flag is in the off state, but not to capture and store the received audio data, and the audio data is silently discarded. The dual-write buffer module 203 is used to start capturing audio data by setting the recording flag to the on state when a trigger signal is received from the trigger source, and to write the captured audio data to the memory buffer and the disk temporary file at the same time. The local end-side recognition module 204 is used to stop capturing audio data when it receives a release signal or a re-trigger signal from the trigger source, and send the buffered audio data into the local end-side offline speech recognition model for recognition to obtain offline recognition results. The instruction execution module 205 is used to transmit the recognition result to the application system running on the current terminal, and the application system executes the corresponding operation instruction according to the recognition result.

[0131] In an optional implementation, writing the captured audio data simultaneously to a memory buffer and a temporary disk file includes: At the start of recording, in response to receiving the trigger signal, the historical data in the memory buffer is cleared, and the temporary disk file is created in the operating system's temporary file directory; During the recording phase, each captured audio data block is appended to the end of the dynamic array of the memory buffer by reference, and the same audio data block is appended to the temporary disk file by streaming and a disk flushing operation is performed.

[0132] The system of the present invention corresponds to the method described above, and the specific implementation of the system will not be repeated here.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-latency voice input method based on a resident audio stream, characterized in that, include: S1. When the voice input system starts, an audio input stream is created and kept active. The audio input stream continuously receives audio data from the audio acquisition device through a callback mode. The lifecycle of the audio input stream is independent of the recording flag and does not change with the start or stop of the recording state. S2. When the recording flag is off, the audio input stream remains running, but the received audio data is not captured or stored, and the audio data is silently discarded. S3. When a trigger signal is received from the trigger source, the recording flag is set to the on state to start capturing audio data, and the captured audio data is written to the memory buffer and the disk temporary file at the same time. S4. When a release signal or a re-trigger signal is received from the trigger source, stop capturing audio data and send the buffered audio data to the local terminal-side offline speech recognition model for recognition to obtain the offline recognition result. S5. The recognition result is transmitted to the application system running on the current terminal, and the application system executes the corresponding operation instructions based on the recognition result.

2. The method according to claim 1, characterized in that, The step of simultaneously writing the captured audio data to a memory buffer and a temporary disk file includes: At the start of recording, in response to receiving the trigger signal, the historical data in the memory buffer is cleared, and the temporary disk file is created in the operating system's temporary file directory; During the recording phase, each captured audio data block is appended to the end of the dynamic array of the memory buffer by reference, and the same audio data block is appended to the temporary disk file by streaming and a disk flushing operation is performed.

3. The method according to claim 2, characterized in that, Also includes: During the normal recording end phase, in response to receiving the release signal or re-trigger signal from the trigger source, the temporary disk file is closed and the file header information is completed, and the completed temporary disk file is moved to the preset output path.

4. The method according to claim 2, characterized in that, Also includes: If the program terminates abnormally during recording, the audio data already written to the temporary file on the disk is retained in the temporary file directory of the operating system for use after the voice input system restarts.

5. The method according to claim 1, characterized in that: The trigger source includes at least one of the following: a global hotkey on the keyboard, a physical button on the vehicle steering wheel, and a function button on the audio acquisition device itself.

6. The method according to claim 5, characterized in that: The keyboard global hotkeys include two concurrent modes: long press mode and switching mode. In the long press mode, when the user presses the hotkey, audio data capture begins. While the user continues to press and hold the hotkey, the recording flag remains on. When the user releases the hotkey, the recording flag is set to off, thus stopping the capture. In the switching mode, when the user presses the hotkey for the first time, audio data capture begins, the recording flag is set to the on state and maintained, and the user can continue speaking without holding down the button. When the user presses the same hotkey again, the recording flag is set to the off state, thus stopping the capture. Both modes are handled simultaneously through a single global keyboard listener instance. The listener contains event dispatch logic and mutual exclusion logic. The mutual exclusion logic ensures that when one mode is in an active recording state, the triggering event of the other mode is ignored, thereby avoiding recording state conflicts.

7. The method according to claim 1, characterized in that, Between S3 and S4, a streaming identification step is also included, and in S4, the offline identification result is used to replace the streaming identification result: The streaming recognition steps include: during recording, resampling the captured audio data in real time to the sampling rate and channel number format required by the streaming speech recognition model, and then feeding it to the streaming speech recognition model in units of fixed-size audio blocks; the streaming speech recognition model achieves contextual continuity across audio blocks by maintaining a key-value cache in the attention mechanism; whenever an audio block is fed, the streaming speech recognition model returns the current partial recognition result, which is displayed on the user interface in real time; S4 includes: when a release signal or a re-trigger signal is received from the trigger source, stopping the capture of audio data, and sending the complete audio file of this recording to an offline speech recognition model for re-recognition to obtain an offline recognition result; replacing the part of the recognition result previously displayed on the user interface with the offline recognition result, and outputting the offline recognition result as the final recognition result.

8. The method according to claim 7, characterized in that, Before sending the buffered audio data into the local offline speech recognition model for recognition, an automatic segmentation step for speech activity detection is also included. The automatic segmentation step for voice activity detection includes: Determine whether the total duration of the recorded audio exceeds a preset duration threshold. If it does not exceed the duration threshold, the complete audio is directly sent to the local offline speech recognition model. If it exceeds the duration threshold, the speech activity detection and segmentation process is started. The speech activity detection segmentation process includes: loading a pre-trained speech activity detection model; feeding the complete audio frame by frame into the speech activity detection model with a fixed window size to obtain the speech probability of each frame; marking each frame as a speech frame or a silence frame according to a preset confidence threshold; segmenting the speech into a segment at the position when the duration of consecutive silence frames exceeds the minimum silence threshold; applying a maximum duration limit constraint to a single speech segment; forcibly segmenting the speech segment at the maximum duration limit when no silence interval exceeding the minimum silence threshold occurs within the maximum duration limit; independently feeding each segmented speech segment into the local offline speech recognition model for recognition; and sequentially concatenating the recognized text according to the original time order of each segment to form a complete final recognition result.

9. The method according to claim 7, characterized in that: In stage S2, the audio data received by the audio input stream is not discarded, but written to a fixed-capacity circular buffer in a first-in-first-out manner. When the circular buffer is full, the oldest data is overwritten. In stage S3, the existing audio data in the circular buffer is spliced ​​to the audio front end of the current recording.

10. A low-latency voice input system based on a resident audio stream, characterized in that, include: The resident audio stream module is used to create and keep the audio input stream active when the voice input system starts. The audio input stream continuously receives audio data from the audio acquisition device through a callback mode. The lifecycle of the audio input stream is independent of the recording flag and does not change with the start or stop of the recording state. The silent discard module is used to keep the audio input stream running when the recording flag is in the off state, but not to capture and store the received audio data, and the audio data is silently discarded. The dual-write buffer module is used to start capturing audio data by setting the recording flag to the on state when a trigger signal is received from the trigger source, and simultaneously write the captured audio data to a memory buffer and a temporary disk file. The local end-side recognition module is used to stop capturing audio data when it receives a release signal or a re-trigger signal from the trigger source, and send the buffered audio data into the local end-side offline speech recognition model for recognition to obtain offline recognition results. The instruction execution module is used to transmit the recognition result to the application system running on the current terminal, and the application system executes the corresponding operation instructions according to the recognition result.

Images