Display device and voice interaction method

Abstract

This application provides a display device and a voice interaction method. In the standby state of the display device, a coprocessor continues to run and stores ambient audio collected by a microphone array in its buffer. Once the coprocessor detects a wake-up word, it immediately wakes the main chip from sleep mode and re-establishes communication with the server. Simultaneously, it starts a voice activity detection service to locate the user's voice following the wake-up word from the buffer. Because the user's voice is pre-cached during the main chip's wake-up process, there is no risk of data loss. This allows the user to speak a voice command simultaneously with issuing the wake-up word, completing voice wake-up and voice command input in a single interaction, significantly improving the user experience.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a voice interaction method. Background Technology

[0002] Display devices refer to terminal devices capable of outputting specific display images, such as smart TVs, mobile terminals, smart advertising screens, and projectors. Taking smart TVs as an example, smart TVs are television products based on Internet application technologies, possessing open operating systems and chips, and having open application platforms. They enable two-way human-computer interaction and integrate multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] In standby mode, display devices disable certain modules and services to reduce power consumption, such as shutting down the main chip, turning off the screen, and enabling far-field voice services. However, to allow users to wake up the device via far-field voice while in standby mode, the display device retains modules and services related to voice wake-up.

[0004] However, in standby mode, since far-field voice services are disabled, users can only wake up the device by voice. To further interact with the display device via voice, they must wait for the display device to wake up and then wake up the far-field voice service again by voice. Therefore, in standby mode, users need to interact with the display device multiple times before they can engage in voice interaction. Summary of the Invention

[0005] This application provides a display device and a voice interaction method, which can solve the problem that users need to interact with the display device multiple times in standby mode before they can perform voice interaction.

[0006] In a first aspect, this application provides a display device, comprising: Display; communication device for communicating with a server; main chip configured to run far-field voice services in operation; microphone array; coprocessor with buffer; controller configured to: In standby mode, the main chip is controlled to enter sleep mode, and the communication device is controlled to disconnect from the server. The coprocessor remains running and acquires ambient audio via the microphone array. The acquired ambient audio is stored in the buffer, and the coprocessor performs wake-up word recognition on the ambient audio in the buffer. After the coprocessor recognizes the wake-up word, the main chip is restored from sleep mode to running mode. Communication with the server is established via the communication device, and a voice activity detection service is started to detect user voice following the wake-up word in the buffer. After establishing communication with the server, the user voice is read from the buffer and uploaded to the server. Upon receiving a target instruction corresponding to the user voice from the server, the corresponding task is executed in response to the target instruction.

[0007] Based on this, while the display device is in standby mode, the coprocessor continues to run and stores the ambient audio captured by the microphone array in its buffer. Once the coprocessor detects a wake-up word, it immediately wakes the main chip from sleep mode and re-establishes communication with the server. Simultaneously, it starts a voice activity detection service to locate the user's voice following the wake-up word from the buffer. Since the user's voice is pre-cached during the main chip's wake-up process, there is no risk of data loss. This allows the user to speak a voice command simultaneously with issuing the wake-up word, completing voice wake-up and voice command input in a single interaction, significantly improving the user experience.

[0008] In some embodiments of this application, the buffer is a circular buffer; the controller stores the collected ambient audio in the buffer and is specifically configured to: identify a write pointer in the buffer; write the collected ambient audio at the position indicated by the write pointer, wherein if the amount of ambient audio data already written reaches the maximum memory of the buffer, start writing ambient audio from the first data currently written in the buffer and continue writing ambient audio in an overwrite manner; after writing ambient audio, update the position indicated by the write pointer, wherein the write pointer indicates the last data written in the buffer.

[0009] Based on this, a circular buffer structure is adopted. After the buffer is full, the latest collected environmental audio is written in a way that overwrites the earliest data. This allows for the continuous storage of audio data within a recent period while maintaining extremely low power consumption of the coprocessor. This ensures that the wake word and subsequent user speech can be completely read in the buffer, providing a reliable data foundation for subsequent voice activity detection and uploading.

[0010] In some embodiments of this application, after writing ambient audio, the controller is further configured to update the position indicated by the read pointer, wherein the read pointer indicates the data written first in the buffer.

[0011] Based on this, by maintaining the read pointer, the data written first in the buffer can be located at any time, which makes it easy to accurately determine the starting position of reading when a specific range of audio data needs to be read from the buffer, thus ensuring the orderliness and correctness of data reading.

[0012] In some embodiments of this application, the controller stores the collected ambient audio in the buffer, and the coprocessor performs wake word recognition on the ambient audio in the buffer. Specifically, the controller is configured to: preprocess the collected ambient audio and store the preprocessed ambient audio in the buffer; the coprocessor calls a first model to input the preprocessed ambient audio into the first model, and the first model performs wake word detection on the preprocessed ambient audio and outputs the detection result.

[0013] Therefore, preprocessing the ambient audio before writing it to the buffer can reduce noise interference, improve the accuracy of wake word detection, and effectively reduce the amount of data stored in the buffer, saving buffer space. By using a coprocessor to call a dedicated first model for wake word detection, efficient and accurate wake word recognition can be achieved with extremely low power consumption.

[0014] In some embodiments of this application, the controller is invoked by the coprocessor to call a first model to input preprocessed ambient audio into the first model. The first model performs wake word detection on the preprocessed ambient audio and outputs the detection result. Specifically, the first model is configured to: calculate the matching degree between the audio features of the preprocessed ambient audio and the audio template of the wake word; and output the detection result based on the matching degree and a confidence threshold. Wherein, if the matching degree is greater than or equal to the confidence threshold, the detection result is that a wake word has been detected; if the matching degree is less than the confidence threshold, the detection result is that no wake word has been detected.

[0015] Based on this, the first model calculates the matching degree between audio features and wake word audio templates, compares it with the confidence threshold, and outputs the judgment result of whether a wake word is detected, thereby achieving accurate recognition of wake words, effectively reducing false wake-up rate, and ensuring reasonable use of power in standby mode.

[0016] In some embodiments of this application, after the controller recognizes the wake-up word through the coprocessor, it initiates a voice activity detection service to detect user voice in the buffer following the wake-up word. Specifically, this is configured as follows: after the coprocessor recognizes the wake-up word, the coprocessor marks a first point at the end position of the wake-up word in the buffer; the voice activity detection service detects ambient audio in the buffer to determine the start and end positions of the voice; after recognizing the first point, the voice activity detection service marks a second point at the position corresponding to the first point and a third point at the end position of the voice; wherein the data between the second point and the third point is the user voice.

[0017] Based on this, after the coprocessor recognizes the wake word, it immediately marks the end position of the wake word in the buffer (first point). Then, the voice activity detection service accurately locates the start and end positions of the user's voice (second to third points), thereby accurately extracting valid user voice data from the buffer, avoiding the uploading of invalid data, and improving the efficiency and accuracy of voice recognition.

[0018] In some embodiments of this application, after establishing communication with the server, the controller reads the user's voice from the buffer and uploads the user's voice to the server. Specifically, the controller is configured to: after establishing communication with the server, identify the second point and the third point in the buffer; read the user's voice from the buffer based on the second point and the third point; and upload the user's voice to the server.

[0019] Based on this, after the main chip completes wake-up and establishes communication with the server, the user's voice data can be accurately read by recognizing the second and third points in the buffer, and then uploaded to the server for recognition and processing, ensuring the integrity and timeliness of voice interaction.

[0020] In some embodiments of this application, the first point is a locking point. When writing ambient audio to the buffer, if the position to be written coincides with the position indicated by the first point, the writing operation is stopped.

[0021] Based on this, by setting the first point as the lock point, it is ensured that when new ambient audio continues to be written to the circular buffer, the user voice data starting from the end position of the wake-up word will not be overwritten, thus ensuring the integrity of the user voice data and preventing the critical audio data after wake-up from being overwritten and lost due to the buffer being full.

[0022] In some embodiments of this application, after establishing communication with the server, the controller reads the user's voice from the buffer and uploads the user's voice to the server. Specifically, the controller is configured to: after establishing communication with the server, obtain the user's voice from the buffer by the far-field voice service; and upload the obtained user's voice to the server by the far-field voice service.

[0023] Based on this, when the main chip resumes operation and starts the far-field voice service, the far-field voice service actively retrieves the user's voice from the coprocessor's buffer and uploads it to the server. This fully utilizes the voice processing capabilities of the far-field voice service, achieving seamless connection between standby wake-up scenarios and normal voice interaction scenarios, and improving the overall fluency of the voice interaction experience.

[0024] Secondly, this application also provides a voice interaction method applied to any of the display devices described in the first aspect, the display device including a main chip, a communication device, a microphone array, and a coprocessor; wherein, the main chip runs a far-field voice service in operation, and the coprocessor has a buffer. The method includes: In standby mode, the main chip is controlled to enter sleep mode, and the communication device is controlled to disconnect from the server. The coprocessor remains running and acquires ambient audio via the microphone array. The acquired ambient audio is stored in the buffer, and the coprocessor performs wake-up word recognition on the ambient audio in the buffer. After the coprocessor recognizes the wake-up word, the main chip is restored from sleep mode to running mode. Communication with the server is established via the communication device, and a voice activity detection service is initiated to detect user voice in the ambient audio following the wake-up word in the buffer. After establishing communication with the server, the user voice is read from the buffer and uploaded to the server. Upon receiving a target instruction corresponding to the user voice from the server, the corresponding task is executed in response to the target instruction.

[0025] Based on this, while the display device is in standby mode, the coprocessor continues to run and stores the ambient audio captured by the microphone array in its buffer. Once the coprocessor detects a wake-up word, it immediately wakes the main chip from sleep mode and re-establishes communication with the server. Simultaneously, it starts a voice activity detection service to locate the user's voice following the wake-up word from the buffer. Since the user's voice is pre-cached during the main chip's wake-up process, there is no risk of data loss. This allows the user to speak a voice command simultaneously with issuing the wake-up word, completing voice wake-up and voice command input in a single interaction, significantly improving the user experience. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the operation scenario between the display device 200 and the control device 100 in an embodiment of this application; Figure 2 This is a hardware configuration block diagram of the display device 200 in this embodiment of the application; Figure 3 This is a diagram showing the operating system configuration of the display device 200 in this embodiment of the application; Figure 4 This is a flowchart illustrating the far-field voice interaction between the display device 200 and the user in standby mode, as described in this application embodiment. Figure 5 This is a timing diagram showing the far-field voice interaction between the display device 200 and the user in an embodiment of this application; Figure 6A and Figure 6B This is a schematic diagram of the structure of the circular buffer in the embodiments of this application; Figure 7 This is a flowchart illustrating how the display device 200 stores the collected ambient audio into a buffer via a coprocessor, as described in this embodiment of the application. Figure 8 This is a flowchart illustrating how the display device 200 identifies a wake word via a coprocessor in an embodiment of this application. Figure 9 This is a flowchart illustrating the process of the first model recognizing wake words in an embodiment of this application; Figure 10 This is a flowchart illustrating how the display device 200 detects user voice through a voice activity detection service in an embodiment of this application. Figure 11 This is a schematic diagram of the marker points in the buffer in an embodiment of this application; Figure 12 This is a flowchart illustrating the process of display device 200 uploading user voice in an embodiment of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0030] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0031] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0032] In this application embodiment, "display device" refers to any device with screen display and data processing capabilities. For example, display devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices.

[0033] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch, voice, mobile terminal 300, and control device. For example, the control device can be a remote control 100, a stylus, a gamepad, etc.

[0034] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0035] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0036] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0037] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a user input interface 280, a memory, and a power supply.

[0038] In some embodiments, the communication device 220 is a component for communicating with external devices or the server 400 according to various communication protocol types. The display device 200 may be equipped with multiple communication devices 220 depending on the supported communication methods. The communication devices 220 can enable the display device 200 to communicate with the external devices or the server 400 via wireless or wired connections.

[0039] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0040] In this embodiment, the detector 230 further includes a microphone array for acquiring the user's voice and ambient audio. The microphone array can include various forms such as linear arrays and circular arrays. Through signal processing between multiple microphones, sound source localization and beamforming can be achieved, improving the quality of far-field voice acquisition.

[0041] In some embodiments, device interface 240 is used to connect to an external device.

[0042] In some embodiments, the controller 250 is used to control the overall operation of the display device 200. The controller 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device 200 and responds to user operations through various software control programs stored in memory.

[0043] In this embodiment, the controller 250 works in conjunction with the main chip. The main chip, in its operational state, is responsible for running far-field voice services and performing complex speech recognition and natural language processing tasks.

[0044] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0045] In some embodiments, the display 260 is used to receive and display image signals output from the controller 250. The display 260 may include display function components for presenting images and driving components for driving image display.

[0046] In some embodiments, a user can input user commands on a graphical user interface (GUI) displayed on a display 260, and a user input interface 280 can receive user commands through the GUI.

[0047] In some embodiments, the audio output device 270 may be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200.

[0048] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0049] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0050] The operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices 200.

[0051] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system can be divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Runtime Library layer, and the Kernel layer.

[0052] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run the applications and interact with the user based on the applications.

[0053] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0054] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0055] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management.

[0056] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0057] In standby mode, display device 200 disables some modules and services to reduce power consumption, such as turning off the main chip, turning off the screen, and enabling far-field voice services. However, to allow users to wake up the device via far-field voice in standby mode, the display device retains modules and services related to voice wake-up.

[0058] In this embodiment of the application, in order to support the voice wake-up function of the display device 200 in standby mode, the controller 250 also works in cooperation with the coprocessor.

[0059] In some embodiments, the coprocessor may include a low-power digital signal processor (DSP). The coprocessor remains operational in standby mode, responsible for continuously acquiring ambient audio and performing lightweight wake-word detection. Its power consumption is far lower than that of the main chip, making it suitable for long-term operation in standby mode.

[0060] In some embodiments, the coprocessor and the main chip can be two independent devices.

[0061] In other embodiments, the coprocessor may be part of the main chip. In standby mode, the coprocessor remains powered on and operational.

[0062] In standby mode, the main chip is in a dormant state. For example, when the main chip is powered off, services running on the main chip, such as far-field voice services, are correspondingly disabled. At this time, because the coprocessor remains running, the device wake-up service running on the coprocessor is also running. Therefore, the display device 200 can only support users waking up the device via voice through the device wake-up service. If the user wants to further interact with the display device 200 via voice, they need to wait for the display device 200 to be woken up before waking up the far-field voice service again via voice. That is, the far-field voice service must be started separately before voice interaction with the display device 200 can occur. Therefore, in standby mode, users need to interact with the display device 200 multiple times before they can engage in voice interaction.

[0063] To address the aforementioned issues, the coprocessor of the display device 200 in this embodiment can maintain operation in standby mode and cache the collected ambient audio. This allows it to directly invoke the far-field voice service run by the main chip after the device is woken up. The far-field voice service then recognizes and responds to the user's voice commands based on the cached ambient audio.

[0064] To support voice interaction in standby mode, the operating system architecture of this application embodiment also includes voice interaction-related service modules at the system level, including but not limited to: a far-field voice service running on the main chip, a wake-word detection module running on the coprocessor, and a voice activity detection (VAD) service. Specifically, the far-field voice service is responsible for coordinating and managing user voice uploaded to the server while the main chip is running; the wake-word detection module is responsible for continuously performing lightweight wake-word detection on the ambient audio in the buffer while the main chip is in sleep mode; and the voice activity detection service is responsible for detecting the start and end positions of human voice in the audio in the buffer after the wake-word is recognized, accurately locating the range of user voice and providing accurate boundary information for subsequent reading and uploading of user voice.

[0065] In some embodiments, the buffer remains operational even in standby mode, enabling normal data reading and writing.

[0066] In some embodiments, based on Figure 2 and Figure 3 The hardware and software architecture shown indicates that the display device 200 supports one-time device wake-up in standby mode and can directly interact with the user via far-field voice.

[0067] Figure 4 This is a flowchart illustrating the far-field voice interaction between the display device 200 and the user in standby mode, as described in this application embodiment. Figure 4 As shown, the specific process is as follows: In step S401, in standby mode, the main chip is controlled to enter sleep mode, the communication device is controlled to disconnect from the server, and the coprocessor is kept running and ambient audio is collected through the microphone array.

[0068] In some embodiments, the standby state of the display device 200 may be triggered by the user.

[0069] In one example, the user presses the power button on the remote control 100 or the standby button on the display device 200 to put the display device 200 into standby mode.

[0070] In other embodiments, the standby state of the display device 200 may be automatically triggered by the device.

[0071] In one example, the display device 200 automatically enters standby mode after not receiving any user operation for an extended period of time; or, the display device 200 automatically enters standby mode after detecting that the system time has reached a specific time period (such as late at night).

[0072] It is understood that this application does not restrict the method of entering standby mode. Regardless of the method of entering standby mode, the technical solution provided in this application is equally applicable and has good versatility in various scenarios.

[0073] In response to a standby command, the display device 200 controls the main chip to enter a sleep state and controls the communication device to disconnect from the server 400, thereby reducing the power consumption of the display device 200. After the main chip enters the sleep state, the far-field voice service run by the main chip also becomes disabled.

[0074] Furthermore, in response to a standby command, the display device 200 will keep the coprocessor running and the microphone array powered on to continuously capture ambient audio. This allows the display device 200 to still support functions such as device wake-up via far-field voice commands even in standby mode.

[0075] In some embodiments, the microphone array may include multiple microphones. The microphone array can be used to locate the direction of the sound source and enhance the speech signal in a specific direction through techniques such as beamforming, while suppressing noise in other directions and improving the quality of speech acquisition. This can effectively improve the accuracy of subsequent wake-up word recognition and reduce the false wake-up rate.

[0076] Figure 5 This is a timing diagram of the display device 200 interacting with the user in the far-field voice interaction in this embodiment of the application.

[0077] Combination Figure 5 Upon receiving a standby command from the user, the display device 200 enters a standby state. When the display device 200 enters standby state, the controller 250 controls the main chip to enter a hibernation state and controls the communication device 220 to disconnect from the server 400, thereby reducing overall power consumption. Simultaneously, the controller 250 keeps the coprocessor running and continuously collects ambient audio through the microphone array to continuously monitor the user's voice commands while maintaining low power consumption.

[0078] In step S402, the collected ambient audio is stored in a buffer, and the coprocessor performs wake word recognition on the ambient audio in the buffer.

[0079] In some embodiments, the coprocessor has a dedicated buffer for storing ambient audio data over a period of time, providing data support for wake-up word detection and post-wake-up voice data extraction.

[0080] In some embodiments, the buffer can be a pre-allocated, fixed-size, contiguous memory region in Static Random Access Memory (SRAM). Because SRAM offers advantages such as high speed and relatively controllable power consumption, this buffer is more suitable for data reading and writing in standby mode.

[0081] In some embodiments, the buffer is a circular buffer.

[0082] Figure 6A and Figure 6B This is a schematic diagram of the structure of the circular buffer in an embodiment of this application. Figure 6A and Figure 6B As shown, a circular buffer is a circular data structure with its head and tail connected. Its memory space is fixed, and data writing and reading are managed by write pointers (head) and read pointers (tail).

[0083] Figure 7 This is a flowchart illustrating how the display device 200 stores the collected ambient audio into a buffer via a coprocessor, as described in this embodiment of the application. The specific steps are as follows: Step S701: Identify the write pointer in the buffer.

[0084] When storing new ambient audio, the coprocessor first identifies the write pointer in the buffer, and then determines the location where the new ambient audio is written based on the position indicated by the write pointer.

[0085] Step S702: Write the acquired ambient audio at the position indicated by the write pointer.

[0086] In some embodiments, if the amount of ambient audio data already written does not reach the maximum memory of the buffer, the already written ambient audio will not be overwritten when new ambient audio is written.

[0087] In other embodiments, if the amount of ambient audio data already written reaches the maximum memory of the buffer, when writing new ambient audio, the ambient audio is written starting from the data written first in the buffer and continues to be written in an overwrite manner to ensure that the data stored in the buffer is always the ambient audio collected within the last n seconds, thus ensuring the timeliness of the stored ambient audio.

[0088] Step S703: After writing the ambient audio, update the position indicated by the write pointer.

[0089] Understandably, the write pointer indicates the last data written to the buffer. In other words, newly acquired ambient audio will continue to be stored following the last written data.

[0090] In some embodiments, when the amount of data in the buffer has not reached the maximum memory, the write pointer moves backward in a circular direction and writes the newly acquired audio data to the position after the write pointer.

[0091] In other embodiments, when the amount of data in the buffer reaches the maximum memory, the write pointer returns to the starting position to continue writing the latest acquired audio data by overwriting the earliest written data. This ensures that the buffer always stores the latest audio data within a certain period of time, enabling continuous recording of real-time audio streams without consuming a large amount of memory resources.

[0092] Based on the above embodiments, a circular buffer structure is adopted. After the buffer is full, the latest collected environmental audio is written in a way that overwrites the earliest data. This allows for the continuous storage of audio data within a recent period while maintaining extremely low power consumption of the coprocessor. This ensures that the wake word and subsequent user speech can be completely read in the buffer, providing a reliable data foundation for subsequent voice activity detection and uploading.

[0093] In some embodiments, after the display device 200 writes ambient audio into the buffer via a coprocessor, it updates the position indicated by the read pointer, where the read pointer indicates the earliest data written in the buffer. The read pointer indicates the audio data at the earliest point in time currently retained in the buffer, facilitating subsequent reading of audio data within a specific time range. By simultaneously maintaining both the write and read pointers, the effective data range in the circular buffer can be precisely managed, avoiding data read / write chaos.

[0094] Understandably, if the data already written to the buffer has not reached the buffer's maximum memory, the read pointer indicates the first data written to the buffer after the display device 200 enters standby mode. (Refer to...) Figure 6A The write and read pointers are shown. If the data written to the buffer has reached the maximum memory of the buffer, the position indicated by the read pointer coincides with the position indicated by the write pointer. (Refer to...) Figure 6B The write pointer and read pointer are shown.

[0095] Figure 8 This is a flowchart illustrating how the display device 200 identifies the wake word via a coprocessor in an embodiment of this application. The specific steps are as follows: Step S801: Preprocess the collected environmental audio and store the preprocessed environmental audio in a buffer.

[0096] In some embodiments, the coprocessor preprocesses the acquired ambient audio before writing it into the buffer.

[0097] In some embodiments, preprocessing may include: merging the acquired multi-channel microphone signals to obtain enhanced single or multi-channel audio signals; filtering the audio signals to remove low-frequency and high-frequency noise; applying automatic gain control to the audio signals to facilitate subsequent audio data processing; normalizing the audio signals to ensure amplitude consistency; and extracting features from the audio signals, such as extracting Mel-Frequency Cepstral Coefficients (MFCCs), to reduce the amount of stored data and adapt to the input format of subsequent models. Preprocessing improves the quality of audio data stored in the buffer while reducing the computational load on the coprocessor during wake-word detection, further reducing standby power consumption while ensuring wake-word recognition accuracy.

[0098] In step S802, the coprocessor calls the first model to input the preprocessed environmental audio into the first model, and the first model performs wake word detection on the preprocessed environmental audio and outputs the detection result.

[0099] In some embodiments, the first model is a lightweight wake word detection model.

[0100] Figure 9 This is a flowchart illustrating the process of the first model recognizing wake words in an embodiment of this application. The specific steps are as follows: Step S901: The first model calculates the matching degree between the audio features of the preprocessed environmental audio and the audio template of the wake word.

[0101] In some embodiments, the audio characteristics of the ambient audio may be MFCC, etc.

[0102] Step S902: The first model outputs the detection results based on the matching degree and confidence threshold.

[0103] Specifically, if the matching degree is greater than or equal to the confidence threshold, the detection result is that a wake word has been detected; if the matching degree is less than the confidence threshold, the detection result is that no wake word has been detected.

[0104] In some embodiments, as the audio data input to the first model increases, if the audio data corresponds to a wake word, the matching degree, or confidence degree, calculated by the first model will gradually increase. If the confidence degree gradually increases to a peak value, and the peak value can exceed a preset confidence threshold, the first model can determine that a wake word has been detected.

[0105] In some embodiments, the confidence threshold can be dynamically configured according to the actual application scenario to achieve an optimal balance between wake-up sensitivity and false wake-up rate. A higher confidence threshold results in a lower false wake-up rate, but also a corresponding decrease in wake-up sensitivity, potentially leading to missed wake-ups in noisy environments. Conversely, a lower confidence threshold results in higher wake-up sensitivity, but also a corresponding increase in the false wake-up rate, potentially causing the ambient audio (not a wake-up word) to be misinterpreted as a wake-up word, unnecessarily waking up the main chip and increasing power consumption. In specific implementations, the confidence threshold can be reasonably configured based on factors such as the usage environment of the display device 200 and user preferences to meet the usage requirements in different scenarios.

[0106] Combination Figure 5 While maintaining low-power operation, the coprocessor continuously writes the ambient audio captured by the microphone array into its internal buffer. Specifically, after receiving the ambient audio from the microphone array, the coprocessor preprocesses the audio, writes the preprocessed audio to the buffer, and updates the write and read pointers in the buffer. The coprocessor then calls the first model, inputting the ambient audio from the buffer into the first model to output the detection result of the wake word.

[0107] While the display device 200 is in standby mode, the coprocessor continuously executes a loop process of acquiring audio, writing to the buffer, and detecting the wake word until the wake word is detected.

[0108] Step S403: After the coprocessor recognizes the wake word, the main chip is restored from sleep state to running state, communication is established with the server through the communication device, and the voice activity detection service is started to detect the user voice in the buffer located after the wake word.

[0109] Combination Figure 5If the first model detects a wake-up word, the coprocessor will send a wake-up command to the main chip based on the detection result, thereby waking up the main chip from its sleep state. After waking up, the main chip can control the communication device 220 to re-establish a communication connection with the server 400, such as by reconnecting to the server 400.

[0110] Of course, the above operations can be performed in parallel to minimize the overall time delay from the coprocessor recognizing the wake word to the main chip completing the wake-up and establishing communication with the server.

[0111] After recognizing the wake word through the coprocessor, the display device 200 also starts the Voice Activity Detection Service (VAD) to detect the user's voice following the wake word in the buffer, which corresponds to the user's input voice command.

[0112] Figure 10 This is a flowchart illustrating how the display device 200 detects user voice through a voice activity detection service in this embodiment of the application. The specific steps are as follows: Step S1001: After the wake word is detected, the coprocessor marks the first point at the end position of the wake word in the buffer.

[0113] Figure 11 This is a schematic diagram of the marker points in the buffer in an embodiment of this application. Figure 11 As shown, the first point corresponds to the end position of the wake word in the buffer.

[0114] Combination Figure 5 After the coprocessor recognizes the wake word, it marks the first point at the end of the wake word in the buffer.

[0115] The audio data written to the buffer after the end of the wake word is the voice content emitted by the user after saying the wake word. This voice content may contain the specific voice command that the user wants to execute.

[0116] In some embodiments, the first point is the locking point, which can be denoted as "head_trigger".

[0117] When writing ambient audio to the buffer, if the position to be written coincides with the position indicated by the first point, the writing operation stops. That is, when the write pointer moves forward in the circular buffer and is about to reach the position indicated by the first point (lock point), the writing of new ambient audio stops to ensure that the user's voice data starting from the first point is not overwritten by the newly written audio data, ensuring that critical voice data is not lost and providing a guarantee for the complete extraction of the user's voice.

[0118] In step S1002, the voice activity detection service detects the ambient audio in the buffer to determine the start and end positions of the voice.

[0119] After the voice activity detection service is started, it detects the ambient audio in the buffer. By analyzing the characteristics of audio frames such as energy, zero-crossing rate, and fundamental frequency, it accurately determines whether each audio frame contains human voice activity, and thus determines the start and end positions of the human voice.

[0120] In step S1003, after the first point is identified, the voice activity detection service marks the second point at the position corresponding to the first point and marks the third point at the end position of the voice.

[0121] The data between the second and third points is the user voice data precisely extracted from the buffer. This data excludes the wake word itself and the environmental noise before the wake word, and only contains the valid voice commands issued by the user after saying the wake word.

[0122] In some embodiments, the second point can be denoted as "t_utt_start" and the third point can be denoted as "t_utt_end". See also... Figure 11 Points 2 and 3 are shown in the diagram.

[0123] In some embodiments, after the third point is successfully marked, i.e. after the user's voice ends, the first point can be unlocked, and the normal write operation of the circular buffer can be resumed to save memory and provide sufficient buffer space for subsequent audio acquisition, ensuring that the display device 200 can continue to work normally after the end of this voice interaction.

[0124] Combination Figure 5 After the voice activity detection service is started, it performs voice detection on the data in the buffer, and after recognizing the first point, it marks the second point at the position indicated by the first point, and marks the third point at the end position of the voice.

[0125] Step S404: After establishing communication with the server, read the user's voice from the buffer and upload the user's voice to the server.

[0126] After the main chip completes wake-up and successfully establishes a communication connection with the server 400 through the communication device 220, the display device 200 reads the user's voice from the buffer and uploads the user's voice to the server 400 so that the server 400 can accurately identify the target command corresponding to the user's voice.

[0127] Figure 12 This is a flowchart illustrating the process of the display device 200 uploading user voice data in this embodiment of the application. The specific steps include: In step S1201, after establishing communication with the server, the far-field voice service retrieves the user's voice from the buffer in the coprocessor.

[0128] After the main chip resumes operation, the far-field voice service it runs also starts and resumes operation.

[0129] Combination Figure 5 After establishing communication with server 400, the far-field voice service obtains the user's voice from the coprocessor's buffer through the communication interface with the coprocessor.

[0130] In some embodiments, the coprocessor identifies a second point and a third point in a buffer, and based on the second point and the third point, reads the data located between the two points, i.e., the user's voice, from the buffer, and feeds the user's voice back to the far-field voice service.

[0131] In step S1202, the far-field voice service uploads the acquired user voice to the server.

[0132] By having the far-field voice service uniformly handle the uploading of user voice data, the existing capabilities of the far-field voice service in communicating with the server and processing voice data can be fully utilized to achieve a unified processing flow for standby wake-up voice interaction and normal operation voice interaction. There is no need to design additional upload logic for standby wake-up scenarios, which reduces the complexity of system implementation and ensures the consistency of the voice interaction flow after standby wake-up with the voice interaction flow under normal operation, which is beneficial to system maintenance and expansion.

[0133] Step S405: After receiving the target instruction corresponding to the user's voice from the server, execute the corresponding task in response to the target instruction.

[0134] After receiving the uploaded user voice, server 400 performs Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) on it, generates a target instruction corresponding to the user's voice intent, and feeds the target instruction back to display device 200.

[0135] The controller 250 of the display device 200 responds to the received target command and controls the display device 200 to perform corresponding tasks, such as playing specified content, adjusting volume, switching channels, querying weather information, and controlling smart home devices.

[0136] Combination Figure 5After receiving the target instruction and completing the corresponding task, the voice interaction process ends. The display device 200 can decide to maintain the running state or re-enter the standby state according to the nature of the task, so that the user can carry out subsequent interactions.

[0137] Based on the above embodiments, when a user interacts with the display device 200 via voice while it is in standby mode, the user only needs to say a wake-up word followed by a voice command. The coprocessor of the display device 200 can detect the wake-up word and simultaneously retain the user's voice data after the wake-up word in a buffer. After the main chip completes wake-up and establishes communication with the server, it reads the user's voice from the buffer and uploads it to the server for recognition. The entire process is transparent to the user, who does not need to wait for the device to wake up before issuing another voice command. This achieves a simple user experience of "one wake-up, one interaction," effectively solving the problem of lost user voice due to main chip wake-up delay and the need for multiple interactions in traditional solutions.

[0138] In some embodiments, if the voice activity detection service does not detect valid user voice in the buffer after the first point (i.e., the user only says the wake-up word without immediately saying a voice command), it can switch to the normal far-field voice interaction mode after the main chip completes the wake-up, and wait for the user to interact with the display device 200 again by voice in normal operation, thereby ensuring the robustness and integrity of the voice interaction process and avoiding abnormal interruption of the process due to the failure to detect user voice.

[0139] In some embodiments, the size of the buffer can be configured appropriately according to the actual application scenario. The buffer capacity needs to be sufficient to store the complete audio data from the start of the wake-up word to the end of the user's voice, while also taking into account the time delay required for the main chip to recover from sleep mode to running mode and establish communication with the server. During this time delay, the coprocessor continues to collect and store audio into the buffer, so the buffer size should be able to accommodate at least the amount of audio data within this time period to ensure the complete preservation of the user's voice. In practical design, the minimum required buffer capacity can be calculated based on parameters such as the wake-up delay of the main chip, the network connection establishment time, the microphone sampling rate, and the encoding format, and a certain margin should be reserved on this basis to cope with various uncertainties in actual use.

[0140] This application also discloses a voice interaction method that can be applied to a display device 200.

[0141] In some embodiments, in standby mode, the main chip is controlled to enter sleep mode, and the communication device is controlled to disconnect from the server. The coprocessor remains running and acquires ambient audio via a microphone array. The acquired ambient audio is stored in a buffer, and the coprocessor performs wake-up word recognition on the ambient audio in the buffer. After the coprocessor recognizes the wake-up word, the main chip is restored from sleep mode to running mode, communication is established with the server via the communication device, and a voice activity detection service is started to detect user voice following the wake-up word in the buffer. After establishing communication with the server, user voice is read from the buffer and uploaded to the server. Upon receiving a target instruction corresponding to the user voice from the server, the corresponding task is executed in response to the target instruction.

[0142] Based on this, while the display device is in standby mode, the coprocessor continues to run and stores the ambient audio captured by the microphone array in its buffer. Once the coprocessor detects a wake-up word, it immediately wakes the main chip from sleep mode and re-establishes communication with the server. Simultaneously, it starts a voice activity detection service to locate the user's voice following the wake-up word from the buffer. Since the user's voice is pre-cached during the main chip's wake-up process, there is no risk of data loss. This allows the user to speak a voice command simultaneously with issuing the wake-up word, completing voice wake-up and voice command input in a single interaction, significantly improving the user experience.

[0143] In some embodiments of this application, the buffer is a circular buffer; storing the acquired ambient audio into the buffer includes: identifying a write pointer in the buffer; writing the acquired ambient audio at the position indicated by the write pointer, wherein if the amount of ambient audio data already written reaches the maximum memory of the buffer, the ambient audio is continued to be written in an overwrite manner, starting from the data currently written first in the buffer; after writing the ambient audio, updating the position indicated by the write pointer, wherein the write pointer indicates the data last written in the buffer.

[0144] Based on this, a circular buffer structure is adopted. After the buffer is full, the latest collected environmental audio is written in a way that overwrites the earliest data. This allows for the continuous storage of audio data within a recent period while maintaining extremely low power consumption of the coprocessor. This ensures that the wake word and subsequent user speech can be completely read in the buffer, providing a reliable data foundation for subsequent voice activity detection and uploading.

[0145] In some embodiments of this application, after writing ambient audio, the controller is further configured to update the position indicated by the read pointer, wherein the read pointer indicates the data written first in the buffer.

[0146] Based on this, by maintaining the read pointer, the data written first in the buffer can be located at any time, which makes it easy to accurately determine the starting position of reading when a specific range of audio data needs to be read from the buffer, thus ensuring the orderliness and correctness of data reading.

[0147] In some embodiments of this application, the controller stores the collected ambient audio in the buffer, and the coprocessor performs wake word recognition on the ambient audio in the buffer. Specifically, the controller is configured to: preprocess the collected ambient audio and store the preprocessed ambient audio in the buffer; the coprocessor calls a first model to input the preprocessed ambient audio into the first model, and the first model performs wake word detection on the preprocessed ambient audio and outputs the detection result.

[0148] Therefore, preprocessing the ambient audio before writing it to the buffer can reduce noise interference, improve the accuracy of wake word detection, and effectively reduce the amount of data stored in the buffer, saving buffer space. By using a coprocessor to call a dedicated first model for wake word detection, efficient and accurate wake word recognition can be achieved with extremely low power consumption.

[0149] In some embodiments of this application, the controller is invoked by the coprocessor to call a first model to input preprocessed ambient audio into the first model. The first model performs wake word detection on the preprocessed ambient audio and outputs the detection result. Specifically, the first model is configured to: calculate the matching degree between the audio features of the preprocessed ambient audio and the audio template of the wake word; and output the detection result based on the matching degree and a confidence threshold. Wherein, if the matching degree is greater than or equal to the confidence threshold, the detection result is that a wake word has been detected; if the matching degree is less than the confidence threshold, the detection result is that no wake word has been detected.

[0150] Based on this, the first model calculates the matching degree between audio features and wake word audio templates, compares it with the confidence threshold, and outputs the judgment result of whether a wake word is detected, thereby achieving accurate recognition of wake words, effectively reducing false wake-up rate, and ensuring reasonable use of power in standby mode.

[0151] In some embodiments of this application, after the controller recognizes the wake-up word through the coprocessor, it initiates a voice activity detection service to detect user voice in the buffer following the wake-up word. Specifically, this is configured as follows: after the coprocessor recognizes the wake-up word, the coprocessor marks a first point at the end position of the wake-up word in the buffer; the voice activity detection service detects ambient audio in the buffer to determine the start and end positions of the voice; after recognizing the first point, the voice activity detection service marks a second point at the position corresponding to the first point and a third point at the end position of the voice; wherein the data between the second point and the third point is the user voice.

[0152] Based on this, after the coprocessor recognizes the wake word, it immediately marks the end position of the wake word in the buffer (first point). Then, the voice activity detection service (VAD) accurately locates the start and end positions of the user's voice (second to third points), thereby accurately extracting valid user voice data from the buffer, avoiding the uploading of invalid data, and improving the efficiency and accuracy of voice recognition.

[0153] In some embodiments of this application, after establishing communication with the server, the controller reads the user's voice from the buffer and uploads the user's voice to the server. Specifically, the controller is configured to: after establishing communication with the server, identify the second point and the third point in the buffer; read the user's voice from the buffer based on the second point and the third point; and upload the user's voice to the server.

[0154] Based on this, after the main chip completes wake-up and establishes communication with the server, the user's voice data can be accurately read by recognizing the second and third points in the buffer, and then uploaded to the server for recognition and processing, ensuring the integrity and timeliness of voice interaction.

[0155] In some embodiments of this application, the first point is a locking point. When writing ambient audio to the buffer, if the position to be written coincides with the position indicated by the first point, the writing operation is stopped.

[0156] Based on this, by setting the first point as the lock point, it is ensured that when new ambient audio continues to be written to the circular buffer, the user voice data starting from the end position of the wake-up word will not be overwritten, thus ensuring the integrity of the user voice data and preventing the critical audio data after wake-up from being overwritten and lost due to the buffer being full.

[0157] In some embodiments of this application, after establishing communication with the server, the controller reads the user's voice from the buffer and uploads the user's voice to the server. Specifically, the controller is configured to: after establishing communication with the server, obtain the user's voice from the buffer by the far-field voice service; and upload the obtained user's voice to the server by the far-field voice service.

[0158] Based on this, when the main chip resumes operation and starts the far-field voice service, the far-field voice service actively retrieves the user's voice from the coprocessor's buffer and uploads it to the server. This fully utilizes the voice processing capabilities of the far-field voice service, achieving seamless connection between standby wake-up scenarios and normal voice interaction scenarios, and improving the overall fluency of the voice interaction experience.

[0159] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the embodiments.

Claims

1. A display device, characterized in that, include: monitor; A communication device used to communicate with a server; The main chip is configured to run far-field voice services in operation. Microphone array; A coprocessor with a buffer; The controller is configured as follows: In standby mode, the main chip is controlled to enter sleep mode, the communication device is controlled to disconnect from the server, and the coprocessor is kept running and ambient audio is collected through the microphone array. The collected ambient audio is stored in the buffer, and the coprocessor performs wake word recognition on the ambient audio in the buffer. After the coprocessor recognizes the wake word, the main chip is restored from sleep state to running state, communication is established with the server through the communication device, and the voice activity detection service is started to detect the user voice in the buffer following the wake word; After establishing communication with the server, the user's voice is read from the buffer and uploaded to the server; Upon receiving the target instruction corresponding to the user's voice from the server, the system executes the corresponding task in response to the target instruction.

2. The display device according to claim 1, characterized in that, The buffer is a circular buffer; The controller stores the collected ambient audio into the buffer, and is specifically configured as follows: Identify the write pointer in the buffer; The captured ambient audio is written at the position indicated by the write pointer. If the amount of ambient audio data already written reaches the maximum memory of the buffer, the ambient audio is continued to be written in an overwrite manner, starting from the data written first in the buffer. After writing the ambient audio, the position indicated by the write pointer is updated, wherein the write pointer indicates the last data written in the buffer.

3. The display device according to claim 2, characterized in that, After writing the ambient audio, the controller is further configured to: Update the position indicated by the read pointer, where the read pointer indicates the data written first in the buffer.

4. The display device according to claim 1, characterized in that, The controller stores the collected ambient audio into the buffer, and the coprocessor performs wake word recognition on the ambient audio in the buffer, specifically configured as follows: The collected ambient audio is preprocessed, and the preprocessed ambient audio is stored in the buffer. The coprocessor calls the first model to input the preprocessed ambient audio into the first model, and the first model performs wake word detection on the preprocessed ambient audio and outputs the detection result.

5. The display device according to claim 4, characterized in that, The controller, invoked by the coprocessor, calls a first model to input preprocessed ambient audio. The first model then performs wake word detection on the preprocessed ambient audio and outputs the detection result. Specifically, it is configured as follows: The matching degree between the audio features of the preprocessed environmental audio and the audio template of the wake word is calculated by the first model; The first model outputs the detection result based on the matching degree and the confidence threshold; wherein, if the matching degree is greater than or equal to the confidence threshold, the detection result is that a wake word has been detected; If the matching degree is less than the confidence threshold, the detection result is that no wake word was detected.

6. The display device according to claim 1, characterized in that, After the controller recognizes the wake word through the coprocessor, it initiates a voice activity detection service to detect user voice in the buffer following the wake word. Specifically, this is configured as follows: After the coprocessor identifies the wake word, the coprocessor marks a first point at the end position of the wake word in the buffer. The voice activity detection service detects the ambient audio in the buffer to determine the start and end positions of the human voice. After the voice activity detection service identifies the first point, it marks a second point at the position corresponding to the first point and marks a third point at the end position of the human voice; wherein the data between the second point and the third point is the user's voice.

7. The display device according to claim 6, characterized in that, After establishing communication with the server, the controller reads the user's voice from the buffer and uploads the user's voice to the server, specifically configured as follows: After establishing communication with the server, the second point and the third point are identified in the buffer; Based on the second and third points, the user's voice is read from the buffer; The user's voice is uploaded to the server.

8. The display device according to claim 6 or 7, characterized in that, The first point is a locking point. When writing ambient audio to the buffer, if the position to be written coincides with the position indicated by the first point, the writing operation stops.

9. The display device according to claim 1, characterized in that, After establishing communication with the server, the controller reads the user's voice from the buffer and uploads the user's voice to the server, specifically configured as follows: After establishing communication with the server, the far-field voice service retrieves the user's voice from the buffer from the coprocessor; The far-field voice service uploads the acquired user voice to the server.

10. A voice interaction method, characterized in that, The invention is applied to a display device, which includes a main chip, a communication device, a microphone array, and a coprocessor; wherein the main chip runs a far-field voice service in operation, and the coprocessor has a buffer. The method includes: In standby mode, the main chip is controlled to enter sleep mode, the communication device is controlled to disconnect from the server, and the coprocessor is kept running and ambient audio is collected through the microphone array. The collected ambient audio is stored in the buffer, and the coprocessor performs wake word recognition on the ambient audio in the buffer. After the coprocessor recognizes the wake word, the main chip is restored from sleep state to running state, communication is established with the server through the communication device, and the voice activity detection service is started to detect the user's voice in the ambient audio following the wake word in the buffer. After establishing communication with the server, the user's voice is read from the buffer and uploaded to the server; Upon receiving the target instruction corresponding to the user's voice from the server, the system executes the corresponding task in response to the target instruction.

Images