Real-time and file-based audio data processing
By dynamically switching between real-time and batch data processing modes, electronic devices effectively manage audio data processing, addressing inefficiencies in low-cost equipment and enhancing user experience.
Patent Information
- Application Number
- JP2023520198
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-10-02
- Filing Date
- 2021-10-01
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2041-10-01
AI Technical Summary
Existing electronic devices with limited communication, computing, and storage capabilities face delays in processing voice-activated functions due to inefficiencies in data processing and transfer mechanisms, particularly in low-cost television equipment.
Implementing a combination of real-time and batch data processing modes for audio data, where devices dynamically switch between these modes based on their processing capabilities, ensuring efficient handling of audio data samples.
This approach enhances data processing efficiency, improves audio quality, and streamlines communication, thereby reducing delays and improving the user experience for voice-activated features.
Smart Images

Figure 0007588223000001 
Figure 0007588223000002 
Figure 0007588223000003
Abstract
Description
[Technical field]
[0001] REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 086,953, filed October 2, 2020, which is incorporated by reference in its entirety.
[0002] Technical Field This application relates generally to audio data transfer and processing, including, but not limited to, methods and systems for providing real-time and file-based audio data processing to facilitate data transfer and voice recognition in electronic devices. [Background technology]
[0003] background Electronic devices with microphones are widely used as assistant devices to collect voice inputs from users and initiate different voice-activated functions according to the voice inputs. For example, many remote control devices coupled to digital television devices are configured to integrate microphones. The voice inputs of these remote control devices are streamed to the digital television device and at least partially processed by the digital television device. The digital television device may submit the voice inputs (pre-processed or not) to a remote server system for additional audio processing. User requests are extracted from the voice inputs to initiate voice-activated functions as a result of audio processing in the television device and / or the remote server system. Any deficiencies in the communication, cache, and processing capabilities of the television device may cause delays in initiating voice-activated functions. This often occurs in low-cost television devices with limited capabilities. It would be beneficial to have a more efficient data processing and transfer mechanism than currently exists to compensate for the deficiencies in the communication, cache, or processing capabilities of these devices. Summary of the Invention
[0004] overview The present application relates to processing and forwarding audio data received from an electronic device having a microphone (e.g., a remote control device, an assistant device). The electronic device is coupled to another electronic device having audio processing capabilities (e.g., a television device) or coupled to a server having audio processing capabilities. These two electronic devices are connected through a communication channel. The audio data is transferred in real time through the communication channel and processed by the receiving electronic device in real time or in batches depending on whether the communication, computation, and storage of the receiving electronic device can support processing of audio data samples in real time. Thereby, the real-time audio data processing is supplemented by batch audio data processing, especially in some electronic devices that do not always have sufficient resources to communicate, cache, or process audio data in real time.
[0005] Specifically, in one aspect, a method for processing audio data, e.g., switching from a real-time data processing mode to a batch data processing mode, is realized. The method includes a second electronic device (e.g., a television device) receiving a first sequence of audio data samples and a second sequence of audio data samples from a first electronic device (e.g., a remote control device). The second sequence of audio data samples follows the first sequence of audio data samples in an audio signal captured by a microphone of the first electronic device. The method further includes the second electronic device processing the first sequence of audio data samples according to the real-time data processing mode and determining that the second electronic device cannot support processing of the audio data samples in the real-time data processing mode. The method further includes caching the second sequence of audio data samples in a buffer of the second electronic device according to a determination that the second electronic device cannot support processing of the audio data samples in the real-time data processing mode and generating a data file including the second sequence of audio data samples in the batch data processing mode.
[0006] Alternatively, in another aspect, a method for processing audio data, e.g., switching from a batch data processing mode to a real-time data processing mode, is realized. The method includes a second electronic device (e.g., a television device) receiving a first sequence of audio data samples and a second sequence of audio data samples from a first electronic device (e.g., a remote control device). The second sequence of audio data samples follows the first sequence of audio data samples in an audio signal captured by a microphone of the first electronic device. The method further includes the second electronic device processing the first sequence of audio data samples according to the batch data processing mode, which further includes caching the first sequence of audio data samples in a buffer of the second electronic device and generating a data file including the first sequence of audio data samples. The method further includes determining that the second electronic device can support processing of the audio data samples in the real-time data processing mode. The method further includes, in accordance with determining that the second electronic device can support processing of the audio data samples in the real-time data processing mode, the second electronic device processing the second sequence of audio data samples in accordance with the real-time data processing mode.
[0007] A non-transitory computer readable medium has instructions stored thereon that, when executed by one or more processors, cause the processors to perform any of the methods described above. An electronic device includes one or more processors and a memory having instructions stored thereon that, when executed by the one or more processors, cause the processors to perform any of the methods described above.
[0008] BRIEF DESCRIPTION OF THE DRAWINGS For a better understanding of the various implementations described, please refer to the following description of implementations in conjunction with the following drawings, in which like reference numbers refer to corresponding parts and in which: [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 illustrates an exemplary media environment in which networked TV devices, remote control devices, and server systems interact with each other via one or more communication networks, according to some implementations. [Diagram 2] A diagram showing an example audio data transfer path between a first electronic device and a second electronic device according to some implementations. [Diagram 3] 1 is a schematic diagram illustrating an example audio data processing process for switching from a real-time data processing mode to a batch data processing mode, according to some implementations. [Figure 4] FIG. 2 is a schematic diagram illustrating an example audio data processing process for switching from a batch data processing mode to a real-time data processing mode, according to some implementations. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example voice assistant process initiated by a user action or voice input, according to some implementations. [Figure 6] A diagram illustrating an exemplary remote control device configured to transfer audio data to a television device according to some implementations. [Figure 7] 1 is a flowchart of a method for dynamically processing audio data in two audio data processing modes according to some implementations. [Figure 8] 1 is a flowchart of another method for dynamically processing audio data in two audio data processing modes according to some implementations.
[0010] Like reference numerals refer to corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Explanation of the implementation example Many electronic devices (e.g., remote control devices, voice-activated displays, or speaker devices) include a microphone that is used to collect voice input from the environment in which the electronic device is located. Such electronic devices may be configured to automatically collect voice input upon detecting a hotword in an audio signal or in response to a user pressing a dedicated assistant button on the electronic device. Upon receiving the voice input, the electronic device communicates the voice input to a remote server system (e.g., an assistant server) via one or more communication networks, and the remote server system recognizes the user request in the voice input and responds to the user request. In one example, the electronic device includes a remote control device, and the voice input is initiated to control a network-connected television (TV) device coupled to the remote control device. The remote control device transmits the voice input to the remote server system via the TV device, and the voice input is processed at the TV device before the TV device transmits the voice input to the remote server system. During this process of processing the voice input, the TV device uses its communication, computing, and storage capabilities to bridge the remote control device and the remote server system.
[0012] The audio data delivered to the audio manager of the TV device may be different from that collected by the microphone of the remote control device. This occurs due to various factors, such as data packet loss and delay through the communication channel coupling the remote control device and the TV device, and the processor load of the TV device. In various implementations of the present application, a combination of real-time data processing and file-based batch data processing is implemented in a second electronic device (e.g., a TV device) to process the audio data collected by the microphone of the first electronic device (e.g., a remote control device). In some implementations, real-time data processing is prioritized over file-based batch data processing. If it is determined that at least one of the communication capabilities, computation capabilities, and storage capabilities of the second electronic device cannot support the processing of the audio data samples in real time, the subsequent data samples are cached and organized into a data file in the second electronic device (e.g., processed by an audio data processing module different from the audio manager). Alternatively, if it is determined that the communication, computing, and storage capabilities of the second electronic device can support processing of audio data samples in real time, the subsequent data samples are processed into data packets in real time by the second electronic device (e.g., processed by an audio manager that is part of the operating system of the second electronic device). Such a controlled audio data transfer process streamlines data processing and communication in the second electronic device and improves the performance, audio quality, and user experience associated with voice input that initiates a user interaction with the electronic device.
[0013] Reference will now be made in detail to the implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various implementations being described. However, it will be apparent to those skilled in the art that the various implementations being described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the implementations.
[0014] 1 is an exemplary media environment 100 in which a networked TV device 102, a remote control device 104, and a server system 106 interact with each other via one or more communication networks 180, according to some implementations. The media environment 100 corresponds to a virtual user domain created and hosted by the server system 106, the virtual user domain including a plurality of user accounts. For each user account, the server system 106 is coupled to a content source 110 and one or more media devices 102 and 116-126, and configured to stream media content provided by the content source 110 for review by the user via the respective user account. Optionally, the content source 110 includes one or more of an advertisement source, an electronic program guide (EPG) source, and a media content source.
[0015] Specifically, one or more media devices associated with a user and a user account are disposed within the media environment 100 to provide the user with media content stored in and streamed from a content source 110. The content source 110 is optionally a third-party media content source or an internal media source hosted by the server system 106. In some implementations, the one or more media devices include a networked TV device 102 that directly streams media content from a remote content source or integrates an embedded casting unit configured to stream media content for display to its audience. The networked TV device 102 is communicatively coupled to a dedicated remote control device 104 and / or electronic devices having remote control applications (e.g., a mobile phone 122, a tablet computer 124, a laptop computer 126, an assistant device 138). The dedicated remote control device 104 may be disposed in proximity to the TV device 102 and configured to communicate with the TV device 102 using pulses of digitally encoded infrared signals. Alternatively, in some situations, a dedicated remote control device 104 or an electronic device having a remote control application may be configured to communicate with the TV device 102 via the communications network 180 (i.e., via a short-range communications link, a local area network, and / or a wide area network) and need not be in physical proximity to the TV device 102.
[0016] The networked TV device 102 includes one or more processors and memory that stores instructions for execution by the one or more processors. The instructions stored in the networked TV device 102 include one or more of an integrated TV application, a local content casting application, a remote control application, an assistant application, and one or more media playback applications associated with the content source 110. These applications are user applications distinct from the operating system of the TV device 102 and are optionally linked to a user account in a virtual user domain of the media environment 100. In addition, the networked TV device 102 includes an audio manager (e.g., 234 of FIG. 2) integrated into its operating system to process audio data at the data packet level.
[0017] Alternatively, in some implementations, media devices disposed in the media environment 100 include a display device 116 that outputs media content directly to a viewer, and a casting device 118 that is coupled to the display device 116 and configured to stream media content to the display device 116. Examples of the display device 116 include, but are not limited to, a television (TV) display device and a music player. Examples of the casting device 118 include, but are not limited to, a set-top box (STB), a DVD player, and a TV box. In this example shown in FIG. 1, the display device 116 includes a TV display that is hardwired to a DVD player or set-top box 118. In contrast, in some implementations, media devices disposed in the media environment 100 include a computer screen 120A that outputs media content to a viewer, and a desktop computer 120B that streams media content to the computer screen 120A. In some implementations, media devices disposed in the media environment 100 include mobile devices, such as a mobile phone 122, a tablet computer 124, and a laptop computer 126. Each of the media devices 118 - 126 includes one or more media playback applications configured to receive and play media content items provided by the content source 110 or by internal media sources associated with the server system 106 .
[0018] The server system 106 includes a unified media platform (I-JWP) 128 configured to manage media content recommendations and streaming to one or more media devices in the media environment 100. Media content recommendations generated by the I-AAP 128 are presented on the networked TV device 102 via a server-side TV application 134 that enables display of media content on the unified TV application on the TV device 102 in response to a user selection from the media content recommendations. Additionally, the UNIP 128 may also function as a centralized media content management module configured to provide media content recommendations to other media devices 118-126 in addition to the TV device 102. In some implementations, activity data associated with each user account is collected from the TV application 134 and the casting service module 136 and used to personalize media content recommendations provided to users of the user account.
[0019] In some implementations, a user account of a virtual user domain hosted by server system 106 is also associated with one or more of other types of devices, such as a network-connected assistant device 138 installed in media environment 100, in addition to one or more of media devices 102, 104, and 116-126. Examples of assistant devices 138 include speaker assistant device 142 and display assistant device 144. Speaker assistant device 142 can collect voice input, recognize user commands from the voice input, and perform actions in response to the user commands (e.g., playing music, answering questions). Display assistant device 144 can collect audio and / or video input, recognize user commands from the audio and / or video input, and perform actions in response to the user commands (e.g., playing music, presenting images or video clips, answering questions). Each of the assistant devices 138 is optionally managed by a dedicated device application or a general-purpose user application (e.g., a web browser) and linked to a user account in a virtual domain in conjunction with an integrated TV application on the network-connected TV device 102.
[0020] Further, in some implementations, the server system 106 optionally includes an assistant module 140 operating with artificial intelligence. The assistant module 140 is configured to recognize user requests from voice input collected by the microphone and initiate actions such as searching the Internet, scheduling events and alarms, adjusting hardware settings, presenting public or private information, playing media content items, interactively conversing with the user, purchasing products, transferring money, etc. The microphone is integrated in any of the media devices 102, 104, and 116-126 and assistant device 138 disposed in the media environment 100. In some implementations, the assistant module 140 is coupled to a speech recognition module 160 configured to process the voice input collected by the microphone and identify user requests from the voice input using, for example, natural language processing (NLP) algorithms.
[0021] In some implementations, the server system 106 includes a device and application registry 150 configured to store information for one or more user accounts managed by the server system 106, as well as information for user devices and applications associated with each of the one or more user accounts. For example, the device and application registry 150 stores information for the networked TV devices 102, remote control devices 104, media devices 116-126, and assistant devices 138, as well as information for corresponding integrated TV applications, remote control applications, media playback applications, and dedicated device applications associated with the assistant devices 138.
[0022] Optionally, these media devices and assistant devices associated with the same user account are distributed across different regions. Optionally, these devices are located in the same physical location. Each media or assistant device communicates with another device or server system 106 using one or more communication networks 180. The communication network 180 used can be one or more networks with one or more types of topology, including, but not limited to, the Internet, an intranet, a local area network (LAN), a cellular network, an Ethernet, a storage area network (SAN), a telephone network, a Bluetooth personal area network (PAN), and the like. In some implementations, two or more devices in a sub-network are coupled via a wired connection, while at least some of the devices in the same sub-network are coupled via a local wireless communication network (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other wireless communication networks).
[0023] In various implementations, a first electronic device (e.g., remote control device 104, any of media devices 116-126, assistant device 138) having a microphone is coupled to a second electronic device (e.g., TV device 102 or any of media devices 116-126) having one or more processors and a memory that stores instructions to be executed by the one or more processors via a communication channel. The first electronic device captures an audio signal using its microphone. The audio signal is sampled into a first sequence of audio data samples and a second sequence of audio data samples that follows the first sequence of audio data samples. Optionally, the first and second sequences of audio data samples are recorded in the same recording session or during two separate recording sessions. Each recording session is activated by a respective user action with the first electronic device (e.g., a user pressing the assistant button 606 in FIG. 6). The first electronic device transfers both the first and second sequences of audio data samples to the second electronic device, which then processes each sequence of audio data samples via one of a real-time data processing mode and a batch data processing mode.
[0024] During or after transferring the first sequence of audio data samples, the second electronic device determines whether the communication, computation and storage capabilities of the second electronic device can support processing of the audio data samples in a real-time data processing mode. If at least one of the communication, computation and storage capabilities of the second electronic device cannot support processing of the audio data samples in the real-time data processing mode, the second electronic device processes the second sequence of audio data samples in a batch data processing mode (e.g., by an audio data processing module 238 installed in its operating system or as a user application in FIG. 2). In contrast, if the second electronic device can support processing of the audio data samples in the real-time data processing mode, the second electronic device processes the second sequence of audio data samples in the real-time data processing mode (e.g., by an audio manager 234 integrated in its operating system in FIG. 2). Thus, the batch data processing mode is activated according to a determination that the second electronic device cannot support processing of the audio data samples in the real-time data processing mode, for example, when an error rate or latency of the audio data exceeds a corresponding error or latency tolerance value.
[0025] 2 is an example audio data transfer path 200 between a first electronic device 202 and a second electronic device 204 according to some implementations. The first electronic device 202 has or is coupled to a microphone 206 configured to capture an audio signal 220, and the second electronic device 204 is coupled to the first electronic device 202 via a communication channel 208. The audio signal 220 is sampled at the first electronic device 202, and the audio data samples are transferred from the first electronic device 202 to the second electronic device 204 via the communication channel 208. The communication channel 208 is enabled by one or more communication networks 180, including, but not limited to, local wireless communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and other wireless communication networks). In this example audio data transfer path 200, a communication channel 208 is formed via a Bluetooth communication link collectively enabled by a first Bluetooth stack 208A of a first electronic device 202 and a second Bluetooth stack 208B of a second electronic device 204.
[0026] The first electronic device 202 includes an audio streaming module 210 configured to obtain audio data samples of an audio signal 220 captured by a microphone 206 and organize the audio data samples to be transferred to a second electronic device via a communication channel 208. In some implementations, the audio streaming module 210 groups a subset of the audio data samples into an ordered sequence of audio data packets. Each data packet includes one or more consecutive audio data samples and optionally has a preamble, a message header, encoded packet data, a dummy field, and an integrity check field according to a predefined data format (e.g., MPEG-4 HE-ACC codec format, EVRC voice codec format). At the input to the first Bluetooth stack 208A, the sequence of the ordered audio data packets is sequentially arranged into a stream of audio data 230 for transmission over the communication channel 208. In some implementations, each data packet in the same data packet sequence corresponds to a predefined data format, and the two different data packet sequences optionally correspond to the same data format or different data formats.
[0027] The second electronic device 204 includes three levels of programs: a kernel 216 on a hardware abstraction layer (HAL), a device firmware 218, and application and service programs 222. The kernel 216 on the HAL and the device firmware 218 are part of the operating system of the second electronic device 204, and the application and service programs 222 are outside the operating system and are installed by a manufacturer or a user to realize dedicated computer operations (e.g., gaming, web browsing, document editing, media playback). In some implementations, after receiving operational data, optionally originating from the first electronic device 202 or any other electronic device via the second Bluetooth stack 208B, the second electronic device 204 passes the received operational data to an input event identifier 226 of the kernel 216 on the HAL. The input event identifier 226 identifies the operational data from the data received by the second Bluetooth stack 208B and provides the identified operational data to an input dispatcher 228 of the device firmware 218. The input dispatcher 228 assigns the action data to an assistant application 232 installed on the second electronic device 204 to recognize user requests in the action data and to initiate actions in response to the user requests.
[0028] In some implementations, a stream of audio data 232 is collected by the microphone 206 of the first electronic device 202 and dispatched by the kernel / HAL 216 directly to a remote control application 236 associated with the first electronic device 202. The remote control application 236 cooperates (260) with an audio manager 234 of the device firmware 218 and an assistant application 232 installed on the second electronic device 204 to recognize and respond to user requests in the stream of audio data 230. The audio manager 234 implements a real-time data processing mode and processes the audio data 230 at the data packet level. That is, the audio manager 234 is configured to identify audio data samples from data packets within a sequence of multiple ordered audio data packets received from the first electronic device 202, compensate for erroneous, missing, and out-of-order data packets, organize the received audio data samples into another sequence of audio data packets, and pass (262) the sequence of audio data packets to the assistant application 232 for subsequent audio processing or transfer.
[0029] Alternatively, in some implementations, the second electronic device 204 includes a file-based audio data processing module 238 that implements a batch data processing mode and processes the audio data 230 received from the first electronic device 202 into a data file. The file-based audio data processing module 238 is either integrated (238A) into the device firmware 218 (i.e., the operating system) or installed (238B) as a user application in the application and service programs 222. The remote control application 236 provides (270 or 280) the audio data 230 to the processing module 238. The processing module 238 is configured to identify audio data samples from data packets in a sequence of a plurality of ordered audio data packets received from the first electronic device 202, cache the data packets in a data file, and provide (272 or 282) the data file to the assistant application 232 for later audio processing or transfer. The second electronic device 204 further includes an audio data buffer 214 for storing the data file.
[0030] The audio data samples are alternately processed by the audio manager 234 and the processing module 238 in a real-time data processing mode and a batch data processing mode, respectively. The second electronic device 204 switches between these two audio data processing modes depending on whether the communication, computation, and storage capabilities of the second electronic device 204 can support real-time transfer of the audio data samples. In some implementations, such determination and mode switching is realized dynamically while the sequence of ordered audio data packets is being processed by the audio manager 234 or the processing module 238. Alternatively, such determination and mode switching is realized between two separate recording sessions, i.e., between two separate sequences of audio data packets, each independently processed by one of the real-time and batch data processing modes. Each recording session is optionally activated by a user action on the first electronic device 202 or a voice activation detected from the audio data 230 collected by the electronic device 202.
[0031] More specifically, upon receiving the stream of audio data 230, the second electronic device 204 monitors the audio data 230 to determine whether the communication, computation, and storage capabilities of the second electronic device 204 can support processing of the audio data samples in a real-time data processing mode. For example, the second electronic device 204 determines in real-time whether a data sample latency of a data sample in the audio data 230 exceeds a latency tolerance, e.g., before or after the audio manager 234 processes the data sample in the audio data 230. In another example, the second electronic device 204 determines in real-time whether a rate of missing audio data samples of data samples in the audio data 230 exceeds a missing rate tolerance or whether a rate of out-of-order audio data samples of data samples in the audio data 230 exceeds an out-of-order rate tolerance, before or after the audio manager 234 processes the data sample in the audio data 230. In some implementations, the second electronic device 204 monitors its central processing unit (CPU) utilization and determines that it cannot support processing of audio data samples in real time if the CPU utilization exceeds a predetermined utilization rate (e.g., 85%). According to the determination result, the second electronic device 204 selects the audio manager 234 or the file-based audio data processing module 238 to process the stream of audio data 230 in real time or batch, respectively.
[0032] In some implementations, the second electronic device 204 is configured to locally process the output of the audio manager 234 or the processing module 238 to recognize a user request therefrom for the purpose of protecting user privacy, and optionally provide the processed output to the remote control application 236 for controlling the first electronic device 202. Alternatively, in some implementations, the second electronic device 204 is configured to pre-process the output of the audio manager 234 or the processing module 238 before transmitting it to the remote server system 106 to recognize a user request therefrom. Alternatively, in some implementations, for example when the second electronic device 204 is intended to be a low-cost device, the second electronic device 204 has limited voice recognition capabilities. The second electronic device 204 is configured to transmit the entire output of the audio manager 234 or the processing module 238 to the remote server system 106 and rely on the server system 106 to recognize a user request therefrom.
[0033] 3 is a schematic diagram illustrating an example audio data processing process 300 for switching from a real-time data processing mode to a batch data processing mode according to some implementations. An audio signal 220 is captured using a microphone 206 of a first electronic device 202 and sampled at an audio sampling rate to obtain a first sequence 302 of audio data samples and a second sequence 304 of audio data samples following the first sequence 302 of audio data samples. The first sequence 302 of audio data samples is transferred to the second electronic device 204 via a communication channel 208. In some implementations, the audio data samples in the first sequence 302 and the second sequence 304 are grouped into a plurality of audio data packets, each data packet including one or more consecutive audio data samples. Optionally, the second sequence 304 of audio data samples immediately follows the first sequence 302. Optionally, the second sequence 304 of audio data samples is separated from the first sequence 302 by an interruption. The audio data packets are streamed to the second electronic device 204. In some circumstances, one or more data packets are reordered or discarded from the first sequence 302 and the second sequence 304 in the course of being transferred over the communication channel 208.
[0034] The audio data packets 308 are grouped from the first sequence of audio data samples 302 and processed by the second electronic device 204 in a real-time processing mode, and erroneous, missing, and out-of-order data packets in the audio data packets 308 may both occur and be corrected. That is, one or more of the data packets 308 are discarded or reordered by the second electronic device 204 (specifically, by the audio manager 234). During or after processing the first sequence of audio data samples 302, the second electronic device 204 determines that at least one of the communication, computing, and storage capabilities of the second electronic device 204 cannot support processing of the audio data samples in the real-time data processing mode. In response to such a determination (at time t F At (a), the second electronic device 204 aborts the real-time data processing mode and initiates a batch data processing mode to process the second sequence of audio data samples 304. Specifically, the second electronic device 204 caches the second sequence of audio data samples 304 in the audio data buffer 214 and generates a data file 310 including the second sequence of audio data samples 304 in the batch data processing mode. In some implementations, the second electronic device 204 determines a corresponding capability to support processing of the audio data samples in the real-time data processing mode based on at least one of a data sample latency, a rate of missing audio data samples, a rate of out-of-order audio data samples, and a CPU utilization associated with the second electronic device 204.
[0035] In some implementations, the second electronic device 204 ceases processing the second sequence 304 of audio data samples in accordance with a determination that the communication, computational, and storage capabilities of the second electronic device 204 can support processing of the audio data samples in a real-time data processing mode. Alternatively, in some implementations, the second electronic device 204 is configured to limit the second sequence 304 to include a predefined number of audio data samples. The second electronic device 204 processes the second sequence 304 of audio data samples to be cached in the first data file 310 when the predefined number is reached. The second electronic device 204 then organizes the third sequence 306 of audio data samples, which immediately follows the second sequence 304 of audio data samples, into the second data file 312. The second data file 312 is transferred when the predefined number of audio data samples is included in the third sequence 306 of audio data samples or when it is determined that the second electronic device 204 can support processing of the audio data samples in a real-time data processing mode.
[0036] Under some circumstances, the number of current audio data samples included in the data file 310 or 312 has not reached the predefined number when it is determined that the second electronic device 204 can support processing of the audio data samples in a real-time data processing mode. According to the determination, the data file 310 or 312 may be immediately transferred with the current number of audio data samples. Optionally, the data file 310 or 312 is aborted from being transferred and the current number of audio data samples are reorganized into data packets for real-time audio data processing by the audio manager 234 of the second electronic device 204.
[0037] In some implementations, the data file 310 including the first sequence of processed audio data samples 308 and the second sequence of audio data samples is transferred (320) to the server system 106. The first sequence of processed audio data samples 302 has a first data transfer rate corresponding to a real-time data processing mode, and the second sequence of processed audio data samples 304 has a second data transfer rate corresponding to a batch data processing mode. The second data transfer rate is greater than the first data transfer rate. In some implementations, the first data transfer rate is less than an audio sampling rate of the audio signal 220, and the second data transfer rate is greater than the audio sampling rate.
[0038] 4 is a schematic diagram illustrating an example audio data processing process 400 for switching from a batch data processing mode to a real-time data processing mode according to some implementations. The audio signal 220 is captured using a microphone of the first electronic device 202 and sampled at an audio sampling rate to obtain a first sequence of audio data samples 402 and a second sequence of audio data samples 404 following the first sequence of audio data samples 402. Both the first sequence of audio data samples 402 and the second sequence of audio data samples 404 are transferred to the second electronic device 204 via the communication channel 208. The first sequence of audio data samples 302 is processed by the second electronic device 204 according to the batch data processing mode. The audio data samples in the first sequence 402 are cached in the audio data buffer 214 of the first electronic device 202 and organized into a data file 406.
[0039] During or after processing the first sequence of audio data samples 402, the second electronic device 204 detects or determines that the second electronic device 204 can support processing of the audio data samples in a real-time data processing mode. A In accordance with such determination (at B t , the second electronic device 204 continues to add more audio data samples to the first sequence 402 until it reaches a predefined number at time t . After the second electronic device 204 has completed caching the first sequence of audio data samples 402 in the data file 406, it begins to process the second sequence of audio data samples 404, which is collected subsequent to the first sequence of audio data samples 402, in a real-time data processing mode. Alternatively, in some implementations, the second electronic device 204 may continue to process the second sequence of audio data samples 404 in a real-time data processing mode (at time t C Pursuant to the determination at time t, the second electronic device 204 stops adding audio data samples to the first sequence 402, regardless of whether the number of audio data samples for the first sequence 402 has reached a predefined number. C 402, thereby terminating the batch data processing mode. The second electronic device 204 immediately begins transferring the second sequence of audio data samples 404 collected subsequent to the first sequence of audio data samples 402 in the real-time data processing mode. Furthermore, in some implementations (not shown), the second electronic device 204 may transfer the second sequence of audio data samples 404 collected subsequent to the first sequence of audio data samples 402 (e.g., at time t C At the determination in, the second electronic device 204 aborts processing the first sequence of audio data samples 402 in the batch data processing mode and immediately begins processing the first sequence of audio data samples 402 into data packets 408 in the real-time data processing mode. After the first sequence of audio data samples 402 is transferred, the second electronic device 204 continues processing the second sequence of audio data samples 404 in the real-time data processing mode.
[0040] Each of the first sequences of audio data samples 302 and 402 optionally starts or is in the middle of the stream of audio data 230 transmitted to the second electronic device 204. Similarly, each of the second sequences of audio data samples 304 and 404 and the third sequence of audio data samples 306 optionally is the last sequence in the stream of audio data 230 transmitted to the second electronic device 204 or is in the middle of the stream of audio data 230. Note that in some implementations, the first sequence of audio data samples 302 or 402 is not immediately preceded by the second sequence of audio data samples 304 or 404. The first and second sequences are captured during two separate recording sessions separated by an interruption. The second electronic device 204 determines whether it can support processing of the audio data samples in a real-time data processing mode during the interruption separating the two recording sessions.
[0041] FIG. 5 is a schematic diagram illustrating an example voice assistant process 500 initiated by a user action or voice input according to some implementations. The voice assistant process 500 is cooperatively implemented by the first electronic device 202 and the second electronic device 204. In some implementations, the first electronic device 202 includes a physical assistant button (e.g., the assistant button 606 in FIG. 6), and a user can request to start or end a voice assistant function via a user action on the assistant button. For example, the user presses the physical assistant button to start the voice assistant process 500 (also called a recording session). While the user presses the assistant button, audio signals are continuously collected by the microphone 206 of the first electronic device 202 for sampling, transferring, and recognizing voice data through the voice assistant process 500. The microphone 206 does not stop collecting audio signals until the user releases the assistant button to complete the corresponding recording session.
[0042] In some implementations, in response to detecting the first user action, the first electronic device 202 sends an assistant call request 502 to the remote control application 236 and the assistant application 232 of the second electronic device 204. In response to the assistant call request 502, the assistant application 232 verifies that the first electronic device 202 is authorized to implement the voice assistant process 500 with the second electronic device 204 and sends an instruction to start the assistant 504 to the remote control application 236. In response to the instruction to start the assistant 504, the remote control application 236 sends a microphone open instruction 506 to the first electronic device 202. After the microphone 206 of the first electronic device 202 is turned on, audio data samples 230 are collected and transferred to the second electronic device 204. After being transferred to the second electronic device 204, the audio data samples 230 are processed (520 and 530) by the audio manager 234 associated with a real-time data processing mode or by the file-based audio data processing module 238 associated with a batch data processing mode. In some implementations, the real-time data processing mode and the batch data processing mode alternate dynamically in the second electronic device 204 based on, for example, a data sample latency, a rate of missing audio data samples, a rate of out-of-order audio data samples, or a CPU utilization associated with the second electronic device 204. In some implementations, only one of the real-time data processing mode and the batch data processing mode is activated during each recording section that is activated in response to detecting a first user action.
[0043] In some implementations, the assistant application 232 sends a start recording instruction 508 to the remote control application 236 so that the remote control application 236 may control the first electronic device 202 to capture audio data collected by the first electronic device 202. The start recording instruction 508 is optionally issued together with the start assistant instruction 504 and configured to trigger the open microphone instruction 506. In response to the start recording instruction 508, the audio data samples 230 are recorded by the second electronic device 204 after being transferred from the first electronic device 202. Alternatively, in some embodiments, the start recording instruction 508' is issued by the assistant application after a subset of the audio data samples 230 is received by the second electronic device 204. The start recording instruction 508' may be issued based on the content of the subset of audio data samples (e.g., a user request in the content), and the subset of the audio data samples 230 is not recorded by the second electronic device 204.
[0044] In some implementations, the assistant application 232 sends a command 510 to stop recording to both the audio manager 234 and the remote control application 236, which may then issue a microphone close command 512 to control the first electronic device 202 to turn off the microphone 206. Alternatively, in some implementations, the user of the first electronic device 202 terminates the first user action that initiated the voice assistant process 500 or applies a second user action (e.g., release of the assistant button) to terminate the voice assistant process 500. In response to the second user action, the microphone 206 of the first electronic device 202 is turned off and a request 514 to terminate the assistant is sent to the remote control application 236 and the assistant application 232 of the second electronic device 204.
[0045] 3 and 4, the microphone 206 of the first electronic device 202 is turned on and configured to begin capturing an audio signal at a different time relative to a first user press. In some implementations, the first electronic device 202 receives a first user action requesting recording of an audio signal at a first time t1, and its microphone 206 is turned on to capture an audio signal immediately in response to the first user action at a second time t2, regardless of whether a microphone open command 506 is issued from the second electronic device 204. The second time t2 is substantially simultaneous with the first time t1. Alternatively, in some implementations, the first electronic device 202 receives a first user action requesting recording of an audio signal at a first time t1'. The audio signal is captured by the microphone at a second time t2 that precedes the first time t1'. Audio data corresponding to the duration between times t2 and t1′ is cached but still transferred to the second electronic device 204 in response to the first user action. Additionally, in some implementations, the first electronic device 202 receives a first user action requesting recording of an audio signal at a first time t1″ and begins capturing the audio signal at a second time t2 following the first time t1″, e.g., delayed by a predefined buffer time (e.g., 5 seconds) from the first time t1″.
[0046] Note that in some implementations, the first electronic device 202 waits to receive an audio data request including a microphone open command 506 from the second electronic device 204 during the duration between receiving the first user action time (t1, t1', or t1") and transferring the captured audio data to the second electronic device 204 at t2. The second electronic device 204 obtains authorization to send the audio data request in response to the first user action, which authorization is granted by the assistant application 232 of the second electronic device 204 or by the remote server system 106.
[0047] In some circumstances, the voice assistant process 500 is initiated by a voice input. The microphone 206 of the first electronic device 202 is configured to continuously collect audio signals and provide corresponding audio data to the second electronic device 204 regardless of whether the first electronic device 202 is in a sleep mode or an active mode. In the sleep mode, the audio data is not processed to recognize any user request for a voice assistant function used to control a media device or a user application until one or more predefined hot words (e.g., "Hi Google") are detected to enable the active mode. The second electronic device 204 is configured to locally detect one or more predefined hot words in the audio data and initiate the voice assistant process 500 in response to the detection of the one or more predefined hot words.
[0048] Referring to FIG. 5, in some embodiments, the assistant call request 502 includes one or more predefined hot words. The second electronic device 204 confirms the assistant call request 502 when it detects the hot words in the audio data received from the first electronic device 202. In response to the assistant call request 502, the assistant application 232 verifies that the first electronic device 202 is authorized to perform the voice assistant process 500 with the second electronic device 204 and sends an instruction 504 to start the assistant to the remote control application 236. The second electronic device 204 is controlled to operate in an active mode to locally process the audio data samples 230 in a real-time data processing mode or a batch data processing mode to identify additional user requests. Furthermore, in some implementations, the assistant application 232 identifies a user request to stop recording (e.g., "Google, stop recording" in FIG. 3 and FIG. 4) in the audio data 230 provided by the first electronic device. The assistant application 232 generates an instruction to stop recording 510 of the audio data samples 230 and sends the instruction 510 to the audio manager 234 and the remote control application 236 so that the remote control application 236 can issue a microphone close instruction 512 to control the first electronic device 202 to turn off its microphone. By these means, the voice assistant process 500 is started and ended based on the audio data captured by the first electronic device 202 without any physical user action.
[0049] 3, in one example, a request 314 to stop capturing the audio signal is recognized in the audio signal 220 and sent from the second electronic device 204 to the first electronic device 202. In response to the request 314, the second electronic device 204 completes processing of the first data file 310 including the second sequence of data samples 304, but aborts transferring the second data file 312 including the third sequence of data samples 306 that immediately follows the second sequence 304. In contrast, referring to FIG. 4, in another example, upon receiving the request 314, the second electronic device 204 aborts processing of subsequent data packets following the request 314 in the second sequence 404 of audio data samples.
[0050] FIG. 6 illustrates an exemplary remote control device 104 configured to transfer audio data to a television device, according to some implementations. A number of user buttons on the remote control device 104 include one or more of a power button 602, a home button 604, an assistant button 606, a cycle button 608 (also referred to as a play / cycle button), a previous button 610, a next button 612, a preview / backdrop button 614, and a volume control button 616. A user actuation on the power button 602 powers on a media device coupled to the remote control device 104 if the media device is off prior to the user actuation, and powers off the media device if the media device is on prior to the user actuation. A user actuation on the home button 604 controls the media device coupled to the remote control device 104 to display a home screen. For example, the home screen displays a particular advertisement clip or a randomly selected media program provided by a given Internet content channel. In some implementations, the power button 602 or the home button 604 functions as a quickcast button configured to enable instant playback of media content provided by a particular Internet content channel.
[0051] A user action on the assistant button 606 controls the microphone 206 integrated in the remote control device 104 to collect audio signals in the media environment 100, and a user request is extracted from the audio signal to control one or more media playback devices (e.g., TV device 102) located in the media environment 100. In some implementations, the microphone 206 of the remote control device 104 starts collecting audio signals from the environment 100 when the assistant button 606 is pressed for a first short period of time, and a second short press or user request is applied to stop collecting the audio signals. Alternatively, in some implementations, the microphone 206 of the remote control device 104 only collects audio signals from the environment 100 when the assistant button 606 is pressed, and stops collecting audio signals when the assistant button 606 is released. Additionally, in some implementations, the microphone 206 of the remote control device 104 continuously captures audio signals from the environment 100, and the audio signals include one or more predefined hot words and / or user requests. The user request may be used to control the remote control device 104 or one or more media devices or applications coupled to the remote control device 104 .
[0052] In this application, audio data is processed in a second electronic device 204 (e.g., a network-connected television device 102) having two audio data processing modes including a real-time data processing mode and a batch data processing mode. The second electronic device 204 determines whether the communication, cache, and processing capabilities of the second electronic device 204 can support processing of the audio data samples in the real-time data processing mode. In accordance with the determination that the second electronic device 204 can support processing of the audio data samples in the real-time data processing mode, the second electronic device 204 processes the subsequent audio data samples at a data packet level in accordance with the real-time data processing mode. In accordance with the determination that the second electronic device 204 cannot support processing of the audio data samples in the real-time data processing mode, the second electronic device 204 caches the subsequent audio data samples in a buffer of the second electronic device 204 and generates a data file including these audio data samples in the batch data processing mode. In some implementations, the determination and mode enablement are realized during a break between two recording sessions. Alternatively, in some implementations, the decision and mode switching is accomplished dynamically during the same recording session.
[0053] FIG. 7 is a flowchart of a method 700 for dynamically processing audio data in two audio data processing modes (from a real-time data processing mode to a batch data processing mode) according to some implementations. The method 700 is performed by the first electronic device 202 and the second electronic device 204 and is optionally governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the respective electronic devices. Each of the operations illustrated in FIG. 7 may correspond to instructions stored in a computer memory or a non-transitory computer-readable storage medium. The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, or other non-volatile memory device. The instructions stored in the computer-readable storage medium may include one or more of source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations of the method 700 may be combined and / or the order of some operations may be changed.
[0054] An audio signal is captured (702) using the microphone 206 of the first electronic device 202. The first electronic device 202 obtains (704) a first sequence of audio data samples 302 and a second sequence of audio data samples 304 following the first sequence of audio data samples 302 from the audio signal and transfers the first sequence of audio data samples 302 to the second electronic device 204 via the communication channel 208 according to the real-time data processing mode. The second electronic device 204 receives (706) the first sequence of audio data samples 302 and the second sequence of audio data samples 304 from the first electronic device 202 via the communication channel 208. During or after the first sequence of audio data samples 302, the second electronic device 204 determines (708) that the second electronic device 204 cannot support processing of the audio data samples in the real-time data processing mode. In accordance with a determination that the second electronic device 204 cannot support processing of the audio data samples in the real-time data processing mode, the second electronic device 204 caches (712) the second sequence of audio data samples 304 in a buffer of the second electronic device and generates (714) a data file 310 including the second sequence of audio data samples 304 in a batch data processing mode.
[0055] In some implementations, the second electronic device 204 transfers the processed first sequence of audio data samples and a data file including the second sequence of audio data samples to the server system 106 (e.g., 540 in FIG. 5). The first sequence of audio data samples 302 has a first data transfer rate corresponding to a real-time data processing mode, and the second sequence of audio data samples 304 has a second data transfer rate corresponding to a batch data processing mode. The second data transfer rate is greater than the first data transfer rate. Furthermore, in some examples, the audio signal captured by the first electronic device 202 is sampled at an audio sampling rate to obtain the first sequence of audio data samples 302 and the second sequence of audio data samples 304. The first data transfer rate is less than the audio sampling rate, and the second data transfer rate is greater than the audio sampling rate.
[0056] In some implementations, the audio data samples in the first sequence 302 are grouped into multiple audio data packets in a real-time data processing mode. Each audio data packet includes one or more consecutive audio data samples, optionally organized according to a consistent data format. The multiple audio data packets are streamed to the server system 106.
[0057] In some implementations, the second electronic device 204 is determined not to support forwarding of audio data samples in a real-time data processing mode based on at least one of a data sample latency, a rate of missing audio data samples, and a rate of out-of-order audio data samples associated with the second electronic device 204. Specifically, in one example, a data sample latency of a subset of the first sequence of processed audio data samples exceeds a latency tolerance. In another example, a rate of missing audio data samples of the first sequence of processed audio data samples exceeds a missing rate tolerance. In yet another example, a rate of out-of-order audio data samples of the first sequence of processed audio data samples exceeds an out-of-order rate tolerance.
[0058] In some implementations, a first user action requesting recording of an audio signal is received at the first electronic device 202. The audio signal is captured in response to the first user action. For example, the first user action is a press of an assistant button on the first electronic device 202. The press initiates a process for obtaining approval by the assistant application 232 of the second electronic device 204 or the server system 106. Upon receiving approval, the audio signal is captured, processed, and recorded. Specifically, in one example, the first electronic device 202 receives a first user action requesting recording of an audio signal at a first time t1', and capturing the audio signal is initiated at a second time t2 following the first time t1'. The second time t2 is delayed by a predefined buffer time from the first time. Specifically, in some circumstances, in response to the first user action, the first electronic device 202 receives an audio data request from the second electronic device 204. The second electronic device 204 is configured to obtain authorization to send the audio data request in response to a first user action. Transferring the first sequence of audio data samples is initiated in response to the audio data request.
[0059] In some implementations, the data file 310 includes a first data file 310. After generating the first data file 310, the second electronic device 204 continues in a batch data processing mode to generate a second data file 312 including a third sequence of audio data samples 306. The third sequence of audio data samples 306 immediately follows the second sequence of audio data samples 304 in the audio signal, with each of the second and third sequences of audio data samples 304 and 306 having a predefined number of data samples.
[0060] In some implementations, the second electronic device 204 is configured to forward the first sequence 302 and the second sequence 304 of processed audio data samples to the server system 106 for audio processing (e.g., speech recognition). The server system 106 hosts a virtual user domain that includes a user account. The first electronic device 202 and the second electronic device 204 are linked to the user account.
[0061] Alternatively, in some implementations, the audio signal includes one or more predefined hot words or user requests. The second electronic device 204 is configured to locally process the first and second sequences of audio data samples to identify the one or more predefined hot words or user requests in the audio signal. Furthermore, in some implementations, the user request includes a request to stop capturing the audio signal. The request to stop capturing the audio signal is recognized by the second electronic device 204 and provided thereby to the first electronic device 202. In response to the request, the first electronic device 202 aborts transferring the sequence of audio data samples following the second sequence of audio data samples 304.
[0062] In some implementations, while transferring the second sequence of data samples 304, the second electronic device 204 receives a second user action to stop capturing the audio signal. In response to the second user action, the second electronic device 204 aborts receiving the sequence of audio data samples that immediately follows the second sequence of audio data samples 304.
[0063] FIG. 8 is a flowchart of a method 800 for dynamically processing audio data in two audio data processing modes (from a batch data processing mode to a real-time data processing mode) according to some implementations. The method 800 is performed by the first electronic device 202 and the second electronic device 204 and is optionally governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the respective electronic devices. Each of the operations illustrated in FIG. 8 may correspond to instructions stored in a computer memory or a non-transitory computer-readable storage medium. The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, or other non-volatile memory device. The instructions stored in the computer-readable storage medium may include one or more of source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations of the method 800 may be combined and / or the order of some operations may be changed.
[0064] An audio signal is captured (802) using a microphone of the first electronic device 202. The first electronic device 202 obtains (804) a first sequence of audio data samples 402 and a second sequence of audio data samples 404 that follows the first sequence of audio data samples 402 in the audio signal. The second electronic device 204 receives (806) the first sequence of audio data samples 402 and the second sequence of audio data samples 404 from the first electronic device 202. The second electronic device 204 processes the first sequence of audio data samples 402 according to a batch data processing mode, including caching (810) the first sequence of audio data samples 402 in a buffer of the second electronic device and generating (812) a data file including the first sequence of audio data samples 402. During or after processing the first sequence of audio data samples, the second electronic device 204 determines that the second electronic device 204 can support processing of the audio data samples in a real-time data processing mode (814). In accordance with determining that the second electronic device 204 can support processing of the audio data samples in the real-time data processing mode, the second electronic device 204 processes the second sequence of audio data samples in accordance with the real-time data processing mode (816).
[0065] In some implementations, the audio data samples 402 and 404 are transferred to the server system 106. The first sequence of audio data samples 402 has a first data transfer rate corresponding to a real-time data processing mode, and the second sequence of audio data samples 404 has a second data transfer rate corresponding to a batch data processing mode. The first data transfer rate is greater than the second data transfer rate. Furthermore, in some examples, the second data transfer rate is slower than an audio sampling rate, and the first data transfer rate is greater than the audio sampling rate. In some implementations, the audio data samples in the second sequence 404 are grouped into multiple audio data packets in the real-time data processing mode, and the multiple audio data packets are optionally streamed in real time to the server system 106.
[0066] 7 and 8, in some implementations, the first electronic device 202 includes a remote control device 104 and the second electronic device 204 includes a networked TV device 102 configured to be controlled by the remote control device 104. In some implementations, the second electronic device 204 includes one or more processors and memory that stores one or more programs configured to implement an Android operating system and one or more user applications on the second electronic device 204. In some implementations, the first electronic device 202 is battery operated.
[0067] It should be understood that the particular order in which the operations in each of Figures 7 and 8 are described is merely illustrative and is not intended to indicate that the described order is the only order in which the operations may be performed. Those skilled in the art will recognize various ways to display information items and focus content within an integrated user interface as described herein. In addition, it should be noted that details described with respect to one of methods 700, 750, 800, and 850 are also applicable in a similar manner to any other of methods 700, 750, 800, and 850. For the sake of brevity, similar details will not be repeated.
[0068] It will also be understood that, although terms such as first, second, etc. are used in some examples herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another element. For example, a first electronic device may be referred to as a second electronic device, and similarly, a second electronic device may be referred to as a first electronic device, without departing from the scope of various implementations described. Although the first electronic device and the second electronic device are both electronic devices, they are not the same electronic device.
[0069] The terminology used in the description of the various implementations described herein is intended to describe the particular implementations only and is not intended to be limiting. As used in the description of the various described implementations and in the claims, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly dictates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprise", "include", "comprises", and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0070] As used herein, the word "if" is to be interpreted as meaning "when" or "when" or "response to determining" or "response to detecting" or "pursuant to a determination that," depending on the context, as appropriate. Similarly, the phrase "when determined" or "when [a defined condition or event] is detected" is to be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [a defined condition or event]" or "in response to detecting [a defined condition or event]" or "pursuant to a determination that [a defined condition or event] has been detected," depending on the context, as appropriate.
[0071] Although various figures show some logical stages in a particular order, stages that are not order dependent may be rearranged and other stages may be combined or broken down. While some rearrangements or other groupings are specifically mentioned, other rearrangements or groupings will be apparent to those of ordinary skill in the art, and thus the rearrangements and groupings presented herein are not an exhaustive list of alternatives. Furthermore, it should be recognized that the logical stages may be implemented in hardware, firmware, software, or any combination thereof.
[0072] The foregoing description has been described with reference to specific implementations for purposes of illustration. However, the above exemplary description is not intended to be exhaustive or to limit the claims to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The implementations have been selected to best explain the principles underlying the claims and their practical application, thereby enabling others skilled in the art to best use the implementations with various modifications as appropriate for the particular application contemplated.
Claims
1. 1. A method for processing audio data, comprising the steps of: and a second electronic device receiving from a first electronic device a first sequence of audio data samples and a second sequence of audio data samples, the second sequence of audio data samples following the first sequence of audio data samples in an audio signal captured by a microphone of the first electronic device, the method further comprising: said second electronic device processing said first sequence of audio data samples according to a real-time data processing mode; determining that the second electronic device cannot support processing of audio data samples in the real-time data processing mode; in accordance with said determining that the second electronic device cannot support processing of audio data samples in said real-time data processing mode, caching the second sequence of audio data samples in a buffer of the second electronic device, and generating a data file including the second sequence of audio data samples in a batch data processing mode.
2. transferring the processed first sequence of audio data samples and the data file including the second sequence of audio data samples to a server system; 2. The method of claim 1, wherein the first sequence of audio data samples has a first data rate corresponding to the real-time data processing mode, and the second sequence of audio data samples has a second data rate corresponding to the batch data processing mode.
3. The method of claim 2 , wherein the audio signal is sampled at an audio sampling rate to obtain the first sequence of audio data samples and the second sequence of audio data samples.
4. The method further includes grouping the audio data samples in the first sequence into a plurality of audio data packets, each audio data packet including one or more consecutive audio data samples, the method further comprising: The method of any one of claims 1 to 3, comprising streaming the plurality of audio data packets to a server system.
5. Determining that the second electronic device is incapable of supporting processing of the audio data samples in the real-time data processing mode includes: determining that a data sample latency of a subset of the first sequence of processed audio data samples exceeds a latency tolerance; determining that a rate of dropped audio data samples of the first sequence of processed audio data samples exceeds a drop rate tolerance; and 5. The method of claim 1, further comprising at least one of: determining that a rate of out-of-order audio data samples of the first sequence of processed audio data samples exceeds an out-of-order rate tolerance value.
6. 6. The method of claim 1, wherein the second electronic device is determined not to support transfer of audio data samples in the real-time data processing mode based on at least one of a data sample latency associated with the second electronic device, a rate of missing audio data samples, and a rate of out-of-order audio data samples.
7. 7. The method of claim 1, further comprising the first electronic device receiving a first user action requesting recording of the audio signal, the audio signal being captured in response to the first user action.
8. 5. The method of claim 1, wherein the first electronic device receives a first user action requesting recording of the audio signal at a first time, and capturing the audio signal is initiated at a second time following the first time.
9. and obtaining authorization to send a request for audio data in response to the first user action, and the second electronic device generating a request for audio data. The method of claim 7 or 8, wherein receiving the first sequence of audio data samples is in response to the request for audio data.
10. The data files include a first data file; 10. The method of claim 1, further comprising: after generating the first data file, the second electronic device continuing in the batch data processing mode to generate a second data file comprising a third sequence of audio data samples, the third sequence of audio data samples immediately following the second sequence of audio data samples in the audio signal, and each of the second sequence of audio data samples and the third sequence of audio data samples having a predefined number of data samples.
11. the second electronic device is configured to transfer the first sequence of processed audio data samples and the second sequence of audio data samples to a server system for audio processing; The server system hosts a virtual user domain containing user accounts; The method of any one of claims 1 to 10, wherein the first electronic device and the second electronic device are linked to the user account.
12. 12. The method of claim 1, wherein the audio signal comprises one or more predefined hot words or user requests, and the second electronic device is configured to locally process the first sequence of audio data samples and the second sequence of audio data samples to identify the one or more predefined hot words or user requests in the audio signal.
13. The user request includes a request to stop capturing the audio signal, and the method further comprises:
13. The method of claim 12, further comprising the second electronic device generating a request to stop capturing the audio signal, the first electronic device being configured to, in response to the request, abort transferring a sequence of audio data samples that follows the second sequence of audio data samples.
14. The method further comprises: receiving a second user action to cease capturing the audio signal while transferring the second sequence of data samples; and in response to the second user action, aborting receiving a sequence of audio data samples that immediately follows the second sequence of audio data samples.
15. 15. The method of any one of claims 1 to 14, wherein the first electronic device comprises a remote control device and the second electronic device comprises a networked television device configured to be controlled by the remote control device.
16. 16. The method of any one of claims 1 to 15, wherein the second electronic device includes one or more processors and memory storing one or more programs configured to implement an Android operating system and one or more user applications on the second electronic device.
17. 17. The method of claim 16, wherein the Android operating system includes an audio manager module having instructions for processing the first sequence of audio data samples according to the real-time data processing mode.
18. 17. The method of claim 16, wherein the one or more user applications of the second electronic device include a file-based audio data processing module for caching the second sequence of audio data samples in the buffer and generating the data file including the second sequence of audio data samples in the batch data processing mode.
19. 19. The method of claim 1, wherein it is determined that the second electronic device does not support processing of audio data samples in the real-time data processing mode during a recording session activated by a user action, and both the first sequence of data samples and the second sequence of data samples are processed during the recording session.
20. 17. The method of claim 1, wherein the second electronic device is determined to not support processing of audio data samples in the real-time data processing mode at an interruption separating two separate recording sessions activated by two separate user actions, and the first sequence of data samples and the second sequence of data samples are captured during the two separate recording sessions.
21. 1. A method for processing audio data, comprising the steps of: and a second electronic device receiving from a first electronic device a first sequence of audio data samples and a second sequence of audio data samples, the second sequence of audio data samples following the first sequence of audio data samples in an audio signal captured by a microphone of the first electronic device, the method further comprising: and the second electronic device processing the first sequence of audio data samples according to a batch data processing mode, which includes caching the first sequence of audio data samples in a buffer of the second electronic device, and generating a data file including the first sequence of audio data samples, the method further comprising: determining that the second electronic device is capable of supporting processing of audio data samples in a real-time data processing mode; and in accordance with the determination that the second electronic device can support processing of audio data samples in the real-time data processing mode, the second electronic device processing a second sequence of the audio data samples in accordance with the real-time data processing mode.
22. 1. An electronic device comprising: one or more processors; and a memory storing instructions which, when executed by said one or more processors, cause said one or more processors to carry out the method of any one of claims 1 to 6 and 21.
23. A program causing one or more processors to carry out the method according to any one of claims 1 to 21.
Citation Information
Patent Citations
Voice data collection system
JP2017220807A
Relay device, relay method, and program
JP2019169812A
Data structure pooling of voice-activated data packets
JP2019505023A
Analyzing Electronic Conversations for Presentation in Alternative Interfaces
JP2020521995A
