Synchronous playback of audio information received from other networks

Abstract

Disclosed herein are playback devices, groups of playback devices, and methods for operating and grouping playback devices so that playback devices in a group of playback devices communicating over a first network play back audio content received via a second network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 898,194, filed on September 10, 2019, entitled “Synchronizing Playback of Audio Information Received from Unreliable Audio Information Sources,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to consumer products and, more particularly, to methods, systems, products, features, services, and other elements related to media playback or some aspect thereof. Background Art

[0004] The options for accessing and listening to digital audio in an external speaker setup were limited until 2002, when Sonos began developing a new playback system. Sonos then filed one of its first patent applications in 2003, entitled "Method for Synchronizing Audio Playback between Multiple Networked Devices," and began offering its first media playback systems for sale in 2005. The Sonos wireless home audio system enables people to experience music from many sources via one or more networked playback devices. Through a software control application installed on a controller (e.g., a smart phone, tablet computer, computer, voice input device), people can play what they want in any room with a networked playback device. Media content (e.g., songs, podcasts, video sounds) can be streamed to the playback device so that each room with a playback device can play back corresponding different media content. In addition, rooms can be grouped together to play back the same media content synchronously, and / or the same media content can be listened to synchronously in all rooms. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The features, aspects and advantages of the presently disclosed technology may be better understood with reference to the following description, appended claims and drawings listed below. Those skilled in the relevant art will appreciate that the features shown in the drawings are for illustrative purposes and that variations including different and / or additional features and arrangements thereof are possible.

[0006] Figure 1A A partial cutaway view of an environment having a media playback system configured in accordance with aspects of the disclosed technology is shown.

[0007] Figure 1B Shows Figure 1A A schematic diagram of a media playback system and one or more networks.

[0008] Figure 1C A block diagram of a playback device is shown.

[0009] Figure 1D A block diagram of a playback device is shown.

[0010] Figure 1E A block diagram of a network microphone device is shown.

[0011] Figure 1F A block diagram of a network microphone device is shown.

[0012] Figure 1G A block diagram of a playback device is shown.

[0013] Figure 1H A partial schematic diagram of a control device is shown.

[0014] Figure 1I to Figure 1L A schematic diagram showing the corresponding media playback system area.

[0015] Figure 1M A schematic diagram showing the media playback system area.

[0016] Figure 2A A front isometric view of a playback device configured in accordance with aspects of the disclosed technology is shown.

[0017] Figure 2B Shows Figure 3A Front isometric view of a playback device without grille.

[0018] Figure 2C Shows Figure 2A Exploded view of the playback device.

[0019] Figure 3A A front view of a network microphone device configured in accordance with aspects of the disclosed technology is shown.

[0020] Figure 3B Shows Figure 3A Side isometric view of a web microphone device.

[0021] Figure 3C Shows Figure 3A and Figure 3B An exploded view of the network microphone device.

[0022] Figure 3D Shows Figure 3B An enlarged view of a portion of.

[0023] Figure 3EShows FIG. 3A to FIG. 3D Block diagram of a network microphone device.

[0024] Figure 3F A schematic diagram of an example speech input is shown.

[0025] Figure 4A-4D Schematic diagrams of control devices at various stages of operation are shown in accordance with aspects of the disclosed technology.

[0026] Figure 5 A front view of the control device is shown.

[0027] Figure 6 A message flow diagram of a media playback system is shown.

[0028] Figure 7 An example configuration of two playback devices playing audio received from a source in synchronization with each other is shown in accordance with some embodiments.

[0029] Figure 8 An example method performed by a playback device for playing audio content from multiple sources is shown in accordance with some embodiments.

[0030] Fig. 9 An example method performed by a playback device according to some embodiments is shown.

[0031] Fig.10 Example configurations of one or more playback devices and one or more control devices that exchange metadata associated with media content are shown.

[0032] The drawings are for the purpose of illustrating example embodiments, but a person skilled in the art will understand that the technology disclosed herein is not limited to the arrangements and / or instrumentalities shown in the drawings. DETAILED DESCRIPTION

[0033] 1. Overview

[0034] It is desirable for a media playback system to play (i) media content from many different sources (e.g., audio streaming services, video streaming services, audio or video sources, voice assistant services (VAS), doorbells, intercoms, telephones, and other sources), and (ii) media content via many different types of input interfaces (e.g., WIFI, Bluetooth, digital line-in, analog line-in, optical line-in, and other types of input interfaces). However, while a media playback system configured to play media content from many different sources via many different types of input interfaces enables a user to enjoy music, audio, video, and other media content from virtually any type of media source, coordinating the synchronized playback of audio content from such a diverse set of media sources and interface types between multiple playback devices (including audio, video, and audio / video playback devices) in a media playback system in a consistent and reliable manner presents various technical challenges.

[0035] Therefore, SONOS has been continuously developing new technologies to address this technical challenge and expand the types of input interfaces that can be used to receive media content. For example, SONOS has developed a virtual line input (VLI) to facilitate interoperability between media playback systems. Through VLI, a first media playback system from a first manufacturer may be able to synchronize playback with a second media playback system from a second, different manufacturer, which would otherwise be incompatible with the first media playback system. Therefore, users can form a synchronization group across any combination of multiple different media playback systems and playback devices. Such functionality is disclosed in particular in U.S. Patent No. 10,452,345, entitled “Media Playback System with Virtual Line-In,” issued on October 22, 2019, the entire contents of which are incorporated herein by reference.

[0036] Based on these previous innovations, SONOS has realized that traditional playback devices that support playback of audio received via Bluetooth (e.g., classic Bluetooth, Bluetooth Low Energy (BLE), etc.) are generally only able to synchronize playback of audio with other playback devices on the same Bluetooth network. Given the range limitations of Bluetooth networks, this architecture is based on the assumption that both the playback device and the audio source are very close. Although this assumption of device proximity may be correct in some multi-device situations (e.g., synchronized playback of audio through two wireless earbuds worn by a user, each earbud connected to a smart phone in the user's pocket via Bluetooth), the assumption of device proximity does not hold for other scenarios. For example, this range limitation is problematic for synchronized playback of audio across an entire space (e.g., a house, a commercial establishment, etc.), at least because the distance between the playback device and / or the audio source may be quite large (e.g., more than 10 meters). Therefore, it may not be feasible for all of the grouped playback devices and audio sources to be on a public Bluetooth network. As a result, traditional techniques for synchronized playback of audio via Bluetooth may not be easily used for true multi-room synchronized playback.

[0037] Therefore, aspects of the present disclosure relate to new technologies for synchronous playback of audio received via a personal area network (PAN) such as a Bluetooth network, which can solve one or more of the above-mentioned technical challenges. In some cases, the audio received by the playback device via the PAN is repacked and sent to other playback devices via different networks such as a wireless local area network (WLAN) for synchronous playback. In this case, the playback device can support synchronous playback with any other playback device connected to the same different network (e.g., the same WLAN). Therefore, the range limitation of the PAN can be eliminated by bridging the audio from the PAN to another network that may have a much larger range than the PAN by the playback device. As a result, the user can stream audio from the computing device to a first playback device (e.g., a playback device close to the user) via the PAN, and have the audio played back on any arbitrary combination of playback devices on the same WLAN as the first playback device. In addition, the user can stream audio to the first playback device via the PAN for playback by one or more second playback devices (e.g., playback devices that may not have physical radio hardware supporting communication via the PAN) that may not directly support playback via the PAN. For example, a user may stream audio via a Bluetooth connection to a portable playback device that supports Bluetooth communications for playback on one or more fixed playback devices that do not support Bluetooth communications.

[0038] Should be understood that the synchronous playback of the audio received by the PAN may produce additional technical challenges. In some cases, such an architecture may produce the technical problems related to the synchronization of metadata associated with the media between the devices connected to different networks. For example, a media playback system can have any number of control devices, and each control device can display metadata about the media played back on any set of one or more playback devices in the media playback system. Such a control device is unlikely to directly access the media content (and / or any metadata associated with this media content) and is sent to the PAN through which the playback device passes. As a result, such a control device will turn a blind eye to the media playback activities on the media playback system.

[0039] Thus, in some cases, a playback device that bridges media content from the PAN to the WLAN for synchronized playback with other playback devices on the WLAN may also function to bridge metadata associated with the media content from the PAN to the WLAN for transmission to other devices on the WLAN (e.g., control devices, playback devices, etc.). Thus, any device on the WLAN may output (e.g., via a display) the status of such playback activity (e.g., that one or more playback devices are playing back a particular audio track received via the PAN).

[0040] Another technical challenge that may arise from supporting synchronized playback of audio received over a PAN includes maintaining a high degree of reliability. For example, a media source can typically provide media content to a media playback system in a consistent manner via a reliable communication link (e.g., via a wired link (e.g., an electrical cable, optical cable, and other wired connection) and / or a robust wireless link (e.g., a WIFI wireless link). However, wireless links over a PAN typically do not provide as consistent and / or reliable media content as WIFI links, at least in part because WIFI links tend to operate at higher transmission powers and have better wireless link budgets than Bluetooth links. In addition, WIFI transmission protocols tend to have better error detection and correction and more robust and complex retransmission capabilities than Bluetooth transmission protocols.

[0041] When a media playback system inconsistently and / or unreliably receives media content from a media source, the media playback system may not be able to reliably coordinate the synchronized playback of the received media content in a consistent and reliable manner because frequent packet loss and lost packets of media content can cause playback devices in the media playback system to malfunction (e.g., play media content that is "out of sync" with each other, generate error messages, interrupt playback, etc.). When playback devices are playing audio content that is out of sync with each other, users hear an echo effect caused by playback devices playing back portions of the audio content at different times. And for audio content with corresponding video content, when a playback device plays audio content that is out of sync with a display (e.g., a television, video monitor, or other display device), viewers can see lip sync delays, where the audio sounds are out of sync with the corresponding video.

[0042] Therefore, in some cases, the playback device may treat the audio received via PAN (or other unreliable sources) differently from the audio received via another mechanism (e.g., wired connection, WLAN, etc.). In this case, the playback device that bridges the media content from PAN to other networks (e.g., WLAN) can perform one or more additional operations related to the media content received via other mechanisms to enhance the reliability of the media playback system. For example, the playback device that receives audio via PAN can intelligently identify the lost packets and create empty packets to send to other playback devices from synchronous playback, so that other playback devices receive consistent information streams. Additionally (or alternatively), if multiple packets are continuously discarded on PAN, the playback device that receives audio via PAN can adopt a resynchronization process to avoid echoing when the audio is restarted. In addition, the playback device can monitor the state of the connection with the media source through PAN, and if the connection quality is lower than a threshold (e.g., the signal strength is lower than a threshold, the number of packets lost or otherwise discarded exceeds a threshold, etc.), directly or indirectly output one or more messages to the user to warn the user. For example, the playback device may send one or more messages to the device acting as the media source (e.g., over the PAN or over another connection between the playback device and the media source such as a WLAN) that triggers a notification (e.g., a pop-up on the user's smartphone) to notify them of the poor connection quality and / or provide one or more suggestions for the user on how to improve the connection quality (e.g., suggesting that the user move the device acting as the media source closer to the playback device that is receiving audio over the PAN).

[0043] The technology disclosed and described herein overcomes these and other technical problems that may be generated when supporting the synchronous playback of audio content received via another network (e.g., another wireless network) such as a personal area network (PAN). In some embodiments disclosed herein, a first playback device receives a first data stream including a first series of frames, and the first series of frames include encoded audio information. In some embodiments, and as further described herein, the first playback device is and / or includes at least one or more functions of a group coordinator that performs a synchronization group. As described herein, a synchronization group is a group of playback devices configured to play audio synchronously with each other in a group. The synchronization group may include the first playback device, but may not include the first playback device. That is, the first playback device may perform group coordinator functions, but may not play audio synchronously with other playback devices in the synchronization group. In addition, although many examples disclosed herein describe frames, systems and methods are equally applicable to packets, units, or other data structures (and systems and methods may be implemented with them) that are suitable for sending data between a transmitter and a receiver.

[0044] In operation, each frame in the first data stream has a first framing format. In some embodiments, one or more frames in the first data stream also include a sequence number or other identifier that specifies the order of frames in the first data stream. Frames sent through a data packet network (e.g., Ethernet, WIFI, or other data network) may arrive out of order, so in some embodiments, the first playback device uses a sequence number or other identifier to reassemble the frames in the first data stream in the correct order before performing further processing. In some embodiments, the sequence number or other identifier that specifies the order of the frame is (or at least includes) a timestamp indicating the time when the frame was created. The frame creation time can be used as a sequence number based on the following assumption: the frame is created in the order in which it should be processed and / or played later. And as further described herein, in some protocols, a frame or packet includes both a sequence identifier and a timestamp, wherein the sequence identifier can be used to reorder frames / packets and / or identify lost frames / packets, and wherein the timestamp (alone or in combination with the timestamps of adjacent frames / packets) can be used to determine the duration required to play back audio information within the frame / packet.

[0045] After a first playback device (which may be a group coordinator of a synchronization group) receives each frame in a first data stream (incoming data stream), the first playback device extracts the encoded audio information from the each incoming frame. And as the first playback device continues to receive the first data stream (incoming data stream), the first playback device generates a second data stream (i.e., an outgoing data stream, sometimes referred to as a second series of frames) including the extracted audio information and sends it to at least one second playback device (and possibly to other playback devices). In operation, each frame in the second data stream (outgoing data stream) has a second framing format that is different from the first framing format of the frames in the first data stream (incoming data stream).

[0046] The frames in the second data stream include (i) portions of the encoded audio information extracted from the respective frames of the first data stream and (ii) playback timing information including the playback timing of the portions of the encoded audio information extracted from the first data stream. As described herein, in some embodiments, the playback timing information is new timing information (e.g., a new timestamp) generated by a first playback device (or a group coordinator) and sent to at least a second playback device for the purpose of coordinating the synchronous playback of audio content of the first playback device and at least a second playback device. In operation, the first playback device plays the audio content synchronously with at least a second playback device using the portions of the encoded audio information extracted from the first data stream and the playback timing (based on the encoded audio information).

[0047] In operation, a first playback device (i) generates respective frames of a second data stream (outgoing data stream) that include one or more portions of encoded audio information extracted by the first playback device from respective frames of the first data stream (incoming data stream) received by the first playback device only a few milliseconds previously, and (ii) plays the portion of the encoded audio information in synchronization with at least a second playback device only a few milliseconds after sending respective frames of the second data stream that include the portion of the encoded audio information to at least the second playback device. For example, in some embodiments, for each portion of the encoded audio information, and within only a few milliseconds (e.g., between about 20 milliseconds and about 300 milliseconds), the first playback device (i) receives a frame (of the first data stream) that includes the portion of the audio information, (ii) extracts the portion of the audio information from the frame (of the first data stream), (iii) generates and sends a frame (of the second data stream) to at least a second playback device, and (iv) plays the audio content in synchronization with at least the second playback device (and possibly with additional playback devices) (based on the portion of the audio information).

[0048] The above-mentioned ability to receive a first frame (or packet or unit) stream having a first framing format (or packet format or unit format) and generate a second frame (or packet or unit) stream having a second framing format (or packet format or unit format) enables a first playback device to receive audio information and (based on the audio information) play audio content from multiple audio sources, thereby enabling one aspect of the disclosed system and method to be implemented by enabling the first playback device to use a common format (e.g., the second framing format) to stream audio information and playback timing information to at least a second playback device (and potentially also to additional playback devices), regardless of (i) the format of the first frame stream, and / or (ii) sequence identifiers, timestamps, or other timing and / or sequence information in the first frame stream.

[0049] Because the first playback device generates and sends the second data stream while receiving the first data stream, and because the time between the first playback device receiving a portion of the audio information via the first data stream and sending the same portion of the audio information via the second data stream is very short (e.g., typically only about 20 milliseconds to about 300 milliseconds), any inconsistencies in the first data stream (e.g., delays, lost frames) and / or errors in the audio information received via the first data stream introduce technical challenges that inhibit the first playback device's ability to consistently and reliably generate and send the second series of frames to at least a second playback device.

[0050] Specifically, a first playback device may not send to a second playback device portions of the audio information that the first playback device did not receive and / or did not receive in sufficient time to process and stream to the second playback device for synchronized playback.

[0051] Therefore, to prevent inconsistencies, delays, and errors that may be present in the first data stream from causing similar inconsistencies in the second data stream, in some embodiments, the first playback device additionally (i) determines whether the first data stream is losing one or more portions of encoded audio information, and (ii) in response to determining that the first data stream is losing one or more portions of encoded audio information, generates one or more empty portions corresponding to the one or more lost portions of the encoded audio information. Losing one or more portions of encoded audio information includes losing a packet that includes a portion of the audio information because (i) a packet / frame that includes the audio information is not received, or (ii) a packet / frame that includes the audio information is received, but the audio information in the packet / frame is corrupted and cannot be used (e.g., an error check performed by the receiver determines that the audio information has errors).

[0052] And when the first playback device generates the second data stream, the first playback device inserts one or more empty portions into the second data stream at the location where the lost portion of the audio information should have existed. And, when the first playback device generates the playback timing of the audio information of the second data stream, the first playback device allocates the playback time of one or more corresponding empty portions in the second data stream. In addition, when the audio information and playback timing of the second data stream are used to play the audio content through at least the second playback device (based on the audio information in the second data stream), the first playback device uses (i) a portion of the audio information extracted from the first data stream, (ii) the playback timing of the portion of the audio information extracted from the first data stream, (iii) the empty portion corresponding to the lost portion of the audio information lost from the first data stream, and (iv) the playback timing of the empty portion inserted into the second data stream. In operation, the method enables (i) a first playback device to generate and send a consistent second data stream to at least a second playback device (and possibly additional playback devices), and (ii) at least the second playback device (and possibly additional playback devices) to receive a consistent data stream (i.e., the second data stream) from the first playback device.

[0053] In some embodiments, the first playback device additionally or alternatively monitors the incoming data stream to detect problems with the incoming data stream, such as (i) dropped packets (e.g., there are gaps in the sequence numbers of packets in the incoming data stream), or perhaps more than a threshold amount or number of dropped packets, and / or (ii) more than some threshold amount of packets that arrive too late to be processed and distributed to the second playback device (and possible additional playback devices) in a timely manner (e.g., the determined playback time of the audio in the packets is too close to the current time or may be in the past) for the second playback device (and possible additional playback devices) to play the audio content in the packets in synchronization with the other playback devices in the synchronization group.

[0054] In response to detecting one or more of the above-mentioned problems of the incoming data stream, the first playback device (i) sends one or more commands to other playback devices in the synchronization group, which causes those other playback devices to "refresh" (i.e., delete) at least some (and possibly all) of the buffers of the audio packets (including audio content) currently queued for playback, and (ii) reset the "start time" (i.e., reference start time) from which the playback timing of the audio content received via the incoming data stream is restarted. As described above and further described herein, the playback devices in the synchronization group use the playback timing generated by the first playback device (sometimes referred to as the group coordinator of the synchronization group) to play the audio content in sync with each other. In this way, by refreshing the playback device buffer of the queued audio packets for playback, the first playback device actually resynchronizes the audio playback through the synchronization group. In operation, resynchronizing the audio playback through the synchronization group in this way is similar to how the first playback device initially synchronizes the playback audio through the synchronization group. By resynchronizing playback of audio content through synchronization groups in the manner described herein, some embodiments are able to identify problems in an incoming data stream (received from an unreliable source) before lost and / or late packets cause problems and / or delays to propagate through the synchronization group and interrupt playback.

[0055] These and other aspects of the technical solutions disclosed herein enable a first playback device to (i) receive a first frame stream (having a first format) including encoded audio information and other control / timing information, (ii) generate a second frame stream (having a second format) including encoded audio information and playback timing information and send it to one or more additional playback devices (i.e., at least the second playback device and possible additional playback devices), and (iii) play audio content in synchronization with the one or more additional playback devices based on the encoded audio information, even when the first data stream is unreliable or inconsistent.

[0056] For example, some embodiments disclosed herein relate to a first playback device comprising one or more processors and a tangible, non-transitory computer-readable medium comprising instructions that, when executed, cause the first playback device to perform computing functions related to playing audio content in synchronization with one or more other playback devices.

[0057] Although some examples described herein may involve functions being performed by given actors (e.g., "users," "listeners," and / or other entities), it should be understood that this is for purposes of explanation only. The claims should not be interpreted as requiring any such example actors to perform an action unless the language of the claim itself expressly requires it.

[0058] In the drawings, like reference numerals identify substantially similar and / or identical elements. To facilitate discussion of any particular element, the most significant digit(s) in the reference numeral refer to the drawing in which the element is first introduced. For example, element 110a is shown in FIG. Figure 1A The present invention is introduced and discussed for the first time. Many of the details, dimensions, angles, and other features shown in the drawings are merely illustrative of specific embodiments of the disclosed technology. Therefore, other embodiments may have other details, dimensions, angles, and features without departing from the spirit or scope of the present disclosure. In addition, one of ordinary skill in the art will understand that other embodiments of the various disclosed technologies may be practiced without several of the details described below.

[0059] 2. Suitable operating environment

[0060] Figure 1A 1 is a partial cutaway view of a media playback system 100 distributed in an environment 101 (e.g., a house). The media playback system 100 includes one or more playback devices 110 (identified as playback devices 110a-110n, respectively), one or more network microphone devices ("NMDs") 120 (identified as NMDs 120a-120c, respectively), and one or more control devices 130 (identified as control devices 130a and 130b, respectively).

[0061] As used herein, the term "playback device" may generally refer to a network device configured to receive, process, and output the data of a media playback system. For example, a playback device may be a network device that receives and processes audio content. In some embodiments, the playback device includes one or more transducers or loudspeakers powered by one or more amplifiers. However, in other embodiments, the playback device includes one of a loudspeaker and an amplifier (or none of them). For example, the playback device may include one or more amplifiers that are configured to drive one or more loudspeakers outside the playback device via corresponding wires or cables.

[0062] Furthermore, as used herein, the term NMD (i.e., "network microphone device") may generally refer to a network device configured for audio detection. In some embodiments, the NMD is a standalone device configured primarily for audio detection. In other embodiments, the NMD is incorporated into a playback device (or vice versa).

[0063] The term “control device” may generally refer to a network device that is configured to perform functions related to facilitating user access, control, and configuration of the media playback system 100 .

[0064] Each of the playback devices 110 is configured to receive audio signals or data from one or more media sources (e.g., one or more remote servers, one or more local devices) and to play back the received audio signals or data as sound. One or more NMDs 120 are configured to receive spoken commands, and one or more control devices 130 are configured to receive user input. In response to the received spoken commands and / or user input, the media playback system 100 can play back audio via one or more playback devices 110. In some embodiments, the playback device 110 is configured to start playback of media content in response to a trigger. For example, one or more playback devices 110 can be configured to play back a morning playlist when an associated trigger condition (e.g., the presence of a user in the kitchen, detecting coffee machine operation) is detected. In some embodiments, for example, the media playback system 100 is configured to play back audio from a first playback device (e.g., playback device 100a) synchronously with a second playback device (e.g., playback device 100b). Referring to FIG. 10A and FIG. 10B , the playback device 110 may be configured to play back audio from a first playback device (e.g., playback device 100a) synchronously with a second playback device (e.g., playback device 100b). Figure 1B-1L The interactions between the playback device 110, the NMD 120, and / or the control device 130 of the media playback system 100 configured according to various embodiments of the present disclosure are described in more detail.

[0065] exist Figure 1A In the illustrated embodiment, environment 101 includes a home having multiple rooms, spaces, and / or playback areas, including (clockwise from top left) a master bathroom 101a, a master bedroom 101b, a second bedroom 101c, a family room or den 101d, an office 101e, a living room 101f, a dining room 101g, a kitchen 101h, and an outdoor balcony 101i. Although certain embodiments and examples are described below in the context of a home environment, the techniques described herein may be implemented in other types of environments. In some embodiments, for example, media playback system 100 may be implemented in one or more commercial settings (e.g., a restaurant, a shopping mall, an airport, a hotel, a retail store, or other store), one or more vehicles (e.g., a sports utility vehicle, a bus, a car, a ship, a boat, an airplane), multiple environments (e.g., a combination of home and vehicle environments), and / or other suitable environments where multi-zone audio may be desired.

[0066] The media playback system 100 may include one or more playback zones, some of which may correspond to rooms in the environment 101. The media playback system 100 may be established with one or more playback zones, and additional zones may be added or removed thereafter to form, for example, a playback environment. Figure 1AConfiguration shown. Each area can be named according to a different room or space (e.g., office 101e, master bathroom 101a, master bedroom 101b, second bedroom 101c, kitchen 101h, dining room 101g, living room 101f and / or outdoor balcony 101i). In some aspects, a single playback area can include multiple rooms or spaces. In some aspects, a single room or space can include multiple playback areas.

[0067] exist Figure 1A In the illustrated embodiment, the master bathroom 101a, the second bedroom 101c, the office 101e, the living room 101f, the dining room 101g, the kitchen 101h, and the outdoor courtyard 101i each include a playback device 110, and the master bedroom 101b and the study 101d include multiple playback devices 110. In the master bedroom 101b, the playback devices 110i and 110m can be configured to, for example, play back audio content in sync with individual playback devices in the playback devices 110, bound playback regions, merged playback devices, and / or any combination thereof. Similarly, in the study 101d, the playback devices 110h-110j can be configured to, for example, play back audio content in sync with individual playback devices in the playback devices 110, one or more bound playback devices, and / or one or more merged playback devices. Referring now to, for example Figure 1B and Figure 1E as well as Figure 1I-1M Additional details regarding bound and merged playback devices are described.

[0068] In some aspects, one or more playback areas in environment 101 can play different audio content. For example, a user can be grilling in courtyard 101i and listening to hip-hop music played by playback device 110c, while another user is preparing food in kitchen 101h and listening to classical music played by playback device 110b. In another example, a playback area can play the same audio content synchronously with another playback area. For example, a user can be listening to playback device 110f playing the same music as the hip-hop music played by playback device 110c on courtyard 101i in office 101e. In some aspects, playback devices 110c and 110f play hip-hop music synchronously, so that the user feels that the audio content is played seamlessly (or at least substantially seamlessly) when moving between different playback areas. Additional details regarding synchronization of audio playback between playback devices and / or zones may be found, for example, in U.S. Patent No. 8,234,395, entitled “System and method for synchronizing operations among a plurality of independently clocked digital data processing devices,” the entire contents of which are incorporated herein by reference.

[0069] a. Suitable media playback system

[0070] Figure 1B 1 is a schematic diagram of a media playback system 100 and a cloud network 102. Figure 1B Some devices of the media playback system 100 and the cloud network 102 are omitted. One or more communication links 103 (hereinafter referred to as “links 103 ”) communicatively couple the media playback system 100 and the cloud network 102 .

[0071] The link 103 may include, for example, one or more wired networks, one or more wireless networks, one or more wide area networks (WANs), one or more local area networks (LANs), one or more personal area networks (PANs), one or more telecommunication networks (e.g., one or more global system for mobile (GSM) networks, code division multiple access (CDMA) networks, long term evolution (LTE) networks, 5G communication networks, and / or other suitable data transmission protocol networks), etc. The cloud network 102 is configured to deliver media content (e.g., audio content, video content, photos, social media content) to the media playback system 100 in response to a request sent from the media playback system 100 via the link 103. In some embodiments, the cloud network 102 is also configured to receive data (e.g., voice input data) from the media playback system 100 and send commands and / or media content to the media playback system 100 accordingly.

[0072] The cloud network 102 includes computing devices 106 (identified as a first computing device 106a, a second computing device 106b, and a third computing device 106c, respectively). The computing devices 106 may include various computers or servers, such as a media streaming service server that stores audio and / or other media content, a voice service server, a social media server, a media playback system control server, etc. In some embodiments, one or more computing devices 106 include modules of a single computer or server. In certain embodiments, one or more computing devices 106 include one or more modules, computers, and / or servers. In addition, although the cloud network 102 is described above in the context of a single cloud network, in some embodiments, the cloud network 102 includes multiple cloud networks including communicatively coupled computing devices. In addition, although in Figure 1B 1. Cloud network 102 is shown in FIG. 1 as having three computing devices 106, but in some embodiments, cloud network 102 includes fewer than (or more than) three computing devices 106.

[0073] The media playback system 100 is configured to receive media content from the network 102 via a link 103. The received media content may include, for example, a uniform resource identifier (URI) and / or a uniform resource locator (URL). For example, in some examples, the media playback system 100 may stream, download, or otherwise obtain data from a URI or URL corresponding to the received media content. The network 104 communicatively couples the link 103 with at least a portion of the devices (e.g., one or more of the playback device 110, NMD 120, and / or control device 130) of the media playback system 100. The network 104 may include, for example, a wireless network (e.g., a WiFi network, a Bluetooth network, a Z-Wave network, a ZigBee network, and / or other suitable wireless communication protocol networks) and / or a wired network (e.g., a network including Ethernet, a universal serial bus (USB), and / or other suitable wired communications). As will be appreciated by those of ordinary skill in the art, as used herein, "WiFi" may refer to several different communication protocols that transmit at 2.4 megahertz (GHz), 5 GHz, and / or other suitable frequencies, including, for example, Institute of Electrical and Electronics Engineers (IEEE) 802.11a, 802.11b, 802.11g, 802.11n, 802.11ac, 802.11ac, 802.11ad, 802.11af, 802.11ah, 802.11ai, 802.11aj, 802.11aq, 802.11ax, 802.11ay, 802.15, and the like.

[0074] In some embodiments, network 104 includes a dedicated communication network, and media playback system 100 uses this dedicated communication network to send messages between various devices and / or to send media content to and from a media content source (e.g., one or more computing devices 106). In some embodiments, network 104 is configured to be accessible only to devices in media playback system 100, thereby reducing interference and competition with other home devices. However, in other embodiments, network 104 includes an existing home communication network (e.g., a home WIFI network). In some embodiments, link 103 and network 104 include one or more identical networks. In some aspects, for example, link 103 and network 104 include a telecommunications network (e.g., an LTE network, a 5G network). In addition, in some embodiments, media playback system 100 is implemented without network 104, and devices including media playback system 100 can communicate with each other, for example, via one or more direct connections, PANs, telecommunications networks, and / or other suitable communication links.

[0075] In some embodiments, audio content sources may be added or removed periodically in the media playback system 100. For example, in some embodiments, the media playback system 100 indexes media items when one or more media content sources are updated, added to, and / or removed from the media playback system 100. The media playback system 100 may scan identifiable media items in some or all folders and / or directories accessible to the playback device 110, and generate or update a media content database including metadata (e.g., title, artist, album, track length) and other associated information (e.g., URI, URL) for each identifiable media item found. For example, in some embodiments, the media content database is stored on one or more of the playback device 110, the network microphone device 120, and / or the control device 130.

[0076] exist Figure 1BIn the illustrated embodiment, playback devices 1101 and 110m comprise a group 107a. Playback devices 1101 and 110m may be temporarily or permanently placed in different rooms in a home and grouped together in group 107a based on user input received at a control device 130a and / or another control device 130 in the media playback system 100. When arranged in group 107a, playback devices 1101 and 110m may be configured to synchronously play back the same or similar audio content from one or more audio content sources. In some embodiments, for example, group 107a comprises a binding region, wherein playback devices 1101 and 110m respectively comprise a left audio channel and a right audio channel of multi-channel audio content, thereby producing or enhancing a stereo effect of the audio content. In some embodiments, group 107a comprises an additional playback device 110. However, in other embodiments, the media playback system 100 omits the arrangement of group 107a and / or other groupings of playback devices 110. Referring below to Figures 1I to 1M Additional details regarding groups and other arrangements of playback devices are described in greater detail.

[0077] The media playback system 100 includes NMDs 120a and 120d, each of which includes one or more microphones configured to receive voice utterances from a user. Figure 1B In the illustrated embodiment, NMD 120a is a standalone device and NMD 120d is integrated into playback device 110n. NMD 120a is configured, for example, to receive voice input 121 from user 123. In some embodiments, NMD 120a sends data associated with the received voice input 121 to a voice assistant service (VAS), which is configured to (i) process the received voice input data and (ii) send corresponding commands to media playback system 100. In some aspects, for example, computing device 106c includes a VAS (e.g., composed of Computing device 106c may receive voice input data from NMD 120a via network 104 and link 103. In response to receiving the voice input data, computing device 106c processes the voice input data (i.e., “Play Hey Jude by The Beatles”) and determines that the processed voice input includes a command to play a song (e.g., “Hey Jude”). Therefore, computing device 106c sends a command to media playback system 100 to play back “Hey Jude” by The Beatles from an appropriate media service on one or more playback devices 110 (e.g., via one or more computing devices 106).

[0078] b. Suitable playback equipment

[0079] Figure 1C 1 is a block diagram of a playback device 110a including an input / output 111. The input / output 111 may include an analog I / O 111a (e.g., one or more wires, cables, and / or other suitable communication links configured to carry analog signals) and / or a digital I / O 111b (e.g., one or more wires, cables, or other suitable communication links configured to carry digital signals). In some embodiments, the analog I / O 111a is an audio line input connection, including, for example, an automatically detected 3.5mm audio line input connection. In some embodiments, the digital I / O 111b includes a Sony / Philips Digital Interface Format (S / PDIF) communication interface and / or cable and / or a Toshiba Link (TOSLINK) cable. In some embodiments, the digital I / O 111b includes a High Definition Multimedia Interface (HDMI) interface and / or cable. In some embodiments, the digital I / O 111b includes one or more wireless communication links, including, for example, radio frequency (RF), infrared, WIFI, Bluetooth, or other suitable communication protocols. In some embodiments, analog I / O 111a and digital I / O 111b include interfaces (eg, ports, plugs, jacks) configured to receive connectors of cables for transmitting analog and digital signals, respectively, without necessarily including cables.

[0080] The playback device 110a can, for example, receive media content (e.g., audio content including music and / or other sounds) from a local audio source 105 via an input / output 111 (e.g., a cable, wire, PAN, Bluetooth connection, an adhoc wired or wireless communication network, and / or other suitable communication link). The local audio source 105 can include, for example, a mobile device (e.g., a smart phone, a tablet computer, a laptop computer) or other suitable audio components (e.g., a television, a desktop computer, an amplifier, a phonograph, a Blu-ray player, a memory storing digital media files). In some aspects, the local audio source 105 includes a local music library on a smart phone, a computer, a network-attached storage (NAS), and / or other suitable devices configured to store media files. In some embodiments, one or more of the playback device 110, NMD 120, and / or the control device 130 include a local audio source 105. However, in other embodiments, the media playback system completely omits the local audio source 105. In some embodiments, the playback device 110a does not include an input / output 111, and receives all audio content via the network 104.

[0081] The playback device 110a also includes an electronic device 112, a user interface 113 (e.g., one or more buttons, knobs, dials, touch-sensitive surfaces, displays, touch screens), and one or more transducers 114 (hereinafter referred to as "transducers 114"). The electronic device 112 is configured to receive the input / output 111, the network 104 ( Figure 1B ) receives audio from an audio source (e.g., local audio source 105), amplifies the received audio, and outputs the amplified audio for playback via one or more transducers 114. In some embodiments, playback device 110a optionally includes one or more microphones 115 (e.g., a single microphone, multiple microphones, microphone array) (hereinafter referred to as "microphone 115"). In some embodiments, for example, playback device 110a with one or more optional microphones 115 can operate as an NMD that is configured to receive voice input from a user and perform one or more operations accordingly based on the received voice input.

[0082] exist Figure 1C In the illustrated embodiment, the electronic device 112 includes one or more processors 112a (hereinafter referred to as "processor 112a"), a memory 112b, a software component 112c, a network interface 112d, one or more audio processing components 112g (hereinafter referred to as "audio component 112g"), one or more audio amplifiers 112h (hereinafter referred to as "amplifier 112h"), and a power source 112i (e.g., one or more power supplies, power cords, power outlets, batteries, inductor coils, Power over Ethernet (POE) interfaces, and / or other suitable power sources). In some embodiments, the electronic device 112 optionally includes one or more other components 112j (e.g., one or more sensors, a video display, a touch screen, a battery charging base).

[0083] The processor 112a may include a clock-driven computing component configured to process data, and the memory 112b may include a computer-readable medium (e.g., a tangible, non-transitory computer-readable medium, a data storage device loaded with one or more software components 112c) configured to store instructions for performing various operations and / or functions. The processor 112a is configured to execute the instructions stored on the memory 112b to perform one or more operations. The operations may include, for example, causing the playback device 110a to receive audio from an audio source (e.g., one or more computing devices 106a-106c ( Figure 1B)) and / or another playback device 110 to retrieve the audio information. In some embodiments, the operation also includes causing the playback device 110a to send the audio information to another playback device 110a and / or other devices (e.g., one of the NMDs 120). Certain embodiments include the operation of causing the playback device 110a to pair with another of the one or more playback devices 110 to enable a multi-channel audio environment (e.g., stereo pair, bound region).

[0084] The processor 112a may also be configured to perform operations to cause the playback device 110a to synchronize playback of audio content with another of the one or more playback devices 110. As will be appreciated by one of ordinary skill in the art, during synchronized playback of audio content on multiple playback devices, a listener will preferably be unable to perceive a time delay difference between the playback of audio content by the playback device 110a and the other one or more other playback devices 110. Additional details regarding synchronization of audio playback between playback devices may be found, for example, in U.S. Pat. No. 8,234,395, which is incorporated herein by reference.

[0085] In some embodiments, the memory 112b may also be configured to store data associated with the playback device 110a, such as one or more regions and / or region groups of which the playback device 110a is a member, audio sources accessible to the playback device 110a, and / or playback queues with which the playback device 110a (and / or another of the one or more playback devices) may be associated. The stored data may include one or more state variables that are periodically updated and used to describe the state of the playback device 110a. The memory 112b may also include data associated with the state of one or more other devices of the media playback system 100 (e.g., playback device 110, NMD 120, control device 130). In some aspects, for example, the state data is shared during a predetermined time interval (e.g., every 5 seconds, every 10 seconds, every 60 seconds) between at least a portion of the devices of the media playback system 100 so that one or more devices have the latest data associated with the media playback system 100.

[0086] The network interface 112d is configured to facilitate communication between the playback device 110a and a data network (eg, link 103 and / or network 104). Figure 1B)) on one or more other devices on the network interface 112d. The network interface 112d is configured to send and receive data corresponding to media content (e.g., audio content, video content, text, photos) and other signals (e.g., non-transient signals) including digital packet data, which includes an Internet Protocol (IP)-based source address and / or an IP-based destination address. The network interface 112d can parse the digital packet data so that the electronic device 112 correctly receives and processes data destined for the playback device 110a.

[0087] exist Figure 1C In the illustrated embodiment, the network interface 112d includes one or more wireless interfaces 112e (hereinafter referred to as "wireless interfaces 112e"). The wireless interface 112e (e.g., a suitable interface including one or more antennas) can be configured to wirelessly communicate with one or more other devices (e.g., one or more of other playback devices 110, NMD 120, and / or control device 130) that are communicatively coupled to the network 104 according to a suitable wireless communication protocol (e.g., WIFI, Bluetooth, LTE). Figure 1B ). In some embodiments, the network interface 112d optionally includes a wired interface 112f (e.g., an interface or socket configured to receive a network cable such as an Ethernet, USB-A, USB-C, and / or Thunderbolt cable), which is configured to communicate through a wired connection with other devices according to a suitable wired communication protocol. In some embodiments, the network interface 112d includes the wired interface 112f and does not include the wireless interface 112e. In some embodiments, the electronic device 112 completely excludes the network interface 112d and sends and receives media content and / or other data via another communication path (e.g., input / output 111).

[0088] The audio processing component 112g is configured to process and / or filter data including media content received by the electronic device 112 (e.g., via the input / output 111 and / or the network interface 112d) to generate an output audio signal. In some embodiments, the audio processing component 112g includes, for example, one or more digital-to-analog converters (DACs), audio preprocessing components, audio enhancement components, digital signal processors (DSPs), and / or other suitable audio processing components, modules, circuits, etc. In some embodiments, one or more of the audio processing components 112g may include one or more subcomponents of the processor 112a. In some embodiments, the electronic device 112 omits the audio processing component 112g. In some aspects, for example, the processor 112a executes instructions stored on the memory 112b to perform audio processing operations to generate an output audio signal.

[0089] The amplifier 112h is configured to receive and amplify the audio output signal generated by the audio processing component 112g and / or the processor 112a. The amplifier 112h may include an electronic device and / or component configured to amplify the audio signal to a level sufficient to drive one or more transducers 114. In some embodiments, for example, the amplifier 112h includes one or more switches or class D power amplifiers. However, in other embodiments, the amplifier includes one or more other types of power amplifiers (e.g., linear gain power amplifiers, class A amplifiers, class B amplifiers, class AB amplifiers, class C amplifiers, class D amplifiers, class E amplifiers, class F amplifiers, class G and / or class H amplifiers and / or other suitable types of power amplifiers). In some embodiments, the amplifier 112h includes a suitable combination of two or more of the aforementioned types of power amplifiers. In addition, in some embodiments, each amplifier in the amplifier 112h corresponds to each transducer in the transducer 114. However, in other embodiments, the electronic device 112 includes a single amplifier in the amplifier 112h, which is configured to output the amplified audio signal to multiple transducers 114. In some other embodiments, the electronic device 112 omits the amplifier 112h.

[0090] The transducer 114 (e.g., one or more speakers and / or speaker drivers) receives the amplified audio signal from the amplifier 112h and presents or outputs the amplified audio signal as sound (e.g., audible sound waves having a frequency between approximately 20 hertz (Hz) and 20 kilohertz (kHz)). In some embodiments, the transducer 114 may include a single transducer. However, in other embodiments, the transducer 114 includes a plurality of audio transducers. In some embodiments, the transducer 114 includes more than one type of transducer. For example, the transducer 114 may include one or more low-frequency transducers (e.g., subwoofers, woofers), mid-frequency transducers (e.g., mid-range transducers, mid-frequency woofers), and one or more high-frequency transducers (e.g., one or more tweeters). As used herein, "low frequency" may generally refer to audible frequencies below about 500 Hz, "mid-frequency" may generally refer to audible frequencies between about 500 Hz and about 2 kHz, and "high frequency" may generally refer to audible frequencies above 2 kHz. However, in some embodiments, one or more of the transducers 114 include transducers that do not adhere to the aforementioned frequency ranges. For example, one of the transducers 114 may include a mid-bass transducer configured to output sound at a frequency between about 200 Hz and about 5 kHz.

[0091] For example, SONOS currently offers (or has offered) certain playback devices for sale, including, for example, "SONOS ONE", "PLAY:1", "PLAY:3", "PLAY:5", "PLAYBAR", "CONNECT:AMP", "CONNECT", and "SUB". Other suitable playback devices may be used in addition or alternatively to implement the playback devices of the example embodiments disclosed herein. In addition, those of ordinary skill in the art will understand that the playback devices are not limited to the examples described herein or the SGNGS product offerings. In some embodiments, for example, one or more playback devices 110 include wired or wireless headphones (e.g., ear-hook headphones, on-ear headphones, in-ear headphones). In other embodiments, one or more of the playback devices 110 include a docking station and / or an interface configured to interact with a docking station for a personal mobile media playback device. In some embodiments, the playback device can be integrated into another device or component, such as a television, lighting fixture, or some other device for use indoors or outdoors. In some embodiments, the playback device omits a user interface and / or one or more transducers. For example, Figure 1D is a block diagram of a playback device 110 p including input / output 111 and electronics 112 , but without a user interface 113 or transducer 114 .

[0092] Figure 1E is a block diagram of a bundled playback device 110q including a playback device 110i (eg, a subwoofer) ( Figure 1A ) Sound-bound playback device 110a ( Figure 1C In the illustrated embodiment, playback devices 110a and 110i are separate playback devices in playback device 110 housed in separate housings. However, in some embodiments, bound playback device 110q includes a single housing that houses both playback devices 110a and 110i. Bound playback device 110q may be configured to be coupled to an unbound playback device (e.g., Figure 1C playback device 110a) and / or a paired or bound playback device (e.g., Figure 1B In some embodiments, for example, playback device 110a is a full-range playback device configured to present low-frequency, mid-frequency, and high-frequency audio content, and playback device 110i is a subwoofer configured to present low-frequency audio content. In some aspects, when bound to a first playback device, playback device 110a is configured to present only mid-frequency and high-frequency components of specific audio content, while playback device 110i presents low-frequency components of specific audio content. In some embodiments, bound playback device 110q includes additional playback devices and / or another bound playback device. Figure 2A-Figure 3D Additional playback device embodiments are described in more detail.

[0093] c. Suitable Network Microphone Device (NMD)

[0094] Figure 1F NMD 120a( Figure 1A and Figure 1B ). The NMD 120a includes one or more voice processing components 124 (hereinafter referred to as "voice components 124") and a playback device 110a ( Figure 1C ) described above, including a processor 112a, a memory 112b, and a microphone 115. The NMD 120a optionally includes a playback device 110a ( Figure 1C ) other components in, for example, user interface 113 and / or transducer 114. In some embodiments, NMD 120a is configured as a media playback device (e.g., one or more playback devices 110) and also includes, for example, an audio processing component 112g ( Figure 1C ), transducer 114, and / or one or more of other playback device components. In some embodiments, NMD 120a includes an Internet of Things (IoT) device, such as a thermostat, an alarm panel, a fire and / or smoke detector, etc. In some embodiments, NMD 120a includes microphone 115, voice processing 124, and the above Figure 1B In some aspects, for example, the NMD 120a includes a processor 112a and a memory 112b ( Figure 1B ), while omitting one or more other components of the electronic device 112. In some embodiments, the NMD 120a includes additional components (e.g., one or more sensors, cameras, thermometers, barometers, hygrometers).

[0095] In some embodiments, the NMD may be integrated into the playback device. Figure 1G 1 is a block diagram of a playback device 110r including an NMD 120d. The playback device 110r may include many or all of the components of the playback device 110a, and also includes a microphone 115 and a voice processing 124 ( Figure 1F ). The playback device 110r optionally includes an integrated control device 130c. The control device 130c may include, for example, a user interface (eg, Figure 1B However, in other embodiments, the playback device 110r is controlled from another control device (e.g., Figure 1B The control device 130a) receives the command. Figure 3A-Figure 3FAdditional NMD embodiments are described in more detail.

[0096] Reference again Figure 1F , the microphone 115 is configured to receive signals from the environment (e.g. Figure 1A The system may acquire, capture and / or receive sound in an environment 101) and / or a room in which the NMD 120a is located. The received sound may include, for example, voice utterances, audio played back by the NMD 120a and / or another playback device, background voices, ambient sounds, etc. The microphone 115 converts the received sound into electrical signals to generate microphone data. The voice processing 124 receives and analyzes the microphone data to determine whether there is voice input in the microphone data. The voice input may include, for example, an activation word followed by a voice utterance including a user request. As will be understood by those of ordinary skill in the art, an activation word is a word or other audio prompt that represents a user voice input. For example, in a query When using VAS, the user might say the activation word "Alexa". Other examples include VAS's "Ok, Google" and the “Hey, Siri” by VAS.

[0097] After detecting the activation word, the voice processing 124 monitors the microphone data for user requests accompanying the voice input. The user request may include, for example, controlling a third-party device (e.g., a thermostat (e.g., thermostats), lighting equipment (e.g., PHILIPS lighting equipment) or media playback equipment (e.g. For example, a user may say the activation word "Alexa" followed by the utterance "set the thermostat to 68 degrees" to set a home (e.g., Figure 1A The user may speak the same activation word followed by the utterance "light up the living room" to turn on the lighting in the living room area of ​​the home. The user may similarly speak the activation word followed by a request to play a particular song, album, or music playlist on a playback device in the home. Additional description of receiving and processing voice input data may be found below. Figure 3A-Figure 3F A more detailed description can be found in .

[0098] d. Appropriate control equipment

[0099] Figure 1H is the control device 130a ( Figure 1A and Figure 1B). As used herein, the term "control device" may be used interchangeably with "controller" or "control system". The control device 130a is configured to receive user input related to the media playback system 100, and in response, cause one or more devices in the media playback system 100 to perform an action or operation corresponding to the user input. In the illustrated embodiment, the control device 130a comprises a smart phone (e.g., an iPhone) on which a media playback system controller application software is installed. TM , Android phone). In some embodiments, the control device 130a includes, for example, a tablet computer (e.g., iPad TM ), a computer (e.g., a laptop computer, a desktop computer), and / or other suitable devices (e.g., a television, a car audio head unit, an IoT device). In some embodiments, the control device 130a comprises a dedicated controller for the media playback system 100. In other embodiments, as described above with respect to Figure 1G As depicted, control device 130a is integrated into another device in media playback system 100 (eg, one or more of playback device 110, NMD 120, and / or other suitable devices configured to communicate over a network).

[0100] The control device 130a includes an electronic device 132, a user interface 133, one or more speakers 134, and one or more microphones 135. The electronic device 132 includes one or more processors 132a (hereinafter referred to as "processor 132a"), a memory 132b, a software component 132c, and a network interface 132d. The processor 132a can be configured to perform functions related to facilitating user access, control, and configuration of the media playback system 100. The memory 132b can include a data storage device that can be loaded with one or more software components that can be executed by the processor 302 to perform those functions. The software component 132c can include an application and / or other executable software configured to facilitate control of the media playback system 100. The memory 112b can be configured to store, for example, the software component 132c, the media playback system controller application software, and / or other data associated with the media playback system 100 and the user.

[0101] The network interface 132d is configured to facilitate network communications between the control device 130a and one or more other devices in the media playback system 100 and / or one or more remote devices. In some embodiments, the network interface 132d is configured to operate according to one or more suitable communication industry standards (e.g., infrared, radio, wired standards including IEEE 802.3, wireless standards including IEEE 802.11a, 802.11b, 802.11g, 802.11n, 802.11ac, 802.15, 4G, LTE). The network interface 132d can be configured, for example, to provide communication to the playback device 110, the NMD 120, the other control devices in the control device 130, Figure 1B The network interface 132d may send and / or receive data from one of the computing devices 106 of the network interface 132d, as well as devices including one or more other media playback systems. The data sent and / or received may include, for example, playback device control commands, state variables, playback regions and / or region group configurations. For example, based on user input received at the user interface 133, the network interface 132d may send playback device control commands (e.g., volume control, audio playback control, audio content selection) from the control device 304 to one or more playback devices. The network interface 132d may also send and / or receive configuration changes, such as adding / removing one or more playback devices to / from a region; adding / removing one or more regions to / from a region group; forming a bound or merged player; separating one or more playback devices from a bound or merged player, etc. Reference may be made below. Figures 1I to 1M Find additional descriptions of regions and groups.

[0102] The user interface 133 is configured to receive user input and can facilitate control of the media playback system 100. The user interface 133 includes media content art 133a (e.g., album cover, lyrics, video), playback status indicators 133b (e.g., elapsed time and / or remaining time indicators), a media content information area 133c, a playback control area 133d, and a region indicator 133e. The media content information area 133c may include a display of relevant information (e.g., title, artist, album, genre, release year) about the media content currently being played and / or the media content in the queue or playlist. The playback control area 133d may include selectable (e.g., via touch input and / or via a cursor or other suitable selector) icons to cause one or more playback devices in a selected playback region or region group to perform a playback action, such as play or pause, fast forward, fast rewind, skip to next, skip to previous, enter / exit shuffle mode, enter / exit repeat mode, enter / exit cross-fade mode, etc. The playback control area 133d may also include selectable icons for modifying equalization settings, playback volume, and / or other suitable playback actions. In the illustrated embodiment, the user interface 133 includes icons presented on a smart phone (e.g., an iPhone). TM , Android phone). However, in some embodiments, other user interfaces of varying formats, styles, and interaction sequences may alternatively be implemented on one or more network devices to provide similar control access to the media playback system.

[0103] One or more speakers 134 (e.g., one or more transducers) can be configured to output sound to the user of the control device 130a. In some embodiments, the one or more speakers include individual transducers, and each transducer is configured to output low frequency, medium frequency and / or high frequency accordingly. In some aspects, for example, the control device 130a is configured as a playback device (e.g., one of the playback devices 110). Similarly, in some embodiments, the control device 130a is configured as an NMD (e.g., one of the NMD120), which receives voice commands and other sounds via one or more microphones 135.

[0104] The one or more microphones 135 may include, for example, one or more condenser microphones, electret condenser microphones, dynamic microphones, and / or other suitable types of microphones or transducers. In some embodiments, two or more microphones 135 may be arranged to capture location information of an audio source (e.g., speech, audible sound) and / or be configured to facilitate filtering of background noise. Additionally, in some embodiments, the control device 130a is configured to function as a playback device and an NMD. However, in other embodiments, the control device 130a omits one or more speakers 134 and / or one or more microphones 135. For example, the control device 130a may include a device (e.g., a thermostat, an IoT device, a network device) that includes a portion of the electronic device 132 and a user interface 133 (e.g., a touch screen) without any speakers or microphones. Referring below to Figure 4A-4D and Figure 5 Additional control device embodiments are described in more detail.

[0105] e. Appropriate playback device configuration

[0106] Figure 1L to Figure 1M An example configuration of playback devices in regions and region groups is shown. Figure 1M In one example, a single playback device may belong to one area. For example, the second bedroom 101c ( Figure 1A ) may belong to region C. In some embodiments described below, multiple playback devices may be "bound" to form a "bound pair" that together form a single region. For example, playback device 1101 (e.g., a left playback device) may be bound to playback device 1101 (e.g., a left playback device) to form region A. Bound playback devices may have different playback responsibilities (e.g., channel responsibilities). In another embodiment described below, multiple playback devices may be merged to form a single region. For example, playback device 110h (e.g., a front playback device) may be merged with playback device 110i (e.g., a subwoofer) and playback devices 110j and 110k (e.g., left surround speakers and right surround speakers, respectively) to form a single region D. In another example, playback devices 110g and 110h may be merged to form a merged group or region group 108b. The merged playback devices 110g and 110h may not be specifically assigned different playback responsibilities. That is, in addition to being able to synchronously play audio content, the merged playback devices 110h and 110i can also individually play audio content as if they were not merged.

[0107] Each region in the media playback system 100 can be provided as a single user interface (UI) entity for control. For example, region A can be provided as a single entity named Master Bathroom. Region B can be provided as a single entity named Master Bedroom. Region C can be provided as a single entity named Second Bedroom.

[0108] Bound playback devices can have different playback responsibilities, for example, responsibilities for certain audio channels. Fig. 1I As shown, playback devices 1101 and 110m can be bound to produce or enhance the stereo effect of audio content. In this example, playback device 1101 can be configured to play the left channel audio component, and playback device 110k can be configured to play the right channel audio component. In some embodiments, this stereo binding can be referred to as "pairing."

[0109] Additionally, the bundled playback device may have additional and / or different corresponding speaker drivers. Figure 1J As shown, a playback device named front 110h can be bound to a playback device named subwoofer 110i. The front device 110h can be configured to render the mid-high frequency range, and the subwoofer device 110i can be configured to render the low frequencies. However, when not bound, the front device 110h can be configured to render the entire frequency range. As another example, Figure 1K The front device 110h and the subwoofer device 110i are shown to be further bound to the left playback device 110j and the right playback device 110k, respectively. In some embodiments, the left device 110j and the right device 102k can be configured to form the surround or "satellite" channels of the home theater system. The bound playback devices 110h, 110i, 110j and 110k can form a single D area ( Figure 1M ).

[0110] The merged playback devices may not yet have playback responsibilities assigned, and each may present the full range of audio content that the corresponding playback device is capable of playing back. However, the merged devices may be represented as a single UI entity (i.e., a region as discussed above). For example, playback devices 110a and 110n in the master bathroom have a single UI entity for region A. In one embodiment, playback devices 110a and 110n may each output the full range of audio content that each corresponding playback device 110a and 110n is capable of synchronously playing back.

[0111] In some embodiments, an NMD may be bound or merged with another device to form a region. For example, NMD 120b may be bound with playback device 110e, and the two together form region F, which is named "Living Room". In other embodiments, a standalone network microphone device may be in a region by itself. However, in other embodiments, a standalone network microphone device may not be associated with a region. Additional details on associating a network microphone device and a playback device as a designated device or a default device may be found, for example, in previously referenced U.S. Patent Application No. 15 / 438,749.

[0112] You can group regions of individual, bonded, and / or merged devices together to form region groups. For example, refer to Figure 1M , region A can be grouped with region B to form a regional group that includes the two regions. Similarly, region G can be grouped with region H to form regional group 108b. As another example, region A can be grouped with one or more other regions CI. AI regions can be grouped and ungrouped in a variety of ways. For example, three, four, five or more (e.g., all) of the AI ​​regions can be grouped together. As described in previously referenced U.S. Patent No. 8,234,395, when grouped together, regions of individual and / or bound playback devices can play back audio in sync with each other. Playback devices can be dynamically grouped and ungrouped to form new or different groups that play back audio content in sync.

[0113] In various embodiments, the regions in the environment may be the default names of the regions within the group, or a combination of region names within the region group. For example, the region group 108 b may be assigned a name, such as “restaurant+kitchen”, such as Figure 1M In some embodiments, the regional group may also be named a unique name selected by the user.

[0114] Certain data may be stored in the memory of the playback device as one or more state variables (e.g., Figure 1C The memory 112b) is periodically updated and used to describe the state of the playback zone, playback device, and / or zone group associated therewith. The memory may also include data associated with the state of other devices of the media system and may be shared between devices from time to time so that one or more of the devices have the latest data associated with the system.

[0115] In some embodiments, the memory may store instances of various variable types associated with states. The variable instances may be stored with identifiers (e.g., tags) corresponding to the types. For example, certain identifiers may be a first type "a1" for identifying playback devices for a region, a second type "b1" for identifying playback devices that can be bound in the region, and a third type "c1" for identifying a region group to which the region may belong. As a related example, an identifier associated with the second bedroom 101c may indicate that the playback device is the only playback device in region C, rather than a playback device in a region group. An identifier associated with the study may indicate that the study is not grouped with other regions, but includes bound playback devices 110h-110k. An identifier associated with the restaurant may indicate that the restaurant is part of the restaurant+kitchen region group 108b, and that devices 110b and 110d are grouped together ( Figure 1L ). Since the kitchen is part of the restaurant+kitchen region group 108b, the identifier associated with the kitchen may indicate the same or similar information. Other example region variables and identifiers are described below.

[0116] In yet another example, the media playback system 100 may store other associated variables or identifiers representing regions and region groups, such as identifiers associated with regions, such as Figure 1M As shown. Regions can refer to clusters of regional groups and / or clusters of regions that are not in regional groups. For example, Figure 1M An upper region 109a including the AD region and a lower region 109b including the EI region are shown. In one aspect, regions can be used to call clusters of region groups and / or regions that share one or more regions and / or region groups of another cluster. On the other hand, this is different from a region group, which does not share regions with another region group. Other examples of techniques for implementing regions can be found in, for example, U.S. Application No. 15 / 682,506, entitled “Room Association Based on Name,” filed on August 21, 2017, and U.S. Patent No. 8,483,853, entitled “Controlling and manipulating groupings in a multi-zone media system,” filed on September 11, 2007. Each of these applications is incorporated herein by reference in its entirety. In some embodiments, the media playback system 100 may not implement regions, in which case the system may not store variables associated with regions.

[0117] 3. Example System and Equipment

[0118] Figure 2Ais a front isometric view of a playback device 210 configured in accordance with aspects of the disclosed technology. Figure 2B is a front isometric view of playback device 210 without grille 216e. Figure 2C is an exploded view of the playback device 210. Figure 2A-2C , playback device 210 includes housing 216, which includes upper portion 216a, right or first side 216b, lower portion 216c, left or second side 216d, grille 216e and rear portion 216f. A plurality of fasteners 216g (e.g., one or more screws, rivets, clips) attach frame 216h to housing 216. Cavity 216j ( Figure 2C ) is configured to receive the frame 216h and the electronic device 212. The frame 216h is configured to carry a plurality of transducers 214 (in Figure 2B 214a-214f, respectively.) The electronic device 212 (e.g., Figure 1C The electronic device 112 is configured to receive audio content from an audio source and send electrical signals corresponding to the audio content to the transducer 214 for playback.

[0119] The transducer 214 is configured to receive electrical signals from the electronic device 112 and is further configured to convert the received electrical signals into audible sounds during playback. For example, the transducers 214a-214c (e.g., tweeters) can be configured to output high frequency sounds (e.g., sound waves with a frequency greater than about 2 kHz). The transducers 214d-214f (e.g., mid-range woofers, woofers, mid-range speakers) can be configured to output sounds at frequencies lower than the frequencies of the transducers 214a-214c (e.g., sound waves with a frequency lower than about 2 kHz). In some embodiments, the playback device 210 includes a Figure 2A-2C The transducers shown are different from multiple transducers. For example, as shown below Figures 3A-3C As further described in detail, playback device 210 may include less than six transducers (e.g., one, two, three). However, in other embodiments, playback device 210 includes more than six transducers (e.g., nine, ten). In addition, in some embodiments, all or a portion of transducers 214 are configured to operate as a phased array to desirably adjust (e.g., narrow or widen) the radiation pattern of transducer 214, thereby changing the user's perception of the sound emitted from playback device 210.

[0120] exist Figure 2A-2CIn the illustrated embodiment of the present invention, filter 216i is axially aligned with transducer 214b. Filter 216i can be configured to desirably attenuate a predetermined frequency range output by transducer 214b to improve the sound quality and perceived sound level of the co-output of transducers 214. However, in some embodiments, playback device 210 omits filter 216i. In other embodiments, playback device 210 includes one or more additional filters aligned with transducer 214b and / or at least another of transducers 214.

[0121] Figure 3A and Figure 3B 3 are front isometric side view and right isometric side view, respectively, of an NMD 320 configured in accordance with an embodiment of the disclosed technology. Figure 3C This is an exploded view of the NMD 320. Figure 3D yes Figure 3B , including the user interface 313 of the NMD 320. First, refer to Figure 3A-3C , NMD 320 includes a housing 316 including an upper portion 316a, a lower portion 316b, and a middle portion 316c (e.g., a grille). A plurality of ports, holes, or apertures 316d in the upper portion 316a allow sound to pass to one or more microphones 315 ( Figure 3C ). One or more microphones 316 are configured to receive sound via hole 316d and generate an electrical signal based on the received sound. In the illustrated embodiment, the frame 316e ( Figure 3C ) surrounds cavities 316f and 316g, which are configured to accommodate a first transducer 314a (e.g., a tweeter) and a second transducer 314b (e.g., a mid-range woofer, a midrange speaker, a woofer), respectively. However, in other embodiments, NMD 320 includes a single transducer, or more than two (e.g., two, five, six) transducers. In some embodiments, NMD 320 omits transducers 314a and 314b entirely.

[0122] Electronic equipment 312( Figure 3C ) includes components configured to drive transducers 314a and 314b and further configured to analyze audio data corresponding to electrical signals generated by one or more microphones 315. For example, in some embodiments, electronic device 312 includes the above reference Figure 1C In some embodiments, the electronic device 312 includes many or all of the components of the electronic device 112 described above. Figure 1FComponents are depicted, for example, one or more processors 112a, memory 112b, software components 112c, network interfaces 112d, etc. In some embodiments, electronic device 312 includes additional suitable components (eg, proximity sensors or other sensors).

[0123] Reference Figure 3D , the user interface 313 includes a plurality of control surfaces (e.g., buttons, knobs, capacitive surfaces), the control surface including a first control surface 313a (e.g., the previous control), a second control surface 313b (e.g., the next control) and a third control surface 313c (e.g., play and / or pause control). The fourth control surface 313d is configured to receive touch inputs corresponding to activation and deactivation of one or more microphones 315. The first indicator 313e (e.g., one or more light-emitting diodes (LEDs) or another suitable illuminator) can be configured to illuminate only when one or more microphones 315 are activated. The second indicator 313f (e.g., one or more LEDs) can be configured to remain lit during normal operation, and flash or otherwise change from lit to indicate that voice activity is detected. In some embodiments, the user interface 313 includes additional or fewer control surfaces and illuminators. In one embodiment, for example, the user interface 313 includes a first indicator 313e, and the second indicator 313f is omitted. Furthermore, in some embodiments, NMD 320 includes a playback device and a control device, and user interface 313 includes a user interface of the control device.

[0124] Refer to Figure 3A-3D , NMD 320 is configured to receive voice commands from one or more adjacent users via one or more microphones 315. Figure 1B As described, one or more microphones 315 can acquire, capture or record nearby sounds (e.g., an area within 10m or less of NMD 320) and send electrical signals corresponding to the recorded sounds to electronic device 312. Electronic device 312 can process the electrical signals and can analyze the resulting audio data to determine the presence of one or more voice commands (e.g., one or more activation words). In some embodiments, for example, after detecting one or more suitable voice commands, NMD 320 is configured to send a signal to another device and / or a remote server (e.g., Figure 1BThe remote server may analyze the audio data, determine an appropriate action based on the voice command, and send a message to the NMD 320 to perform the appropriate action. For example, the user may say "Sonos, play Michael Jackson". The NMD 320 may record the user's voice utterance via one or more microphones 315, determine the presence of a voice command, and send a message to the remote server (e.g., Figure 1B The remote server may analyze the audio data and determine an action corresponding to the command. The remote server may then send a command to the NMD 320 to perform the determined action (e.g., play back audio content related to Michael Jackson). The NMD 320 may receive the command and play back audio content related to Michael Jackson from the media content source. Figure 1B As described, suitable content sources may include those via a LAN (e.g., Figure 1B 104) communicatively coupled to a device or storage of NMD 320, a remote server (e.g., Figure 1B However, in some embodiments, the NMD 320 determines and / or performs one or more actions corresponding to one or more voice commands without intervention or involvement of an external device, computer, or server.

[0125] Figure 3E 31 is a functional block diagram illustrating additional features of NMD 320 in accordance with aspects of the present disclosure. NMD 320 includes components configured to facilitate voice command capture, including: a voice activity detector component 312k, a beamformer component 312l, an acoustic echo cancellation (AEC) and / or self-sound suppression component 312m, an activation word detector component 312n, and a speech / voice conversion component 312o (e.g., speech to text and text to speech). Figure 3E In the illustrated embodiment, the aforementioned components 312k-312o are shown as separate components. However, in some embodiments, one or more of the components 312k-312o are subcomponents of the processor 112a.

[0126] The beamforming and self-sound suppression components 3121 and 312m are configured to detect audio signals and determine various aspects of the speech input represented in the detected audio signals, such as direction, amplitude, spectrum, etc. The voice activity detector active component 312k is operably coupled to the beamforming and AEC components 3121 and 312m and is configured to determine one or more directions in which voice activity is likely to have occurred in the detected audio signal. Potential voice directions can be identified by monitoring metrics that distinguish voice from other sounds. Such metrics can include, for example, energy within the voice band relative to background noise and entropy within the voice band (which is a measure of spectral structure). As will be appreciated by those of ordinary skill in the art, voice typically has lower entropy than most common background noise.

[0127] The activation word detector component 312n is configured to monitor and analyze the received audio to determine whether there are any activation words (e.g., wake-up words) in the received audio. The activation word detector component 312n can use an activation word detection algorithm to analyze the received audio. If the activation word detector 312n detects an activation word, the NMD 320 can process the voice input contained in the received audio. The example activation word detection algorithm accepts audio as input and provides an indication of whether an activation word is present in the audio. Many first-party and third-party activation word detection algorithms are known and commercially available. For example, an operator of a voice service can make its algorithm available to third-party devices. Alternatively, the algorithm can be trained to detect certain activation words. In some embodiments, the activation word detector 312n runs multiple activation word detection algorithms on the received audio simultaneously (or substantially simultaneously). As described above, different voice services (e.g., AMAZON's APPLE or MICROSOFT ) can use different activation words to invoke their respective voice services. To support multiple services, the activation word detector 312n can run the received audio through the activation word detection algorithm in parallel for each supported voice service.

[0128] The speech / text conversion component 312o can facilitate processing by converting speech in the speech input into text. In some embodiments, the electronic device 312 may include speech recognition software trained for a specific user or a specific set of users associated with a family. Such speech recognition software can implement a speech processing algorithm tuned to a specific voice profile. Tuning to a specific voice profile can require less computationally intensive algorithms than traditional voice activation services, which are typically sampled from a wide range of user bases and various requests that are not targeted at a media playback system.

[0129] Figure 3F328 is a schematic diagram of an example voice input 328 captured by NMD 320 according to aspects of the present disclosure. Voice input 328 may include an activation word portion 328a and a voice utterance portion 328b. In some embodiments, activation word 557a may be a known activation word, such as, for example, the word "AMAZON". 328a. In some embodiments, the voice input 328 may not include an activation word. In some embodiments, the network microphone device may output an audible and / or visual response upon detecting the activation word portion 328a. Additionally or alternatively, the NMB may output an audible and / or visual response after processing the voice input and / or a series of voice inputs.

[0130] The speech utterance portion 328b may include, for example, one or more spoken commands (recognized as first command 328c and second command 328e, respectively) and one or more spoken keywords (recognized as first keyword 328d and second keyword 328f, respectively). In one example, the first command 328c may be a command to play music, such as a specific song, album, playlist, etc. In this example, the keyword may be a region or regions (e.g., Figure 1A In some examples, the speech utterance portion 328b may include other information, such as a detected pause (e.g., a non-speech period) between words spoken by the user, such as Figure 3F This pause can mark the location of a separate command, keyword or other information spoken by the user within the speech utterance portion 328b.

[0131] In some embodiments, the media playback system 100 is configured to temporarily reduce the volume of the audio content it is playing while detecting the activation word portion 557a. The media playback system 100 can restore the volume after processing the voice input 328, such as Figure 3F Such a process may be referred to as ducking, an example of which is disclosed in U.S. Patent Application No. 15 / 438,749, which is incorporated herein by reference in its entirety.

[0132] Figure 4A-4D is a control device 430 (e.g., Figure 1H A schematic diagram of a control device 130a, a smart phone, a tablet, a dedicated control device, an IoT device, and / or another suitable device) is shown, showing corresponding user interface displays in various operating states. The first user interface display 431a ( Figure 4A) includes a display name 433a (i.e., "Room"). The selected group area 433b displays audio content information (e.g., artist name, track name, album cover) of the audio content played back in the selected group and / or region. Group areas 433c and 433d display the corresponding group and / or region name, as well as audio content information of the audio content played back or next to be played back in the playback queue of the corresponding group or region. The audio content area 433e includes information related to the audio content in the selected group and / or region (i.e., the group and / or region indicated in the selected group area 433b). The lower display area 433f is configured to receive touch input to display one or more other user interface displays. For example, if the user selects "Browse" in the lower display area 433f, the control device 430 can be configured to output a second user interface display 431b ( Figure 4B ), the second user interface display 431b includes multiple music services 433g (e.g., Spotify, Tunein Radio, Apple Music, Pandora, Amazon, TV, local music, line-in), through which the user can browse media content, and the user can play the music via one or more playback devices (e.g., Figure 1A Alternatively, if the user selects “My Sonos” in the lower display area 433f, the control device 430 may be configured to output a third user interface display 431c ( Figure 4C ). The first media content area 433h may include a graphical representation (e.g., album cover) corresponding to each album, station, or playlist. The second media content area 433i may include a graphical representation (e.g., album cover) corresponding to each song, track, or other media content. If the user selects the graphical representation 433j ( Figure 4C ), the control device 430 can be configured to start playing back the audio content corresponding to the graphical representation 433j and output a fourth user interface display 431d, which includes an enlarged version of the graphical representation 433j, media content information 433k (e.g., track name, artist, album), transport controls 433m (e.g., play, previous, next, pause, volume), and an indication 433n of the currently selected group and / or region name.

[0133] Figure 55 is a schematic diagram of a control device 530 (e.g., a laptop computer, a desktop computer). The control device 530 includes a transducer 534, a microphone 535, and a camera 536. The user interface 531 includes a transmission control area 533a, a playback status area 533b, a playback area area 533c, a playback queue area 533d, and a media content source area 533e. The transmission control area includes one or more controls for controlling media playback, including, for example, volume, previous, play / pause, next, repeat, random play, track position, fade in and fade out, equalization, etc. The audio content source area 533e includes a list of one or more media content sources, from which a user can select a media item for playback and / or add to a playback queue.

[0134] The playback area 533b may include a representation of the playback area within the media playback system 100 ( Figure 1A and Figure 1B ). In some embodiments, the graphical representation of the playback region may be selectable to bring up additional selectable icons to manage or configure the playback region in the media playback system, for example, the creation of bound regions, the creation of regional groups, the separation of regional groups, the renaming of regional groups, etc. In the illustrated embodiment, a "grouping" icon may be provided within each graphical representation of the playback region. The "grouping" icon provided within the graphical representation of a particular region may be selectable to bring up options for selecting one or more other regions in the media playback system to be grouped with the particular region. Once grouped, playback devices in a region that has been grouped with the particular region may be configured to play audio content synchronously with playback devices in the particular region. Similarly, a "grouping" icon may be provided within the graphical representation of a regional group. In the illustrated embodiment, the "grouping" icon may be selectable to bring up options for deselecting one or more regions in the regional group to be removed from the regional group. In some embodiments, the control device 530 includes other interactions and implementations for grouping and degrouping regions via the user interface 531. In some embodiments, the representation of the playback zone in the playback zone area 533b may be dynamically updated when the playback zone or zone group configuration is modified.

[0135] The playback status area 533c includes a graphical representation of the audio content currently being played, previously played, or scheduled to be played next in the selected playback region or region group. The selected playback region or region group can be visually distinguished on the user interface (e.g., in the playback region area 533b and / or the playback queue area 533d). The graphical representation can include the track name, artist name, album name, album year, track length, and other relevant information, which can be useful for the user to understand when the user controls the media playback system 100 via the user interface 531.

[0136] Playback queue area 533d can comprise the graphical representation of the audio content in the playback queue associated with selected playback region or regional group.In certain embodiments, each playback region or regional group can be associated with playback queue, and this playback queue comprises the corresponding information of zero or more audio items of this playback region or regional group playback.For example, each audio item in playback queue can comprise uniform resource identifier (URI), uniform resource locator (URL) or some other identifiers, and it can be used for finding and / or obtaining audio items from local audio content source or networked audio content source by the playback device in playback region or regional group, and may be played back for playback device.In certain embodiments, for example, playlist can be added to playback queue, in this case, the information corresponding to each audio item in playlist can be added to playback queue.In certain embodiments, the audio items in playback queue can be saved as playlist.In certain embodiments, when playback region or regional group are continuously playing streaming audio content (for example, internet radio, it can continuously play until stopped), rather than when having the discrete audio items of playback duration, playback queue can be empty or filled but "unused". In some embodiments, a playback queue may include Internet radio and / or other streaming audio content items and be "in use" while a playback zone or zone group is playing those items.

[0137] When a playback zone or zone group is "grouped" or "ungrouped", the playback queues associated with the affected playback zone or zone group can be cleared or reassociated. For example, if a first playback zone including a first playback queue is grouped with a second playback zone including a second playback queue, the established zone group can have an associated playback queue (which is initially empty), contain audio items from the first playback queue (e.g., if the second playback zone is added to the first playback zone), or contain audio items from the second playback queue (e.g., if the first playback zone is added to the second playback zone), or contain a combination of audio items from both the first playback queue and the second playback queue. Subsequently, if the established zone group is ungrouped, the resulting first playback zone can be reassociated with the previous first playback queue, or associated with a new playback queue that is empty or contains audio items from a playback queue associated with a zone group established before the established zone group was ungrouped. Similarly, the resulting second playback zone can be reassociated with the previous second playback queue, or associated with a new playback queue that is empty or contains audio items from a playback queue associated with a zone grouping established before the established zone grouping was ungrouped.

[0138] Figure 6 is a diagram showing a media playback system 100 ( Figure 1A-1M)'s message flow diagram for data exchange between devices.

[0139] At step 650a, the media playback system 100 receives an indication of selected media content (e.g., one or more songs, albums, playlists, podcasts, videos, radio stations) via the control device 130a. The selected media content may include, for example, locally stored on one or more devices connected to the media playback system (e.g., Figure 1C media items on the audio source 105) and / or stored on one or more media service servers ( Figure 1B In response to receiving the indication of the selected media content, the control device 130a sends a message 651a ( Figures 1A-1C ) to add the selected media content to the playback queue on the playback device 110a.

[0140] At step 650b, playback device 110a receives message 651a and adds the selected media content to a playback queue for playback.

[0141] At step 650c, control device 130a receives an input corresponding to the command for playback selected media content. In response to receiving an input corresponding to the command for playback selected media content, control device 130a sends message 651b to playback device 110a, causing playback device 110a to playback selected media content. In response to receiving message 651b, playback device 110a sends message 651c to first computing device 106a, to request selected media content. In response to receiving message 651c, first computing device 106a sends message 651d, which includes data (e.g., audio data, video data, URL, URI) corresponding to the requested media content.

[0142] At step 650d, the playback device 110a receives the message 651d having data corresponding to the requested media content and plays back the associated media content.

[0143] At step 650e, playback device 110a optionally causes one or more other devices to playback the selected media content. In one example, playback device 110a is one of the binding areas of two or more players ( Figure 1M). Playback device 110a can receive the selected media content and send all or part of the media content to other devices in the binding zone. In another example, playback device 110a is a coordinator of the group and is configured to send and receive timing information from one or more other devices in the group. One or more other devices in the group can receive the selected media content from first computing device 106a and, in response to a message from playback device 110a, start playback of the selected media content so that all devices in the group synchronously play back the selected media content.

[0144] IV. Overview of Example Embodiments

[0145] As described above, it is desirable for a media playback system to play (i) media content from many different sources (e.g., audio streaming services, video streaming services, audio or video sources, voice assistant services (VAS), doorbells, intercoms, telephones, and other sources), and (ii) media content via many different types of input interfaces (e.g., WIFI, Bluetooth, digital line-in, analog line-in, optical line-in, and other types of input interfaces). However, while a media playback system configured to play media content from many different sources via many different types of input interfaces enables a user to enjoy music, audio, video, and other media content from virtually any type of media source, coordinating the synchronized playback of audio content from such a diverse set of media sources and interface types between multiple playback devices (including audio, video, and audio / video playback devices) in a media playback system in a consistent and reliable manner presents a technical challenge because the consistency and reliability of media content delivery varies by media source and input interface type.

[0146] In some example embodiments, a first playback device (i) receives a first frame stream including encoded audio information from an audio source (described below) via a Bluetooth connection, (ii) extracts the encoded audio information from the received Bluetooth frames, (iii) generates a second frame stream including the extracted audio information and playback timing information of the extracted encoded audio information and sends it to one or more additional playback devices via a WIFI connection, and (iv) decodes the audio information and plays the audio content in synchronization with the one or more additional playback devices using the encoded audio information and playback timing in the second frame stream. This example embodiment is sometimes referred to herein as a Bluetooth line-in configuration.

[0147] For Bluetooth line-in configuration, Bluetooth frames are typically sent in a manner similar to User Datagram Protocol (UDP) packets, in that the Bluetooth transmitter sends each frame to the Bluetooth receiver once, and the Bluetooth receiver then detects the frame and extracts data from it. In addition, if the Bluetooth receiver fails to receive a frame or determines that the data within the frame is corrupted, the Bluetooth receiver typically does not have the opportunity to ask the Bluetooth transmitter to resend the Bluetooth frame. The 2.4 GHz band in which Bluetooth transmitters and receivers operate is extremely congested due to many devices, and therefore, wireless interference and Bluetooth frame loss are common. For example, other devices that also operate in and near the 2.4 GHz band include (i) cordless phones, microwave ovens, baby monitors, and garage door controllers in homes, (ii) wireless devices configured to use 2.4 GHz WIFI networks, (iii) wireless devices configured to use LTE wireless networks, and (iv) other Bluetooth devices.

[0148] In operation, frequent Bluetooth frame loss and transmission errors (e.g., caused by 2.4GHz network congestion or other forms of interference) can make the Bluetooth link between the Bluetooth audio information source and the first playback device inconsistent and unreliable. And when the first playback device (e.g., acting as a synchronization group coordinator) is distributing the audio information received via the Bluetooth link to one or more additional playback devices (e.g., group members) for synchronized playback (where the group coordinator and the group members form a synchronization group), the first playback device (group coordinator) and the one or more additional playback devices (group members) may fail (e.g., be out of sync with each other) due to the delay between the group coordinator transmitting the frames of the second data stream, and thus delay the group members from receiving the frames of the second data stream. As a result, after the group members encounter a delay in receiving the frames of the second data stream including audio information and playback timing from the group coordinator, when the group members recover from the failure (e.g., resynchronize the playback of their audio content with the group coordinator), there may be playback anomalies (e.g., echoes) during playback.

[0149] As further described below, in order to avoid or at least improve the problem of synchronized playback of audio content by the synchronization group, which problem is due to the unreliable and inconsistent reception of audio information (e.g., audio information received from the audio source via the Bluetooth line input interface) by the group coordinator via the audio source, some embodiments disclosed and described herein include the group coordinator, in particular (i) generating null data for lost or damaged audio information, (ii) generating playback timing information for the generated null data, (iii) sending the null data and the playback timing of the null data to one or more group members, and (iv) using the null data and the playback timing of the null data, in combination with the audio information and the playback timing of the audio information, to synchronize the playback of audio content with one or more group members based on the audio information and the null data. In some embodiments, in response to determining and / or detecting one or more problems with the incoming data stream of audio information via the audio source, the group coordinator additionally or alternatively resynchronizes the playback of the audio content by the synchronization group members.

[0150] 5. Technical features

[0151] In some embodiments, at least some aspects of the technical solution are derived from the technical structure and organization of audio information, playback timing and clock timing information used by playback devices to play audio content from an audio source in synchronization with each other, including how different playback devices generate playback timing based on clock timing and play audio content based on playback timing and clock timing.

[0152] Therefore, in order to help understand certain aspects of the disclosed technical solution, some technical details of audio information, playback timing, and clock timing information are described below, as well as how a playback device generates and / or uses playback timing and clock timing for playing audio content. Unless otherwise specified, the technical details of the audio information, playback timing, and clock timing information described below are for reference in this article. Figure 7 , Figure 8 and Fig. 9 The examples shown and described are the same or at least substantially the same.

[0153] a. Audio content

[0154] The audio content may be any type of audio content now known or later developed. For example, in some embodiments, the audio content includes one or more of any of the following: (i) streaming music or other audio obtained from a streaming service (e.g., Spotify, Pandora, or other streaming services); (ii) streaming music or other audio from a local music library, such as a music library stored on a user's laptop, desktop computer, smart phone, tablet computer, home server, or other computing device now known or later developed; (iii) audio content related to video content, such as audio associated with a television program or movie received from any of the following devices: a television, set-top box, digital video recorder, digital video disc player, streaming video service, or any other source of audiovisual media content now known or later developed; (iv) text-to-speech or other audible content from a voice assistant service (VAS), such as Amazon Alexa or other VAS services now known or later developed; (v) audio content from a doorbell or intercom system (e.g., Nest, Ring, or other doorbell or intercom systems now known or later developed); and / or (vi) audio content from a phone, video phone, video / teleconferencing system, or other application configured to allow users to communicate with each other via audio and / or video.

[0155] In operation, a group coordinator (sometimes referred to as a "source" playback device) obtains any of the above-described types of audio content from an audio source via an interface on the playback device (e.g., one of the source playback device's network interfaces, a "line-in" analog interface, a digital audio interface, or any other interface suitable for receiving audio content in a digital or analog format now known or later developed).

[0156] An audio source is any system, device, or application that generates, provides, or otherwise makes any of the above audio content available to a playback device. For example, in some embodiments, the audio source includes any one or more of the following: a streaming (audio, video) service, a digital media server or other computing system, a VAS service, a television, a cable set-top box, a streaming media player (e.g., AppleTV, Roku, a game console), a CD / DVD player, a doorbell, an intercom, a telephone, a tablet computer, or any other source of digital audio content.

[0157] The playback device that receives or otherwise obtains audio information from an audio source for playback and / or distributes to other playback devices is sometimes referred to as a group coordinator or "source" playback device in this article. A function of the group coordinator of a synchronization group is to process the audio information received for playback and / or distribute to the group member of the synchronization group for synchronous playback. In certain embodiments, the group coordinator sends the processed audio information to all other group members in the synchronization group that the group coordinator is configured to control. In certain embodiments, the group coordinator sends the audio information to a multicast network address, and all other playback devices (that is, the group member of the synchronization group) configured to play audio content receive the audio information via the multicast address. In certain embodiments, the group coordinator sends the audio information to each unicast network address of each other playback device (that is, the group member) configured to play audio content, and each other playback device (that is, the group member) configured to play audio content receives the audio information via its unicast address.

[0158] As described above, in some Bluetooth line-in embodiments (and other embodiments), the group coordinator receives audio information from an audio source in digital form, for example, via a packet stream (sometimes referred to herein as a first data stream or an incoming data stream). In some embodiments, each packet in the first data stream has a sequence number or other identifier that specifies the order of the packets. In operation, the group coordinator uses the sequence number or other identifier to detect lost packets and / or to reassemble the packets of the first data stream in the correct order before performing further processing. In some embodiments, the sequence number or other identifier that specifies the order of the packets is (or at least includes) a timestamp indicating the time when the packet was created. The packet creation time can be used as a sequence number based on the assumption that the packets are created in the order in which they should be played out later.

[0159] For example, in the Bluetooth line-in embodiment, each packet includes both a timestamp and a sequence number. The timestamp is used to place the incoming packets of audio information in the correct order, and the sequence number is primarily used to detect packet loss. In operation, the sequence number of each Real-time Transport Protocol (RTP) packet sent from the Bluetooth transmitter is incremented by 1, and the timestamp is incremented by the time "covered" by the RTP packet. In the case where a portion of the audio content is split into multiple RTP packets, multiple RTP packets may have the same timestamp.

[0160] In some embodiments, the group coordinator does not change the sequence number or identifier of the received packets during processing. In some embodiments, the group coordinator reorders at least a first set of packets in the packet stream based on the sequence identifier of each packet, extracts audio information from the received packets, reassembles a bit stream of the audio information from the received packets, and then repackets the reassembled bit stream into a second set of packets (sometimes referred to herein as a second data stream), wherein the packets in the second set of packets have sequence numbers and / or timestamps that are different from the sequence numbers and / or timestamps of the packets in the first set of packets (or first data stream).

[0161] In some embodiments, the length of each packet in the second data stream is different (i.e., shorter or longer) than the length of each packet in the first data stream. In some embodiments, reassembling the bitstream of the incoming packets and then subsequently regrouping the reassembled bitstream into different sets of packets facilitates the unified processing and / or transmission of audio content by the group coordinator and group members, and the group members receive the audio information from the group coordinator. However, for some delay-sensitive audio information, reassembly and regrouping may be undesirable, and therefore, in some embodiments, the group coordinator may not perform reassembly and regrouping on some (or all) of the audio information received before playing the audio information and / or sending the audio information to other playback devices / group members.

[0162] In some embodiments, the audio source provides the audio information in digital form to the group coordinator, for example via a digital line input interface. In such embodiments, the group coordinator groups the digital audio into packets of audio information before sending the audio information to other playback devices. In some embodiments, each packet includes a sequence number or other identifier so that when other playback devices receive the audio information, those other playback devices will be able to reliably arrange the received packets in the correct order before performing further packet processing.

[0163] After obtaining the audio information from an audio source or from another playback device, the playback device in some embodiments performs one or more of the following operations: (i) plays audio content based on the audio information alone, (ii) plays content based on the audio information synchronously with one or more additional playback devices, and / or (iii) sends the audio information to one or more other playback devices.

[0164] b. Playback timing

[0165] The playback device disclosed and described herein uses playback timing to play audio content synchronously with each other.Based on the configuration of playback device in media playback network and / or synchronization group, each playback device can generate playback timing, and / or playback audio content according to playback timing.The source playback device (acting as synchronization group coordinator) that generates the playback timing of audio content also sends the generated playback timing to all playback devices (synchronization group members) that are configured to play audio content.In certain embodiments, the synchronization group coordinator sends playback timing to the multicast network address, and all other playback devices (synchronization group members) that are configured to play audio content receive playback timing via this multicast address.In certain embodiments, the synchronization group coordinator alternatively sends playback timing to each unicast network address of each synchronization group member, and each synchronization group member receives playback timing via its own unicast address.

[0166] In some embodiments, playback timing is generated for each frame (or packet) of audio information. As described above, in some embodiments, audio information is encapsulated in a series of frames (or packets), wherein each frame (or packet) includes a portion of the audio information. In some embodiments, the playback timing of audio information includes the playback time of each frame (or packet) of audio information. In some embodiments, the playback timing of each frame (or packet) is included in the frame (or packet), for example, in the header of the frame (or packet), in the extended header of the frame (or packet), and / or in the payload portion of the frame (or packet). As further described herein, each frame (or packet) may include one or more parts and / or one or more empty parts of audio information.

[0167] In some embodiments, the playback time of each frame (or packet) is identified within a timestamp or other indication. In such embodiments, the timestamp (or other indication) represents the time at which one or more portions of audio information and / or one or more empty portions within the respective frame (or packet) are played. In some embodiments, the timestamp or other indication is associated with a "start" time that is used as a reference start time from which playback timing is generated for each audio program and / or synchronized playback session.

[0168] In operation, when the playback timing of each frame (or packet) is generated, the playback timing of each frame (or packet) is the future time relative to the "current clock time" of the reference clock, and the playback timing of each frame (or packet) is generated at the current clock time. In some embodiments, the playback timing for creating the first frame (or packet) including audio content is used for the "current clock time" played back in a synchronous playback session is the "start" time. In some embodiments, if (or when) the group coordinator determines that the audio playback of the playback device in the synchronization group needs to be resynchronized, the group coordinator resynchronizes the playback by setting a new "start" time and generating a new playback timing for each frame (or packet) including audio content. As described in more detail below, the reference clock can be a "local" clock of the group coordinator or a "remote" clock of a separate network device (e.g., another playback device, a computing device or another network device, which is configured to provide a playback device to generate a playback timing and / or a clock timing for playback audio content).

[0169] In operation, a playback device responsible for playing a specific audio content will play back a portion (or one or more empty portions, if applicable) of the specific audio content within each frame (or packet) at a playback time specified by the playback timing of the respective frames (or packets), the specified playback time being adjusted to accommodate the difference between the clock timing information and the clock of the playback device responsible for playing the audio content, as described in more detail below.

[0170] c. Clock timing

[0171] The playback device disclosed and described herein generates playback timing of audio information using clock timing, and plays audio content based on the audio information and the generated playback timing.

[0172] In some embodiments, the group coordinator uses the clock timing of a reference clock (e.g., a device clock, a digital audio converter clock, a playback time reference clock, or any other clock) to generate playback timing of audio information received by the group coordinator from an audio source. The reference clock can be a "local" clock of the group coordinator or a "remote" clock of a separate network device (e.g., another playback device, a computing device, or another network device that is configured to provide (i) clock timing used by the group coordinator to generate playback timing and / or (ii) clock timing used by the group coordinator and group members to play back audio content).

[0173] In some embodiments, all playback devices (i.e., all group members in a synchronization group) responsible for synchronously playing a specific audio content use the same clock timing of the same reference clock to synchronize playback of the specific audio content with each other. In some embodiments, the playback devices use the same clock timing to play the audio content used to generate the playback timing of the audio content. The reference clock can be a local clock of the group coordinator, but the reference clock can also be the clock of different devices, and different devices include different playback devices.

[0174] In operation, the device generating the clock timing also sends the clock timing to all playback devices in the network that need to use the clock timing to generate playback timing and / or playback audio content. In some embodiments, the device generating the clock timing sends the clock timing to a multicast network address, and all playback devices configured to generate playback timing (e.g., group coordinator) and / or play audio content (e.g., group coordinator and / or group members) receive the clock timing via the multicast address. In some embodiments, the device alternately sends the clock timing to each unicast network address of each playback device in the synchronization group.

[0175] In some embodiments, the device generating the clock timing is a playback device configured to be used as a group coordinator of the synchronization group. And in operation, the group coordinator of the synchronization group sends the clock timing to all group members of the synchronization group. As described herein, the group coordinator and the group members all use the clock timing and the playback timing to play the audio content synchronously with each other.

[0176] d. The playback timing is generated by the group coordinator

[0177] In some embodiments, a playback device is configured as a group coordinator for a synchronization group. In such an embodiment, the group coordinator: (i) generates playback timing for audio information based on the clock timing of the group coordinator's local clock, and (ii) sends the generated playback timing to all other group members in the synchronization group. In operation, when generating playback timing for each frame (or packet), the group coordinator adds a "timing advance" to the current clock time of the group coordinator's local clock that the group coordinator is using to generate the playback timing. As described above, when starting a synchronized playback session, in some embodiments, the group coordinator generates playback timing based on a "start" time, which is a first "current clock time", to which the group coordinator adds the "timing advance" to generate the playback time of the first frame (or packet) in the frame (or packet) stream that the group coordinator sends to the playback devices in the synchronization group at the start of the synchronized playback session.

[0178] In some embodiments, the "timing advance" is based on an amount of time greater than or equal to the sum of: (i) the network transmission time required for frames and / or packets comprising audio information sent from the group coordinator to reach all other group members, and (ii) the amount of time required for all other group members to process frames / packets received from the group coordinator for playback.

[0179] In some embodiments, the group coordinator determines the timing advance by sending one or more test packets to one or more (or possibly all) of the other group members and then receiving a test response packet in return from those one or more group members. In some embodiments, the group coordinator and one or more group members negotiate the timing advance via multiple test and response messages. In some embodiments with more than two group members, the group coordinator determines the timing advance by exchanging test and response messages with all group members, and then sets a timing advance that is sufficient for the group member with the longest total time of network transmission time and packet processing time.

[0180] In some embodiments, the timing is advanced less than about 50 milliseconds. In some embodiments, the timing is advanced less than about 20 milliseconds to 30 milliseconds. And in other embodiments, the timing is advanced less than about 10 milliseconds. In some embodiments, the timing is advanced and remains unchanged after being determined, or remains unchanged at least in the duration of the synchronous playback session. In other embodiments, the group coordinator can change the timing in response to a request from a group member, and the request indicates that a larger timing is needed in advance (for example, because the group member has not received a packet including a portion of the audio content until after one or more other group members have played the portion of the audio content), or a shorter timing will be enough in advance (for example, because the group member is buffering more packets than the packets required for providing consistent, reliable playback, and the packets include the portion of the audio content).

[0181] As described in more detail below, all playback devices configured to play audio content in synchronization will use playback timing and clock timing to play the audio content in synchronization with each other.

[0182] e. Generate playback timing using the clock timing of the remote clock

[0183] In some embodiments, the group coordinator can generate the playback timing of the audio information based on the clock timing of the remote clock of another network device, such as another playback device, another computing device (e.g., a smart phone, a laptop, a media server, or other computing devices that can be configured to provide sufficient clock timing for the group coordinator to generate playback timing and / or playback audio content. Generating playback timing based on the clock timing of the remote clock of another network device is more complex than generating playback timing based on the clock timing of the local clock in embodiments where the same clock timing is used for both (i) generating playback timing and (ii) playing audio content based on the playback timing.

[0184] In an embodiment in which a group coordinator generates playback timing of audio information based on the clock timing of a remote clock, the playback timing of individual frames (or packets) is based on (i) a "timing offset" between (a) a local clock of the group coordinator used to generate the playback timing and (b) clock timing information of the remote reference clock, and (ii) a "timing advance" based on an amount of time greater than or equal to the sum of: (a) the network transmission time required for a packet transmitted from the group coordinator to reach the group members and (b) the amount of time required for all of these group members to process frames and / or packets including audio information received from the group coordinator for playback.

[0185] For each frame (or packet) containing part of the audio information, the group coordinator generates the playback timing of each frame (or packet) by adding the sum of the "timing offset" and the "timing advance" to the current time of the local clock of the group coordinator used by the group coordinator to generate the playback timing of the audio information. In operation, the "timing offset" can be a positive offset or a negative offset, depending on whether the local clock of the group coordinator is ahead of or behind the remote clock that provides the clock timing. The "timing advance" is a positive number because it represents a future time relative to the local clock time, which is adjusted by the "timing offset". As described above, when a synchronized playback session is started, in some embodiments, the group coordinator generates the playback timing based on the "start" time, which is the first "current clock time", and the group coordinator adds the sum of the "timing advance" and the "timing offset" to the first "current clock time" to generate the playback time of the first frame (or packet) in the frame (or packet) stream that the group coordinator sends to the playback devices in the synchronized group at the start of the synchronized playback session.

[0186] By adding the sum of the "timing advance" and the "timing offset" to the current time of the group coordinator's local clock that the group coordinator is using to generate the playback timing of the audio information, the group coordinator is actually generating a playback time relative to the remote clock.

[0187] In some embodiments, and as described above, the "timing advance" is based on an amount of time greater than or equal to the sum of: (i) the network transmission time required for frames and / or packets comprising audio information sent from the group coordinator to reach all other group members, and (ii) the amount of time required for all other group members to process frames / packets received from the source playback device for playback.

[0188] In certain embodiments, the group coordinator determines the timing in advance via the signaling between the group coordinator and one or more group members, as described above. In addition, in certain embodiments, the timing is in advance less than about 50 milliseconds, less than about 20 milliseconds to 30 milliseconds or less than about 10 milliseconds, depending on the audio content playback delay requirement, because different audio contents may have different delay requirements. For example, the audio content with associated video content may have a delay requirement lower than the audio content without associated video content, because the audio content associated with the video content must be synchronized with its corresponding video content, and the audio content not associated with the video content does not need to be synchronized with any corresponding video content. In certain embodiments, the timing remains unchanged in advance after being determined, or remains unchanged at least in the duration of the synchronous playback session. And in certain embodiments, the group coordinator can change the timing in advance based on the future signaling between the group coordinator (generating playback timing) and one or more group members (playing audio content with playback timing).

[0189] As described in more detail below, all playback devices configured to play audio content in synchronization will use playback timing and clock timing to play the audio content in synchronization with each other.

[0190] f. Use local playback timing and local clock timing to play audio content

[0191] In some embodiments, the group coordinator is configured to play audio content synchronously with one or more group members. And if the group coordinator uses the clock timing of the local clock of the group coordinator to generate playback timing, the group coordinator will use the locally generated playback timing and the locally generated clock timing to play the audio content. In operation, when the local clock used by the group coordinator to generate playback timing reaches the time specified by the playback timing of each frame (or packet), the group coordinator plays each frame (or packet) including a portion (or empty portion, if applicable) of the audio information.

[0192] For example, recall that when generating playback timing for each frame (or packet), the group coordinator device adds a "timing advance" to the current clock time of the reference clock used to generate the playback timing. In this case, the reference clock used to generate the playback timing is the local clock of the group coordinator. Therefore, if the timing advance for each frame is, for example, 30 milliseconds, the group coordinator plays a portion (e.g., a sample or a set of samples) of the audio information in each frame (or packet) (or plays an empty portion, if applicable) 30 milliseconds after creating the playback timing for that frame (or packet).

[0193] In this way, the group coordinator plays the audio content based on the audio information by using the locally generated playback timing and the clock timing of the local reference clock of the group coordinator. As further described below, by playing the portion of the audio information of each frame and / or packet when the clock time of the local reference clock reaches the playback timing of the respective frame or packet, the group coordinator plays the portion (or empty portion) of the audio content corresponding to the audio information in the respective frame and / or packet in synchronization with other group members in the synchronization group.

[0194] g. Use local playback timing and remote clock timing to play audio content

[0195] As mentioned above, in some embodiments, the group coordinator generates the playback timing of the audio information based on the clock timing of the remote clock, the remote clock, i.e., the clock of another network device separated from the group coordinator, the other network device such as another playback device, another computing device (e.g., smart phone, laptop computer, media server, or other computing devices that can be configured to provide enough for the playback device to generate the playback timing and / or playback audio content clock timing). Because the group coordinator uses the clock timing of the remote clock to generate the playback timing of the audio content, the group coordinator also uses the clock timing of the remote clock to play the audio content. In this way, the group coordinator uses the playback timing generated locally and the clock timing of the remote clock to play the audio content.

[0196] Recall that in embodiments where the group coordinator generates playback timing for audio content based on the clock timing of a remote clock, the group coordinator generates playback timing for each frame (or packet) based on: (i) a "timing offset" based on the difference between (a) the local clock of the group coordinator and (b) the clock timing information of the remote clock, and (ii) a "timing advance" comprising an amount of time greater than or equal to the sum of: (a) the network transmission time required for the frame / packet transmitted from the group coordinator to reach all group members and (b) the amount of time required for all group members to process the frames and / or packets including the audio information received from the group coordinator for playback. And recall also that the group coordinator sends the generated playback timing to all group members in the synchronization group responsible for synchronously playing the audio content.

[0197] In this scenario, in order to play each frame (or group) of the audio content synchronously with one or more other playback devices, the group coordinator subtracts the "timing offset" from the playback timing of each frame (or group) to generate a "local" playback time for playing the audio content based on the audio information in each frame (or group). After generating the "local" playback time for playing the portion of the audio content corresponding to the audio information in each frame (or group), when the local clock (which is the same local clock used by the group coordinator to generate the playback time) that the group coordinator is using to play the audio content reaches the "local" playback time of each frame (or group), the group coordinator plays the portion of the audio content corresponding to the audio information in each frame (or group). By subtracting the "timing offset" from the playback time to generate the "local" playback time of each frame, the group coordinator references the clock timing of the remote clock to effectively play the portion of the audio content corresponding to the audio information in the frame / group.

[0198] h. Use remote playback timing and local clock timing to play audio content

[0199] Recall that in some embodiments, the group coordinator sends audio information and playback timing of the audio information to one or more group members. If the group member that receives the audio information and playback timing from the group coordinator (i.e., the receiving group member) is the same group member as the group member that provides the group coordinator with the clock timing information used by the group coordinator to generate the playback timing, then in this case, the receiving group member uses the audio information and the playback timing received from the group coordinator (i.e., the remote playback timing) and the group member's own clock timing (i.e., the local clock timing) to play the audio content. Because the group coordinator uses the clock timing of the receiving group member's clock to generate the playback timing, the receiving group member also uses the clock timing of its local clock to play the audio content. In this way, the receiving group member uses the remote playback timing (i.e., the playback timing of the group coordinator) and the clock timing of its local clock (i.e., its local clock timing) to play the audio content.

[0200] In order to play individual frames (or packets) of audio information synchronously with the group coordinator (and each other playback device that receives playback timing from the group coordinator and clock timing from the receiving group member), the receiving group member (i) receives frames (or packets) including portions of audio information (and empty portions, if applicable) from the group coordinator, (ii) receives playback timing of the audio information (and empty portions, if applicable) from the group coordinator (for example, in frame and / or packet headers of frames and / or packets including portions of audio information and empty content, or possibly separately from frames and / or packets including portions of audio information and empty content), and (iii) plays the portions of the audio information in the individual frames (or packets) when the local clock used by the receiving group member to generate clock timing reaches the playback time specified in the playback timing of the individual frames (or packets) received from the group coordinator.

[0201] Because the group coordinator uses a "timing offset" (i.e., the difference between the clock timing of the receiving group member and the clock timing of the group coordinator in this scenario) when generating the playback timing, and because this "timing offset" has accounted for the time difference between the group coordinator and the receiving group member, in this case, the receiving group member playback includes the individual frame (or packet) information of the audio part (and the empty part, if applicable) when the local clock of the receiving group member (used to generate the clock timing) reaches the playback time of the individual frames (or packets) specified in the playback time of the individual frames (or packets).

[0202] And because the receiving group members play frames (or packets) including portions of the audio information (and empty portions, if applicable) according to the playback timing, and because the group coordinator plays the same frames (or packets) including portions of the audio information (and empty portions, if applicable) according to the playback time and the determined "timing offset", the receiving group members and the group coordinator play the same frames (or packets) including the audio information corresponding to the same portion of the audio content synchronously (i.e., simultaneously or substantially simultaneously).

[0203] i. Use remote playback timing and remote clock timing to play audio content

[0204] Recall that in some embodiments, the source playback device (e.g., which in many cases may be a group coordinator) sends audio information and playback timing of the audio information to one or more other playback devices in the synchronization group. And also recall that in some embodiments, the network device that provides clock timing may be a different device from the playback device that provides audio information and playback timing (i.e., the source playback device, which in many cases may be a group coordinator). The playback device that receives audio information, playback timing, and clock timing from one or more other devices is configured to play audio content using the playback timing of the device that provides playback timing (i.e., remote playback timing) and the clock timing of the clock of the device that provides clock timing (i.e., remote clock timing). In this way, in this case, the receiving playback device plays audio content based on the audio information by using the remote playback timing and the remote clock timing.

[0205] In order to synchronize the playback of individual frames (or packets) of audio information with each other playback device responsible for synchronized playback of the audio content, the receiving playback device (i) receives the frames (or packets) including the portion of the audio information, (ii) receives the playback timing of the audio information (e.g., in the frame and / or packet header of the frames and / or packets including the portion of the audio information, or possibly separately from the frames and / or packets including the portion of the audio information and empty content), (iii) receives the clock timing, and (iv) plays the portion of the audio information in the individual frames (or packets) when the local clock of the receiving playback device for audio content playback reaches the playback time specified in the playback timing of the individual frames (or packets), the playback time being adjusted by the "timing offset".

[0206] In operation, after the receiving playback device receives the clock timing, the receiving device determines a "timing offset" for the receiving playback device. This "timing offset" includes (or at least corresponds to) the difference between a "reference" clock used to generate the clock timing and a "local" clock used by the receiving playback device to play the audio content. In operation, each playback device that receives the clock timing from another device calculates its own "timing offset" based on the difference between its local clock and the clock timing, and therefore, the "timing offset" determined by each playback device is specific to that particular playback device.

[0207] In some embodiments, when playing back audio content, the receiving playback device generates new playback timing (specific to the receiving playback device) for each frame (or packet) of the audio information (and the empty portion, when applicable) by adding the previously determined "timing offset" to the playback timing of each received frame (or packet) that includes a portion of the audio information and / or an empty portion. In this way, the receiving playback device converts the playback timing of the received audio information / empty portion to the "local" playback timing of the receiving playback device. Because each receiving playback device calculates its own "timing offset", the determined "local" playback timing of each frame of each receiving playback device is specific to that particular playback device.

[0208] And when the "local" clock that the receiving playback device is using to play back the audio content reaches the "local" playback time of each frame (or packet), the receiving playback device plays the audio information (or part thereof) or empty content (or part thereof) associated with the each frame (or packet). As described above, in some embodiments, the playback timing of a particular frame (or packet) is in the header of the frame (or packet). In other embodiments, the playback timing of each frame (or packet) is sent separately from the frame (or packet) that includes the audio content.

[0209] Because the receiving playback device plays frames (or packets) including portions of audio information and / or empty content (if applicable) according to the playback timing relative to the clock timing adjusted by the “timing offset”, and because the device providing playback timing generates the playback timing of these frames (or packets) relative to the clock timing and plays the same frames (or packets) including portions of audio information (or empty content) according to the playback timing and the “timing offset” determined by it, the receiving playback device and the device providing playback timing (e.g., the group coordinator in some embodiments) play the same frames (or packets) including the same portions of audio information and / or empty portions synchronously (i.e., simultaneously or substantially simultaneously).

[0210] VI. Example Embodiments

[0211] The example embodiments described below illustrate synchronized playback of audio information received from an unreliable audio information source by playback devices in various networked scenarios in some embodiments.

[0212] Figure 7 An example configuration 700 is shown of two playback devices 704, 706 playing audio content received from an unreliable audio information source 702 in sync with each other in accordance with some embodiments.

[0213] Figure 7 The configuration 700 in FIG. 7 includes a first playback device 704, a second playback device 706, and an audio source 702. The arrows shown in the configuration 700 indicate the signal flow of timing information and media content.

[0214] Playback devices 704 and 706 are the same or similar to the playback devices shown and described herein. In some embodiments, one or both of playback devices 704 and 706 are or include networked microphone devices.

[0215] In some embodiments, the first playback device 704 and the second playback device 706 are configured in a synchronization group, wherein the first playback device 704 and the second playback device 706 play the same audio content in synchronization with each other, as previously described.

[0216] In some embodiments, the first playback device 704 and the second playback device 706 are configured as a stereo pair, wherein the first playback device 704 is configured to play the left channel of the stereo audio content and the second playback device 706 is configured to play the right channel of the stereo audio content (or vice versa).

[0217] And in other embodiments, the first playback device 704 and the second playback device 706 are configured as a bound pair, where the first playback device 704 plays a first set of frequency components of the audio content, and the second playback device 706 plays a second set of frequency components of the audio content.

[0218] And in still other embodiments, the first playback device 704 and the second playback device 706 are configured as other grouping arrangements now known or later developed, wherein the first playback device 704 and the second playback device 706 play the same audio content of different channels and / or sets of frequency components in synchronization with each other.

[0219] In some embodiments, the first playback device 704 and the second playback device 706 need not be grouped in any particular manner (e.g., a synchronization group, a stereo pair, a bonded pair, a merged device, etc.) to play audio content in sync with each other. Instead, in some embodiments, even if the first playback device 704 and the second playback device 706 are not in a grouped configuration, but are merely communicatively coupled to each other via a network (e.g., in the same home), the first playback device 704 and the second playback device 706 can play certain audio content in sync with each other (e.g., a home intercom, phone / doorbell notifications, etc.) according to the shared playback timing and clock timing described herein.

[0220] In some embodiments, audio source 702 is the same as or similar to any audio source disclosed and described herein.

[0221] In the example configuration 700, the first playback device 704 is the "source" playback device for the first audio content 750, and in some embodiments, serves as a group coordinator for a sync group that includes the first playback device 704 and the second playback device 706. The sync group in the example 700 is shown as including only the first playback device 704 and the second playback device 706. In operation, a sync group may include more than two playback devices.

[0222] In operation, the first playback device 704 (i) receives audio information 750 from the audio source 702 via a communication link between the audio source 702 and the first playback device 704, (ii) provides the audio information 752 to the second playback device 706 (and possible additional playback devices (not shown)), and (iii) plays audio content in synchronization with the second playback device 706 (and possible additional playback devices (not shown)) based on the audio information 752.

[0223] In some embodiments, the communication link between the audio source 702 and the first playback device 704 is a Bluetooth communication link. However, the communication link between the audio source 702 and the first playback device 704 can be any communication link now known or later developed suitable for transmitting audio data.

[0224] In operation, Figure 7 In the illustrated example 700, a source 702 transmits a data stream including packets of encoded audio information 750 to a first playback device 704 via a Bluetooth wireless link. In some embodiments, the data stream includes frames or units of audio information 750. In some embodiments, the data stream includes any other type of data encapsulation format now known or later developed suitable for transmitting data.

[0225] The first playback device 704 receives a packet stream including the encoded audio information 750 from the source 702, and extracts the encoded audio information 750 from the incoming data stream on a packet-by-packet basis. That is, when the first playback device 704 receives a first packet including the encoded audio information 750 from the source 702, the first playback device 704 extracts a portion of the encoded audio information 750 contained within the packet. The first playback device 704 continues to extract the encoded audio information 750 from the packets on a packet-by-packet (or frame-by-frame or unit-by-unit, depending on how the encoded audio information 750 is encapsulated) basis as the packets arrive (or substantially as they arrive).

[0226] As the first playback device 704 continues to receive the first data stream including the encoded audio information 750 (i.e., the incoming data stream) and extract the encoded audio information 750 from the incoming data stream, the first playback device 704 generates a second data stream (i.e., the outgoing data stream) including a portion of the extracted audio information 752 and sends it to at least the second playback device 706 (and possibly also to an additional playback device (not shown)). In example 700, the first playback device 704 sends the second data stream (the outgoing data stream) including a portion of the extracted audio information 752 to the second playback device 706 via a WIFI communication link. However, the second data stream can be transmitted via any type of communication link now known or later developed that is suitable for transmitting data. In some embodiments, the first playback device 704 (i) receives the incoming data stream from the audio source 702 via a first wireless interface, and (ii) sends the outgoing data stream to at least the second playback device 706 via a second wireless interface different from the first wireless interface.

[0227] In example 700, because Bluetooth and WIFI are different transmission protocols, the outgoing data stream has a different framing format than the incoming data stream. In this way, the first playback device 704 (i) receives audio information 750 via an incoming data stream having a first framing format (e.g., Bluetooth), and (ii) generates audio information 752 and sends it to one or more other playback devices via an outgoing data stream having a second framing format (e.g., Wi-Fi).

[0228] Because both Bluetooth and WIFI operate in the 2.4 GHz band, and because WIFI tends to have a higher transmission power level than Bluetooth, the transmission of the outgoing data stream by the first playback device 704 via WIFI may interfere with the reception of the incoming data stream by the first playback device 704 via Bluetooth. Thus, in some embodiments, the first playback device 704 (i) receives the audio information 750 via the incoming data stream via Bluetooth operating in the 2.4 GHz band, and (ii) generates and sends the audio information 752 to one or more other playback devices via the outgoing data stream via WIFI operating in the 5.0 GHz band. However, in some embodiments, the 5.0 GHz WIFI band may not be available for transmission, for example, one or more playback devices and / or other network infrastructure components do not support 5.0 GHz WIFI, the 5.0 GHz WIFI band is too congested with other network traffic, and / or one or more playback devices are too far away to receive 5.0 GHz band WIFI transmissions from the first playback device 704 in a consistent and reliable manner. In such a case, in some embodiments, and to avoid or at least reduce the possibility of interference between the incoming data stream and the outgoing data stream, the first playback device 704 (i) receives the audio information 750 via the incoming data stream via Bluetooth operating at a first portion of the 2.4 GHz frequency band, and (ii) generates and transmits the audio information 752 to one or more other playback devices via the outgoing data stream via WIFI operating at a second portion of the 2.4 GHz frequency band. Preferably, the first portion of the 2.4 GHz frequency band does not overlap with the second portion of the 2.4 GHz frequency band, however, such a non-overlapping configuration is not required.

[0229] In operation, the outgoing data stream includes (i) a portion of the encoded audio information extracted from the incoming data stream and (ii) playback timing information, wherein the playback timing information includes the playback timing of the portion of the encoded audio information extracted from the incoming data stream. In some embodiments, the outgoing data stream may additionally include clock timing information. However, in some embodiments, the first playback device 704 may send the clock timing information to the second playback device 706 separately from the outgoing data stream.

[0230] Specifically, in addition to the portion of the audio information 752, the first playback device 704 also sends the playback timing 754 and the clock timing 756 of the portion of the audio information 752 to the second playback device 706 (and possible additional playback devices (not shown)). In operation, the first playback device 704 uses the playback timing 754 and the clock timing 756 to play the audio content synchronously with at least the second playback device 706 based on the audio information 752, as described herein.

[0231] In the example configuration 700 , and in conjunction with generating the outgoing data stream, the first playback device 704 generates a playback timing 754 of the audio information 752 using its own clock timing 756 (ie, the clock timing of the clock of the first playback device 704 ).

[0232] In some embodiments, the first playback device 704 generates the playback timing 754 of the audio information 752 by adding a "timing advance" to the current clock time of a local reference clock used by the first playback device 704 to generate the playback timing 754 of the audio information 752. In some embodiments, the playback timing of a frame or packet of the audio information 752 corresponds to a future time (relative to the reference clock time) at which the portion of the audio information 752 in the packet (or frame) will be played back.

[0233] In some embodiments, the timing advance that the first playback device 704 adds to its own clock timing 756 to generate a future time for the playback timing 754 is based on an amount of time greater than or equal to the sum of: (i) the network transmission time required for a packet sent from the first playback device 704 to reach the second playback device 706 and (ii) the amount of time required for the second playback device 706 to process the received packet for playback.

[0234] In some embodiments, the first playback device 704 determines the timing advance by sending one or more test packets to the second playback device 706 and receiving a test response packet from the second playback device 706 in response to the test packet. In some embodiments, the first playback device 704 and the second playback device 706 can negotiate the timing advance via one or more test and response messages. In some embodiments with more than two playback devices, the first playback device 704 determines the timing advance by exchanging test and response messages with one or more (or all) playback devices, and then sets a timing advance that is sufficient for the playback device with the longest total time of network transmission time and packet processing time.

[0235] In some embodiments, the timing advance is less than about 50 milliseconds. In some embodiments, the timing advance is less than about 20 milliseconds to 30 milliseconds. And in yet other embodiments, the timing advance is less than about 10 milliseconds. In some embodiments, the timing advance remains unchanged after being determined, or at least remains unchanged during the duration of the synchronized playback session, or may remain unchanged during the duration that the synchronization group exists or is in a particular configuration (recall that synchronization groups can be formed, changed, dissolved, etc.).

[0236] In other embodiments, the first playback device 704 can change the timing advance in response to a request from a receiving playback device indicating that (i) a larger timing advance is required, for example, because the receiving playback device is receiving packets including the audio portion of the audio information 752 after the time at which the portion was played (specified by the playback timing 754 for the portion of the audio information 752), or because the receiving playback device is receiving the portion of the audio information 752 after some other device has already played the portion of the audio information, or (ii) a shorter timing advance will be sufficient, for example, because the receiving device is buffering more packets including portions of the audio information than are required to provide consistent, reliable playback.

[0237] After determining the playback timing 754 of the packets, frames and / or samples of the audio information 752, the first playback device 704 sends the packets / frames / samples of the audio information 752 and the first playback timing 754 of the packets / frames / samples of the audio information 752 to the second playback device 706 via the WIFI communication link. Because in the example configuration 700, the first playback device 704 uses its own local clock timing to generate the playback timing 754 of the audio information 752, the first playback device 704 also sends its clock timing 756 to the second playback device 706.

[0238] The second playback device 706 generates a "timing offset" using the clock timing 756 of the first playback device 704 and the clock timing of the second playback device 706 itself. Because the second playback device 706 uses the playback timing 754 and clock timing 756 received from the first playback device 704 to play the audio information 752, in this case the second playback device 706 uses the remote playback timing and remote clock timing to play the audio content based on the audio information 752.

[0239] In some embodiments, one or more (or all) of the following steps occur in conjunction with a second playback device 706 that generates a "timing offset" based on the clock timing 756 of the first playback device 704 and the second playback device 706's own clock timing.

[0240] First, the first playback device 704 generates a clock timing indication 756 (e.g., a timestamp or other indication) at time t and sends the clock timing indication 756 to the second playback device 706. Next, when the second playback device 706 receives the clock timing indication 756 from the first playback device 704, the second playback device 706 determines the difference between the local clock of the second playback device 706 and the time t indicated in the clock timing indication 756. The second playback device 706 uses this determined “timing offset” based on a portion of the audio information 752 to synchronize playback of audio content with the first playback device 704.

[0241] In some embodiments, at the beginning of the synchronized playback session, the first playback device 704 sends the clock timing 756 to the second playback device 706 at least once. In some embodiments, because the clocks of the first playback device 704 and the second playback device 706 are susceptible to clock drift (e.g., frequency and / or phase drift), the first playback device 704 sends the clock timing 756 to the second playback device 706 periodically or at least more than a few times during the synchronized playback session. For example, the first playback device 704 may send the clock timing 756 to the second playback device 706 every several milliseconds, every tens of milliseconds, every hundreds of milliseconds, every few seconds, every few minutes, etc.

[0242] After sending at least some frames (or packets) including at least a portion of the audio information 752, the first playback timing 754, and the clock timing 756 to the second playback device 706, the first playback device 704 and the second playback device 706 perform the following functions to play back the audio content in synchronization with each other.

[0243] When the local clock of the first playback device 704 used to generate the first playback timing 754 reaches the time specified in the first playback timing 754 of the respective frame (or packet) including the audio information 752, the first playback device 704 plays the respective frame (or packet) including the portion of the audio information 752. For example, recall that when the playback timing of the respective frame (or packet) is generated, the "source" playback device (i.e., Figure 7 The first playback device 704 in the playback device 706 adds the "timing advance" to the current clock time of the local reference clock used to generate the playback timing. Therefore, if the timing advance of each frame is, for example, 30 milliseconds, the first playback device 704 ultimately plays the audio content corresponding to the portion (e.g., sample or set of samples) of the audio information 752 in each frame (or packet) 30 milliseconds after creating the playback time of each frame (or packet).

[0244] And., when the current time of the local clock (local clock of the second playback device 706) that the second playback device 706 is using to play the audio information 752 reaches the first playback timing 754 of the frame (or grouping), taking into account the previously determined "timing offset", the second playback device 706 plays the audio content corresponding to the audio information 752 in each frame (or grouping).

[0245] Recall that the second playback device 706 calculates (or otherwise determines) a "timing offset" that corresponds to the difference between: (a) the "reference" clock of the first playback device 704 used to generate the clock timing 756 (and the first playback timing 754) of the first playback device 704 and (b) the "local" clock of the second playback device 706 used to play the audio information 752. The second playback device 706 uses this timing offset when playing the audio content based on the audio information 752.

[0246] Specifically, by adding the previously determined "timing offset" to the playback timing of the playback time (or packet) of each frame received from the first playback device 702, the second playback device 706 generates a new playback timing (specific to the second playback device 706) for each frame (or packet) of the audio information 752. In this way, the second playback device 704 converts the playback timing 754 of the audio information 752 received from the first playback device 704 to the "local" playback timing of the second playback device 706. And when the "local" clock of the second playback device 706 that is being used to play back the audio content reaches the determined "local" playback time of each frame (or packet), the second playback device 706 plays the audio information 752 (or part thereof) associated with the each frame (or packet). As described above, in some embodiments, the playback timing of a specific frame (or packet) is in the header of the frame (or packet). In other embodiments, the playback timing of each frame (or packet) is sent separately from the frame (or packet) including the audio content.

[0247] Because the first playback device 704 plays the audio content corresponding to the frames (or packets) including the portion of the audio information 752 according to the playback timing 754, and because the second playback device 706 also plays the audio content corresponding to the frames (or packets) including the portion of the audio information 752 according to the same playback timing 754 and the timing offset, the audio content played by the first playback device 704 and the audio content played by the second playback device 706 are played synchronously with each other. That is, the first playback device 704 and the second playback device 706 play the audio content synchronously with each other based on the playback timing 754 and the clock timing 756.

[0248] The above-described ability to receive a first frame (or packet or unit) stream having a first framing format (or packet format or unit format) and to generate a second frame (or packet or unit) stream having a second framing format (or packet format or unit format) enables the first playback device 704 to receive audio information 750 and play audio content from a variety of audio sources, thereby enabling one aspect of the disclosed systems and methods to be implemented by enabling the first playback device 704 to use a common format (e.g., a second framing format) to stream audio information 752 and playback timing information 754 to at least a second playback device 706 (and potentially also to additional playback devices), regardless of (i) the format of the first frame stream, and / or (ii) a sequence identifier, timestamp, or other timing and / or sequence information in the first frame stream.

[0249] As described above, some types of communication links are more robust than other types of communication links. For example, a WIFI wireless link typically provides more consistent and reliable media content delivery than a Bluetooth wireless link, because a WIFI link tends to operate at a higher transmission power than a Bluetooth link and has a better wireless link margin, and protocols typically used with WIFI (e.g., TCP / IP) tend to have better error detection and correction and more robust and complex retransmission capabilities than protocols typically used with Bluetooth (e.g., UDP / IP). In operation, the first playback device 704 cannot send to the second playback device 706 audio information that the first playback device 704 did not receive and / or did not receive in sufficient time to process and stream to the second playback device 706 for synchronized playback.

[0250] Therefore, in order to prevent inconsistencies, delays, and errors that may be present in the incoming data stream (from source 702) from causing similar inconsistencies in the outgoing data stream (with at least the second playback device 706), in some embodiments, the first playback device 704 additionally (i) determines whether the incoming data stream is missing one or more portions of the encoded audio information 750, and (ii) in response to determining that the incoming data stream is missing one or more portions of the encoded audio information 750, generates one or more empty portions corresponding to the one or more missing portions of the encoded audio information 750.

[0251] In some embodiments, based on one or more of the following analyses: (i) analysis of a sequence identifier corresponding to the encoded audio information 750 in the stream received from the audio source 702, and / or (ii) analysis of a timestamp corresponding to the encoded audio information 750 in the stream received from the audio source 702, the first playback device 704 determines whether the packet stream including the encoded audio information 750 received from the audio source 702 is losing one or more portions of the encoded audio information 750.

[0252] And when the first playback device 704 generates the outgoing data stream, the first playback device 704 inserts one or more empty portions into the outgoing data stream, where portions of the audio information 752 are at locations where the missing portions of the audio information would have been, so that the first playback device 704 receives those portions in the incoming data stream. And, when the first playback device generates playback timing 754 of the audio information 752 of the outgoing data stream, the first playback device 704 also generates and assigns playback times for one or more corresponding empty portions in the second data stream in the same manner as the first playback device 704 assigned playback times for portions of the audio information 752.

[0253] In some embodiments, the first playback device 704 determines the duration of each empty portion by examining the sequence number and timestamp included in the packets of audio information 750 received from the audio source 702. For example, each RTP packet includes a timestamp and a sequence number. The playback device 704 uses the Bluetooth timestamp and / or sequence number in each received RTP packet to detect packet loss and place the incoming RTP packets in the correct order (sequence).

[0254] In some embodiments, the first playback device 704 uses RTP sequence numbers to detect packet loss. The sequence number is increased by one for each RTP packet sent by the audio source 702, so if the first playback device 704 loses sequence numbers in the incoming data stream, the first playback device 704 is losing packets, i.e., the lost sequence numbers correspond to "lost" RTP packets.

[0255] The timestamps in the incoming RTP packets increase with the time that each RTP packet "covers." If a frame (or other portion) of audio information 750 is split into several RTP packets, these several packets may have the same timestamp, but each packet will have a different sequence number.

[0256] In some embodiments, in order to determine the duration of the empty portion, the first playback device 704 compares the timestamp of the RTP packet before the "lost packet" and the timestamp of the RTP packet after the "lost" packet (based on the sequence number), and determines the duration of the empty portion based at least in part on a time difference corresponding to the difference between the timestamp of the RTP packet before the "lost packet" and the timestamp of the RTP packet after the "lost" packet.

[0257] By the way, in embodiments where the playback timing 754 includes timestamps, the timestamps in the playback timing 754 of the outgoing data stream are different from the timestamps in the incoming RTP packets (including the audio information 750) received from the audio source 702. As further described herein, for embodiments where the playback timing 754 includes timestamps, the timestamps in the playback timing 754 of each packet (or frame or unit) indicate the time at which one or more portions (or empty portions, when applicable) of the audio information 752 corresponding to the timestamp in the playback timing 754 are played. In contrast, at least in embodiments where the incoming data stream is received over a Bluetooth link, the timestamps in the incoming RTP packets received from the audio source 702 refer to the time "covered" by the respective RTP packets. In some embodiments, the timestamps in the incoming packets received from the audio source 702 refer to the time when the packets were created by the audio source 702, rather than the time when the packets are to be played by the playback device.

[0258] In addition, when generating audio content using the audio information 752 and playback timing 754 of the outgoing data stream and playing the audio content (based on the audio information 752 of the outgoing data stream) synchronously with at least the second playback device 706, the first playback device 704 uses (i) a portion of the audio information 752 in the outgoing data stream, (ii) the playback timing 754 of the portion of the audio information 752 in the outgoing data stream, (iii) an empty portion corresponding to a missing portion of the audio information 750 that is lost from the incoming data stream, and (iv) the playback time 754 of the empty portion inserted into the outgoing data stream. Specifically, when the clock used by the first playback device 704 to generate the clock timing 756 reaches the time specified by the playback timing 754 of the portion of the audio information 752, the first playback device 704 (i) plays the audio content corresponding to each portion of the audio information 752 at the time specified by the playback timing 754 of the portion of the audio information 752, and (ii) when the clock used by the first playback device 704 to generate the clock timing 756 reaches the time specified by the playback timing 754 of the empty portion, plays each empty portion at the time specified in the playback timing 754 of the empty portion.

[0259] In operation, when using the audio information 752 and playback timing 754 received from the first playback device 704 to play audio content (based on the audio information 752) in synchronization with at least the first playback device 704, the second playback device 706 uses (i) a portion of the audio information 752 received from the first playback device 704, (ii) the playback timing 754 received from the first playback device 704, (iii) an empty portion generated by the first playback device 704 and inserted into the data stream received from the first playback device 704, and (iv) the playback timing 754 of the empty portion. Specifically, the second playback device 706 (i) plays the audio content corresponding to each portion of the audio information 752 at the time specified by the playback timing 754 for that portion of the audio information 752, which is adjusted by the previously determined "timing offset" between the clock time of the second playback device 706 and the clock timing 756 received from the first playback device 704, and (ii) plays each empty portion at the time specified in the playback timing 754 for the empty portion, which is adjusted by the previously determined "timing offset" between the clock time of the second playback device 706 and the clock timing 756 received from the first playback device 704.

[0260] In operation, the method enables (i) the first playback device 704 to generate and send a consistent outgoing data stream to at least the second playback device 706 (and possible additional playback devices), and (ii) at least the second playback device 706 (and possible additional playback devices) to receive a consistent stream from the first playback device 704 (i.e., the outgoing data stream generated by the first playback device 704).

[0261] Although example 700 illustrates and describes a scenario in which a first playback device 704 sends audio information 752, empty portions, playback timing 754, and clock timing 756 to a second playback device 706, in other embodiments, the first playback device 704 sends audio information 752, empty portions, playback timing 754, and clock timing 756 to one or more additional playback devices.

[0262] For example, in an embodiment where the first playback device 704 is or includes a surround sound controller (e.g., a sound bar) that distributes audio information to multiple surround sound playback devices, the first playback device 704 can be configured to send the audio information 752 (or portions thereof), the null portion, the playback timing 754, and the clock timing 756 to one or more additional playback devices, including but not limited to (i) one or more left satellite playback devices, (ii) one or more right satellite playback (iii) one or more subwoofer playback devices, (iv) one or more overhead satellite playback devices, and / or (v) additional playback devices in a synchronization group, configured to play back surround sound content in synchronization with the sound bar and speakers and other playback devices in the synchronization group.

[0263] In addition, although example 700 shows and describes a scenario in which the first playback device 704 generates playback timing 754 based on its local clock timing, in other embodiments, the first playback device 704 can generate playback timing 754 based on a remote clock timing (as described above). Similarly, the first playback device 704 can receive and distribute playback timing 754 and clock timing information 756 received from another device (as described above).

[0264] In some embodiments, in addition to generating and inserting the null portions and playback timing 754 of the null portions into the outgoing data stream that the first playback device 704 sends to the second playback device 706 (and possibly to additional playback devices (not shown)), the first playback device 704 additionally notifies the second playback device 706 that the packet stream including the audio information 752 may include more than a threshold number of null portions. And in some embodiments, the first playback device 704 may additionally or alternatively notify the second playback device 706 that the time interval between when the second playback device 706 receives a portion of the encoded audio information 752 and when the second playback device 706 receives a subsequent portion of the encoded audio information 752 may exceed a threshold duration. In operation, the first playback device 704 may send one or more control messages to notify the second playback device 706 that the second playback device 706 may receive more than a threshold number of null portions and / or that the second playback device 706 may experience a delay between receiving portions of the audio information from the first playback device 704.

[0265] In some embodiments, in response to receiving one or more control messages from the first playback device 704 regarding the possibility of receiving more than a threshold number of empty portions and / or encountering delays between portions of received audio information 752, the second playback device 706 switches to operating in a resync delay off mode.

[0266] During normal operation, the second playback device 706 may initiate a resynchronization process in response to any one or more of the following situations: (i) failing to receive portions of the audio information 752 from the first playback device 704 within a first threshold duration, (ii) encountering a delay between portions of the audio information 752 received from the first playback device 704 that exceeds a second threshold duration, (iii) receiving more than a first threshold number of out-of-order portions of the audio portion information 752, and / or (iv) receiving (or failing to receive) portions of the audio information 752 from the first playback device 704 in a manner that affects the ability of the second playback device 706 to consistently and / or reliably reassemble portions of the audio information 752, and / or play audio content based on the audio information 752.

[0267] In some embodiments, the resynchronization process may include, among other functions, determining a new “timing offset” between (a) the clock timing information 756 received from the first playback device 704 and (b) the local clock of the second playback device 706. Recall that the second playback device 706 uses the “timing offset” to convert the playback timing 754 information received from the first playback device 704 into a “local playback timing” specific to the second playback device 706. In some embodiments, the resynchronization process may additionally or alternatively include “refreshing,” deleting, de-indexing, or otherwise removing audio information including audio samples with associated playback timing information that were previously (i) received by the second playback device 706 from the first playback device 704 and (ii) stored in the buffer of the second playback device 706 and queued for playback by the second playback device 706 according to the playback timing, e.g., when the local clock of the second playback device 706 advances to the playback time of the audio samples stored in the buffer of the second playback device 706.

[0268] When the second playback device 706 performs the resynchronization process during synchronized playback, in some cases, the listener may hear an echo effect or brief silence ("interruption") via the second playback device 706. For example, a change in the "timing offset" may cause the second playback device 704 to (i) skip (not play back) a portion of the audio information 752 that results in an interruption (e.g., because the clock of the second playback device 706 has been running faster than the clock of the first playback device 704, resulting in an increase in the timing offset), or (ii) may repeat (play again) a portion of the audio information 752 that results in an echo (e.g., because the clock of the second playback device 706 has been running slower than the clock of the first playback device 704, resulting in a decrease in the timing offset). Although brief echoes or interruptions may not be distracting if they occur rarely, frequent echoes and interruptions may be very annoying to the listener.

[0269] The resync delay off mode helps avoid unnecessary resynchronization at least in part by increasing a threshold that would otherwise trigger the second playback device 706 to initiate a resynchronization process with the first playback device 704. In operation, when the second playback device 706 is switched to operate in the resync delay off mode, the second playback device 706 may initiate a resynchronization process with the first playback device 704 in response to any one or more of the following: (i) failing to receive portions of the audio information 752 from the first playback device 704 within some third threshold duration that is greater than the first threshold duration used during normal operation (described above), ii) encountering a delay between portions of the audio information 752 received from the first playback device 704 that exceeds a fourth threshold duration that is greater than the first threshold duration used during normal operation (described above), a second threshold duration used during normal operation, (iii) receiving more than a second threshold number of out-of-order portions of the audio information 752, the second threshold number of out-of-order portions of the audio information 752 being greater than the first threshold number of out-of-order portions of the audio information 752 used during normal operation, and / or (iv) receiving (or failing to receive) more (compared to normal operation) portions of the audio information 752 from the first playback device 704 in a manner that affects the ability of the second reply device 706 to consistently and / or reliably reassemble portions of the audio information 752, and / or play audio content based on the audio information 752.

[0270] In some embodiments, the normal operation mode and the resync delay off mode are equivalent to the first operation mode and the second operation mode (or are corresponding to the functional aspects of the first operation mode and the second operation mode). In operation, the synchronization group members operate in the first operation mode while the group coordinator (or other "source" device) is receiving audio information from a reliable audio source (e.g., from a WIFI source, a wired source, or any scenario in which the group coordinator receives audio information in a consistent and reliable manner), and the synchronization group members operate in the second operation mode while the group coordinator (or other "source" device) is receiving audio information from an unreliable audio source (e.g., from a Bluetooth source or any scenario in which the group coordinator receives audio information in an inconsistent and / or unreliable manner).

[0271] VII. Example Method

[0272] Figure 8 An example method 800 is shown for performing by a playback device for playing audio content from an unreliable audio source according to some embodiments. The playback device may be the same or similar to any playback device disclosed and / or described herein, including but not limited to the playback device described in reference to Figure 7 A first playback device 704 is described.

[0273] Method 800 begins at block 802, which includes receiving a first data stream including a first series of frames (or packets), wherein each frame (or packet) in the first series of frames (or packets) has a first format (e.g., a first packet format and / or a first framing format), and wherein the first series of frames (or packets) include encoded audio data. In some embodiments, the encoded audio data is the same or similar to any audio data disclosed and described herein, including but not limited to reference Figure 7 Described audio information 750. In some embodiments, the first data stream is a stream of RTP packets sent via a Bluetooth communication link.

[0274] Next, the method 800 proceeds to block 804, which includes extracting encoded audio data from the first series of frames (or packets).

[0275] In some embodiments, method 800 optionally includes blocks 806 and 808 .

[0276] At optional block 806, method 800 includes determining whether a first series of frames (or packets) including encoded audio data is missing one or more portions of the encoded audio data. In some embodiments, determining whether the first series of frames (or packets) is missing one or more portions of the encoded audio data includes checking one or more of: (i) a sequence identifier in each frame (or packet) of the series of frames (or packets) received from the audio source and / or (ii) a timestamp in each frame (or packet) of the series of frames (or packets) received from the audio source. In operation, checking the sequence identifier and / or timestamp in the incoming frames (or packets) to identify missing frames (or packets) can be performed according to any method disclosed and described herein for identifying missing or lost frames.

[0277] At optional block 808, method 800 includes, in response to determining that the first series of frames (or packets) including the encoded audio data are losing one or more portions of the encoded audio data, generating one or more empty portions corresponding to the one or more lost portions of the encoded audio data. In operation, the duration of each empty portion may be determined according to any method disclosed and described herein for determining a duration corresponding to a lost frame / packet, including but not limited to comparing sequence identifiers and / or timestamps of frames / packets before the lost packet and frames / packets after the lost frame / packet.

[0278] Next, method 800 proceeds to box 810, which includes generating a second series of frames (or packets), wherein each frame (or packet) in the second series of frames (or packets) has a second framing format (or a similar protocol-specific format), and wherein the second series of frames (or packets) include one or more of the following: (i) a portion of the encoded audio data extracted from the first series of frames (or packets), (ii) one or more empty portions corresponding to the lost portion of the encoded audio data (if any), (iii) playback timing information, wherein the playback timing information includes: (iii-a) playback timing of the portion of the encoded audio data extracted from the first series of frames (or packets), and (iii-b) one or more playback times of one or more empty portions corresponding to the one or more lost portions of the encoded audio data (if any). In some embodiments, each frame (or grouping) in the second series of frames (or groups) includes: (i) at least a portion of the encoded audio data extracted from the first series of frames (or groups), and (ii) the playback time of at least a portion of the encoded audio data extracted from the first series of frames (or groups) contained in each frame (or grouping) in the second series of frames (or groups).

[0279] In some embodiments, generating the second series of frames (or packets) includes generating playback timing of audio data contained within the frames (or packets). In some embodiments, generating playback timing information includes generating playback timing of (i) portions of the encoded audio data extracted from the first series of frames (or packets) and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data.

[0280] In some embodiments, generating playback timing of i) portions of encoded audio data extracted from a first series of frames (or packets) and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data includes: (A) generating playback times for respective portions of the encoded audio data, wherein the playback times for respective portions of the encoded audio data include future playback times relative to a clock of a first playback device, at which the first playback device will play the respective portions of the encoded audio data; and (B) generating playback times for respective empty portions, wherein the playback time information for respective empty portions includes future playback times relative to a clock of the first playback device, at which the first playback device will play the respective empty portions.

[0281] In some embodiments, the first framing / packetization format corresponds to a Bluetooth format and the second framing / packetization format corresponds to a WIFI format. In other embodiments, the first framing / packetization format may correspond to a different protocol other than Bluetooth and the second framing / packetization format may correspond to a different protocol other than WIFI. In some embodiments, receiving a first data stream comprising a first series of frames (or packets) comprises receiving the first data stream via a first wireless interface. And in some embodiments, sending a second data stream comprising a second series of frames (or packets) to a second playback device comprises sending the second data stream via a second wireless interface.

[0282] In some embodiments, method 800 includes optional block 812, which includes notifying the second playback device that the second series of frames (or packets) may include more than a threshold number of empty portions. In some embodiments, optional block 812 may additionally or alternatively include notifying the second playback device that a time interval between when the second playback device receives a portion of the encoded audio data and when the second playback device receives a subsequent portion of the encoded audio data may exceed a threshold duration.

[0283] Next, method 800 proceeds to block 814, which includes sending a second data stream (from block 810) including a second series of frames (or packets) to a second playback device. In some embodiments, and as described above, each frame in the second series of frames (or packets) has a second series of frame / packet format, wherein the second series of frames (or packets) includes one or more of the following: (i) a portion of the encoded audio data extracted from the first series of frames (or packets), (ii) one or more empty portions corresponding to the missing portion of the encoded audio data (if any), (iii) playback timing information, wherein the playback timing information includes: (iii-a) playback timing of the portion of the encoded audio data extracted from the first series of frames (or packets), and (iii-b) one or more playback times of the one or more empty portions corresponding to the one or more missing portions of the encoded audio data (if any).

[0284] Next, method 800 proceeds to block 816, which includes using playback timing information to synchronize the audio content with the second playback device based on the portions and empty portions (if any) of the encoded audio data extracted from the first series of frames (or packets). In some embodiments, according to any synchronized playback method disclosed and described herein, the first playback device uses playback timing to synchronize the audio content with the second playback device based on the audio information portions and empty portions (if any).

[0285] For example, in some embodiments, using playback timing information to play audio content synchronously with a second playback device based on portions of encoded audio data extracted from a first series of frames (or packets) includes: (A) playing portions of the encoded audio data extracted from the first series of frames (or packets) when the clock of the first playback device reaches a time indicated by the playback time of the portions of the encoded audio data; and (B) playing each empty portion when the clock of the first playback device reaches a time indicated by the playback time of each empty portion.

[0286] Fig. 9 An example method 900 is shown that may be performed by a playback device in accordance with some embodiments.

[0287] The playback device may be the same as or similar to any playback device disclosed and / or described herein, including but not limited to reference Figure 7 A first playback device 704 is depicted. In some embodiments, and in the example method 900, the playback device is, includes, or at least performs some functionality of a sync group coordinator.

[0288] Method 900 begins at block 902, which includes a group coordinator setting a start time to generate playback timing for synchronous playback of audio information contained in a first data packet stream received from an audio source. As previously described, in some embodiments, the start time is a reference time for the group coordinator to start generating playback timing of audio content to be synchronously played back by the synchronization group.

[0289] For example, a group coordinator in some embodiments described herein receives a first data stream including a first series of frames (or packets), wherein each frame (or packet) in the first series of frames (or packets) has a first format (e.g., a first packet format and / or a first framing format), and wherein the first series of frames (or packets) include encoded audio data. In some embodiments, the group coordinator also generates a second series of frames (or packets), wherein each frame (or packet) in the second series of frames (or packets) has a second framing format (or a similar protocol-specific format), and wherein the second series of frames (or packets) include one or more of the following: (i) a portion of the encoded audio data extracted from the first series of frames (or packets), (ii) one or more empty portions corresponding to the lost portion of the encoded audio data (if any), (iii) playback timing information, wherein the playback timing information includes: (iii-a) playback timing of the portion of the encoded audio data extracted from the first series of frames (or packets), and (iii-b) one or more playback times of one or more empty portions corresponding to one or more lost portions of the encoded audio data (if any). In some embodiments, each frame (or grouping) in the second series of frames (or groups) includes: (i) at least a portion of the encoded audio data extracted from the first series of frames (or groups), and (ii) the playback time of at least a portion of the encoded audio data extracted from the first series of frames (or groups) contained in each frame (or grouping) in the second series of frames (or groups).

[0290] Next, method 900 proceeds to block 904, which includes the group coordinator determining whether the first data packet stream is losing one or more packets. In some embodiments, the group coordinator determines whether the first data packet stream is losing one or more data packets by analyzing sequence numbers (or other overhead information) in packets of the first packet stream. In some embodiments, the group coordinator determines whether the first data packet stream is losing one or more packets includes the group coordinator determining whether the first data packet stream is losing more than a threshold number of packets.

[0291] If at block 904, the group coordinator determines that the first data stream of data packets (or at least a portion of the first data stream that has been received by the group coordinator) is losing more than a threshold number of packets, then the method 900 proceeds to block 906, which includes the group coordinator (i) refreshing (or otherwise removing, deleting and / or de-indexing) packets queued for transmission to the synchronization group via the second data stream (described above) and / or audio samples queued in buffers for playback by the group coordinator according to corresponding playback timings previously determined by the group coordinator, and / or (ii) instructing playback devices in the synchronization group (sometimes referred to as synchronization group members) to each refresh audio samples queued in respective buffers for playback according to corresponding playback timings that were previously received from the group coordinator (via the second data stream sent to the group members by the group coordinator). The method 900 then returns to block 902, which includes the group coordinator setting a new start time to resynchronize the synchronization group playback audio content.

[0292] However, if at block 904, the group coordinator determines that the first data packet stream (or at least a portion of the first data stream that has been received by the group coordinator) is not losing more than a threshold number of packets, then method 904 proceeds to block 908, which includes the group coordinator determining, for the audio content in each packet, a playback time for the audio content within the packet. Recall that the playback time is the future time at which a playback device in the synchronization group will play the audio content within a particular packet (or frame, unit, etc.). In operation, the group coordinator determines the playback time for the audio content in the packet according to any method and / or process disclosed and described herein for determining playback timing.

[0293] Next, method 900 proceeds to frame 910, it comprises group coordinator determines whether the playback time of this audio content is one of the following for the audio content in each grouping: (i) relative to the clock of group coordinator for playing audio content is past, or (ii) less than threshold time amount in the future.If the playback time of this audio content is past, then the playback device in synchronization group may not play back this audio content synchronously with each other according to the playback time of this audio content.Similarly, if the playback time of this audio content is less than the threshold time amount in the future, then the playback device in synchronization group may not have enough time to receive, process and play back this audio content synchronously with each other.This situation may occur, for example, if group coordinator encounters interruption or delay when receiving the first packet stream, and then receives data packet burst, after which group coordinator needs to reorder the received packet burst, extract audio information from the received packet burst, generate the part including the extracted audio information and the second packet stream for sending to the playback timing of other playback devices, and then send the second packet stream to synchronization group.In this case, other playback devices in synchronization group may not have enough time to receive, process and play back this audio content synchronously with each other.

[0294] If, at block 910, the group coordinator determines that the playback time of the audio content in each group is one of: (i) in the past relative to the group coordinator's clock for playing the audio content, or (ii) less than a threshold amount of time in the future, then method 900 proceeds to block 906, which includes the group coordinator (i) refreshing (or otherwise removing, deleting, and / or de-indexing) the audio samples queued for transmission to the synchronization group via the second data stream (described above) and / or queued in the buffer for playback by the group coordinator according to the corresponding playback timing previously determined by the group coordinator, and / or (ii) instructing other synchronization group members to each refresh the audio samples queued in the respective buffers for playback that were previously received from the group coordinator (via the second data stream sent to the group members by the group coordinator). Then, method 900 returns to block 902, which includes the group coordinator setting a new start time to resynchronize the synchronization group playback audio content.

[0295] However, if at block 910, the group coordinator determines that the playback time of the audio content in each packet is not one of: (i) in the past relative to the clock used by the group coordinator to play the audio content, or (ii) less than a threshold amount of time in the future, then the method 900 returns to one of blocks 904 or 908. In operation, in some embodiments, the functionality of block 904 occurs in parallel or at least substantially in parallel with the functionality of blocks 908 and 910. As a result, the group coordinator determines that one (or both) of the following two situations is sufficient for the method 900 to proceed to block 906 and then return to block 902: the incoming data stream is losing one or more packets (or more than a threshold number of incoming packets), and / or the playback time of one or more portions of the audio content is one of the past or less than a threshold amount of time in the future.

[0296] In this manner, method 900 enables a group coordinator to determine if / when playback resynchronization should occur, as well as how to resynchronize playback among a group of playback devices configured to play audio content in sync with one another as described herein.

[0297] 8. Metadata Exchange Across Multiple Networks

[0298] As mentioned above, aspects of the present disclosure relate to technologies for exchanging metadata associated with media content between multiple devices across multiple networks. For example, a playback device can receive media content and metadata associated with the media content from a media source via a PAN (e.g., a Bluetooth network). In this example, the playback device can send metadata associated with the media content to one or more other devices (e.g., one or more control devices and / or playback devices) via a WLAN. In addition, in some cases, the playback device can send media content to one or more other playback devices on the WLAN (e.g., one or more other playback devices that are designated to play back media content synchronously with other playback devices alone or with other playback devices). Therefore, access to metadata associated with the media content being transmitted by a PAN can be provided to one or more devices that are not connected to a PAN.

[0299] Fig.10 An example configuration 1000 of one or more playback devices and one or more control devices exchanging metadata associated with media content is shown. As shown, the configuration 1000 includes a media source device 1002 (e.g., a user device such as a smart phone, a tablet computer, a laptop computer, etc.), which communicates with a playback device 1006 via a first network 1004 (e.g., a PAN such as a Bluetooth network). The playback device 1006 is also connected to a second network 1012 (e.g., a WLAN such as a WIFI network) to facilitate communication with one or more control devices (e.g., user devices such as smart phones, tablet computers, laptop computers, etc.), one or more control devices are shown as control devices 1008a to 1008n, and one or more playback devices are shown as playback devices 1010a to 1010n. In some cases, the media source device 1002 may also be connected to the second network 1012. In other cases, the media source device 1002 may not be connected to the second network 1012.

[0300] Playback device 1006 can receive media content and metadata associated with the media content for playback from media source device via first network 1004. Examples of metadata associated with media content include: title, artist, album, track length, current search position in track, content source (e.g., SPOTIFY, AMAZONMUSIC, APPLEMUSIC, local storage, etc.) and album cover. For example, when a user starts to play back an album on media source device 1002, playback device 1006 can receive the first metadata set associated with the album (e.g., album name, album cover, first track name, first track duration, etc.) when starting playback. After starting playback, playback device 1006 can receive additional metadata when one or more events associated with media content occur. For example, when current track is changed (e.g., by user input or after previous track naturally ends), playback device 1006 can receive additional metadata such as the name and / or duration of new track.

[0301] After playback device 1006 receives metadata associated with the media content to be played back, playback device 1006 may send metadata associated with the media content to one or more of control devices 1008a to 1008n via second network 1012. In some cases, playback device 1006 may enhance metadata before sending to one or more control devices. For example, metadata received from media source device 1002 may include an album title but not an album cover image. In this example, playback device 1006 may access the associated album cover image from one or more remote servers (e.g., connected to playback device 1006 via a WAN such as the Internet) using the album title. In other cases, playback device 1006 may relay metadata associated with the media content to one or more control devices without enhancement. One or more of control devices 1008a to 1008n may present the received metadata to a user. For example, one or more control devices 1008a-1008n may present a graphical user interface (GUI) to a user that includes an indication of which playback devices in the media playback system are playing back media content received via the first network 1004, and / or an indication of what is being played back (e.g., one or more pieces of metadata associated with the media content).

[0302] In some cases, playback device 1006 may send metadata to one or more other playback devices 1010a to 1010n via second network 1012. For example, playback device 1006 may send metadata to all other playback devices connected to second network 1012 (e.g., all other playback devices in the system). In other examples, playback device 1006 may send metadata to only a subset of other playback devices connected to second network 1012. To illustrate, playback device 1006 may send metadata to only those other playback devices that are currently (e.g., synchronized with playback device 1006) playing back media content from media source device 1002. In another scenario, playback device 1006 may send metadata to only those other playback devices that have certain characteristics (e.g., those other playback devices that include a display or other output device through which metadata can be transmitted to a user).

[0303] It should be understood that in some cases, playback device 1006 can send metadata back to media source device 1002 via second network 1012. For example, media source device 1002 can be a user device that executes a software application associated with a media playback system that facilitates playback and control of playback device 1006 and / or playback devices 1010a to 1010n via second network 1012. In some cases, the software application may not be able to directly access the media content (and / or associated metadata) being streamed via first network 1004 (e.g., the media is being streamed from another application on the user device). Therefore, the software application may not know that the media is being played back on the media playback system. Therefore, playback device 1006 can notify such software application on the user device via second network 1012.

[0304] IX. Conclusion

[0305] The above discussion of playback devices, controller devices, playback zone configurations, and media / audio content sources only provides some examples of operating environments in which the functions and methods described below can be implemented. Configurations and other operating environments of media playback systems, playback devices, and network devices not explicitly described herein may also be applicable and suitable for the implementation of the functions and methods.

[0306] The above description discloses, among other things, various example systems, methods, devices, and articles of manufacture, including, among other things, firmware and / or software executed on hardware. It should be understood that these examples are merely illustrative and should not be considered restrictive. For example, it is contemplated that any or all of these firmware, hardware, and / or software aspects or components may be implemented exclusively in hardware, exclusively in software, exclusively in firmware, or in any combination of hardware, software, and / or firmware. Therefore, the examples provided are not the only way to implement these systems, methods, devices, and / or articles of manufacture.

[0307] In addition, references herein to "embodiments" mean that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one exemplary embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Therefore, it should be explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0308] This specification is presented primarily in terms of illustrative environments, systems, processes, steps, logic blocks, processing, and other symbolic representations that are directly or indirectly similar to the operations of data processing devices coupled to a network. Those skilled in the art typically use these processing descriptions and representations to communicate their work content to other technicians of the field. Various specific details are set forth to provide a thorough understanding of the present disclosure. However, those skilled in the art will appreciate that the present disclosure can be implemented without specific, specific details. In other instances, well-known methods, processes, components, and circuits are not described to avoid unnecessarily obscuring aspects of the embodiments. Therefore, the scope of the present disclosure is defined by the appended claims, rather than the description of the above embodiments.

[0309] When any of the following claims is understood to cover a pure software and / or firmware implementation, at least one element in at least one example is expressly defined herein to include a non-transitory tangible medium storing the software and / or firmware, such as a memory, DVD, CD, Blu-ray, etc.

[0310] Example Features

[0311] (Feature 1) A first playback device includes: one or more processors; and a tangible, non-transitory computer-readable medium including instructions stored therein, wherein the instructions, when executed, cause the first playback device to perform functions including: receiving a first data stream including a first series of frames, wherein each packet in the first series of frames has a first format, and wherein the first series of frames includes encoded audio data; extracting the encoded audio data from the first series of frames; generating a second series of frames, wherein each frame in the second series of frames has a second format, and wherein the second series of frames includes (i) a portion of the encoded audio data extracted from the first series of frames and (ii) playback timing information, wherein the playback timing information includes playback timing of the portion of the encoded audio data extracted from the first series of frames; sending a second data stream including the second series of frames to a second playback device; and using the playback timing information to play audio content in synchronization with the second playback device based on the portion of the encoded audio data extracted from the first series of frames.

[0312] (Feature 2) A first playback device according to Feature 1, wherein the function further includes: determining whether a first series of frames including encoded audio data is losing one or more portions of the encoded audio data; and in response to determining that the first series of frames including encoded audio data is losing one or more portions of the encoded audio data, generating one or more empty portions corresponding to the one or more lost portions of the encoded audio data, wherein the second series of frames also includes one or more empty portions corresponding to the one or more lost portions of the encoded audio data, and wherein the playback timing information also includes one or more playback times of the one or more empty portions corresponding to the one or more lost portions of the encoded audio data.

[0313] (Feature 3) A first playback device according to Feature 2, wherein whether a first series of frames including encoded audio data is losing one or more portions of the encoded audio data is determined based on one or more of (i) analysis of a sequence identifier corresponding to the encoded audio data received via the first data stream, or (ii) analysis of a timestamp corresponding to the encoded audio data received via the first data stream.

[0314] (Feature 4) The first playback device according to Feature 2, wherein the function further includes: notifying the second playback device that the second series of frames may include more than a threshold number of empty portions.

[0315] (Feature 5) A first playback device according to Feature 1, wherein the function also includes: generating playback timing of (i) a portion of the encoded audio data extracted from the first series of frames and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data.

[0316] (Feature 6) A first playback device according to Feature 5, wherein generating the playback timing of (i) portions of the encoded audio data extracted from the first series of frames and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data includes: generating playback times for respective portions of the encoded audio data, wherein the playback times for respective portions of the encoded audio data include future playback times relative to a clock of the first playback device, at which the first playback device will play the respective portions of the encoded audio data; and generating playback times for respective empty portions, wherein the playback times for the respective empty portions include future playback times relative to a clock of the first playback device, at which the first playback device will play the respective empty portions.

[0317] (Feature 7) According to the first playback device described in Feature 6, using playback timing information to play audio content synchronously with the second playback device based on parts of the encoded audio data extracted from the first series of frames includes: when the clock of the first playback device reaches the time indicated by the playback time of the respective parts of the encoded audio data, playing the respective parts of the encoded audio data extracted from the first series of frames; and when the clock of the first playback device reaches the time indicated by the playback time of the respective empty parts, playing the respective empty parts.

[0318] (Feature 8) The first playback device according to Feature 1, wherein the first format is associated with a Bluetooth transmission protocol, and wherein the second format is associated with a WIFI transmission protocol.

[0319] (Feature 9) A first playback device according to Feature 1, wherein the function also includes: notifying the second playback device that a time interval between when the second playback device receives a portion of the encoded audio data and when the second playback device receives a subsequent portion of the encoded audio data may exceed a threshold duration.

[0320] (Feature 10) A first playback device according to Feature 1, wherein each frame in the second series of frames includes: (i) at least a portion of the encoded audio data extracted from the first series of frames, and (ii) the playback time of at least a portion of the encoded audio data extracted from the first series of frames contained in each frame in the second series of frames.

[0321] (Feature 11) A first playback device according to Feature 1, wherein receiving a first data stream including a first series of frames includes receiving the first data stream via a first wireless interface, and wherein sending a second data stream including a second series of frames to a second playback device includes sending the second data stream via a second wireless interface.

[0322] (Feature 12) A tangible, non-transitory computer-readable medium includes instructions stored therein, wherein the instructions, when executed, cause a first playback device to perform functions including: receiving a first data stream including a first series of frames, wherein each packet in the first series of frames has a first format, and wherein the first series of frames includes encoded audio data; extracting the encoded audio data from the first series of frames; generating a second series of frames, wherein each frame in the second series of frames has a second format, and wherein the second series of frames includes (i) a portion of the encoded audio data extracted from the first series of frames and (ii) playback timing information, wherein the playback timing information includes playback timing of the portion of the encoded audio data extracted from the first series of frames; sending a second data stream including the second series of frames to a second playback device; and using the playback timing information to play audio content in synchronization with the second playback device based on the portion of the encoded audio data extracted from the first series of frames.

[0323] (Feature 13) A tangible, non-transitory computer-readable medium according to Feature 12, wherein the function also includes: determining whether a first series of frames including encoded audio data is losing one or more portions of the encoded audio data; and in response to determining that the first series of frames including encoded audio data is losing one or more portions of the encoded audio data, generating one or more empty portions corresponding to the one or more lost portions of the encoded audio data, wherein the second series of frames also includes one or more empty portions corresponding to the one or more lost portions of the encoded audio data, and wherein the playback timing information also includes one or more playback times of the one or more empty portions corresponding to the one or more lost portions of the encoded audio data.

[0324] (Feature 14) A tangible, non-transitory computer-readable medium according to Feature 13, wherein a determination is made as to whether a first series of frames comprising encoded audio data is missing one or more portions of the encoded audio data based on one or more of (i) an analysis of a sequence identifier corresponding to the encoded audio data received via the first data stream, or (ii) an analysis of a timestamp corresponding to the encoded audio data received via the first data stream.

[0325] (Feature 15) A tangible, non-transitory computer-readable medium according to Feature 13, wherein the functions also include one or more of the following: (i) notifying a second playback device that the second series of frames may include more than a threshold number of empty portions, or (ii) notifying the second playback device that a time interval between when the second playback device receives a portion of the encoded audio data and when the second playback device receives a subsequent portion of the encoded audio data may exceed a threshold duration.

[0326] (Feature 16) A tangible, non-transitory computer-readable medium according to Feature 1, wherein the functions also include: generating playback timing of (i) portions of the encoded audio data extracted from the first series of frames and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data.

[0327] (Feature 17) A tangible, non-transitory computer-readable medium according to Feature 16, wherein generating playback timing of (i) portions of encoded audio data extracted from a first series of frames and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data includes: generating playback times for respective portions of the encoded audio data, wherein the playback times for respective portions of the encoded audio data include future playback times relative to a clock of a first playback device, at which the first playback device will play the respective portions of the encoded audio data; and generating playback times for respective empty portions, wherein the playback times for the respective empty portions include future playback times relative to a clock of the first playback device, at which the first playback device will play the respective empty portions.

[0328] (Feature 18) A tangible, non-transitory computer-readable medium according to Feature 17, wherein using playback timing information to play audio content synchronously with a second playback device based on portions of encoded audio data extracted from a first series of frames includes: playing the portions of encoded audio data extracted from the first series of frames when the clock of the first playback device reaches a time indicated by the playback time of the portions of the encoded audio data; and playing the empty portions when the clock of the first playback device reaches a time indicated by the playback time of the empty portions.

[0329] (Feature 19) The tangible, non-transitory computer-readable medium of Feature 12, wherein the first format is associated with a Bluetooth transmission protocol, and wherein the second format is associated with a WIFI transmission protocol.

[0330] (Feature 20) A tangible, non-transitory computer-readable medium according to Feature 12, wherein each frame in the second series of frames includes: (i) at least a portion of the encoded audio data extracted from the first series of frames, and (ii) a playback time of at least a portion of the encoded audio data extracted from the first series of frames contained within each frame in the second series of frames.

[0331] (Feature 21) The first playback device includes: a first wireless communication interface configured to facilitate communication via at least one personal area network (PAN); a second wireless communication interface configured to facilitate communication via at least one wireless local area network (WLAN); one or more processors; at least one non-transitory computer-readable medium including program instructions, wherein the program instructions are executable by at least one processor so that the first playback device is configured to perform the following operations: using the first wireless communication interface, receiving a first data stream including a first series of frames from a first user device, wherein each packet in the first series of frames has a first format, and wherein the first series of frames includes encoded audio data; extracting the encoded audio data from the first series of frames; generating a second series of frames, wherein each frame in the second series of frames has a second format, and wherein the second series of frames includes (i) a portion of the encoded audio data extracted from the first series of frames and (ii) playback timing information, wherein the playback timing information includes playback timing of the portion of the encoded audio data extracted from the first series of frames; using the second wireless communication interface, sending a second data stream including the second series of frames to a second playback device; and using the playback timing information to synchronize the playback of audio content with the second playback device based on the portion of the encoded audio data extracted from the first series of frames.

[0332] (Feature 22) The first playback device according to Feature 21, wherein the first format is associated with a Bluetooth transmission protocol, and wherein the second format is associated with a WIFI transmission protocol.

[0333] (Feature 23) A first playback device according to any one of Features 21 and 22, wherein each frame in the second series of frames includes: (i) at least a portion of the encoded audio data extracted from the first series of frames, and (ii) the playback time of at least a portion of the encoded audio data extracted from the first series of frames contained in each frame in the second series of frames.

[0334] (Feature 24) A first playback device according to any one of Features 21 to 23, wherein at least one non-transitory computer-readable medium also includes program instructions that can be executed by at least one processor so that the first playback device is configured to perform the following operations: receive first metadata associated with the encoded audio content from a first user device using a first communication interface; and based on the first metadata, send second metadata associated with the encoded audio content to a second user device using a second communication interface.

[0335] (Feature 25) The first playback device according to Feature 24, wherein at least one of the first metadata or the second metadata includes at least one of the following: artist name, song title, song art, song duration, album title, album cover, or album duration.

[0336] (Feature 26) The first playback device according to any one of Features 24 and 25, wherein the first metadata is the same as the second metadata.

[0337] (Feature 27) The first playback device according to any one of Features 24 and 25, wherein the first metadata is different from the second metadata.

[0338] (Feature 28) A first playback device according to any one of Features 21 to 27, wherein at least one non-transitory computer-readable medium also includes program instructions that can be executed by at least one processor so that the first playback device is configured to perform the following operations: determine whether a first series of frames including encoded audio data is losing one or more portions of the encoded audio data; and in response to determining that the first series of frames including encoded audio data is losing one or more portions of the encoded audio data, generate one or more empty portions corresponding to the one or more lost portions of the encoded audio data, wherein the second series of frames also includes one or more empty portions corresponding to the one or more lost portions of the encoded audio data, and wherein the playback timing information also includes one or more playback times of the one or more empty portions corresponding to the one or more lost portions of the encoded audio data.

[0339] (Feature 29) A first playback device according to Feature 28, wherein the program instructions executable by at least one processor so that the first playback device is configured to determine whether a first series of frames including encoded audio data is losing one or more portions of the encoded audio data include the following program instructions, which are executable by at least one processor so that the first playback device is configured to perform the following operations: determining whether the first series of frames including encoded audio data is losing one or more portions of the encoded audio data based on one or more of (i) analysis of a sequence identifier corresponding to the encoded audio data received via the first data stream, or (ii) analysis of a timestamp corresponding to the encoded audio data received via the first data stream.

[0340] (Feature 30) A first playback device according to any one of Features 21 to 29, wherein at least one non-transitory computer-readable medium also includes program instructions that can be executed by at least one processor so that the first playback device is configured to perform the following operations: generate playback timing of (i) portions of encoded audio data extracted from the first series of frames and (ii) one or more empty portions corresponding to one or more missing portions of the encoded audio data.

[0341] (Feature 31) A first playback device according to Feature 30, wherein program instructions executable by at least one processor so that the first playback device is configured to generate playback timing information include the following program instructions, and the program instructions executable by at least one processor so that the first playback device is configured to perform the following operations: generating playback times for respective portions of the encoded audio data, wherein the playback times for respective portions of the encoded audio data include future playback times relative to a clock of the first playback device, at which the first playback device will play the respective portions of the encoded audio data; and generating playback times for respective empty portions, wherein the playback times for respective empty portions include future playback times relative to a clock of the first playback device, at which the first playback device will play the respective empty portions.

[0342] (Feature 32) A first playback device according to any one of Features 21 to 31, wherein program instructions executable by at least one processor so that the first playback device is configured to use playback timing information to play audio content include program instructions executable by at least one processor so that the first playback device is configured to perform the following operations: when the clock of the first playback device reaches the time indicated by the playback time of the respective parts of the encoded audio data, play the respective parts of the encoded audio data extracted from the first series of frames; and when the clock of the first playback device reaches the time indicated by the playback time of the respective empty parts, play the respective empty parts.

[0343] (Feature 33) A first playback device according to any one of Features 21 to 32, wherein each frame in the second series of frames includes: (i) at least a portion of the encoded audio data extracted from the first series of frames, and (ii) a playback time of at least a portion of the encoded audio data extracted from the first series of frames contained in each frame in the second series of frames.

[0344] (Feature 34) A method for operating a first playback device, the method comprising: using a first communication interface configured to facilitate communication via at least one personal area network, receiving a first data stream comprising a first series of frames, wherein each packet in the first series of frames has a first format, and wherein the first series of frames comprises encoded audio data; extracting the encoded audio data from the first series of frames; generating a second series of frames, wherein each frame in the second series of frames has a second format, and wherein the second series of frames comprises (i) a portion of the encoded audio data extracted from the first series of frames and (ii) playback timing information, wherein the playback timing information comprises playback timing of the portion of the encoded audio data extracted from the first series of frames; using a second communication interface configured to facilitate communication via at least one wireless local area network, sending a second data stream comprising the second series of frames to a second playback device; and using the playback timing information to play audio content in synchronization with the second playback device based on the portion of the encoded audio data extracted from the first series of frames.

[0345] (Feature 35) At least one non-transitory computer-readable medium comprising program instructions executable by at least one processor so that a first playback device is configured to perform the following operations: using a first wireless communication interface of the first playback device, receiving a first data stream comprising a first series of frames from a first user device, wherein each packet in the first series of frames has a first format, and wherein the first series of frames comprises encoded audio data; extracting the encoded audio data from the first series of frames; generating a second series of frames, wherein each frame in the second series of frames has a second format, and wherein the second series of frames comprises (i) a portion of the encoded audio data extracted from the first series of frames and (ii) playback timing information, wherein the playback timing information comprises playback timing of the portion of the encoded audio data extracted from the first series of frames; using a second wireless communication interface of the first playback device, sending a second data stream comprising the second series of frames to a second playback device; and using the playback timing information to play audio content in synchronization with the second playback device based on the portion of the encoded audio data extracted from the first series of frames.

[0346] (Feature 36) At least one non-transitory computer-readable medium according to Feature 35, wherein each frame in the second series of frames includes: (i) at least a portion of the encoded audio data extracted from the first series of frames, and (ii) a playback time of at least a portion of the encoded audio data extracted from the first series of frames contained in each frame in the second series of frames.

[0347] (Feature 37) At least one non-transitory computer-readable medium according to any one of Features 35 and 36, wherein the at least one non-transitory computer-readable medium also includes program instructions that can be executed by at least one processor so that the first playback device is configured to perform the following operations: using a first communication interface to receive first metadata associated with the encoded audio content from a first user device; and based on the first metadata, using a second communication interface to send second metadata associated with the encoded audio content to a second user device.

[0348] (Feature 38) At least one non-transitory computer-readable medium according to Feature 37, wherein at least one of the first metadata or the second metadata includes at least one of the following: artist name, song title, song art, song duration, album title, album cover, or album duration.

[0349] (Feature 39) At least one non-transitory computer-readable medium according to any one of Features 37 and 38, wherein the first metadata is the same as the second metadata.

[0350] (Feature 40) The at least one non-transitory computer-readable medium according to any one of Features 37 and 38, wherein the first metadata is different from the second metadata.

[0351] (Feature 41) A first playback device comprises: a first wireless communication interface configured to facilitate communication via at least one personal area network (PAN); a second wireless communication interface configured to facilitate communication via at least one wireless local area network (WLAN); one or more processors; and at least one non-transitory computer-readable medium comprising program instructions, wherein the program instructions are executable by at least one processor such that the first playback device is configured to perform the following operations: using the first wireless communication interface, receiving a first data stream comprising a first series of frames from a first user device, wherein each packet in the first series of frames has a first format, and wherein the first series of frames comprises encoded audio data; and receiving from a first user device a first data stream comprising a first series of frames; wherein each packet in the first series of frames has a first format, and wherein the first series of frames comprises encoded audio data; and The invention relates to a method for extracting encoded audio data from a series of frames; generating a second series of frames, wherein each frame in the second series of frames has a second format, and wherein the second series of frames includes (i) a portion of the encoded audio data extracted from the first series of frames and (ii) playback timing information, wherein the playback timing information includes the playback timing of the portion of the encoded audio data extracted from the first series of frames; using a second wireless communication interface, sending a second data stream including the second series of frames to a second playback device; using the first communication interface to receive first metadata associated with the encoded audio content from the first user device; and based on the first metadata, using the second communication interface to send second metadata associated with the encoded audio content to the second user device.

[0352] (Feature 42) The first playback device according to Feature 41, wherein the first playback device does not play back the audio content based on the encoded audio data.

[0353] (Feature 43) The first playback device according to Feature 41, wherein the first playback device plays back the audio content at a minimum volume level (eg, zero volume level, silence, etc.) based on the portion of the encoded audio data.

[0354] (Feature 44) The first playback device according to any one of Features 21 to 33 and 41 to 43, wherein the first playback device is a portable playback device including a battery.

[0355] (Feature 45) The first playback device according to any one of Features 21 to 33 and 41 to 43, wherein the first playback device is a wearable device configured to be worn around a portion of the object.

[0356] (Feature 46) The first playback device according to Feature 45, wherein the first playback device is a headphone device.

[0357] (Feature 47) The first playback device according to any one of Features 21 to 33 and Features 41 to 46, wherein the first playback device is a screenless playback device (eg, does not include a display screen).

Claims

1. A method for a first playback device, the method comprising: receiving, using a first communication interface configured to facilitate communication over at least one personal area network, a first data stream comprising a first series of frames including encoded audio data, wherein the first series of frames has a first format, extracting the encoded audio data from the first series of frames; generating a second series of frames having a second format, the second format being different from the first format, the second series of frames including (i) a portion of the encoded audio data extracted from the first series of frames, and (ii) playback timing information starting from a reference start time, the playback timing information including a playback timing of the portion of the encoded audio data extracted from the first series of frames; sending a second data stream including the second series of frames to a second playback device using a second communication interface configured to facilitate communication over at least one wireless local area network; Playing the audio content synchronously with the second playback device based on the following: (i) a portion of said encoded audio data extracted from said first series of frames; as well as (ii) the playback timing information; determining whether the first series of frames including the encoded audio data is missing more than a threshold amount of portions of the encoded audio data while playing audio content in synchronization with the second playback device based on the portions of the encoded audio data extracted from the first series of frames and the playback timing information; as well as When the first series of frames is missing more than a threshold number of portions of the encoded audio data, resynchronizing playback of the audio content with the second playback device based on: (i) (a) deleting data packets queued for transmission to the second playback device and / or deleting audio data extracted from the first series of frames queued in a buffer for playback by the first playback device, and (b) sending one or more commands to the second playback device to cause the second playback device to delete at least some of the second series of frames queued in its current buffer for playback, and (ii) resetting the reference start time, and regenerating the playback timing information of the second series of frames starting from the reference start time.

2. The method according to claim 1, wherein: The first format is associated with a Bluetooth transmission protocol, and The second format is associated with the WIFI transmission protocol.

3. The method according to claim 1 or 2, wherein: Each frame in the second series of frames comprises: at least a portion of the encoded audio data extracted from the first series of frames; and A playback time of at least a portion of the encoded audio data extracted from the first series of frames contained within each frame of the second series of frames.

4. The method according to claim 1, wherein: Generating the second series of frames includes: when the first series of frames including the encoded audio data loses one or more portions of the encoded audio data, generating one or more empty portions and corresponding playback timing information corresponding to the lost one or more portions of the encoded audio data.

5. The method of claim 1 , further comprising determining whether the first series of frames comprising the encoded audio data is missing more than a threshold amount of portions of the encoded audio data based on one or more of: (i) analysis of a sequence identifier corresponding to said encoded audio data received via said first data stream, or (ii) analysis of time stamps corresponding to the encoded audio data received via the first data stream.

6. The method according to claim 1, further comprising: The playback timing information is generated for (i) portions of the encoded audio data extracted from the first series of frames and (ii) one or more empty portions corresponding to lost one or more portions of the encoded audio data.

7. The method according to claim 6, wherein: Generating the playback timing information includes: generating playback times for respective portions of the encoded audio data, wherein the playback times for respective portions of the encoded audio data comprise future playback times relative to a clock of the first playback device at which the first playback device will play the respective portions of the encoded audio data; and A playback time for each empty portion is generated, wherein the playback time for each empty portion includes a future playback time relative to a clock of the first playback device at which the first playback device will play the each empty portion.

8. The method according to claim 7, wherein: Playing the audio content based on the playback timing information includes: playing the respective portions of the encoded audio data extracted from the first series of frames when the clock of the first playback device reaches the time indicated by the playback time of the respective portions of the encoded audio data; and When the clock of the first playback device reaches the time indicated by the playback time of the respective empty sections, the respective empty sections are played.

9. The method according to any one of claims 4 to 8, wherein: Each frame in the second series of frames comprises: at least a portion of the encoded audio data extracted from the first series of frames; and A playback time of at least a portion of the encoded audio data extracted from the first series of frames contained within each frame of the second series of frames. 10 . A non-transitory computer-readable medium comprising program instructions executable by at least one processor to cause a first playback device to be configured to perform the method of any one of claims 1 to 9 .

11. A first playback device, comprising: a first wireless communication interface configured to facilitate communication via at least one personal area network PAN; A second wireless communication interface, configured to facilitate communication via at least one wireless local area network WLAN; one or more processors; as well as At least one non-transitory computer-readable medium comprising program instructions executable by the one or more processors to cause the first playback device to be configured to perform the method of any one of claims 1 to 9.

Images