Multiplexing of application channels on an isochronous stream

Abstract

A first device is provided. The first device includes an audio source, a sensor, and a processor. The audio source generates audio data, and the sensor captures sensor data. The processor generates a data packet that includes an audio data set generated by the audio source and a sensor data set captured by the sensor. In some examples, the data packet may also include audio payload length data and / or sensor payload length data, audio channel identification data and / or sensor channel identification data, and / or audio time offset data and / or sensor time offset data. The audio data set may have a first duration, and the sensor data set may have a second duration that is longer than the first duration. The processor transmits the data packet to a second device configured to reconstruct the audio data set and the sensor data set by demultiplexing the data packet.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 366,041, filed on June 8, 2022, entitled "Multiplexing Application Channels on Isochronous Streams", which is hereby incorporated by reference in its entirety.

Background Art

[0002] Immersive audio rendered in virtual reality or augmented reality applications often requires the capture of data in multiple modes by wearable audio devices. This captured data must be wirelessly transmitted to a central device (such as a smartphone) for further processing to provide an immersive audio experience to the user.

Summary of the Invention

[0003] This disclosure generally pertains to the transmission of two or more types of data by isochronous streams via a multiplexing scheme. These types of data may include a variety of audio data (such as data captured by a microphone or data used by an acoustic transducer to generate audio) and / or non-audio data (such as data collected by different types of non-audio sensors). The multiplexing scheme can incorporate time offset values ​​to enable temporally accurate demultiplexing and reconstruction of the data transmitted by the isochronous stream. The multiplexing scheme may also incorporate data relating to the payload length of the data packets and channel identification information relating to the source of the captured data. The isochronous stream may be a connected isochronous stream (CIS) or a broadcast isochronous stream (BIS), depending on the application. Sensor data may include a wide variety of different data types, such as motion data captured by an inertial measurement unit (IMU).

[0004] In general, in one embodiment, a first device is provided. The first device includes an audio source. The audio source is configured to generate audio data.

[0005] The first device further includes a sensor, which is configured to acquire sensor data. For example, the sensor may be an inertial measurement unit (IMU). In addition to this example, the sensor data may be motion data.

[0006] The first device further includes a processor, which is configured to generate data packets. The data packets include an audio dataset generated by an audio source and a sensor dataset captured by a sensor. In one example, the data packets may further include audio payload length data and / or sensor payload length data. In another example, the data packets may further include audio channel identification data and / or sensor channel identification data. In yet another example, the data packets may further include audio time offset data and / or sensor time offset data. In yet another example, the audio dataset may have a first lifetime, and the sensor dataset may have a second lifetime that is longer than the first lifetime.

[0007] The processor is further configured to transmit data packets to a second device. The second device is configured to reconstruct audio and sensor datasets by demultiplexing the data packets. For example, data packets may be transmitted via a Bluetooth Connected isochronous stream or a Bluetooth Broadcast isochronous stream.

[0008] In one example, the first device could be a wearable audio device, and the second device could be a central device. In an alternative example, the first device could be a central device, and the second device could be a wearable audio device.

[0009] For example, the processor is further configured to (1) receive an audio data acknowledgment before the first lifetime expires, (2) generate a second data packet containing a sensor dataset, and (3) transmit the second data packet to a second device.

[0010] For example, the processor is further configured to (1) generate a second audio dataset via an audio source after the audio dataset has finished, (2) generate a second data packet containing the second audio dataset and the sensor dataset via the processor of the first device, and (3) transmit the second data packet to the second device. In addition to this example, the processor may be further configured to (1) capture a second sensor dataset via the sensor of the first device after the sensor dataset has finished, (2) generate a third data packet containing the second audio dataset and the second sensor dataset, and (3) transmit the third data packet to the second device.

[0011] In general, in another embodiment, a method for transmitting data is provided. The method includes capturing an audio dataset via an audio source of a first device.

[0012] The method further includes acquiring a sensor dataset via the sensors of the first device.

[0013] The method further comprises generating data packets via the processor of a first device. The data packets include an audio dataset and a sensor dataset. In one example, the data packets may further include audio payload length data and / or sensor payload length data. In another example, the data packets may further include audio channel identification data and / or sensor channel identification data. In yet another example, the data packets may further include audio time offset data and / or sensor time offset data. In yet another example, the audio dataset has a first duration, and the sensor dataset has a second duration that is longer than the first duration.

[0014] The method further includes transmitting data packets to a second device via a transceiver of a first device. For example, the data packets may be transmitted via a Bluetooth connected isochronous stream or a Bluetooth broadcast isochronous stream.

[0015] The method further includes receiving data packets via the transceiver of a second device.

[0016] The method further includes reconstructing the audio dataset and sensor dataset by demultiplexing the data packets via the processor of a second device.

[0017] For example, the method may further include (1) receiving an audio data acknowledgment via a transceiver of a first device before the first lifetime ends, (2) generating a second data packet containing a sensor dataset via a processor of the first device, and (3) transmitting the second data packet to a central device via a transceiver of the first device.

[0018] For example, the method may further include (1) generating a second audio dataset via an audio source after the audio dataset has finished, (2) generating a second data packet containing the second audio dataset and the sensor dataset via the processor of the first device, and (3) transmitting the second data packet to the second device via the transceiver of the first device. In addition to this example, the method may further include (1) capturing a second sensor dataset via the sensor of the first device after the sensor dataset has finished, (2) generating a third data packet containing the second audio dataset and the second sensor dataset via the processor of the first device, and (3) transmitting the third data packet to the second device via the transceiver of the first device.

[0019] In various embodiments, a processor or controller may be associated with one or more storage media (collectively referred to herein as “memory,” and include, for example, volatile and non-volatile computer memories such as ROM, RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, magnetic tapes, flash memory, OTP-ROMs, SSDs, HDDs, etc.). In some implementations, the storage media may be encrypted with one or more programs that, when running on one or more processors and / or controllers, perform at least some of the functions described herein. Various storage media may be fixed within the processor or controller or be portable, thereby allowing one or more programs stored on the storage media to be loaded into the processor or controller to perform various embodiments described herein. The terms “program” or “computer program” are used herein in a general sense to refer to any type of computer code (e.g., software or microcode) that may be used to program one or more processors or controllers.

[0020] It should be understood that all combinations of the aforementioned concepts and any additional concepts discussed in more detail below (provided that such concepts are not mutually contradictory) are intended to be part of the subject matter of the invention disclosed herein. Specifically, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter of the invention disclosed herein. It should also be understood that terms expressly used herein, which may also appear in any disclosure incorporated by reference, should be given meanings that most coincide with the specific concepts disclosed herein.

[0021] Other features and advantages will become apparent from this specification and the claims.

[0022] In the drawings, the same reference numerals generally refer to the same parts throughout different figures. Also, the drawings are not necessarily to scale; instead, emphasis is generally placed on illustrating the principles of the various embodiments.

Brief Description of the Drawings

[0023] [Figure 1] FIG. 1 is a schematic diagram of a first embodiment of a system for wireless communication by way of example. [Figure 2] FIG. 2 is a schematic diagram of a second embodiment of a system for wireless communication by way of example. [Figure 3] FIG. 3 is a flowchart showing multiplexing and demultiplexing using the Bluetooth protocol by way of example. [Figure 4] FIG. 4 illustrates a data packet including a connected isochronous stream (CIS) multiplexing header by way of example. [Figure 5] FIG. 5 illustrates a data packet including a super service data unit (SSDU) having a CIS multiplexing header by way of example. [Figure 6] FIG. 6 schematically illustrates an example of a Bluetooth isochronous adaption layer (ISOAL) by way of example. [Figure 7] FIG. 7 schematically illustrates a further example of a Bluetooth ISOAL having additional layers by way of example. [Figure 8] FIG. 8 is a flowchart illustrating wireless communication between a wearable audio device and a central device by way of example. [Figure 9] FIG. 9 is a further flowchart illustrating wireless communication between a wearable audio device and a central device by way of example. [Figure 10] FIG. 10 is a schematic diagram of a first device by way of example. [Figure 11] FIG. 11 is a schematic diagram of a second device by way of example. [Figure 12]This is a flowchart illustrating an example of a method for transmitting data. [Figure 13] This is another flowchart illustrating a method for transmitting data, as an example. [Figure 14] This is a further flowchart illustrating an example of a method for transmitting data. [Modes for carrying out the invention]

[0024] This disclosure generally pertains to the transmission of two or more types of data over an isochronous stream via a multiplexing scheme. These types of data may include a variety of audio data (such as data captured by a microphone or data used by an acoustic transducer to generate audio) and / or non-audio data (such as data collected by various types of non-audio sensors). The multiplexing scheme can incorporate time offset values ​​to enable temporally accurate demultiplexing and reconstruction of the data transmitted over the isochronous stream. The multiplexing scheme can also incorporate data relating to the payload length of the data packets and channel identification information relating to the source of the captured data.

[0025] In one non-limiting example, a wearable audio headset is used with a personal computer (PC) for gaming purposes. The headset may connect to the PC via an isochronous stream. The headset includes a microphone for capturing the user's voice as audio data and sensors, such as an inertial measurement unit (IMU), for capturing the user's head movements as sensor data. The headset's processor can multiplex the audio data with the sensor data so that the headset transmits data packets containing both audio and sensor data to the PC via an isochronous stream. The data packets also include time offset data indicating when each payload of audio or sensor data was captured. The PC receives the data packets, demultiplexes the packets, and uses the time offset to reconstruct the received audio and sensor data at the appropriate time. In further examples, other types of audio or non-audio data may be multiplexed and demultiplexed.

[0026] In another non-limiting example, a vehicle's onboard computing system is used in conjunction with a pair of wireless earphones worn by the vehicle's driver. The wireless earphones may be connected to the onboard computing system via an isochronous stream. The onboard computing system includes audio sources for generating audio data corresponding to a navigation subsystem, an entertainment subsystem, or any other onboard subsystem capable of generating audio. The onboard computing system also includes sensors configured to capture sensor data regarding the vehicle's movement (such as speed or direction). The processor of the onboard computing system can multiplex the audio data with the sensor data so that the onboard computing system transmits data packets containing both audio data and sensor data to the wireless earphones via an isochronous stream. The data packets may also include time offset data indicating when each payload of audio data or sensor data was captured. The wireless earphones receive the data packets, demultiplex the packets, and use the time offset to reconstruct the received audio and sensor data at the appropriate time. Thus, the multiplexed audio and sensor data transmitted from the onboard computing system to the wireless earphones can be used to create an immersive audio experience incorporating the vehicle's movement.

[0027] As used in this application, the term “wearable audio device” is intended to mean a device that fits around, on, inside, or near the ear (including open-ear audio devices worn on the user’s head or shoulder) and radiates acoustic energy into or toward the ear, in addition to its ordinary meaning or the meaning known to those skilled in the art. Wearable audio devices may also be referred to as headphones, earphones, earpieces, headsets, earbuds, or sports headphones, and may be wired or wireless. A wearable audio device includes an acoustic driver that converts audio signals into acoustic energy. This acoustic driver may be housed in an earcup. While some of the following figures and descriptions show a single wearable audio device having a pair of earcups (each containing an acoustic driver), it should be understood that a wearable audio device may also be a single standalone unit having only one earcup. Each earcup of a wearable audio device can be mechanically connected to another earcup or headphone, for example, by a headband and / or by leads that transmit audio signals to an acoustic transducer within the earcup or headphone. A wearable audio device may include components for wirelessly receiving audio signals. A wearable audio device may include components for an active noise reduction (ANR) system. A wearable audio device may also include other functions, such as a microphone, so that the device can function as a headset. Figure 1 shows examples of in-ear headphones, eyeglasses, and over-ear headsets, but in other examples, a wearable audio device can be an on-ear, around-ear, behind-ear, or near-ear headset.In some examples, wearable audio devices can be open-ear devices that include acoustic drivers that radiate acoustic energy towards the ear while leaving the ear open to the ear environment and surroundings.

[0028] As used herein, the term “connected isochronous stream” is intended to include, in addition to its ordinary meaning or the meaning known to those skilled in the art, an isochronous data stream that utilizes a pre-established point-to-point communication link via LE audio between a source device (also known as a central device or master device) and one or more audio devices (also known as peripheral devices or slave devices). In other words, a connected isochronous stream can provide an isochronous audio stream that utilizes at least one established trusted communication channel and / or at least one verified communication channel between the source device and each of any audio devices.

[0029] As used herein, the term “broadcast isochronous stream” is intended to include its ordinary meaning or the meaning known to those skilled in the art, as well as to refer to an isochronous data stream between a source device transmitting data and an audio device receiving data, without the need to establish a pre-established communication link and without the need to transmit or receive acknowledgments or denials.

[0030] The following explanation should be read with reference to Figures 1 to 14. Figure 1 is a schematic diagram of the components of System 10 according to the present disclosure. In the non-limiting example of Figure 1, System 10 includes at least one first device 100 and a second device 200. As shown in Figure 1, at least one first device 100 may be embodied as a wearable audio device, while the second device 200 may be embodied as a central device. However, as will be demonstrated in subsequent examples, the first device 100 may instead be embodied as a central device, while the second device 200 may be embodied as a wearable audio device. Additionally, in some examples illustrated in Figure 1, System 10 includes a plurality of first devices 100A to 100C (collectively referred to as “First Device 100” or “Pl. Pl. First Device 100”). The second device 200 is intended to be a device capable of establishing a wireless connection, e.g., a wireless connection 138 (described later), with at least one first device 100. While illustrated as a smartphone, the second device 200 can be selected from at least one of the following: a tablet, a smart hub, a media hub, a stereo hub, a soundbar, a headphone case, or any device capable of transmitting or broadcasting wireless data (described later) to at least one wearable device 100. Furthermore, while illustrated as a pair of true wireless earbuds 100A, a glasses form factor device 100B, and an over-ear form factor headset, the first device 100 can be selected from any device capable of transmitting data to and / or receiving wireless data from the second device 200. In some examples, each first device 100 is intended to be a device capable of rendering audible acoustic energy, e.g., a device capable of rendering audio data, based on wireless data received from the second device 200.

[0031] Each device in System 10, i.e., the first device 100 and the second device 200, can establish one or more wireless connections 138A-138C (collectively referred to as “wireless connections 138”) between the second device 200 and each first device 100 using their respective communication modules and / or transceivers. Each wireless connection 138 can be used to transmit and / or receive wireless data via one or more wireless data streams 140A-140C (collectively referred to as “wireless data streams 140”). In some examples, these wireless connections 138 include establishing one or more data streams between the second device 200 and each first device 100, where each data stream is an isochronous data stream, e.g., a connected isochronous stream using the LE audio standard. In other examples, the wireless connections 138 include a broadcast isochronous stream, i.e., the second device 200 broadcasts data in one or more isochronous data streams that are received by one or more wireless audio devices 100. For example, a second device 200 may be configured to generate, broadcast, or, in some cases, wirelessly transmit a wireless data stream 140 that is received by each first device 100. Alternatively, the first device 100 may also transmit the wireless data stream 140 via a broadcast isochronous stream. The streams established by each wireless connection 138 may use various wireless data protocols, standards, or transmission methods, such as the Bluetooth Low Energy Protocol or LE Audio standards.

[0032] Figure 2 illustrates a modified form of the system 10 in Figure 1. In the non-limiting example of Figure 2, the first device 100 is embodied as a central device, while the second device 200 is embodied as a wearable audio device. In the example of Figure 2, the first device 100 may be an in-vehicle computing system incorporating embodiments such as a navigation subsystem and an entertainment subsystem. The second device 200 is embodied as a pair of wireless earphones that can be worn by the driver of the vehicle. A wireless connection 138 enables the in-vehicle computing system to transmit a wireless data stream 140 to the wireless earphones.

[0033] Figure 3 illustrates an example of multiplexing and demultiplexing of data transmitted by a wireless data stream 140, such as a Bluetooth isochronous stream. In this example, multiple applications of a first device 100, such as a wearable audio device, capture data. These applications may be associated with different sensors, such as a microphone and a motion sensor. A multiplexer combines portions of the data collected by each application into a single packet. The packet is transmitted via a transport 140, such as an isochronous stream, and received by a second device 200, such as a PC or smartphone. The PC or smartphone reconstructs the data captured by the first device 100 and provides the data to the appropriate application on the second device 200. In some examples, the multiplexing-demultiplexing scheme can be bidirectional, allowing the second device 200 to transmit data packets containing multiple types of data to be demultiplexed by the first device 100.

[0034] The purpose of this disclosure is to create a scheme for isochronous channels, such as a Connected Isochronous Stream (CIS) channel, in which each packet includes one or more channel identifiers and length headers, so that the isochronous channel carries packets containing data from multiple sources, such as data collected by microphones and motion sensors. An Isochronous Adaptive Layer (ISOAL) may be used to segment the packets.

[0035] Figure 4 illustrates a data packet 106 created by multiplexing using a basic CIS multiplexing header. The illustrated packet includes a CIS header 142, isochronic adaptive layer (ISOAL) frame data 144, a first CIS multiplexing header 152, a first information payload 108, a second CIS multiplexing header 154, and a second information payload 110. In one example, the first CIS multiplexing header 152 and the first information payload 108 may correspond to audio data generated by an audio source 102. In some examples, the audio source may be a microphone configured to capture external audio. In other examples, the audio source 102 may be a hardware or software interface configured to receive audio information, such as compressed media files from an entertainment or navigation subsystem, from another aspect of the first device 100. The second CIS multiplexing header 154 and the second information payload 110 may correspond to motion data captured by a non-audio sensor 104, such as a motion sensor. In some examples, the motion sensor may be an inertial measurement unit (IMU). The first CIS multiplexing header 152 includes the first service data unit (SDU) length 112 and the first channel identification data 116, while the second CIS multiplexing header 154 includes the second SDU length 114 and the first channel identification data 118. The service data unit (SDU) lengths 112 and 114 indicate the lengths of the corresponding payloads 108 and 110, while the channel identification data 116 and 118 indicate the application or data type corresponding to the payload. In the non-restrictive example in Figure 4, the ISOAL frame data 144 is 24 bits, the SDU length data 112 and 114 are 8 to 16 bits, and the channel identification data 116 and 118 are 8 bits.

[0036] However, when transmitting payloads containing data from different sources (microphone, motion sensor, etc.) in the same packet, a timing offset is needed to indicate when the data from each payload was captured. Due to different devices with varying processing speeds or refresh rates, or different sensors on the same device, the data from each payload 108, 110 may be captured at different times. By incorporating a timing offset corresponding to each payload, the receiver can map all the data back to their original time domain based on the reference clock of the isochronous channel. Examples of timing offsets 120, 122 are illustrated in Figure 5 as components of the CIS multiplexing headers 152, 154 of a super SDU (SSDU). In Figure 5, timing offsets 120, 122 represent when payloads 108, 110 were captured relative to an anchor point. The anchor point can correspond to the timing of a data packet received by the first device 100 from a second device 200 in an isochronous event, or any other relevant point in time.

[0037] Figure 6 illustrates an existing ISOAL architecture for creating CIS packets from multiple types of application data, while Figure 7 illustrates a new layer in this architecture for segmentation and reassembly, retransmission and flow control, and encapsulation. Specifically, this new layer enables more efficient retransmission of payloads based on the lifespan of data within the payload, as different types of data may have different lifespans. For example, audio data may end after 10 milliseconds (in low-latency gaming applications, for example), while sensor data (such as motion sensor data) may end every 15 milliseconds. Therefore, in this example, a data channel may consist of a lifespan (or flash timeout) of 5 milliseconds, repeating audio data twice and sensor data three times before refreshing. Furthermore, if audio data is acknowledged as received at the 8-millisecond mark, but sensor data is not acknowledged as received, the audio data is removed but not replaced with new data until the 10-millisecond mark is reached. During this period between 8 and 10 milliseconds, the isochronous channel conserves energy by continuing to transmit packets containing only sensor data (without audio data) and retransmitting only the data that has not yet been received.

[0038] Figures 8 and 9 are flowcharts illustrating some of the scenarios considered with reference to Figures 6 and 7. In the embodiments of Figures 8 and 9, the first device 100 may be a wearable audio device (such as an audio headset), and the second device may be a central device (such as a smartphone). In other embodiments of Figures 8 and 9, the first device 100 may be a central device (such as an in-vehicle computing system), and the second device may be a wearable audio device (such as a pair of wireless earphones).

[0039] In Figure 8, the first device 100 wirelessly transmits a first data packet 106 (such as the data packet 106 shown in Figure 5). The data packet 106 includes an audio data set 108 and a sensor data set 110. The sensor data set 110 may correspond to a motion sensor or other non-audio sensor. The audio data 108 has a first duration 124 of 10 milliseconds, and the sensor data 110 has a second duration 126 of 15 milliseconds. The first data packet 106 is received by the second device 200. In response, the second device 200 transmits an audio data acknowledgment 128 to the first device 100. Since the second device 200 has already acknowledged the reception of the audio data set 108, the first device 100 then transmits a second data packet 130 containing only the sensor data 110.

[0040] In Figure 9, the first device 100 again wirelessly transmits a first data packet 106 containing an audio data set 108 and a sensor data set 110. Then, the first device 100 generates a second data packet 130 after the first lifetime 124 has ended but before the second lifetime 126 has ended. Thus, the second data packet 130 contains a second audio data set 132 and the first sensor data set 110. Then, the first device 100 generates a third data packet 136 after both the first and second lifetimes 124 and 126 have ended. Thus, the third data packet 136 contains a second audio data set 132 and a second sensor data set 134. As time progresses, new audio and sensor data are recycled into the transmitted data packets according to the data lifetimes 124 and 126.

[0041] Figure 10 schematically illustrates one of the first devices 100 shown earlier in Figures 1 and 2. The first device 100 may be a wearable audio device as shown in Figure 1, or the first device may be a central device as shown in Figure 2. As shown in the non-limiting example of Figure 1, the first device 100 may be embodied in the form factor of in-ear headphones, eyeglasses, or an over-ear headset. As shown in the non-limiting example of Figure 2, the first device 100 may be embodied as an in-vehicle computing system. The first device 100 includes an audio source 102, a sensor 104, a processor 125, memory 175, and a transceiver 185. The audio source 102 may be embodied as a microphone for capturing audio, or as a hardware or software interface for receiving audio information. Memory 175 is configured to store a first data packet 106, a second data packet 130, a first audio dataset 108, a second audio dataset 132, a first sensor dataset 110, a second sensor dataset 134, audio payload length 112, sensor payload length 114, first channel identification data 116, second channel identification data 118, audio time offset data 120, sensor time offset data 122, first data lifetime 124, second data lifetime 126, data acknowledgment 128, and a third data packet 136. Processor 125 is configured to run one or more applications, such as app 1 111, app 2 113 to app N 1NN, as shown in Figure 3. In a non-limiting example, processor 125 may be configured to multiplex the first audio dataset 108 and the first sensor dataset 110 to produce the first data packet 106.

[0042] Figure 11 schematically illustrates the second device 200 shown earlier in Figures 1 and 2. As shown in the non-limiting example of Figure 1, the second device 200 may be a smartphone. As shown in the non-limiting example of Figure 2, the second device 200 may be a pair of wireless earphones. The second device 200 includes a processor 225, memory 275, and transceiver 285. Memory 275 is configured to store a first data packet 106, a second data packet 130, a first audio dataset 108, a second audio dataset 132, a first sensor dataset 110, a second sensor dataset 134, audio payload length 112, sensor payload length 114, first channel identification data 116, second channel identification data 118, audio time offset data 120, sensor time offset data 122, first data lifetime 124, second data lifetime 126, data acknowledgment 128, and a third data packet 136. Processor 225 is configured to run one or more applications, such as app 1 211, app 2 213, and app Y 2YY, as shown in Figure 3. In a non-limiting example, processor 225 may be configured to demultiplex the first data packet 106 to reconstruct the first audio dataset 108 and the first sensor dataset 110 for further processing.

[0043] Figures 12 to 14 are flowcharts of a method 900 for transmitting data according to various embodiments of the present invention. Referring to Figures 1 to 14, the method 900 includes, in step 902, generating an audio dataset 108 via an audio source 102 of a first device 100.

[0044] Method 900 further includes, in step 904, acquiring a sensor dataset 110 via the sensor 104 of the first device 100.

[0045] Method 900 further includes generating a data packet 106 via the processor 125 of the first device 100 in step 906. The data packet 106 includes an audio data set 108 and a sensor data set 110. In one example, the data packet 106 may further include audio payload length data 112 and / or sensor payload length data 114. In another example, the data packet 106 may further include audio channel identification data 116 and / or sensor channel identification data 118. In yet another example, the data packet 106 may further include audio time offset data 120 and / or sensor time offset data 122. In yet another example, the audio data set 108 has a first duration 124, and the sensor data set 110 has a second duration 126 which is longer than the first duration 124.

[0046] Method 900 further includes, in step 908, transmitting a data packet 106 to a second device 200 via the transceiver 185 of the first device 100. For example, the data packet 106 may be transmitted via a Bluetooth Connected isochronous stream or a Bluetooth Broadcast isochronous stream.

[0047] Method 900 further includes, in step 910, receiving a data packet 106 via the transceiver 285 of the second device 200.

[0048] Method 900 further includes, in step 912, reconstructing the audio dataset 108 and the sensor dataset 110 by demultiplexing the data packet 106 via the processor 225 of the second device 200.

[0049] For example, method 900 may further include, in optional steps 914, 916, and 918, (1) receiving an audio data acknowledgment 128 via the transceiver 185 of the first device 100 before the first lifetime 124 ends; (2) generating a second data packet 130 containing a sensor dataset 110 via the processor 125 of the first device 100; and (3) transmitting the second data packet 130 to the second device 200 via the transceiver 185 of the first device 100.

[0050] For example, method 900 may further include, in optional steps 920, 922, and 924, (1) generating a second audio dataset 132 via audio source 102 after audio dataset 108 has finished, (2) generating a second data packet 130 including the second audio dataset 132 and sensor dataset 110 via processor 125 of first device 100, and (3) transmitting the second data packet 130 to second device 200 via transceiver 185 of first device 100. In addition to this example, method 900 may further include, in optional steps 926, 928, and 930, (1) acquiring a second sensor dataset 134 via sensor 926 of the first device 100 after the sensor dataset 110 has been completed; (2) generating a third data packet 136 via processor 125 of the first device 100, which includes a second audio dataset 132 and a second sensor dataset 134; and (3) transmitting the third data packet 136 to the second device 200 via transceiver 185 of the first device 100.

[0051] All definitions defined and used herein should be understood to govern dictionary definitions, definitions in documents incorporated by reference, and / or the ordinary meanings of the defined terms.

[0052] As used herein and in the claims, the indefinite articles "a" and "an" should be understood to mean "at least one" unless otherwise explicitly indicated.

[0053] As used herein and in the claims, the phrase “and / or” should be understood to mean “either or both” of the elements thus combined, that is, elements that exist concomitantly in some cases and separately in others. Multiple elements listed in “and / or” should be interpreted in the same way, that is, “one or more” of the elements thus combined. Other elements may exist optionally, in addition to the elements specifically identified by the “and / or” clause, whether related to or unrelated to the specifically identified elements.

[0054] Where used herein and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items within a list, “or” or “and / or” should be interpreted as inclusive, that is, including not only at least one of the number of elements or list, but more than one, and optionally including items not in the additional list. “One of” or “exactly one of” or, where used in the claims, “consisting of” should be explicitly indicated elsewhere, mean including exactly one element of the number of elements or list. In general, where used herein, the term “or” should be interpreted only as indicating an exclusive choice (i.e., “one or the other, but not both”) when preceded by an exclusive term such as “either,” “one of,” “one of,” or “exactly one of.”

[0055] As used herein and in the claims, the phrase “at least one” with respect to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one of each and all elements specifically listed in the list of elements, nor excluding any combination of elements in the list of elements. This definition also allows elements to exist optionally other than those specifically identified in the list of elements to which the phrase “at least one” refers, whether related to or unrelated to the specifically identified elements.

[0056] Unless otherwise explicitly indicated, in any method claimed herein that includes more than one step or action, the order of the steps or actions of the method is not necessarily limited to the order in which the steps or actions of the method are enumerated.

[0057] In the claims and in the above specification, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” and “composed of” should be understood as unrestrictive, that is, including but not limiting. Only the transitional phrases “consisting of” and “consisting essentially of” are restrictive or semi-restrictive, respectively.

[0058] The above-described examples of the subject matter can be implemented in any of many ways. For example, some embodiments can be implemented using hardware, software, or a combination thereof. If at least a portion of any embodiment is implemented as software, the software code can run on any suitable processor or set of processors, whether it is provided on a single device or computer, or distributed across multiple devices / computers.

[0059] This disclosure can be implemented as a system, method, and / or computer program product at any conceivable level of technical detail. The computer program product may include a computer-readable storage medium (or more) having computer-readable program instructions thereon for causing a processor to execute aspects of this disclosure.

[0060] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices on which instructions are recorded, as well as any preferred combination thereof. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal itself, such as a freely propagating electromagnetic wave like a radio wave, an electromagnetic wave propagating through a transmission medium like a waveguide (for example, a light pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device, via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network, transfers these computer-readable program instructions, and stores them in a computer-readable storage medium within the respective computing / processing device.

[0062] The computer-readable program instructions for performing the operations of the Disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, procedural programming languages ​​such as the C programming language, or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or fully on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet Service Provider). In some examples, electronic circuits, including, for instance, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions in order to perform aspects of the present disclosure.

[0063] Aspects of the present disclosure are described herein with reference to illustrative flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products described herein. It will be understood that each block in the illustrative flowcharts and / or block diagrams, and combinations of blocks in the illustrative flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0064] Computer-readable program instructions may be provided to a processor of a dedicated-purpose computer or other programmable data processing device to manufacture a machine, thereby creating means for performing functions / operations specified in one or more blocks of a flowchart and / or block diagram, through which the instructions executed via the processor of the computer or other programmable data processing device. Furthermore, these computer-readable program instructions may be stored in a computer-readable storage medium that can be instructed to function in a particular way to a computer, programmable data processing device, and / or other device, thereby including a computer-readable storage medium in which the instructions are stored in a manufactured article having instructions that perform a manner of function / operation specified in a flowchart and / or block diagram or block.

[0065] Computer-readable program instructions can also be loaded into a computer, another programmable data processing device, or another device to cause a series of operational steps on the computer, another programmable device, or another device to generate a computer realization process in which instructions executed on the computer, another programmable device, or another device perform a function / action specified in one or more blocks of a flowchart and / or block diagram.

[0066] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of assumed implementations of the systems, methods, and computer program products in the various examples of this disclosure. In this regard, each block in the flowchart or block diagram may correspond to a module, segment, or portion of an instruction, containing one or more executable instructions for performing a specified logical function. In some alternative implementations, the functions described in the blocks may occur in the order shown in the figures. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or, in some cases, the blocks may be executed in reverse order, depending on the function involved. Furthermore, it should be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented in a dedicated hardware-based system that performs a specific function or operates or executes a combination of dedicated hardware and computer instructions.

[0067] Other implementations are within the scope of the following claims, as well as any other claims to which the applicant may have rights.

[0068] While various examples have been described and illustrated in this specification, those skilled in the art will readily conceive of various other means and / or structures to implement functions and / or results and / or obtain one or more advantages described herein, and each of such modifications and / or variations will be considered within the scope of the examples described herein. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials and configurations described herein are illustrative, and furthermore, that actual parameters, dimensions, materials and / or configurations will depend on the specific application or the application in which the teachings of the present invention are used. Those skilled in the art will be able to recognize or confirm many equivalents to the specific examples described herein simply by performing routine experiments. Therefore, it should be understood that the examples described herein are presented for illustrative purposes only, and that examples can be implemented in ways other than those explicitly described and claimed within the scope of the appended claims and their equivalents. The examples in this disclosure cover each individual feature, system, article, material, kit and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the inventions of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0069] 10 Systems 100 First device 100A~C First device 102 Audio Sources 104 Sensors 106 data packets 108 payloads 110 payloads 112 Audio payload length data 114 Sensor payload length data 116 First channel identification data 118 Second channel identification data 120 Timing Offset 122 Timing Offset 124 First Lifetime 125 processors 126 Second Life Term 128 Audio data acknowledgment 130 Second data packet 132 Second audio dataset 134 Second Sensor Dataset 136 Third data packet 138 Wireless connection 138A~138C Wireless Connection 140 Wireless Data Streams 140A~140C Wireless Data Stream 142 CIS Header 144 frame data 152 First CIS Multiplexing Header 154 Second CIS Multiplexing Header 175 memory 185 Transceiver 200 Second device 225 processors 275 memory 285 Transceiver 900 methods 902 Steps 904 Steps 906 steps 908 steps 910 steps 912 steps 914 steps 916 steps 920 steps 922 steps 924 steps 926 steps 928 steps

Claims

1. The first device, An audio source configured to generate audio data, A sensor configured to acquire sensor data, It is a processor, A data packet is generated, which includes an audio dataset generated by the audio source and a sensor dataset acquired by the sensor. A second device, configured to reconstruct the audio dataset and the sensor dataset by demultiplexing the data packets, comprises a processor configured to transmit the data packets to the second device, The audio dataset has a first duration, the sensor dataset has a second duration longer than the first duration, and the processor Before the first duration described above ends, an audio data acknowledgment is received. A second data packet containing the aforementioned sensor dataset is generated. A first device further configured to transmit the second data packet to the second device.

2. The first device according to claim 1, wherein the first device is a wearable audio device and the second device is a central device.

3. The first device according to claim 1, wherein the first device is a central device and the second device is a wearable audio device.

4. The first device according to claim 1, wherein the data packet further includes audio payload length data and / or sensor payload length data.

5. The first device according to claim 1, wherein the data packet further includes audio channel identification data and / or sensor channel identification data.

6. The first device according to claim 1, wherein the data packet further includes audio time offset data and / or sensor time offset data.

7. The first device according to claim 1, wherein the sensor is an inertial measuring unit (IMU) and the sensor data is motion data.

8. The first device according to claim 1, wherein the data packets are transmitted via a Bluetooth connected isochronous stream or a Bluetooth broadcast isochronous stream.

9. The audio dataset has a first duration, the sensor dataset has a second duration longer than the first duration, and the processor After the aforementioned audio dataset is completed, a second audio dataset is generated via the aforementioned audio source. A second data packet, including the second audio dataset and the sensor dataset, is generated via the processor of the first device. The first device according to claim 1, further configured to transmit the second data packet to the second device.

10. The aforementioned processor, After the aforementioned sensor data set is completed, the second sensor data set is acquired via the sensor of the first device. A third data packet is generated, which includes the second audio dataset and the second sensor dataset. The first device according to claim 9, further configured to transmit the third data packet to the second device.

11. A method for transmitting data, To generate an audio dataset via the audio source of the first device, The sensor data set is acquired via the sensor of the first device, The first device generates a data packet via its processor, the data packet comprising the audio dataset and the sensor dataset. The data packet is transmitted to the second device via the transceiver of the first device, The data packet is received via the transceiver of the second device, This includes reconstructing the audio dataset and the sensor dataset by demultiplexing the data packets via the processor of the second device, The audio dataset has a first duration, and the sensor dataset has a second duration that is longer than the first duration. Receiving an audio data acknowledgment via the transceiver of the first device before the first lifetime ends, The first device generates a second data packet containing the sensor dataset via the processor, A method further comprising transmitting the second data packet to the second device via the transceiver of the first device.

12. The method according to claim 11, wherein the data packet further includes audio payload length data and / or sensor payload length data.

13. The method according to claim 11, wherein the data packet includes audio channel identification data and / or sensor channel identification data.

14. The method according to claim 11, wherein the data packet further includes audio time offset data and / or sensor time offset data.

15. The method according to claim 11, wherein the data packets are transmitted via a Bluetooth connected isochronous stream or a Bluetooth broadcast isochronous stream.

16. After the aforementioned audio dataset is completed, a second audio dataset is generated via the aforementioned audio source. The first device generates a second data packet including the second audio dataset and the sensor dataset via the processor of the first device, The method according to claim 11, further comprising transmitting the second data packet to the second device via the transceiver of the first device.

17. After the aforementioned sensor data set is completed, a second sensor data set is acquired via the sensor of the first device. A third data packet, including the second audio dataset and the second sensor dataset, is generated via the processor of the first device. The method according to claim 16, further comprising transmitting the third data packet to the second device via the transceiver of the first device.

Images