Multiplexing application channels on synchronization streams

Abstract

A first device is provided. The first device includes an audio source, a sensor, and a processor. The audio source generates audio data, and the sensor captures sensor data. The processor generates a data packet including an audio data set generated by the audio source and a sensor data set captured by the sensor. In some examples, the data packet may also include audio payload length data and / or sensor payload length data, audio channel identification data and / or sensor channel identification data and / or audio time offset data and / or sensor time offset data. The audio data set may have a first lifetime, and the sensor data set has a second lifetime that is longer than the first lifetime. The processor transmits the data packet to a second device configured to reconstruct the audio data set and the sensor data set by demultiplexing the data packet.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 366,041, filed on June 8, 2022, entitled “Multiplexing Application Channels on Isochronous Streams,” which is incorporated herein by reference in its entirety. Background Art

[0003] Immersive audio presented in virtual reality or augmented reality applications typically requires data from multiple modes to be captured by wearable audio devices. This captured data must be wirelessly transmitted to a central device (such as a smartphone) for further processing to provide an immersive audio experience to the user. Summary of the invention

[0004] The present disclosure generally relates to sending two or more types of data on a synchronous stream via a multiplexing scheme. These types of data may include various audio data (such as data captured by a microphone or used by an acoustic transducer to generate audio) and / or non-audio data (such as data collected by different types of non-audio sensors). The multiplexing scheme can be combined with a time offset value to allow time-accurate demultiplexing and reconstruction of the data sent through the synchronous stream. The multiplexing scheme can also incorporate data about the payload length of the data packet and channel identification information about the source of the captured data. Depending on the application, the synchronous stream can be a connection synchronous stream (CIS) or a broadcast synchronous stream (BIS). Sensor data may include a large number of different data types, such as, for example, motion data captured by an inertial measurement unit (IMU).

[0005] Generally speaking, in one aspect, a first device is provided. The first device includes an audio source. The audio source is configured to generate audio data.

[0006] The first device also includes a sensor. The sensor is configured to capture sensor data. According to an example, the sensor can be an inertial measurement unit (IMU). Further for this example, the sensor data can be motion data.

[0007] The first device also includes a processor. The processor is configured to generate a data packet. The data packet includes an audio data set generated by an audio source and a sensor data set captured by a sensor. According to an example, the data packet may also include audio payload length data and / or sensor payload length data. According to a further example, the data packet may also include audio channel identification data and / or sensor channel identification data. According to a further example, the data packet also includes audio time offset data and / or sensor time offset data. According to a further example, the audio data set may have a first lifetime, and the sensor data set may have a second lifetime longer than the first lifetime.

[0008] The processor is also configured to send the data packet to the second device. The second device is configured to reconstruct the audio data set and the sensor data set by demultiplexing the data packet. According to an example, the data packet can be sent via a Bluetooth connection synchronization stream or a Bluetooth broadcast synchronization stream.

[0009] According to one example, the first device may be a wearable audio device and the second device is a central device. In an alternative example, the first device may be a central device and the second device is a wearable audio device.

[0010] According to one example, the processor is further configured to: (1) receive an audio data confirmation before the first lifetime expires; (2) generate a second data packet including the sensor data set; and (3) send the second data packet to the second device.

[0011] According to an example, the processor is further configured to: (1) generate a second audio data set via an audio source after the audio data set expires; (2) generate a second data packet including the second audio data set and the sensor data set via the processor of the first device; and (3) send the second data packet to the second device. Further with respect to this example, the processor may also be configured to: (1) capture a second sensor data set via a sensor of the first device after the sensor data set expires; (2) generate a third data packet including the second audio data set and the second sensor data set; and (3) send the third data packet to the second device.

[0012] In general, in another aspect, a method for transmitting data is provided. The method includes capturing an audio data set via an audio source of a first device.

[0013] The method also includes capturing a sensor data set via a sensor of the first device.

[0014] The method also includes generating a data packet via a processor of the first device. The data packet includes an audio data set and a sensor data set. According to one example, the data packet may also include audio payload length data and / or sensor payload length data. According to another example, the data packet may also include audio channel identification data and / or sensor channel identification data. According to a further example, the data packet may also include audio time offset data and / or sensor time offset data. According to a further example, the audio data set has a first lifespan, and the sensor data set has a second lifespan longer than the first lifespan.

[0015] The method further includes sending the data packet to the second device via the transceiver of the first device.According to an example, the data packet may be sent via a Bluetooth connection synchronization flow or a Bluetooth broadcast synchronization flow.

[0016] The method also includes receiving, via a transceiver of the second device, a data packet.

[0017] The method also includes reconstructing, via a processor of the second device, the audio data set and the sensor data set by demultiplexing the data packets.

[0018] According to one example, the method may also include: (1) receiving an audio data confirmation via a transceiver of the first device before the first lifetime expires; (2) generating a second data packet including a sensor data set via a processor of the first device; and (3) sending the second data packet to a central device via the transceiver of the first device.

[0019] According to an example, the method may also include: (1) generating a second audio data set via an audio source after the audio data set expires; (2) generating a second data packet including the second audio data set and the sensor data set via a processor of the first device; and (3) sending the second data packet to the second device via a transceiver of the first device. Further with respect to this example, the method may also include: (1) capturing a second sensor data set via a sensor of the first device after the sensor data set expires; (2) generating a third data packet including the second audio data set and the second sensor data set via a processor of the first device; and (3) sending the third data packet to the second device via the transceiver of the first device.

[0020] In various implementations, a processor or controller may be associated with one or more storage media (collectively referred to herein as "memory," e.g., volatile and non-volatile computer memory, such as ROM, RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, tapes, flash memory, OTP-ROMs, SSDs, HDDs, etc.). In some implementations, the storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform at least some of the functions discussed herein. The various storage media may be fixed within a processor or controller or may be removable so that one or more programs stored thereon may be loaded into a processor or controller to implement various aspects as discussed herein. The term "program" or "computer program" is used herein in a general sense to refer to any type of computer code (e.g., software or microcode) that can be used to program one or more processors or controllers.

[0021] It should be understood that all combinations of the foregoing concepts and the additional concepts discussed in more detail below (assuming such concepts are not mutually inconsistent) are considered to be part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the inventive subject matter disclosed herein. It should also be understood that terms explicitly adopted herein that may also appear in any disclosure incorporated by reference should be given the meaning that best fits the specific concepts disclosed herein.

[0022] Other features and advantages will be apparent from the detailed description and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In the drawings, like reference characters generally refer to the same parts throughout the different views. Furthermore, the drawings are not necessarily drawn to scale, emphasis instead generally being placed upon illustrating the principles of the various embodiments.

[0024] Figure 1 is a schematic diagram of a first implementation of a system for wireless communication according to an example.

[0025] Figure 2 is a schematic diagram of a second embodiment of a system for wireless communication according to an example.

[0026] Figure 3 is a flow chart illustrating multiplexing and demultiplexing using the Bluetooth protocol according to one example.

[0027] Figure 4 A data packet including a connection synchronization stream (CIS) multiplexing header according to one example is illustrated.

[0028] Figure 5 A data packet including a super service data unit (SSDU) with a CIS multiplexing header according to an example is illustrated.

[0029] Figure 6 An example Bluetooth synchronization adaptation layer (ISOAL) according to one example is schematically illustrated.

[0030] Figure 7 Another example Bluetooth ISOAL with additional layers according to an example is schematically illustrated.

[0031] Figure 8 is a flow chart illustrating wireless communication between a wearable audio device and a central device according to one example.

[0032] Fig. 9 is yet another flow chart illustrating wireless communication between a wearable audio device and a central device according to an example.

[0033] Fig.10 is a schematic diagram of a first device according to an example.

[0034] Fig.11 is a schematic diagram of a second device according to an example.

[0035] Fig.12 is a flow chart of a method for sending data according to one example.

[0036] Fig.13 is another flow chart of a method for transmitting data according to an example.

[0037] Fig.14 is another flow chart of a method for sending data according to an example. DETAILED DESCRIPTION

[0038] The present disclosure generally relates to sending two or more types of data on a synchronous stream via a multiplexing scheme. These types of data may include various audio data (such as data captured by a microphone or used by an acoustic transducer to generate audio) and / or non-audio data (such as data collected by various types of non-audio sensors). The multiplexing scheme can incorporate time offset values, thereby allowing time-accurate demultiplexing and reconstruction of the data sent over the synchronous stream. The multiplexing scheme can also incorporate data about the payload length of the data packet and channel identification information about the source of the captured data.

[0039] In one non-limiting example, a wearable audio headset is used with a personal computer (PC) for gaming purposes. The headset can be connected to the PC via a synchronous stream. The headset includes a microphone for capturing the user's voice as audio data, and a sensor (such as an inertial measurement unit (IMU)) for capturing the movement of the user's head as sensor data. The processor of the headset can multiplex the audio data with the sensor data so that the headset sends a data packet containing both audio data and sensor data to the PC via a synchronous stream. The data packet also includes time offset data indicating when each payload of audio data or sensor data is captured. The PC receives the data packet, demultiplexes the packet, and uses the time offset to reconstruct the received audio and sensor data with appropriate timing. In other examples, other types of audio or non-audio data can be multiplexed and demultiplexed.

[0040] In another non-limiting example, an onboard computing system of a vehicle is used with a pair of wireless earbuds worn by a driver of the vehicle. The wireless earbuds can be connected to the onboard computing system via a synchronous stream. The onboard computing system includes an audio source for generating audio data corresponding to a navigation subsystem, an entertainment subsystem, or any other onboard subsystem capable of generating audio. The onboard computing system also includes a sensor configured to capture sensor data related to the motion of the vehicle (such as speed or direction). The processor of the onboard computing system can multiplex the audio data with the sensor data so that the onboard computing system sends a data packet containing both audio and sensor data to the wireless earbuds via a synchronous stream. The data packet may also include time offset data indicating when each payload of audio data or sensor data is captured. The wireless earbuds receive the data packets, demultiplex the packets, and use the time offset to reconstruct the received audio and sensor data with appropriate timing. Therefore, the multiplexed audio and sensor data transmitted from the onboard computing system to the wireless earbuds can be used to create an immersive audio experience combined with vehicle motion.

[0041] The term "wearable audio device" as used in this application, in addition to its ordinary meaning or meaning known to those skilled in the art, is also intended to mean a device that fits around the ear, on the ear, in the ear, or near the ear (including open-ear audio devices worn on the user's head or shoulders) and a device that radiates sound energy into or toward the ear. Wearable audio devices are sometimes referred to as earphones, earphones, earphones, headphones, earbuds, or sports headphones, and can be wired or wireless. The wearable audio device includes an acoustic driver for converting an audio signal into sound energy. The acoustic driver can be housed in an earmuff. Although some of the following figures and descriptions may show a single wearable audio device with a pair of earmuffs (each including an acoustic driver), it should be understood that the wearable audio device can be a single independent unit with only one earmuff. Each earmuff of the wearable audio device can be mechanically connected to another earmuff or earphone, for example, by a headband and / or by a lead that conducts the audio signal to the acoustic driver in the earmuff or earphone. The wearable audio device may include a component for wirelessly receiving an audio signal. Wearable audio devices may include components of an active noise reduction (ANR) system. Wearable audio devices may also include other features, such as a microphone, so that the wearable device can be used as a headset. Figure 1 Examples of in-ear earphone form factors, eyeglass form factors, and circumaural headphones are shown, but in other examples, the wearable audio device may be an on-ear, around-the-ear, behind-the-ear, or near-the-ear headphone. In some examples, the wearable audio device may be an open-ear device that includes an acoustic driver to radiate sound energy toward the ear while leaving the ear open to its surroundings and environment.

[0042] The term "connection synchronization stream" as used herein, in addition to including its ordinary meaning or its meaning known to those skilled in the art, is also intended to refer to a synchronized data stream that utilizes a pre-established point-to-point communication link over LE audio between, for example, a source device (also referred to as a central device or master device) and an audio device or multiple audio devices (also referred to as peripheral devices or slave devices). In other words, the connection synchronization stream may provide a synchronized audio stream that utilizes at least one established reliable communication channel and / or at least one confirmed communication channel between the source device and any corresponding audio device.

[0043] The term "broadcast synchronous stream" as used herein, in addition to including its ordinary meaning or meaning known to those skilled in the art, is also intended to refer to a synchronous data stream that does not require a pre-established communication link between a source device sending data and an audio device receiving data and does not require the sending or receiving of confirmation or negative confirmation.

[0044] In view of Figures 1 to 14 Read the description below. Figure 1is a schematic diagram of components of a system 10 according to the present disclosure. Figure 1 In a non-limiting example, the system 10 includes at least a first device 100 and a second device 200. Figure 1 As shown, at least one first device 100 may be embodied as a wearable audio device, and the second device 200 may be embodied as a central device. However, as will be shown in subsequent examples, the first device 100 may alternatively be embodied as a central device, and the second device 200 may be embodied as a wearable audio device. Figure 1 In some examples illustrated in , the system 10 includes multiple first devices 100A to 100C (collectively referred to as "first device 100" or "multiple first devices 100"). The second device 200 is intended to be a device capable of establishing a wireless connection (e.g., wireless connection 138 (discussed below)) with at least one first device 100. Although illustrated as a smart phone, it should be understood that the second device 200 can be selected from at least one of a tablet computer, a smarthub, a media center, a stereo center, a sound bar, a headphone shell, or any device capable of sending or broadcasting wireless data (discussed below) to at least one wearable device 100. In addition, although illustrated as a pair of truly wireless earbuds 100A, a glasses form factor device 100B, and a circumaural form factor headset, it should be understood that the first device 100 can be selected from any device capable of sending data to the second device 200 and / or receiving wireless data from the second device. In some examples, each first device 100 is intended to be a device capable of reproducing audible sound energy (eg, reproducing audio data) based on wireless data received from the second device 200 .

[0045] Each device of the system 10, that is, each first device 100 and the second device 200, can use their respective communication modules and / or transceivers to establish one or more wireless connections 138A to 138C (collectively referred to as "wireless connections 138") between the second device 200 and each first device 100. Each wireless connection 138 can be used to send and / or receive wireless data via one or more wireless data streams 140A to 140C (collectively referred to as "wireless data streams 140"). In some examples, these wireless connections 138 include establishing one or more data streams between the second device 200 and each first device 100, wherein each data stream is a synchronous data stream, such as a connection synchronization stream using the LE audio standard. In other examples, the wireless connection 138 includes a broadcast synchronization stream, that is, wherein the second device 200 broadcasts data in one or more synchronous data streams received by one or more wireless audio devices 100. For example, the second device 200 may be configured to generate, broadcast, or otherwise wirelessly transmit a wireless data stream 140 received by each first device 100. Alternatively, the first device 100 can also transmit the wireless data stream 140 via a broadcast synchronization stream. The stream established on each wireless connection 138 can use various wireless data protocols, standards, or transmission methods, such as the Bluetooth low energy protocol or the LE audio standard.

[0046] Figure 2 Illustrated Figure 1 A variation of system 10. Figure 2 In a non-limiting example, the first device 100 is embodied as a central device, and the second device 200 is embodied as a wearable audio device. Figure 2 In the example of FIG. 1 , the first device 100 may be an onboard computing system of a vehicle, which incorporates aspects such as a navigation subsystem, an entertainment subsystem, etc. The second device 200 is embodied as a pair of wireless earbuds that may be worn by the driver of the vehicle. The wireless connection 138 enables the onboard computing system to send a wireless data stream 140 to the wireless earbuds.

[0047] Figure 3An example of multiplexing and demultiplexing data sent via a wireless data stream 140 such as a Bluetooth synchronization stream is illustrated. In this example, multiple applications of a first device 100 (such as a wearable audio device) capture data. These applications can be associated with different sensors (such as microphones, motion sensors, etc.). The multiplexer combines the portions of the data collected by each application in a single packet. These packets are sent via a transmission 140 such as a synchronization stream and are received by a second device 200 such as a PC or a smart phone. The PC or smart phone reconstructs the data captured by the first device 100 and provides the data to the appropriate application of the second device 200. In some examples, the multiplexing-demultiplexing scheme can be bidirectional, allowing the second device 200 to also send data packets containing multiple types of data to be demultiplexed by the first device 100.

[0048] The goal of the present disclosure is to create a scheme for isochronous channels, such as connected isochronous stream (CIS) channels, where each packet contains one or more channel identifiers and a length header, so that the isochronous channel sends packets containing data from multiple sources (such as data collected by microphones and motion sensors). A synchronization adaptation layer (ISOAL) can be used to segment the packets.

[0049] Figure 4 The data packet 106 created by multiplexing using the basic CIS multiplexing header is illustrated. The illustrated packet includes a CIS header 142, a synchronization adaptation layer (ISOAL) frame data 144, a first CIS multiplexing header 152, a first information payload 108, a second CIS multiplexing header 154, and a second information payload 110. In one example, the first CIS multiplexing header 152 and the first information payload 108 may correspond to the audio data generated by the audio source 102. In some examples, the audio source may be a microphone configured to capture external audio. In other examples, the audio source 102 may be a hardware or software interface configured to receive audio information (such as a compressed media file from an entertainment or navigation subsystem) from other aspects of the first device 100. The second CIS multiplexing header 154 and the second information payload 110 may correspond to motion data captured by a non-audio sensor 104 (such as a motion sensor). In some examples, the motion sensor may be an inertial measurement unit (IMU). The first CIS multiplexing header 152 includes a first service data unit (SDU) length 112 and a first channel identification data 116, and the second CIS multiplexing header 154 includes a second SDU length 114 and a first channel identification data 118. The service data unit (SDU) lengths 112, 114 indicate the lengths of the corresponding payloads 108, 110, and the channel identification data 116, 118 indicate the application or data type corresponding to the payload. Figure 4In the non-limiting example of , the ISOAL frame data 144 is 24 bits, the SDU length data 112, 114 is 8 bits to 16 bits, and the channel identification data 116, 118 is 8 bits.

[0050] However, when sending payloads containing data from different sources (microphones, motion sensors, etc.) in the same packet, a timing offset is needed to indicate when the data from each payload was captured. Since different devices or different sensors of the same device have varying processing speeds or refresh rates, the data for each payload 108, 110 may have been captured at different times. Incorporating a timing offset corresponding to each payload allows the receiver to map all the data back into its original time domain based on the reference clock of the synchronization channel. Example timing offsets 120, 122 are shown in FIG. Figure 5 Components of the CIS multiplexing header 152, 154 illustrated as a super SDU (SSDU). Figure 5 In FIG. 1 , the timing offsets 120, 122 represent the time at which the payloads 108, 110 are captured relative to an anchor point. The anchor point may correspond to the timing at which the first device 100 receives a data packet from the second device 200 in a synchronization event, or any other relevant point in time.

[0051] Figure 6 The existing ISOAL architecture for creating CIS packets from various types of application data is illustrated. Figure 7 A new layer for segmentation and reorganization, retransmission and flow control and encapsulation in the architecture is illustrated. In particular, the new layer enables to resend the payload more efficiently based on the life of the data in the payload, because different types of data may have different lifespans. For example, audio data may expire after 10 milliseconds (such as in low-latency game applications), while sensor data (such as motion sensor data) may expire every 15 milliseconds. Therefore, in this example, the data channel can be configured with a lifespan (or refresh timeout) of 5 milliseconds, while repeating the audio data twice and repeating the sensor data three times before refreshing. In addition, if the audio data is confirmed to be received at the 8 millisecond mark, but the sensor data has not yet been confirmed to be received, the audio data will be deleted but not replaced by new data until the 10 millisecond mark has been reached. During this time period between 8 milliseconds and 10 milliseconds, the synchronization channel will continue to send packets with only sensor data (no audio data), thereby saving energy by only resending data that has not yet been received.

[0052] Figure 8 and Fig. 9 This is an example reference Figure 6 and Figure 7 Flowcharts of some of the scenarios discussed. Figure 8 and Fig. 9In an embodiment of the present invention, the first device 100 may be a wearable audio device (such as an audio headset) and the second device may be a central device (such as a smart phone). Figure 8 and Fig. 9 In other embodiments, the first device 100 may be a central device (such as an in-vehicle computing system) and the second device may be a wearable audio device (such as a pair of wireless earbuds).

[0053] exist Figure 8 In the example, the first device 100 wirelessly sends a first data packet 106 (such as Figure 5 100 ). The data packet 106 includes an audio data set 108 and a sensor data set 110. The sensor data set 110 may correspond to a motion sensor or other non-audio sensor. The audio data 108 has a first lifetime 124 of 10 milliseconds, while the sensor data 110 has a second lifetime 126 of 15 milliseconds. The first data packet 106 is received by the second device 200. In response, the second device 200 sends an audio data confirmation 128 to the first device 100. The first device 100 then sends a second data packet 130 having only the sensor data 110, because the second device 200 has confirmed receipt of the audio data set 108.

[0054] exist Fig. 9 , the first device 100 again wirelessly transmits the first data packet 106 including the audio data set 108 and the sensor data set 110. Then, the first device 100 generates a second data packet 130 after the first lifetime 124 expires but before the second lifetime 126 expires. Therefore, the second data packet 130 includes the second audio data set 132 and the first sensor data set 110. Then, the first device 100 generates a third data packet 136 after both the first lifetime 124 and the second lifetime 126 expire. Therefore, the third data packet 136 includes the second audio data set 132 and the second sensor data set 134. Over time, new audio and sensor data will be cycled into the transmitted data packets according to the lifetimes 124, 126 of the data.

[0055] Fig.10 This schematically illustrates the previous Figure 1 and Figure 2 The first device 100 may be one of the first devices 100 depicted in FIG. Figure 1 The wearable audio device shown, or the first device may be as shown Figure 2 The central device shown. Figure 1 As shown in the non-limiting examples of , the first device 100 can be embodied as an in-ear headphone form factor, a glasses form factor, or a circumaural headphone. Figure 2As shown in the non-limiting example of , the first device 100 can also be embodied as an in-vehicle computing system. The first device 100 includes an audio source 102, a sensor 104, a processor 125, a memory 175, and a transceiver 185. The audio source 102 can be embodied as a microphone for capturing audio or a hardware or software interface for receiving audio information. The memory 175 is configured to store a first data packet 106, a second data packet 130, a first audio data set 108, a second audio data set 132, a first sensor data set 110, a second sensor data set 134, an audio payload length 112, a sensor payload length 114, a first channel identification data 116, a second channel identification data 118, an audio time offset data 120, a sensor time offset data 122, a first data lifetime 124, a second data lifetime 126, a data confirmation 128, and a third data packet 136. The processor 125 is configured to execute one or more applications, such as Figure 3 App 1 111 , app 2 113 through app N 1NN are shown. In a non-limiting example, the processor 125 may be configured to multiplex the first audio data set 108 and the first sensor data set 110 to generate the first data packet 106 .

[0056] Fig.11 This schematically illustrates the previous Figure 1 and 2 The second device 200 is shown in the figure. Figure 1 As shown in the non-limiting example of , the second device 200 may be a smart phone. Figure 2 As shown in the non-limiting example of , the second device 200 can be a pair of wireless earbuds. The second device 200 includes a processor 225, a memory 275, and a transceiver 285. The memory 275 is configured to store the first data packet 106, the second data packet 130, the first audio data set 108, the second audio data set 132, the first sensor data set 110, the second sensor data set 134, the audio payload length 112, the sensor payload length 114, the first channel identification data 116, the second channel identification data 118, the audio time offset data 120, the sensor time offset data 122, the first data lifetime 124, the second data lifetime 126, the data confirmation 128, and the third data packet 136. The processor 225 is configured to execute one or more applications, such as Figure 3 App 1 211, app 2 213 through app Y2YY are shown. In a non-limiting example, the processor 225 may be configured to demultiplex the first data packet 106 to reconstruct the first audio data set 108 and the first sensor data set 110 for further processing.

[0057] Figure 12 to Figure 14is a flow chart of a method 900 for sending data according to various embodiments of the present invention. Figures 1 to 14 , the method 900 includes generating an audio data set 108 via an audio source 102 of a first device 100 in step 902 .

[0058] The method 900 also includes capturing the sensor data set 110 via the sensor 104 of the first device 100 in step 904 .

[0059] The method 900 also includes generating a data packet 106 via the processor 125 of the first device 100 in step 906. The data packet 106 includes an audio data set 108 and a sensor data set 110. According to one example, the data packet 106 may also include audio payload length data 112 and / or sensor payload length data 114. According to another example, the data packet 106 may also include audio channel identification data 116 and / or sensor channel identification data 118. According to another example, the data packet 106 may also include audio time offset data 120 and / or sensor time offset data 122. According to a further example, the audio data set 108 has a first lifetime 124, and the sensor data set 110 has a second lifetime 126 that is longer than the first lifetime 124.

[0060] The method 900 further includes: in step 908, sending the data packet 106 to the second device 200 via the transceiver 185 of the first device 100. According to an example, the data packet 106 may be sent via a Bluetooth connection synchronization flow or a Bluetooth broadcast synchronization flow.

[0061] The method 900 also includes receiving the data packet 106 via the transceiver 285 of the second device 200 in step 910 .

[0062] The method 900 further includes reconstructing the audio data set 108 and the sensor data set 110 by demultiplexing the data packets 106 via the processor 225 of the second device 200 in step 912 .

[0063] According to an example, method 900 may also include in optional steps 914, 916 and 918, respectively: (1) receiving an audio data confirmation 128 via the transceiver 185 of the first device 100 before the first lifetime 124 expires; (2) generating a second data packet 130 including the sensor data set 110 via the processor 125 of the first device 100; and (3) sending the second data packet 130 to the second device 200 via the transceiver 185 of the first device 100.

[0064] According to an example, the method 900 may further include, in optional steps 920, 922, and 924, respectively: (1) generating a second audio data set 132 via the audio source 102 after the audio data set 108 expires; (2) generating a second data packet 130 including the second audio data set 132 and the sensor data set 110 via the processor 125 of the first device 100; and (3) sending the second data packet 130 to the second device 200 via the transceiver 185 of the first device 100. Further with respect to this example, the method 900 may further include, in optional steps 926, 928, and 930, respectively: (1) capturing a second sensor data set 134 via the sensor 926 of the first device 100 after the sensor data set 110 expires; (2) generating a third data packet 136 including the second audio data set 132 and the second sensor data set 134 via the processor 125 of the first device 100; and (3) sending the third data packet 136 to the second device 200 via the transceiver 185 of the first device 100.

[0065] All definitions, as defined and used herein, should be understood to encompass dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0066] Unless expressly indicated to the contrary, the indefinite articles "a" and "an" as used herein in the specification and claims should be understood to mean "at least one".

[0067] As used herein in the specification and claims, the phrase "and / or" should be understood to mean "either one or both of the elements so combined," i.e., elements that are present in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be understood in the same manner, i.e., "one or more of the elements so combined." In addition to the elements explicitly identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to those elements explicitly identified.

[0068] As used herein in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be understood to be inclusive, i.e., including at least one element and more than one element in a plurality of elements or element lists, and optional additional unlisted items. Only the terms (such as "only one of ... in " or "exactly one of ... in " or "consisting of ... (when used in the claims)) that are clearly pointed out in the opposite direction will refer to including exactly one element in a plurality of elements or element lists. Generally speaking, when it is followed by an exclusive term (such as "any one", "one of ... in ", "only one of ... in " or "exactly one of ... in "), the term "or" as used herein should only be understood to indicate exclusive alternatives (that is, "one or another but not two").

[0069] As used herein in the specification and claims, the phrase "at least one" (with respect to a list of one or more elements) should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but does not necessarily include at least one element of each element specifically listed within the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those explicitly identified within the list of elements to which the phrase "at least one" refers, whether related or unrelated to those elements explicitly identified.

[0070] It should also be understood that in any method claimed herein that includes more than one step or action, the order of the method steps or actions is not necessarily limited to the order in which the method steps or actions are described unless explicitly stated to the contrary.

[0071] In the claims and the above description, all transitional phrases (such as "include", "comprising", "carrying", "having", "containing", "involving", "containing", "consisting of", etc.) should be understood as open-ended, i.e., meaning including but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" should be closed or semi-closed transitional phrases, respectively.

[0072] The above examples of the described subject matter can be implemented in any of a variety of ways. For example, some aspects can be implemented using hardware, software, or a combination thereof. When any aspect is implemented at least in part in software, the software code can be executed on any suitable processor or set of processors, whether provided in a single device or single computer or distributed between multiple devices / computers.

[0073] The present disclosure may be implemented as a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include one (or more) computer-readable storage media having computer-readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.

[0074] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. An incomplete list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be understood as a transient signal itself, such as a radio wave or other free-propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal sent by a wire.

[0075] Computer-readable program instructions as described herein can be downloaded to corresponding computing / processing devices from computer-readable storage media via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards computer-readable program instructions to be stored in the computer-readable storage medium in the corresponding computing / processing device.

[0076] The computer-readable program instructions for implementing the operations of the present disclosure may be any of source code or object code written in any combination of assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit system, or one or more programming languages ​​(including: object-oriented programming languages, such as Smalltalk, C++, etc.; and procedural programming languages, such as "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, and partially on a remote computer or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some examples, an electronic circuit system including, for example, a programmable logic circuit system, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to perform various aspects of the present disclosure by using state information of the computer-readable program instructions to personalize the electronic circuit system.

[0077] Various aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to examples of the present disclosure. It should be understood that each box in the flowchart illustration and / or block diagram and the combination of boxes in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0078] Computer-readable program instructions may be provided to a processor of a special-purpose computer or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct the computer, programmable data processing device, and / or other device to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or block diagram or box.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible specific implementation architecture, functions and operations of the systems, methods and computer program products according to various examples of the present disclosure. In this regard, each box in the flowchart or block diagram may represent an instruction module, an instruction fragment or an instruction portion, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the box can occur in a sequence not described in the accompanying drawings. For example, depending on the functions involved, the two boxes shown in succession can actually be executed substantially simultaneously, or the boxes can sometimes be executed in reverse order. It should also be noted that each box in the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or implements a combination of dedicated hardware and computer instructions.

[0081] Other implementations are within the scope of the following claims and other claims that the applicant may claim.

[0082] Although various examples have been described and illustrated herein, a person of ordinary skill in the art will readily envision a variety of other devices and / or structures for performing the functions described herein and / or obtaining one or more of the results and / or advantages described herein, and each of such variations and / or modifications is considered to be within the scope of the examples described herein. More generally, a person of ordinary skill in the art will readily understand that all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and that actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using the teachings of the present invention. A person of ordinary skill in the art will recognize or be able to determine many equivalents of the specific examples described herein using only routine experiments. Therefore, it should be understood that the above examples are presented by way of example only, and within the scope of the appended claims and their equivalents, examples may be practiced in a manner different from that specifically described and claimed. The examples disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits and / or methods is included within the scope of the present disclosure if such features, systems, articles, materials, kits and / or methods are not mutually inconsistent.

Claims

1. A first device, comprising: an audio source configured to generate audio data; a sensor configured to capture sensor data; and A processor, the processor being configured to: generating a data packet, wherein the data packet includes an audio data set generated by the audio source and a sensor data set captured by the sensor; as well as The data packets are sent to a second device, wherein the second device is configured to reconstruct the audio data set and the sensor data set by demultiplexing the data packets. 2 . The first device of claim 1 , wherein the first device is a wearable audio device, and wherein the second device is a central device.

3. The first device of claim 1, wherein the first device is a central device, and wherein the second device is a wearable audio device. 4 . The first device of claim 1 , wherein the data packet further comprises audio payload length data and / or sensor payload length data.

5. The first device of claim 1, wherein the data packet further comprises audio channel identification data and / or sensor channel identification data.

6. The first device of claim 1, wherein the data packet further comprises audio time offset data and / or sensor time offset data.

7. The first device of claim 1, wherein the sensor is an inertial measurement unit (IMU), and wherein the sensor data is motion data.

8. The first device of claim 1, wherein the data packet is sent via a Bluetooth connection synchronization stream or a Bluetooth broadcast synchronization stream.

9. The first device of claim 1 , wherein the audio data set has a first lifetime and the sensor data set has a second lifetime longer than the first lifetime, and wherein the processor is further configured to: receiving an audio data confirmation before expiration of said first lifetime; generating a second data packet including the sensor data set; and The second data packet is sent to the second device.

10. The first device of claim 1, wherein the audio data set has a first lifetime and the sensor data set has a second lifetime longer than the first lifetime, and wherein the processor is further configured to: generating, via the audio source, a second audio data set after expiration of the audio data set; generating, via the processor of the first device, a second data packet comprising the second audio data set and the sensor data set; as well as The second data packet is sent to the second device.

11. The first device according to claim 10, wherein the processor is further configured to: capturing, via the sensor of the first device, a second sensor data set after expiration of the sensor data set; generating a third data packet comprising the second audio data set and the second sensor data set; and The third data packet is sent to the second device.

12. A method for sending data, the method comprising: generating an audio data set via an audio source of a first device; capturing a sensor data set via a sensor of the first device; generating, via a processor of the first device, a data packet, wherein the data packet includes the audio data set and the sensor data set; sending the data packet to a second device via a transceiver of the first device; receiving the data packet via a transceiver of the second device; as well as The audio data set and the sensor data set are reconstructed by demultiplexing the data packets via a processor of the second device.

13. The method of claim 12, wherein the data packet further comprises audio payload length data and / or sensor payload length data.

14. The method of claim 12, wherein the data packets include audio channel identification data and / or sensor channel identification data.

15. The method of claim 12, wherein the data packets further include audio time offset data and / or sensor time offset data.

16. The method of claim 12, wherein the data packets are sent via a Bluetooth connection synchronization stream or a Bluetooth broadcast synchronization stream.

17. The method of claim 12, wherein the audio data set has a first lifetime and the sensor data set has a second lifetime that is longer than the first lifetime.

18. The method according to claim 17, further comprising: receiving, via the transceiver of the first device, an audio data acknowledgement prior to expiration of the first lifetime; generating, via the processor of the first device, a second data packet comprising the sensor data set; as well as The second data packet is sent to the second device via the transceiver of the first device.

19. The method according to claim 17, further comprising: generating, via the audio source, a second audio data set after expiration of the audio data set; generating, via the processor of the first device, a second data packet comprising the second audio data set and the sensor data set; as well as The second data packet is sent to the second device via the transceiver of the first device.

20. The method according to claim 19, further comprising: capturing, via the sensor of the first device, a second sensor data set after expiration of the sensor data set; generating, via the processor of the first device, a third data packet comprising the second audio data set and the second sensor data set; as well as The third data packet is sent to the second device via the transceiver of the first device.

Images