Spatial audio playback device and method

Abstract

A spatial audio playing device, in communication connection with a head-mounted device, comprising: a transceiving unit configured to communicate with the head-mounted device through wireless communication, and configured to receive motion information of the head-mounted device on a motion sensing signal, and receive information or transmission standard of the wireless communication from the head-mounted device; and a processing unit connected to the transceiving unit, configured to predict head information after a preset time according to the motion information, adjust spatial audio after the preset time according to the predicted head information to generate adjusted spatial audio, and transmit the adjusted spatial audio to the head-mounted device through the transceiving unit; and wherein the preset time changes according to the information or the transmission standard of the wireless communication.

Description

[Technical Field] This disclosure relates to spatial audio playback devices and methods. [Background Technology] Spatial audio is a sound playback technology widely used in playback devices. Playback devices can sense sound from all directions without requiring a specific multi-speaker setup. Therefore, spatial audio can provide users with a unique, immersive audio experience.

[0001] In existing technologies, achieving spatial audio effects requires the assistance of head-tracking technology from wearable devices. The goal is for the playback device to dynamically sense and simulate the surround sound effect of spatial audio as the user moves their head, thus achieving high real-time performance. Therefore, reducing the latency of spatial audio is a key factor in improving the user experience.

[0002] However, due to limitations in playback devices and encoding technologies, playback devices inevitably introduce a certain degree of latency during audio transmission, thus affecting the user's experience of spatial audio. For example, the delay from the end of a user's head movement to the change and convergence of sound often significantly reduces the quality of the user's spatial audio experience. Therefore, improving the user's spatial audio experience is a crucial issue. [Summary of the Invention] Some embodiments of this disclosure provide a spatial audio playback device communicatively connected to a head-mounted device, comprising: a transceiver unit, communicating with the head-mounted device via wireless communication, configured to receive motion information from the head-mounted device on motion sensing signals, and to receive information or transmission standards from the head-mounted device via the wireless communication; and a processing unit, connected to the transceiver unit, configured to predict head information after a preset time based on the motion information, adjust spatial audio based on the predicted head information after the preset time to generate adjusted spatial audio, and transmit the adjusted spatial audio to the head-mounted device via the transceiver unit; wherein the preset time changes according to the information or the transmission standards of the wireless communication.

[0003] Other embodiments of this disclosure provide a spatial audio playback device communicatively connected to a head-mounted device, comprising: a transceiver unit, communicating wirelessly with the head-mounted device and configured to receive motion information of the head-mounted device on a motion sensing signal; and a processing unit connected to the transceiver unit, configured to predict head information after a preset time based on the motion information, and adjust spatial audio based on the predicted head information after the preset time to generate adjusted spatial audio; wherein the preset time is calculated based on a signal delay time period of the spatial audio playback device and / or the head-mounted device.

[0004] Further embodiments of this disclosure provide a method for playing spatial audio, including: receiving motion information from a head-mounted device and wireless communication information or transmission standards; predicting head information after a preset time based on the motion information; adjusting the spatial audio after the preset time based on the predicted head information to generate adjusted spatial audio; and transmitting the adjusted spatial audio to the head-mounted device; wherein the preset time changes according to the wireless communication information or transmission standards. [Attached Image Description] Embodiments of this disclosure will now be described by way of example only, with reference to the accompanying drawings, in which: Figure 1 A schematic diagram illustrating a spatial audio playback device according to an embodiment of the present disclosure.

[0005] Figure 2 A schematic diagram illustrating the signal delay time period according to an embodiment of the present disclosure.

[0006] Figure 3 A flowchart illustrating the signal transmission and processing of a plurality of sub-processing units of a processing unit according to an embodiment of the present disclosure.

[0007] Figure 4 A flowchart illustrating a spatial audio playback method according to an embodiment of the present disclosure.

Detailed Implementation Methods

[0008] Figure 1 A schematic diagram of a spatial audio playback device 110 according to an embodiment of the present invention is shown. The spatial audio playback device 110 is communicatively connected to a head-mounted device 120, and the spatial audio playback device 110 includes a transceiver unit 111 and a processing unit 112.

[0009] The transceiver unit 111 may be a transceiver circuit that supports wired networks (such as fiber optic, Ethernet, or TV cable) or wireless networks (such as Wi-Fi, mobile networks, and Bluetooth). In one embodiment, the communication transceiver unit 111 includes (but is not limited to) components such as a digital-to-analog converter, an amplifier, an antenna, and a mixer, depending on their type, but the embodiment is not limited thereto. It is worth noting that even though the spatial audio playback device 110 primarily uses Bluetooth to communicate with the headset 120 in the following paragraphs, the embodiment is not limited thereto.

[0010] The processing unit 112 may include a central processing unit (CPU), a microprocessor (MCU), or a field-programmable gate array (FPGA), or may have a chip with data computation capabilities. It is worth noting that in the embodiments described in the following paragraphs, the processing unit 112 is further connected to at least one storage device (not shown in the figures). The storage device may be memory, such as read-only memory (ROM), erasable programmable read-only memory (EPROM) or other non-volatile memory, random access memory (RAM) or other volatile memory, hard disk drive, semiconductor memory, or other storage devices, used to store various programs and information as well as other data, but the embodiments are not limited thereto.

[0011] The head-mounted device 120 may be a set of headphones for user use, but the embodiments described are not limited thereto. The head-mounted device 120 can exchange data with the transceiver unit 111 and the processing unit 112 via a specific communication protocol. In some embodiments, the processing unit 112 is equipped with associated Bluetooth communication circuitry / modules, and the head-mounted device 120 can exchange data with the processing unit 112 via the Bluetooth Low Energy (BLE) protocol. In some embodiments using the BLE protocol for data exchange, the data packets transmitted by the head-mounted device 120 may include, for example, preambles, addresses, headers, payloads, and cyclic redundancy check (CRC) codes. The definition / content of each part can be found in the relevant BLE specifications, and will not be described in detail here.

[0012] In one embodiment, the transceiver unit 111 communicates with the head-mounted device 120 via wireless communication to receive motion information from the head-mounted device on motion sensing signals, and to receive wireless communication information or transmission standards from the head-mounted device 120.

[0013] The motion sensing signal is a sensing signal obtained using at least one sensor in the head-mounted device 120. The motion information of the head-mounted device on the motion sensing signal is configured to represent head movement-related information when the user wears the device. The wireless communication information or transmission standard configuration of the head-mounted device 120 is configured to enable the spatial audio playback device 110 to obtain Bluetooth communication-related content of the head-mounted device 120 during communication.

[0014] It is worth noting that the motion sensing signal in this embodiment of the invention is a continuous signal, while motion information is specific information at a specific time on the continuous signal of the motion sensing signal. That is, the motion sensing signal received by the transceiver unit may include a plurality of motion information. The motion information represents multiple motion information of the user at multiple specific times.

[0015] Processing unit 112 is connected to transceiver unit 111 and is configured to predict head information after a preset time based on motion information, and adjust spatial audio based on the head information after the preset time to generate adjusted spatial audio to transceiver unit 111. Furthermore, the adjusted spatial audio is transmitted from transceiver unit to head-mounted device, and the preset time changes according to the wireless communication information or transmission standard.

[0016] In one embodiment, the preset time varies depending on the information or transmission standard of the wireless communication. That is, even if the head-mounted device 120 uses various different information or transmission standards of wireless communication to transmit motion sensing signals, the processing unit 112 can calculate and set different preset times accordingly. Therefore, the processing unit 112 can reduce latency under different information or transmission standards of wireless communication.

[0017] The wireless communication information or transmission standard transmitted in the embodiments includes Bluetooth codec or Bluetooth profile. Bluetooth codec may include sub-band codec (SBC), advanced audio codec (AAC), lossless digital audio codec (LDAC), low-complexity communication codec (LC3), and low-energy audio (LEAudio), but the embodiments are not limited to these.

[0018] In addition, Bluetooth profiles may include advanced audio distribution profile (A2DP), clear voice capture (CVC), audio video remote control profile (AVRCP), intercom profile (ICP), human interface device profile (HID), headset profile (HSP), object push profile (OPP), etc., but the embodiments described are not limited to these.

[0019] It is worth noting that the spatial audio playback device 110 may include multiple Bluetooth codes or Bluetooth profiles, and can automatically switch between different Bluetooth codes or profiles. Users can also select a specific Bluetooth code or profile for the spatial audio playback device 110. Furthermore, different Bluetooth codes or profiles require their own encoding time when performing spatial information-related processing. This encoding time is highly correlated with the time required to process and transmit spatial audio.

[0020] In one embodiment, the preset time is ±10% of the signal delay time between the spatial audio playback device 110 and the head-mounted device 120. For example, if the spatial audio playback device 110 predicts that the signal delay time between the spatial audio playback device 110 and the head-mounted device 120 is 50 milliseconds, then the preset time range is ±5 milliseconds of 50 milliseconds, that is, between 45 milliseconds and 55 milliseconds.

[0021] Therefore, the preset time can be considered as the time error range after adjusting the signal delay period. For example, if the signal delay period between the spatial audio playback device 110 and the head-mounted device 120 is 50 milliseconds, the processing unit 112 predicts the head position after a preset time (45 milliseconds to 55 milliseconds) using motion sensing signals. The processing unit 112 obtains the user's head information after the preset time in order to adjust the spatial audio, generate adjusted spatial audio, and transmit it to the transceiver unit 111. Based on the above, the signal delay period can be effectively reduced by calculating and setting the preset time.

[0022] It is worth noting that the audio playback device 100 calculates and sets different preset times based on different Bluetooth codecs, and adjusts the signal delay time period of the spatial audio accordingly. Furthermore, the audio playback device 100 can adjust the same spatial audio to different signal delay time periods based on the encoding complexity of the Bluetooth codec. Generally speaking, the higher the encoding complexity of the Bluetooth codec, the shorter the predictable prediction time of the audio playback device 100, and the smaller the controllable signal delay time period range. Conversely, the lower the encoding complexity of the Bluetooth codec, the longer the predictable prediction time of the audio playback device 100, and the larger the controllable signal delay time period range.

[0023] The signal delay time period of the spatial audio playback device 110 and the head-mounted device 120 may include multiple times during which the spatial audio playback device 110 and the head-mounted device 120 process, transmit, and receive signals and spatial audio. The following paragraphs will specifically describe the multiple times or time differences configured to combine the signal delay time period.

[0024] Figure 2 A schematic diagram of a signal delay time period according to an embodiment of the present disclosure is shown. Figure 2 A schematic diagram of the signal delay time period in the diagram can be derived from... Figure 1 The spatial audio playback device 110 is implemented in the middle.

[0025] In one embodiment, the signal delay time period of the spatial audio playback device 110 is the time difference between the time it takes for the head-mounted device 120 to transmit motion information of the head-mounted device and the time it takes for the head-mounted device 120 to receive the adjusted spatial audio. (See reference...) Figure 2 The head-mounted device 120 transmits motion information at time t0 and receives adjusted spatial audio at time t5. Based on the above, the time difference T1 (not shown in the figure) can be represented by subtracting time t0 from time t5. That is, the time difference T1 is the signal delay period after the spatial audio playback device 110 processes or adjusts the spatial audio.

[0026] In one embodiment, the signal delay time period of the spatial audio playback device 110 is the time difference between the time it takes for the transceiver unit 111 to receive motion information from the head-mounted device and the time it takes for the transceiver unit 111 to transmit the adjusted spatial audio. (See reference...) Figure 2 The head-mounted device 120 receives motion information at time t1 and transmits adjusted spatial audio at time t4. Based on the above, the time difference T2 (not shown in the figure) can be represented by subtracting time t1 from time t4. The time difference T2 in the above paragraph is shorter than the time difference T1 and is included within the time difference T1. That is, the time difference T2 is part of the signal delay period after the spatial audio playback device 110 processes or adjusts the spatial audio.

[0027] In one embodiment, the signal delay time period of the spatial audio playback device 110 includes the signal transmission and processing time of the transceiver unit 111 and the processing unit 112. The signals transmitted and processed between the transceiver unit 111 and the processing unit 112 may include: motion information of the head-mounted device transmitted from the transceiver unit 111 to the processing unit 112, and adjusted spatial audio generated by the processing unit 112 and transmitted to the transceiver unit 111. (See reference...) Figure 2 At time t2, processing unit 112 receives motion information from the head-mounted device transmitted by transceiver unit 111, and after a series of processing steps, generates adjusted spatial audio at time t3 and transmits it to transceiver unit 111. Based on the above, time difference T3 (not shown in the figure) can be represented by subtracting time t2 from time t3. Time difference T3 is shorter than time differences T1 and T2 in the above paragraphs and is included within time differences T1 and T2. That is, time difference T3 is part of the signal delay period after spatial audio playback device 110 processes or adjusts spatial audio.

[0028] In one embodiment, the processing unit 112 includes a plurality of sub-processing units, and the signal delay time period includes the signal transmission and processing time of one or more sub-processing units. Specifically, the plurality of sub-processing units are interconnected, electrically connected, or communicatively connected to each other to collaboratively process the process of adjusting spatial audio into adjusted spatial audio. Similar to the paragraphs above, the signal transmission and processing time of the plurality of sub-processing units is a portion of the signal delay time period after the spatial audio playback device 110 processes or adjusts the spatial audio.

[0029] Figure 3 A schematic diagram illustrating signal transmission and processing of a plurality of sub-processing units of a processing unit according to an embodiment of the present disclosure is shown. The schematic diagram of signal transmission and processing can be understood through... Figure 1 The spatial audio playback device 110 is implemented by a processing unit 112. The following paragraphs will describe in detail the signal transmission and processing performed by the plurality of sub-processing units of the processing unit 112.

[0030] Please see Figure 3 The processing unit 112 comprises a plurality of sub-processing units, including: a Bluetooth driver unit, a human interface device (HID), a dynamic data sensor, a head information tracking unit, an audio effect conversion unit, a spatial audio output unit, etc., but this embodiment is not limited thereto. Furthermore, the HID includes a Bluetooth HID registration unit and an HID raw data sensor, but this embodiment is not limited thereto.

[0031] First, during the process of the spatial audio playback device 110 receiving and processing motion information from the head-mounted device 120, the processing unit 112 first triggers the Bluetooth driver unit of the spatial audio playback device 110 to initiate the processing program for the signal received via Bluetooth. Next, the Bluetooth driver unit transmits the motion information to the Bluetooth HID sensing registration unit in the HID, and the HID transmits the motion information to the motion data sensor via the HID raw data sensor. Then, the motion data sensor transmits the motion information to the head information tracking unit. The head information tracking unit is configured to analyze the motion information to obtain head information analysis results.

[0032] Next, the head information tracking unit transmits the head information analysis results to the audio effects conversion unit, which performs spatial audio conversion based on the analysis results. Specifically, the audio effects conversion unit calculates a preset time based on the head information analysis results and predicts the head information after the preset time. Then, the audio effects conversion unit adjusts the spatial audio to the adjusted spatial audio based on the head information after the preset time.

[0033] Furthermore, when the spatial audio playback device 110 transmits the adjusted spatial audio to the head-mounted device 120, the spatial audio output unit first obtains the adjusted spatial audio through the audio effects conversion unit, and then transmits the adjusted spatial audio to the Bluetooth driver unit of the spatial audio playback device 110. Finally, the Bluetooth driver unit transmits the adjusted spatial audio to the head-mounted device 120 through the transceiver unit 111.

[0034] In some embodiments, the motion information of the head-mounted device on the motion sensing signal of the spatial audio playback device 110 includes instantaneous head position information or instantaneous head posture information. Specifically, the head-mounted device 120 includes at least one sensor, which includes a positioning module. The sensor can obtain the user's head position information or posture information in space at the instant the user's head moves in space. The aforementioned position information or posture information can be absolute position / posture information located in space, or relative position / posture information relative to the entire space or other components, but this embodiment is not limited to this.

[0035] In other embodiments, the motion sensing signal of the spatial audio playback device 110 includes continuous head position information or continuous head posture information. Specifically, the head-mounted device 120 includes at least one sensor, which includes a positioning module. The sensor can acquire multiple position or posture information of the user's head in space during the time period in which the user's head moves in space, and record continuous information of multiple position or posture information. The aforementioned position or posture information can be absolute position / posture information located in space, or relative position / posture information relative to the entire space or other components, but this embodiment is not limited to this.

[0036] Figure 4 A flowchart of a spatial audio playback method 400 according to an embodiment of the present disclosure is shown. Figure 4 The spatial audio playback method in the middle can be made by Figure 1 This is implemented using a spatial audio playback device 110. In one embodiment, in step 410, the transceiver unit 111 of the spatial audio playback device 110 receives motion information from the motion sensing signal of the head-mounted device 120, as well as wireless communication information or transmission standards. In step 420, the processing unit 112 of the spatial audio playback device 110 predicts head information after a preset time based on the motion information. In step 430, the processing unit 112 of the spatial audio playback device 110 adjusts the spatial audio based on the head information after the preset time to generate adjusted spatial audio. In step 440, the processing unit 112 of the spatial audio playback device 110 transmits the adjusted spatial audio to the head-mounted device 120 through the transceiver unit 111.

[0037] In one embodiment, the adjusted spatial audio is transmitted from the transceiver unit to the head-mounted device, and the preset time varies according to the wireless communication information or the wireless communication transmission standard.

[0038] In one embodiment, in the spatial audio playback method 400, the preset time predicted by the processing unit 112 is ±10% of the signal delay time between the spatial audio playback device 110 and the head-mounted device 120. Based on this, the preset time can be regarded as the time error range after adjusting for the signal delay time.

[0039] In one embodiment, the transmission standard for the information or the wireless communication includes a Bluetooth codec or a Bluetooth profile. It is noteworthy that various Bluetooth codecs or Bluetooth profiles can automatically switch between different Bluetooth codecs or different Bluetooth profiles, or can be selected / switched by the user. Furthermore, different Bluetooth codecs or different Bluetooth profiles have their own encoding times when performing spatial information-related processing.

[0040] In one embodiment, the signal delay time period of the head-mounted device is the time difference between the time it takes to receive motion information from the head-mounted device and the time it takes to transmit the adjusted spatial audio in the spatial audio playback method. This time difference may include multiple times during which the spatial audio playback device and the head-mounted device process, transmit, and receive signals and spatial audio.

[0041] Based on the above, some embodiments of the present invention optimize the user's latency setting from the start to the end of movement by controlling spatial audio, making the user experience of spatial audio close to zero latency. Specifically, some embodiments of the present invention use motion information of the head-mounted device on the motion sensing signal transmitted by the user's head-mounted device to predict head information after a preset time, so as to effectively improve the user's listening experience by tracking changes in the user's head. In addition, some embodiments of the present invention can support multiple different wireless communication information or transmission standards to adjust the spatial audio latency time accordingly, wherein the preset time changes according to the wireless communication information or transmission standard. Through the above, some embodiments of the present invention can appropriately adjust the possible latency period of spatial audio according to different playback devices and the user's head posture at different times and locations, so as to effectively improve the real-time performance of spatial audio.

[0042] In the claims, the word "comprising" does not exclude other components or steps, and the indefinite articles "a" or "described" do not exclude a plurality. A processor or other unit may perform the functions of multiple items described in the claims. Furthermore, any reference marks in the claims should not be construed as limiting their scope. Additionally, specific and preferred aspects disclosed are set forth in the appended independent claims. Combinations of features in the dependent and / or independent claims may be appropriately combined, and not limited to those described in the claims.

[0043] All orientations and arrangements of the components shown herein are for illustrative purposes only. Furthermore, those skilled in the art will understand that in alternative embodiments, the functionality of several components may be performed by fewer components or a single component. Similarly, in some embodiments, any functional component may perform fewer or different operations than those described in the embodiments. Moreover, functional components shown as independent for illustrative purposes may be incorporated into other functional components in a particular implementation.

[0044] While the subject matter has been described with reference to preferred embodiments, those skilled in the art will readily understand that various changes and / or modifications can be made to the subject matter without departing from its spirit or scope. For example, each claim may depend on any or all of the claims in a multi-dependent manner, even if not originally so claimed.

Claims

1. A spatial audio playback device, communicatively connected to a head-mounted device, comprising: The transceiver unit communicates with the head-mounted device via wireless communication and is configured to receive motion information from the head-mounted device on motion sensing signals and receive information or transmission standards from the head-mounted device via wireless communication. as well as A processing unit, connected to the transceiver unit, is configured to predict head information after a preset time based on the motion information, adjust spatial audio based on the predicted head information after the preset time to generate adjusted spatial audio, and transmit the adjusted spatial audio to the head-mounted device via the transceiver unit; and The preset time is changed according to the information or the transmission standard of the wireless communication.

2. The spatial audio playback device as claimed in claim 1, wherein the transmission standard of the information or the wireless communication includes Bluetooth codec type or Bluetooth profile.

3. The spatial audio playback device as claimed in claim 1, wherein the preset time is calculated based on the signal delay time period of the spatial audio playback device and / or the head-mounted device.

4. The spatial audio playback device as claimed in claim 3, wherein the signal delay time period is the time difference between the time when the head-mounted device transmits the motion information and the time when the head-mounted device receives the adjusted spatial audio.

5. The spatial audio playback device as claimed in claim 3, wherein the signal delay time period is the time difference between the time when the head-mounted device sends the motion information and the time when the head-mounted device receives the adjusted spatial audio.

6. The spatial audio playback device as claimed in claim 3, wherein the signal delay time period includes the signal transmission and processing time of the transceiver unit and the processing unit.

7. The spatial audio playback device of claim 3, wherein the processing unit comprises a plurality of sub-processing units, and the signal delay time period comprises the signal transmission and processing time of one or more sub-processing units.

8. The spatial audio playback device of claim 1, wherein the motion information on the motion sensing signal of the head-mounted device includes instantaneous head position information or instantaneous head posture information.

9. The spatial audio playback device as claimed in claim 1, wherein the motion sensing signal includes continuous head position information or continuous head posture information.

10. A spatial audio playback device, communicatively connected to a head-mounted device, comprising: The transceiver unit communicates with the head-mounted device via wireless communication and is configured to receive motion information from the head-mounted device in motion sensing signals. as well as The processing unit, connected to the transceiver unit, is configured to predict head information after a preset time based on the motion information, and adjust the spatial audio after the preset time based on the predicted head information to generate adjusted spatial audio. as well as The preset time is calculated based on the signal delay time period of the spatial audio playback device and / or the head-mounted device.

11. The spatial audio playback device of claim 10, wherein the signal delay time period is the time difference between the time when the head-mounted device transmits the motion information and the time when the head-mounted device receives the adjusted spatial audio.

12. The spatial audio playback device of claim 10, wherein the signal delay time period is the time difference between the time when the head-mounted device sends the motion information and the time when the head-mounted device receives the adjusted spatial audio.

13. The spatial audio playback device of claim 10, wherein the signal delay time period includes the signal transmission and processing time of the transceiver unit and the processing unit.

14. The spatial audio playback device of claim 10, wherein the signal delay time period includes the signal transmission and processing time of the transceiver unit, the processing unit, and the head-mounted device.

15. The spatial audio playback device of claim 10, wherein the processing unit comprises a plurality of sub-processing units, and the signal delay time period comprises the signal transmission and processing time of one or more sub-processing units.

16. The spatial audio playback device of claim 10, wherein the motion information of the head-mounted device on the motion sensing signal includes instantaneous head position information or instantaneous head posture information.

17. The spatial audio playback device of claim 10, wherein the motion sensing signal includes continuous head position information or continuous head posture information.

18. A method for playing spatial audio, comprising: Receive motion information and wireless communication information or transmission standards from head-mounted devices; Predict head information after a preset time based on the motion information; Based on the predicted head information, the spatial audio is adjusted after the preset time to generate adjusted spatial audio; as well as The adjusted spatial audio is transmitted to the head-mounted device; The preset time varies depending on the wireless communication information or transmission standard.

19. The method of claim 18, wherein the preset time is calculated based on the signal delay time period between the spatial audio playback device and the head-mounted device.

20. The method of claim 19, wherein the information or transmission standard of the wireless communication includes a Bluetooth codec type or a Bluetooth profile.

Images