Apparatus and method for time synchronization of multi-channel audio signals of multiple devices on basis of bluetooth connection

Abstract

According to various embodiments of the present disclosure, provided is an operation method of a source device in a short-range wireless communication system, the method comprising the steps of: transmitting, to one or more external speaker devices, a request for information on a second delay of the one or more external speaker devices; receiving the information on the second delay from the one or more external speaker devices; determining a first delay for an embedded speaker of the source device on the basis of the second delay in order to synchronize audio output among the embedded speaker of the source device and the one or more external speaker devices; and transmitting a first audio data stream to the one or more external speaker devices for reproduction synchronized with a second audio data stream output from the embedded speaker on the basis of the determined first delay.

Description

Device and method for time synchronization of multi-channel audio signals of multiple devices based on Bluetooth connection

[0001] The present disclosure relates to a device and method for precisely synchronizing audio signals between a plurality of wired and wireless speakers in a multi-channel audio environment using Bluetooth communication technology. Specifically, the present disclosure relates to a device and method for precisely synchronizing audio playback timing between a source device (e.g., a TV or a sound bar) and a plurality of external wireless speakers based on a Bluetooth audio transmission structure, thereby minimizing sound distortion based on a listener's position and realizing high-quality surround audio.

[0002]

[0003] Multichannel audio systems are designed to deliver three-dimensional surround sound, including not only left and right channels, but also rear, center, and woofer channels. These systems require time synchronization of audio signals between multiple speakers, typically achieved by adjusting the timing of the signals transmitted from the source device to each speaker.

[0004] However, conventional multi-channel audio technology simply synchronizes the output timing between speakers, failing to compensate for delays caused by distance differences between speakers and the listener relative to the listener's position. This can result in misaligned sound. In particular, in environments where wired and wireless speakers are mixed, accurate synchronization based on the listener's position is difficult to achieve due to differences in transmission paths and internal processing delays between each speaker.

[0005] To address this, it is necessary to know in advance all path delays from the source device to the listener, but in real environments, accurate absolute delay measurement is difficult and complex due to manufacturing variations between devices, network delays, and differences in installation environments.

[0006] To overcome these problems, the present invention proposes a new time synchronization technique that measures the arrival time of each speaker at the listener's position and, based on the information, allows a source device to adjust individual delays for each speaker, thereby providing a synchronized audio environment from the listener's perspective.

[0007]

[0008] In order to solve the above-described problem, the present disclosure provides a device and method for precisely synchronizing audio signals between a plurality of wired and wireless speakers in a multi-channel audio environment using Bluetooth communication technology.

[0009] The present disclosure provides a device and method for precisely aligning audio playback timing between a source device and a plurality of external wireless speakers based on a Bluetooth audio transmission structure, thereby minimizing sound distortion based on a listener position and realizing high-quality surround audio.

[0010] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0011]

[0012] According to various embodiments of the present disclosure, a method of operating a source device in a short-range wireless communication system is provided, comprising: transmitting a request for information on a second delay of the one or more external speaker devices to one or more external speaker devices; receiving information on the second delay from the one or more external speaker devices; determining a first delay for an internal speaker of the source device based on the second delay to synchronize audio output between the internal speaker of the source device and the one or more external speaker devices; and transmitting a first audio data stream to the one or more external speaker devices for synchronized playback with a second audio data stream output from the internal speaker based on the determined first delay.

[0013] According to various embodiments of the present disclosure, a method for operating an external speaker device in a short-range wireless communication system is provided, comprising: receiving a request for information on a second delay of the external speaker device from a source device; transmitting information on the second delay to the source device; and receiving a first audio data stream from the source device for synchronized reproduction with a second audio data stream output from the built-in speaker based on a first delay for the built-in speaker of the source device, wherein the first delay for the built-in speaker of the source device is based on the second delay for synchronizing audio output between the built-in speaker of the source device and the external speaker device.

[0014] According to various embodiments of the present disclosure, in a short-range wireless communication system, a source device includes: a first processor corresponding to a host stack; a second processor corresponding to a controller stack; a memory; an input device corresponding to a user interface (UI); an output device corresponding to the UI; and a transceiver, wherein the host stack and the controller stack are connected by a Host Controller Interface (HCI), and the memory stores instructions for performing operations based on what is executed by the first processor and the second processor, the operations comprising: requesting information on a second delay of the one or more external speaker devices from one or more external speaker devices; receiving information on the second delay from the one or more external speaker devices; determining a first delay for an internal speaker of the source device based on the second delay to synchronize audio output between an internal speaker of the source device and the one or more external speaker devices; And a source device is provided, comprising a step of transmitting a first audio data stream to the one or more external speaker devices for synchronized playback with a second audio data stream output from the built-in speaker based on the determined first delay.

[0015]

[0016] In order to solve the above-described problem, the present disclosure can provide a device and method for precisely synchronizing audio signals between a plurality of wired and wireless speakers in a multi-channel audio environment using Bluetooth communication technology.

[0017] The present disclosure provides a device and method for precisely aligning audio playback timing between a source device and a plurality of external wireless speakers based on a Bluetooth audio transmission structure, thereby minimizing sound distortion based on a listener position and realizing high-quality surround audio.

[0018]

[0019] The accompanying drawings are intended to aid in understanding the present disclosure and, together with detailed descriptions, may provide embodiments of the present disclosure. However, the technical features of the present disclosure are not limited to specific drawings, and the features disclosed in each drawing may be combined with each other to form new embodiments. Reference numerals in each drawing may indicate structural elements.

[0020] FIG. 1 is a schematic diagram showing an example of a wireless communication system using the Bluetooth low energy technology proposed in the present disclosure.

[0021] FIG. 2 shows an example of an internal block diagram of a device capable of implementing the methods proposed in the present disclosure.

[0022] FIG. 3 illustrates an example of a Bluetooth communication architecture to which the methods proposed in the present disclosure can be applied.

[0023] Figure 4 shows an example of the structure of the GATT (Generic Attribute Profile) of Bluetooth low energy.

[0024] FIG. 5 is a flowchart illustrating an example of a connection procedure method in Bluetooth low energy technology to which various embodiments of the present disclosure can be applied.

[0025] FIG. 6 illustrates an example of a system structure in which multi-channel audio signals are transmitted between a BLE-based TV transmitting device and multiple BLE audio receiving speakers.

[0026] FIG. 7 illustrates an example of a compensation procedure for synchronizing image processing delay and audio processing delay in a Bluetooth low-latency audio system proposed in the present disclosure.

[0027] FIG. 8 illustrates an example of a test stream-based calibration procedure for measuring and adjusting the sound pressure level of each speaker based on a user position in a BLE-based audio system proposed in the present disclosure.

[0028] FIG. 9 illustrates an example of an inter-speaker synchronization correction structure for correcting an audio synchronization error due to a difference in distance between a TV speaker and a rear speaker in a multi-channel audio system proposed in the present disclosure.

[0029] FIG. 10 illustrates an example of a speaker-to-speaker synchronization structure for correcting sound synchronization errors due to distance differences between rear speakers in a multi-channel audio system proposed in the present disclosure.

[0030] FIG. 11 illustrates an example of a structure in which each speaker outputs a different audio channel (1 kHz, 2 kHz, etc.) in a Bluetooth-based multi-channel audio environment proposed in the present disclosure, and a microphone device at the listener's position receives the same.

[0031] FIG. 12 illustrates an example of a process of measuring the arrival time difference of test sounds from each speaker received by a microphone device at a listener's position in a multi-channel audio system proposed in the present disclosure and transmitting the difference to a source device (TV).

[0032] FIG. 13 illustrates an example of a structure in which a third device, such as a Bluetooth remote control or a smartphone, measures the time at which each speaker receives a test tone, and a source device adjusts the delay for each speaker based on this in a multi-channel audio system proposed in the present disclosure.

[0033] FIG. 14 illustrates an example of a structure that independently defines the presentation delay of a built-in speaker in a source device (TV) and matches the total audio delay with Bluetooth speakers in a multi-channel audio system proposed in the present disclosure.

[0034] FIG. 15 illustrates a multi-channel audio transmission structure between a source device (such as a sound bar) and multiple wired / wireless speakers in a multi-channel audio system proposed in the present disclosure, and illustrates an example of a configuration in which each speaker is connected via various paths such as HDMI, Opt, and Bluetooth.

[0035] FIG. 16 illustrates an example of an audio output timing control structure for synchronized audio output between speakers in a multi-channel audio system proposed in the present disclosure.

[0036] FIG. 17 illustrates an example of an operation process of a source device such as a TV or sound bar according to various embodiments of the present disclosure.

[0037] FIG. 18 illustrates an example of an operation process of a speaker according to various embodiments of the present disclosure.

[0038]

[0039] In various embodiments of the present disclosure, “A or B” may mean “only A,” “only B,” or “both A and B.” In other words, in various embodiments of the present disclosure, “A or B” may be interpreted as “A and / or B.” For example, in various embodiments of the present disclosure, “A, B or C” may mean “only A,” “only B,” “only C,” or “any combination of A, B and C.”

[0040] In various embodiments of the present disclosure, a slash ( / ) or a comma may mean "and / or." For example, "A / B" may mean "A and / or B." Accordingly, "A / B" may mean "only A," "only B," or "both A and B." For example, "A, B, C" may mean "A, B, or C."

[0041] In various embodiments of the present disclosure, “at least one of A and B” may mean “only A,” “only B,” or “both A and B.” Furthermore, in various embodiments of the present disclosure, the expressions “at least one of A or B” or “at least one of A and / or B” may be interpreted as equivalent to “at least one of A and B.”

[0042] Additionally, in various embodiments of the present disclosure, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and / or C” can mean “at least one of A, B and C.”

[0043]

[0044] Figure 1 is a schematic diagram showing an example of a wireless communication system using the Bluetooth low energy technology proposed in this specification.

[0045] A wireless communication system (100) includes at least one server device (Server Device, 120) and at least one client device (Client Device, 110).

[0046] The server device and client device perform Bluetooth communication using Bluetooth Low Energy (BLE, hereinafter referred to as 'BLE' for convenience) technology.

[0047] First, BLE technology has a relatively small duty cycle compared to Bluetooth BR / EDR (Basic Rate / Enhanced Data Rate) technology, allows for low-cost production, and significantly reduces power consumption through low-speed data transmission rates, enabling operation for more than a year using a coin cell battery.

[0048] Additionally, BLE technology simplifies the connection process between devices and is designed to have a smaller packet size compared to Bluetooth BR / EDR technology.

[0049] In BLE technology, (1) the number of RF channels is 40, (2) the data transmission speed supports 1 Mbps, (3) the topology is a scatternet structure, (4) the latency is 3 ms, (5) the maximum current is 15 mA or less, (6) the output power is 10 mW (10 dBm) or less, and (7) it is mainly used in applications such as mobile phones, watches, sports, healthcare, sensors, and device control.

[0050] The server device (120) can operate as a client device in a relationship with other devices, and the client device can operate as a server device in a relationship with other devices. That is, in a BLE communication system, any one device can operate as a server device or a client device, and, if necessary, can operate as a server device and a client device simultaneously.

[0051] The above server device (120) may be expressed as a data service device, a client device, a slave device, a server, a conductor, a host device, a gateway, a sensing device, a monitoring device, a first device, a second device, etc.

[0052] The above client device (110) can be expressed as a server device (master device), a master, a client, a member, a sensor device, a sink device, a collector, a third device, a fourth device, etc.

[0053] The server device and the client device are the main components of the wireless communication system, and the wireless communication system may include other components in addition to the server device and the client device.

[0054] The above server device refers to a device that receives data from a client device and performs direct communication with the client device, thereby providing data to the client device through a response when receiving a data request from the client device.

[0055] Additionally, the server device sends a notification message and an indication message to the client device to provide data information to the client device. Additionally, when the server device transmits an indication message to the client device, it receives a confirmation message corresponding to the indication message from the client.

[0056] In addition, the server device can provide data information to a user through an output unit (Display Unit) or receive a request input from a user through an input unit (User Input Interface) in the process of transmitting and receiving notifications, instructions, and confirmation messages to and from a client device.

[0057] Additionally, the server device can read data from a memory unit or write new data to the memory unit during the process of transmitting and receiving messages with the client device.

[0058] Additionally, one server device can be connected to multiple client devices, and can easily reconnect (or connect) to the client devices by utilizing bonding information.

[0059] The above client device (120) refers to a device that requests data information and data transmission from a server device.

[0060] The client device receives data from the server device through a notification message, an instruction message, etc., and when receiving an instruction message from the server device, sends a confirmation message in response to the instruction message.

[0061] Likewise, the client device can provide information to the user through an output unit or receive input from the user through an input unit during the process of transmitting and receiving messages with the server device.

[0062] Additionally, the client device can read data from memory or write new data to the memory during the process of transmitting and receiving messages with the server device.

[0063] Hardware components such as the output section, input section, and memory of the above server device and client device will be examined in detail in Fig. 2.

[0064] Additionally, the wireless communication system can establish a Personal Area Network (PAN) via Bluetooth technology. For example, the wireless communication system can establish a private piconet between devices, enabling the quick and secure exchange of files, documents, and other information.

[0065] FIG. 2 shows an example of an internal block diagram of a device capable of implementing the methods proposed in this specification.

[0066] As illustrated in FIG. 2, the server device (110) includes an input unit (User Input Interface, 112), a power supply unit (Power Supply Unit, 113), a control unit (Control Unit, 114), a memory unit (Memory Unit, 115), a network interface (Network Interface, 116) including a Bluetooth interface, storage (Storage, 117), an output unit (Display Unit, 118), and a multimedia module (Multi Media Module, 119).

[0067] The input unit (User Input Interface, 112), power supply unit (Power Supply Unit, 113), control unit (Control Unit, 114), memory (Memory Unit, 115), network interface (Network Interface, 116) including Bluetooth interface, storage (Storage, 117), output unit (Display Unit, 118), and multimedia module (Multi media Module, 119) are functionally interconnected to perform the method proposed in this specification.

[0068] In addition, as illustrated in FIG. 2, the client devices (#1 and #2) (120) include an input unit (User Input Interface, 122), a power supply unit (Power Supply Unit, 123), a control unit (Control Unit, 124), a memory unit (Memory Unit, 125), a network interface (Network Interface, 126) including a Bluetooth interface, storage (Storage, 127), an output unit (Display Unit, 128), and a multimedia module (Multi media Module, 129).

[0069] The above input unit (User Input Interface, 122), power supply unit (Power Supply Unit, 123), control unit (Control Unit, 124), memory (Memory Unit, 125), network interface (Network Interface, 126) including Bluetooth interface, storage (Storage, 127), output unit (Display Unit, 128), and multimedia module (Multi media Module, 129) are functionally interconnected to perform the method proposed in this specification.

[0070] The above network interface (116, 126) refers to a unit (or module) that can transmit data, such as request / response, command, notification, instruction / confirmation messages, etc., between devices using Bluetooth technology.

[0071] The above memory (115, 125) is a unit implemented in various types of devices, and refers to a unit in which various types of data are stored. In addition, the storage (117, 127) refers to a unit that performs a function similar to memory.

[0072] The above control unit (114, 124) refers to a module that controls the overall operation of the server device (110) or client device (120), and controls to request transmission of a message to a network interface or to process a received message.

[0073] The above control unit (114, 124) may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and / or a data processing device.

[0074] The above memory (115, 125) may include a ROM (read-only memory), a RAM (random access memory), a flash memory, a memory card, a storage medium, and / or other storage devices.

[0075] The above memory (115, 125) may be internal or external to the processor (114, 124) and may be connected to the processor (114, 124) by various well-known means.

[0076] The above output unit (118, 128) refers to a module for providing device status information and message exchange information to the user through a screen.

[0077] The above power supply unit (power supply unit, 113, 123) refers to a module that receives external power and internal power under the control of the control unit and supplies the power required for the operation of each component.

[0078] As discussed above, BLE technology has a small duty cycle and can significantly reduce power consumption through low data rates.

[0079] Figure 3 shows an example of a Bluetooth communication architecture to which the methods proposed in this specification can be applied.

[0080] Specifically, FIG. 3 shows an example of the architecture of Bluetooth LE (Low Energy).

[0081] As illustrated in FIG. 3, the BLE architecture includes a controller stack (Controller stackACK) operable to handle timing-critical wireless device interfaces and a host stack (Host stackACK) operable to handle high level data.

[0082] The above Controller stack may be referred to as a Controller, but to avoid confusion with the processor, which is an internal component of the device mentioned in FIG. 2 above, it will be referred to as a Controller stackACK hereinafter.

[0083] First, the controller stack may be implemented using a communication module that may include a Bluetooth radio, and a processor module that may include a processing device, such as a microprocessor.

[0084] The host stack can be implemented as part of an OS running on a processor module, or as an instantiation of a package on top of the OS.

[0085] In some cases, the controller stack and the host stack may operate or execute on the same processing device within a processor module.

[0086] The host stack includes GAP (Generic Access Profile, 310), GATT-based Profiles (320), GATT (Generic Attribute Profile, 330), ATT (Attribute Protocol, 340), SM (Security Manager, 350), and L2CAP (Logical Link Control and Adaptation Protocol, 360). However, the host stack is not limited to this and may include various protocols and profiles.

[0087] The host stack multiplexes various protocols, profiles, etc. provided above Bluetooth using L2CAP.

[0088] First, L2CAP (Logical Link Control and Adaptation Protocol,360) provides a single bidirectional channel for transmitting data to a specific protocol or profile.

[0089] L2CAP may be capable of multiplexing data between upper layer protocols, segmenting and reassembling packages, and managing multicast data transmission.

[0090] BLE uses three fixed channels (one for signaling CH, one for Security Manager, and one for Attribute protocol).

[0091] On the other hand, BR / EDR (Basic Rate / Enhanced Data Rate) uses dynamic channels and supports protocol service multiplexer, retransmission, streaming mode, etc.

[0092] SM (Security Manager, 350) is a protocol for authenticating devices and providing key distribution.

[0093] ATT (Attribute Protocol, 340) defines the rules for accessing data from other devices in a server-client architecture. ATT has six message types: Request, Response, Command, Notification, Indication, and Confirmation.

[0094] That is, ① Request and Response messages: The Request message is a message for requesting specific information from a client device to a server device, and the Response message is a response message to the Request message, and refers to a message transmitted from the server device to the client device.

[0095] ② Command message: A message sent from a client device to a server device to instruct a specific action command. The server device does not send a response to the Command message to the client device.

[0096] ③ Notification message: A message sent from a server device to a client device for notification of events, etc.; the client device does not send a confirmation message for the Notification message to the server device.

[0097] ④ Indication and Confirm messages: Messages sent from the server device to the client device for notification of events, etc. Unlike the Notification message, the client device sends a confirmation message for the Indication message to the server device.

[0098] GAP (Generic Access Profile) is a new layer implemented for BLE technology, which is used to control role selection for communication between BLE devices and how multi-profile operation occurs.

[0099] Additionally, GAP is primarily used in device discovery, connection creation, and security procedures, defines how to provide information to users, and defines the types of attributes as follows:

[0100] ① Service: Defines the basic operation of the device through a combination of data-related behaviors.

[0101] ② Include: Defines the relationship between services

[0102] ③ Characteristics: Data values ​​used in the service

[0103] ④ Behavior: Computer-readable format defined as UUID (Universal Unique Identifier, value type)

[0104] GATT-based Profiles are profiles that depend on GATT and are primarily applied to BLE devices. GATT-based Profiles include Battery, Time, FindMe, Proximity, Time, and Object Delivery Service. The specifics of GATT-based Profiles are as follows.

[0105] Battery: How to exchange battery information

[0106] Time: A method for exchanging time information

[0107] FindMe: Distance-based alarm service

[0108] Proximity: How to Exchange Battery Information

[0109] Time: A method for exchanging time information

[0110] GATT can function as a protocol that describes how ATT is used when composing services. For example, GATT can be used to specify how ATT attributes are grouped into services, and can be used to describe the features associated with services.

[0111] Therefore, GATT and ATT can use features to describe the state and services of a device, how features relate to each other, and how they are used.

[0112] The controller stack includes a physical layer (390), a link layer (380), and a host controller interface (370).

[0113] The physical layer (wireless transmission / reception module, 390) is a layer that transmits and receives 2.4 GHz wireless signals and uses GFSK (Gaussian Frequency Shift Keying) modulation and a frequency hopping technique consisting of 40 RF channels.

[0114] The link layer (380) transmits or receives Bluetooth packets.

[0115] Additionally, the link layer performs advertising and scanning functions using three advertising channels, then creates a connection between devices, and provides the ability to send and receive data packets of up to 42 bytes through 37 data channels.

[0116] HCI (Host Controller Interface) provides an interface between the Host stack and the Controller stack, allowing the Host stack to provide commands and data to the Controller stack, and allowing the Controller stack to provide events and data to the Host stack.

[0117] Below, we will briefly look at the procedures of Bluetooth Low Energy (BLE) technology.

[0118] BLE procedures can be divided into device filtering procedures, advertising procedures, scanning procedures, discovering procedures, and connecting procedures.

[0119] Device Filtering Procedure

[0120] Device filtering procedures are a way to reduce the number of devices that respond to requests, instructions, notifications, etc. in the controller stack.

[0121] When a request is received from any device, since it is unnecessary to respond to it, the controller stack can control the power consumption of the BLE controller stack by reducing the number of requests transmitted.

[0122] An advertising device or scanning device may perform the above device filtering procedure to limit the devices that receive advertising packets, scan requests, or connection requests.

[0123] Here, an advertising device refers to a device that transmits an advertising event, i.e., performs an advertisement, and is also expressed as an advertiser.

[0124] A scanning device is a device that performs scanning and transmits scan requests.

[0125] In BLE, when a scanning device receives some advertising packets from an advertising device, the scanning device must send a scan request to the advertising device.

[0126] However, if a device filtering procedure is used and transmission of scan requests is unnecessary, the scanning device may ignore advertising packets transmitted from the advertising device.

[0127] Device filtering procedures may also be used during the connection request process. If device filtering is used during the connection request process, the connection request is ignored, eliminating the need to send a response to the connection request.

[0128] Advertising Procedure

[0129] An advertising device performs an advertising procedure to perform a non-directional broadcast to devices within the area.

[0130] Here, non-directional broadcast refers to broadcast in all directions rather than broadcasting in a specific direction.

[0131] In contrast, a directional broadcast refers to a broadcast in a specific direction. A non-directional broadcast occurs without a connection procedure between an advertising device and a device in a listening (or listening) state (hereinafter referred to as a listening device).

[0132] The advertising process is used to establish a Bluetooth connection with a nearby initiating device.

[0133] Alternatively, the advertising procedure may be used to provide periodic broadcasts of user data to scanning devices listening on the advertising channel.

[0134] In the advertising process, all advertisements (or advertising events) are broadcast through the advertising physical channel.

[0135] Advertising devices can receive scan requests from listening devices that are listening to obtain additional user data from the advertising device. The advertising device transmits a response to the scan request to the device that sent the scan request over the same advertising physical channel as the advertising physical channel that received the scan request.

[0136] Broadcast user data sent as part of advertising packets is dynamic data, whereas scan response data is typically static data.

[0137] An advertising device can receive a connection request from an initiating device on an advertising (broadcast) physical channel. If the advertising device uses a connectable advertising event and the initiating device is not filtered by the device filtering procedure, the advertising device stops advertising and enters connected mode. The advertising device can resume advertising after entering connected mode.

[0138] Scanning Procedure

[0139] A device performing scanning, i.e., a scanning device, performs a scanning procedure to listen for non-directional broadcasts of user data from advertising devices using an advertising physical channel.

[0140] The scanning device transmits a scan request to the advertising device via the advertising physical channel to request additional data from the advertising device. The advertising device transmits a scan response, which is a response to the scan request, including the additional data requested from the scanning device via the advertising physical channel.

[0141] The above scanning procedure can be used while connecting with other BLE devices in a BLE piconet.

[0142] If the scanning device is in initiator mode, which means that it can receive a broadcasted advertising event and initiate a connection request, the scanning device can initiate a Bluetooth connection with the advertising device by sending a connection request to the advertising device over the advertising physical channel.

[0143] When a scanning device sends a connection request to an advertising device, the scanning device stops initiator mode scanning for further broadcasts and enters connection mode.

[0144] Discovery Procedure

[0145] Devices capable of Bluetooth communication (hereinafter referred to as “Bluetooth devices”) perform advertising and scanning procedures to discover nearby devices or to be discovered by other devices within a given area.

[0146] The discovery process is performed asymmetrically. A Bluetooth device attempting to discover other devices in its vicinity is called a discovering device and listens for devices advertising scannable advertising events. A Bluetooth device discovered and available to other devices is called a discoverable device and actively broadcasts advertising events over the advertising (broadcast) physical channel to make it scannable.

[0147] Both the discovering device and the discoverable device may already be connected to other Bluetooth devices in the piconet.

[0148] Connecting Procedure

[0149] The connection procedure is asymmetric, requiring one Bluetooth device to perform an advertising procedure while another Bluetooth device performs a scanning procedure.

[0150] That is, the advertising process can be targeted, resulting in only one device responding to the advertisement. After receiving an accessible advertising event from the advertising device, a connection can be initiated by sending a connection request to the advertising device via the advertising (broadcast) physical channel.

[0151] Next, we will briefly look at the operating states in BLE technology, namely Advertising State, Scanning State, Initiating State, and Connection State.

[0152] Advertising State

[0153] The Link Layer (LL) enters the advertising state at the direction of the host (stack). When the Link Layer is in the advertising state, it transmits advertising Packet Data Units (PDUs) in advertising events.

[0154] Each advertising event consists of at least one advertising PDU, which are transmitted via the advertising channel indices used. An advertising event may terminate when each advertising PDU has been transmitted via the advertising channel indices used, or may terminate earlier if the advertising device needs to free up space for other functions.

[0155] Scanning State

[0156] The link layer enters the scanning state at the direction of the host (stack). In the scanning state, the link layer listens for advertising channel indices.

[0157] There are two types of scanning states: passive scanning and active scanning, and each scanning type is determined by the host.

[0158] No separate time or advertising channel index is defined for performing scanning.

[0159] During the scanning state, the link layer listens for advertising channel indices for the duration of the scanWindow. The scanInterval is defined as the interval between the start points of two consecutive scan windows.

[0160] The link layer must listen for the completion of all scan intervals in the scan window, as directed by the host, provided there are no scheduling conflicts. In each scan window, the link layer must scan a different advertising channel index. The link layer uses all available advertising channel indices.

[0161] In passive scanning, the link layer only receives packets and does not transmit any packets.

[0162] When actively scanning, the link layer listens to the advertising device for advertising PDUs and depending on the advertising PDU type, may request additional information about the advertising device.

[0163] Initiating State

[0164] The link layer enters the initiated state at the direction of the host (stack).

[0165] When the link layer is in the initiating state, the link layer listens for advertising channel indices.

[0166] During the initiation state, the link layer listens for advertising channel indices during the scan window period.

[0167] connection state

[0168] The link layer enters a connected state when the device performing the connection request, i.e., the initiating device, sends a CONNECT_REQ PDU to the advertising device, or when the advertising device receives a CONNECT_REQ PDU from the initiating device.

[0169] Once a connection enters the connected state, it is considered established. However, the connection need not be considered established at the time it enters the connected state. The only difference between a newly created connection and an established connection is the link-layer connection supervision timeout value.

[0170] When two devices are connected, they act in different roles.

[0171] The link layer that performs the master role is called the master, and the link layer that performs the slave role is called the slave. The master controls the timing of connection events, and connection events are the points in time when the master and slave are synchronized.

[0172] Below, we will briefly examine the packets defined in the Bluetooth interface. BLE devices use the packets defined below.

[0173] Packet Format

[0174] The Link Layer has only one packet format, which is used for both advertising channel packets and data channel packets.

[0175] Each packet consists of four fields: Preamble, Access Address, PDU, and CRC.

[0176] When a packet is transmitted on an advertising physical channel, the PDU will be an advertising channel PDU, and when a packet is transmitted on a data physical channel, the PDU will be a data channel PDU.

[0177] Advertising Channel PDU

[0178] Advertising channel PDUs (Packet Data Units) have a 16-bit header and payloads of various sizes.

[0179] The PDU type field of the advertising channel PDU included in the header indicates the PDU type as defined in Table 1 below.

[0180] PDU TypePDU NameChannelPermitted PHYsLE 1MLE 2MLE Coded0000bADV_INDPrimary AdvertisingO0001bADV_DIRECT_INDPrimary AdvertisingO0010bADV_NONCONN_INDPrimary AdvertisingO0011bSCAN_REQPrimary AdvertisingOAUX_SCAN_REQSecondary AdvertisingOOO0100bSCAN_RSPPrimary AdvertisingO0101bCONNECT_INDPrimary AdvertisingOAUX_CONNECT_REQSecondary AdvertisingOOO0110bADV_SCAN_INDPrimary AdvertisingO

[0181] Advertising PDU

[0182] The advertising channel PDU types below are called advertising PDUs and are used in specific events.

[0183] ADV_IND: Connectable non-directional advertising event

[0184] ADV_DIRECT_IND: Connectable directional advertising event

[0185] ADV_NONCONN_IND: Non-directional ad event that is not reachable

[0186] ADV_SCAN_IND: Scannable non-directional ad event

[0187] The above PDUs are transmitted by the link layer in the advertising state and received by the link layer in the scanning state or initiating state.

[0188] Scanning PDU

[0189] The advertising channel PDU type below is called a scanning PDU and is used in the conditions described below.

[0190] SCAN_REQ: Sent by the link layer in scanning state and received by the link layer in advertising state.

[0191] SCAN_RSP: Sent by the link layer in advertising state and received by the link layer in scanning state.

[0192] Initiating PDU

[0193] The advertising channel PDU type below is called an initiation PDU.

[0194] CONNECT_REQ: Sent by the link layer in the initiating state and received by the link layer in the advertising state.

[0195] Data Channel PDU

[0196] A data channel PDU has a 16-bit header, a payload of variable size, and may include a Message Integrity Check (MIC) field.

[0197] The procedures, states, packet formats, etc. in BLE technology discussed above can be applied to perform the methods proposed in this specification.

[0198]

[0199] Figure 4 shows an example of the structure of the GATT (Generic Attribute Profile) of Bluetooth low energy.

[0200] Referring to Figure 4, a structure for exchanging profile data of Bluetooth low energy can be examined.

[0201] Specifically, GATT (Generic Attribute Profile) defines how to exchange data using services and characteristics between Bluetooth LE devices.

[0202] Typically, a peripheral device (e.g., a sensor device) acts as a GATT server and has definitions for services and characteristics.

[0203] To read or write data, a GATT client sends a data request to a GATT server, and all operations (transactions) are initiated by the GATT client and receive responses from the GATT server.

[0204] The GATT-based operation structure used in Bluetooth LE is based on profiles, services, and characteristics, and can form a vertical structure as shown in Fig. 5.

[0205] The above profile consists of one or more services, and the services may consist of one or more characteristics or other services.

[0206] The above service serves to logically divide data into units and may include one or more characteristics or other services. Each service has a 16-bit or 128-bit identifier called a Universal Unique Identifier (UUID).

[0207] The above characteristic is the lowest unit in the GATT-based operation structure. The characteristic contains only one piece of data and, similar to the above service, has a 16-bit or 128-bit UUID.

[0208] The above characteristics are defined by the values ​​of various pieces of information, and each piece of information requires one attribute. Multiple consecutive attributes can be used for the above characteristics.

[0209] The above attribute consists of four components and has the following meanings:

[0210] - handle: address of the property

[0211] - Type: Type of property

[0212] - Value: The value of the property

[0213] - Permission: Access rights to properties

[0214]

[0215] FIG. 5 is a flowchart showing an example of a connection procedure method in Bluetooth low energy technology to which the present invention can be applied.

[0216] The server transmits advertising messages to the client through three advertising channels (S5010).

[0217] A server may be called an Advertiser before connection, and a Master after connection. An example of such a server may be a sensor (e.g., a temperature sensor).

[0218] Additionally, a client may be referred to as a "Scanner" before connection and as a "Slave" after connection. An example of a client may be a smartphone.

[0219] As previously discussed, Bluetooth communicates across 40 channels over the 2.4 GHz band. Of these 40 channels, three are advertising channels, used to exchange various advertising packets and other packets used to establish a connection.

[0220] The remaining 37 channels are used for data exchange after connection as data channels.

[0221] After receiving the advertisement message, the client may send a Scan Request message to the server to obtain additional data (e.g., server device name, etc.) from the server.

[0222] In this case, the server transmits a Scan Response message containing additional data in response to a Scan Request message to the client.

[0223] Here, the Scan Request message and the Scan Response message are the end of an advertising packet, and the advertising packet can only contain user data of 31 bytes or less.

[0224] Therefore, if there is data whose size is larger than 3 bytes but the overhead is too high to send the data by establishing a connection, the data is sent in two parts using a scan request message / scan response message.

[0225] Next, the client sends a connection request message to the server to establish a Bluetooth connection with the server (S5020).

[0226] Through this, a Link Layer (LL) connection is established between the server and the client.

[0227] Afterwards, the server and client perform security establishment procedures.

[0228] The secure establishment procedure may be interpreted as or performed incorporating Secure Simple Pairing.

[0229] That is, the security establishment procedure can be carried out through Phase 1 to Phase 3.

[0230] Specifically, a pairing procedure (phase 1) is performed between the server and the client (S5030).

[0231] The pairing procedure involves the client sending a pairing request message to the server, and the server sending a pairing response message to the client.

[0232] The pairing process involves exchanging authentication requirements, input / output capabilities, and key size information between devices. This information is used to determine which key generation method to use in Phase 2.

[0233] Next, as phase 2, legacy pairing or secure connections are performed between the server and the client (S5040).

[0234] In Phase 2, a 128-bit temporary key and a short term key (STK) are generated to perform legacy pairing.

[0235] - Temporary Key: Key created to generate STK

[0236] - Short Term Key (STK): Key value used to create an encrypted connection between devices.

[0237] If a secure connection is performed in Phase 2, a 128-bit Long Term Key (LTK) is generated.

[0238] - Long Term Key (LTK): A key value used not only for encrypted connections between devices but also for future connections.

[0239] Next, as phase 3, a key distribution procedure is performed between the server and the client (S5050).

[0240] This establishes a secure connection between the server and client, forming an encrypted link to enable data transmission and reception.

[0241] Isochronous Channel General

[0242] For audio signals, you can see that audio streaming data or audio data occurs periodically at Idle Event Interval intervals.

[0243] Audio data occurs periodically (or at specific time intervals) depending on its characteristics. Here, a specific time period during which audio data occurs periodically can be expressed as an Idle Event Interval. Each audio data is transmitted during each Idle Event Interval. Furthermore, each audio data can be transmitted throughout the entire Idle Event Interval or a portion of the Idle Event Interval. When transmitting audio streaming data that occurs periodically or regularly using the BLE mechanism, advertising and scanning procedures, communication procedures, and disconnection procedures must be performed each time the generated audio data is transmitted or received. However, audio data is generally generated periodically, and regardless of the amount of data, a latency guarantee for audio data transmission is essential.

[0244] However, there is a problem that latency occurs in audio data transmission when advertising and scanning procedures, communication procedures, and disconnection procedures must be performed every time new audio data is transmitted.

[0245] Audio data transmission through hearing aids (HA) or headsets can achieve higher energy efficiency by utilizing BLE technology rather than Bluetooth BR / EDR technology because the amount of data generated is relatively small. However, as previously discussed, the Data Channel Process of BLE technology requires advertising and connection for each data transmission, which results in a large overhead in data transmission. In particular, the Latency Guarantee, which is absolutely necessary for audio data transmission, cannot be guaranteed.

[0246] In addition, since the Data Channel Process of BLE technology aims to increase energy efficiency by transmitting data that occurs sporadically only when necessary and inducing deep sleep of the BLE device in other time domains, it may be difficult to apply the Data Channel Process of BLE technology to the transmission of audio data that occurs periodically.

[0247] Definition of Isochronous Channels and Related Mechanisms

[0248] A new channel, the Isochronous Channel, is defined to transmit periodically occurring data using BLE technology.

[0249] An isochronous channel is a channel used to transmit isochronous data between devices that use isochronous streams (e.g., Conductor-Member).

[0250] Isochronous data refers to data that is transmitted periodically or regularly at specific time intervals.

[0251] That is, an isochronous channel may refer to a channel through which periodically occurring data, such as audio data or voice data, is transmitted and received in BLE technology. In addition, the isochronous channel may refer to a channel through which data generated based on a user input of a game user's controller device is transmitted and received in a gaming scenario. The isochronous channel may be used to transmit and receive data with a single member, a set of one or more coordinated members, or multiple members. In addition, the isochronous channel corresponds to a flushing channel that may be used to transmit and receive important data in an isochronous stream, such as audio streaming, or in another time domain.

[0252]

[0253] Composition of various embodiments of the present disclosure

[0254] Problems with prior art

[0255] In multichannel audio, if only the time synchronization between speakers is adjusted, the sound image will be distorted due to the time delay caused by the difference in distance between the speakers and the listener.

[0256] In order to synchronize wired and wireless speakers at the listener's position in a multi-channel audio environment, the source device must know all delay times from the source to the listener's position, and it is not easy to accurately set the delay time due to variations between products or within the same product.

[0257]

[0258] Summary of various embodiments of the present disclosure

[0259] Various embodiments of the present disclosure provide a method for synchronizing audio at a listener's position in a multichannel audio environment.

[0260] Various embodiments of the present disclosure provide a method for synchronizing audio at a listener's position by measuring only the listener's delay time at the source device.

[0261] Various embodiments of the present disclosure provide a method for synchronizing audio by fixing a wired audio delay time and changing a wireless audio delay time, or by fixing a wireless audio delay time and changing a wired audio delay time, when building a multi-channel audio environment using wired and wireless speakers.

[0262]

[0263] Effects of various embodiments of the present disclosure

[0264] Through various embodiments of the present disclosure, the sound distortion effect caused by time delay due to the difference in distance between each speaker from the listener's perspective is resolved.

[0265] Various embodiments of the present disclosure enable synchronization from the source device to any product, from the source device to the output speaker.

[0266] Various embodiments of the present disclosure enable synchronization of multi-channel audio without having to know the delay time of third-party products on a Sound Bar or TV.

[0267] Various embodiments of the present disclosure enable synchronization of multichannel audio when audio delay is known.

[0268]

[0269] FIG. 6 illustrates an example of a system structure in which multi-channel audio signals are transmitted between a BLE-based TV transmitting device and multiple BLE audio receiving speakers.

[0270] Figure 6 illustrates an example system architecture that wirelessly transmits multi-channel audio signals between a TV transmitter and multiple audio receiver speakers using BLE (Bluetooth Low Energy) technology. The system of Figure 6 configures the TV as an audio transmitter (Initiator) and each speaker as an audio receiver (Acceptor), enabling a multi-channel audio environment without wired connections. It utilizes BLE's ISO channel, making it suitable for audio data transmission where time synchronization is critical.

[0271] The TV LE Audio Tx Device includes a video display, an audio decoder / encoder, a controller, and a BLE Tx interface. The audio decoder / encoder converts surround audio formats such as Dolby or DTS into PCM and encodes it into the LC3 codec suitable for BLE transmission. It also supports mono unicast transmission. The TV wirelessly transmits the processed audio signal to external speakers via the BLE Tx interface. Configuration information can be communicated bidirectionally, while the actual audio stream is transmitted unidirectionally.

[0272] The Speaker LE Audio Rx Device consists of a BLE Tx / Rx interface, an audio decoder, an LC3 decoder, a controller, and a speaker driver. Each speaker receives BLE audio data wirelessly transmitted from the TV through the BLE Tx / Rx interface. The received data is restored to the original audio signal through the LC3 decoder and audio decoder, and this signal is output as sound through the speaker driver. Speakers with Tx / Rx interfaces can be used for exchanging setting information with the TV or receiving control signals.

[0273] The system in Figure 6 operates by processing multichannel audio signals, such as 5.1 channels, from a TV and assigning and transmitting them to wireless speakers, either individually or in groups. Each speaker receives and plays audio data for its assigned channel, allowing users to experience a wireless surround sound system without the need for complex audio cable connections. This offers the advantage of increased installation flexibility and a cleaner user experience.

[0274]

[0275] FIG. 7 illustrates an example of a compensation procedure for synchronizing image processing delay and audio processing delay in a Bluetooth low-latency audio system proposed in the present disclosure.

[0276] Video and audio signals each have their own unique delays as they pass through various stages of decoding, processing, and output. This delay difference can cause misalignment between video and audio, making it essential to synchronize the two for a natural audiovisual experience.

[0277] The video processing path involves compressed data being decoded, then undergoing image enhancement and display output. This process can include various processes such as frame doubling, anti-aliasing, and TCON (Timing Controller), which, as shown in the diagram, can result in a relatively large delay of approximately 100 ms or more. This is the result of a combination of the physical delay required to display the image on screen and the processing delay.

[0278] The audio processing path involves decoding compressed audio data, then going through the Bluetooth encoding / transmission / decoding process, and possibly applying additional audio enhancements. This includes various audio-related processes such as buffering, pop filtering, woofer boost, and active noise cancellation. These processes introduce delays into the audio signal, and the diagram shows this audio processing delay at approximately 30ms or more. This delay is relatively short compared to video delay.

[0279] There is a significant difference between video processing delay and audio processing delay. The compensation procedure presented in Figure 7 aims to overcome this delay difference and synchronize video and audio. It utilizes information such as the Presentation Time Stamp (PTS) to determine the original timing of the video and audio data, intentionally delaying the output of the audio signal with a shorter delay or making other timing adjustments to ensure that the video and audio are output nearly simultaneously.

[0280]

[0281] FIG. 8 illustrates an example of a test stream-based calibration procedure for measuring and adjusting the sound pressure level of each speaker based on a user position in a BLE-based audio system proposed in the present disclosure.

[0282] Figure 8 illustrates an example of a test stream-based calibration procedure for measuring and adjusting the sound pressure level (SPL) from each speaker relative to the user's listening position (sweet spot) in a Bluetooth low-latency audio system. This process aims to optimize balanced sound from each speaker in a multi-speaker audio system at the user's primary seating position. This procedure utilizes a measurement device, such as a smartphone, to achieve this goal.

[0283] The calibration process begins with playing a test audio stream to each speaker. As shown in the figure, the user's smartphone or control device individually sends a command to play the test stream to each speaker. Each speaker then plays its own corresponding test sound via a unicast connection. This test stream may be a signal with a specific frequency range or characteristics designed for loudness measurement.

[0284] While the test stream plays through each speaker, the user measures the sound pressure level (SPL) arriving at each speaker using a sound pressure measurement device, such as a smartphone, at their primary listening position (the "Sweet Spot"). The figure shows the user sitting in the center, holding the smartphone, and measuring the SPL values ​​coming from each speaker. The measured SPL values ​​for each speaker position may vary depending on factors such as the distance from the speaker, the speaker's performance, and the surrounding environment (e.g., SPL=60, SPL=80, SPL=30).

[0285] Based on the measured sound pressure level of each speaker, the system adjusts the final audio output. By adjusting the volume or gain of each speaker to compensate for differences in measured SPL values, the system ensures that all speakers sound at the intended relative loudness when the user is listening in the "Sweet Spot." This process compensates for volume differences caused by individual speaker locations or characteristics, and optimizes the overall sound field of the multichannel audio system to the user's location, providing a more immersive sound experience.

[0286]

[0287] FIG. 9 illustrates an example of an inter-speaker synchronization correction structure for correcting an audio synchronization error due to a difference in distance between a TV speaker and a rear speaker in a multi-channel audio system proposed in the present disclosure.

[0288] FIG. 10 illustrates an example of a speaker-to-speaker synchronization structure for correcting sound synchronization errors due to distance differences between rear speakers in a multi-channel audio system proposed in the present disclosure.

[0289] Figures 9 and 10 illustrate various examples of inter-speaker synchronization structures for compensating for acoustic synchronization errors caused by distance differences between multiple speakers and the user's listening position in a multichannel audio system. Since sound propagates through the air at a finite speed, if each speaker is located at a different distance from the listener, the time it takes for the sound to reach the listening position also varies. This difference in sound arrival times can distort the accurate spatial perception and sound image localization of multichannel sound.

[0290] Figure 9 illustrates the effect of the distance difference between the TV speakers (which typically serve as front speakers) and the rear speakers on audio synchronization. Figure 9 shows that the rear speakers are located further away from the listener than the TV speakers (e.g., 3 meters), causing sound from the rear speakers to arrive at the listening position later than sound from the TV speakers. Even though the system transmits audio signals simultaneously, the listener experiences a temporal mismatch due to the physical distance difference.

[0291] Figure 10 illustrates the effect of distance differences between rear speakers on synchronization. The left and right rear speakers may not be positioned exactly the same distance from the listener, and the figure assumes one rear speaker is slightly further away than the other (e.g., a distance difference of 3 meters). Even these slight distance differences between speakers can cause sound arrival times to vary, and especially in sophisticated surround sound environments, these errors can lead to distortion of the sound image.

[0292] To compensate for this audio synchronization error caused by distance differences, the system measures the distance between each speaker and the listening position or calculates the difference in sound arrival times based on a preset value. The system then performs compensation by intentionally delaying the audio signal output from speakers closer to the listening position by the calculated difference. In other words, by adjusting the timing of sound from all speakers to arrive at the listening position, the system ensures that even in multichannel systems with speakers in different locations, users experience a precise and consistent soundstage as designed.

[0293]

[0294] Figures 11 and 12 illustrate a method of measuring the delay time difference between speakers as a differentiation technique by measuring the user (remote control) microphone input time difference.

[0295] FIG. 11 illustrates an example of a structure in which each speaker outputs a different audio channel (1 kHz, 2 kHz, etc.) in a Bluetooth-based multi-channel audio environment proposed in the present disclosure, and a microphone device at the listener's position receives the same.

[0296] FIG. 11 illustrates a process in which multiple speaker devices output test tones of different audio frequencies (e.g., 1 kHz, 2 kHz, etc.) and a microphone device (e.g., a Bluetooth remote control or a smartphone) at a listener's position receives them in a Bluetooth-based multi-channel audio environment according to an embodiment of the present disclosure. As illustrated in FIG. 11, a source device such as a TV or a soundbar outputs audio signals to the front left speaker (CH1: FL) and the front right speaker (CH2: FR), while simultaneously, the rear left speaker (CH3: RL) and the rear right speaker (CH4: RR) output audio signals delayed by 2 ms and 4 ms, respectively, via Bluetooth-based wireless communication. A Bluetooth remote control (or smartphone) placed at the listener's position receives the test tones output from each speaker and can detect the difference in the arrival times of the audio signals for each channel. For example, CH1 and CH2 are received with a delay of 1 ms, CH3 with a delay of 2 ms, and CH4 with a delay of 4 ms. These measurements are utilized in a subsequent correction step.

[0297]

[0298] FIG. 12 illustrates an example of a process of measuring the arrival time difference of test sounds from each speaker received by a microphone device at a listener's position in a multi-channel audio system proposed in the present disclosure and transmitting the difference to a source device (TV).

[0299] FIG. 12 illustrates a series of steps in a multi-channel audio system according to one embodiment of the present disclosure, in which a microphone device at a listener's position measures the arrival time difference for a test tone of each speaker and transmits the measured value to a source device (TV or sound bar).

[0300] The delay time difference between speakers can be measured by the time difference between the user's (remote control's) microphone reception sound. When sound is requested simultaneously or at equal intervals from the source device and played through each speaker, the delay time for the audio coming from each speaker from the listener's perspective can be measured to determine the sync difference between speakers.

[0301] Step #0: Speakers can be distinguished by changing the audio frequency band (1kHz, 2kHz, 3kHz, etc.) (Example: Speaker 1: Transmits 1kHz sound, Speaker 2: Transmits 2kHz sound, Speaker 3: Transmits 3kHz sound, Speaker 4: Transmits 4kHz sound)

[0302] Step #1: The TV sends a test sound request to the speakers, and the speakers play the sound. In a multichannel environment, the source device sends test tones of different frequencies (e.g., CH1 = 1 kHz, CH2 = 2 kHz, CH3 = 3 kHz, CH4 = 4 kHz) to each speaker. These test signals can be transmitted simultaneously or sequentially.

[0303] 1) The source device requests simultaneous sound transmission to all speakers, and each speaker reproduces Hz-band sound.

[0304] 2) Send sequential Hz-band sound playback requests to each speaker at equal intervals (e.g. 10ms) between source devices.

[0305] Step #2: The listener's device (phone, remote control microphone) collects the delays that vary from speaker to speaker. A Bluetooth remote control or smartphone microphone placed at the listener's location receives test tones corresponding to each frequency band and measures the arrival time (delay) for each channel. For example, CH1 and CH2 are received with a 1ms delay, CH3 with a 2ms delay, and CH4 with a 4ms delay.

[0306] 1kHz sound (Speaker1) comes in the fastest.

[0307] The 2kHz sound (Speaker2) arrives 1ms later than the 1kHz sound.

[0308] 3kHz sound (Speaker3) arrives 2ms later than 1kHz.

[0309] 4kHz sound (Speaker3) arrives 4ms later than 1kHz.

[0310] Step #3: The listener device (phone, remote control microphone) transmits the sound delay difference for each speaker (by frequency band) to the source device (TV). The microphone device transmits the measured arrival time information to the source device.

[0311] 1kHz is 1ms, 2kHz is 1ms, 3kHz is 2ms, and 4kHz is 4ms.

[0312] Step #4: On the source device, 1) transmit audio by compensating for the delay difference between speakers or 2) synchronize all speakers by adding a compensation value to the audio playback delay (e.g., Presentation Delay) for each speaker. The source device sets the compensated audio playback delay (Presentation Delay) for each speaker channel based on the received delay information. For example, by adding an audio delay of 3ms to CH1 and CH2, 2ms to CH3, and 0ms to CH4, the audio output timing of all speakers is synchronized with respect to the listener.

[0313] Speaker 1: Transmit audio with 3ms delay

[0314] Speaker 2: Transmit audio with a 3ms delay

[0315] Speaker 3: Transmit audio with 1ms delay

[0316] Speaker 3: Transmit audio with 0ms delay

[0317]

[0318] The advantage of this method is that it doesn't require a wired or wireless connection between the source device and the speakers. Because the delay is measured from the listener's perspective, it can also be the most accurate method. The disadvantage of this method is that it requires excellent audio performance (microphone, measurement capabilities, etc.) to measure the listener's delay.

[0319] This method enables precise listener-centric time synchronization regardless of whether the speakers are wired or wireless, their location distance, or the network path.

[0320]

[0321] FIG. 13 illustrates an example of a structure in which a third device, such as a Bluetooth remote control or a smartphone, measures the time at which each speaker receives a test tone, and a source device adjusts the delay for each speaker based on this in a multi-channel audio system proposed in the present disclosure.

[0322] FIG. 13 is a sequence diagram illustrating a process in which a third device (Client), such as a Bluetooth remote control or a smartphone, measures the time at which each speaker receives a test tone, and a source device (Source device) individually corrects the delay of each speaker based on the measurement, in a multi-channel audio system according to one embodiment of the present disclosure.

[0323] CHMeasured Delay(ms)Audio Delay(ms)#113#213#321#440

[0324]

[0325] Table 2 shows, according to one embodiment of the present disclosure, the measured delay time (Measured Delay) for each speaker channel (CH1 to CH4) in a Bluetooth-based multi-channel audio system through a third device (e.g., a Bluetooth remote control or a smartphone microphone) at the listener's position, and the audio playback delay (Audio Delay) value set in the source device based on the measured delay time.

[0326] More specifically, each column in Table 2 represents:

[0327] CH (Channel Number): A channel number that distinguishes each audio output speaker, for example, CH1 may correspond to the front left (FL), CH2 to the front right (FR), CH3 to the rear left (RL), and CH4 to the rear right (RR) speaker.

[0328] Measured Delay (ms): The relative delay time measured from the arrival of the test tone for each channel at the listener's position. For example, CH1 and CH2 are measured as 1 ms, CH3 as 2 ms, and CH4 as 4 ms.

[0329] Audio Delay (ms): This is an additional audio presentation delay set by the source device for each channel to synchronize all speakers. Based on the latest arrival, CH4 (4ms), the compensation values ​​are set to 3ms for CH1 and CH2, 1ms for CH3, and 0ms for CH4.

[0330] In this way, the present disclosure measures the delay for each speaker based on the listener's position, and based on the delay information, the source device precisely adjusts the audio playback timing for each speaker, thereby enabling synchronization of the entire audio system.

[0331] The embodiment illustrated in Fig. 13 comprises the following steps:

[0332] (1) Transmitting Test Tone: The third device (Client) transmits LE Audio data packets containing test tones of different frequencies (e.g., 1kHz, 2kHz, 3kHz, 4kHz) to each speaker (Server: FL, FR, RL, RR) via the Soundbar. These test tones are transmitted by each speaker on its own assigned frequency.

[0333] (2) Test tone playback: Each speaker plays its assigned test tone. For example, FL sequentially outputs a test sound of 1 kHz, FR sequentially outputs a test sound of 2 kHz, RL sequentially outputs a test sound of 3 kHz, and RR sequentially outputs a test sound of 4 kHz.

[0334] (3) Test tone reception and delay measurement: The third device compares the timing of receiving test tones output from multiple speakers and detects the delay difference between the speaker arrival times. As a measurement example, FL and FR are received with a delay of 1 ms, RL with a delay of 2 ms, and RR with a delay of 4 ms.

[0335] (4) Transmitting delay compensation value: The third device transmits the measured delay information to the source device (e.g., TV or sound bar), and the source device sets an additional presentation delay for each speaker based on this.

[0336] (5) Setting the delay compensation value: For example, to synchronize based on the latest arriving RR channel (4 ms), additional delay values ​​of 3 ms are individually set for FL and FR, 1 ms for RL, and 0 ms for RR.

[0337] This method provides a technique for precisely synchronizing all audio channels based on the listener's position, regardless of whether the speakers are wired or wireless or the distance between them. It is particularly compatible with Bluetooth-based multichannel audio environments, and offers the advantage of allowing users to automatically experience the audio synchronization effect without the need for separate calibration.

[0338]

[0339] The present disclosure proposes a technology for precisely implementing audio synchronization (inter-speaker sync) between a TV (source device) and a Bluetooth speaker (sink device). As illustrated in the drawing, a conventional test involved connecting two TVs and Bluetooth (BT) speakers in a laboratory environment. However, the delay time of the BT speakers based on BR / EDR was large, and even when the Audio Delay was manually set to 0, a delay difference existed at the actual audio output time, resulting in an echo phenomenon.

[0340] Additionally, the sound pressure level (SPL) standard was not applied to the volume control, which resulted in an imbalance in the actual output volume. For example, at volume 1, the sound from the BT speaker was louder than the TV speaker, at volume 10, the sound levels between the TV and BT speakers were similar, and at volume 30, the sound from the TV speaker was louder than the BT speaker.

[0341] To solve these problems, the present disclosure proposes the following proposed technology (To-Be): that is, by fixing the built-in speaker delay value of the TV (TV Source Speaker Presentation Delay) to a certain degree and adjusting the delay value of the Bluetooth speaker accordingly, the audio output timing between the TV speaker and the Bluetooth speaker is synchronized.

[0342] More specifically, it is a method of synchronizing the audio output of all speaker channels to the listener by individually defining the presentation delay set to the TV's built-in speakers according to the actual timing of playback by Bluetooth speakers.

[0343] This method can also be effectively applied in a Bluetooth-based multi-channel audio environment, and even if the TV itself is unaware of the Bluetooth speaker delay, it can ensure the sound quality and precise synchronization of the entire system by adjusting the Bluetooth speaker output timing based on a fixed built-in speaker delay.

[0344]

[0345] FIG. 14 illustrates an example of a structure that independently defines the presentation delay of a built-in speaker in a source device (TV) and matches the total audio delay with Bluetooth speakers in a multi-channel audio system proposed in the present disclosure.

[0346] FIG. 14 illustrates an example of a structure for independently setting a presentation delay for a built-in speaker of a source device (e.g., a TV) and synchronizing audio output timing with external Bluetooth (BLE) speakers based on the presentation delay in a multi-channel audio system according to an embodiment of the present disclosure.

[0347] In traditional audio systems, the delay times of Bluetooth speakers differ from those of TV speakers, which can compromise audio consistency or cause echo effects from the listener's perspective. To address these issues, the present disclosure proposes a structure that sets the presentation delay (TV Source Speaker Presentation Delay) of the TV speakers to a reasonable level and synchronizes the audio delay with the total audio delay of the Bluetooth speakers.

[0348] More specifically, in Fig. 14, the total delay within the TV (TV Internal Delay) includes the following components:

[0349] PCM audio data output from the Multi-CH Audio Processor passes through the BUS delay, Preparation Delay, and Audio Renderer / Audio Processing process, and is finally output to the TV Speaker (CH L, CH R) with the TV Source Speaker Presentation Delay applied. At this time, the final output timing of the TV Speaker (TV Speaker Total Delay) is defined as follows:

[0350] TV Speaker Total Delay = Audio processing delay + TV Source Speaker Presentation Delay

[0351] Meanwhile, audio signals transmitted via Bluetooth path go through the following processes: Encoding & Audio Processing; Low Latency (LL) transmission and reception; Transport Latency; Decoding & Audio Processing; Audio Out after final Presentation Delay application

[0352] In total, Bluetooth Total System Delay is calculated as follows:

[0353] Bluetooth Total System Delay = Audio Processing Time + Transport Latency + Presentation Delay

[0354] At this time, the Total Audio System Delay is defined as follows:

[0355] Total Audio System Delay = BUS Delay + Preparation Delay + TV Speaker Total Delay = BUS Delay + Preparation Delay + Bluetooth Total System Delay

[0356] That is, the TV calculates the overall system delay based on the final audio output timing of the built-in speakers and Bluetooth speakers, and independently sets the TV speaker delay value to synchronize with the playback timing of the Bluetooth speakers, thereby configuring all channels to be precisely synchronized from the listener's perspective.

[0357] The phrase “Set individual presentation_delay of this TV speaker” in Figure 14 emphasizes that a presentation delay dedicated to the TV speaker can be individually set within the source device to align with the Bluetooth audio path.

[0358] This structure can cope with external speaker delay even in a Bluetooth-based audio environment, and can realize precise audio synchronization that does not impede the user experience even in a mixed wired / wireless environment.

[0359]

[0360] FIG. 15 illustrates a multi-channel audio transmission structure between a source device (such as a sound bar) and multiple wired / wireless speakers in a multi-channel audio system proposed in the present disclosure, and illustrates an example of a configuration in which each speaker is connected via various paths such as HDMI, Opt, and Bluetooth.

[0361] FIG. 15 is a diagram schematically illustrating an audio transmission structure of a multi-channel audio system according to one embodiment of the present disclosure.

[0362] As illustrated in Fig. 15, the audio source device receives various wired and wireless inputs such as HDMI, optical, and Bluetooth (BT) from external devices such as TVs and mobile phones, and processes multi-channel audio through a multi-channel audio processing device in the form of a sound bar. The multi-channel audio processing device supports multi-channel speaker outputs such as front left and right (Front L / R), center, and woofer, and transmits audio signals to multiple wired / wireless speakers (SPK).

[0363] A multichannel audio processing device transmits audio for each channel to multiple speakers via a multichannel path. These speakers can be connected to the soundbar via various methods, either wired or wireless (e.g., Bluetooth). Thus, Fig. 15 illustrates an example of a multichannel audio system with various input interfaces and a multichannel audio output structure.

[0364]

[0365] FIG. 16 illustrates an example of an audio output timing control structure for synchronized audio output between speakers in a multi-channel audio system proposed in the present disclosure.

[0366] FIG. 16 is a diagram illustrating an example of a timing control structure for audio output synchronization between a soundbar speaker and a BLE (Bluetooth Low Energy) speaker in a multi-channel audio system according to one embodiment of the present disclosure.

[0367] First, the soundbar contains an internal multi-channel audio processor (PCM), and internal bus delay and preparation delay occur based on the PCM data output point (Anchor Point for Audio Input). This is defined as the soundbar internal delay (SoundBar Internal Delay).

[0368] After that, the Audio Renderer performs audio processing, and the TV Source Speaker Presentation Delay is added. Each speaker in the soundbar (e.g. CH L, CH R) additionally has its own individual Presentation Delay, which sets the final output timing of the soundbar speakers (SoundBar Speaker Total Delay).

[0369] Meanwhile, the Bluetooth speaker receives audio signals through the Bluetooth audio path, which consists of the following delay elements: Audio Processing Time (Unicast Client / Audio Source / Broadcast Source), Transport Latency, Presentation Delay (Unicast Server / Audio Sink / Broadcast Sink).

[0370] The overall delay in a Bluetooth audio flow is defined as Bluetooth Total System Delay.

[0371] In this embodiment, the TV Source Speaker Presentation Delay is fixed, and the Delay of the soundbar speaker is additionally defined based on the Bluetooth Delay, so that the built-in speakers of the soundbar are synchronized with the playback timing of the Bluetooth speakers. This enables Inter Speaker Sync that considers the total delay time of each path after the Anchor Point in the entire audio system.

[0372] That is, the following formula applies:

[0373] TV Speaker Total Delay = Audio Processing Delay + TV Source Speaker Presentation Delay = Bluetooth Total System Delay

[0374] Total Audio System Delay = BUS Delay + Preparation Delay + TV Speaker Total Delay = BUS Delay + Preparation Delay + Bluetooth Total System Delay

[0375] Figure 16 visually organizes such time delay items to explain a method of adjusting and synchronizing audio output timing in a multi-channel audio system.

[0376]

[0377] [TV device claim information]

[0378] The embodiments described below are specifically described with reference to FIG. 17 in terms of the operation of a source device (TV, soundbar, etc.). The methods described below are distinguished for convenience of explanation, and it is understood that some components of one method may be substituted for some components of another method, or may be applied in combination with each other, as long as they are not mutually exclusive.

[0379] FIG. 17 illustrates an example of an operation process of a source device such as a TV or sound bar according to various embodiments of the present disclosure.

[0380] According to various embodiments of the present disclosure, a method is provided that is performed by a source device such as a TV or sound bar that supports a short-range communication system such as Bluetooth.

[0381] The source device includes a first processor corresponding to a host stack; a second processor corresponding to a first controller stack; a memory; and a transceiver. The host stack and the controller stack are connected via a Host Controller Interface (HCI).

[0382] At step S1701, the source device transmits a request for information on a second delay of the one or more external speaker devices to one or more external speaker devices.

[0383] In step S1702, the source device receives information about the second delay from the one or more external speaker devices.

[0384] At step S1703, the source device determines a first delay for the built-in speaker of the source device based on the second delay to synchronize audio output between the built-in speaker of the source device and the one or more external speaker devices.

[0385] In step S1704, the source device transmits the first audio data stream to the one or more external speaker devices for synchronized playback with the second audio data stream output from the built-in speaker based on the determined first delay.

[0386]

[0387] According to various embodiments of the present disclosure, the second delay information may include a transport latency and a presentation delay associated with the one or more external speaker devices.

[0388] According to various embodiments of the present disclosure, the first delay may be determined such that the first delay of the built-in speaker of the source device is synchronized with the second delay of the one or more external speaker devices.

[0389] According to various embodiments of the present disclosure, the first delay of the built-in speaker may include a conventional audio processing delay and an added source speaker presentation delay.

[0390] According to various embodiments of the present disclosure, the embodiment of FIG. 17 may further include the step of receiving, from a third device located at a listener position, information regarding a measured first arrival time for a first test tone output from the one or more external speaker devices. The determination of the first delay may additionally be based on the received first arrival time.

[0391] According to various embodiments of the present disclosure, the embodiment of FIG. 17 may further include: determining a delay difference based on the first arrival time and the second arrival time of the second test tone output from the built-in speaker to the third device; and transmitting a control signal for adjusting a presentation delay based on the delay difference to the one or more external speaker devices.

[0392] According to various embodiments of the present disclosure, the embodiment of FIG. 17 may further include: determining a second presentation delay for the one or more external speaker devices based on a fixed delay of the built-in speaker of the source device; and transmitting the second presentation delay to the one or more external speaker devices to adjust an audio output timing of the one or more external speaker devices.

[0393]

[0394] According to various embodiments of the present disclosure, an electronic device is provided. The source device includes a first processor corresponding to a host stack; a second processor corresponding to a first controller stack; a memory; and a transceiver. The host stack and the controller stack are connected via a Host Controller Interface (HCI). The memory may be configured to store instructions for performing an operating method of the source device according to FIG. 17 based on instructions executed by the first processor and the second processor.

[0395]

[0396] According to various embodiments of the present disclosure, a control device for controlling a source device is provided. The control device includes at least one processor and at least one memory operably connected to the at least one processor. The at least one memory may be configured to store instructions for performing an operating method of the source device according to FIG. 17 based on instructions executed by the at least one processor.

[0397]

[0398] According to various embodiments of the present disclosure, one or more non-transitory computer-readable media (CRM) storing one or more commands are provided. The one or more commands, when executed by one or more processors, perform operations, and the operations may include a method of operating a source device according to FIG. 17.

[0399]

[0400] [Speaker claim related explanation]

[0401] The embodiments described below are specifically described with reference to FIG. 18 in terms of speaker operation. The methods described below are distinguished for convenience of explanation, and it is understood that some components of one method may be substituted for or combined with some components of another method, as long as they are not mutually exclusive.

[0402] FIG. 18 illustrates an example of an operation process of a speaker according to various embodiments of the present disclosure.

[0403] According to various embodiments of the present disclosure, a method is provided that is performed by a speaker that supports a short-range communication system such as Bluetooth.

[0404] The speaker includes a third processor corresponding to a host stack; a fourth processor corresponding to a third controller stack; memory; and a transceiver. The host stack and the controller stack are connected via a Host Controller Interface (HCI).

[0405] At step S1801, the speaker receives a request for information on the second delay of the external speaker device from the source device.

[0406] At step S1802, the speaker transmits information about the second delay to the source device.

[0407] In step S1803, the speaker receives a first audio data stream from the source device for synchronized playback with a second audio data stream output from the built-in speaker based on a first delay for the built-in speaker of the source device. In order to synchronize audio output between the built-in speaker of the source device and the external speaker device, the first delay for the built-in speaker of the source device is based on the second delay.

[0408]

[0409] According to various embodiments of the present disclosure, the information of the second delay may include transport latency and presentation delay associated with the external speaker device.

[0410] According to various embodiments of the present disclosure, the first delay may be determined such that the first delay of the built-in speaker of the source device is synchronized with the second delay of the external speaker device.

[0411] According to various embodiments of the present disclosure, the first delay of the built-in speaker may include a conventional audio processing delay and an added source speaker presentation delay.

[0412] According to various embodiments of the present disclosure, the determination of the first delay may additionally be based on a first arrival time. The first arrival time may be measured from a third device located at the listener's position with respect to a first test tone output from the one or more external speaker devices.

[0413] According to various embodiments of the present disclosure, the embodiment of FIG. 18 may further include a step of receiving a control signal from the source device for adjusting a presentation delay based on a delay difference. The delay difference may be based on the first arrival time and a second arrival time of a second test tone output from the built-in speaker to the third device.

[0414] According to various embodiments of the present disclosure, the embodiment of FIG. 18 may further include a step of receiving a second presentation delay from the source device for adjusting an audio output timing of the external speaker device. The second presentation delay may be based on a fixed delay of the built-in speaker of the source device.

[0415]

[0416] According to various embodiments of the present disclosure, a speaker is provided. The speaker includes a third processor corresponding to a host stack; a fourth processor corresponding to a third controller stack; a memory; a transceiver; and a speaker device. The host stack and the controller stack are connected via a Host Controller Interface (HCI). The memory may be configured to store instructions for performing a method of operating a speaker according to FIG. 18 based on instructions executed by the third processor and the fourth processor.

[0417]

[0418] According to various embodiments of the present disclosure, a control device for controlling a speaker is provided. The control device includes at least one processor and at least one memory operably connected to the at least one processor. The at least one memory may be configured to store instructions for performing a method for operating a speaker according to FIG. 18 based on instructions executed by the at least one processor.

[0419]

[0420] According to various embodiments of the present disclosure, one or more non-transitory computer-readable media (CRM) storing one or more instructions are provided. The one or more instructions, when executed by one or more processors, perform operations, and the operations may include a method of operating a speaker according to FIG. 18.

[0421]

[0422] The claims described in the various embodiments of the present disclosure may be combined in various ways. For example, the technical features of the method claims of the various embodiments of the present disclosure may be combined and implemented as a device, and the technical features of the device claims of the various embodiments of the present disclosure may be combined and implemented as a method. Furthermore, the technical features of the method claims of the various embodiments of the present disclosure may be combined and implemented as a device, and the technical features of the method claims of the various embodiments of the present disclosure may be combined and implemented as a method.

Claims

1. In a method of operating a source device in a short-range wireless communication system, A step of transmitting a request for information on a second delay of the one or more external speaker devices to one or more external speaker devices; A step of receiving information of the second delay from the one or more external speaker devices; determining a first delay for the built-in speaker of the source device based on the second delay to synchronize audio output between the built-in speaker of the source device and the one or more external speaker devices; and A step of transmitting a first audio data stream to the one or more external speaker devices for synchronized playback with a second audio data stream output from the built-in speaker based on the determined first delay, method.

2. In paragraph 1, The information of the second delay includes a transport latency and a presentation delay associated with the one or more external speaker devices. method.

3. In paragraph 1, The first delay is determined so that the first delay of the built-in speaker of the source device is synchronized with the second delay of the one or more external speaker devices. method.

4. In paragraph 3, The first delay of the built-in speaker includes the existing audio processing delay and the added source speaker presentation delay. method.

5. In paragraph 1, Further comprising the step of receiving information of a first arrival time measured for a first test tone output from said one or more external speaker devices from a third device at a listener position, The determination of the above first delay is additionally based on the received first arrival time, method.

6. In paragraph 5, A step of determining a delay difference based on the first arrival time and the second arrival time for the third device of the second test tone output from the built-in speaker; and Further comprising the step of transmitting a control signal for adjusting a presentation delay based on the delay difference to the one or more external speaker devices. method.

7. In paragraph 1, determining a second presentation delay for the one or more external speaker devices based on a fixed delay of the built-in speaker of the source device; and Further comprising the step of transmitting the second presentation delay to the one or more external speaker devices to adjust audio output timing of the one or more external speaker devices. method.

8. In a method for operating an external speaker device in a short-range wireless communication system, A step of receiving a request for information on a second delay of the external speaker device from a source device; A step of transmitting information of the second delay to the source device; A step of receiving a first audio data stream from the source device for synchronized playback with a second audio data stream output from the built-in speaker based on a first delay for the built-in speaker of the source device, In order to synchronize audio output between the built-in speaker of the source device and the external speaker device, the first delay for the built-in speaker of the source device is based on the second delay. method.

9. In paragraph 8, The information of the second delay includes a transport latency and a presentation delay associated with the external speaker device. method.

10. In paragraph 8, The first delay is determined so that the first delay of the built-in speaker of the source device is synchronized with the second delay of the external speaker device. method.

11. In paragraph 10, The first delay of the built-in speaker includes the existing audio processing delay and the added source speaker presentation delay. method.

12. In paragraph 8, The determination of the above first delay is additionally based on the first arrival time, The first arrival time is measured from a third device at the listener position for the first test tone output from the one or more external speaker devices. method.

13. In paragraph 12, Further comprising the step of receiving a control signal for adjusting the presentation delay based on the delay difference from the source device, The delay difference is based on the first arrival time and the second arrival time of the second test tone output from the built-in speaker to the third device. method.

14. In paragraph 8, Further comprising the step of receiving a second presentation delay from the source device for adjusting the audio output timing of the external speaker device, The second presentation delay is based on a fixed delay of the built-in speaker of the source device. method.

15. In a short-range wireless communication system, in a source device, A first processor corresponding to a host stack; a second processor corresponding to a controller stack; a memory; an input device corresponding to a user interface (UI); an output device corresponding to the UI; and a transceiver. The above host stack and the above controller stack are connected through HCI (Host Controller Interface), The above memory stores instructions for performing operations based on what is executed by the first processor and the second processor, The above actions are, A step of requesting information on a second delay of one or more external speaker devices from one or more external speaker devices; A step of receiving information of the second delay from the one or more external speaker devices; determining a first delay for the built-in speaker of the source device based on the second delay to synchronize audio output between the built-in speaker of the source device and the one or more external speaker devices; and A step of transmitting a first audio data stream to the one or more external speaker devices for synchronized playback with a second audio data stream output from the built-in speaker based on the determined first delay, Source device.

16. In paragraph 15, The information of the second delay includes a transport latency and a presentation delay associated with the one or more external speaker devices. Source device.

17. In paragraph 15, The first delay is determined so that the first delay of the built-in speaker of the source device is synchronized with the second delay of the one or more external speaker devices. Source device.

18. In paragraph 17, The first delay of the built-in speaker includes the existing audio processing delay and the added source speaker presentation delay. Source device.

19. In paragraph 1, The above operations further include: receiving, from a third device at a listener position, information about a first arrival time measured for a first test tone output from the one or more external speaker devices; The determination of the above first delay is additionally based on the received first arrival time, Source device.

20. In paragraph 19, The above actions are: A step of determining a delay difference based on the first arrival time and the second arrival time for the third device of the second test tone output from the built-in speaker; and Further comprising the step of transmitting a control signal for adjusting a presentation delay based on the delay difference to the one or more external speaker devices. Source device.

Images