A method of audio enhancement and corresponding apparatus

Abstract

This invention provides a method for audio enhancement, headphones, a headphone system, and a user equipment. The method includes: using the user equipment to determine a first predetermined synchronization delay and notifying a first headphone of the first predetermined synchronization delay; using the user equipment to determine a second predetermined synchronization delay and notifying a second headphone of the second predetermined synchronization delay; using the user equipment to receive first uplink audio data from the first headphone and second uplink audio data from the second headphone, wherein the timing of the first uplink audio data and the timing of the second uplink audio data are controlled by a first digital signal processing circuit and a second digital signal processing circuit, respectively, based on a first reference point for the first headphone and the second headphone.

Description

Technical Field

[0001] This invention relates to audio control, and more specifically, to a method and related apparatus for audio enhancement using timing control, such as audio digital signal processing (DSP) circuits, headphones, and user equipment (UE). Background Technology

[0002] According to relevant technologies, some wireless headphones can be implemented in very small sizes and can be referred to as wireless earbuds. Wireless headphone manufacturers may attempt to implement more functions in small wireless headphones (such as wireless earbuds) with built-in microphones. However, some problems may arise. For example, the hardware resources in small wireless headphones may be very limited. In addition, more computation corresponding to more functions may lead to more power consumption. Therefore, small wireless headphones may need to be charged frequently. One or more conventional methods can be proposed to try to solve these problems, but may lead to additional problems, such as some side effects. Therefore, a novel method and related architecture that does not introduce side effects or is unlikely to introduce side effects is needed to enhance the audio control of electronic systems. Summary of the Invention

[0003] This invention provides a method for audio enhancement, headphones, headphone systems, and user equipment.

[0004] In one embodiment, the present invention provides an audio enhancement method for a user equipment wirelessly connected to a first headset and a second headset, the method comprising: using the user equipment to determine a first predetermined synchronization delay and notifying the first predetermined synchronization delay to the first headset, so that a first digital signal processing circuit in the first headset determines a synchronization point for the first headset based on a first time point of a first event and the first predetermined synchronization delay; using the user equipment to determine a second predetermined synchronization delay and notifying the second predetermined synchronization delay to the second headset, so that a second digital signal processing circuit in the second headset determines the synchronization point for the second headset based on a second time point of a second event and the second predetermined synchronization delay; using the user equipment to receive first uplink audio data from the first headset and second uplink audio data from the second headset, wherein the timing of the first uplink audio data and the timing of the second uplink audio data are controlled by the first digital signal processing circuit and the second digital signal processing circuit respectively based on a first reference point for the first headset and the second headset, wherein the first reference point is determined at least based on the synchronization point.

[0005] In another embodiment, the present invention provides an earphone wirelessly connected to a user equipment, characterized in that the earphone includes: a wireless communication interface circuit configured to enable the earphone to perform wireless communication with the user equipment; an audio input device configured to input an audio wave to generate input audio data; an audio output device configured to output an audio wave according to the output audio data; and a first digital signal processing circuit coupled to the wireless communication interface circuit, the audio input device, and the audio output circuit, configured to perform signal processing on either the input audio data and the output audio data of the earphone; wherein the first digital signal processing circuit determines a synchronization point for the earphone based on a first time point of a first event and a first predetermined synchronization delay, wherein the first predetermined synchronization delay is determined by the user equipment; wherein the first digital signal processing determines a first reference point for the first earphone based on the synchronization point and a second predetermined synchronization delay, wherein the second predetermined synchronization delay is greater than the first predetermined synchronization delay; and wherein the first digital signal processing circuit controls the timing of a first uplink audio data from the earphone to the user equipment based on the first reference point for the earphone.

[0006] In another embodiment, the present invention provides an earphone system including a first earphone and a second earphone, wherein the first earphone and the first and second earphones are wirelessly connected to a user device, wherein the first earphone includes: a first wireless communication interface circuit configured to enable the first earphone to perform wireless communication with the user device; a first audio input device configured to input an audio wave to generate input audio data; a first audio output device configured to output an audio wave according to the output audio data; and a first digital signal processing circuit coupled to the first wireless communication interface circuit, the first audio input device, and the first audio output circuit, configured to perform signal processing on either the input audio data and the output audio data of the first earphone; wherein the second earphone includes: a second wireless communication interface circuit configured to enable the second earphone to perform wireless communication with the user device; a second audio input device configured to input an audio wave to generate input audio data; a second audio output device configured to output an audio wave according to the output audio data; and a second digital signal processing circuit coupled to the second wireless communication interface circuit, the second audio input device, and the second audio output circuit, configured to perform signal processing on either the input audio data and the output audio data of the first earphone ... The device is configured to perform signal processing on either the input audio data or the output audio data of the second headset; wherein the first digital signal processing circuit determines a synchronization point for the first headset based on a first time point of a first event and a first predetermined synchronization delay, wherein the first predetermined synchronization delay is determined by the user equipment; wherein the second digital signal processing circuit determines the synchronization point for the second headset based on a second time point of a second event and a second predetermined synchronization delay, wherein the second predetermined synchronization delay is determined by the user equipment; wherein the first digital signal processing circuit determines a first reference point for the first headset based on the synchronization point and a third predetermined synchronization delay; wherein the second digital signal processing circuit determines the first reference point for the second headset based on the synchronization point and the third predetermined synchronization delay, wherein the third predetermined synchronization delay is greater than the first predetermined synchronization delay and the second predetermined synchronization delay; and wherein the first digital signal processing circuit and the second digital signal processing circuit control the timing of a first uplink audio data from the first headset to the user equipment and the timing of a second uplink audio data from the second headset to the user equipment, respectively, based on the first reference points for the first headset and the second headset.

[0007] In another embodiment, the present invention provides a user equipment wirelessly connected to a first earpiece and a second earpiece, characterized in that it includes: a wireless communication interface circuit configured to enable the user equipment to wirelessly communicate with the first earpiece and the second earpiece; and an audio digital signal processing circuit coupled to the wireless communication interface circuit and configured to perform signal processing for the user equipment; wherein the audio digital signal processing circuit is configured to: determine a first predetermined synchronization delay and notify the first earpiece of the first predetermined synchronization delay, so that a first digital signal processing circuit in the first earpiece determines a synchronization point for the first earpiece based on a first time point of a first event and the first predetermined synchronization delay; A second predetermined synchronization delay is determined and notified to the second earpiece, so that the second digital signal processing circuit in the second earpiece determines the synchronization point for the second earpiece based on the second time point of the second event and the second predetermined synchronization delay; first uplink audio data is received from the first earpiece and second uplink audio data is received from the second earpiece, wherein the timing of the first uplink audio data and the timing of the second uplink audio data are controlled by the first digital signal processing circuit and the second digital signal processing circuit respectively based on a first reference point for the first earpiece and the second earpiece, wherein the first reference point is determined at least based on the synchronization point.

[0008] In summary, the embodiments provided by this invention can achieve audio enhancement. Attached Figure Description

[0009] Figure 1 This is an icon of an electronic system according to an embodiment of the present invention.

[0010] Figure 2 An audio processing control scheme for performing audio enhancement by means of timing control is illustrated in an embodiment of the present invention.

[0011] Figure 3 The present invention illustrates a frame-based error prevention and control scheme for a method of performing audio enhancement by means of timing control.

[0012] Figure 4 An embodiment of the present invention is illustrated. Figure 3 The following are some implementation details of the frame-based error prevention and control scheme.

[0013] Figure 5 An example of a frame-based error that may occur when headphone audio scheduling controls are temporarily disabled is shown.

[0014] Figure 6According to embodiments of the present invention, a sample-based error prevention and control scheme is shown for a method of audio enhancement using timing control.

[0015] Figure 7 According to embodiments of the present invention, a UE-schedule-aware headset timing control scheme is shown as a method for performing audio enhancement by means of timing control.

[0016] Figure 8 According to embodiments of the present invention, a noise cancellation control scheme for a method of audio enhancement using timing control is shown.

[0017] Figure 9 The present invention illustrates the workflow of a method for audio enhancement using timing control.

[0018] Figure 10 According to another embodiment of the present invention, the workflow of a method for audio enhancement by means of timing control is shown. Detailed Implementation

[0019] Certain terms are used in the specification and claims to refer to specific components. Those skilled in the art will understand that hardware manufacturers may use different names to refer to the same component. This specification and claims do not distinguish components based on differences in name, but rather on differences in function. The terms "comprising" and "including" used throughout the specification and claims are open-ended and should be interpreted as "comprising but not limited to." "Substantially" means that, within an acceptable margin of error, those skilled in the art can solve the technical problem and substantially achieve the technical effect within a certain margin of error. Furthermore, the term "coupled" here includes any direct and indirect electrical connection means. Therefore, if a first device is described as coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means. The following description is a preferred mode for carrying out the invention and is intended to illustrate the spirit of the invention rather than to limit the scope of protection of the invention. The scope of protection of the invention shall be determined by the claims.

[0020] The following description represents the preferred embodiments of the present invention. These descriptions are intended to illustrate the general principles of the invention and not to limit it. The scope of protection of the invention should be determined based on the claims.

[0021] Figure 1This is an icon representing an electronic system according to an embodiment of the present invention, wherein the electronic system may include a UE 100, a first earpiece such as earpiece 160, and a second earpiece such as earpiece 180, earpieces 160 and 180 being wirelessly connected to the UE 100. For ease of understanding, the UE 100 may be implemented as a multi-functional mobile phone, and earpieces 160 and 180 may be implemented as small wireless earpieces (e.g., wireless earbuds) with built-in microphones, but the invention is not limited thereto. The electronic system is configured to offload some processing from earpieces 160 and 180 to the UE 100 without degrading the performance of some processing.

[0022] exist Figure 1 In the illustrated architecture, UE 100 may include at least one processor (e.g., one or more processors) such as an Application Processor (AP) 101, a modem 102 coupled to AP 101, and one or more antennas coupled to modem 102. AP 101 may include a Micro Control Unit (MCU) such as AP MCU 110, audio DSP circuitry 120, and wireless communication interface circuitry such as Bluetooth (BT) interface (IF) circuitry 130. Audio DSP circuitry 120 may include independent processing circuitry 121 and dependent processing circuitry 122 (labeled "independent processing circuitry" and "dependent processing circuitry," respectively). The MCU such as AP MCU 110, audio DSP circuitry 120, and wireless communication interface circuitry such as BT IF circuitry 130 may be coupled to each other via internal connections within AP 101.

[0023] Furthermore, the first earphone, such as earphone 160, may include a first DSP circuit, such as audio DSP circuit 162, an audio input device 168, an audio output device 169, and a wireless communication interface circuit, such as BTIF circuit 170. Figure 1As shown in the upper right corner, audio input device 168, audio output device 169, and wireless communication interface circuitry such as BT IF circuitry 170 may be coupled to a first DSP circuit such as audio DSP circuitry 162. For example, the first DSP circuitry such as audio DSP circuitry 162 may include multiple sub-circuits, such as a Low Complexity Communications Codec (LC3) processing circuitry 164 and a first part-one processing circuitry 166 (labeled "LC3" and "Part 1" respectively for simplicity). Audio input device 168 may include at least one microphone (e.g., one or more microphones), and audio output device 169 may include at least one speaker (e.g., one or more speakers).

[0024] Furthermore, the second headset, such as headset 180, may include a second DSP circuit, such as audio DSP circuit 182, an audio input device 188, an audio output device 189, and a wireless communication interface circuit, such as BTIF circuit 190. Figure 1 As shown in the lower right corner, audio input device 188, audio output device 189, and wireless communication interface circuitry such as BT IF circuitry 190 may be coupled to a second DSP circuit such as audio DSP circuitry 182. For example, the second DSP circuitry such as audio DSP circuitry 182 may include multiple sub-circuits, such as LC3 processing circuitry 184 and first-part processing circuitry 186 (labeled "LC3" and "part 1" respectively for simplicity), audio input device 188 may include at least one microphone (e.g., one or more microphones), and audio output device 189 may include at least one speaker (e.g., one or more speakers).

[0025] AP 101 can be used to control the operation of UE 100, and modem 102 enables UE 100 to perform wireless communication with at least one network (e.g., one or more networks) to allow UE 100 to communicate with another UE through the aforementioned at least one network. For example, a user of UE 100 and a user of another UE can be regarded as a near-end user and a remote user, respectively, and the two users can use UE 100 and the other UE to make calls. In addition, the APMCU 110 runs multiple program modules (e.g., one or more audio drivers 111 and one or more speech drivers 112) that can be used to control the AP 101 to control the operation of the UE 100. The audio DSP circuit 120 can be used to perform signal processing such as audio data processing. Wireless communication interface circuits such as the BT IF circuit 130 can be used to communicate wirelessly with the headset 160 (e.g., the BT IF circuit 170 therein) and the headset 180 (e.g., the BT IF circuit 190 therein) to allow the user of the UE 100 to maintain a hands-free conversation with a remote user with the help of the headsets 160 and 180.

[0026] In a first headset such as headset 160, a wireless communication interface circuit such as BT IF circuit 170 enables headset 160 to perform wireless communication with UE 100 (e.g., BT IF circuit 130 therein). Audio input device 168 can be used to input audio waves to generate input audio data for headset 160. As an example, the input audio data for headset 160 may represent an initial version (e.g., raw audio data) corresponding to uplink audio data for headset 160, which UE 100 can transmit to other UEs via at least one of the aforementioned networks. Furthermore, audio output device 169 can be used to output audio waves based on the output audio data for headset 160. For example, UE 100 can receive downlink audio data corresponding to headset 160 from other UEs through at least one network as described above, and the output audio data of headset 160 can represent a preprocessed version of the downlink audio data corresponding to headset 160. UE 100 can preprocess the downlink audio data corresponding to headset 160 to generate its preprocessed result and send it to headset 160 as a preprocessed version of the downlink audio data, but the invention is not limited thereto. As another example, the output audio data of headset 160 can represent a forwarded version of the downlink audio data corresponding to headset 160. Furthermore, a first DSP circuit, such as audio DSP circuit 162, can be used to perform signal processing, such as audio data processing, on either the input audio data or the output audio data of headset 160.

[0027] Furthermore, both audio DSP circuit 120 and audio DSP circuit 162 (e.g., LC3 processing circuit 164 therein) can perform LC3 processing, such as decoding and encoding of LC3 frames, to allow BTIF circuit 130 in UE 100 and BTIF circuit 170 in headset 160 to communicate with each other via LC3 frames. Regarding the first uplink audio data path of the near-end user-side electronic system (e.g., a data path starting at headset 160, passing through UE 100, and reaching at least one of the aforementioned networks), audio DSP circuit 162 can utilize first-part processing circuit 166 to perform first-part processing, such as first audio data processing, on input audio data (e.g., raw audio data) from audio input device 168 to produce a first processing result as an intermediate version of the uplink audio data corresponding to headset 160, and utilize BTIF circuit 170 to transmit the intermediate version of the uplink audio data corresponding to headset 160 to UE 100 via one or more LC3 frames. UE 100 may use the second part of the processing circuit 126 (referred to as "part 2" for brevity) in the dependent processing circuit 122 of the audio DSP circuit 120 to perform second part processing (e.g., second audio data processing) on ​​an intermediate version of the uplink audio data corresponding to the headset 160 to produce a second processing result as the uplink audio data corresponding to the headset 160 (e.g., its processed version), and use the modem 102 to transmit the uplink audio data corresponding to the headset 160 to another UE through the aforementioned at least one network. Regarding the first downlink audio path of the near-end user-side electronic system (e.g., the data path from the aforementioned at least one network through UE 100 to the headset 160), UE 100 may use the audio DSP circuit 120 to perform signal processing such as LC3 processing on the downlink audio data corresponding to the headset 160, and use the BT IT circuit 130 to transmit the downlink audio data corresponding to the headset 160 to the headset 160 via one or more LC3 frames. The headphones 160 can use the audio DSP circuit 162 to perform signal processing such as volume adjustment and audio quality adjustment on the downlink audio data corresponding to the headphones 160 to generate output audio data of the headphones 160 for playback by the audio output device 169.

[0028] In a second headset, such as headset 180, a wireless communication interface circuit, such as BT IF circuit 190, enables headset 180 to perform wireless communication with UE 100 (e.g., BT IF circuit 130 therein). Audio input device 188 can be used to input audio waves to generate input audio data for headset 180. For example, the input audio data for headset 180 may represent an initial version (e.g., raw audio data) of uplink audio data corresponding to headset 180, which UE 100 can transmit to other UEs via at least one of the aforementioned networks. Furthermore, audio output device 189 can be used to output audio waves based on the output audio data of headset 180. For example, UE 100 can receive downlink audio data corresponding to headset 180 from other UEs through at least one of the networks described above, and the output audio data of headset 180 can represent a preprocessed version of the downlink audio data corresponding to headset 180. UE 100 can preprocess the downlink audio data corresponding to headset 180 to generate its preprocessed result and send it to headset 180 as a preprocessed version of the downlink audio data, but the invention is not limited thereto. As another example, the output audio data of headset 180 can represent a forwarded version of the downlink audio data corresponding to headset 180. Furthermore, a second DSP circuit, such as audio DSP circuit 182, can be used to perform signal processing, such as audio data processing, on either the input audio data or the output audio data of headset 180.

[0029] Furthermore, both audio DSP circuit 120 and audio DSP circuit 182 (e.g., LC3 processing circuit 184 therein) can perform LC3 processing, such as decoding and encoding of LC3 frames, to allow BT IF circuit 130 in UE 100 and BT IF circuit 190 in headset 180 to communicate with each other via LC3 frames. Regarding the second uplink audio data path of the near-end user-side electronic system (e.g., a data path starting at headset 180, passing through UE 100, and reaching at least one of the aforementioned networks), audio DSP circuit 182 can utilize first-part processing circuit 186 to perform first-part processing, such as first audio data processing, on the input audio data (e.g., raw audio data) from audio input device 188 to produce a first processing result as an intermediate version of the uplink audio data corresponding to headset 180, and utilize BT IF circuit 190 to transmit the intermediate version of the uplink audio data corresponding to headset 180 to UE 100 via one or more LC3 frames. UE 100 may use the second part of the processing circuit 126 (referred to as "part 2" for brevity) in the dependent processing circuit 122 of the audio DSP circuit 120 to perform second part processing (e.g., second audio data processing) on ​​an intermediate version of the uplink audio data corresponding to the headset 180 to produce a second processing result as the uplink audio data corresponding to the headset 180 (e.g., its processed version), and transmit the uplink audio data corresponding to the headset 180 to another UE via the at least one of the aforementioned networks using the modem 102. Regarding the second downlink audio path of the near-end user-side electronic system (e.g., the data path from the at least one aforementioned network through UE 100 to the headset 180), UE 100 may use the audio DSP circuit 120 to perform signal processing such as LC3 processing on the downlink audio data corresponding to the headset 180, and use the BT IT circuit 130 to transmit the downlink audio data corresponding to the headset 180 to the headset 180 via one or more LC3 frames. The headphones 180 can use the audio DSP circuit 182 to perform signal processing such as volume adjustment and audio quality adjustment on the downlink audio data corresponding to the headphones 180 to generate output audio data of the headphones 180 for playback by the audio output device 189.

[0030] based on Figure 1The architecture shown describes an electronic system configured to offload some processing of headphones 160 and 180 to the UE 100 without reducing the performance of some processing. Furthermore, a second processing circuit 126 in the dependent processing circuit 122 of the audio DSP circuit 120 performs a second processing, such as second audio data processing, for headphones 160 and 180 without reducing the performance of the second audio data processing. This second processing can be an example of the aforementioned partial processing, but the invention is not limited thereto. According to some embodiments of the invention, the UE 100 can enable the independent processing circuit 121 and disable the dependent processing circuit 122, and utilize the independent processing circuit 121 to perform a first processing (e.g., first audio data processing) and a second processing (e.g., second audio data processing) for any one of multiple wireless headphones. For example, the first set of wireless earphones in a plurality of wireless earphones can be implemented with sub-circuits that are the same as or similar to those of earphones 160 and 180, respectively, and the UE 100 can be arranged to disable the first portion of the processing circuitry 166 and 186 of the first set of wireless earphones. In another example, the second set of wireless earphones in a plurality of wireless earphones can be implemented with sub-circuits that are similar to those of earphones 160 and 180, respectively, but it is not necessary to implement the first portion of the processing circuitry 166 and 186 in the second set of wireless earphones.

[0031] According to some embodiments, the communication system may include the electronic system of the present invention and at least one of the aforementioned networks, and the communication system may further include other UEs, which may be implemented in the same or similar manner as UE 100. In the communication system, the uplink audio data of UE 100 can be used as the downlink audio data of other UEs, and the uplink audio data of other UEs can be used as the downlink audio data of UE 100. Furthermore, the first and second uplink audio data paths of UE 100 can respectively point to the first and second downlink audio data paths of other UEs, and the first and second uplink audio data paths of other UEs can respectively point to the first and second downlink audio data paths of UE 100. For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0032] According to some embodiments, BT IF circuits 130, 170, and 190 may conform to the BT specification. More specifically, BT IF circuits 130, 170, and 190 may be implemented according to Bluetooth Low Energy (BLE) technology to make UE100, headsets 160, and 180 BLE compatible. For example, headsets 160 and 180 may be implemented as BLE headsets. For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0033] Figure 2 An audio processing control scheme for performing audio enhancement using timing control is illustrated in an embodiment of the present invention. This method can be used... Figure 1 The illustrated electronic system, further, allows this method to be used with UE 100 and a first and second headset (e.g., headsets 160 and 180) wirelessly connected to UE 100. According to the audio processing control scheme of the invention, the first processing circuits 166 and 186 can be implemented as acoustic echo cancellation (AEC) processing circuits 266 and 286 (denoted as "AEC" for simplicity), respectively, for performing AEC processing, and the second processing circuit 126 can be implemented as a noise reduction (NR) processing circuit 226 (denoted as "NR" for simplicity) for performing NR processing. The associated numerals can be changed accordingly in response to architectural variations. For example, UE100, AP 101, audio DSP circuits 120, 162 and 182, dependency processing circuit 122, and headphones 160 and 180 can be replaced by UE 200, AP 201, audio DSP circuits 220, 262 and 282, dependency processing circuit 222, and headphones 260 and 280, respectively. For the sake of brevity, similar descriptions to other embodiments are not repeated here.

[0034] exist Figure 2 In the architecture, the first part of the processing (e.g., the first audio data processing) can be implemented through AEC processing, while the second part of the processing (e.g., the second audio data processing) can be implemented through NR processing, but the invention is not limited thereto. According to some embodiments, the first part of the processing (e.g., the first audio data processing) and / or the second part of the processing (e.g., the second audio data processing) can be changed.

[0035] Figure 3According to embodiments of the present invention, a frame-based error prevention control scheme is illustrated for a method of performing audio enhancement by means of timing control. For ease of understanding, a multi-function mobile phone UE 300 can be used as an example of UE 100, and the headset 360 with microphone #1 (hereinafter referred to as "Mic#1") and the headset 380 with microphone #2 (hereinafter referred to as "Mic#2") embedded in UE 300 can be used as examples of headset 160 with audio input device 168 and headset 180 with audio input device 188 embedded in UE 100, respectively. For BT communication between UE 300 and headsets 360 and 380, UE 300 can be considered as the master side (e.g., BT master device), while headsets 360 and 380 can be considered as the slave side (e.g., BT slave device).

[0036] like Figure 3 As shown, UE 100, such as UE 300, can perform appropriate timing control on multiple time slots, and headsets 160 and 180, such as headsets 360 and 380, can each perform their own timing control with the assistance of UE 100, such as UE 300. The horizontal axis can represent time, and the multiple time slots can be divided into units of predetermined time lengths, such as 10 milliseconds (ms), but the invention is not limited thereto. For example, the predetermined time length can vary, and more specifically, can be equal to any one of one or more other predetermined values. Furthermore, an uplink audio data sequence {M1}, such as uplink audio data M1(1), M1(2), etc., can serve as an example of uplink audio data corresponding to headset 160 (e.g., headset 360). An uplink audio data sequence {M2}, such as uplink audio data M2(1), M2(2), etc., can serve as an example of uplink audio data corresponding to headset 180 (e.g., headset 380). For ease of understanding, an uplink audio data sequence group {M1(i), M2(i)} with the same index i can represent uplink audio data sampled at the same time. The index i can be an integer, but the present invention is not limited thereto.

[0037] In the first time slot of a plurality of time slots, for example, the time slot starting from a first reference point (e.g., a Connected Isochronous Group (CIG) reference point), BT IF circuit 130 (e.g., the BT IF circuit of UE 300, denoted as "Bluetooth" for brevity) can receive uplink audio data M1(1) from headset 160 (e.g., headset 360) and uplink audio data M2(1) from headset 180 (e.g., headset 380) for transmission to audio DSP circuit 120 (e.g., the audio DSP circuit of UE 300, denoted as "audio DSP" for brevity) at the beginning of the next time slot (e.g., the second time slot of a plurality of time slots), BT IF circuit 130 (e.g., the BT IF circuit of UE 300, denoted as "Bluetooth" for brevity) and BT IF circuit 130 (e.g., the BT IF circuit of UE 300) at the beginning of the next time slot (e.g., the second time slot of a plurality of time slots). The IF circuit (labeled "Bluetooth" for brevity) can receive uplink audio data M1(2) from headset 160 (e.g., headset 360) and uplink audio data M2(2) from headset 180 (e.g., headset 380) for transmission to audio DSP circuit 120 (e.g., audio DSP circuit of UE 300, labeled "audio DSP" for brevity) at the start of the next time slot (e.g., the third time slot in a plurality of time slots); the rest can be deduced by analogy.

[0038] Under the control of UE 100, such as UE 300, headsets 160 and 180, such as headsets 360 and 380, can know at least a portion (e.g., part or all) of the scheduling on the master device side, such as UE scheduling. More specifically, headsets 160 and 180, such as headsets 360 and 380, can convert the downlink timing reference to the uplink timing reference to correctly perform the relevant timing control on the slave device side. Finally, the uplink audio data M1(1) and M2(1) corresponding to the first moment (e.g., the first sampling period) can be prepared in time on the slave device side, for example, before the start time of the Connected Isochronous Stream (CIS) event of the uplink audio data M1(1) and M2(1), respectively. Furthermore, the uplink audio data M1(1) and M2(1) can be sent from the slave device side to the master device side in the same time slot, so as to allow the audio DSP circuit 120 (e.g., the audio DSP circuit of UE 300, referred to as "audio DSP" for brevity) to perform audio data processing on the uplink audio data M1(1) and M2(1) corresponding to the same time (e.g., the same sampling time period); the uplink audio data M1(2) and M2(2) corresponding to the second time (e.g., the second sampling time period) can be prepared in a timely manner on the slave device side, for example, before the start time of the CIS event of the uplink audio data M1(2) and M2(2) respectively. Furthermore, uplink audio data M1(2) and M2(2) can be transmitted from the slave device side to the master device side in the same time slot, allowing the audio DSP circuit 120 (e.g., the audio DSP circuit of UE 300, referred to as "audio DSP" for brevity) to perform audio data processing on uplink audio data M1(1) and M2(1) corresponding to the same time (e.g., the same sampling time period); the rest can be deduced by analogy. Therefore, the method and related apparatus of the present invention can prevent any frame-based errors from occurring, and more specifically, can ensure that there is no difference between the various transmission times of the respective uplink audio data M1(i) and M2(i) of Mic#1 and Mic#2 (e.g., uplink audio data M1(1) and M2(1)). For the sake of brevity, similar descriptions of this embodiment and other embodiments will not be repeated here.

[0039] According to some embodiments, uplink audio data M1(i) and M2(i) (e.g., uplink audio data M1(1) and M2(1)) may represent a set of stereo audio data corresponding to the same moment (e.g., the same sampling time period). For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0040] According to some embodiments, a first earphone such as earphone 160 (e.g., earphone 360) and a second earphone such as earphone 180 (e.g., earphone 380) may be referred to as earphone #1 and #2, respectively, and an audio DSP circuit 120 (e.g., the audio DSP circuit in UE 300), a first DSP circuit such as audio DSP circuit 162 (e.g., the audio DSP circuit in earphone 360), and a second DSP circuit such as audio DSP circuit 182 (e.g., the audio DSP circuit in earphone 380) may be referred to as DSP circuit #0, #1, and #2, respectively.

[0041] Figure 4 An embodiment of the present invention is illustrated. Figure 3 The following are some implementation details of the frame-based error prevention and control scheme. For ease of understanding, a CIG event may include multiple CIS events (e.g., N). EVENT CIS events), the event count N for multiple CIS events. EVENT It can be an integer greater than 1. More specifically, multiple CIS events can include a CIS event for the earliest CIS, at least one intermediate CIS event (e.g., a CIS event for an intermediate CIS), and a CIS event for the latest CIS, wherein the corresponding synchronization delays {CIS_Sync_Delay} of the multiple CIS events are, for example, synchronization delays CIS_Sync_Delay(1), CIS_Sync_Delay(2), and CIS_Sync_Delay(N). EVENT The CIS, intermediate CIS, and latest CIS (labeled for brevity as "CIS_Sync_Delay corresponding to the earliest CIS", "CIS_Sync_Delay corresponding to the intermediate CIS", and "CIS_Sync_Delay corresponding to the latest CIS"), and the CIS_Sync_Delay of the CIS event can be determined by UE 100, such as UE 300, according to the scheduling on the master device side (e.g., UE scheduling). In addition, uplink audio data sequence groups {M1(i), M2(i)} with the same index i (e.g., uplink audio data M1(1) and M2(1)) can represent uplink audio data sampled at the same time. The CIS event of the earliest CIS and the CIS event of the intermediate CIS can be used as examples of the CIS events of uplink audio data M1(i) and M2(i), respectively, but the present invention is not limited thereto.

[0042] For example, electronic systems can enhance audio using timing control; more specifically, they can perform the following operations:

[0043] (1) The DSP circuit #1 in the earphone #1 can determine the synchronization point (e.g., the CIG synchronization point of the CIG event) for the earphone #1 based on the first time point of the first event (e.g., the CIS event of the earliest CIS) and the first predetermined synchronization delay (e.g., the synchronization delay corresponding to the earliest CIS CIS_Sync_Delay(1)), wherein the first predetermined synchronization delay is determined by UE 100 such as UE 300;

[0044] (2) The DSP circuit #2 in the earphone #2 can determine the synchronization point (e.g., the CIG synchronization point of the CIG event) for the earphone #2 based on the second time point of the second event (e.g., the CIS event of the intermediate CIS) and the second predetermined synchronization delay (e.g., the synchronization delay corresponding to the intermediate CIS, CIS_Sync_Delay(2)) of the second event (e.g., the CIS event of the intermediate CIS), wherein the second predetermined synchronization delay is determined by UE 100 such as UE 300.

[0045] (3) DSP circuit #1 can determine a first reference point (e.g., CIG reference point) for earphone #1 based on the synchronization point (e.g., CIG synchronization point of CIG event) and a third predetermined synchronization delay (e.g., CIG synchronization delay of CIG event CIG_Sync_Delay), wherein the third predetermined synchronization delay is greater than either the first predetermined synchronization delay or the second predetermined synchronization delay (e.g., CIG_Sync_Delay > CIS_Sync_Delay(1) and CIG_Sync_Delay > CIS_Sync_Delay(2)), and the third predetermined synchronization delay is also determined by UE 100 such as UE 300;

[0046] (4) The DSP circuit #2 can determine a first reference point (e.g., a CIG reference point) for the earphone #2 based on the synchronization point (e.g., the CIG synchronization point of a CIG event) and a third predetermined synchronization delay (e.g., the synchronization delay of a CIG event, CIG_Sync_DelayCIG); and

[0047] (5) DSP circuit #1 and DSP circuit #2 can control the timing of uplink audio data M1(i) (e.g., uplink audio data M1(1)) from earphone #1 to UE 100 such as UE 300 and uplink audio data M2(i) (e.g., uplink audio data M2(1)) from earphone #2 to UE 100 such as UE 300 according to the first reference point (e.g., CIG reference point) used for earphone #1 and earphone #2 respectively;

[0048] Earphones #1 and #2 can each perform their own timing control with the assistance of UE 100, such as UE 300. UE 100, such as UE 300, can send a first predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(1) corresponding to the earliest CIS) to earphone #1 in advance, and can send a second predetermined synchronization delay (e.g., CIS_Sync_Delay(2) corresponding to the intermediate CIS) to earphone #2 in advance, and can send a third predetermined synchronization delay (e.g., the CIG synchronization delay CIG_Sync_Delay for CIG events) to earphones #1 and #2 in advance, so that earphones #1 and #2 can be aware of the scheduling on the master device side (e.g., UE scheduling), thereby enabling earphones #1 and #2 to correctly perform their own timing control.

[0049] like Figure 4As shown, the synchronization point (e.g., the CIG synchronization point of a CIG event) is later than each of the first time point (e.g., the start time of the CIS event of the earliest CIS) and the second time point (e.g., the start time of the CIS event of the intermediate CIS), wherein the time difference between the synchronization point and the first time point is equal to the first predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(1) corresponding to the earliest CIS), and the time difference between the synchronization point and the second time point is equal to the second predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(2) corresponding to the intermediate CIS). According to the frame-based error prevention control scheme, the synchronization point (e.g., the CIG synchronization point of the CIG event) can be used as the downlink playback reference time point, and can also be used as the intermediate reference point for the headphones #1 (e.g., DSP circuit #1) and headphones #2 (e.g., DSP circuit #2) to perform timing control on the uplink audio data M1(i) (e.g., uplink audio data M1(1)) and uplink audio data M2(i) (e.g., uplink audio data M2(1)) on the slave device side, respectively. The headphone #1 (e.g., DSP circuit #1) can determine a synchronization point (e.g., the CIG synchronization point of a CIG event) at least based on a first predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(1) corresponding to the earliest CIS) and use this synchronization point as the time point to trigger the playback of downlink audio data S1(i) (e.g., downlink audio data S1(1)) corresponding to the headphone #1, and further convert the downlink timing reference into an uplink timing reference. More specifically, the CIG reference point is used as the first reference point for timing control of uplink audio data M1(i) (e.g., uplink audio data M1(1)) to prevent frame-based errors. Similarly, the earphone #2 (e.g., DSP circuit #2) can determine a synchronization point (e.g., the CIG synchronization point of a CIG event) at least based on a second predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(2) corresponding to the intermediate CIS), and use this synchronization point as the time point to trigger the playback of downlink audio data S2(i) (e.g., downlink audio data S2(1)) corresponding to the earphone #2, and further convert the downlink timing reference to an uplink timing reference. More specifically, the CIG reference point is used as the first reference point for timing control of the uplink audio data M2(i) (e.g., uplink audio data M2(1)) to prevent frame-based errors. For the sake of brevity, similar descriptions to those of other embodiments are not repeated here.

[0050] According to some embodiments, the downlink playback reference point can represent the downlink playback reference time point of BLE True Wireless Stereo (TWS). For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0051] Figure 5 An example of a frame-based error that may occur when headphone audio scheduling control is temporarily disabled is shown. In this case, headphone #1 (e.g., DSP circuit #1) may not be aware of the scheduling on the master device side (e.g., UE scheduling), and therefore may not be able to prepare uplink audio data M1(i) (e.g., uplink audio data M1(1)) in time. Figure 5 As shown, the uplink audio data M1(i) (e.g., uplink audio data M1(1)) that should be sent from the slave device to the master device in the i-th time slot (e.g., the first time slot) may be replaced by previous uplink audio data, such as uplink audio data M1(i-1) (e.g., uplink audio data M1(0)). As a result, frame-based errors occur, and there are differences between the various transmission times of the respective uplink audio data M1(i) and M2(i) (e.g., uplink audio data M1(1) and M2(1)) of Mic#1 and Mic#2.

[0052] According to some embodiments, earphones #1 and #2 on the device side can monitor the sampling time offset {Offset1} (e.g., the sampling time offset Offset1(i) of uplink audio data M1(i)) and the sampling time offset {Offset2} (e.g., the sampling time offset Offset2(i) of uplink audio data M2(i)) of uplink audio data sequence {M1}, respectively, and provide the sampling time offsets {Offset1} and {Offset2} to the master device side, so that UE 100, such as UE 300, can perform sampling time calibration on uplink audio data sequence {M1} and uplink audio data sequence {M2} according to the sampling time offsets {Offset1} and {Offset2}, respectively, to prevent any frame-based errors, wherein the sampling time calibration may include audio sample interpolation. For example, before the first time point of the first event (e.g., the earliest CIS event of CIS), the headphone #1 can use the DSP circuit #1 to complete the processing of the uplink audio data M1(i) (e.g., uplink audio data M1(1)), such as LC3 processing and the first part processing of the uplink audio data M1(i), and then send the uplink audio data M1(i) (e.g., uplink audio data M1(1)) and the sampling time offset Offset1(i) of the uplink audio data M1(i) to UE 100 such as UE 300 in the first secondary timeslot in the i-th timeslot (e.g., the first timeslot). For example, before the second time point of the second event (e.g., the CIS event of the intermediate CIS), the headphone #2 can use the DSP circuit #2 to process the uplink audio data M2(i) (e.g., uplink audio data M2(1)), such as LC3 processing and processing of the first part of the uplink audio data M2(i), and then send the uplink audio data M2(i) (e.g., uplink audio data M2(1)) and the sampling time offset Offset2(i) of the uplink audio data M2(i) to UE 100 such as UE 300 in the second auxiliary time slot in the i-th time slot (e.g., the first time slot).

[0053] Figure 6 According to embodiments of the present invention, a sample-based error prevention and control scheme is shown for a method of audio enhancement using timing control. For ease of understanding, a complex array of vertical line segments corresponding to multiple audio frames {Frame1} (e.g., audio frames Frame1(1), Frame1(2), and Frame1(3)) is described. Figure 6The upper half of the horizontal axis (e.g., the time axis) is shown to represent audio samples of the uplink audio data sequence {M1} (e.g., uplink audio data M1(1), M1(2), and M1(3)), and complex array vertical line segments corresponding to multiple audio frames {Frame2} (e.g., audio frames Frame2(1), Frame2(2), and Frame2(3)). Figure 6 The lower half of the horizontal axis (e.g., the time axis) is shown to represent audio samples of the uplink audio data sequence {M2} (e.g., uplink audio data M2(1), M2(2) and M2(3)), respectively.

[0054] Ideally, neither of the clock sources in earphone #1 nor earphone #2 should have clock frequency drift. For example, earphone #1 can periodically generate audio frames {Frame1} (e.g., audio frames Frame1(1), Frame1(2), Frame1(3), etc.), while earphone #2 can periodically generate audio frames {Frame2} (e.g., audio frames Frame2(1), Frame2(2), Frame2(3), etc.). In addition, the sampling time ranges of audio frames Frame1(1), Frame1(2), Frame1(3), etc. can be equal to each other, and the sampling time ranges of audio frames Frame2(1), Frame2(2), Frame2(3), etc. can be equal to each other, wherein the sampling time range of any one of audio frames Frame1(1), Frame1(2), Frame1(3), etc. can be equal to the sampling time range of any one of audio frames Frame2(1), Frame2(2), Frame2(3), etc.

[0055] Although clock frequency drift may occur in, for example Figure 6In the actual scenario shown, earphones #1 and #2 on the device side can monitor the sampling time offset Offset1(i) (such as sampling time offset Offset1(1), Offset1(2), Offset1(3) etc.) and sampling time offset Offset2(i) (such as sampling time offset Offset2(1), Offset2(2), Offset2(3) etc.) respectively, and provide sampling time offset Offset1(i) and Offset2(i) to the master device side so that UE 100, such as UE300, can perform sampling time calibration (e.g., audio sample interpolation) on uplink audio data M1(i) and M2(i) respectively according to sampling time offset Offset1(i) and Offset2(i) so that the audio samples of uplink audio data M1(i) and M2(i) are calibrated relative to time (e.g., as if these audio samples were sampled according to a stable and common clock source on the master device side) to prevent any sample-based errors from occurring.

[0056] For example, before the first time point of the first event (e.g., the earliest CIS event of CIS), the headset #1 can use the DSP circuit #1 to process the uplink audio data M1(i) (e.g., uplink audio data M1(3)). Then, in the first secondary time slot of the i-th time slot (e.g., the third time slot), the uplink audio data M1(i) (e.g., uplink audio data M1(3)) and the sampling time offset Offset1(i) of the uplink audio data M1(i) (e.g., the sampling time offset Offset1(3) of the uplink audio data M1(i), which is marked as "Offset1 of Mic #1 in headset #1" for easy understanding) are sent to UE 100 such as UE 300. For example, before the second time point of the second event (e.g., the CIS event of the intermediate CIS), the earphone #2 can use the DSP circuit #2 to process the uplink audio data M2(i) (e.g., uplink audio data M2(3)). Then, in the second secondary time slot of the i-th time slot (e.g., the third time slot), the uplink audio data M2(i) (e.g., uplink audio data M2(3)) and the sampling time offset Offset2(i) of the uplink audio data M2(i) (e.g., the sampling time offset Offset2(3) of the uplink audio data M2(3), which is marked as "Offset2 of Mic #2 in earphone #2" for easy understanding) are sent to UE 100 such as UE 300.

[0057] like Figure 6 As shown, the sampling time offset Offset1(i) (e.g., sampling time offset Offset1(3), denoted as "Offset1(Offset1) of Mic#1 in Headphone #1") can represent the time difference between the start sampling time of a first reference point (e.g., CIG reference point) and multiple audio samples within uplink audio data M1(i) (e.g., uplink audio data M1(3)), for example, the time difference between the CIG reference point and the start time of the sampling time range of audio frame Frame1(i) (e.g., audio frame Frame1(3)). And sampling time offset Offset2(i) (e.g., sampling time offset Offset2(3), labeled “Offset2(Offset2) of Mic#2 in Headphone #2”) can represent the time difference between the start sampling time of multiple audio samples within the first reference point (e.g., CIG reference point) and the uplink audio data M2(i) (e.g., uplink audio data M2(3)), such as the time difference between the CIG reference point and the start time of the sampling time range of the audio frame Frame2(i) (e.g., audio frame Frame2(3)).

[0058] According to a sample-based error prevention and control scheme, UE 100, such as UE 300, can utilize DSP circuit #0 (e.g., audio DSP circuit 120 in UE 100) to perform sampling time calibration, such as audio sample interpolation, on either uplink audio data M1(i) (e.g., uplink audio data M1(i)) or uplink audio data M2(i) (e.g., uplink audio data M2(i)) based on sampling time offset Offset1(i) (e.g., sampling time offset Offset1(3), denoted as "offset 1(Offset1) of Mic #1 in Headphone #1") and sampling time offset Offset2(i) (e.g., sampling time offset Offset2(3), denoted as "offset 2(Offset2) of Mic #2 in Headphone #2") to prevent sample-based errors. For the sake of brevity, similar descriptions to those of other embodiments are not repeated here.

[0059] According to some embodiments, in response to the failure to successfully receive any uplink audio data in uplink audio data M1(i) (e.g., uplink audio data M1(3)) and uplink audio data M2(i) (e.g., uplink audio data M2(3)), UE 100, such as UE 300, may use DSP circuit #0 (e.g., audio DSP circuit 120 in UE 100) to convert the received uplink audio data (e.g., other uplink audio data) in uplink audio data M1(i) (e.g., uplink audio data M1(3)) and uplink audio data M2(i) (e.g., uplink audio data M2(3)) into simulated uplink audio data as a substitute for any uplink audio data that was not successfully received. For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0060] According to some embodiments, the first time point of the first event (e.g., the CIS event of the earliest CIS) and the second time point of the second event (e.g., the CIS event of the intermediate CIS) can respectively represent the start time points of the first auxiliary time slot and the second auxiliary time slot within the i-th time slot (e.g., the i-th isochronous (ISO) interval) of the same audio data frame on the master device side. For example, UE 100 of UE 300 can send (TX) downlink audio data S1(i) to headset #1 and receive (RX) uplink audio data M1(i) from headset #1 in the first auxiliary time slot within the i-th time slot, and can send (TX) downlink audio data S2(i) to headset #2 and receive (RX) uplink audio data M2(i) from headset #2 in the second auxiliary time slot within the i-th time slot. However, the present invention is not limited thereto. For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0061] Figure 7 This invention discloses a UE-schedule-aware headset timing control scheme for performing audio enhancement using timing control. For ease of understanding, the i-th time slot, for example, the i-th ISO interval, can be represented as... Figure 7 The ISO intervals shown are as follows: Figure 7The diagram shows the case where i=1. According to the UE-schedule-aware headset timing control scheme, uplink audio data M1(i) and M2(i) can be transmitted from the slave device side to the master device side in a timely manner. More specifically, uplink audio data M1(i) and M2(i) can be transmitted from headset #1 and #2 to UE 100, such as UE300, respectively in the i-th time slot (e.g., the i-th ISO interval) of multiple time slots (e.g., multiple ISO intervals). Furthermore, the electronic system can utilize DSP circuit #1 and DSP circuit #2 to control the timing of uplink audio data M1(i) (e.g., uplink audio data M1(1)) from earphone #1 to UE 100 (e.g., UE 300) and the timing of uplink audio data M2(i) (e.g., uplink audio data M2(1)) from earphone #2 to UE 100 (e.g., UE 300) based on a first reference point (e.g., CIG reference point) for earphone #1 and earphone #2, respectively.

[0062] For example, the electronic system can utilize DSP circuit #1 (e.g., audio DSP circuit 162 in headset 360, denoted as "audio DSP" for brevity) to complete the processing of uplink audio data M1(i) (e.g., uplink audio data M1(1)) before the first time point (more specifically, before the time point of anchor 1(i) (e.g., anchor 1(1)) of headset #1) and complete the transmission of uplink audio data M1(i) from DSP circuit #1 to BT IF circuit (denoted as "Bluetooth" for brevity) in headset #1, and then transmit uplink audio data M1(i) (e.g., uplink audio data M1(1)) to UE 100 (e.g., UE 300) in the first auxiliary time slot within the i-th time slot. The operation of preparing uplink audio data M1(i) (e.g., uplink audio M1(1)) from the device side can include:

[0063] (1) The analog-to-digital converter (ADC) in the DSP circuit #1 is used to perform analog-to-digital conversion (marked as “A2D”) in the first sub-processing time period Δ1(1) of the first processing time period Δ1;

[0064] (2) The LC3 processing circuit and the first part processing circuit in DSP circuit #1 are used to perform LC3 processing and first part processing (for simplicity, this is marked as "frame-based processing") in the second sub-processing time period Δ1(2) of the first processing time period Δ1; and

[0065] (3) The uplink audio data M1(i) (e.g., uplink audio data MI(1)) is sent to the BT IF circuit in the headset #1 (for simplicity, labeled as "to Bluetooth") during the third sub-processing time period Δ1(3) of the first processing time period Δ1.

[0066] Where Δ1=Δ1(1)+Δ1(2)+Δ1(3).

[0067] For example, the electronic system can utilize DSP circuit #2 (e.g., audio DSP circuit 182 in headset 380, denoted as "audio DSP" for brevity) to complete the processing of uplink audio data M2(i) (e.g., uplink audio data M2(1)) before the second time point (more specifically, before the time point of anchor 2(i) (e.g., anchor 2(1)) of headset #2) and complete the transmission of uplink audio data M2(i) from DSP circuit #2 to BT IF circuit (denoted as "BT" for brevity) in headset #2, and then transmit uplink audio data M2(i) (e.g., uplink audio data M2(1)) to UE 100 (e.g., UE 300) in the second auxiliary time slot within the i-th time slot. The operation of preparing uplink audio data M2(i) (e.g., uplink audio M2(1)) from the device side can include:

[0068] (1) The ADC in DSP circuit #2 performs analog-to-digital conversion (marked as “A2D”) in the first sub-processing time period Δ2(1) of the second processing time period Δ2;

[0069] (2) The LC3 processing circuit and the first part processing circuit in DSP circuit #2 are used to perform LC3 processing and first part processing (for simplicity, this is marked as "frame-based processing") in the second sub-processing time period Δ2(2) of the second processing time period Δ2; and

[0070] (3) The uplink audio data M2(i) (e.g., uplink audio data M2(1)) is sent to the BT IF circuit in the headset #2 (for simplicity, labeled "to Bluetooth") during the third sub-processing time period Δ2(3) of the second processing time period Δ2.

[0071] Where Δ2=Δ2(1)+Δ2(2)+Δ2(3).

[0072] Furthermore, the hardware (HW) #1 of earphone #1 (e.g., the ADC within DSP circuit #1) and the hardware (HW) #2 of earphone #2 (e.g., the ADC within DSP circuit #2) can simultaneously receive audio samples, and the first processing time period Δ1 and the second processing time period Δ2 can be measured from the time point when the audio samples of the corresponding uplink audio data M1(i) and M2(i) are received from HW #1 and HW #2. The end time point of the first processing time period Δ1 should be earlier than the anchor point Anchor1(i) of earphone #1, and the end time point of the second processing time period Δ2 should be earlier than the anchor point Anchor2(i) of earphone #2, wherein the end time points of the first processing time period Δ1 and the second processing time period Δ2 need not be earlier than the anchor point Anchor0(i) of UE 100, such as UE 300. For example, for the case of i=1 (e.g., the time point when HW #1 and HW #2 receive the audio samples of the corresponding uplink audio data M1(1) and M2(1) can be Figure 7 The end time of the first processing time period Δ1 should be earlier than the anchor point Anchor1(1) of earphone #1, and the end time of the second processing time period Δ2 should be earlier than the anchor point Anchor2(1) of earphone #2. The end time of each of the first and second processing time periods Δ1 and Δ2 need not be earlier than UE 100 (e.g., UE 100). Anchor0(1) of UE 100 (e.g., UE 300); for the case of i=2 (e.g., the time point when HW#1 and HW#2 receive the audio samples of the corresponding uplink audio data M1(2) and M2(2) is usually later than the time point when HW#1 and HW#2 receive the audio samples of the corresponding uplink audio data M1(1) and M2(1)), the end time point of the first processing time period Δ1 should be earlier than the anchor1(2) of the headphone #1, and the end time point of the second processing time period Δ2 should be earlier than the anchor2(2) of the headphone #2, wherein the end time points of the first processing time period Δ1 and the second processing time period Δ2 need not be earlier than the anchor0(2) of UE 100 (e.g., UE 300); the rest can be deduced by analogy. For the sake of brevity, similar descriptions of this embodiment and other embodiments will not be repeated here.

[0073] According to some embodiments, UE 100, such as UE 300, can determine the time point of anchor point Anchor1(i) of earphone #1 and the time point of anchor point Anchor2(i) of earphone #2 respectively as the deadlines for preparing uplink audio data M1(i) and M2(i) on the slave device side based on the scheduling (e.g., UE scheduling) of earphone #1 and earphone #2 respectively on the master device side, so as to allow earphone #1 and #2 to perform their own timing control with the help of UE 100, such as UE 300. For example, the preparation of uplink audio data M1(i) and M2(i) is completed on the slave device side before the time points of anchor point Anchor1(i) and anchor point Anchor2(i) determined on the master device side, but the present invention is not limited thereto. Furthermore, the time point of anchor1(i) can be equal to or earlier than the first auxiliary time slot within the i-th time slot starting from anchor0(i) on the master device side (e.g., the auxiliary time slot used to perform the first set of TX and RX operations, such as...). Figure 7 The lower half shows the start time points of the TX operation of S1(i) and the RX operation of M1(i) when i=1, and the time point of the anchor point Anchor2(i) can be equal to or earlier than the second auxiliary time slot in the i-th time slot (e.g., the auxiliary time slot used to perform the second set of TX and RX operations, such as...). Figure 7 The lower half shows the start time of the TX operation of S2(i) and the RX operation of M2(i) when i=1.

[0074] The first set of TX and RX operations, such as the TX operation of S1(i) and the RX operation of M1(i) on the master device side, can be considered as the first CIS event in the i-th time slot. The second set of TX and RX operations, such as the TX operation of S2(i) and the RX operation of M2(i) on the master device side, can be considered as the second CIS event in the i-th time slot. The CIG event in the i-th time slot can include the first CIS event and the second CIS event. Since the RX and TX operations on the slave device side correspond to the TX and RX operations on the master device side, a set of RX and TX operations on the slave device side, such as the RX operation of S1(i) and the TX operation of M1(i), can be considered as the CIS event of stream #1 of earphone #1. Another set of RX and TX operations on the slave device side, such as the RX operation of S2(i) and the TX operation of M2(i), can be considered as the CIS event of stream #2 of earphone #2. For the sake of brevity, similar descriptions of these embodiments and other embodiments will not be repeated here.

[0075] Figure 8 According to embodiments of the present invention, a noise cancellation control scheme for a method of audio enhancement using timing control is illustrated. For example... Figure 8 As shown in the left half, when a user is making a hands-free call with a remote user using headsets #1 and #2, noise sources (such as someone speaking loudly, standing or walking nearby) may generate significant noise. Figure 8 As shown in the right half, UE 100, such as UE 300, can perform acoustic beamforming, and more specifically, eliminate background noise based on the time difference between the respective noise receptions of Mic#1 and Mic#2. For example, the noise received by Mic#1 can overlap with the user's speech along the horizontal axis (e.g., the time axis) in a first manner, and the noise received by Mic#2 can overlap with the user's speech along the horizontal axis (e.g., the time axis) in a second manner, where the time difference can represent the difference between a first time range of the noise received by Mic#1 and a second time range of the noise received by Mic#2, for example, the time offset between the noise wave of the noise received by Mic#1 and the noise wave of the noise received by Mic#2. DSP circuit #0 (e.g., the second part of the processing circuit therein) can perform second part processing (e.g., NR processing) on ​​uplink audio data M1(i) and M2(i). More specifically, DSP circuit #0 can identify noise in each of the uplink audio data M1(i) and M2(i) at least according to the time difference, and remove noise from each of the uplink audio data M1(i) and M2(i).

[0076] Since uplink audio data M1(i) and M2(i) can be transmitted in a timely manner from the slave device side (e.g., headphones #1 and #2) to the master device side (e.g., UE 100 such as UE 300), and since no frame-based or sample-based errors occur in the electronic system, the method and related apparatus of the present invention can guarantee that the second part of the processing (e.g., NR processing) will be successfully executed without being hindered by any errors (e.g., any frame-based errors or any sample-based errors). For the sake of brevity, similar descriptions of this embodiment and other embodiments will not be repeated here.

[0077] like Figure 8 As shown, earphones #1 and #2 (e.g., earphones 160 and 180 such as earphones 360 and 380) can represent the left and right earphones of the user, respectively, such as earphones worn in the user's left and right ears, but the invention is not limited thereto.

[0078] Figure 9 The present invention illustrates the workflow of a method for audio enhancement using timing control.

[0079] In step S11, the electronic system may use the DSP circuit #1 in the earphone #1 to determine the synchronization point (e.g., the CIG synchronization point of the CIG event) for the earphone #1 based on the first time point of the first event (e.g., the CIS event of the earliest CIS) and the first predetermined synchronization delay (e.g., the synchronization delay corresponding to the earliest CIS, CIS_Sync_Delay(1)), wherein the first predetermined synchronization delay is determined by UE 100 such as UE 300.

[0080] In step S12, the electronic system may use DSP circuit #1 to determine a first reference point (e.g., CIG reference point) for earphone #1 based on a synchronization point (e.g., CIG synchronization point of a CIG event) and a third predetermined synchronization delay (e.g., CIG synchronization delay of a CIG event, CIG_Sync_Delay), wherein the third predetermined synchronization delay is determined by UE 100, such as UE 300.

[0081] In step S13, the electronic system may use DSP circuit #1 to control the timing of uplink audio data M1(i) (e.g., uplink audio data M1(1)) from earphone #1 to UE 100 such as UE 300 according to a first reference point (e.g., CIG reference point) for earphone #1.

[0082] In step S21, the electronic system may use the DSP circuit #2 in the earphone #2 to determine the synchronization point (e.g., the CIG synchronization point of the CIG event) for the earphone #2 based on the second time point of the second event (e.g., the CIS event of the intermediate CIS) and the second predetermined synchronization delay (e.g., the synchronization delay corresponding to the intermediate CIS, CIS_Sync_Delay(2)), wherein the second predetermined synchronization delay is determined by UE 100 such as UE 300.

[0083] In step S22, the electronic system can use the DSP circuit #2 to determine a first reference point (e.g., a CIG reference point) for the earphone #2 based on a synchronization point (e.g., the CIG synchronization point of a CIG event) and a third predetermined synchronization delay (e.g., the CIG synchronization delay CIG_Sync_Delay of a CIG event).

[0084] In step S23, the electronic system may use DSP circuit #2 to control the timing of uplink audio data M2(i) (e.g., uplink audio data M2(1)) from earphone #2 to UE 100 such as UE 300 according to a first reference point (e.g., CIG reference point) for earphone #2.

[0085] like Figure 9As shown, the electronic system can perform parallel processing. More specifically, the operations of UE scheduling awareness timing control for earphone #1 (e.g., operations in steps S11-S13) and UE scheduling awareness timing control for earphone #2 (e.g., operations in steps S21-S23) are performed in parallel. For the sake of brevity, similar descriptions to those in other embodiments will not be repeated here.

[0086] For ease of understanding, this method can be used Figure 9 The workflow shown is for illustrative purposes only, but the invention is not limited thereto. According to some embodiments, it is possible to... Figure 9 Add, delete, or change one or more steps in the workflow shown.

[0087] Figure 10 According to another embodiment of the present invention, the workflow of a method for audio enhancement by means of timing control is shown.

[0088] In step S31, the electronic system may use UE 100, such as UE 300, to determine a first predetermined synchronization delay (corresponding to the synchronization delay CIS_Sync_Delay(1) of the earliest CIS) and notify the earphone #1 of the first predetermined synchronization delay, wherein the DSP circuit #1 in the earphone #1 may determine a synchronization point (e.g., the CIG synchronization point of the CIG event) for the earphone #1 based on the first time point of the first event (e.g., the CIS event of the earliest CIS) and the first predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(1) of the earliest CIS).

[0089] In step S32, the electronic system may use UE 100, such as UE 300, to determine a second predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(2) corresponding to the intermediate CIS) and notify the earphone #2 of the second predetermined synchronization delay, wherein the DSP circuit #2 in the earphone #2 may determine a synchronization point (e.g., the CIG synchronization point of the CIG event) for the earphone #2 based on the second time point of the second event (e.g., the CIS event of the intermediate CIS) and the second predetermined synchronization delay (e.g., the synchronization delay CIS_Sync_Delay(2) corresponding to the intermediate CIS).

[0090] In step S33, the electronic system may use UE 100, such as UE 300, to determine a third predetermined synchronization delay (e.g., CIG synchronization delay CIG_Sync_Delay for a CIG event) and notify the first and second earpieces of the third predetermined synchronization delay respectively, for determining the first reference point respectively. DSP circuit #1 can determine the first reference point (e.g., a CIG reference point) for earpiece #1 based on the synchronization point (e.g., the CIG synchronization point for a CIG event) and the third predetermined synchronization delay (e.g., the CIG synchronization delay CIG_Sync_Delay for a CIG event). DSP circuit #2 can determine the first reference point (e.g., a CIG reference point) for earpiece #2 based on the synchronization point (e.g., the CIG synchronization point for a CIG event) and the third predetermined synchronization delay (e.g., the CIG synchronization delay CIG_Sync_Delay for a CIG event).

[0091] In step S34, the electronic system may use UE 100, such as UE 300, to receive uplink audio data M1(i) (e.g., uplink audio data M1(1)) from earphone #1 and uplink audio data M2(i) (e.g., uplink audio data M2(1)) from earphone #2, wherein DSP circuit #1 can control the timing of uplink audio data M1(i) (e.g., uplink audio data M1(1)) from earphone #1 to UE 100, such as UE 300, according to a first reference point (e.g., CIG reference point) for earphone #1, and DSP circuit #2 can control the timing of uplink audio data M2(i) (e.g., uplink audio data M2(1)) from earphone #2 to UE 100, such as UE 300, according to a first reference point (e.g., CIG reference point) for earphone #2.

[0092] For ease of understanding, this method can be used Figure 10 The workflow shown is for illustrative purposes only, but the invention is not limited thereto. According to some embodiments, it is possible to... Figure 10 Add, delete, or change one or more steps in the workflow shown.

[0093] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the scope of the invention. Any person skilled in the art can make some modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for audio enhancement, characterized in that, The method is used for a user device that wirelessly connects to a first earphone and a second earphone, and the method includes: The user equipment is used to determine a first predetermined synchronization delay and the first predetermined synchronization delay is notified to the first headset, so that the first digital signal processing circuit in the first headset determines a synchronization point for the first headset based on the first time point of the first event and the first predetermined synchronization delay. The user equipment is used to determine a second predetermined synchronization delay and the second predetermined synchronization delay is notified to the second headset, so that the second digital signal processing circuit in the second headset determines the synchronization point for the second headset based on the second time point of the second event and the second predetermined synchronization delay; The user equipment receives first uplink audio data from the first headset and second uplink audio data from the second headset, wherein the timing of the first uplink audio data and the timing of the second uplink audio data are controlled by the first digital signal processing circuit and the second digital signal processing circuit respectively according to a first reference point for the first headset and the second headset, wherein the first reference point is determined at least according to the synchronization point.

2. The method as described in claim 1, characterized in that, This synchronization point serves as a reference time point for downlink playback.

3. The method as described in claim 2, characterized in that, This downlink playback reference time point represents the downlink playback reference time point for Bluetooth Low Energy true wireless stereo.

4. The method as described in claim 1, characterized in that, The synchronization point is later than each of the first time point and the second time point, and the time difference between the synchronization point and the first time point is equal to the first predetermined synchronization delay, and the time difference between the synchronization point and the second time point is equal to the second predetermined synchronization delay.

5. The method as described in claim 1, characterized in that, This first reference point is used as the reference time point for uplink audio data.

6. The method as described in claim 1, characterized in that, Also includes: The user equipment is used to determine a third predetermined synchronization delay and the third predetermined synchronization delay is notified to the first headset and the second headset respectively, so that the first digital signal processing circuit and the second digital signal processing circuit can determine the first reference point respectively.

7. The method as described in claim 6, characterized in that, The first reference point is determined by the first digital signal processing circuit and the second digital signal processing circuit respectively based on the synchronization point and the third predetermined synchronization delay.

8. The method as described in claim 6, characterized in that, The first reference point is earlier than the synchronization point, and the time difference between the synchronization point and the first reference point is equal to the third predetermined synchronization delay.

9. The method as described in claim 1, characterized in that, The first time point and the second time point respectively represent the start time points of the first auxiliary time slot and the second auxiliary time slot within the time slot corresponding to the same audio data frame.

10. The method as described in claim 1, characterized in that, The first uplink audio data and the second uplink audio data are transmitted to the user equipment in the first time slot of a plurality of time slots.

11. The method as described in claim 10, characterized in that: The first uplink audio data is transmitted to the user equipment in the first auxiliary time slot within the first time slot, wherein the processing of the first uplink audio data is completed by the first digital signal processing circuit before the first time point; and The second uplink audio data is transmitted to the user equipment in the second auxiliary time slot within the first time slot, wherein the processing of the second uplink audio data is completed by the second digital signal processing circuit before the second time point.

12. The method as described in claim 1, characterized in that, The first uplink audio data and the second uplink audio data represent a set of stereo audio data corresponding to the same sampling time period.

13. The method as described in claim 1, characterized in that: The first uplink audio data and the first sampling time offset of the first uplink audio data are sent to the user equipment in the first auxiliary time slot within the first time slot, wherein the processing of the first uplink audio data is completed by the first digital signal processing circuit before the first time point; and The second uplink audio data and the second sampling time offset of the second uplink audio data are sent to the user equipment in the second auxiliary time slot within the first time slot, wherein the processing of the second uplink audio data is completed by the second digital signal processing circuit before the second time point.

14. The method as described in claim 13, characterized in that, The first sampling time offset represents the time difference between the first reference point and the start sampling time of multiple audio samples in the first uplink audio data, and the second sampling time offset represents the time difference between the first reference point and the start sampling time of multiple audio samples in the second uplink audio data.

15. The method as described in claim 13, characterized in that, Also includes: The user equipment performs sampling time calibration on either the first uplink audio data or the second uplink audio data based on the first sampling time offset and the second sampling time offset.

16. The method as described in claim 13, characterized in that, Also includes: In response to the failure to successfully receive any uplink audio data in the first uplink audio data and the second uplink audio data, the user equipment uses the first sampling time offset and the second sampling time offset to convert the received uplink audio data in the first uplink audio data and the second uplink audio data into simulated uplink audio data as a substitute for any uplink audio data that was not successfully received.

17. A pair of headphones that are wirelessly connected to a user device, characterized in that, The headphones include: The wireless communication interface circuit is configured to enable the headset to perform wireless communication with the user equipment; An audio input device is configured to input audio waves to generate input audio data. An audio output device is configured to output an audio wave based on output audio data; and A first digital signal processing circuit, coupled to the wireless communication interface circuit, the audio input device, and the audio output circuit, is configured to perform signal processing on either the input audio data or the output audio data of the headphones. The first digital signal processing circuit determines a synchronization point for the earphone based on a first time point of the first event and a first predetermined synchronization delay, wherein the first predetermined synchronization delay is determined by the user equipment. The first digital signal processing determines a first reference point for the first earphone based on the synchronization point and a second predetermined synchronization delay, wherein the second predetermined synchronization delay is greater than the first predetermined synchronization delay; and The first digital signal processing circuit controls the timing of the first uplink audio data from the headset to the user equipment based on the first reference point used for the headset.

18. A headphone system comprising a first earpiece and a second earpiece, wherein the first earpiece and the second earpiece are wirelessly connected to a user equipment, wherein the first earpiece includes: A first wireless communication interface circuit is configured to enable the first earpiece to perform wireless communication with the user equipment. The first audio input device is configured to input audio waves to generate input audio data; The first audio output device is configured to output audio waves based on output audio data; and A first digital signal processing circuit, coupled to the first wireless communication interface circuit, the first audio input device, and the first audio output circuit, is configured to perform signal processing on either the input audio data or the output audio data of the first earphone. The second earphone includes: The second wireless communication interface circuit is configured to enable the second earpiece to perform wireless communication with the user equipment. The second audio input device is configured to input audio waves to generate input audio data; The second audio output device is configured to output audio waves based on the output audio data; and A second digital signal processing circuit, coupled to the second wireless communication interface circuit, the second audio input device, and the second audio output circuit, is configured to perform signal processing on either the input audio data or the output audio data of the second earphone. The first digital signal processing circuit determines a synchronization point for the first earphone based on the first time point of the first event and a first predetermined synchronization delay, wherein the first predetermined synchronization delay is determined by the user equipment. The second digital signal processing circuit determines the synchronization point for the second earphone based on the second time point of the second event and the second predetermined synchronization delay, wherein the second predetermined synchronization delay is determined by the user equipment. The first digital signal processing circuit determines a first reference point for the first earphone based on the synchronization point and the third predetermined synchronization delay. The second digital signal processing circuit determines the first reference point for the second earphone based on the synchronization point and the third predetermined synchronization delay, wherein the third predetermined synchronization delay is greater than both the first predetermined synchronization delay and the second predetermined synchronization delay; and The first digital signal processing circuit and the second digital signal processing circuit control the timing of the first uplink audio data from the first earphone to the user equipment and the timing of the second uplink audio data from the second earphone to the user equipment, respectively, based on the first reference point used for the first earphone and the second earphone.

19. A user equipment, wirelessly connected to a first earpiece and a second earpiece, characterized in that, include: The wireless communication interface circuit is configured to enable the user equipment to communicate wirelessly with the first earphone and the second earphone; An audio digital signal processing circuit, coupled to the wireless communication interface circuit, is configured to perform signal processing for the user equipment; The audio digital signal processing circuit is configured as follows: A first predetermined synchronization delay is determined and the first predetermined synchronization delay is notified to the first earphone, so that the first digital signal processing circuit in the first earphone determines a synchronization point for the first earphone based on the first time point of the first event and the first predetermined synchronization delay; A second predetermined synchronization delay is determined and the second predetermined synchronization delay is notified to the second earpiece, so that the second digital signal processing circuit in the second earpiece determines the synchronization point for the second earpiece based on the second time point of the second event and the second predetermined synchronization delay; The first uplink audio data is received from the first earphone and the second uplink audio data is received from the second earphone, wherein the timing of the first uplink audio data and the timing of the second uplink audio data are controlled by the first digital signal processing circuit and the second digital signal processing circuit respectively according to a first reference point for the first earphone and the second earphone, wherein the first reference point is determined at least according to the synchronization point.

20. The user equipment as claimed in claim 19, characterized in that, The audio digital signal processing circuit is also configured as follows: The first uplink audio data and the first sampling time offset of the first uplink audio data are received in the first auxiliary time slot within the first time slot, and the second uplink audio data and the second sampling time offset of the second uplink audio data are received in the second auxiliary time slot within the first time slot; The processing of the first uplink audio data is completed by the first digital signal processing circuit before the first time point, and the processing of the second uplink audio data is completed by the second digital signal processing circuit before the second time point.

21. The user equipment as claimed in claim 20, characterized in that, The audio digital signal processing circuit is also configured to perform sampling time calibration on either the first uplink audio data or the second uplink audio data based on the first sampling time offset and the second sampling time offset.

22. The user equipment as claimed in claim 19, characterized in that, The synchronization point is later than each of the first time point and the second time point, and the time difference between the synchronization point and the first time point is equal to the first predetermined synchronization delay, and the time difference between the synchronization point and the second time point is equal to the second predetermined synchronization delay.

23. The user equipment as claimed in claim 19, characterized in that, The audio digital signal processing circuit is also configured to determine a third predetermined synchronization delay and notify the first earphone and the second earphone of the third predetermined synchronization delay respectively, so that the first digital signal processing circuit and the second digital signal processing circuit can determine the first reference point respectively.

24. The user equipment as claimed in claim 23, characterized in that, The first reference point is earlier than the synchronization point, and the time difference between the synchronization point and the first reference point is equal to the third predetermined synchronization delay.

25. The user equipment as claimed in claim 19, characterized in that, The audio digital signal processing circuit is also configured to receive the first uplink audio data in a first auxiliary time slot within the first time slot and to receive the second uplink audio data in a second auxiliary time slot within the first time slot; The processing of the first uplink audio data is completed by the first digital signal processing circuit before the first time point, and the processing of the second uplink audio data is completed by the second digital signal processing circuit before the second time point.

Images