Systems, methods, and non-transitory computer-readable storage media with sound adjustment capabilities

Abstract

A system with sound adjustment capabilities. The system includes a headset, a first speaker, and a processor. The first speaker is detachable from the headset. The processor detects multiple positions and orientations of the headset and the first speaker to determine whether the first speaker is detached from the headset. The processor is also configured to modulate a first audio signal using at least one first filter or at least one second filter to generate a filtered first audio signal. At least one first filter is used when the first speaker is coupled to the headset, and at least one second filter is used when the first speaker is detached from the headset. The filtered first audio signal is configured to drive the first speaker. This provides two different configurations, a headphone configuration and a speaker configuration, improving communication efficiency.

Description

Technical Field

[0001] This disclosure relates to the processing of audio signals. More particularly, this disclosure relates to a system with sound adjustment capabilities, a method for adjusting sound, and a non-transitory computer-readable storage medium. Background Technology

[0002] Virtual reality (VR) is a technology that uses computers to simulate a three-dimensional virtual world, providing users with sensory simulations such as sight, hearing, and touch. Headphones are typically integrated into VR devices to provide an immersive binaural audio experience. However, not only are real-world sounds blocked by the headphones, but others also cannot hear the sounds provided to the user through the headphones, making communication between the user and their colleagues or teammates difficult. Summary of the Invention

[0003] This disclosure provides a system with sound adjustment capabilities. The system includes a head-mounted device, a first speaker, and at least one processor. The first speaker is detachable from the head-mounted device. The at least one processor is configured to detect multiple positions and orientations of the head-mounted device and the first speaker to determine whether the first speaker has been detached from the head-mounted device. The at least one processor is also configured to modulate a first audio signal using at least one first filter or at least one second filter to generate a filtered first audio signal. The at least one processor uses at least one first filter in response to the first speaker being coupled to the head-mounted device, and uses at least one second filter in response to the first speaker being detached from the head-mounted device. The filtered first audio signal is transmitted to the first speaker to drive the first speaker.

[0004] In some embodiments, the at least one processor is configured to adjust the first audio signal at one or more frequencies to generate a sound, wherein the sound generated by the first speaker based on the filtered first audio signal has an enhanced frequency response at the entrance to a user's ear compared to the sound generated by the first speaker based on the unfiltered audio signal.

[0005] In some embodiments, the at least one first filter includes a headphone effect filter for eliminating distortion caused at least in part by a circuit when the headset and the first speaker are coupled together.

[0006] In some embodiments, the at least one second filter includes a speaker effect filter for eliminating at least part of the distortion caused by a circuit when the headset and the first speaker are disconnected from the headset.

[0007] In some embodiments, the at least one processor is configured to select a coefficient for the first speaker in the speaker effect filter based on a distance between the first speaker and the head-mounted device.

[0008] In some embodiments, a storage device is further included, wherein, in response to the first speaker being coupled to the head-mounted device, the at least one processor is configured to obtain an actual frequency response of an echo of a sound generated by the first speaker based on a reference audio signal, and in response to a difference between the actual frequency response and an ideal frequency response stored in the storage device, the at least one processor is configured to apply a position compensation filter of the at least one first filter to the first audio signal, wherein the position compensation filter is configured to render the echo such that the echo has a modulated frequency response substantially the same as the ideal frequency response.

[0009] In some embodiments, a second speaker detachable from the head-mount device is further included, wherein in response to the first speaker and the second speaker being coupled to opposite first and second terminals of the head-mount device, respectively, and in response to the at least one processor determining that the filtered first audio signal has a channel corresponding to the second terminal, the at least one processor is configured to transmit a filtered second audio signal previously transmitted to the second speaker to the first speaker and to transmit the filtered first audio signal to the second speaker.

[0010] In some embodiments, a second speaker detachable from the head-mounted device is further included, wherein in response to the first speaker and the second speaker being detached from the head-mounted device and positioned at a first position and a second position respectively, the head-mounted device being substantially located between the first position and the second position, and in response to the at least one processor determining that the filtered first audio signal has a channel corresponding to the second position, the at least one processor is configured to transmit a filtered second audio signal previously transmitted to the second speaker to the first speaker and to transmit the filtered first audio signal to the second speaker.

[0011] In some embodiments, at least one second filter includes a crosstalk cancellation filter and a head correlation transfer function (HRTF) filter.

[0012] In some embodiments, at least one processor is configured to obtain coefficients in the crosstalk cancellation filter and the HRTF filter based on the plurality of locations and the plurality of orientations.

[0013] This disclosure provides a method for adjusting sound. The method is applicable to a head-mounted device and a first speaker system detachable from the head-mounted device, comprising the following operations: detecting multiple positions and multiple orientations of the head-mounted device and the first speaker, and determining whether the first speaker is detached from the head-mounted device; modulating a first audio signal using at least one first filter or at least one second filter to generate a filtered first audio signal, wherein at least one first filter is used in response to the first speaker being coupled to the head-mounted device, and at least one second filter is used in response to the first speaker being detached from the head-mounted device; and transmitting the filtered first audio signal to the first speaker to drive the first speaker.

[0014] In some embodiments, modulating the first audio signal includes modulating the first audio signal at one or more frequencies to generate a sound, wherein the sound generated by the first speaker based on the filtered first audio signal has an enhanced frequency response at the entrance to a user's ear compared to the sound generated by the first speaker based on the unfiltered audio signal.

[0015] In some embodiments, at least one first filter includes a headphone effect filter for eliminating distortion caused at least in part by a circuit that couples the headset and the first speaker to each other.

[0016] In some embodiments, at least one second filter includes a speaker effect filter for eliminating distortion caused at least in part by a circuit of the headset and the first speaker separated from the headset.

[0017] In some embodiments, the coefficients for the first speaker in the speaker effect filter are selected based on a distance between the first speaker and the head-mounted device.

[0018] In some embodiments, the system further includes a storage, wherein modulating the first audio signal further includes: in response to the first speaker being coupled to the head-mounted device, obtaining an actual frequency response of an echo of a sound generated by the first speaker according to a reference audio signal; and in response to a difference between the actual frequency response and an ideal frequency response stored in the storage, applying a position compensation filter of the at least one first filter to the first audio signal, wherein the position compensation filter is configured to render the echo such that the echo has a modulated frequency response substantially the same as the ideal frequency response.

[0019] In some embodiments, the system further includes a second speaker detachable from the head-mounted device, characterized in that the method further includes: in response to the first speaker and the second speaker being coupled to a first and a second terminal opposite to each other on the head-mounted device, and in response to the filtered first audio signal having a channel corresponding to the second terminal, transmitting a filtered second audio signal previously transmitted to the second speaker to the first speaker, and transmitting the filtered first audio signal to the second speaker.

[0020] In some embodiments, the system further includes a second speaker detachable from the head-mounted device, characterized in that the method further includes: in response to the first speaker and the second speaker being detached from the head-mounted device and respectively located at a first position and a second position, the head-mounted device being substantially located between the first position and the second position, and in response to the filtered first audio signal having a channel corresponding to the second position, transmitting a filtered second audio signal previously transmitted to the second speaker to the first speaker, and transmitting the filtered first audio signal to the second speaker.

[0021] In some embodiments, the at least one second filter includes a crosstalk cancellation filter and a head correlation transfer function (HRTF) filter.

[0022] This disclosure provides a non-transitory computer-readable storage medium storing a plurality of computer-readable instructions for controlling a system including at least one processor, a head-mounted device, and a first speaker detachable from the head-mounted device. When the plurality of computer-readable instructions are executed by at least one processor, the at least one processor can perform: detecting a plurality of positions and a plurality of orientations of the head-mounted device and the first speaker to determine whether the first speaker is detached from the head-mounted device; modulating a first audio signal to generate a filtered first audio signal by passing at least one first filter or at least one second filter, wherein at least one first filter is used in response to the first speaker being coupled to the head-mounted device, and at least one second filter is used in response to the first speaker being detached from the head-mounted device; and transmitting the filtered first audio signal to the first speaker to drive the first speaker.

[0023] It should be understood that the foregoing general description and the following detailed description are merely examples and are intended to provide a further explanation of the claimed disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the specification, serve to explain the technical solutions of the embodiments of the present invention.

[0025] Figure 1A schematic side view of a system with sound adjustment capability according to an embodiment of the present disclosure is shown.

[0026] Figure 2 An embodiment according to the present disclosure is shown. Figure 1 Simplified functional block diagram of the system.

[0027] Figure 3 A flowchart of a method for adjusting sound according to an embodiment of the present disclosure is shown.

[0028] Figure 4 An example diagram of the frequency response of an earphone configuration worn on a mannequin head according to an embodiment of the present disclosure is shown.

[0029] Figure 5 An example diagram of an adaptive filter according to an embodiment of the present disclosure is shown.

[0030] Figure 6 An example diagram of the frequency response of an earphone configuration worn on a user's head according to an embodiment of the present disclosure is shown.

[0031] Figure 7 It shows the result of Figure 1 Example diagram of the virtual environment provided by the head-mounted device.

[0032] Figure 8 It shows the result of Figure 1 Another example diagram of the virtual environment provided by the head-mounted device.

[0033] Symbol explanation:

[0034] 23: Headphone Effect Filter

[0035] 24: Speaker Effect Filter

[0036] 25: Position Compensation Filter

[0037] 26: Crosstalk Cancellation Filter

[0038] 27: Head-related transfer function filter

[0039] 100: System

[0040] 110: Head-mounted device

[0041] 112: Display Module

[0042] 114: First terminal

[0043] 116: Second terminal

[0044] 120A: First loudspeaker

[0045] 120B: Second speaker

[0046] 130: Control element

[0047] 210: Communication Interface

[0048] 220: Position tracking circuit

[0049] 230: Communication Interface

[0050] 240: Position tracking circuit

[0051] 250: Audio output circuit

[0052] 300: Method

[0053] S301-S310 Operation

[0054] 410: Dummy Head

[0055] 420: Actual Frequency Response

[0056] 430: Sensor

[0057] 440: Ideal Frequency Response

[0058] 510: Adaptive Filter

[0059] 610: User

[0060] 620a, 620b, and 620c: Actual Frequency Response

[0061] 630: Ideal Frequency Response

[0062] 640: Ideal Location

[0063] 650a, 650b, and 650c: Position

[0064] 700: Virtual Environment

[0065] 710: The First Virtual Sound Source

[0066] 720: Second Virtual Sound Source

[0067] PA: First position

[0068] PB: Second position

[0069] asA: First audio signal

[0070] asB: Second audio signal

[0071] F_asA: The first audio signal after filtering

[0072] F_asB: The filtered second audio signal Detailed Implementation

[0073] The spirit of this application will be clearly explained below with illustrations and detailed description. Anyone skilled in the art can make changes and modifications based on the technology taught in this application after understanding the embodiments of this application, without departing from the spirit and scope of this application.

[0074] The language used herein is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. Singular forms such as “a,” “this,” “this,” “the,” and “the”, as used herein, also include multiple forms.

[0075] The terms "coupled" or "connected" as used in this article can refer to two or more components or devices making direct physical contact with each other, or making indirect physical contact with each other, or to two or more components or devices operating or moving together.

[0076] The terms "include", "include", "have", "contain", etc., used in this article are all open-ended terms, meaning they include but are not limited to.

[0077] The use of "and / or" in this document includes any or all of the things mentioned.

[0078] Unless otherwise specified, the terms used herein generally have their ordinary meaning in the context of the art, the application, and the specific content thereof. Certain terms used to describe this application will be discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art in describing this application.

[0079] Figure 1 This is a schematic side view of a system 100 with sound adjustment capabilities according to an embodiment of the present disclosure. The system 100 includes a head-mounted device 110, a first speaker 120A, a second speaker 120B, and a control element 130 including at least one processor. In this embodiment, the head-mounted device 110 is an augmented reality (AR) device and / or a virtual reality (VR) device, which includes a display module 112 to project virtual objects into the user's field of vision in an augmented reality (AR) application and / or to provide the user with an immersive virtual environment in a virtual reality (VR) application. In some embodiments, the head-mounted device 110 may also be implemented as a headband portion of headphones.

[0080] The first speaker 120A and the second speaker 120B are coupled to a first terminal 114 and a second terminal 116 respectively disposed on opposite surfaces of the head-mounted device 110, and are detachable from the head-mounted device 110. When the first speaker 120A and the second speaker 120B are coupled to the head-mounted device 110, the first speaker 120A and the second speaker 120B are positioned at the entrances of the left and right ear canals of the corresponding user. On the other hand, when the first speaker 120A and the second speaker 120B are detached from the head-mounted device 110, the first speaker 120A and the second speaker 120B can be operated as speakers capable of providing stereo sound to the user wearing the head-mounted device 110.

[0081] Control element 130 is configured to provide video signals to head-mounted device 110 to drive display module 112, and to modulate first audio signal asA and second audio signal asB (e.g., Figure 2 (As shown). The modulation can be achieved by applying a filter to the first audio signal asA and the second audio signal asB to generate a filtered first audio signal F_asA and filtered second audio signals F_asB, which are used to drive the first speaker 120A and the second speaker 120B, respectively. The filtering path described above can be performed by the control element 130, which will be described in detail in later paragraphs. The control element 130 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic elements. In some embodiments, the control element 130 may include one or more components that are partially or wholly integrated into the head-mounted device 110, that is, the head-mounted device 110 can be an integrated head-mounted device with sufficient computing power.

[0082] Figure 2 According to an embodiment of this disclosure Figure 1 A simplified functional block diagram of the system. The head-mounted device 110 includes a communication interface 210, a position tracking circuit 220, and a display module 112. The head-mounted device 110 is communicatively coupled to the control element 130 via the communication interface 210 to receive video signals. The position tracking circuit 220 generates position and orientation information for processing by the control element 130, enabling the control element 130 to determine the precise position and orientation of the head-mounted device 110 in the physical environment.

[0083] The first speaker 120A and the second speaker 120B are similar to each other, therefore, only the components and connections of the first speaker 120A will be described in detail below. The first speaker 120A includes a communication interface 230, a position tracking circuit 240, and an audio output circuit 250. The communication interface 230 is configured to communicate with the control element 130 to receive a filtered first audio signal F_asA therefrom. In some embodiments, the communication interface 230 is used to communicate with the communication interface 210 of the head-mounted device 110 to indirectly receive the filtered first audio signal F_asA through the head-mounted device 110. The position tracking circuit 240 is used to generate position information and orientation information, which are processed by the control element 130 so that the control element 130 can determine the position and orientation of the first speaker 120A relative to the head-mounted device 110. The audio output circuit 250 is configured to generate sound based on the filtered first audio signal F_asA.

[0084] In some embodiments, communication interfaces 210 and 230 may be wired or wireless interfaces, such as Bluetooth, ZigBee, or Ethernet.

[0085] In some embodiments, location tracking circuits 220 and 240 may include a plurality of optical sensors configured to sense invisible light (e.g., infrared light) emitted by a plurality of base stations (e.g., lighthouses) arranged in a physical environment.

[0086] In some embodiments, position tracking circuits 220 and 240 may be radio frequency (RF) transceivers suitable for ultra-wideband positioning. For example, position tracking circuits 220 and 240 may communicate with each other via ultra-wideband signals, thereby obtaining the position and orientation of the first speaker 120A relative to the head-mounted device 110 via time-of-flight ranging.

[0087] Control element 130 is configured to receive a first audio signal asA and a second audio signal asB, wherein the first audio signal asA and the second audio signal asB respectively carry audio data of the first speaker 120A and the second speaker 120B. Control element 130 is also configured to apply one or more filters to the first audio signal asA and the second audio signal asB according to the connection state of the first speaker 120A and the second speaker 120B (i.e., coupled to or disconnected from the head-mounted device 110) to change the first audio signal asA and the second audio signal asB at one or more frequencies. Such filters include, but are not limited to, a headphone effect filter 23, a loudspeaker effect filter 24, a position compensation filter 25, a crosstalk cancellation filter 26, and a head-related transfer function (HRTF) filter 27, which can be stored in a memory accessible by control element 130.

[0088] Figure 3 This is a flowchart of a sound adjustment method 300 according to an embodiment of the present disclosure. Any combination of the features of method 300 or any other method described herein can be implemented by instructions and stored in a non-transitory computer-readable medium. When, for example, by Figure 1 When at least one processor of the control element 130 is executed, the instructions may cause some or all of such methods to be executed. It should be understood that any method discussed herein may include more or fewer operations than shown in the flowchart, and these operations may be executed in any appropriate order.

[0089] In operation S301, position and orientation information of the head-mounted device 110, the first speaker 120A, and the second speaker 120B are obtained, for example, via position tracking circuits 220 and 240. In some embodiments, one or more sensors, such as accelerometers and gyroscopes, may be incorporated into these devices of system 100 to help provide orientation information.

[0090] In operation S302, it is determined whether the first speaker 120A and the second speaker 120B are physically coupled to the head-mounted device 110. For example, the control element 130 can receive and process position information and orientation information to determine the position of the first speaker 120A and the second speaker 120B relative to the head-mounted device 110. The control element 130 can select filters applied to the first audio signal asA and the second audio signal asB based on the connection status of the first speaker 120A and the second speaker 120B.

[0091] When the first speaker 120A and the second speaker 120B are coupled to the headband device 110 for operation as headphones, operations S303-S306 can be performed, applying at least one of the headphone effect filter 23 and the position compensation filter 25 to process the first audio signal asA and the second audio signal asB. On the other hand, when the first speaker 120A and the second speaker 120B are separated from the headband device 110 for operation as speakers, operations S307-S310 can be performed, applying at least one of the speaker effect filter 24, the crosstalk cancellation filter 26, and the HRTF filter 27.

[0092] In operation S303, headphone effect filter 23 is applied to the first audio signal asA and the second audio signal asB. Headphone effect filter 23 is configured to reduce sound distortion caused by the coupling of the first speaker 120A and the second speaker 120B to the headphone device 110 (hereinafter referred to as the "headphone configuration"), wherein the distortion is at least partially caused by the circuitry of the headphone configuration (i.e., the circuitry used to couple the headphone device 110, the first speaker 120A, and the second speaker 120B to each other).

[0093] Figure 4 This is an example of the frequency response when an earphone configuration according to an embodiment of the present disclosure is worn on a dummy head 410. Figure 5 This is an example diagram of an adaptive filter 510 according to an embodiment of the present disclosure. (See reference...) Figure 4 and Figure 5 An exemplary method for generating the headphone effect filter 23 is described below. First, headphones are fitted onto a dummy head 410, and the actual frequency response 420 of the first speaker 120A is obtained via a sensor 430 in the left ear canal of the dummy head 410. Next, this actual frequency response 420 is input as input x(n) to an adaptive filter 510 to adjust the coefficients of the adaptive filter 510. When the output of the adaptive filter 510... Essentially matching the ideal frequency response 440 (by...) Figure 5 When the ideal output y(n) is represented in the headphone effect filter 23, the coefficients of the adaptive filter 510 are stored in the headphone effect filter 23 for the coefficients of the first speaker 120A. Figure 5 The interference v(n) in the headphone effect filter 23 can be any unwanted noise, such as noise from the power supply. The coefficients for the second speaker 120B in the headphone effect filter 23 can be obtained in a similar manner to those described for the first speaker 120A, and therefore these descriptions are omitted here. In some embodiments, the headphone effect filter 23 can also be generated using a neural network model, which can be achieved by using the actual frequency response 420 as input to the neural network.

[0094] In some embodiments, the first and second audio signals asA and asB filtered by the headphone effect filter 23 can be provided to the first and second speakers 120A and 120B, respectively, as the filtered first and second audio signals F_asA and F_asB, or the first and second audio signals asA and asB can be further processed by one or more operations S304-S306. By comparing the actual frequency response 420 with the ideal frequency response 440, it can be confirmed that the sound generated by the first and second audio signals asA and asB filtered by the headphone effect filter 23 has reduced the distortion at the entrance of the user's ear canal compared to the sound generated by the unfiltered audio signal. More specifically, the sound generated by the first and second audio signals asA and asB filtered by the headphone effect filter 23 has an enhanced (i.e., flat) frequency response compared to the sound generated by the unfiltered audio signal.

[0095] In operation S304, the position information and orientation information are used to determine whether the first speaker 120A and the second speaker 120B are coupled to the correct terminals of the head-mounted device 110. The control element 130 can check whether the positions of the first speaker 120A and the second speaker 120B correspond to the channels of the filtered first audio signal F_asA and the filtered second audio signal F_asA.

[0096] For example, the filtered first audio signal F_asA may correspond to the right channel, and control element 130 may check whether the first speaker 120A is coupled to the second terminal 116 (e.g., the right terminal corresponding to the right channel). The filtered second audio signal F_asB may correspond to the left channel, and control element 130 may check whether the second speaker 120B is coupled to the first terminal 114 (e.g., the left terminal corresponding to the left channel). If the determination result of operation S304 is "yes", then operation 305 is omitted and operation S306 can be performed. If the determination result of operation S304 is "no" (e.g., ...), Figure 4 If the headphone configuration results in a "no" result, then operation S305 can be performed.

[0097] In operation S305, the filtered first audio signal F_asA and the filtered second audio signal F_asB received by the first speaker 120A and the second speaker 120B, respectively, can be exchanged. For example, the control element 130 can transmit the filtered first audio signal F_asA previously transmitted to the first speaker 120A to the second speaker 120B, and the filtered second audio signal F_asB previously transmitted to the second speaker 120B to the first speaker 120A. Accordingly, the system 100 allows the user to couple the first and second speakers 120A and 120B to the headset 110 in any manner without distorting the sound, enabling rapid assembly of the headphone configuration to maintain an immersive experience.

[0098] In operation S306, position compensation can be performed on the first audio signal asA and the second audio signal asB after being filtered by the headphone effect filter 23. Figure 6 This is an example diagram showing the frequency response of an earphone configuration worn on the head of a user 610 according to an embodiment of this disclosure. (Refer to...) Figure 6 An exemplary method for position compensation is illustrated below. First, control element 130 obtains the actual frequency response 620a of the sound echo generated by first speaker 120A according to a reference audio signal. Such an echo can be received by an audio sensor (e.g., a microphone) of first speaker 120A. Next, if the actual frequency response 620a differs significantly from an ideal frequency response 630 stored in memory accessible to control element 130, control element 130 can generate a position compensation filter 25 based on the actual frequency response 620a and the ideal frequency response 630. The position compensation filter 25 is configured to modify the reference signal at one or more frequencies so that the echo has a modified frequency response substantially the same as the ideal frequency response 630. The coefficients for the first speaker 120A in the position compensation filter 25 can be obtained by using a method similar to... Figure 5 The position compensation filter 25 is generated using the adaptive filter discussed herein, but this disclosure is not limited thereto. In some embodiments, the position compensation filter 25 may be generated by a neural network by using the actual frequency response 620a as input to the neural network.

[0099] The ideal frequency response 630 can be considered as the frequency response obtained at the ideal position 640 corresponding to the entrance of the user's ear canal, and the difference between the actual frequency response 620a and the ideal frequency response 630 is because the position 650a of the first speaker 120A deviates from the ideal position 640. Figure 6As shown, different positions 650a, 650b, and 650c of the first speaker 120A may result in different actual frequency responses 620a, 620b, and 620c for the aforementioned echo. Therefore, the control element 130 can adaptively adjust the coefficients in the position compensation filter 25 for the first speaker 120A according to the current position of the first speaker 120A. The coefficients in the position compensation filter 25 for the second speaker 120B can also be obtained in the same manner as for the first speaker 120A, so related descriptions are omitted.

[0100] The first and second audio signals asA and asB, processed by operations S303-S306, are output by control element 130 as filtered first and second audio signals F_asA and F_asB, respectively. Therefore, the user does not need to adjust the first and second speakers 120A and 120B to the absolutely correct position every time they connect the first and second speakers 120A and 120B to the head-mounted device 110, because the system 100 can automatically compensate for the audio according to the user's wearing condition.

[0101] Refer again Figure 3 The filtering process of the first speaker 120A and the second speaker 120B (hereinafter referred to as the "speaker configuration") after being separated from the head-mounted device 110 will be described in detail below.

[0102] In operation S307, speaker effect filter 24 is applied to the first audio signal asA and the second audio signal asB. Speaker effect filter 24 is configured to eliminate distortion caused at least partially by the circuitry of the speaker configuration (e.g., including circuitry separating the headset 110, the first speaker 120A, and the second speaker 120B) to obtain a flat frequency response. The coefficients in speaker effect filter 24 for the first speaker 120A can be generated by an exemplary method comprising the steps of: (1) placing the first speaker 120A in a non-echo chamber, (2) obtaining the actual frequency response of the sound generated by the first speaker 120A, and (3) using a method similar to a reference... Figure 5 The adaptive filter discussed obtains filter coefficients for the first loudspeaker 120A based on the actual frequency response and the ideal frequency response stored in a memory accessible to the control element 130.

[0103] Different distances between the user and the first speaker 120A may result in different frequency responses and may require different levels of filtering. In some embodiments, multiple sets of coefficients for the speaker effect filter 24 can be generated using the method described above, and the control element 130 can select one set of coefficients as the coefficients in the speaker effect filter 24 for the first speaker 120A based on the distance between the first speaker 120A and the head-mounted device 110. The coefficients in the speaker effect filter 24 for the second speaker 120B can be generated in a similar manner, therefore, related descriptions are omitted.

[0104] In some embodiments, the first and second audio signals asA and asB filtered by the speaker effect filter 24 can be provided to the first and second speakers 120A and 120B respectively as the filtered first and second audio signals F_asA and F_asB, or the first and second audio signals F_asA and F_asB can be further processed by one or more of operations S308-S310.

[0105] In operation S308, it is determined whether the first speaker 120A and the second speaker 120B are in positions corresponding to the channels of the filtered first audio signal F_asA and the filtered second audio signal F_asB received by them. Figure 7 It shows the result of Figure 1 A schematic diagram of the virtual environment 700 provided by the head-mounted device 110 is provided to illustrate operation S308. When the first virtual sound source 710 is at a first position PA in the physical environment, the filtered second audio signal F_asB may have a channel corresponding to the first virtual sound source 710, which is configured to be heard by the user. When the second virtual sound source 720 is at a second position PB in the physical environment, the filtered first audio signal F_asA may have a channel corresponding to the second virtual sound source 720, which is configured to be heard by the user. The head-mounted device 110 may be located approximately between the first position PA and the second position PB. In this case, the control element 130 may check whether the first speaker 120A corresponds to (e.g., is close to) the second position PB specified by the filtered first audio signal F_asA, and whether the second speaker 120B corresponds to (e.g., is close to) the first position PA specified by the filtered second audio signal F_asB. If the result of operation S308 is "yes", then operation S309 can be omitted and operation S310 can be performed. If the result of operation S308 is "no" (for example, Figure 7 If the speaker configuration results in a "no" result, then operation S309 can be performed.

[0106] In operation S309, the filtered first audio signal F_asA and the filtered second audio signal F_asB received by the first speaker 120A and the second speaker 120B respectively can be exchanged. Figure 8 It shows the result of Figure 1 Another example diagram illustrating the virtual environment provided by a head-mounted device. (See diagram for reference.) Figure 8 As shown, the filtered first audio signal F_asA, having a channel corresponding to the second position PB, is transmitted to the second speaker 120B located at the second position PB instead of the first speaker 120A. The filtered second audio signal F_asB, having a channel corresponding to the first position PA, is transmitted to the first speaker 120A located at the first position PA instead of the second speaker 120B.

[0107] In operation S310, crosstalk cancellation filter 26 and HRTF filter 27 are applied to the first audio signal asA and the second audio signal asB after being filtered by speaker effect filter 24. Crosstalk cancellation filter 26 can render the sound of the first speaker 120A and the second speaker 120B as if they were in a headphone configuration to provide realistic binaural sound. For example, in Figure 8 In this case, with the first speaker 120A on the user's left side, the crosstalk cancellation filter 26 can reduce the portion of the sound from the first speaker 120A that reaches the user's right ear. The HRTF filter 27 is configured to render the sound of the first speaker 120A and the second speaker 120B as if they were produced by the first speaker 120A and the second speaker 120B, which are symmetrically placed on both sides of the head-mounted device 110.

[0108] The position and orientation of the speaker relative to the user can affect the interaural time difference (ITD), interaural level difference (ILD), and frequency response of the sound source reaching the listener's ears. Therefore, in some embodiments, the control element 130 can use an adaptive filter to obtain the coefficients of the crosstalk cancellation filter 26 and the HRTF filter 27 based on the position and orientation of the head-mounted device 110, the first speaker 120A, and the second speaker 120B, similar to the reference filter. Figure 5 The discussion in the middle.

[0109] The first and second audio signals asA and asB processed by operations S307-S310 can be output by control element 130 as filtered first and second audio signals F_asA and F_asB, respectively. Accordingly, system 100 allows the user to place the first speaker 120A and the second speaker 120B in any position and orientation without distorting the sound, enabling rapid speaker configuration to maintain an immersive experience. Furthermore, the speaker configuration allows the user to hear sounds from the physical environment and broadcast sound to others, helping to improve communication efficiency in various situations (e.g., meetings or games).

[0110] Although this application discloses the embodiments as described above, it is not intended to limit this application. Those skilled in the art can make various modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application shall be determined by the appended claims.

Claims

1. A system having sound adjustment capability, characterized by, Comprising: a head-mounted device comprising a display module to project a virtual object; a first speaker, wherein the first speaker is detachable from the head-mounted device; a second speaker, wherein the second speaker is detachable from the head-mounted device; and at least one processor configured to detect a plurality of positions and a plurality of orientations of the head-mounted device, the first speaker, and the second speaker to determine whether the first speaker is detached from the head-mounted device and whether the second speaker is detached from the head-mounted device, and to generate a filtered first audio signal by filtering a first audio signal with at least one first filter or at least one second filter, and to generate a filtered second audio signal by filtering a second audio signal with the at least one first filter or the at least one second filter, wherein the at least one first filter is configured to respond to the first speaker being coupled to the head-mounted device, and the at least one first filter is configured to respond to the second speaker being coupled to the head-mounted device, wherein the at least one second filter is configured to respond to the first speaker being detached from the head-mounted device, and the at least one second filter is configured to respond to the second speaker being detached from the head-mounted device, wherein the at least one processor is configured to determine whether the positions of the first speaker and the second speaker correspond to correct channels, respectively, based on a virtual sound source position corresponding to the virtual object and the plurality of positions and the plurality of orientations, wherein the at least one processor is configured to transmit the filtered first audio signal to the first speaker to drive the first speaker and the filtered second audio signal to the second speaker to drive the second speaker when the positions of the first speaker and the second speaker correspond to the correct channels, respectively. wherein the at least one processor is configured to adjust the first audio signal at one or more frequencies to generate a sound, wherein the sound generated by the first speaker based on the filtered first audio signal has an enhanced frequency response at an entrance of a user's ear compared to a sound generated by the first speaker based on an unfiltered audio signal.

2. The system having sound adjustment capability as claimed in claim 1, wherein, wherein the at least one first filter comprises a headphone effect filter configured to cancel distortion caused at least in part by a circuit of the head-mounted device and the first speaker when coupled together.

3. The system having sound adjustment capability as claimed in claim 1, wherein, wherein the at least one second filter comprises a speaker effect filter configured to cancel distortion caused at least in part by a circuit of the head-mounted device and the first speaker when detached from the head-mounted device.

4. The system having sound adjustment capability as claimed in claim 1, wherein, wherein the at least one processor is configured to select coefficients for the first speaker in the speaker effect filter based on a distance between the first speaker and the head-mounted device.

5. The system having sound adjustment capability as claimed in claim 4, wherein, further comprising a memory, wherein in response to the first speaker being coupled to the head-mounted device, the at least one processor is configured to obtain an actual frequency response of an echo of a sound generated by the first speaker based on a reference audio signal, 6. The system having sound adjustment capability as claimed in claim 1, wherein, ​ In response to a difference between the actual frequency response and an ideal frequency response stored in the memory, the at least one processor is configured to apply a position compensation filter of the at least one first filter to the first audio signal, wherein the position compensation filter is configured to render the echo to have a modulated frequency response that is substantially the same as the ideal frequency response.

7. The system having sound adjustment capability as claimed in claim 1, wherein, In response to the first speaker and the second speaker being coupled to opposite first and second terminals of the headset, respectively, the at least one processor is configured to, in response to determining that the positions of the first speaker and the second speaker do not correspond to the correct ones of the plurality of channels, and in response to the at least one processor determining that the filtered first audio signal has the channel corresponding to the second terminal, transmit the filtered second audio signal previously transmitted to the second speaker to the first speaker and transmit the filtered first audio signal to the second speaker.

8. The system having sound adjustment capability as claimed in claim 1, wherein, In response to the first speaker and the second speaker being uncoupled from the headset and positioned at a first position and a second position, respectively, the headset being substantially between the first position and the second position, the at least one processor is configured to, in response to determining that the positions of the first speaker and the second speaker do not correspond to the correct ones of the plurality of channels, and in response to the at least one processor determining that the filtered first audio signal has the channel corresponding to the second position, transmit the filtered second audio signal previously transmitted to the second speaker to the first speaker and transmit the filtered first audio signal to the second speaker.

9. The system having sound adjustment capability as claimed in claim 1, wherein, The at least one second filter includes a crosstalk cancellation filter and a head-related transfer function (HRTF) filter.

10. The system having sound adjustment capability as claimed in claim 9, wherein, The at least one processor is configured to obtain coefficients in the crosstalk cancellation filter and the HRTF filter based on the plurality of positions and the plurality of orientations.

11. A method of adjusting sound, suitable for use in a system comprising a head-wearable device and a first speaker and a second speaker detachable from the head-wearable device, characterized in that, The method includes: detecting a plurality of positions and a plurality of orientations of the headset and the first speaker to determine whether the first speaker is uncoupled from the headset; detecting a plurality of positions and a plurality of orientations of the headset and the second speaker to determine whether the second speaker is uncoupled from the headset; modulating a first audio signal by at least one first filter or at least one second filter to generate a filtered first audio signal, wherein the at least one first filter is used in response to the first speaker being coupled to the headset, and the at least one second filter is used in response to the first speaker being uncoupled from the headset; modulating a second audio signal by the at least one first filter or the at least one second filter to generate a filtered second audio signal, wherein the at least one first filter is used in response to the second speaker being coupled to the headset, and the at least one second filter is used in response to the second speaker being uncoupled from the headset; determining whether the first speaker and the second speaker are positioned to correspond to the correct plurality of sound channels based on a virtual sound source location of a virtual object projected by the head-wearable device and the plurality of locations and the plurality of orientations; and transmitting the filtered first audio signal to the first speaker to drive the first speaker and transmitting the filtered second audio signal to the second speaker to drive the second speaker when the first speaker and the second speaker are determined to be positioned to correspond to the correct plurality of sound channels.

12. The method of adjusting sound of claim 11, wherein, wherein modifying the first audio signal includes modifying the first audio signal at one or more frequencies to generate a sound, wherein the sound generated by the first speaker from the filtered first audio signal has an enhanced frequency response at an entrance to an ear of a user compared to a sound generated by the first speaker from the unfiltered audio signal.

13. The method of adjusting sound of claim 11, wherein, wherein the at least one first filter includes a headphone effect filter configured to remove distortion caused at least in part by a circuit coupling the head-wearable device and the first speaker to one another.

14. The method of adjusting sound of claim 11, wherein, wherein the at least one second filter includes a speaker effect filter configured to remove distortion caused at least in part by a circuit coupling the head-wearable device and the first speaker apart from one another.

15. The method of adjusting sound of claim 14, wherein, wherein a coefficient of the speaker effect filter for the first speaker is selected based on a distance between the first speaker and the head-wearable device.

16. The method of adjusting sound of claim 11, wherein, wherein the system further includes a memory, and wherein modifying the first audio signal further includes: obtaining an actual frequency response of an echo of a sound generated by the first speaker from a reference audio signal in response to the first speaker being coupled to the head-wearable device; and applying a position compensation filter of the at least one first filter to the first audio signal in response to a difference between the actual frequency response and an ideal frequency response stored in the memory, wherein the position compensation filter is configured to render the echo such that the echo has a modified frequency response that is substantially the same as the ideal frequency response.

17. The method of adjusting sound of claim 11, wherein, The method further includes: transmitting the filtered second audio signal to the first speaker and transmitting the filtered first audio signal to the second speaker in response to the first speaker and the second speaker being coupled to a first and a second terminal, respectively, on opposite sides of the head-wearable device, when the first speaker and the second speaker are determined not to be positioned to correspond to the correct plurality of sound channels and in response to the filtered first audio signal having the sound channel corresponding to the second terminal.

18. The method of adjusting sound of claim 11, wherein, The method further includes: In response to the first speaker and the second speaker being detached from the headset and positioned at a first position and a second position, respectively, the headset is positioned substantially between the first position and the second position, determining that the positions of the first speaker and the second speaker do not correspond to the correct channels, and in response to the filtered first audio signal having the channels corresponding to the second position, transmitting the filtered second audio signal previously transmitted to the second speaker to the first speaker and transmitting the filtered first audio signal to the second speaker.

19. The method of adjusting sound of claim 11, wherein, wherein the at least one second filter comprises a crosstalk cancellation filter and a head-related transfer function (HRTF) filter.

20. A non-transitory computer-readable storage medium storing a plurality of computer-readable instructions for controlling a system comprising at least one processor, a head-mounted device, and a first speaker and a second speaker detachable from the head-mounted device, wherein when the plurality of computer-readable instructions are executed by the at least one processor, the system is caused to perform: The at least one processor is configured to perform: detecting a plurality of positions and a plurality of orientations of the headset and the first speaker to determine whether the first speaker is detached from the headset; detecting a plurality of positions and a plurality of orientations of the headset and the second speaker to determine whether the second speaker is detached from the headset; filtering a first audio signal through at least one first filter or at least one second filter to generate a filtered first audio signal, wherein the at least one first filter is configured to respond to the first speaker being coupled to the headset and the at least one second filter is configured to respond to the first speaker being detached from the headset; filtering a second audio signal through the at least one first filter or the at least one second filter to generate a filtered second audio signal, wherein the at least one first filter is configured to respond to the second speaker being coupled to the headset and the at least one second filter is configured to respond to the second speaker being detached from the headset; determining whether the positions of the first speaker and the second speaker correspond to the correct channels according to a virtual sound source position of a virtual object projected by the headset and the plurality of positions and the plurality of orientations; and in response to the first speaker and the second speaker being detached from the headset and positioned at a first position and a second position, respectively, the headset is positioned substantially between the first position and the second position, determining that the positions of the first speaker and the second speaker do not correspond to the correct channels, and in response to the filtered first audio signal having the channels corresponding to the second position, transmitting the filtered second audio signal previously transmitted to the second speaker to the first speaker and transmitting the filtered first audio signal to the second speaker. ​

Images