Ultrasonic-based audio playback method and ultrasonic-based electronic device

Abstract

This invention application discloses an audio playback method based on ultrasound and an electronic device based on ultrasound. The audio playback method includes: receiving an ultrasonic signal through a microphone; processing the ultrasonic signal to obtain characteristic values ​​of the ultrasonic signal; generating time difference information based on the characteristic values; and controlling the audio delay according to the time difference information.

Description

Technical Field

[0001] This invention application relates to audio signals, and more particularly to playback delay compensation for audio signals and volume control of speakers (sometimes also called loudspeakers). Background Technology

[0002] Figure 1 This is a schematic diagram illustrating the transmission of audio signals over a network. Assuming a remote device (with a microphone) and a near-end device (with a speaker 160) are connected via a network, the remote audio 110 may experience a microphone / audio delay 120, a network delay 130, an audio delay 140, and a system delay 150 on the near-end device before being played by the speaker 160. The microphone / audio delay 120 refers to the time required for the microphone to pick up sound and generate an audio signal, or more specifically, for audio processing (such as noise cancellation). The audio delay 140 refers to the time required for the near-end device's audio software to process the audio signal. The system delay 150 refers to the time required for the near-end device's operating system to process the audio signal.

[0003] When multiple devices receive and play the same remote audio in the same space, the sounds played by these devices may be out of sync due to their different network latency (130), audio latency (140), and system latency (150), resulting in a poor user experience. Furthermore, if the playback volumes from these devices are all too loud, the overall environment will become too noisy, and even howling may occur, making it difficult for the user to hear the content of the broadcast audio. Summary of the Invention

[0004] One object of the present invention is to provide an ultrasonic-based audio playback method and an ultrasonic-based electronic device to improve upon the shortcomings of the prior art.

[0005] This invention discloses an audio playback method based on ultrasound, comprising: receiving an ultrasound signal through a microphone; processing the ultrasound signal to obtain feature values ​​of the ultrasound signal; generating time difference information based on the feature values; and delaying the playback of the audio signal according to the time difference information.

[0006] The present invention also discloses an ultrasonic-based audio playback method, comprising: receiving an ultrasonic signal played by a microphone device; processing the ultrasonic signal to obtain the energy and phase difference of the ultrasonic signal; estimating the distance and direction of the device based on the energy and the phase difference; and adjusting the volume of a speaker or sending a volume adjustment signal based on the distance and the direction.

[0007] The present invention also discloses an ultrasonic-based electronic device, comprising: a microphone; memory for storing a plurality of program codes or program instructions; and a computing circuit coupled to the microphone and the memory for executing the plurality of program codes or program instructions to perform the following steps: receiving an ultrasonic signal through the microphone; processing the ultrasonic signal to obtain a feature value of the ultrasonic signal; generating time difference information based on the feature value; and delaying the playback of an audio signal according to the time difference information.

[0008] The features, implementation, and effects of this invention will be described in detail below with reference to the accompanying drawings. Attached Figure Description

[0009] To make the above-mentioned and other objects, features, advantages and embodiments of this invention application clearer and easier to understand, the following description will be made in conjunction with the accompanying drawings.

[0010] Figure 1 This is a diagram illustrating the transmission of audio signals over a network;

[0011] Figure 2 This is a functional block diagram of one embodiment of the electronic device of the present invention;

[0012] Figure 3 This is a flowchart of an embodiment of the ultrasonic-based audio playback method of the present invention;

[0013] Figure 4 This is a flowchart of one embodiment of step S330;

[0014] Figure 5 This is a flowchart of another embodiment of step S330;

[0015] Figure 6 This is a flowchart of another embodiment of the ultrasonic-based audio playback method of the present invention;

[0016] as well as

[0017] Figure 7 This is a schematic diagram showing the distance and direction between the near-end device and the local device.

[0018] Symbol Explanation

[0019] 160, 250: Speakers

[0020] 110: Remote Audio

[0021] 120: Microphone / Audio Delay

[0022] 130: Network latency

[0023] 140: Sound Delay

[0024] 150: System latency

[0025] 200: Electronic devices

[0026] 210: Network transceiver circuit

[0027] 220: Calculation Circuit

[0028] 230: Memory

[0029] 240: Microphone

[0030] S_aud_fr: Remote audio signal

[0031] S_ult_lc: Local ultrasonic signal

[0032] S_aud_lc: Local audio signal

[0033] S_ult_nr: Proximal ultrasound signal

[0034] S_aud_nr: Near-end audio signal

[0035] Ta_r, Tb_r: Arrival time information

[0036] Ta_t, Tb_t: Playback time information

[0037] 710, 730: Proximal devices

[0038] 720: Local device

[0039] DB: Broadcasting Direction

[0040] DL: Online Direction

[0041] DD: Distance

[0042] SV_adj, SV_adj1, SV_adj2: Volume adjustment signals

[0043] S310, S320, S330, S340, S410, S420, S510, S520, S530, S540, S550, S610, S620, S630, S640, S650, S660: Steps Detailed Implementation

[0044] The technical terms used in the following description are conventional terms in this technical field. If this specification provides explanations or definitions for certain terms, the explanations or definitions in this specification shall prevail.

[0045] The disclosure of this invention application includes an ultrasound-based audio playback method and an ultrasound-based electronic device. Since some components of the electronic device of this invention may be known individually, details of known components will be omitted in the following description without affecting the full disclosure and implementability of the invention. Furthermore, some or all of the processes of the ultrasound-based audio playback method of this invention can be in the form of software and / or firmware, and can be executed by the electronic device of this invention or its equivalent. Without affecting the full disclosure and implementability of the method invention, the following description of the method invention will focus on the steps rather than the hardware.

[0046] Figure 2 This is a functional block diagram of an embodiment of the electronic device of the present invention. The electronic device 200 includes a network transceiver circuit 210, a computing circuit 220, a memory 230, a microphone 240, and a speaker 250. The computing circuit 220 receives a remote audio signal S_aud_fr from the network via the network transceiver circuit 210, plays a local ultrasonic signal S_ult_lc and a local audio signal S_aud_lc via the speaker 250, and receives a near-end ultrasonic signal S_ult_nr and a near-end audio signal S_aud_nr via the microphone 240. In this specification, "near-end" refers to the space where electronic device 200 is located (e.g., a conference room or classroom), where users of electronic device 200 can directly hear the near-end audio signal S_aud_nr played by other devices (i.e., near-end devices) in the same space; "far-end" refers to a space other than where electronic device 200 is located (e.g., a conference room or classroom in another location), where users of electronic device 200 cannot directly hear the far-end audio signal S_aud_fr in that space (e.g., generated by users in that space and transmitted to electronic device 200 via a remote device and network); "local" refers to electronic device 200 itself.

[0047] The near-end device and local device referred to in this specification may be a computer (such as a desktop computer or laptop computer), a portable electronic device (such as a tablet computer or mobile phone), or other electronic products with network connectivity and including a microphone and speaker.

[0048] Please see Figure 3 , Figure 3 This is a flowchart of an embodiment of the ultrasonic-based audio playback method of the present invention, which includes the following steps.

[0049] Step S310: The computing circuit 220 of the electronic device 200 receives the near-end ultrasound signal S_ult_nr played by the near-end device (i.e., other devices in the same space as the electronic device 200) via the microphone 240.

[0050] Step S320: The calculation circuit 220 processes the near-end ultrasound signal S_ult_nr to obtain the characteristic values ​​of the near-end ultrasound signal S_ult_nr. For example, the characteristic values ​​are the energy or frequency of the near-end ultrasound signal S_ult_nr.

[0051] Step S330: The calculation circuit 220 generates time difference information based on the characteristic values ​​of the near-end ultrasonic signal S_ult_nr. The time difference information can be used to represent the audio signal delay time between the near-end device and the electronic device 200.

[0052] Step S340: Delay the playback of the local audio signal S_aud_lc based on the time difference information to minimize the delay between the local audio signal S_aud_lc and the near-end audio signal S_aud_nr, thereby providing a better user experience for the user of the electronic device 200. For example, if the time difference information shows a time difference of 1 second, the calculation circuit 220 will delay the received remote audio signal S_aud_fr by 1 second before playing it.

[0053] Please see Figure 4 , Figure 4 This is a flowchart of one embodiment of step S330, which includes the following sub-steps.

[0054] Step S410: The calculation circuit 220 calculates the distance based on the energy of the near-end ultrasonic signal S_ult_nr. Assuming the near-end device sends the near-end ultrasonic signal S_ult_nr at a default energy (e.g., 40 dB, which the electronic device 200 knows in advance), the local device (i.e., the electronic device 200) can determine the distance between the near-end device and the local device by comparing the energy of the received near-end ultrasonic signal S_ult_nr with the default energy. In other words, in this embodiment, the characteristic value of step S330 is the energy of the ultrasonic signal. In some embodiments, the energy of the ultrasonic signal can be represented by the sound pressure level (SPL) of the ultrasonic signal; the higher the energy of the ultrasonic signal, the higher the SPL. Generally, if other factors are not considered, the relationship between the SPL and the distance is as follows:

[0055]

[0056] Where SPL_1 represents the sound measured at the first point, with a distance of d_1 from the sound source, and SPL_2 represents the sound measured at the second point, with a distance of d_2 from the sound source. Based on equation (1) and the characteristic value (i.e., sound pressure level or energy) of the near-end ultrasonic signal S_ult_nr, the calculation circuit 220 can determine the distance between the near-end device and the electronic device 200. Those skilled in the art can obtain the sound pressure level of the ultrasonic signal from the following documents: https: / / en.wikipedia.org / wiki / Sound_pressure and https: / / en.wikipedia.org / wiki / Inverse-square_law.

[0057] Step S420: The calculation circuit 220 calculates the time difference information based on the distance and the speed of sound. More specifically, the calculation circuit 220 divides the distance by the speed of sound (e.g., the speed of sound in air at room temperature) to know the time it takes for the near-end ultrasonic signal to travel from the near-end device to the local device. This time is the time difference information in step S330.

[0058] Please see Figure 5 , Figure 5 This is a flowchart of another embodiment of step S330. In this embodiment, the near-end device performs frequency-shift keying on the near-end ultrasound signal S_ult_nr, that is, different frequencies represent different numbers, for example, 21kHz represents the number "0", 22kHz represents the number "1", and so on. In this way, the near-end ultrasound signal S_ult_nr can carry its system playback time (i.e., the time the near-end device plays the near-end ultrasound signal S_ult_nr). Step S330 includes the following sub-steps.

[0059] Step S510: The calculation circuit 220 records the arrival time information Ta_r of the near-end ultrasonic signal S_ult_nr, which is the time point at which the electronic device 200 receives the near-end ultrasonic signal S_ult_nr.

[0060] Step S520: The calculation circuit 220 demodulates the near-end ultrasonic signal S_ult_nr to obtain the playback time information Ta_t of the near-end ultrasonic signal S_ult_nr (i.e., the time point at which the near-end device plays the near-end ultrasonic signal S_ult_nr). In other words, in this embodiment, the characteristic value of step S330 is the frequency of the ultrasonic signal. Demodulation is well known to those skilled in the art and will not be described in detail here.

[0061] Step S530: The calculation circuit 220 plays the local ultrasonic signal S_ult_lc through the speaker 250 and records the playback time information Tb_t of the local ultrasonic signal S_ult_lc (that is, the time point when the electronic device 200 plays the local ultrasonic signal S_ult_lc).

[0062] Step S540: The calculation circuit 220 receives the local ultrasonic signal S_ult_lc through the microphone 240 and records the arrival time information Tb_r of the local ultrasonic signal S_ult_lc, which is the time when the electronic device 200 receives the local ultrasonic signal S_ult_lc.

[0063] Step S550: Calculate the time difference information based on the playback time information Ta_t of the near-end ultrasound signal S_ult_nr, the arrival time information Ta_r of the near-end ultrasound signal S_ult_nr, the playback time information Tb_t of the local ultrasound signal S_ult_lc, and the arrival time information Tb_r of the local ultrasound signal S_ult_lc. More specifically, the time difference information = (Ta_r - Ta_t) - (Tb_r - Tb_t).

[0064] Compared to Figure 4 Implementation examples, Figure 5 The embodiments described above can obtain more accurate time difference information. More specifically, the time difference between the near-end device playing a sound wave (e.g., an ultrasonic signal or an audible audio signal) and the local device receiving the sound wave (i.e., the user simulating the local device hearing the audio signal) includes the near-end device's audio delay, the near-end device's system delay, the path delay (i.e., the time it takes for the sound wave to travel through the air from the near-end device to the local device), and the local device's microphone / audio delay; the time difference between the local device playing a sound wave (e.g., an ultrasonic signal or an audible audio signal) and the local device receiving the sound wave (i.e., the user simulating the local device hearing the audio signal) includes the local device's audio delay, the local device's system delay, and the local device's microphone / audio delay. That is to say, Figure 5 The embodiment takes all the above-mentioned delays into account, so more accurate time difference information can be obtained.

[0065] Please see Figure 6 , Figure 6 The flowchart of another embodiment of the ultrasonic-based audio playback method of the present invention includes the following steps.

[0066] Step S610: The calculation circuit 220 receives the near-end ultrasound signal S_ult_nr through the microphone 240.

[0067] Step S620: The calculation circuit 220 processes the near-end ultrasound signal S_ult_nr to obtain the sound pressure level and phase difference of the near-end ultrasound signal S_ult_nr (please refer to: https: / / en.wikipedia.org / wiki / Phase_(waves)).

[0068] Step S630: The calculation circuit 220 estimates the distance and direction of the near-end device (i.e., estimates the source information about the near-end ultrasonic signal S_ult_nr) based on the sound pressure level and phase difference of the near-end ultrasonic signal S_ult_nr. The calculation circuit 220 can estimate the distance of the near-end device using the sound pressure level based on equation (1), and estimate the direction of the near-end device using the phase difference based on beamforming technology. For beamforming technology, please refer to https: / / en.wikipedia.org / wiki / Beamforming and http: / / www.labbookpages.co.uk / audio / beamforming / delaySum.html, so it will not be described in detail here. In some embodiments, such as Figure 7 As shown, the direction of the near-end device can be represented by the angle θ (0°≤θ≤180°) between the broadcast direction DB of the near-end device 710 (i.e., the direction of the speaker (not shown) of the near-end device 710) and the connection direction DL. The connection direction DL refers to the straight line from the near-end device 710 to the local device 720, and specifically the straight line between the speaker of the near-end device 710 and the local device 720. The distance between the near-end device 710 and the local device 720 is represented by the distance DD.

[0069] Step S640: The calculation circuit 220 adjusts its own volume or sends a volume adjustment signal to the near-end device based on distance and direction. For example, when the distance DD and / or the angle θ is small (large), the near-end device 710 lowers (raises) the volume of its own speaker, and / or instructs the near-end device 710 to lower (raise) the volume using a volume adjustment signal SV_adj, thereby reducing or preventing howling. The volume adjustment signal SV_adj can be an ultrasonic signal (e.g., carrying control information (e.g., target volume) via frequency offset modulation) or a network signal (e.g., transmitting control information via a wired / wireless network). In some embodiments, the control information can be determined empirically, for example, the memory 230 of the electronic device 200 stores a look-up table, and the calculation circuit 220 retrieves the control information from the look-up table using the distance DD and / or the angle θ as an index.

[0070] When multiple devices (e.g., 3 or more devices, such as near-end device 710, local device 720, and near-end device 730) are implemented in the same space (e.g., a conference room or classroom) Figure 6 In the embodiment, the speakers of the multiple devices can be regarded as a speaker array, and after each device (e.g., local device 720) receives volume adjustment signals (e.g., SV_adj1 and SV_adj2) from other devices (e.g., near-end devices 710 and 730) (i.e., step S650: local device 720 receives the volume adjustment signals sent by near-end devices 710 and 730), it can adjust the volume of its own speaker in a way that adjusts the sound field according to the received volume adjustment signals (i.e., step S660: the calculation circuit 220 of local device 720 adjusts the volume of the speakers according to the received volume adjustment signals, for example, by referring to a lookup table), so that the summed sound field is comfortable for the user.

[0071] For example, in some embodiments, based on the near-end spatial configuration (e.g., space size, space shape, interior items, item materials, etc.) and the distance between the speakers of these devices (e.g., multiple electronic devices 200), these devices (e.g., the computing circuit 220 of electronic device 200) can determine the acoustically bright zone (a high-energy area formed by multiple speakers playing at a specific volume and / or directionally in the space) and the acoustically dark zone (a low-energy area formed by multiple speakers playing at a specific volume in the space) that can be formed in the near-end space using their speakers (e.g., multiple speakers 250). Based on these acoustic bright areas, these acoustic dark areas, and the user's position in the near-end space (e.g., in a typical conference room, users are generally positioned at a certain angle and distance in front of their laptops), these devices (including multiple and single devices) can control their corresponding speakers to play at a specific volume and / or directionality, so that the user (including users of multiple near-end devices and users of a single near-end device) can be covered in the acoustic bright areas (including the acoustic bright areas generated by the combined speakers of multiple near-end devices and the acoustic bright areas generated by the speakers of a single near-end device) rather than the acoustic dark areas.

[0072] In some embodiments, the computing circuit 220 may be executed concurrently. Figure 3 and Figure 6 The steps are to delay and control the volume of the speaker 250 based on the time difference information and the volume adjustment signal of itself / other near-end devices, so that the audio played by the speaker 250 has the effect of delay correction and summing of sound field, but the present invention is not limited thereto.

[0073] In some embodiments, the computing circuit 220 may be a circuit or electronic component with program execution capability, such as a central processing unit, microprocessor, microcontroller, microprocessor unit, digital signal processing circuit (DSP), or equivalent circuit. The computing circuit 220 executes program code or program instructions stored in memory 230. Figures 3-6 The steps are as follows. In other embodiments, those skilled in the art can design the computing circuit 220 based on the above disclosure. That is, the computing circuit 220 can be an application-specific integrated circuit (ASIC) or implemented by circuits or hardware such as a programmable logic device (PLD).

[0074] The ultrasonic-based audio playback method of this invention can adjust the sound delay to synchronize sounds in the same space as much as possible, and / or adjust the sound volume to make the overall sound field in the space more comfortable. Compared with traditional technologies, this invention can prevent feedback in the space, thus improving the user experience.

[0075] Since those skilled in the art can understand the implementation details and variations of the method invention in this case through the disclosure of the device invention, to avoid redundancy, repeated descriptions are omitted here without affecting the disclosure requirements and implementability of the method invention. Please note that the shapes, sizes, and proportions of the components in the illustrations disclosed above are merely illustrative and are intended to help those skilled in the art understand the invention, and are not intended to limit the invention. Furthermore, in some embodiments, the order of the steps mentioned in the flowcharts disclosed above may be adjusted according to actual operation, and they may even be performed simultaneously or partially simultaneously.

[0076] Although the contents of this application have been disclosed above through specific embodiments, these embodiments are not intended to limit the contents of this application. Those skilled in the art can modify or adjust the technical solutions of this application based on the explicit or implicit contents of this application without departing from the concept and scope of this application. All such changes may fall within the scope of patent protection sought by this application. In other words, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. An audio playback method based on ultrasound, characterized in that, The ultrasound-based audio playback method includes: The first electronic device receives an ultrasonic signal through its microphone. The ultrasonic signal is a near-end ultrasonic signal emitted by the second electronic device. The second electronic device is in the same space as the first electronic device and is used to play the same remote audio. The second electronic device uses a specific sound pressure level or frequency offset modulation method to make the near-end ultrasonic signal carry the first playback time information when it is played. The ultrasonic signal is processed to obtain its characteristic values; Time difference information is generated based on this feature value; as well as The playback of the audio signal of the remote audio by the first electronic device is delayed based on the time difference information; The step of generating the time difference information based on the feature value includes: Record the first arrival time information of the near-end ultrasonic signal; Demodulate the near-end ultrasound signal to obtain the first playback time information of the near-end ultrasound signal; The local ultrasonic signal is played through a first electronic device, and the second playback time information of the local ultrasonic signal is recorded. The local ultrasonic signal is received through the microphone of the first electronic device, and the second arrival time information of the local ultrasonic signal is recorded; and The time difference information is calculated based on the first playback time information, the first arrival time information, the second playback time information, and the second arrival time information.

2. The method as described in claim 1, characterized in that, The step of generating the time difference information based on the feature value includes: The distance is calculated based on the sound pressure level of the ultrasonic signal; and The time difference information is obtained by dividing the distance by the speed of sound.

3. An ultrasonic-based electronic device, characterized in that, The ultrasound-based electronic device includes: microphone; Memory is used to store multiple program codes or program instructions; and A computing circuit, coupled to the microphone and the memory, is used to execute the plurality of program codes or program instructions to perform the following steps: The microphone receives an ultrasonic signal, which is a near-end ultrasonic signal emitted by the second electronic device; the second electronic device is in the same space as the first electronic device and is used to play the same remote audio; the second electronic device uses a specific sound pressure level or frequency offset modulation method to make the near-end ultrasonic signal carry the first playback time information when it is played. The ultrasonic signal is processed to obtain the characteristic values ​​of the ultrasonic signal; Time difference information is generated based on the aforementioned feature values; as well as The playback of the remote audio signal by the electronic device is delayed based on the time difference information; The computing circuit further performs the following steps to complete the step of generating the time difference information based on the feature value: Record the first arrival time information of the near-end ultrasonic signal; Demodulate the near-end ultrasound signal to obtain the first playback time information of the near-end ultrasound signal; The electronic device plays a local ultrasonic signal and records the second playback time information of the local ultrasonic signal. The local ultrasonic signal is received through the microphone, and the second arrival time information of the local ultrasonic signal is recorded; and The time difference information is calculated based on the first playback time information, the first arrival time information, the second playback time information, and the second arrival time information.

4. The electronic device as claimed in claim 3, characterized in that, The computing circuit further performs the following steps to complete the step of generating the time difference information based on the feature value: The distance is calculated based on the sound pressure level of the ultrasonic signal; and The time difference information is obtained by dividing the distance by the speed of sound.

Images