A kind of audio playing method and related equipment

Abstract

The present application discloses an audio playback method. Specifically, an electronic device obtains audio to be played, then separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played, performs crosstalk elimination on the high-frequency component of at least one audio component of at least one musical instrument, delays and aligns the low-frequency component of at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination, and finally plays the high-frequency component and the audio component of human voice through a first speaker pair with a smaller opening angle, and plays the low-frequency component through a second speaker pair. In this way, the robustness of audio playback is improved.

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to an audio playback method, an audio playback device, a speaker, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] As users' requirements for sound reproduction continue to increase, for example, users hope that the reproduced sound can restore the three-dimensional sense of sound in nature, the stereo in audio technology can restore the three-dimensional sense of sound to a certain extent, thereby providing a certain degree of reproduced sound with spatial distribution characteristics such as directional layering.

[0003] In stereo playback, the signal transmitted on the same side may interfere with the signal transmitted on the opposite side, thereby destroying the spatial information contained in the stereo signal.

[0004] Normally, crosstalk cancellation can be used to eliminate the signals on the opposite side, thereby avoiding interference of the signals on the opposite side on the transmission of the signals on the same side, so that the human ear can obtain stereoscopic sound. However, traditional crosstalk cancellation has defects such as poor robustness, possible harmonic distortion and color staining, and poor effect, which affects the user experience. Summary of the invention

[0005] The present application provides a mixed audio processing method, which can perform frequency band processing on mixed audio, improve the robustness of mixed audio processing, and improve the user experience. The present application also provides a device, a speaker, an electronic device, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In the first aspect, the present application provides an audio playback method, which is applied to an electronic device. Specifically, the electronic device obtains the audio to be played, and then separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played, performs crosstalk elimination on the high-frequency component in at least one audio component of at least one musical instrument, and delays and aligns the low-frequency component and the audio component of human voice in at least one audio component of at least one musical instrument with the high-frequency component after crosstalk elimination, and finally plays the high-frequency component and the audio component of human voice through a first speaker pair with a smaller opening angle, and plays the low-frequency component through a second speaker pair. In this way, the electronic device performs different processing on the low-frequency component, the high-frequency component and the human voice according to the frequency, so as to avoid the head-related transfer function as a transmission matrix for crosstalk elimination. The gain calculated at low frequency is large, so crosstalk elimination is not performed on the low-frequency component, thereby reducing the amount of calculation of audio playback and improving the robustness of the system. In addition, the electronic device replays the low-frequency signal through a large opening angle speaker pair and the high-frequency signal and the human voice signal through a small opening angle speaker pair, thereby improving the stability of audio playback and further improving the robustness of audio processing.

[0008] In some possible implementations, separating at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played includes:

[0009] At least one audio component of the at least one musical instrument and an audio component of human voice are separated from the audio to be played according to the instrument type of the at least one musical instrument.

[0010] In this way, the audio can be separated based on frequency according to the type of musical instrument, and the frequency characteristics of different musical instruments can be used to simplify the audio separation process and improve the processing speed.

[0011] In some possible implementations, when the instrument type of the musical instrument is a low-frequency instrument, the method further includes:

[0012] An audio component of the low-frequency instrument is determined as a low-frequency component.

[0013] In this way, the audio component of the low-frequency instrument can be directly determined as the low-frequency component, thereby increasing the speed of audio separation.

[0014] In some possible implementations, when the instrument type of the musical instrument is a wide-band instrument, the method further includes:

[0015] A high frequency component and a low frequency component are separated from at least one audio component of the wide frequency musical instrument.

[0016] In this way, the high-frequency component and the low-frequency component can be further determined from the audio components of the wide-band musical instrument, thereby improving the accuracy of audio separation and enhancing the user experience.

[0017] In some possible implementations, separating the high-frequency component and the low-frequency component from the at least one audio component of the wide-band musical instrument includes:

[0018] A high-frequency component and a low-frequency component are separated from at least one audio component of the wide-band musical instrument by using a filter bank.

[0019] In this way, the electronic device can realize the division of high-frequency components and low-frequency components in the audio components of broadband musical instruments through the filter group, thereby improving the accuracy of the audio components and the effect of audio playback.

[0020] In some possible implementations, when the instrument type of the musical instrument is not any of the preset types, the method further includes:

[0021] At least one audio component of the at least one musical instrument is determined as at least one of a low-frequency component, a middle-frequency component or a high-frequency component through spectral energy distribution.

[0022] In this way, for audio that cannot be divided into frequencies according to the type of instrument, the electronic device can determine to which frequency component the main frequency of the instrument's audio belongs through the spectral energy distribution, so that it can perform corresponding audio processing according to the divided audio components and select the corresponding speaker pair for playback.

[0023] In some possible implementations, the method further includes:

[0024] Eliminate crosstalk of an intermediate frequency component in at least one audio component of the at least one musical instrument;

[0025] The delay alignment of the low-frequency component in at least one audio component of the at least one musical instrument and the audio component of the human voice with the high-frequency component after crosstalk elimination comprises:

[0026] The low-frequency component of at least one audio component of the at least one musical instrument and the audio component of the human voice are delayed and aligned with the high-frequency component and the middle-frequency component after crosstalk elimination.

[0027] In this way, the electronic device can also perform crosstalk elimination processing on the intermediate frequency components in the audio components, so that the method can play audio in various frequency ranges.

[0028] In some possible implementations, the method further includes:

[0029] The intermediate frequency component is played through a third speaker pair, wherein an opening angle of the third speaker pair is greater than an opening angle of the first speaker pair, and an opening angle of the third speaker pair is smaller than an opening angle of the second speaker pair.

[0030] In this way, the electronic device can utilize more types of speaker pairs to enable the playback method to meet a variety of different needs.

[0031] In some possible implementations, determining at least one audio component of the at least one musical instrument as at least one of a low-frequency component, a medium-frequency component, or a high-frequency component through spectral energy distribution includes:

[0032] Converting at least one audio component of the at least one musical instrument from a time domain signal to a frequency domain signal;

[0033] The at least one audio component of the at least one musical instrument is determined as at least one of a low frequency component, a middle frequency component or a high frequency component according to a frequency band range of a frequency domain signal of the at least one audio component of the at least one musical instrument.

[0034] In this way, the electronic device can estimate the energy size in each frequency band range through the frequency band range of the frequency domain signal, determine the type of the audio component according to the energy size, and perform corresponding processing on various types of audio components.

[0035] In some possible implementations, the performing crosstalk elimination on a high frequency component of at least one audio component of the at least one musical instrument includes:

[0036] The high frequency component of the at least one audio component of the at least one musical instrument is crosstalk-cancelled based on the least square method.

[0037] In this way, the electronic device can eliminate the crosstalk of the high-frequency components in the audio components, so that the audio components can meet the binaural effect and improve the user experience.

[0038] In a second aspect, the present application provides an audio playback device, comprising:

[0039] An acquisition module, used to acquire the audio to be played;

[0040] A separation module, used to separate at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played;

[0041] A first processing module, configured to perform crosstalk elimination on a high frequency component of at least one audio component of the at least one musical instrument;

[0042] A second processing module, configured to delay and align a low-frequency component in at least one audio component of the at least one musical instrument and an audio component of the human voice with the high-frequency component after crosstalk elimination;

[0043] The playing module is used to play the high-frequency component and the audio component of the human voice through a first speaker pair, and play the low-frequency component through a second speaker pair, wherein the opening angle of the first speaker pair is smaller than the opening angle of the second speaker pair.

[0044] In some possible implementations, the separation module may be used to:

[0045] At least one audio component of the at least one musical instrument and an audio component of human voice are separated from the audio to be played according to the instrument type of the at least one musical instrument.

[0046] In some possible implementations, when the instrument type of the musical instrument is a low-frequency instrument, the separation module is further configured to:

[0047] An audio component of the low-frequency instrument is determined as a low-frequency component.

[0048] In some possible implementations, when the instrument type of the musical instrument is a wide-band instrument, the separation module is further configured to:

[0049] A high frequency component and a low frequency component are separated from at least one audio component of the wide frequency musical instrument.

[0050] In some possible implementations, the separation module may be used to:

[0051] A high-frequency component and a low-frequency component are separated from at least one audio component of the wide-band musical instrument by using a filter bank.

[0052] In some possible implementations, when the instrument type of the musical instrument is not any of the preset types, the separation module is further configured to:

[0053] At least one audio component of the at least one musical instrument is determined as at least one of a low-frequency component, a middle-frequency component or a high-frequency component through spectral energy distribution.

[0054] In some possible implementations, the first processing module is further configured to:

[0055] Eliminate crosstalk of an intermediate frequency component in at least one audio component of the at least one musical instrument;

[0056] The second processing module is also used for:

[0057] The low-frequency component of at least one audio component of the at least one musical instrument and the audio component of the human voice are delayed and aligned with the high-frequency component and the middle-frequency component after crosstalk elimination.

[0058] In some possible implementations, the playback module is further configured to:

[0059] The intermediate frequency component is played through a third speaker pair, wherein an opening angle of the third speaker pair is greater than an opening angle of the first speaker pair, and an opening angle of the third speaker pair is smaller than an opening angle of the second speaker pair.

[0060] In some possible implementations, the separation module may be used to:

[0061] Converting at least one audio component of the at least one musical instrument from a time domain signal to a frequency domain signal;

[0062] The at least one audio component of the at least one musical instrument is determined as at least one of a low frequency component, a middle frequency component or a high frequency component according to the frequency band range of the frequency domain signal of the at least one audio component of the at least one musical instrument.

[0063] In some possible implementations, the separation module may be used to:

[0064] The high frequency component of the at least one audio component of the at least one musical instrument is crosstalk-cancelled based on the least square method.

[0065] In a third aspect, the present application provides a speaker, the speaker comprising a processor and a memory;

[0066] One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the speaker performs the audio playback method as described in any possible implementation of the first aspect above.

[0067] In a fourth aspect, the present application provides an electronic device, the electronic device comprising one or more processors and a memory;

[0068] Among them, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device performs the audio playback method described in any possible implementation method of the first aspect above.

[0069] In a fifth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the terminal executes the audio playback method as described in any possible design in the first aspect above.

[0070] In a sixth aspect, the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the audio playback method described in any possible design in the first aspect above.

[0071] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0073] Figure 2 A flowchart of an audio playback method provided in an embodiment of the present application;

[0074] Figure 3 A schematic diagram of the size of the speaker angle provided in an embodiment of the present application;

[0075] Figure 4 A schematic diagram of audio playback by four speakers provided in an embodiment of the present application;

[0076] Figure 5 A schematic diagram of audio playback by six speakers provided in an embodiment of the present application;

[0077] Figure 6 A schematic diagram of the structure of a speaker provided in an embodiment of the present application;

[0078] Figure 7 A schematic diagram of the structure of an audio playback device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0080] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0081] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0082] Stereo sound refers to sound with a three-dimensional sense. The sounds emitted by nature are stereo sound, and when these stereo sounds are recorded, amplified, and replayed, all the sounds will be played from the speakers. When the sound waves reach the human ear, due to the difference in the position of the left and right ears, there is a slight difference in the time it takes to reach the left and right ears. In addition, due to the occlusion of the head, there is also a slight difference in the sound pressure of the sound waves reaching the left and right ears. The human brain can determine the location of the sound source through this slight difference.

[0083] When playing sound sources through the same speaker, since all sounds are emitted from the same speaker, the sound will lose its original three-dimensional sense, affecting the user experience. For example, in a concert, users can hear the sounds of different instruments emitted from different positions, but when playing the recording of the concert through the speaker, the spatial sense of the concert scene will be lost, affecting the user experience.

[0084] The binaural effect is an effect in which the user relies on the volume difference, time difference and timbre difference between the two ears to determine the direction of the sound. Audio technology can use the binaural effect to restore the stereoscopic sense of sound to a certain extent. Specifically, stereo sound effects can be generated through the Head Related Transfer Functions (HRTF) technology, so that when the sound is transmitted to the auricle, ear canal and eardrum in the human ear, the listener will have a sense of surround sound.

[0085] However, in stereo playback, the signal transmitted on the same side and the signal transmitted on the opposite side may interfere with each other, destroying the spatial information originally contained in the stereo signal. Normally, the method of crosstalk elimination can be used to eliminate the opposite-side signal, thereby avoiding the interference of the opposite-side signal on the same-side signal transmission. Specifically, HRTF can be used as a transmission matrix to perform crosstalk elimination on the stereo signal. The inventors have found that when HRTF is used as a transmission matrix, there is a problem with the gain in the crosstalk elimination matrix estimated for the stereo signal. When processing the low frequency, the estimated gain is large, which affects the robustness of the processing and the audio playback effect, and it is difficult to meet the user's experience requirements.

[0086] In view of this, the present application provides an audio playback method, which can be performed by an electronic device. The electronic device can be a speaker or other electronic device with a speaker function. Specifically, the electronic device obtains the audio to be played, and then separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played, performs crosstalk elimination on the high-frequency component in at least one audio component of at least one musical instrument, and delays and aligns the low-frequency component in at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination, and then plays the high-frequency component and the audio component of human voice through a first speaker pair with a smaller opening angle, and plays the low-frequency component through a second speaker pair with a larger opening angle, thereby achieving audio playback.

[0087] On the one hand, electronic devices process low-frequency components, high-frequency components and human voices differently according to the frequency, avoiding the large gain calculated by the head-related transfer function as the transmission matrix for crosstalk elimination at low frequencies. Therefore, crosstalk elimination is not performed on low-frequency components, thereby reducing the amount of calculation for audio playback and improving the robustness of the system.

[0088] On the other hand, electronic devices replay low-frequency signals through large-angle speakers and high-frequency signals and vocal signals through small-angle speakers, thereby improving the stability of audio playback and further improving the robustness of audio processing.

[0089] The electronic device may be a terminal, which includes but is not limited to a smart phone, a tablet computer, a laptop computer, a personal digital assistant (PDA), a smart home device or a smart wearable device.

[0090] The following takes the electronic device as a terminal as an example to introduce the electronic device in this embodiment. The structure of the terminal can be as follows: Figure 1 As shown, Figure 1 A schematic diagram of the structure of a terminal provided in an embodiment of the present application.

[0091] like Figure 1As shown, the terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0092] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0093] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0094] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0095] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0096] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0097] In an embodiment of the present application, the processor 110 can obtain the audio to be played, and then separate at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played, crosstalk the high-frequency component in at least one audio component of at least one musical instrument, and delay and align the low-frequency component in at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination, and finally play the high-frequency component and the audio component of human voice through the first speaker pair with a smaller opening angle, and play the low-frequency component through the second speaker pair. The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the terminal. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, music, video and other files are saved in an external non-volatile memory.

[0098] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The non-volatile memory can also store executable programs and store user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.

[0099] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.

[0100] Non-volatile memory may include disk storage devices and flash memory. Flash memory may include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle, single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the potential level of the storage cell, and universal flash storage (UFS), embedded multimedia card (eMMC), etc. according to the storage specification.

[0101] In the embodiment of the present application, the internal memory 121 can store audio to be played.

[0102] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal. While the charging management module 140 is charging the battery 142, it may also power the terminal through the power management module 141.

[0103] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0104] The wireless communication function of the terminal can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0105] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0106] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0107] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0108] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied on the terminal. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0109] In some embodiments, the antenna 1 of the terminal is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0110] The terminal can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor. For example, music playing, recording, etc. The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In an embodiment of the present application, the terminal can control the audio module 170 to play the audio components of high-frequency components and human voice through the first speaker, and play the low-frequency components through the second speaker.

[0111] The speaker 170A, also called a "speaker", is used to convert the audio electrical signal into a sound signal. The terminal can play the processed high-frequency component, middle-frequency component, low-frequency component and human voice through the speaker 170A.

[0112] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0113] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C with his mouth to input the sound signal into the microphone 170C. The terminal can be provided with at least one microphone 170C. In other embodiments, the terminal can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc. The terminal can obtain the audio signal to be processed through the microphone 170C.

[0114] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0115] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the terminal detects the intensity of the touch operation based on the pressure sensor 180A. The terminal can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0116] The gyroscope sensor 180B can be used to determine the motion posture of the terminal. In some embodiments, the angular velocity of the terminal around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0117] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0118] The magnetic sensor 180D includes a Hall sensor. The terminal can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal is a flip phone, the terminal can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0119] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal in all directions (generally three axes). When the terminal is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the terminal posture and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0120] The distance sensor 180F is used to measure the distance. The terminal can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal can use the distance sensor 180F to measure the distance to achieve fast focus.

[0121] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal emits infrared light outward through the light emitting diode. The terminal uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal. When insufficient reflected light is detected, the terminal can determine that there is no object near the terminal. The terminal can use the proximity light sensor 180G to detect that the user holds the terminal close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0122] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal is in a pocket to prevent accidental touches.

[0123] The fingerprint sensor 180H is used to collect fingerprints. The terminal can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0124] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal heats the battery 142 to avoid abnormal shutdown of the terminal due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0125] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal, which is different from the position of the display screen 194.

[0126] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals of the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure beating signals. In some embodiments, the bone conduction sensor 180M can also be set in the earphones to form bone conduction earphones.

[0127] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal may receive key input and generate key signal input related to user settings and function control of the terminal.

[0128] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0129] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0130] The terminal can realize the shooting function through the camera 193, ISP, video codec, GPU, display screen 194 and application processor.

[0131] Camera 193 is used to capture images. Specifically, the object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0132] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0133] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0134] Video codecs are used to compress or decompress digital videos. A terminal can support one or more video codecs. In this way, the terminal can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0135] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of terminals, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0136] The terminal implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0137] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display 194 (liquid crystal display, LCD), an organic light-emitting diode (organic light-emitting diode, OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (active-matrix organic light emitting diode, AMOLED), a flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light-emitting diodes, QLED), etc.

[0138] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the terminal by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal and cannot be separated from the terminal.

[0139] The above is an introduction to the structure of electronic devices. Figure 2 The specific steps of the audio playback method of the embodiment of the present application are introduced.

[0140] S202: The terminal obtains the audio to be played.

[0141] The audio to be played refers to the audio that needs to be played through the terminal. The audio to be played usually includes mixed voices. In this embodiment, the audio to be played includes the audio of at least one musical instrument and human voice audio. In some possible implementations, the audio to be played may include drum sounds, bass sounds, piano sounds, and audio of other musical instruments and human voice audio.

[0142] The terminal can obtain the audio to be played in a variety of ways. For example, the terminal can obtain the audio to be played through communication transmission, and the terminal can also obtain the audio to be played by recording through a recording function.

[0143] S204: The terminal separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played.

[0144] The terminal can separate at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played through the audio source separation system. The audio source separation system can separate audio components corresponding to different audio sources from the mixed audio according to different audio sources.

[0145] The sound source separation system can be implemented based on deep learning. During the training process, the neural network can learn the audio features of different sound source pairs during training, and then extract different sound sources based on the learned features. For example, the sound source separation system is obtained through training of drum sounds, bass sounds, piano sounds and human voices, so the sound source separation system can separate and obtain the audio components of drum sounds, bass sounds, piano sounds, human voices and other musical instruments in the mixed audio. For another example, the sound source separation system is obtained through training of drum sounds, piano sounds and human voices, so the sound source separation system can separate and obtain the audio components of drum sounds, piano sounds, human voices and other musical instruments in the mixed audio. In this embodiment, the example that the sound source separation system is obtained through training of drum sounds, bass sounds, piano sounds and human voices, and can separate and obtain the audio components of drum sounds, bass sounds, piano sounds, human voices and other musical instruments in the mixed audio is introduced.

[0146] Through the sound source separation system, the terminal can separate the audio components of drum sounds, bass sounds, piano sounds, human voices and other musical instruments from the audio to be played.

[0147] S206: The terminal performs crosstalk elimination on a high-frequency component of at least one audio component of at least one musical instrument.

[0148] In some possible implementations, the high-frequency component in the audio component of the musical instrument can be determined by the type of musical instrument. For example, when the sound source separation system separates the audio component of the piccolo from the audio to be played, since the piccolo is a high-frequency instrument and the corresponding frequency range is mainly concentrated in the high frequency, the audio component corresponding to the piccolo can be determined as the high-frequency component, and the crosstalk elimination is performed on the audio component.

[0149] In some other possible implementations, the frequency range of a musical instrument may be relatively wide, and the audio components corresponding to the musical instrument may be high-frequency components, intermediate-frequency components, and low-frequency components, such as a piano. Based on this, the terminal may use a filter group to divide the audio sub-band corresponding to the musical instrument to obtain the low-frequency component and high-frequency component of the musical instrument. For example, the terminal may obtain a low-frequency piano sound through a low-pass filter, or obtain a high-frequency piano sound through a high-pass filter. In the case where the audio components of the piano need to be divided into high-frequency components, intermediate-frequency components, and low-frequency components, the terminal obtains a low-frequency piano sound through a low-pass filter, and obtains a high-frequency piano sound through a high-pass filter, and the rest is an intermediate-frequency piano sound. Further division of the audio components of the musical instrument through a filter group can be processed based on the frequency of the audio, and the computational complexity is low and the processing speed is fast.

[0150] For audio components that are not identified as specific musical instruments or human voices by the sound source separation system, that is, audio components of other musical instruments, the terminal can determine whether the audio components of the musical instrument are high-frequency components or low-frequency components based on the spectrum energy distribution. For example, the terminal can separate the audio components of drum sounds, bass sounds, piano sounds, human voices, and other musical instruments from the audio to be played through the sound source separation system. Under the premise of using the sound source separation system, since the sound source separation system cannot identify the audio components corresponding to the guitar, the audio components of the guitar are the audio components of other musical instruments. The terminal can determine whether the audio components of the guitar are high-frequency components or low-frequency components based on the statistical energy within the frequency band based on the spectrum energy distribution. The judgment based on the spectrum energy distribution of the audio components of the guitar refers to the judgment based on the frequency domain energy of the audio signal of the guitar. Specifically, the terminal can perform Fourier transform (e.g., fast Fourier transform (FFT)) on the time domain signal of the audio component of the guitar to obtain the frequency domain signal of the guitar, and then determine whether the audio signal of the guitar is a low-frequency signal or a high-frequency signal according to the preset frequency band range. In some possible implementations, the terminal then determines whether the audio component of the guitar is a low-frequency component, a medium-frequency component, or a high-frequency component based on a preset frequency band range. For example, the time domain signal of the audio component of the guitar is x(t), and the frequency domain signal of the time domain signal after fast Fourier transform is X(f). The preset frequency band ranges are: low-frequency signal is 0 to 500 Hz (Hz), medium-frequency signal is 500Hz to 6000Hz, and high-frequency signal is greater than 6000Hz. The terminal then calculates the frequency domain energy of the signal in the low-frequency signal range, the medium-frequency signal range, and the high-frequency signal range, respectively, and obtains the frequency domain energy S1 in the low-frequency signal range, the frequency domain energy S2 in the medium-frequency signal range, and the frequency domain energy S3 in the high-frequency signal range, and compares the sizes of S1, S2, and S3 to determine whether the audio component is a low-frequency component, a medium-frequency component, or a high-frequency component.

[0151]

[0152]

[0153]

[0154] Where fs is the sampling rate of the signal.

[0155] In this way, the terminal can perform crosstalk elimination on the high-frequency components in at least one audio component of at least one musical instrument obtained. In audio playback, a dual-channel transmission method is usually adopted. If there is crosstalk between the channels on both sides, it may cause an unnecessary signal to be played on one side while the audio signal is played on the other side, affecting the user experience. In stereo playback, the waveforms of the sound signals on both sides are usually not exactly the same. Due to the existence of crosstalk, the difference between the sound signals on both sides may become smaller, destroying the positioning information in the stereo and affecting the user experience. Normally, the terminal can use a variety of methods to eliminate crosstalk for audio. In this embodiment, a crosstalk elimination method based on the least squares method can be used to process the high-frequency components.

[0156] When at least one audio component of at least one musical instrument includes a mid-frequency component, that is, when the audio component of another musical instrument is determined to be a mid-frequency component, crosstalk elimination is performed on the mid-frequency component, for example, a crosstalk elimination method based on least squares method is used to process the high-frequency component.

[0157] S208: The terminal delays and aligns the low-frequency component of at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination.

[0158] In some possible implementations, the low-frequency component in the audio component of the musical instrument can be determined by the type of the musical instrument. For example, when the sound source separation system separates the audio component of the drum from the audio to be played, since the drum is a low-frequency instrument and the corresponding frequency range is mainly concentrated in the low frequency, the audio component corresponding to the drum can be determined as the low-frequency component.

[0159] Similarly, when the sound source separation system separates the audio component of bass from the audio to be played, since bass is a low-frequency instrument and the corresponding frequency range is mainly concentrated in the low frequency, the audio component corresponding to the bass can be determined as a low-frequency component.

[0160] In some other possible implementations, for a musical instrument with a wider frequency range, such as a piano, the low-frequency component of the audio component of the piano may be obtained by means of a filter bank in S206.

[0161] For audio components that are not identified as specific instrument types or human voices by the sound source separation system, that is, audio components of other instruments, the terminal can determine whether the audio components of the instrument are high-frequency components or low-frequency components based on the spectral energy distribution.

[0162] In this embodiment, since HRTF is used as the transmission matrix, the estimated gain is larger when processing the low frequency, so only the low frequency component is delayed so that the low frequency component can be aligned with the high frequency component after crosstalk elimination. Furthermore, only delaying the low frequency can also avoid the harmonic distortion and color staining problems existing in the low frequency region of the traditional crosstalk elimination.

[0163] For the audio component of human voice, the terminal also only performs delay processing so that the audio component of human voice can be aligned with the high-frequency component after crosstalk elimination.

[0164] S210: The terminal plays the high-frequency component and the audio component of the human voice through the first speaker pair, and plays the low-frequency component through the second speaker pair.

[0165] The opening angle of the first speaker pair is smaller than that of the second speaker pair. That is, the terminal plays the processed high-frequency components and the processed audio components of the human voice through the speaker pair with a smaller opening angle, and plays the processed low-frequency components through the speaker pair with a larger opening angle. Figure 3 As shown, the first speaker pair is 302 and 303 in the figure, and the second speaker pair is 301 and 304 in the figure, and the angle ∠1 formed by 302 and 302 is smaller than the angle ∠2 formed by 301 and 304.

[0166] By adding phase difference, the user's listening area can be estimated based on the relationship between phase difference and distance:

[0167]

[0168] Among them, y max represents the user's listening area, that is, the maximum head deviation distance allowed for the user, c represents the propagation speed of sound, f represents the frequency of the sound source, and θ represents the opening angle of the speaker pair.

[0169] It can be seen from formula (4) that the frequency of the sound source is inversely proportional to the opening angle of the speaker pair. Therefore, in order to obtain the sweet spot for widening the listening experience, that is, the area with the best user experience in this embodiment, the terminal can play the high-frequency component through the first speaker pair with a smaller opening angle, and play the low-frequency component through the second speaker pair with a larger opening angle.

[0170] Since the user's demand for human voice in audio is usually clear sound quality, the terminal can play the audio component of human voice through the first speaker pair with a smaller angle. Exemplarily, when the terminal includes or controls four speakers, the terminal can play the high-frequency component and the audio component of human voice through the two internal speakers (first speaker pair), and play the low-frequency component through the two speakers at both ends (second speaker pair). When the terminal includes or controls six speakers, the terminal can play the high-frequency component and the audio component of human voice through the two internal speakers (first speaker pair), play the mid-frequency component through the two second outer speakers (third speaker pair), and play the low-frequency component through the two speakers at both ends (second speaker pair).

[0171] Through the description of the above content, the present application provides an audio playback method. Specifically, the terminal obtains the audio to be played, separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played, and then performs crosstalk elimination on the high-frequency component in at least one audio component of at least one musical instrument, and delays and aligns the low-frequency component in at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination, and plays the high-frequency component and the audio component of human voice through a first speaker pair with a smaller opening angle, and plays the low-frequency component through a second speaker pair with a larger opening angle, thereby improving the robustness of the audio playback.

[0172] On the one hand, electronic devices process low-frequency components, high-frequency components and human voices differently according to the frequency, avoiding the large gain calculated by the head-related transfer function as the transmission matrix for crosstalk elimination at low frequencies. Therefore, crosstalk elimination is not performed on low-frequency components, thereby reducing the amount of calculation for audio playback and improving the robustness of the system.

[0173] On the other hand, electronic devices replay low-frequency signals through large-angle speakers and high-frequency signals and vocal signals through small-angle speakers, thereby improving the stability of audio playback and further improving the robustness of audio processing.

[0174] The following example takes the case where the sound source separation system can identify drums, bass, piano and human voices, and the terminal can control four speakers. Figure 4 As shown, the audio playback method provided by this application is introduced.

[0175] S202: The terminal obtains the audio to be played.

[0176] The audio to be played includes not only drum sounds, bass sounds, piano sounds and vocals that can be identified by the sound source separation system, but also sounds of other musical instruments that cannot be identified by the sound source separation system, such as guitar sounds, violin sounds and cello sounds.

[0177] S204: The terminal separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played.

[0178] The terminal separates the sounds of drums, bass, piano, vocals and other instruments whose specific instruments cannot be distinguished, such as guitar and violin, through a sound source separation system.

[0179] S206: The terminal performs crosstalk elimination on a high-frequency component of at least one audio component of at least one musical instrument.

[0180] For the drum sounds, bass sounds, piano sounds, human voices and other musical instrument sounds whose specific instruments cannot be distinguished obtained by the sound source system, the terminal can separate the high-frequency components, low-frequency components and medium-frequency components from the audio components according to the type of musical instrument.

[0181] For example, drums and bass are low-frequency instruments, so the terminal may determine the separated drum sounds and bass sounds as low-frequency components.

[0182] The piano is a wide-band instrument, and the terminal can further separate the separated piano sound through a sub-band filter. For example, the terminal can obtain the low-frequency component of the piano sound through a low-pass filter, and obtain the high-frequency component of the piano sound through a high-pass filter.

[0183] For other musical instrument sounds (audio components of other musical instruments) whose specific musical instrument sounds cannot be distinguished, or other audio corresponding to other musical instruments set as high-frequency instruments, low-frequency instruments and wide-band instruments. For example, other musical instrument sounds may include guitar sounds, and guitar sounds may be instrument sounds that the sound source system cannot identify the specific instrument, or may be instrument sounds that the sound source recognition system can identify, but how to process them is not set.

[0184] The terminal can divide the other musical instrument sounds into one of low frequency components, high frequency components or intermediate frequency components according to the frequency band range of the other musical instrument sounds. The terminal can also separate the other musical instrument sounds into low frequency components, high frequency components and intermediate frequency components through a filter bank.

[0185] Thus, the terminal can perform crosstalk cancellation processing based on the least squares method on the high frequency components obtained through the above steps through the crosstalk cancellation unit. When the separated and determined audio components include intermediate frequency components, the terminal can also perform crosstalk cancellation processing based on the least squares method on the intermediate frequency components through the crosstalk cancellation unit.

[0186] S208: The terminal delays and aligns the low-frequency component of at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination.

[0187] In this embodiment, the terminal can delay the drum sounds, bass sounds separated by the sound source system, the low-frequency components of the piano sounds obtained through a low-pass filter, the low-frequency components obtained from other musical instruments, and the audio components of the human voice through a delay unit to align them with the high-frequency components after crosstalk elimination, thereby avoiding delays caused by crosstalk elimination of the high-frequency components.

[0188] S210: The terminal plays the high-frequency component and the audio component of the human voice through the first speaker pair, and plays the low-frequency component through the second speaker pair.

[0189] like Figure 4 As shown, the terminal plays the high-frequency components and the audio components of the human voice through the first speaker pair with a smaller opening angle (402 and 403 in the figure), and plays the low-frequency components through the second speaker pair with a larger opening angle (401 and 404 in the figure). When the audio components of other musical instruments include intermediate frequency components, the terminal plays the intermediate frequency components through the first speaker pair.

[0190] The following example takes the case where the sound source separation system can identify the sound of a piccolo, drums, bass, piano, and human voice, and the terminal can control 6 speakers. Figure 5 As shown, the audio playback method provided by this application is introduced.

[0191] S202: The terminal obtains the audio to be played.

[0192] The audio to be played includes not only the piccolo, drum, bass, piano and human voice that can be identified by the sound source separation system, but also the sounds of other musical instruments that cannot be identified by the sound source separation system.

[0193] S204: The terminal separates at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played.

[0194] The terminal separates the sounds of flute, drums, bass, piano, human voice and other musical instruments whose specific instruments cannot be identified through the sound source separation system.

[0195] S206: The terminal performs crosstalk elimination on a high-frequency component of at least one audio component of at least one musical instrument.

[0196] For the piccolo, drum, bass, piano, human voice and other musical instrument sounds separated by the sound source system, the terminal can separate the high-frequency component, low-frequency component and mid-frequency component from the audio components according to the type of musical instrument.

[0197] Among them, the piccolo is a high-frequency instrument, so the terminal can determine the piccolo sound as a high-frequency component. The drum and bass are low-frequency instruments, so the terminal can determine the separated drum and bass sounds as low-frequency components. The piano is a broadband instrument, and the terminal can further separate the separated piano sound through a sub-band filter to obtain low-frequency components and high-frequency components.

[0198] For the audio components of other musical instruments, the terminal can determine the audio components of other musical instruments as any one of low-frequency components, high-frequency components and medium-frequency components according to the frequency band range corresponding to the audio components of other musical instruments, or obtain the low-frequency components, high-frequency components and medium-frequency components of the audio components of other musical instruments respectively.

[0199] The terminal processes the high frequency component and the intermediate frequency component through the crosstalk cancellation unit based on the least square method.

[0200] S208: The terminal delays and aligns the low-frequency component of at least one audio component of at least one musical instrument and the audio component of human voice with the high-frequency component after crosstalk elimination.

[0201] The terminal can delay the low-frequency components and the audio components of human voice through the delay unit to align them with the high-frequency components after crosstalk elimination, thereby avoiding the delay caused by crosstalk elimination of the high-frequency components.

[0202] S210: The terminal plays the high-frequency component and the audio component of the human voice through the first speaker pair, and plays the low-frequency component through the second speaker pair.

[0203] like Figure 5 As shown, the terminal plays high-frequency components and audio components of human voice through a first speaker pair with a smaller opening angle (503 and 504 in the figure), plays low-frequency components through a second speaker pair with a larger opening angle (501 and 506 in the figure), and plays mid-frequency components through a third speaker pair with an opening angle between the first speaker pair and the second speaker pair.

[0204] The present application also provides a sound system, such as Figure 6 As shown, the terminal may include: multiple speakers 610, one or more processors 620, memory 1130, one or more computer programs 640, and the above-mentioned components may be connected via one or more communication buses 650. The one or more computer programs 640 are stored in the above-mentioned memory 630 and are configured to be executed by the one or more processors 620. The one or more computer programs 640 include instructions, and the above-mentioned instructions can be used to execute as follows Figure 2 The various steps performed by the audio in the corresponding embodiment. The audio may be included in the terminal.

[0205] The embodiment of the present application can divide the audio into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0206] In the case of dividing each functional module into corresponding functional modules, Figure 7 A possible schematic diagram of the terminal involved in the above-mentioned embodiments is shown, and the terminal executes the steps in any method embodiment of the method embodiments of the present application. Figure 7 As shown, the terminal may include: an acquisition module 702, used to acquire audio to be played; a separation module 704, used to separate at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played; a first processing module 706, used to perform crosstalk elimination on the high-frequency component in at least one audio component of the at least one musical instrument; a second processing module 708, used to delay and align the low-frequency component in at least one audio component of the at least one musical instrument and the audio component of the human voice with the high-frequency component after crosstalk elimination; a playing module 710, used to play the high-frequency component and the audio component of the human voice through a first speaker pair, and play the low-frequency component through a second speaker pair, the opening angle of the first speaker pair being smaller than the opening angle of the second speaker pair.

[0207] In some possible implementations, the separation module 704 may be used to:

[0208] At least one audio component of the at least one musical instrument and an audio component of human voice are separated from the audio to be played according to the instrument type of the at least one musical instrument.

[0209] In some possible implementations, when the instrument type of the musical instrument is a low-frequency instrument, the separation module 704 is further configured to:

[0210] An audio component of the low-frequency instrument is determined as a low-frequency component.

[0211] In some possible implementations, when the instrument type of the musical instrument is a broadband instrument, the separation module 704 is further configured to:

[0212] A high frequency component and a low frequency component are separated from at least one audio component of the wide frequency musical instrument.

[0213] In some possible implementations, the separation module 704 may be used to:

[0214] A high-frequency component and a low-frequency component are separated from at least one audio component of the wide-band musical instrument by using a filter bank.

[0215] In some possible implementations, when the instrument type of the musical instrument is not any of the preset types, the separation module 704 is further configured to:

[0216] At least one audio component of the at least one musical instrument is determined as at least one of a low-frequency component, a middle-frequency component or a high-frequency component through spectral energy distribution.

[0217] In some possible implementations, the first processing module 706 is further configured to:

[0218] Eliminate crosstalk of an intermediate frequency component in at least one audio component of the at least one musical instrument;

[0219] The second processing module 708 is further configured to:

[0220] The low-frequency component of at least one audio component of the at least one musical instrument and the audio component of the human voice are delayed and aligned with the high-frequency component and the middle-frequency component after crosstalk elimination.

[0221] In some possible implementations, the playing module 710 is further configured to:

[0222] The intermediate frequency component is played through a third speaker pair, wherein an opening angle of the third speaker pair is greater than an opening angle of the first speaker pair, and an opening angle of the third speaker pair is smaller than an opening angle of the second speaker pair.

[0223] In some possible implementations, the separation module 704 may be used to:

[0224] Converting at least one audio component of the at least one musical instrument from a time domain signal to a frequency domain signal;

[0225] The at least one audio component of the at least one musical instrument is determined as at least one of a low frequency component, a middle frequency component or a high frequency component according to a frequency band range of a frequency domain signal of the at least one audio component of the at least one musical instrument.

[0226] In some possible implementations, the separation module 704 may be used to:

[0227] The high frequency component of the at least one audio component of the at least one musical instrument is crosstalk-cancelled based on the least square method.

[0228] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the audio system so that the audio system executes the corresponding method, which will not be repeated here.

[0229] This embodiment also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes Figure 2 The relevant method steps in the embodiment are used to implement the method in the above embodiment.

[0230] This embodiment also provides a computer program product including instructions, when the computer program product is run on an electronic device, the terminal executes the following Figure 2 The relevant method steps in the embodiment are used to implement the method in the above embodiment.

[0231] In the several embodiments provided in this embodiment, it should be understood that the disclosed terminal and method can be implemented in other ways. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program codes.

[0235] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio playback method, characterized in that: The method comprises: Get the audio to be played; Separating at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played; Eliminate crosstalk of a high frequency component in at least one audio component of the at least one musical instrument; Delaying and aligning a low-frequency component in at least one audio component of the at least one musical instrument and an audio component of the human voice with the high-frequency component after crosstalk elimination; The high frequency component and the audio component of the human voice are played through a first speaker pair, and the low frequency component is played through a second speaker pair, wherein the opening angle of the first speaker pair is smaller than the opening angle of the second speaker pair.

2. The method according to claim 1, characterized in that The step of separating at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played comprises: At least one audio component of the at least one musical instrument and an audio component of human voice are separated from the audio to be played according to the instrument type of the at least one musical instrument.

3. The method according to claim 2, characterized in that When the instrument type of the musical instrument is a low-frequency instrument, the method further includes: An audio component of the low-frequency instrument is determined as a low-frequency component.

4. The method according to claim 2, characterized in that: When the instrument type of the musical instrument is a broadband instrument, the method further includes: A high frequency component and a low frequency component are separated from at least one audio component of the wide frequency musical instrument.

5. The method according to claim 4, characterized in that The step of separating the high-frequency component and the low-frequency component from at least one audio component of the wide-band musical instrument comprises: A high-frequency component and a low-frequency component are separated from at least one audio component of the wide-band musical instrument by using a filter bank.

6. The method according to claim 5, when the instrument type of the musical instrument is not any of the preset types, the method further comprises: At least one audio component of the at least one musical instrument is determined as at least one of a low-frequency component, a middle-frequency component or a high-frequency component through spectral energy distribution.

7. The method according to claim 1, further comprising: Eliminate crosstalk of an intermediate frequency component in at least one audio component of the at least one musical instrument; The delay alignment of the low-frequency component in at least one audio component of the at least one musical instrument and the audio component of the human voice with the high-frequency component after crosstalk elimination comprises: The low-frequency component of at least one audio component of the at least one musical instrument and the audio component of the human voice are delayed and aligned with the high-frequency component and the middle-frequency component after crosstalk elimination.

8. The method according to claim 7, further comprising: The intermediate frequency component is played through a third speaker pair, wherein an opening angle of the third speaker pair is greater than an opening angle of the first speaker pair, and an opening angle of the third speaker pair is smaller than an opening angle of the second speaker pair.

9. The method according to claim 6, characterized in that The step of determining at least one audio component of the at least one musical instrument as at least one of a low-frequency component, a medium-frequency component or a high-frequency component by means of spectral energy distribution comprises: Converting at least one audio component of the at least one musical instrument from a time domain signal to a frequency domain signal; The at least one audio component of the at least one musical instrument is determined as at least one of a low frequency component, a middle frequency component or a high frequency component according to a frequency band range of a frequency domain signal of the at least one audio component of the at least one musical instrument.

10. The method according to claim 1, characterized in that The step of performing crosstalk elimination on the high frequency component of at least one audio component of the at least one musical instrument comprises: The high frequency component of the at least one audio component of the at least one musical instrument is crosstalk-cancelled based on the least square method.

11. An audio playback device, characterized in that: The device comprises: An acquisition module, used to acquire the audio to be played; A separation module, used to separate at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played; A first processing module, configured to perform crosstalk elimination on a high frequency component of at least one audio component of the at least one musical instrument; A second processing module, configured to delay and align a low-frequency component in at least one audio component of the at least one musical instrument and an audio component of the human voice with the high-frequency component after crosstalk elimination; The playing module is used to play the high-frequency component and the audio component of the human voice through a first speaker pair, and play the low-frequency component through a second speaker pair, wherein the opening angle of the first speaker pair is smaller than the opening angle of the second speaker pair.

12. A sound system, characterized in that: The audio system includes a memory and a processor, wherein the memory is used to store one or more programs; and the processor is used to run the one or more programs to implement the following actions: Get the audio to be played; Separating at least one audio component of at least one musical instrument and an audio component of human voice from the audio to be played; Eliminate crosstalk of a high frequency component in at least one audio component of the at least one musical instrument; Delaying and aligning a low-frequency component in at least one audio component of the at least one musical instrument and an audio component of the human voice with the high-frequency component after crosstalk elimination; The high frequency component and the audio component of the human voice are played through a first speaker pair, and the low frequency component is played through a second speaker pair, wherein the opening angle of the first speaker pair is smaller than the opening angle of the second speaker pair.

13. A computer storage medium, characterized in that: The method comprises computer instructions. When the computer instructions are executed on a terminal, the terminal executes the audio playing method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that When the computer program product runs on a computer, the computer executes the audio playing method according to any one of claims 1 to 10.

Images