Audio playing method and device

Abstract

The invention provides an audio playing method and device, which are helpful for restoring the overall tone of audio and improving the audio and video experience of a user. The method comprises the following steps: acquiring auricle information of a user, wherein the auricle information comprises at least one of auricle length, auricle width, auricle thickness or auricle area; based on the auricle information and a first sound effect positioning algorithm, a second sound effect positioning algorithm is determined, the first sound effect positioning algorithm is used for simulating a sound signal in a preset spatial orientation, and the first sound effect positioning algorithm is obtained based on amplitude spectrum data; the amplitude spectrum data is obtained by performing Fourier transform, averaging, envelope extraction and inverse Fourier transform on multiple groups of head-related impulse response (HRIR) data; and processing audio data to be played through the second sound effect positioning algorithm to obtain processed audio data, and playing the processed audio data in a loudspeaker mode.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an audio player method and device. Background Art

[0002] When a terminal device plays audio for a user through an external speaker, the user usually feels that the sound source of the audio heard is at the location of the terminal device, which makes the user feel that the sound field of the audio lacks stereoscopic sense and height sense, affecting the user's audio and video experience. For example, when a laptop plays music for a user through an external speaker, the user feels that the sound field of the music is located on the keyboard of the laptop. In order to improve the user's audio and video experience, the terminal device usually processes the audio data through the head response transfer function (HRTF), and then plays the processed audio data for the user, so that the user feels that the position of the audio sound field becomes higher.

[0003] However, such an audio playback method may change the timbre of the audio, resulting in a poor user experience. Summary of the invention

[0004] The present application provides an audio external speaker method and device, which can improve the height and stereoscopic effect of the audio while restoring the overall timbre of the audio, thereby improving the user's audio and video experience.

[0005] In a first aspect, a method for playing audio externally is provided, comprising: obtaining auricle information of a user, the auricle information comprising at least one of auricle length, auricle width, auricle thickness or auricle area; determining a second sound effect localization algorithm based on the auricle information and a first sound effect localization algorithm, the first sound effect localization algorithm being used to simulate a sound signal in a preset spatial orientation, the first sound effect localization algorithm being obtained based on amplitude spectrum data, the amplitude spectrum data being obtained by Fourier transforming, averaging, envelope extracting and inverse Fourier transforming a plurality of groups of head-related impulse response (HRIR) data; processing audio data to be played by means of the second sound effect localization algorithm to obtain processed audio data, and playing the processed audio data externally.

[0006] The audio external speaker method of the present application first performs Fourier transform, averaging, inverse Fourier transform and envelope extraction on the measured multiple groups of HRIR to obtain a first sound effect positioning algorithm; then, the terminal device calibrates the parameters in the first sound effect positioning algorithm based on the acquired user's auricle information to obtain a second sound effect positioning algorithm, and uses the second sound effect positioning algorithm to process the audio data to be played. Among them, through envelope extraction, the characteristic signal in the frequency domain pulse signal can be extracted, so that the frequency domain average signal is calibrated by the characteristic signal, so that the audio data is processed by the first sound effect positioning algorithm, which helps to restore the overall timbre of the audio; in addition, by calibrating the parameters of the first sound effect positioning algorithm based on the user's auricle information, the accuracy of the second sound effect positioning algorithm can be improved, and the terminal device can process the audio data to be played based on the user's auricle information, thereby adjusting the transmission mode of the sound wave based on the user's personalized biometrics, so that the terminal device can dynamically adapt to the personalized height perception of different users, which helps to restore the overall timbre of the audio and improve the user's audio and video experience.

[0007] It should be understood that the user can be any user of the terminal device, such as the owner of the terminal device. The terminal device can obtain the user's auricle information in any way, such as a camera acquisition method. The auricle information can include two sets of data, one of which is the user's left ear auricle information, and the other is the user's right ear auricle information. The auricle information can also include a set of data, which can be any set of data of the user's left ear auricle information or the right ear auricle information.

[0008] In certain implementations of the first aspect, the first sound effect localization algorithm is obtained based on the linear compensation phase data of the speaker of the terminal device and the amplitude spectrum data.

[0009] It should be understood that the linear compensation phase data can be used to improve the group delay inconsistency problem caused by the acoustic characteristics of the speaker of the terminal device. Linear phase compensation can be performed on the initial phase data of the speaker of the terminal device to obtain linear phase compensated data. The linear compensation phase data can be the data used when linear phase compensation is performed on the initial phase data, that is, the initial phase data and the linear compensation phase data can be synthesized into linear phase compensated data. Since the phase of the speaker of the terminal device will affect the user's perception of the height of the audio sound field, the accuracy of the first sound effect positioning algorithm can be improved by calibrating the sound effect positioning algorithm with linear compensation phase data. In this way, the sound field position of the audio felt by the user is closer to the preset spatial orientation, which can improve the user's audio and video experience.

[0010] In certain implementations of the first aspect, determining the second sound effect localization algorithm based on the auricle information and the first sound effect localization algorithm includes: calibrating the first sound effect localization algorithm based on the auricle information to obtain a calibrated first sound effect localization algorithm; performing interaural time difference ITD calibration and / or interaural intensity difference ILD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm.

[0011] It should be understood that the user's positioning of the sound source is not only related to the user's auricle information, but also related to ITD and ILD. Therefore, performing ITD and / or ILD calibration on the calibrated first sound effect localization algorithm can further improve the accuracy of the second sound effect localization algorithm and restore the overall timbre of the audio.

[0012] In certain implementations of the first aspect, the performing ILD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: performing ILD calibration on the calibrated first sound effect localization algorithm based on the horizontal azimuth angle between the user's head and the terminal device to obtain a third sound effect localization algorithm; and determining the third sound effect localization algorithm as the second sound effect localization algorithm.

[0013] It should be understood that ILD calibration of the calibrated first sound effect localization algorithm can be understood as adjusting the parameters in the calibrated first sound effect localization algorithm. With the center of the user's head as the origin, a two-dimensional coordinate system is established in the plane where the line between the center of the user's head and the sound source is located. The y-axis is a straight line in the vertical direction, and the horizontal azimuth angle can be the angle between the line between the center of the user's head and the sound source and the y-axis. Optionally, when the position of the sound source is in the first quadrant of the two-dimensional coordinate system, the horizontal azimuth angle is a value greater than 0; when the position of the sound source is in the fourth quadrant of the two-dimensional coordinate system, the horizontal azimuth angle is a value less than 0.

[0014] In certain implementations of the first aspect, the performing ITD calibration and ILD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: performing ILD calibration on the calibrated first sound effect localization algorithm based on the horizontal azimuth angle between the user's head and the terminal device to obtain a third sound effect localization algorithm; and performing ITD calibration on the third sound effect localization algorithm based on the horizontal azimuth angle between the user's head and the terminal device, a preset human head radius and a sound speed to obtain the second sound effect localization algorithm.

[0015] It should be understood that by performing ILD calibration and ITD calibration on the calibrated first sound effect localization algorithm in sequence, the obtained second sound effect localization algorithm can process the audio data based on the horizontal azimuth angle and the preset head radius, so that the second sound effect localization algorithm can dynamically adapt to the personalized height perception of different users, thereby improving the accuracy of the sound effect localization algorithm; and enabling the terminal device to process the audio data based on the user's personalized feature information, which helps to restore the overall timbre of the audio and improve the user's audio and video experience.

[0016] In certain implementations of the first aspect, the second sound effect localization algorithm includes a first ILD calibration sound effect localization algorithm and a second ILD calibration sound effect localization algorithm; the ILD calibration of the calibrated first sound effect localization algorithm includes: based on the horizontal azimuth and a preset correspondence, determining a target left adjustment gain parameter from multiple left adjustment gain parameters, and determining a target right adjustment gain parameter from multiple right adjustment gain parameters, the preset correspondence including a correspondence between multiple angles, the multiple left adjustment gain parameters, and the multiple right adjustment gain parameters; using the target left adjustment gain parameter as a coefficient of the calibrated first sound effect localization algorithm to obtain the first ILD calibration sound effect localization algorithm; using the target right adjustment gain parameter as a coefficient of the calibrated first sound effect localization algorithm to obtain the second ILD calibration sound effect localization algorithm.

[0017] It should be understood that the multiple angles may include possible horizontal azimuth angles. For example, if the horizontal azimuth angle is between -90° and 90°, the multiple angles may include angles between -90° and 90°. The preset correspondence relationship includes: a correspondence relationship between multiple angles and multiple left adjustment gain parameters; a correspondence relationship between multiple angles and multiple right adjustment gain parameters, that is, each of the multiple angles corresponds to a left adjustment gain parameter and a right adjustment gain parameter. The horizontal azimuth angle is one of the multiple angles.

[0018] In certain implementations of the first aspect, the audio data to be played is processed by the second sound effect positioning algorithm to obtain processed audio data, and the processed audio data is played externally, including: processing the audio data to be played by the first ILD calibration sound effect positioning algorithm to obtain first processed audio data; processing the audio data to be played by the second ILD calibration sound effect positioning algorithm to obtain second processed audio data; and playing the first processed audio data and the second processed audio data externally.

[0019] It should be understood that the first ILD calibration sound effect positioning algorithm may also be referred to as a left channel height filter, and the second ILD calibration sound effect positioning algorithm may also be referred to as a right channel height filter. The first ILD calibration sound effect positioning algorithm and the second ILD calibration sound effect positioning algorithm may be different, so that the first processed audio data and the second processed audio data may be different, thereby making the audio received by the user's two ears different. By performing ILD calibration on the calibrated first sound effect positioning algorithm to obtain the first ILD calibration sound effect positioning algorithm and the second ILD calibration sound effect positioning algorithm, the accuracy of the sound effect positioning algorithm can be improved, and the terminal device can process the audio played based on the horizontal azimuth angle between the user's head and the terminal device, so that the terminal device can dynamically adapt to the personalized height perception of different users, and process the audio data based on the user's personalized feature information, which helps to restore the overall timbre of the audio and improve the user's audio and video experience.

[0020] In certain implementations of the first aspect, the ITD calibration is performed on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm, including: based on the horizontal azimuth angle between the user's head and the terminal device, a preset head radius and the speed of sound, the ITD calibration is performed on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm.

[0021] It should be understood that the preset head radius is a preset value greater than 0, for example, it can be an average of the head radiuses of multiple users. With the center of the user's head as the origin, a two-dimensional coordinate system is established in the plane where the line between the center of the user's head and the sound source is located, and the y-axis is a straight line in the vertical direction, then the horizontal azimuth angle can be the angle between the line between the center of the user's head and the sound source and the y-axis.

[0022] In certain implementations of the first aspect, the second sound effect localization algorithm includes a first ITD calibration sound effect localization algorithm and a second ITD calibration sound effect localization algorithm, the first ITD calibration sound effect localization algorithm being the calibrated first sound effect localization algorithm; the ITD calibration of the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: determining a first parameter based on the horizontal azimuth angle, the preset human head radius and the sound speed; and calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain the second ITD calibration sound effect localization algorithm.

[0023] It should be understood that the first parameter may be an ITD value calculated based on the horizontal azimuth, a preset head radius and the speed of sound, and the ITD value is used to indicate the time difference of the sound wave being transmitted from the speaker of the terminal device to the two ears. By performing ITD calibration on the calibrated first sound effect localization algorithm, the terminal device can process audio data based on the ITD value, which helps to restore the overall timbre of the audio and improve the user's audio-visual experience.

[0024] In certain implementations of the first aspect, calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain the second ITD calibrated sound effect localization algorithm includes: using the first parameter as a time processing parameter in the calibrated first sound effect localization algorithm to obtain the second ITD calibrated sound effect localization algorithm.

[0025] It should be understood that the time processing parameters refer to parameters used to adjust the time in the calibrated first sound effect localization algorithm.

[0026] In certain implementations of the first aspect, the method further includes: obtaining the head radius of the user; determining a second parameter based on the horizontal azimuth angle, the head radius of the user and the sound speed; calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain the second ITD calibrated sound effect localization algorithm, including: using the difference between the first parameter and the second parameter as a time processing parameter in the calibrated first sound effect localization algorithm to obtain the second ITD calibrated sound effect localization algorithm.

[0027] It should be understood that the determination process of the second parameter can be the same as the determination process of the first parameter. The first sound effect localization algorithm after calibration is further calibrated based on the difference between the first parameter and the second parameter, and the obtained second sound effect localization algorithm enables the terminal device to process audio data based on the user's head radius, so that the terminal device can dynamically adapt to the personalized height perception of different users, which helps to improve the accuracy of the sound effect localization algorithm, restore the overall timbre of the audio, and improve the user's audio-visual experience.

[0028] In certain implementations of the first aspect, obtaining the head radius of the user includes: obtaining the head radius when an external spatial audio switch of the terminal device is in an on state.

[0029] It should be understood that the external spatial audio switch can be a switch for controlling whether the terminal device implements the audio external playback method, and the external spatial audio switch can be set on the system setting page of the terminal device. By setting the external spatial audio switch, the user can control whether the terminal device calibrates the sound effect positioning algorithm, so that the flexibility of the audio external playback method can be improved.

[0030] In certain implementations of the first aspect, determining the second sound effect localization algorithm based on the auricle information and the first sound effect localization algorithm includes: inputting the auricle information into the auricle information encoding algorithm to obtain encoded auricle information; inputting parameters of the first sound effect localization algorithm into the sound effect localization algorithm encoding algorithm to obtain encoded parameters; and determining the second sound effect localization algorithm based on the encoded auricle information and the encoded parameters.

[0031] It should be understood that the auricle information encoding algorithm can also be called an auricle information encoder, which can be used to encode auricle information; the sound effect localization algorithm encoding algorithm can also be called a sound effect localization algorithm encoder, which can be used to encode the parameters of the first sound effect localization algorithm.

[0032] In certain implementations of the first aspect, obtaining the user's auricle information includes: obtaining the auricle information when an external spatial audio switch of the terminal device is in an on state.

[0033] In certain implementations of the first aspect, the method further includes: displaying a first interface, the first interface including a first selection button for turning the external speaker space audio switch on or off; and setting the state of the external speaker space audio switch to an on state or an off state based on the user's selection operation of the first selection button.

[0034] It should be understood that the user's selection operation on the first selection button can be any preset input operation, such as clicking, sliding, etc. When the external spatial audio switch is on, the terminal device can calibrate the sound effect positioning algorithm based on the user's auricle information, thereby improving the accuracy of the sound effect positioning algorithm, restoring the overall timbre of the audio, and improving the user's audio-visual experience. The external spatial audio switch can also be any other name, for example, whether to turn on the personalized height perception of the external spatial audio.

[0035] The method further includes: when the external speaker spatial audio switch is in the on state, displaying a second interface, the second interface including a second selection button for turning the personalized perception switch on or off; based on the user's selection operation on the second selection button, setting the state of the personalized perception switch to the on state or the off state; when the personalized perception switch is in the on state, displaying a third interface, the third interface including a third selection button for selecting a camera acquisition function, a fourth selection button for selecting a user manual input function, a fifth selection button for selecting the acquisition of auricle information, and a sixth selection button for selecting the acquisition of head information, the head information including the head radius of the user; based on the user's first selection operation and the second selection operation, displaying a fourth interface or a fifth interface, the fourth interface including an image of the user acquired by the terminal device through a camera, the fifth interface including an input area for filling in the auricle information or the head information, the first selection operation being a selection operation on the third selection button or the fourth selection button, and the second selection operation being a selection operation on the fifth selection button or the sixth selection button.

[0036] It should be understood that the user's selection operation on the selection button can be any preset input operation, such as clicking, sliding, etc. When the external spatial audio switch is turned off, the terminal device will not process the audio data to be played, and the audio to be played will be directly played externally, and the audio heard by the user lacks height and stereoscopic sense; when the external spatial audio switch is turned on, the terminal device can calibrate the sound effect positioning algorithm, thereby improving the accuracy of the sound effect positioning algorithm, helping to restore the overall timbre of the audio and improving the user's audio and video experience.

[0037] In a second aspect, an audio external speaker device is provided, which is used to execute the method in any possible implementation of the first aspect. Specifically, the device includes a module for executing the method in any possible implementation of the first aspect.

[0038] In a third aspect, another audio external speaker device is provided, including a processor, the processor is coupled to a memory, and can be used to execute instructions in the memory to implement the method in any possible implementation of the first aspect. Optionally, the device also includes a memory. Optionally, the device also includes a communication interface, and the processor is coupled to the communication interface.

[0039] In one implementation, the apparatus is a terminal device. When the apparatus is a terminal device, the communication interface may be a transceiver, or an input / output interface.

[0040] In another implementation, the device is a chip configured in a terminal device. When the device is a chip configured in a terminal device, the communication interface may be an input / output interface.

[0041] In a fourth aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any possible implementation of the first aspect.

[0042] In the specific implementation process, the processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit may be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit may be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit may be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation methods of the processor and various circuits.

[0043] In a fifth aspect, a processing device is provided, including a processor and a memory. The processor is used to read instructions stored in the memory to execute the method in any possible implementation of the first aspect.

[0044] Optionally, the number of the processors is one or more, and the number of the memories is one or more.

[0045] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0046] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. This application does not limit the type of memory and the setting method of the memory and the processor.

[0047] It should be understood that the relevant data interaction process, such as sending indication information, can be a process of outputting indication information from a processor, and receiving capability information can be a process of receiving input capability information from a processor. Specifically, the processed output data can be output to a transmitter, and the input data received by the processor can come from a receiver. Among them, the transmitter and the receiver can be collectively referred to as a transceiver.

[0048] The processing device in the fifth aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0049] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect.

[0050] In a seventh aspect, a computer-readable storage medium is provided, which stores a computer program (also referred to as code, or instructions) which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0052] Figure 2 A software structure block diagram of a terminal device according to an embodiment of the present application;

[0053] Figure 3 A schematic diagram of an application scenario of an audio player method;

[0054] Figure 4 A flowchart of an audio playback method provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of envelope extraction provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a process for obtaining a first sound effect localization algorithm provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a horizontal azimuth angle provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a determination process of a second sound effect localization algorithm provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of a determination process of another second sound effect localization algorithm provided in an embodiment of the present application;

[0060] Fig.10A schematic diagram of a determination process of another second sound effect localization algorithm provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of an interface of a laptop computer displaying a first interface provided by an embodiment of the present application;

[0062] Fig.12 A schematic diagram of an interface of a laptop computer displaying a second interface provided by an embodiment of the present application;

[0063] Fig.13 A schematic diagram of a notebook computer display interface 1 provided in an embodiment of the present application;

[0064] Fig.14 A schematic diagram of a notebook computer display interface 2 provided in an embodiment of the present application;

[0065] Fig.15 A schematic diagram of a notebook computer display interface 3 provided in an embodiment of the present application;

[0066] Fig.16 A schematic diagram of a notebook computer display interface 4 provided in an embodiment of the present application;

[0067] Fig.17 A schematic diagram of a notebook computer display interface 5 provided in an embodiment of the present application;

[0068] Fig.18 A schematic diagram of a notebook computer display interface 6 provided in an embodiment of the present application;

[0069] Fig.19 A schematic diagram of a top view of a user using a laptop computer provided in an embodiment of the present application;

[0070] Fig. 20 A schematic block diagram of an audio external speaker device provided in an embodiment of the present application;

[0071] Fig.21 A schematic block diagram of another audio external speaker device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0073] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first value and the second value are only used to distinguish between different values, and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0074] It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0075] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0076] In order to better understand the terminal device in the embodiment of the present application, Figure 1 The hardware structure of the terminal device in the embodiment of the present application is described in detail.

[0077] Figure 1 This is a schematic diagram of the structure of the terminal device 100 provided in the embodiment of the present application. Figure 1As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0078] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0080] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0081] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0082] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0083] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0084] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0085] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0086] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0087] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0088] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0089] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0090] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0091] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0092] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0093] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0094] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0095] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0096] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0097] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0098] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0099] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0100] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0101] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0102] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0103] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0104] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0105] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0106] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0107] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0108] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0109] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0110] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0111] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0112] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0113] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0114] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0115] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0116] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0117] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0118] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0119] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0120] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0121] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0122] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0123] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0124] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0125] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0126] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0127] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0128] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0129] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0130] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100. The software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the terminal device 100.

[0131] Figure 2 This is a software structure block diagram of the Windows system of the terminal device 100 according to an embodiment of the present application.

[0132] like Figure 2 As shown, the layered architecture of the Windows system is mainly divided into user state and kernel state. The user state can include user application processes, system processes, service processes, etc.; the kernel state can include the kernel and device driver layer, as well as the hardware abstraction layer, etc.

[0133] The user application process can execute a range of applications, such as music, gallery, Bluetooth, WLAN, games, memos, videos, etc.

[0134] System processes and service processes may include input managers, resource managers, view systems, drawing triggers, animation triggers, time managers, task managers, etc., which can provide corresponding services for the execution of user application processes.

[0135] The input manager is used to obtain and transmit various input information from the user.

[0136] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0137] The view system can be responsible for the application interface drawing and event processing.

[0138] The task manager is used to sort the drawing tasks of multiple controls and the multiple drawing tasks of each control, and send the sorting results to the time manager.

[0139] The kernel layer and the device driver layer may include an audio driver, a graphics processing unit (GPU) driver, a display driver, a central processing unit (CPU) driver, etc., and the hardware abstraction layer may include a GPU module, a hardware configuration module, a CPU module, etc., which is not limited in the embodiments of the present application.

[0140] In order to better understand the embodiments of the present application, several terms involved in the embodiments of the present application are explained below.

[0141] 1. Head-response transfer function (HRTF): HRTF is a sound localization algorithm that includes multiple factors that can affect the user's sound source localization and is a function representation of these multiple factors. HRTF can also be called a filter. By processing the audio through this filter, the sound source position perceived by the user can be adjusted.

[0142] 2. Head related impulse responses (HRIR): HRIR is the time domain representation of HRTF. HRIR can be measured using a measuring device in a specific measurement environment. The measurement environment can be an anechoic laboratory that can obtain free sound field signals. The anechoic laboratory is equipped with a speaker whose position can be flexibly changed, a fixed object to be measured, a sensor, and a data acquisition system. The object to be measured can be a human, a special dummy, etc.

[0143] When the terminal device plays audio for the user, the user usually feels that the sound they hear falls on the display screen or keyboard of the terminal device, making the sound field of the external audio lack height and three-dimensionality. At present, in order to improve the user's sound experience, HRTF can be set in the terminal device. In this way, the audio played by the terminal device after HRTF processing makes the height of the sound heard by the user higher and more three-dimensional.

[0144] Combine the following Figure 3 , taking a laptop as an example, the application scenario of the audio player method is described in detail.

[0145] Figure 3 FIG. 3 is a schematic diagram of an application scenario 300 of an audio player method. Figure 3As shown, when the laptop plays audio externally, such as music, the sound field of the music heard by the user is located on the keyboard of the laptop, as shown in FIG. Figure 3 In the audio sound field 301. When a sound effect localization algorithm is provided in the laptop computer, the laptop computer can process the audio data of the music through the sound effect localization algorithm to obtain the external audio data. In this way, when the laptop computer plays the external audio data through the speaker, the audio sound field 302 of the music heard by the user becomes higher than the audio sound field 301, so that the music heard by the user has a higher sense of height and three-dimensionality. Among them, the sound effect localization algorithm can be used to change the transmission mode of the audio data from the laptop computer to the user's ears, so that the position of the sound field of the audio heard by the user changes. For example, the sound effect localization algorithm can be HRTF.

[0146] It should be understood that Figure 3 The application scenario 300 shown in is only an example, wherein the terminal device may also be a mobile phone, a tablet computer, a smart bracelet, or other devices; the system installed in the terminal device may be a Windows system or an Android system, etc.; when the terminal device plays audio processed by a sound effect positioning algorithm, the position of the sound field of the audio heard by the user may also undergo other changes, such as a lower position of the sound field. This application does not specifically limit the specific form of the terminal device, the type of system installed in the terminal device, and the way in which the sound field position perceived by the user changes.

[0147] At present, the HRTF set in the terminal device can be determined in the following way: in a test environment, based on a preset elevation angle and / or horizontal angle, N groups of head related impulse responses (HRIR) are obtained to obtain an HRIR data set; the HRIR data set is Fourier transformed to obtain multiple frequency domain signals of the HRIR data set; the multiple frequency domain signals are averaged to obtain a frequency domain average signal; the frequency domain average signal is inversely Fourier transformed to obtain a time domain pulse signal; the absolute value of the time domain pulse signal can be the target HRTF, and the target HRTF can also be called a height filter. The audio signal is processed by the height filter to obtain an external audio signal. When the external audio signal is played by the speaker of the terminal device, the sound field position of the sound heard by the user can be raised, so that the sound heard by the user has a sense of height and three-dimensionality, thereby improving the user's audio and video experience.

[0148] It can be seen that since HRIR is obtained in a test environment such as an anechoic laboratory that can obtain free sound field signals, and HRTF is also obtained based on the frequency domain average signal of multiple frequency domain signals, therefore, in an environment where the user actually plays audio through a terminal device, since the user's environment is different from the test environment, the user's environment may produce sound wave reflections and other phenomena, thereby hindering the propagation of sound waves; and the user and the subject's auricle information, head information, and head orientation information may be different. Therefore, it is difficult for HRTF to dynamically adapt to the personalized height perception of different users, so that the timbre of the audio heard by the user after HRTF processing may change, affecting the user's audio and video experience.

[0149] In order to solve the above problems, the present application provides an audio playback method, which first performs Fourier transform, averaging, envelope extraction and inverse Fourier transform on multiple groups of HRIRs obtained by measurement to obtain a first sound effect positioning algorithm; then, the terminal device calibrates the parameters of the first sound effect positioning algorithm based on the acquired user's auricle information, the user's head radius, the horizontal azimuth angle between the user's head and the terminal device, etc., to obtain a second sound effect positioning algorithm, and uses the second sound effect positioning algorithm to process the audio data to be played. Among them, through envelope extraction, the characteristic signal in the frequency domain average signal can be extracted, so that the frequency domain average signal is calibrated through the characteristic signal, so that after the audio data is processed by the obtained sound effect localization algorithm, it is helpful to restore the overall timbre of the audio; in addition, the parameters of the first sound effect localization algorithm are calibrated through information such as the user's auricle information, the user's head radius, and the horizontal azimuth angle between the user's head and the terminal device, so that the terminal device can process the audio data to be played based on the user's personalized characteristic information such as the user's auricle information, the user's head radius, and the horizontal azimuth angle between the user's head and the terminal device, thereby adjusting the transmission method of the sound wave based on the user's personalized characteristic information, so that the terminal device can dynamically adapt to the personalized height perception of different users, which helps to restore the overall timbre of the audio and improve the user's audio and video experience.

[0150] Combine the following Figures 4 to 19 The audio playback method of the present application is described in detail. The embodiments shown in the present application can be executed by a terminal device, such as a laptop computer, a mobile phone, a tablet computer, a smart bracelet, etc. The specific form and quantity of each device shown therein are only examples and should not constitute any limitation on the implementation of the method provided in the present application. Below, taking the terminal device as the execution subject as an example, the audio playback method of the embodiment of the present application is described in detail.

[0151] It should be understood that the terminal device can be the terminal device itself, or a chip, a chip system or a processor that supports the terminal device to implement the audio external playback method, or a logic module or software that can implement all or part of the terminal device functions. The hardware structure of the terminal device can be as follows: Figure 1 As shown, the software structure can be as follows Figure 2 As shown, this application does not make any specific limitation to this.

[0152] Figure 4 The present invention provides a flowchart of an audio playback method 400. The method 400 includes the following steps:

[0153] S401. Acquire auricle information of the user, where the auricle information includes at least one of auricle length, auricle width, auricle thickness or auricle area.

[0154] It should be understood that the user can be any user of the terminal device, such as the owner of the terminal device. The terminal device can obtain the user's auricle information in any way, such as a camera acquisition method. The auricle information can include two sets of data, one of which is the user's left ear auricle information, and the other is the user's right ear auricle information. The auricle information can also include a set of data, which can be any set of data of the user's left ear auricle information or the right ear auricle information.

[0155] S402. Determine a second sound effect localization algorithm based on auricle information and a first sound effect localization algorithm. The first sound effect localization algorithm is used to simulate a sound signal in a preset spatial orientation. The first sound effect localization algorithm is based on amplitude spectrum data. The amplitude spectrum data is obtained by Fourier transforming, averaging, extracting envelopes, and inverse Fourier transforming multiple sets of head-related impulse response (HRIR) data.

[0156] It should be understood that the preset spatial orientation may be a preset audio sound field position perceived by the user, such as 302 in scene 300. The first sound effect localization algorithm may be a function obtained based on the preset spatial orientation, and the first sound effect localization algorithm may also be referred to as a first height filter. The preset spatial orientation may be different from the initial spatial orientation, which is the audio sound field position when the terminal device plays audio that has not been processed by the sound effect localization algorithm, such as 301 in scene 300.

[0157] The second sound effect localization algorithm may also be referred to as a second height filter. The second sound effect localization algorithm may be a transfer function obtained by calibrating the first sound effect localization algorithm based on the user's auricle information by the terminal device. The first sound effect localization algorithm is calibrated by the user's auricle information so that the second sound effect localization algorithm can process the audio data based on the user's auricle information, so that the sound field position of the audio processed by the second sound effect localization algorithm perceived by the user is closer to the preset spatial orientation.

[0158] Envelope extraction can also be called envelope extraction, which can be performed by Hilbert-Huang transform (HHT), local peak detection, spline fitting and other methods. Through envelope extraction, characteristic signals such as high-frequency signals and low-frequency signals can be extracted from the frequency domain average signal obtained after inverse Fourier transform. In a specific example, Figure 5 As shown, Figure 5 (a) is a schematic diagram of envelope extraction of the frequency domain average signal of the right channel, wherein curve 501 is the frequency domain average signal before envelope extraction, and after envelope extraction, the right channel analysis signal is obtained, i.e., curve 502; Figure 5 (b) is a schematic diagram of envelope extraction of the frequency domain average signal of the left channel, wherein curve 503 is the frequency domain average signal before envelope extraction, and after envelope extraction, the left channel analysis signal is obtained, namely curve 504.

[0159] Combine the following Figure 6 The determination process of the first sound effect localization algorithm is described in detail.

[0160] Figure 6 A schematic diagram of a process for obtaining a first sound effect localization algorithm provided in an embodiment of the present application. Figure 6 As shown, first, based on the preset height, M HRIR data are obtained to obtain an HRIR data set, where M is a positive integer; Fourier transform is performed on the HRIR data sets to obtain a frequency domain signal data set corresponding to the HRIR data set; the frequency domain signal data set is averaged to obtain a frequency domain average signal, which can be determined by formula (1):

[0161]

[0162] in, is the frequency domain average signal; H k (t,f) is the kth frequency domain signal in the frequency domain signal data set; k is a positive integer greater than or equal to 1 and less than or equal to M, t is the time period, and f is the frequency.

[0163] Then, envelope extraction is performed on the frequency domain average signal, and the frequency domain average signal and the envelope extracted characteristic signal are synthesized into an analytical signal. Optionally, envelope extraction is performed by HHT. The analytical signal can be determined by formula (2):

[0164]

[0165] Among them, H(t, f) is the analytical signal, is the frequency domain average signal, t is the time period, and f is the frequency.

[0166] Then, the analytical signal is subjected to an inverse Fourier transform to obtain a time domain pulse signal. Optionally, the inverse Fourier transform may be an inverse discrete Fourier transform (IDFT), and the time domain pulse signal may be determined by formula (3):

[0167] h(t)=IDFT(H(t,f)) (3)

[0168] Among them, h(t) is the time domain pulse signal, H(t, f) is the analytical signal, t is the time period, and f is the frequency.

[0169] The absolute value of the time domain pulse signal can be called amplitude spectrum data.

[0170] In a possible implementation, the amplitude spectrum data is a first sound effect localization algorithm. The first sound effect localization algorithm can be determined by formula (4):

[0171] f(t)=abs(h(t)) (4)

[0172] Among them, f(t) is the amplitude spectrum data, h(t) is the time domain pulse signal, abs is the absolute value function, and t is the time period.

[0173] The discrete form of the amplitude spectrum data can be shown as formula (5):

[0174] f(n)=abs(h(n)) (5)

[0175] Among them, f(n) is the discrete form of the amplitude spectrum data, h(n) is the discrete form of the time domain pulse signal, and n is the time point.

[0176] S403: Process the audio data to be played by using a second sound effect positioning algorithm to obtain processed audio data, and play the processed audio data by external speaker.

[0177] It should be understood that the audio data to be played can be any audio that can be played by the terminal device, such as music, audio in a video, etc. By processing the audio data to be played through the second sound effect positioning algorithm, the transmission mode of the sound waves corresponding to the audio to be played to the user's ears can be changed, so that the user can feel that the sound field position of the audio to be played is located in a preset spatial orientation.

[0178] The audio external speaker method of the present application first performs Fourier transform, averaging, inverse Fourier transform and envelope extraction on the measured multiple groups of HRIR to obtain a first sound effect positioning algorithm; then, the terminal device calibrates the parameters in the first sound effect positioning algorithm based on the acquired user's auricle information to obtain a second sound effect positioning algorithm, and uses the second sound effect positioning algorithm to process the audio data to be played. Among them, through envelope extraction, the characteristic signal in the frequency domain pulse signal can be extracted, so that the frequency domain average signal is calibrated by the characteristic signal, so that the audio data is processed by the first sound effect positioning algorithm, which helps to restore the overall timbre of the audio; in addition, by calibrating the parameters of the first sound effect positioning algorithm based on the user's auricle information, the accuracy of the second sound effect positioning algorithm can be improved, and the terminal device can process the audio data to be played based on the user's auricle information, thereby adjusting the transmission mode of the sound wave based on the user's personalized biometrics, so that the terminal device can dynamically adapt to the personalized height perception of different users, which helps to restore the overall timbre of the audio and improve the user's audio and video experience.

[0179] As an optional embodiment, the first sound effect localization algorithm is obtained based on the linear compensation phase data and amplitude spectrum data of the speaker of the terminal device.

[0180] It should be understood that the linear compensation phase data can be used to improve the group delay inconsistency problem caused by the acoustic characteristics of the speaker of the terminal device. Linear phase compensation can be performed on the initial phase data of the speaker of the terminal device to obtain linear phase compensated data. The linear compensation phase data can be the data used when linear phase compensation is performed on the initial phase data, that is, the initial phase data and the linear compensation phase data can be synthesized into linear phase compensated data. Since the phase of the speaker of the terminal device will affect the user's perception of the height of the audio sound field, the accuracy of the first sound effect positioning algorithm can be improved by calibrating the sound effect positioning algorithm with linear compensation phase data. In this way, the sound field position of the audio felt by the user is closer to the preset spatial orientation, which can improve the user's audio and video experience.

[0181] In a possible implementation, the first sound effect localization algorithm is determined by formula (6):

[0182] S(n)=abs(h(n)).*e -jp(n) (6)

[0183] Among them, S(n) is the first sound effect localization algorithm, abs(h(n)) is the discrete form of the amplitude spectrum data, p(n) is the linear compensation phase data, and n is the time period.

[0184] As an optional embodiment, S402 can be specifically implemented in the following manner: based on the auricle information, calibrate the first sound effect localization algorithm to obtain a calibrated first sound effect localization algorithm; perform interaural time difference (ITD) calibration and / or interaural level difference (ILD) calibration on the calibrated first sound effect localization algorithm to obtain a second sound effect localization algorithm.

[0185] It should be understood that calibrating the first sound effect localization algorithm to obtain the calibrated first sound effect localization algorithm can be understood as adjusting the parameters in the first sound effect localization algorithm. The user's positioning of the sound source is not only related to the user's auricle information, but also related to ITD and ILD. Therefore, performing ITD and / or ILD calibration on the calibrated first sound effect localization algorithm can further improve the accuracy of the second sound effect localization algorithm and help restore the overall timbre of the audio.

[0186] The ITD and / or ILD calibration of the calibrated first sound effect localization algorithm can be specifically implemented through four situations.

[0187] Case 1: Perform ILD calibration on the calibrated first sound effect localization algorithm to obtain a second sound effect localization algorithm.

[0188] In a possible implementation, the calibrated first sound effect localization algorithm is subjected to ILD calibration, which is specifically implemented in the following manner: based on the horizontal azimuth angle between the user's head and the terminal device, the calibrated first sound effect localization algorithm is subjected to ILD calibration to obtain a third sound effect localization algorithm; and the third sound effect localization algorithm is determined as the second sound effect localization algorithm.

[0189] It should be understood that ILD calibration of the calibrated first sound effect localization algorithm can be understood as adjusting the parameters in the calibrated first sound effect localization algorithm. Taking the center of the user's head as the origin, a two-dimensional coordinate system is established in the plane where the line between the center of the user's head and the sound source is located. The y-axis is a straight line in the vertical direction, and the horizontal azimuth angle can be the angle between the line between the center of the user's head and the sound source and the y-axis. Optionally, when the position of the sound source is in the first quadrant of the two-dimensional coordinate system, the horizontal azimuth angle is a value greater than 0; when the position of the sound source is in the fourth quadrant of the two-dimensional coordinate system, the horizontal azimuth angle is a value less than 0. Exemplarily, Figure 7 A schematic diagram of a horizontal azimuth angle provided in an embodiment of the present application. Figure 7 As shown, point O is the center of the user's head 701, point S is the sound source, point O is the origin, a coordinate system is established in the plane where the OS line is located, the y-axis is the vertical line, and the horizontal azimuth is the angle between the y-axis and the line OS. The sound source can be the center of the speaker of the terminal device.

[0190] In a possible implementation, the second sound effect localization algorithm includes a first ILD calibration sound effect localization algorithm and a second ILD calibration sound effect localization algorithm; performing ILD calibration on the calibrated first sound effect localization algorithm, including: based on the horizontal azimuth and a preset correspondence relationship, determining a target left adjustment gain parameter from multiple left adjustment gain parameters, and determining a target right adjustment gain parameter from multiple right adjustment gain parameters, the preset correspondence relationship including a correspondence between multiple angles, multiple left adjustment gain parameters, and multiple right adjustment gain parameters; using the target left adjustment gain parameter as a coefficient of the calibrated first sound effect localization algorithm to obtain the first ILD calibration sound effect localization algorithm; using the target right adjustment gain parameter as a coefficient of the calibrated first sound effect localization algorithm to obtain the second ILD calibration sound effect localization algorithm.

[0191] It should be understood that the multiple angles may include possible horizontal azimuth angles. For example, if the relative angle between the user's head and the terminal device is between -90° and 90°, the multiple angles may include angles between -90° and 90°. The preset correspondence relationship includes: the correspondence between the multiple angles and the multiple left adjustment gain parameters; the correspondence between the multiple angles and the multiple right adjustment gain parameters, that is, each of the multiple angles corresponds to a left adjustment gain parameter and a right adjustment gain parameter. The horizontal azimuth angle is one of the multiple angles.

[0192] In a possible implementation, the multiple angles may include angles between -90° and 90°, and the preset corresponding relationship is shown in Table 1.

[0193] Table 1 Preset correspondence

[0194]

[0195]

[0196] When the horizontal azimuth angle is -90° among multiple angles, the left adjustment gain parameter is d max , the right adjustment gain parameter is d min ; When the horizontal azimuth angle is greater than -90° and less than 0°, the left adjustment gain parameter is θ' / π(d max -d min )+(d max -d min ) / 2, the right adjustment gain parameter is θ' / π(d min -d max )+(d min -d max ) / 2; when the horizontal azimuth angle is 0°, the left adjustment gain parameter is (d max +d min) / 2, the right adjustment gain parameter is (d max +d min ) / 2; when the horizontal azimuth angle is greater than 0° and less than 90°, the left adjustment gain parameter is -θ' / π(d min -d max )+(d min -d max ) / 2, the right adjustment gain parameter is -θ' / π(d min -d max )+(d min -d max ) / 2; when the horizontal azimuth angle is 90°, the left adjustment gain parameter is d min , the right adjustment gain parameter is d max Among them, d max With d min The sum of is 2, and d max Greater than d min , d max With d min It may change with the size of the terminal device, the size of the display screen of the terminal device, the position information of the speaker of the terminal device, the head radius of the user, etc.; θ' is the horizontal azimuth angle.

[0197] In a possible implementation, the first ILD calibration sound effect localization algorithm is determined by formula (7):

[0198] S L-ILD (n) = α*R(n) (7)

[0199] Among them, S L-ILD (n) is the first ILD calibration sound effect localization algorithm, R(n) is the first sound effect localization algorithm after calibration, and α is the target left adjustment gain parameter. The first sound effect localization algorithm can be S(n) in Formula 6 or f(n) in Formula 5. After S(n) or f(n) is calibrated based on the auricle information, R(n) is obtained.

[0200] In a possible implementation, the second ILD calibration sound effect localization algorithm is determined by formula (8):

[0201] S R-ILD (n) = β * R (n) (8)

[0202] Among them, S R-ILD (n) is the second ILD calibration sound effect localization algorithm, R(n) is the first sound effect localization algorithm after calibration; β is the target right adjustment gain parameter. The first sound effect localization algorithm can be S(n) in formula (6) or f(n) in formula (5). After S(n) or f(n) is calibrated based on the auricle information, R(n) is obtained.

[0203] In a possible implementation, S403 can be specifically implemented in the following manner: processing the audio data to be played through a first ILD calibration sound effect positioning algorithm to obtain first processed audio data; processing the audio data to be played through a second ILD calibration sound effect positioning algorithm to obtain second processed audio data; playing the first processed audio data and the second processed audio data in an external speaker manner.

[0204] It should be understood that the first ILD calibration sound effect positioning algorithm can also be called a left channel height filter, and the second ILD calibration sound effect positioning algorithm can also be called a right channel height filter. The first ILD calibration sound effect positioning algorithm and the second ILD calibration sound effect positioning algorithm may be different, so that the first processed audio data and the second processed audio data may be different, thereby making the audio received by the user's two ears different. By performing ILD calibration on the calibrated first sound effect positioning algorithm, the first ILD calibration sound effect positioning algorithm and the second ILD calibration sound effect positioning algorithm are obtained, so that the first ILD calibration sound effect positioning algorithm and the second ILD calibration sound effect positioning algorithm can process the audio played based on the horizontal azimuth between the user's head and the terminal device, so that the second sound effect positioning algorithm can dynamically adapt to the personalized height perception of different users, which helps to improve the accuracy of the sound effect positioning algorithm, while restoring the overall timbre of the audio and improving the user's audio and video experience.

[0205] Case 2: Perform ITD calibration on the calibrated first sound effect localization algorithm to obtain a second sound effect localization algorithm.

[0206] As an optional embodiment, ITD calibration is performed on the calibrated first sound effect localization algorithm to obtain a second sound effect localization algorithm, including: based on the horizontal azimuth angle between the user's head and the terminal device, a preset head radius and the speed of sound, ITD calibration is performed on the calibrated first sound effect localization algorithm to obtain a second sound effect localization algorithm.

[0207] It should be understood that performing ITD calibration on the calibrated first sound effect localization algorithm can be understood as adjusting the parameters of the calibrated first sound effect localization algorithm. The preset head radius is a preset value greater than 0, for example, it can be an average of the head radii of multiple users.

[0208] In a possible implementation, the second sound effect localization algorithm includes a first ITD calibration sound effect localization algorithm and a second ITD calibration sound effect localization algorithm, the first ITD calibration sound effect localization algorithm being the first sound effect localization algorithm after calibration; performing ITD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm, including: determining a first parameter based on a horizontal azimuth angle, a preset human head radius and a speed of sound; and calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain a second ITD calibration sound effect localization algorithm.

[0209] It should be understood that the first parameter may be an ITD value calculated based on the horizontal azimuth, a preset head radius and the speed of sound, and the ITD value is used to indicate the time difference of the sound wave being transmitted from the speaker of the terminal device to the two ears. By performing ITD calibration on the calibrated first sound effect localization algorithm, the terminal device can process audio data based on the ITD value, which helps to restore the overall timbre of the audio and improve the user's audio-visual experience.

[0210] In a possible implementation, the calibrated first sound effect localization algorithm is calibrated based on the first parameter to obtain a second ITD calibrated sound effect localization algorithm, including: using the first parameter as a time processing parameter in the calibrated first sound effect localization algorithm to obtain a second ITD calibrated sound effect localization algorithm.

[0211] It should be understood that the time processing parameter refers to a parameter used to adjust the time in the first sound effect localization algorithm after calibration. Calibrating the first sound effect localization algorithm after calibration based on the first parameter can be understood as adjusting the parameters of the first sound effect localization algorithm after calibration based on the first parameter, and the parameters obtained after adjustment are the parameters in the second ITD calibrated sound effect localization algorithm.

[0212] In a possible implementation, the first parameter is determined by formula (9):

[0213]

[0214] Among them, δ is the first parameter, r is the preset head radius, θ is the preset horizontal azimuth angle, and c is the speed of sound; wherein the preset horizontal azimuth angle can be 0°.

[0215] The second ITD calibration sound effect localization algorithm is determined by formula (10):

[0216] S R-ITD (n) = R (n-δ) (10)

[0217] Among them, S R-ITD (n) is the second ITD calibration sound effect localization algorithm, which can also be called the right channel height filter, R(n) is the first sound effect localization algorithm after calibration; δ is the time processing parameter. The first sound effect localization algorithm can be S(n) in formula (6) or f(n) in formula (5). After S(n) or f(n) is calibrated based on the auricle information, R(n) is obtained.

[0218] In a possible implementation, the second ITD calibration sound effect localization algorithm is determined by formula (11):

[0219] S L-ITD (n) = R (n + δ) (11)

[0220] Among them, S L-ITD (n) is the second ITD calibration sound effect localization algorithm, which can also be called the left channel height filter, R(n) is the first sound effect localization algorithm after calibration; δ is the time processing parameter. The first sound effect localization algorithm can be S(n) in formula (6) or f(n) in formula (5). After S(n) or f(n) is calibrated based on the auricle information, R(t) is obtained.

[0221] In a possible implementation, method 400 also includes: obtaining the user's head radius; determining a second parameter based on the horizontal azimuth angle, the user's head radius and the speed of sound; calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain a second ITD calibrated sound effect localization algorithm, including: using the difference between the first parameter and the second parameter as a time processing parameter in the calibrated first sound effect localization algorithm to obtain a second ITD calibrated sound effect localization algorithm.

[0222] It should be understood that calibrating the calibrated first sound effect localization algorithm based on the first parameter can be understood as adjusting the parameters of the calibrated first sound effect localization algorithm based on the first parameter, and the parameters obtained after the adjustment are the parameters in the second ITD calibrated sound effect localization algorithm. The time processing parameter can be understood as data used to adjust the time parameter in the first sound effect localization algorithm. The determination process of the second parameter can be the same as the determination process of the first parameter. The calibrated first sound effect localization algorithm is further calibrated based on the difference between the first parameter and the second parameter, and the obtained second sound effect localization algorithm enables the terminal device to process audio data based on the user's head radius, so that the terminal device can dynamically adapt to the personalized height perception of different users. This helps to improve the accuracy of the sound effect localization algorithm, restore the overall timbre of the audio, and improve the user's audio and video experience.

[0223] In a possible implementation, the first parameter is determined by formula (12):

[0224]

[0225] Among them, δ1 is the second parameter, r' is the head radius of the user, θ' is the horizontal azimuth angle, and c is the speed of sound.

[0226] The time processing parameters are determined by formula (13):

[0227]

[0228] Wherein, δ' is a time processing parameter, r' is the head radius of the user, θ is a preset horizontal azimuth angle, θ' is a horizontal azimuth angle, and c is the speed of sound; wherein, the preset horizontal azimuth angle may be 0°.

[0229] The second ITD calibration sound effect localization algorithm is determined by formula (14):

[0230] S R-ITD (n) = R (n - δ') (14)

[0231] Among them, S R-ITD (n) is the second ITD calibration sound effect localization algorithm, which can also be called the right channel height filter, R(n) is the first sound effect localization algorithm after calibration; δ' is the time processing parameter. The first sound effect localization algorithm can be S(n) in formula (6) or f(n) in formula (5). After S(n) or f(n) is calibrated based on the auricle information, R(t) is obtained.

[0232] In a possible implementation, the second ITD calibration sound effect localization algorithm is determined by formula (15):

[0233] S L-ITD (n) = R (n + δ') (15)

[0234] Among them, S L-ITD (n) is the second ITD calibration sound effect localization algorithm, which can also be called the left channel height filter, R(n) is the first sound effect localization algorithm after calibration; δ' is the time processing parameter. The first sound effect localization algorithm can be S(n) in formula (6) or f(n) in formula (5). After S(n) or f(n) is calibrated based on the auricle information, R(t) is obtained.

[0235] In a possible implementation, obtaining the head radius of the user includes: obtaining the head radius when an external spatial audio switch of the terminal device is in an on state.

[0236] It should be understood that the external spatial audio switch can be a switch for controlling whether the terminal device implements the audio external playback method, and the external spatial audio switch can be set on the system setting page of the terminal device. By setting the external spatial audio switch, the user can control whether the terminal device calibrates the sound effect positioning algorithm, so that the flexibility of the audio external playback method can be improved.

[0237] Case 3: The calibrated first sound effect localization algorithm is firstly subjected to ILD calibration and then to ITD calibration to obtain the second sound effect localization algorithm.

[0238] As an optional embodiment, the calibrated first sound effect localization algorithm is subjected to ILD calibration and ITD calibration to obtain a second sound effect localization algorithm, including: based on the horizontal azimuth angle, the calibrated first sound effect localization algorithm is subjected to ILD calibration to obtain a third sound effect localization algorithm; based on the horizontal azimuth angle between the user's head and the terminal device, a preset head radius and the speed of sound, the third sound effect localization algorithm is subjected to ITD calibration to obtain a second sound effect localization algorithm.

[0239] It should be understood that performing ILD calibration and ITD calibration on the first sound effect localization algorithm after calibration can be understood as performing ILD calibration and ITD calibration on the parameters in the first sound effect localization algorithm after calibration, and the parameters obtained after calibration are the parameters in the second sound effect localization algorithm. The processes of ILD calibration and ITD calibration are similar to those of Case 1 and Case 2, respectively, and can refer to the description above, which will not be repeated here.

[0240] In a possible implementation, the third sound effect localization algorithm includes a first ILD calibration sound effect localization algorithm and a second ILD calibration sound effect localization algorithm; the first ILD calibration sound effect localization algorithm is determined by formula (7); and the second ILD calibration sound effect localization algorithm is determined by formula (8).

[0241] The second sound effect positioning algorithm includes a first ILD-ITD calibration sound effect positioning algorithm and a second ILD-ITD calibration sound effect positioning algorithm, wherein the first ILD-ITD calibration sound effect positioning algorithm is the first ILD calibration sound effect positioning algorithm; the second ILD-ITD calibration sound effect positioning algorithm is obtained by ITD calibration of the second ILD calibration sound effect positioning algorithm based on the horizontal azimuth angle between the user's head and the terminal device, a preset head radius and the speed of sound.

[0242] Specifically, the second ILD-ITD calibration sound effect localization algorithm can be determined in the following two ways.

[0243] In one approach, the second ILD-ITD calibration sound localization algorithm is determined by formula (16):

[0244] S R-ILD-ITD (n) = β*R(n-δ') (16)

[0245] Among them, S R-ILD-ITD (n) is the second ILD-ITD calibration sound effect localization algorithm, which can also be called the right channel height filter, and δ' is the time processing parameter. Optionally, when the user's head radius is not obtained, the time processing parameter is δ. β*R(n) is the first ILD calibration sound effect localization algorithm. After calibrating β*R(n) based on the time processing parameter, the second ILD-ITD calibration sound effect localization algorithm is obtained.

[0246] In a possible implementation, the third sound effect localization algorithm includes a first ILD calibration sound effect localization algorithm and a second ILD calibration sound effect localization algorithm; the first ILD calibration sound effect localization algorithm is determined by formula (7); and the second ILD calibration sound effect localization algorithm is determined by formula (8).

[0247] The second sound effect positioning algorithm includes a first ILD-ITD calibration sound effect positioning algorithm and a second ILD-ITD calibration sound effect positioning algorithm, wherein the first ILD-ITD calibration sound effect positioning algorithm is the second ILD calibration sound effect positioning algorithm; the second ILD-ITD calibration sound effect positioning algorithm is obtained by ITD calibration of the first ILD calibration sound effect positioning algorithm based on the horizontal azimuth angle between the user's head and the terminal device, a preset head radius and the speed of sound.

[0248] In another embodiment, the second ILD-ITD calibration sound effect localization algorithm is determined by formula (17):

[0249] S L-ILD-ITD (n) = α*R(n+δ') (17)

[0250] Among them, S L-ILD-ITD (n) is the second ILD-ITD calibration sound effect localization algorithm, which can also be called the left channel height filter, and δ' is the time processing parameter. Optionally, when the user's head radius is not obtained, the time processing parameter is δ. α*R(n) is the first ILD calibration sound effect localization algorithm. After calibrating α*R(n) based on the time processing parameter, the second ILD-ITD calibration sound effect localization algorithm is obtained.

[0251] Case 4: The calibrated first sound effect localization algorithm is first subjected to ITD calibration and then to ILD calibration to obtain a second sound effect localization algorithm.

[0252] As an optional embodiment, the calibrated first sound effect localization algorithm is subjected to ILD calibration and ITD calibration to obtain a second sound effect localization algorithm, including: based on the horizontal azimuth angle between the user's head and the terminal device, a preset head radius and the speed of sound, the calibrated first sound effect localization algorithm is subjected to ITD calibration to obtain a fourth sound effect localization algorithm; based on the horizontal azimuth angle, the fourth sound effect localization algorithm is subjected to ILD calibration to obtain a second sound effect localization algorithm.

[0253] It should be understood that performing ILD calibration and ITD calibration on the first sound effect localization algorithm after calibration can be understood as performing ILD calibration and ITD calibration on the parameters in the first sound effect localization algorithm after calibration, and the parameters obtained after calibration are the parameters in the second sound effect localization algorithm. The process of performing ITD calibration and ITD calibration on the parameters of the first sound effect localization algorithm after calibration is similar to the third case, and the above description can be referred to, which will not be repeated here.

[0254] As an optional embodiment, S402 can be specifically implemented in the following manner: inputting the auricle information and the first sound effect localization algorithm into the neural network model, and outputting the second sound effect localization algorithm.

[0255] In a specific example, the auricle information includes auricle length, auricle width, auricle thickness and auricle area. The determination process of the second sound effect localization algorithm can be as follows: Figure 8 shown. Figure 8 A schematic diagram of a determination process of a second sound effect localization algorithm provided in an embodiment of the present application. Figure 8 As shown, after the parameters of the first sound effect localization algorithm and the auricle information are input into the neural network model, the parameters of the second sound effect localization algorithm are output, and the terminal device can determine the second sound effect localization algorithm based on the parameters of the second sound effect localization algorithm. The second sound effect localization algorithm can be a finite impulse response (FIR) filter with a preset number of points.

[0256] As an optional embodiment, S402 can be specifically implemented in the following manner: inputting the auricle information into the auricle information encoding algorithm to obtain the encoded auricle information; inputting the parameters of the first sound effect localization algorithm into the sound effect localization algorithm encoding algorithm to obtain the encoded parameters; and determining the second sound effect localization algorithm based on the encoded auricle information and the encoded parameters.

[0257] It should be understood that the auricle information encoding algorithm can also be called an auricle information encoder, which can be used to encode auricle information; the sound effect localization algorithm encoding algorithm can also be called a sound effect localization algorithm encoder, which can be used to encode the parameters of the first sound effect localization algorithm.

[0258] In a possible implementation, the neural network model may include an auricle information encoding algorithm, a sound effect localization algorithm encoding algorithm, and a decoding algorithm. The determination process of the second sound effect localization algorithm may be as follows: Fig. 9 shown. Fig. 9 A schematic diagram of another process of determining a second sound effect localization algorithm provided in an embodiment of the present application. Fig. 9 As shown, the parameters of the first sound effect localization algorithm are input into the sound effect localization algorithm encoding algorithm to obtain the encoded parameters, and the encoded parameters are further input into the generation network, and the parameter data output by the generation network is input into the decoding algorithm; the auricle information is input into the auricle information encoding algorithm to obtain the encoded auricle information; the encoded auricle information is input into the decoding algorithm; the decoding algorithm decodes the parameter data output by the generation network and the encoded auricle information to obtain the parameters of the second sound effect localization algorithm, and the terminal device can determine the second sound effect localization algorithm based on the parameters of the second sound effect localization algorithm.

[0259] It should be understood that the generation network can be a convolutional neural network model, for example, a deep learning model (transformer model) based on a self-attention mechanism, etc.

[0260] Fig.10 A schematic diagram of a determination process of another second sound effect localization algorithm provided in an embodiment of the present application. Fig.10 As shown, based on the virtual orientation, an HRIR data set is obtained, and the HRIR data set is successively subjected to Fourier transform, averaging, envelope extraction and inverse Fourier transform to obtain amplitude spectrum data; based on the amplitude spectrum data and linear compensation phase data, a first sound effect localization algorithm is obtained; the parameters of the first sound effect localization algorithm and the auricle information are input into the neural network model to obtain the parameters of the first sound effect localization algorithm after calibration; based on the orientation information, the parameters of the calibrated first sound effect localization algorithm are ILD calibrated to obtain the parameters of the third sound effect localization algorithm; based on the orientation information and the head information, the parameters of the third sound effect localization algorithm are calibrated to obtain the parameters of the second sound effect localization algorithm; the terminal device can obtain the second sound effect localization algorithm based on the parameters of the second sound effect localization algorithm.

[0261] The virtual orientation may be a preset height or an elevation angle corresponding to a preset height. When performing ILD calibration on the parameters in the first sound effect localization algorithm after calibration, the orientation information is the horizontal orientation angle of the user's head relative to the terminal device; when performing ITD calibration on the parameters in the third sound effect localization algorithm, the orientation information is the horizontal orientation angle between the user's head and the terminal device, and the head information is the preset head radius or the preset head radius and the user's head radius.

[0262] It should be understood that the sequence numbers of the above processes do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0263] As an optional embodiment, S401 may be specifically implemented in the following manner: when the external spatial audio switch of the terminal device is in an on state, the auricle information is obtained.

[0264] As an optional embodiment, method 400 also includes: displaying a first interface, the first interface including a first selection button for turning the external spatial audio switch on or off; and setting the state of the external spatial audio switch to an on state or an off state based on the user's selection operation of the first selection button.

[0265] It should be understood that the user's selection operation of the first selection button can be any preset input operation, such as click, slide, etc. When the external spatial audio switch is in the off state, the terminal device will not process the audio data to be played, and the audio to be played is directly played externally, and the audio heard by the user lacks height and stereoscopic sense; when the external spatial audio switch is in the on state, the terminal device can calibrate the sound effect positioning algorithm, thereby improving the accuracy of the sound effect positioning algorithm, helping to restore the overall timbre of the audio and improving the user's audio and video experience.

[0266] It should be understood that the external speaker spatial audio switch may also be any other name, and this application does not make any specific limitation on this.

[0267] In a specific example, the first interface can be as follows Fig.11 shown. Fig.11 The following is a schematic diagram of a notebook computer displaying a first interface provided in an embodiment of the present application. Fig.11 As shown, the first interface displayed by the laptop includes a first selection button 1101, and the user's selection operation on the first selection button 1101 may be clicking the first selection button 1101. Based on the user's selection operation on the first selection button 1101, the state of the external speaker spatial audio switch can be changed.

[0268] As an optional embodiment, when the external spatial audio switch is on, a second interface is displayed, the second interface includes a second selection button for turning the personalized perception switch on or off; based on the user's selection operation on the second selection button, the state of the personalized perception switch is set to the on state or the off state; when the personalized perception switch is on, a third interface is displayed, the third interface includes a third selection button for selecting a camera acquisition function, a fourth selection button for selecting a user manual input function, a fifth selection button for selecting the acquisition of auricle information, and a sixth selection button for selecting the acquisition of head information, the head information includes the user's head radius; based on the user's first selection operation and the second selection operation, a fourth interface or a fifth interface is displayed, the fourth interface includes an image of the user acquired by the terminal device through the camera, the fifth interface includes an input area for filling in auricle information or head information, the first selection operation is a selection operation on the third selection button or the fourth selection button, and the second selection operation is a selection operation on the fifth selection button or the sixth selection button.

[0269] It should be understood that the user's selection operation on the selection button can be any preset input operation, such as clicking, sliding, etc. When the personalized perception switch is in the off state, the terminal device can process the audio data to be played through the first sound effect positioning algorithm; when the personalized perception switch is in the on state, the terminal device can obtain the auricle information and calibrate the first sound effect positioning algorithm based on the auricle information, thereby improving the accuracy of the sound effect positioning algorithm, helping to restore the overall timbre of the audio and improving the user's audio and video experience.

[0270] It should be understood that the personalized perception switch may also be any other name, and this application does not make any specific limitation on this.

[0271] In a specific example, the second interface can be as follows Fig.12 shown. Fig.12 This is a schematic diagram of a laptop computer displaying a second interface provided by an embodiment of the present application. Fig.12 As shown, the second interface displayed by the laptop includes a first selection button and a second selection button 1201. When the external speaker spatial audio switch is turned on, the state of the personalized perception switch can be changed based on the user's selection operation on the second selection button 1201. The user's selection operation on the second selection button 1201 can be clicking the second selection button 1201.

[0272] It should be understood that the third interface may be interface 2 described below, the fourth interface may be interface 3 and / or interface 4 described below, and the fifth interface may be interface 5 and / or interface 6 described below.

[0273] In a possible implementation, before displaying the third interface, when the personalized perception switch is on, a sixth interface is displayed, the sixth interface including a third selection button for selecting a camera acquisition function and a fourth selection button for selecting a user manual input function; based on the user's selection operation of the third selection button and the fourth selection button, the third interface is displayed.

[0274] It should be understood that the sixth interface may be interface 1 described below.

[0275] In a possible implementation, before displaying the third interface, when the personalized perception switch is on, the seventh interface is displayed, and the seventh interface includes a fifth selection button for selecting to collect auricle information and a sixth selection button for selecting to collect head information; based on the user's selection operation of the fifth selection button and the sixth selection button, the third interface is displayed.

[0276] It should be understood that the seventh interface is similar to the sixth interface, with only the third selection button and the fourth selection button in the sixth interface replaced by the fifth selection button and the sixth selection button, which will not be described in detail here.

[0277] Combine the following Figures 13 to 19 When the personalized perception switch is turned on, the interface displayed by the terminal device is described in detail based on different user selection operations.

[0278] When the above-mentioned personalized perception switch is in the on state, interface 1 is displayed, and interface 1 includes a third selection button for selecting a camera acquisition function and a fourth selection button for selecting a user manual input function; based on the user's selection operation of the third selection button or the fourth selection button, the camera acquisition method or the manual input method is selected.

[0279] It should be understood that when the personalized perception switch is turned on, one of the third selection button and the fourth selection button can be in an unselected state, and the other can be in a selected state, and the user selects the camera acquisition method or the manual input method to collect the auricle information and / or the user's head information, and the head information includes the user's head radius. The third selection button and the fourth selection button cannot be selected at the same time, that is, the user cannot select the camera acquisition method and the manual input method at the same time.

[0280] In a specific example, the interface 1 can be as follows Fig.13 shown. Fig.13 This is a schematic diagram of a notebook computer display interface 1 provided in an embodiment of the present application. Fig.13 As shown, the interface 1 displayed by the laptop computer includes a third selection button 1301 and a fourth selection button 1302. When the personalized perception switch is on, based on the user's selection operation on the third selection button 1301, the terminal device will obtain the auricle information and / or the user's head information through camera acquisition; based on the user's selection operation on the fourth selection button 1302, the terminal device will obtain the auricle information and / or the user's head information through manual input.

[0281] In a possible implementation, when the user has selected the camera acquisition method through the third selection button, if the user clicks or slides the fourth selection button, the third selection button will be adjusted to a closed state, and the fourth selection button will be adjusted to an open state, that is, the user's selected method is adjusted to a manual input method. Vice versa, when the user has selected the manual input method through the fourth selection button, if the user clicks or slides the third selection button, the fourth selection button will be adjusted to a closed state, and the third selection button will be adjusted to an open state, that is, the user's selected method is adjusted to a camera acquisition method.

[0282] When the user selects the third selection button, interface 2 is displayed, and interface 2 includes a fifth selection button for selecting to collect auricle information and a sixth selection button for selecting to collect head information; based on the user's selection operation of the fifth selection button or the sixth selection button, selection is made to collect auricle information or collect head information.

[0283] It should be understood that one of the fifth selection button and the sixth selection button can be in an unselected state, and the other can be in a selected state, that is, the user can choose to collect auricle information or collect head information. The fifth selection button and the sixth selection button cannot be selected at the same time, that is, the user cannot choose to collect auricle information and collect head information at the same time.

[0284] In a specific example, interface 2 is as follows Fig.14 shown. Fig.14 This is a schematic diagram of a notebook computer display interface 2 provided in an embodiment of the present application. Fig.14 As shown, the interface 2 displayed by the laptop includes a fifth selection button 1401 and a sixth selection button 1402. Based on the user's selection operation on the fifth selection button 1401, the terminal device will collect auricle information; based on the user's selection operation on the sixth selection button 1402, the terminal device will collect head information.

[0285] When the user selects the third selection button and the fifth selection button, interface 3 is displayed. Interface 3 includes an image of the user acquired by the terminal device through a camera, and the image of the user includes the user's ears.

[0286] It should be understood that the terminal device can analyze and process both ears of the user to determine the auricle information; or, the terminal device can analyze and process one ear of the user to determine the auricle information.

[0287] In a specific example, the interface 3 can be as follows Fig.15 shown. Fig.15 This is a schematic diagram of a notebook computer display interface 3 provided in an embodiment of the present application. Fig.15 As shown, the interface 3 includes an image of the user obtained by the laptop computer through a camera, and the image of the user includes the user's ear. The laptop computer obtains auricle information by analyzing and processing the user's ear.

[0288] When the user selects the third selection button and the sixth selection button, interface 4 is displayed. Interface 4 includes an image of the user acquired by the terminal device through a camera, and the image of the user includes the head of the user.

[0289] It should be understood that the terminal device can analyze and process the user's head to determine the head information, wherein the head information includes the user's head radius.

[0290] In a specific example, the interface 4 can be as follows Fig.16 shown. Fig.16 This is a schematic diagram of a notebook computer display interface 4 provided in an embodiment of the present application. Fig.16 As shown, the interface 4 includes an image of the user obtained by the laptop computer through a camera, and the image of the user includes the head of the user. The laptop computer obtains the head information by analyzing and processing the head of the user.

[0291] When the user selects the fourth selection button and the fifth selection button, an interface 5 is displayed, and the interface 5 includes an input area for filling in auricle information.

[0292] It should be understood that the input area for filling in the auricle information may include at least one of the following: an area for filling in the auricle length, an area for filling in the auricle width, an area for filling in the auricle area, or an area for filling in the auricle thickness.

[0293] In a specific example, the interface 5 can be as follows Fig.17 shown. Fig.17 This is a schematic diagram of a laptop computer display interface 5 provided in an embodiment of the present application. When the user selects the manual input mode through the fourth selection button and selects to collect auricle information through the fifth selection button, the laptop computer displays interface 5. Interface 5 is as shown in FIG. Fig.17 As shown, the interface 5 includes an input area 1701 for filling in auricle information. In the case where the auricle information includes auricle length, auricle width, auricle area and auricle thickness, each type of auricle information corresponds to an input area 1701 respectively.

[0294] When the user selects the fourth selection button and the sixth selection button, interface 6 is displayed. Interface 6 includes an input area for filling in head information. The head information also includes the horizontal distance between the user's head and the terminal device and / or the vertical distance between the user's head and the terminal device.

[0295] Optionally, the horizontal azimuth angle between the user's head and the terminal device can be determined based on the above-mentioned horizontal distance and vertical distance.

[0296] It should be understood that when the head information includes the user's head radius, the above-mentioned horizontal distance and the above-mentioned vertical distance, the input area for filling in the head information includes: an input area for filling in the head radius, an input area for filling in the horizontal distance and an input area for filling in the vertical distance.

[0297] In a specific example, the interface 6 can be as follows Fig.18 shown. Fig.18This is a schematic diagram of a laptop computer display interface 6 provided in an embodiment of the present application. When the user selects the manual input mode through the fourth selection button and selects to collect head information through the sixth selection button, the laptop computer displays interface 6. Interface 6 is as follows Fig.18 As shown, interface 6 includes an input area 1801 for filling in head information. In the case where the head information includes the head radius, the horizontal distance between the user's head and the terminal device, and the vertical distance between the user's head and the terminal device, input area 1801 includes an input area for filling in the head radius, an input area for filling in the horizontal distance between the user's head and the terminal device, and an input area for filling in the vertical distance between the user's head and the terminal device.

[0298] It should be understood that the interfaces corresponding to the user selecting the third selection button and selecting the sixth selection button, the user selecting the fourth selection button and selecting the fifth selection button, and the user selecting the fourth selection button and selecting the sixth selection button are all the same as Fig.14 The interface 2 shown is similar, and only the button selected in the interface 2 is different. Please refer to the interface 2 and will not be repeated here.

[0299] Fig.19 The following is a schematic diagram of a top view of a user using a laptop computer provided in an embodiment of the present application. Fig.19 As shown, when a user is using a laptop computer, the center of the user's head is point A, and the center of the laptop computer's speaker is point B on the keyboard 1901. The straight line passing through point B and parallel to the laptop computer's display screen is straight line AB. The vertical distance between the user's head and the terminal device is the distance from point O to straight line AB, that is, distance h; the horizontal distance between the user's head and the terminal device is distance l. The horizontal azimuth angle can be determined by the following formula: θ'=arctan(l / h); where θ' is the horizontal azimuth angle, l is the horizontal distance, and h is the vertical distance.

[0300] It should be understood that the user can first select the method for obtaining the auricle information and / or head information based on the selection button, and then choose to collect the auricle information or head information based on the selection button; or the user can first choose to collect the auricle information or head information based on the selection button, and then select the method for obtaining the auricle information and / or head information based on the selection button. This application does not make specific limitations on this.

[0301] It should be understood that Figures 11 to 18 The interface diagram shown is only an example, and the interface displayed by the terminal device may include more or less content; the terminal device may be a mobile phone, tablet computer or other device, and this application does not limit the content included in the interface and the specific form of the terminal device.

[0302] Combination of the above Figures 4 to 19 , describes in detail the audio external speaker method of the present application embodiment, and the following is combined with Fig. 20and Fig.21 , describe in detail the audio external speaker device of the embodiment of the present application.

[0303] Fig. 20 Schematic block diagram of an audio external speaker device 2000 provided in an embodiment of the present application. Fig. 20 As shown, the device 2000 includes: an acquisition module 2001 and a processing module 2002.

[0304] In a possible implementation, the apparatus 2000 is used to implement the steps corresponding to the terminal device in the above method 400.

[0305] An acquisition module 2001 is used to acquire the user's auricle information, where the auricle information includes at least one of the auricle length, auricle width, auricle thickness or auricle area; a processing module 2002 is used to determine a second sound effect localization algorithm based on the auricle information and a first sound effect localization algorithm, where the first sound effect localization algorithm is used to simulate a sound signal in a preset spatial orientation, and the first sound effect localization algorithm is obtained based on amplitude spectrum data, where the amplitude spectrum data is obtained by Fourier transforming, averaging, inverse Fourier transforming and envelope extracting multiple groups of head-related impulse response (HRIR) data; the audio data to be played is processed by the second sound effect localization algorithm to obtain processed audio data, and the processed audio data is played externally.

[0306] Optionally, the first sound effect localization algorithm is obtained based on linear compensation phase data and amplitude spectrum data of the speaker of the device 2000.

[0307] Optionally, the processing module 2002 is specifically used to: calibrate the first sound effect localization algorithm based on the auricle information to obtain a calibrated first sound effect localization algorithm; perform interaural time difference ITD calibration and / or interaural intensity difference ILD calibration on the calibrated first sound effect localization algorithm to obtain a second sound effect localization algorithm.

[0308] Optionally, the processing module 2002 is specifically used to: perform ILD calibration on the calibrated first sound effect localization algorithm based on the horizontal azimuth angle of the user's head relative to the device 2000 to obtain a third sound effect localization algorithm; and determine the third sound effect localization algorithm as the second sound effect localization algorithm.

[0309] Optionally, the processing module 2002 is specifically used to: perform ILD calibration on the calibrated first sound effect localization algorithm based on the horizontal azimuth angle of the user's head relative to the device 2000 to obtain a third sound effect localization algorithm; perform ITD calibration on the third sound effect localization algorithm based on the horizontal azimuth angle between the user's head and the device 2000, a preset head radius and the speed of sound to obtain a second sound effect localization algorithm.

[0310] Optionally, the second sound effect localization algorithm includes a first ILD calibration sound effect localization algorithm and a second ILD calibration sound effect localization algorithm; the processing module 2002 is specifically used to: based on the horizontal azimuth angle and a preset correspondence relationship, determine the target left adjustment gain parameter from multiple left adjustment gain parameters, and determine the target right adjustment gain parameter from multiple right adjustment gain parameters, the preset correspondence relationship including the correspondence between multiple angles, multiple left adjustment gain parameters and multiple right adjustment gain parameters; use the target left adjustment gain parameter as the coefficient of the first sound effect localization algorithm after calibration to obtain the first ILD calibration sound effect localization algorithm; use the target right adjustment gain parameter as the coefficient of the first sound effect localization algorithm after calibration to obtain the second ILD calibration sound effect localization algorithm.

[0311] Optionally, the processing module 2002 is specifically used to: process the audio data to be played through a first ILD calibration sound effect positioning algorithm to obtain first processed audio data; process the audio data to be played through a second ILD calibration sound effect positioning algorithm to obtain second processed audio data; and play the first processed audio data and the second processed audio data externally.

[0312] Optionally, the processing module 2002 is specifically used to: perform ITD calibration on the calibrated first sound effect localization algorithm based on the horizontal azimuth angle between the user's head and the device 2000, a preset head radius and a sound speed to obtain a second sound effect localization algorithm.

[0313] Optionally, the second sound effect localization algorithm includes a first ITD calibration sound effect localization algorithm and a second ITD calibration sound effect localization algorithm, and the first ITD calibration sound effect localization algorithm is the first sound effect localization algorithm after calibration; the processing module 2002 is specifically used to: determine the first parameter based on the horizontal azimuth angle, a preset human head radius and the speed of sound; calibrate the first sound effect localization algorithm after calibration based on the first parameter to obtain the second ITD calibration sound effect localization algorithm.

[0314] Optionally, the processing module 2002 is specifically used to: use the first parameter as a time processing parameter in a calibrated first sound effect localization algorithm to obtain a second ITD calibrated sound effect localization algorithm.

[0315] Optionally, the acquisition module 2001 is also used to: obtain the user's head radius; the processing module 2002 is also used to: determine the second parameter based on the horizontal azimuth angle, the user's head radius and the speed of sound; the processing module 2002 is specifically used to: use the difference between the first parameter and the second parameter as the time processing parameter in the calibrated first sound effect localization algorithm to obtain a second ITD calibrated sound effect localization algorithm.

[0316] Optionally, the acquisition module 2001 is specifically used to: acquire the head radius when the external spatial audio switch of the device 2000 is in an on state.

[0317] Optionally, the processing module 2002 is specifically used to: input the auricle information into the auricle information encoding algorithm to obtain the encoded auricle information; input the parameters of the first sound effect localization algorithm into the sound effect localization algorithm encoding algorithm to obtain the encoded parameters; and determine the second sound effect localization algorithm based on the encoded auricle information and the encoded parameters.

[0318] Optionally, the acquisition module 2001 is specifically used to: acquire auricle information when the external speaker spatial audio switch of the device 2000 is in an on state.

[0319] Optionally, the device 2000 also includes a display module 2003 for displaying a first interface, the first interface including a first selection button for turning the external speaker space audio switch on or off; the processing module 2002 is also used to: based on the user's selection operation on the first selection button, set the state of the external speaker space audio switch to an on state or an off state.

[0320] Optionally, the display module 2003 is also used to: when the external spatial audio switch is in the on state, display a second interface, the second interface includes a second selection button for turning the personalized perception switch on or off; the processing module 2002 is also used to: based on the user's selection operation on the second selection button, set the state of the personalized perception switch to the on state or the off state; the display module 2003 is also used to: when the personalized perception switch is in the on state, display a third interface, the third interface includes a third selection button for selecting a camera acquisition function, a fourth selection button for selecting a user manual input function, a fifth selection button for selecting the acquisition of auricle information, and a sixth selection button for selecting the acquisition of head information, the head information includes the user's head radius; based on the user's first selection operation and the second selection operation, display a fourth interface or a fifth interface, the fourth interface includes an image of the user acquired by the device 2000 through the camera, the fifth interface includes an input area for filling in auricle information or head information, the first selection operation is a selection operation on the third selection button or the fourth selection button, and the second selection operation is a selection operation on the fifth selection button or the sixth selection button.

[0321] The above-mentioned apparatus 2000 has the function of implementing the corresponding steps executed by the terminal device in the above-mentioned method; the above-mentioned function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function.

[0322] In the embodiments of the present application, Fig. 20 The device 2000 in the embodiment may also be a chip, such as a SOC. Correspondingly, the processing module 2002 may be a transceiver circuit of the chip, which is not limited here.

[0323] Fig.21 A schematic block diagram of an audio external speaker device 2100 provided in an embodiment of the present application. The device 2100 includes a processor 2101, a transceiver 2102, and a memory 2103. The processor 2101, the transceiver 2102, and the memory 2103 communicate with each other through an internal connection path, the memory 2103 is used to store instructions, and the processor 2101 is used to execute the instructions stored in the memory 2103 to control the transceiver 2102 to send signals and / or receive signals.

[0324] It should be understood that the device 2100 can be specifically a terminal device in the above embodiment, and can be used to execute the various steps and / or processes corresponding to the terminal device in the above method embodiment. Optionally, the memory 2103 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. The processor 2101 may be used to execute instructions stored in the memory, and when the processor 2101 executes the instructions stored in the memory, the processor 2101 is used to execute the various steps and / or processes of the above method embodiment. The transceiver 2102 may include a transmitter and a receiver, the transmitter may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the sending action, and the receiver may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the receiving action.

[0325] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0326] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0327] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method shown in the above method embodiment.

[0328] The present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method shown in the above method embodiment.

[0329] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0330] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0331] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0332] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0333] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0334] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0335] The above is only a specific implementation of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims

1. An audio player method, characterized in that: include: Acquire auricle information of the user, where the auricle information includes at least one of auricle length, auricle width, auricle thickness or auricle area; Determine a second sound effect localization algorithm based on the auricle information and the first sound effect localization algorithm, wherein the first sound effect localization algorithm is used to simulate a sound signal at a preset spatial orientation, and the first sound effect localization algorithm is obtained based on amplitude spectrum data, and the amplitude spectrum data is obtained after a plurality of groups of head-related impulse response HRIR data are subjected to Fourier transformation, averaging, envelope extraction, and inverse Fourier transformation; The audio data to be played is processed by the second sound effect positioning algorithm to obtain processed audio data, and the processed audio data is played in an external speaker manner.

2. The method according to claim 1, characterized in that The first sound effect localization algorithm is obtained based on the linear compensation phase data of the speaker of the terminal device and the amplitude spectrum data.

3. The method according to claim 1 or 2, characterized in that: The determining of the second sound effect localization algorithm based on the auricle information and the first sound effect localization algorithm includes: Based on the auricle information, calibrate the first sound effect localization algorithm to obtain a calibrated first sound effect localization algorithm; The calibrated first sound effect localization algorithm is subjected to interaural time difference ITD calibration and / or interaural intensity difference ILD calibration to obtain the second sound effect localization algorithm.

4. The method according to claim 3, characterized in that The performing ILD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: Based on the horizontal azimuth angle between the user's head and the terminal device, performing ILD calibration on the calibrated first sound effect localization algorithm to obtain a third sound effect localization algorithm; The third sound effect localization algorithm is determined as the second sound effect localization algorithm.

5. The method according to claim 3, characterized in that: The performing ITD calibration and ILD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: Based on the horizontal azimuth angle between the user's head and the terminal device, performing ILD calibration on the calibrated first sound effect localization algorithm to obtain a third sound effect localization algorithm; Based on the horizontal azimuth angle, the preset head radius and the speed of sound, the third sound effect localization algorithm is calibrated by ITD to obtain the second sound effect localization algorithm.

6. The method according to claim 4 or 5, characterized in that: The second sound effect localization algorithm includes a first ILD calibration sound effect localization algorithm and a second ILD calibration sound effect localization algorithm; The performing ILD calibration on the calibrated first sound effect localization algorithm includes: Based on the horizontal azimuth angle and a preset corresponding relationship, determining a target left adjustment gain parameter from a plurality of left adjustment gain parameters, and determining a target right adjustment gain parameter from a plurality of right adjustment gain parameters, wherein the preset corresponding relationship includes a corresponding relationship between a plurality of angles, the plurality of left adjustment gain parameters, and the plurality of right adjustment gain parameters; Using the target left adjustment gain parameter as a coefficient of the calibrated first sound effect localization algorithm to obtain the first ILD calibrated sound effect localization algorithm; The target right adjustment gain parameter is used as a coefficient of the calibrated first sound effect localization algorithm to obtain the second ILD calibrated sound effect localization algorithm.

7. The method according to claim 6, characterized in that The method of processing the audio data to be played by the second sound effect positioning algorithm to obtain processed audio data, and playing the processed audio data by external speaker mode includes: Processing the to-be-played audio data by using the first ILD calibration sound effect positioning algorithm to obtain first processed audio data; Processing the to-be-played audio data by using the second ILD calibration sound effect positioning algorithm to obtain second processed audio data; The first processed audio data and the second processed audio data are played externally.

8. The method according to claim 3, characterized in that The performing ITD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: Based on the horizontal azimuth angle between the user's head and the terminal device, the preset head radius and the speed of sound, the calibrated first sound effect localization algorithm is ITD calibrated to obtain the second sound effect localization algorithm.

9. The method according to claim 8, characterized in that The second sound effect localization algorithm includes a first ITD calibrated sound effect localization algorithm and a second ITD calibrated sound effect localization algorithm, wherein the first ITD calibrated sound effect localization algorithm is the calibrated first sound effect localization algorithm; The performing ITD calibration on the calibrated first sound effect localization algorithm to obtain the second sound effect localization algorithm includes: Determining a first parameter based on the horizontal azimuth angle, the preset head radius and the speed of sound; The calibrated first sound effect localization algorithm is calibrated based on the first parameter to obtain the second ITD calibrated sound effect localization algorithm.

10. The method according to claim 9, characterized in that The step of calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain the second ITD calibrated sound effect localization algorithm includes: The first parameter is used as a time processing parameter in the calibrated first sound effect localization algorithm to obtain the second ITD calibrated sound effect localization algorithm.

11. The method according to claim 9, characterized in that The method further comprises: Get the head radius of the user; determining a second parameter based on the horizontal azimuth angle, the head radius of the user, and the speed of sound; The step of calibrating the calibrated first sound effect localization algorithm based on the first parameter to obtain the second ITD calibrated sound effect localization algorithm includes: The difference between the first parameter and the second parameter is used as a time processing parameter in the calibrated first sound effect localization algorithm to obtain the second ITD calibrated sound effect localization algorithm.

12. The method according to claim 11, characterized in that The obtaining the head radius of the user includes: When the external spatial audio switch of the terminal device is in the turned-on state, the head radius is obtained.

13. The method according to any one of claims 1 to 12, characterized in that The determining of the second sound effect localization algorithm based on the auricle information and the first sound effect localization algorithm includes: Inputting the auricle information into the auricle information encoding algorithm to obtain encoded auricle information; Inputting the parameters of the first sound effect localization algorithm into the sound effect localization algorithm encoding algorithm to obtain encoded parameters; Based on the encoded auricle information and the encoded parameters, the second sound effect localization algorithm is determined.

14. The method according to any one of claims 1 to 13, characterized in that The obtaining of the user's auricle information includes: When the external spatial audio switch of the terminal device is in the turned-on state, the auricle information is obtained.

15. The method according to claim 14, characterized in that The method further comprises: Displaying a first interface, wherein the first interface includes a first selection button for turning on or off the external speaker spatial audio switch; Based on the user's selection operation on the first selection button, the state of the external speaker spatial audio switch is set to an on state or an off state.

16. The method according to claim 15, characterized in that The method further comprises: When the external spatial audio switch is turned on, displaying a second interface, the second interface includes a second selection button for turning the personalized perception switch on or off; Based on the user's selection operation on the second selection button, setting the state of the personalized perception switch to an on state or an off state; When the personalized perception switch is turned on, a third interface is displayed, wherein the third interface includes a third selection button for selecting a camera acquisition function, a fourth selection button for selecting a user manual input function, a fifth selection button for selecting the acquisition of auricle information, and a sixth selection button for selecting the acquisition of head information, wherein the head information includes the head radius of the user; Based on the first selection operation and the second selection operation of the user, a fourth interface or a fifth interface is displayed, the fourth interface includes an image of the user acquired by the terminal device through a camera, the fifth interface includes an input area for filling in the auricle information or the head information, the first selection operation is a selection operation on the third selection button or the fourth selection button, and the second selection operation is a selection operation on the fifth selection button or the sixth selection button.

17. An audio external speaker, characterized in that: Comprising means for performing the method as claimed in any one of claims 1 to 16.

18. An audio external speaker, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the device is caused to execute the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 16.

20. A computer program product, characterized in that The computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 16.

Images