Audio processing method, device, electronic equipment and earphone

Abstract

The present application provides an audio processing method, device, electronic device and headset for headphones, and relates to the field of audio processing technology. The method includes: obtaining the user's head posture data; calculating the corresponding rendering coefficient according to the head posture data; rendering the audio data based on the rendering coefficient to obtain spatial audio data. The audio processing method provided in the present application renders the audio according to the user's real-time head posture, and plays the obtained spatial audio data to the user, so that the audio heard by the user has a sense of space, giving the user an immersive auditory experience; at the same time, by implementing the rendering processing of audio data on the headphones, it can get rid of the limitations brought by audio equipment and audio resources, and greatly reduce the burden on users in terms of cost and ease of use.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to an audio processing method, device, electronic device and earphone for earphones. Background Art

[0002] People's pursuit of quality of life is gradually rising. Driven by this, the functions of electronic products are becoming more and more powerful. Among them, since audio and video occupy an important position in people's leisure time, audio and video related technologies have developed rapidly in recent years, constantly introducing new ideas and bringing a variety of new experiences to users.

[0003] In audio and video technology, how to achieve high-level restoration and give users a more realistic experience through information data has always been a problem that developers are constantly exploring. In order to create an immersive auditory experience, home speakers can already achieve spatial sound field effects such as surround sound and panoramic sound.

[0004] However, when using headphones to listen to audio, due to the limitations of equipment and audio resources in the prior art, users cannot enjoy similar spatial sound field effects. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application provide an audio processing method, device, electronic device and headset for headphones to solve the technical problem in the prior art that spatial sound field effects cannot be provided to users in headphone audio-visual scenarios.

[0006] A first aspect of the present application provides an audio processing method for headphones, comprising: acquiring head posture data of a user; calculating a corresponding rendering coefficient based on the head posture data; and rendering the audio data based on the rendering coefficient to obtain spatial audio data.

[0007] Furthermore, in some embodiments, the step of calculating corresponding rendering coefficients according to the head posture data includes: calculating a current spherical coordinate basis according to the head posture data; and calculating rendering coefficients according to the current spherical coordinate basis and predetermined basis coefficients.

[0008] Further, in some embodiments, the head posture data includes a head pitch angle and a head yaw angle, wherein the step of calculating the current spherical coordinate basis according to the head posture data includes: calculating the current spherical coordinate basis according to the head pitch angle and the head yaw angle.

[0009] Further, in some embodiments, the step of calculating the current spherical coordinate basis according to the head pitch angle and the head roll angle includes: calculating the current spherical coordinate basis according to the head pitch angle and the head roll angle using the following formula:

[0010]

[0011] Where Y is the spherical coordinate basis, θ is the head pitch angle, is the head roll angle, n and m together represent the order of the spherical coordinate basis and its corresponding basis coefficient, and P represents the Legendre polynomial.

[0012] Furthermore, in some embodiments, the audio processing method also includes: determining the user's initial head posture, wherein the step of obtaining the user's head posture data includes: when the user's head posture changes relative to the initial head posture, collecting the head pitch angle and the head yaw angle as head posture data.

[0013] Further, in some embodiments, the current spherical coordinate basis includes a current first group of elements and a current second group of elements, wherein the current spherical coordinate basis is calculated based on the head pitch angle and the head yaw angle, including: calculating the current first group of elements in real time based on the head pitch angle and the head yaw angle; determining the second group of elements corresponding to the head pitch angle and the head yaw angle as the current second group of elements from at least one second group of elements pre-stored in the headset; and obtaining the current spherical coordinate basis by combining the current first group of elements with the current second group of elements.

[0014] In some other embodiments, the step of calculating the corresponding rendering coefficient based on the head posture data includes: determining the spherical coordinate basis corresponding to the head posture data as the current spherical coordinate basis based on multiple pre-stored spherical coordinate bases; and calculating the rendering coefficient based on the current spherical coordinate basis and the pre-determined basis coefficient.

[0015] Furthermore, in some embodiments, the audio processing method also includes: determining the user's initial head posture, wherein the step of obtaining the user's head posture data includes: when the user's head posture changes relative to the initial head posture, collecting the head pitch angle and the head yaw angle as head posture data.

[0016] Further, in some embodiments, the step of determining a spherical coordinate basis corresponding to the head posture data as the current spherical coordinate basis based on a plurality of pre-stored spherical coordinate basis includes: determining a spherical coordinate basis corresponding to the head pitch angle and the head yaw angle as the current spherical coordinate basis based on a plurality of pre-stored spherical coordinate basis.

[0017] Further, in some embodiments, the step of determining a spherical coordinate basis corresponding to the head pitch angle and the head yaw angle as the current spherical coordinate basis based on a plurality of pre-stored spherical coordinate bases includes: searching for the head pitch angle and the head yaw angle among a plurality of pre-stored pitch angles and a plurality of yaw angles; and when it is determined that the head pitch angle is included in the plurality of pitch angles and the head yaw angle is included in the plurality of yaw angles, directly determining the spherical coordinate basis corresponding to both the head pitch angle and the head yaw angle as the current spherical coordinate basis.

[0018] Further, in some embodiments, the step of determining a spherical coordinate basis corresponding to the head pitch angle and the head yaw angle as the current spherical coordinate basis based on a plurality of pre-stored spherical coordinate bases also includes: when it is determined that the current spherical coordinate basis cannot be directly determined from the plurality of spherical coordinate bases, calculating the current spherical coordinate basis by an interpolation algorithm.

[0019] In some embodiments, the earphone includes a left-ear earphone and a right-ear earphone, and the left-ear earphone and the right-ear earphone independently execute the audio processing method, wherein the left-ear earphone is used to calculate the left-ear rendering coefficient according to the head posture data, and render the left-ear audio data based on the left-ear rendering coefficient to obtain the left-ear spatial audio data; the right-ear earphone is used to calculate the right-ear rendering coefficient according to the head posture data, and render the right-ear audio data based on the right-ear rendering coefficient to obtain the right-ear spatial audio data.

[0020] The second aspect of the present application provides an audio processing device, including: an acquisition module for acquiring a user's head posture data; a determination module for calculating a corresponding rendering coefficient based on the head posture data; and a rendering module for rendering the audio data based on the rendering coefficient to obtain spatial audio data.

[0021] The third aspect of the present application provides an electronic device, comprising: a processor; a memory, the memory comprising computer instructions stored thereon, and when the computer instructions are executed by the processor, the processor executes the audio processing method provided in any embodiment of the first aspect of the present application.

[0022] The fourth aspect of the present application provides a headset, comprising: a receiver for receiving audio data; a chip for executing the audio processing method provided by any embodiment of the first aspect of the present application to obtain spatial audio data corresponding to the audio data; and a speaker for playing the spatial audio data.

[0023] Based on the audio processing method, device, electronic device and headphones provided in the present application, by rendering the audio according to the user's real-time head posture to obtain spatial audio data, the user can feel as if he or she is in a spatial sound field when using the headphones, giving the user an immersive auditory experience; at the same time, by implementing the rendering processing of audio data on the headphones, it is possible to get rid of the limitations brought by audio equipment and audio resources, greatly reducing the burden on the user in terms of cost and ease of use.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and should not be construed as limiting the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the drawings constitute a part of the specification and are used together with the embodiments of the present application to explain the present application, and do not constitute a limitation of the present application. Unless otherwise specified, in the drawings, the same symbols and numbers generally represent the same steps or components.

[0026] Figure 1 A schematic diagram of an exemplary audio processing system provided in accordance with an embodiment of the present application.

[0027] Figure 2 A flowchart of an audio processing method provided in one embodiment of the present application.

[0028] Figure 3 A flowchart of an audio processing method provided by another embodiment of the present application.

[0029] Figure 4 A flowchart of an audio processing method provided by another embodiment of the present application.

[0030] Figure 5 A flowchart of an audio processing method provided by another embodiment of the present application.

[0031] Figure 6 A flowchart of an audio processing method provided by another embodiment of the present application.

[0032] Figure 7 A flowchart of an audio processing method provided by another embodiment of the present application.

[0033] Figure 8 A schematic diagram of an exemplary audio processing system provided by another embodiment of the present application.

[0034] Fig. 9 A schematic diagram of the structure of an audio processing device provided in one embodiment of the present application.

[0035] Fig.10 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0036] Fig.11 A schematic diagram of the structure of an earphone provided in one embodiment of the present application. DETAILED DESCRIPTION

[0037] Application Scenario Overview

[0038] This application aims to innovate the basic function of headphones, which is multimedia audio playback.

[0039] In the prior art, the process of achieving audio playback by headphones mainly includes: receiving audio data from an audio device; playing the audio data through a speaker. Each headphone is provided with a speaker, which is placed in the user's ear canal when worn. Therefore, regardless of whether the position of the user's head changes, the sound heard by the user always comes from both sides of the head. In other words, from the user's auditory perception, the relative position and orientation between the head and the sound source are fixed, and the sound image is confined to the narrow point area of ​​the ear canal.

[0040] However, we all know that in real life, the position of most sound sources has nothing to do with the position and posture of our heads. Therefore, in a real spatial sound field, when the position or posture of our head changes, our ears follow the movement of the head and move relative to the sound source, causing the position and orientation of the sound that our ears can perceive to change.

[0041] In order to enable users to enjoy a more realistic audio-visual experience, immersive scenes such as movies with panoramic sound effects have gradually emerged in recent years. However, when users listen to such videos and audios through existing headphones, due to the limitations of existing technology, the sound played by the headphones does not have a sense of direction and cannot provide users with a near-real auditory experience.

[0042] The embodiments of the present application provide an audio processing method for headphones, which aims to improve existing headphones so that the sound played by the headphones has a sense of direction, allowing the user to feel a relative displacement between his or her head and the sound source, and then perceive that the sound comes from a virtual position in the space outside the head, thereby providing the user with a close-to-real, grand spatial sound field effect through the headphones.

[0043] In the embodiments of the present application, in order to restore the orientation of the sound signal in space, the audio data may be processed based on a head related transfer function (HRTF) to obtain the relative direction between the sound and the user.

[0044] HRTF is a sound localization algorithm, which can be understood as a set of filters, including a transfer function for each ear, so it is also called binaural transfer function. This transfer function describes the transmission process of sound waves from the sound source to the ears, and contains the necessary information for the brain to sense the direction of the sound.

[0045] When determining the transfer function for the two ears of a certain user, the following method can be used: first, determine a position point, wherein the position point has a preset distance from each ear and a preset relative orientation; place a sound source at the position point and place a microphone in each ear; play audio data through the sound source and collect audio data through the microphone; calculate the transfer function between the position point and the ear based on the played audio data and the collected audio data.

[0046] If the transfer function is updated in real time in the above manner, it is possible to obtain the changed transfer function by collecting the changed head posture data in real time when the user moves or deflects the head, and then render the audio data based on the changed transfer function to generate real-time changing spatial audio data. By establishing an HRTF transfer function between the location point of the audio device and the user's ears, spatial audio data can be generated when the audio data is played, and the spatial virtual sound image corresponding to the spatial audio data can be imaged in the direction of the audio device relative to the user's position. In this way, when the user moves or deflects the head, he can hear sounds that do not follow the position of the headphones, as if he were in a real spatial sound field.

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0048] It should be understood that, in the present application, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying an order in terms of space, time, logic, etc.

[0049] It should be understood that in the present application, the terms "up", "down", "left", "right", etc., indicating directions or positional relationships, are only used for descriptive purposes, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present application.

[0050] Exemplary Systems

[0051] Figure 1The figure shows a schematic diagram of an exemplary audio processing system provided by an embodiment of the present application. The system includes an audio device and a headset, wherein the audio device sends audio data to the headset, and the headset renders and processes the audio data after receiving it to obtain spatial audio data, and plays the spatial audio data to the user. The headset may include a receiver, a processor, and a speaker, which respectively receive audio data, render the audio data, and play the spatial audio data to the user (human ear).

[0052] It should be understood that the embodiments of the present application do not limit the type of audio device, as long as the audio device can send audio data to the headset. For example, the audio device can be various types of mobile devices, such as mobile phones, laptops, tablet computers, game consoles, etc.

[0053] At the same time, the embodiments of the present application do not limit the type of headphones. For example, when classified according to the communication method, the headphones mentioned in the embodiments of the present application may be wired headphones or wireless headphones. Among them, if a wired headset is used, then before the audio data is rendered, it is necessary to perform an analog-to-digital conversion operation on the audio data. For another example, when classified according to the wearing method, the headphones mentioned in the embodiments of the present application may be headphones, in-ear headphones or semi-in-ear headphones. Further, in some other embodiments, the headphones in the embodiments of the present application may also be devices such as a head-mounted display (Head Mount Display, HMD) with an audio output function.

[0054] In addition, the earphone in the embodiment of the present application may include a left audio channel and a right audio channel. Through the left audio channel and the right audio channel, the audio data from the audio device is presented to the left ear and the right ear of the user respectively.

[0055] Exemplary Methods

[0056] Figure 2 The figure is a flow chart of an audio processing method provided by an embodiment of the present application. The method can be, for example, Figure 1 The audio processing system shown is implemented in the earphone. Figure 2 As shown, the method includes:

[0057] S210: Acquire the user's head posture data.

[0058] Specifically, the head posture data can be used to characterize the user's head movement. For example, the head posture data can include the head pitch angle and the head yaw angle. Among them, through the corresponding mapping relationship, the head pitch angle and the head yaw angle can be used together to characterize the position change information such as the rotation, movement or swing of the user's head in the up and down, left and right, front and back directions.

[0059] When the user wears the headset, the headset can collect the user's head posture data. In one embodiment, the headset can be provided with an acquisition module, which can include one or more sensors such as a motion acceleration sensor (for example, an inertial measurement unit (IMU)), a posture sensor, a gyroscope, and a multi-axis electronic compass. When the user's head moves, the headset can sense the change in head posture in real time through the acquisition module and measure the corresponding head posture data.

[0060] S220: Calculate corresponding rendering coefficients according to the head posture data.

[0061] As mentioned above, the embodiments of the present application can use HRTF to perform convolution calculation on audio data to obtain spatial audio data. Specifically, a rendering filter constructed based on HRTF can be provided in the headset, and after receiving the audio data, the headset can render the audio data through the rendering filter (for example, linear filtering). Among them, the coefficient of the rendering filter is the rendering coefficient, and the rendering coefficient corresponds to the coefficient of HRTF.

[0062] It should be understood that the embodiments of the present application do not limit the type of rendering filter. For example, in some embodiments, the rendering filter may adopt a finite impulse response (FIR) filter. Alternatively, in other embodiments, the rendering filter may also adopt an infinite impulse response (IIR) filter.

[0063] As can be seen from the previous description, HRTF is determined based on the transfer function between the sound source and the ear. When the head posture changes, the transfer function between the sound source and the ear also changes accordingly. Therefore, each head posture has a corresponding rendering coefficient. In other words, by collecting head posture data, the rendering coefficient corresponding to each set of head posture data can be determined.

[0064] Based on this principle, in some embodiments, multiple groups of rendering coefficients corresponding to multiple groups of head postures can be obtained in advance through offline experiments, and these rendering coefficients can be summarized and stored in the headset.

[0065] For example, in an embodiment using the head pitch angle and the head yaw angle as the head posture data, a plurality of pitch angles θ and a plurality of yaw angles θ may be predetermined. And each pitch angle θ and yaw angle The rendering coefficient h corresponding to the combination of θ and θ is constructed to construct a database as shown in Table 1. When searching for the rendering coefficient, the current pitch angle and yaw angle can be collected first. For example, when the pitch angle is θ 1 The yaw angle is When , the corresponding rendering coefficient is h 11 ; When the pitch angle is θ 2 The yaw angle is When , the corresponding rendering coefficient is h 23 .

[0066] Table 1

[0067]

[0068] When storing rendering coefficient data, the pitch angle θ and the yaw angle can be assigned separately according to the required angle accuracy. Set a fixed spatial resolution. For example, set the first spatial resolution between adjacent pitch angles θ to Δθ, and the first spatial resolution between adjacent yaw angles The second spatial resolution between Then, the total data volume M of the rendering coefficients stored in the database can be calculated by the formula shown in Formula 1:

[0069]

[0070] Where N is the order of the rendering filter. According to the range of head movement, the maximum pitch angle is set to 180° and the maximum yaw angle is set to 360°. Δθ and It can be the same or different.

[0071] It is understandable that since the human head is very flexible and has a wide range of movement, there are many possible head postures. In other words, in order to achieve a smooth rendering effect, a large number of head postures need to be determined in advance. In this scheme, if you want to collect more abundant head posture data, you need to convert Δθ and The value of is set to be small, resulting in a large amount of data M of the rendering coefficients that need to be stored. For example, if Δθ and If Δθ is set to 10°, the value of M is 18×36×N=648×N. However, this resolution is too coarse. In order to make the spatial audio more realistic and smooth, it may be necessary to adjust Δθ and However, for example, Δθ and When set to 3°, the value of M is 60×120×N=7200×N, which will result in a very large amount of data that the headset needs to store.

[0072] Due to the limited storage space of the headset, it is difficult to store such a large amount of data. If this offline method of determining the rendering coefficient is adopted, in order to reduce the amount of data storage, it is necessary to reduce the value of M, that is, to reduce the number of corresponding head postures. However, this will result in that in the actual use of the user, there are some head postures that do not have accurate corresponding rendering coefficients, which reduces the accuracy of the audio data rendering process and cannot achieve better rendering effects.

[0073] Therefore, the embodiment of the present application provides a solution for online calculation of rendering coefficients. According to the head posture data collected in real time, real-time calculation is performed in the headset to obtain the corresponding rendering coefficients, and the audio data is rendered based on the rendering coefficients to obtain spatial audio data. This online calculation method greatly reduces the storage burden of the headset, and at the same time can make the rendering effect more delicate and optimize the user experience.

[0074] S230: Render the audio data based on the rendering coefficient to obtain spatial audio data.

[0075] After the rendering coefficient is calculated, the earphone can render the audio data from the audio device through the rendering filter to obtain spatial audio data.

[0076] After the audio data is convolved by the rendering filter, the resulting spatial audio data contains the position information of the audio device relative to the headphones (that is, relative to the user's ears), thereby forming a spatial virtual sound image in the corresponding position. Therefore, when the user turns or moves his head, the sound heard by the user (that is, the spatial audio data played by the headphones) can change relatively according to the position of the head.

[0077] Based on the audio processing method provided in the embodiment of the present application, by rendering the audio according to the user's real-time head posture and playing the obtained spatial audio data to the user, the spatial virtual sound and image corresponding to the spatial audio data can always be imaged at the position of the audio device, thereby achieving realistic spatial virtual sound and image. No matter how the user moves, the subjective feeling of the user when hearing the sound through the headphones is no longer constrained by the headphones themselves, but immersed in the vast spatial sound field, making the sound effects of multimedia videos from any source more three-dimensional and full, enhancing the sense of reality of the "sound" being in the scene, and enriching the user's experience.

[0078] Furthermore, the audio processing method provided in the embodiment of the present application can get rid of the limitations brought by audio devices and audio resources by implementing the rendering processing of audio data on headphones, greatly reducing the burden on users in terms of cost and ease of use. At the same time, compared with setting the rendering unit in the audio device, the solution of the present application can avoid the delay in communication between the audio device and the headphones, quickly respond to the user's real-time head posture to adapt to the current rendering processing, and effectively reduce the cost of rendering operations.

[0079] like Figure 3 As shown, in one embodiment of the present application, Figure 2 Step S220 of the illustrated embodiment may specifically include:

[0080] S3210: Calculate the current spherical coordinate basis according to the head posture data.

[0081] S3220: Calculate rendering coefficients according to the current spherical coordinate basis and predetermined basis coefficients.

[0082] Specifically, the rendering coefficients can be calculated based on the spherical coordinate basis and the basis coefficients, where each set of head postures corresponds to a set of spherical coordinate basis, i.e., a spherical coordinate basis vector; meanwhile, the basis coefficients are a predetermined matrix whose number of columns is equal to the order of the rendering filter and whose number of rows is equal to the number of elements of the spherical coordinate basis vector.

[0083] Since the spherical coordinate basis can be calculated based on the head posture data, in this embodiment, it is only necessary to store the predetermined basis coefficient matrix in the headset, and then the corresponding rendering coefficient can be obtained through online calculation after obtaining the user's head posture data.

[0084] Preferably, in one embodiment, the head posture data may include a head pitch angle and a head roll angle. Figure 4 As shown, in this embodiment, S3210 may specifically include:

[0085] S4210: Calculate the current spherical coordinate basis according to the head pitch angle and the head roll angle.

[0086] Optionally, when calculating the current spherical coordinate basis according to the head pitch angle and the head yaw angle, a formula as shown in Formula 2 may be used for calculation.

[0087]

[0088] Where Y is the spherical coordinate basis, θ is the head pitch angle, is the head roll angle, n and m together represent the order of the spherical coordinate basis and its corresponding basis coefficient, P represents the Legendre polynomial, and i is the imaginary number symbol.

[0089] Specifically, the spherical coordinate basis can be calculated by a linear combination of modes (including monopoles, dipoles, and multipoles). Here, each mode has a corresponding order label, which can be characterized by n and m. For example, n = m = 0 corresponds to the first-order mode (monopole); n = 1, m = 0 corresponds to the second-order mode, n = 1, m = 1 corresponds to the third-order mode (the second and third-order modes are both dipoles); n = 2, m = 0 corresponds to the fourth-order mode... (when n is greater than or equal to 2, the corresponding modes are all multipoles), and so on. All modes can be characterized by n and m, that is, as long as n and m are determined, the corresponding mode can be directly determined, where m is less than or equal to n.

[0090] It can be seen from Formula 2 that the number of elements in the spherical coordinate basis vector is the same as the number of modes involved in the calculation. Therefore, corresponding to each mode, the order of each element in the spherical coordinate basis vector can also be characterized by n and m. For example, in one embodiment, the current spherical coordinate basis can be calculated using the first to third order modes, and the current spherical coordinate basis contains 3 elements; in another embodiment, more modes can be used to calculate the spherical coordinate basis, for example, the current spherical coordinate basis is calculated using the first to sixth order modes, and the current spherical coordinate basis contains 6 elements.

[0091] Correspondingly, when calculating the rendering coefficient according to the spherical coordinate basis and the base coefficient, the formula shown in Formula 3 can be used for calculation.

[0092]

[0093] Wherein, h(k) is the kth element of the rendering coefficient (when the total order of the rendering filter is N, 1≤k≤N), c is the basis coefficient, and L is the value of n and m, which is used to determine the number of modes used when calculating the spherical coordinate basis (i.e., the order of the spherical coordinate basis). Here, the order of the spherical coordinate basis Z = 2L + L (L-1) / 2 + 1. For example, when L = 2, the order Z of the current spherical coordinate basis is 6, that is, the first to sixth order modes are used to calculate the current spherical coordinate basis.

[0094] As mentioned above, the basis coefficient is a predetermined matrix whose number of columns is equal to the order N of the rendering filter and whose number of rows is equal to the order Z of the spherical coordinate basis vector. Therefore, in this embodiment, the amount of data M' that needs to be pre-stored is the number of elements N×Z in the basis coefficient matrix.

[0095] It can be understood that in the calculation method provided in the embodiment of the present application, under normal fitting requirements, there is no need to use too many modes. Preferably, the number of modes can be 6, that is, Z=6. Then, the amount of data that needs to be stored in the embodiment of the present application is M'=6×N, which is much smaller than the amount of data M in the aforementioned embodiment (for example, 7200×N).

[0096] It can be seen that compared with the solution of pre-storing rendering coefficients corresponding to multiple groups of head postures, the solution provided by this embodiment only needs to pre-store a base coefficient matrix with a very small data amount, and can obtain the rendering coefficients corresponding to all head postures through online calculation, thereby greatly reducing the amount of data stored in the headset and alleviating the storage burden of the headset.

[0097] Optionally, in another embodiment, the current spherical coordinate basis may include a current first group of elements and a current second group of elements. Here, S4210 may specifically include:

[0098] According to the head pitch angle and the head roll angle, the current first group of elements is calculated in real time;

[0099] Determine, from at least one second group of elements pre-stored in the headset, a second group of elements corresponding to the head pitch angle and the head roll angle as a current second group of elements;

[0100] Combine the current first group of elements with the current second group of elements to obtain the current spherical coordinate basis.

[0101] Specifically, as mentioned above, the spherical coordinate basis is a vector including multiple elements, each of which can be calculated by the above formula 1. In this embodiment, in order to reduce the burden of online calculation, the elements of the current spherical coordinate basis can be divided into two groups, namely the current first group of elements and the current second group of elements, wherein the current second group of elements can be obtained in advance by offline calculation and stored in the headset, and the remaining current first group of elements are still obtained by online real-time calculation.

[0102] Preferably, since the amount of calculation for the higher-order modes is larger, the elements corresponding to the higher-order modes can be used as the current second group of elements, which are pre-calculated and stored in the earphone, thereby more effectively reducing the amount of online calculation.

[0103] Alternatively, in other embodiments, the elements of the current spherical coordinate basis may be divided into more groups, and an offline or online calculation method may be selected for each group of elements according to actual needs, thereby setting a more suitable amount of calculation and storage for different application scenarios.

[0104] In another embodiment of the present application, Figure 5 As shown, Figure 2 Step S220 of the illustrated embodiment may specifically include:

[0105] S5210: Determine, based on a plurality of pre-stored spherical coordinate bases, a spherical coordinate base corresponding to the head posture data as a current spherical coordinate base.

[0106] S5220: Calculate rendering coefficients according to the current spherical coordinate basis and predetermined basis coefficients.

[0107] In this embodiment, multiple spherical coordinate basis vectors corresponding to multiple groups of head postures can be obtained in advance through offline experiments and stored in the headphones together with a predetermined basis coefficient matrix corresponding to the rendering filter. Then, during the use of the headphones, the current spherical coordinate basis vector corresponding to the current head posture can be obtained based on the pre-stored spherical coordinate basis vector, and the rendering coefficient corresponding to the current head posture can be calculated in real time in combination with the pre-stored basis coefficient matrix.

[0108] Specifically, the pitch angle θ and the yaw angle θ can be calculated with a fixed spatial resolution. The corresponding spherical coordinate basis is stored. The total data volume P of the stored multiple spherical coordinate basis vectors can be calculated by formula 4:

[0109]

[0110] where Z is the number of elements in each spherical coordinate basis vector.

[0111] At the same time, as mentioned above, the number of columns of the basis coefficient matrix is ​​equal to the order N of the rendering filter, and the number of rows is equal to the number of elements Z of the spherical coordinate basis vector. Therefore, the amount of data Q of the basis coefficients that need to be stored in advance (i.e., the number of elements in the basis coefficient matrix) can be calculated by formula 5:

[0112] Q = Z·N (Formula 5)

[0113] At this time, the total amount of data that needs to be stored in advance is shown in Formula 6:

[0114]

[0115] In the calculation method provided in the embodiment of the present application, there is no need to use too many modes. Preferably, the number of modes can be 6, that is, Z=6. For the 3° solution, the total data volume P+Q is (7200+N)×6.

[0116] It can be understood that in order to achieve a better fitting effect, the order of the filter is often large. When performing spatial audio rendering, the order N of the rendering filter needs to be set to, for example, greater than or equal to 10. If the order of the rendering filter is set to N=10, in Δθ and In the 3° solution, the amount of data required to be stored in the aforementioned offline calculation embodiment is 7200×10=72000, while the amount of data required to be stored in this embodiment is (7200+10)×6=43260, which reduces the amount of data storage by more than one third.

[0117] It can be seen that compared with the solution of pre-storing rendering coefficients corresponding to multiple groups of head postures, the solution provided by this embodiment significantly reduces the amount of data stored in the headset, thereby alleviating the storage burden of the headset.

[0118] Furthermore, in one embodiment of the present application, Figure 5 Step S5210 of the illustrated embodiment may specifically include:

[0119] According to a plurality of spherical coordinate bases stored in advance, a spherical coordinate base corresponding to the head pitch angle and the head yaw angle is determined as a current spherical coordinate base.

[0120] As mentioned above, in the embodiment of the present application, the headset may pre-store spherical coordinate bases corresponding to multiple sets of head posture data. In the scheme of using the head pitch angle and the head yaw angle as the head posture data, multiple pitch angles θ, multiple yaw angles θ, and multiple yaw angles θ may be pre-determined. And each pitch angle θ and yaw angle The spherical coordinate basis Y corresponding to the combination of is constructed as shown in Table 2.

[0121] Table 2

[0122]

[0123] After confirming the user's current head pitch angle and head yaw angle, the corresponding spherical coordinate basis can be found in the pre-stored pitch angle and yaw angle as the current spherical coordinate basis.

[0124] Specifically, Figure 6 As shown, this step may include:

[0125] S6210: Searching for a head pitch angle and a head roll angle among a plurality of pre-stored pitch angles and a plurality of roll angles.

[0126] S6220: Determine whether the pre-stored multiple pitch angles and multiple yaw angles include the head pitch angle and the head yaw angle.

[0127] If yes, execute S6230; if no, execute S6240.

[0128] S6230: Directly determine the spherical coordinate basis corresponding to both the head pitch angle and the head roll angle as the current spherical coordinate basis.

[0129] S6240: Obtain the current spherical coordinate basis by calculating through an interpolation algorithm.

[0130] Specifically, if the user's current head pitch angle cannot be found in the predetermined pitch angles, and / or the user's current head yaw angle cannot be found in the predetermined yaw angles, the spherical coordinate bases corresponding to the two pitch angles closest to the current head pitch angle and / or the two yaw angles closest to the current head yaw angle can be found, and the spherical coordinate bases corresponding to the current head pitch angle and head yaw angle can be calculated according to a linear interpolation algorithm.

[0131] In this way, the limitations of predetermined discrete data can be effectively compensated, making the rendering effect more delicate and smooth, giving users a better auditory experience.

[0132] Figure 7 FIG. 2 is a flow chart of an audio processing method provided by another embodiment of the present application. Figure 7 As shown, in this embodiment, Figure 2 The audio processing method shown also includes:

[0133] S7110: Determine the user's initial head posture.

[0134] Specifically, when the user wears the headset, the headset can determine that the user has put on the headset through sensors and the like, and confirm the user's current head posture.

[0135] Here, S210 may specifically include:

[0136] S7120: When the user's head posture changes relative to the initial head posture, the head pitch angle and the head yaw angle are collected as head posture data.

[0137] After confirming the user's initial head posture, the headset can collect the changes in the head pitch angle and head yaw angle only when the user's head posture changes. In this way, the head posture data can be obtained more quickly and the accuracy of the head posture data can be further ensured, providing users with a better spatial audio experience.

[0138] In a preferred embodiment of the present application, as in Figure 8 In the audio processing system shown, the earphone may specifically include a left earphone and a right earphone, and the left earphone and the right earphone respectively and independently execute the audio processing methods provided in the embodiments of the present application.

[0139] Specifically, the audio device can send audio data to the left earphone and the right earphone respectively. Since the left ear and the right ear are located in different positions, the sounds heard by the left ear and the right ear are also different. Therefore, when the user's head moves, the spatial perception of the left ear and the right ear may also change to varying degrees.

[0140] In this embodiment, the left ear earphone can calculate the left ear rendering coefficient according to the head posture data, and render the left ear audio data based on the left ear rendering coefficient to obtain the left ear spatial audio data; at the same time, the right ear earphone can calculate the right ear rendering coefficient according to the head posture data, and render the right ear audio data based on the right ear rendering coefficient to obtain the right ear spatial audio data.

[0141] Based on the audio processing method provided in this embodiment, the rendering processing of audio data can be implemented on the left earphone and the right earphone respectively, thereby improving the accuracy of spatial audio restoration and bringing a more realistic auditory experience to users.

[0142] Exemplary Devices

[0143] Fig. 9 FIG. 9 is a schematic diagram of the structure of an audio processing device 900 provided in an embodiment of the present application. The audio processing device 900 may be provided in the earphone of the aforementioned exemplary audio processing system, for example.

[0144] like Fig. 9 As shown, the audio processing device 900 includes: an acquisition module 910, used to obtain the user's head posture data; a determination module 920, used to calculate the corresponding rendering coefficient based on the head posture data; and a rendering module 930, used to render the audio data based on the rendering coefficient to obtain spatial audio data.

[0145] Specifically, the acquisition module 910 may include one or more sensors such as a motion acceleration sensor (e.g., an inertial measurement unit (IMU)), a posture sensor, a gyroscope, a multi-axis electronic compass, etc. When the user's head moves, the acquisition module can sense the change in the head posture in real time and measure the corresponding head posture data.

[0146] The determination module 920 can calculate the rendering coefficient corresponding to the head posture of the current user according to the head posture data by online calculation.

[0147] After the determination module 920 obtains the rendering coefficient, the rendering module 930 can render the audio data from the audio device through the rendering filter according to the rendering coefficient, thereby obtaining spatial audio data. After the audio data is convolved by the rendering filter, the obtained spatial audio data contains the orientation information, thereby forming a spatial virtual sound image in the corresponding orientation. Therefore, when the user turns or moves his head, he can hear the sound that changes relatively according to the orientation of the head.

[0148] Based on the audio processing device provided in the embodiment of the present application, by rendering the audio according to the user's real-time head posture and playing the obtained spatial audio data to the user, the spatial virtual sound and image corresponding to the spatial audio data can always be imaged at the position of the audio device, thereby achieving realistic spatial virtual sound and image. No matter how the user moves, the subjective feeling of the user when hearing the sound through the headphones is no longer constrained by the headphones themselves, but immersed in the vast spatial sound field, making the sound effects of multimedia videos from any source more three-dimensional and full, enhancing the sense of reality of the "sound" being in the scene, and enriching the user's experience.

[0149] In some embodiments, when the determination module 920 obtains the rendering coefficient by online calculation, it can first calculate the current spherical coordinate basis according to the head posture data, and then further calculate the rendering coefficient according to the current spherical coordinate basis and the predetermined basis coefficient.

[0150] Preferably, the head posture data may include a head pitch angle and a head yaw angle. The determination module 920 may determine the head posture data according to the head pitch angle and the head yaw angle by: Figure 4 The current spherical coordinate basis is calculated by the formula shown in Formula 1 in the method embodiment shown in the figure. Figure 4 Formula 2 in the illustrated method embodiment calculates the rendering coefficient.

[0151] In the solution provided in this embodiment, it is only necessary to pre-store a basis coefficient matrix with a very small amount of data, and the rendering coefficients corresponding to all head postures can be obtained through online calculation, thereby greatly reducing the amount of data stored in the headset and alleviating the storage burden of the headset.

[0152] Optionally, in some embodiments, the current spherical coordinate basis may include a current first group of elements and a current second group of elements. When the determination module 920 calculates the current spherical coordinate basis according to the head pitch angle and the head yaw angle, the current first group of elements may be calculated in real time according to the head pitch angle and the head yaw angle, and at least one second group of elements pre-stored in the headset may be used as the current second group of elements by determining the second group of elements corresponding to the current head pitch angle and the head yaw angle as the current second group of elements, and then combining the current first group of elements with the current second group of elements to obtain the current spherical coordinate basis.

[0153] Specifically, in this embodiment, in order to reduce the burden of online calculation, the elements of the current spherical coordinate basis can be divided into two groups, namely the current first group of elements and the current second group of elements, wherein the current second group of elements can be obtained in advance by offline calculation and stored in the headset, and the remaining current first group of elements are still obtained by online real-time calculation.

[0154] Preferably, since the computational complexity of the higher-order modes is greater, the elements corresponding to the higher-order modes can be used as the current second group of elements, which are pre-calculated and stored in the headset, thereby more effectively reducing the online computational complexity. Alternatively, in other embodiments, the elements of the current spherical coordinate basis can be divided into more groups, and an offline or online computation method can be selected for each group of elements according to actual needs, thereby setting a more suitable computational complexity and storage amount for different application scenarios.

[0155] In some other embodiments, when the determination module 920 obtains the rendering coefficient through online calculation, it can determine the spherical coordinate basis corresponding to the head posture data as the current spherical coordinate basis based on multiple pre-stored spherical coordinate bases, and then calculate the rendering coefficient based on the current spherical coordinate basis and the pre-determined basis coefficient.

[0156] Specifically, the determination module 920 can find the spherical coordinate basis corresponding to the current head posture data in a pre-stored database as the current spherical coordinate basis, and then can calculate the rendering coefficient corresponding to the current user's head posture through online calculation based on the current spherical coordinate basis and the predetermined basis coefficients.

[0157] After the determination module 920 obtains the rendering coefficient, the rendering module 930 can render the audio data from the audio device through the rendering filter according to the rendering coefficient, thereby obtaining spatial audio data. After the audio data is convolved by the rendering filter, the obtained spatial audio data contains the orientation information, thereby forming a spatial virtual sound image in the corresponding orientation. Therefore, when the user turns or moves his head, he can hear the sound that changes relatively according to the orientation of the head.

[0158] Furthermore, in an embodiment of the present application, multiple spherical coordinate basis vectors corresponding to multiple groups of head postures can be obtained in advance through offline experiments, and stored in the headset together with a predetermined basis coefficient matrix corresponding to the rendering filter. Then, during the use of the headset, the current spherical coordinate basis vector corresponding to the current head posture can be obtained according to the pre-stored spherical coordinate basis vector, and the rendering coefficient corresponding to the current head posture can be calculated in real time in combination with the pre-stored basis coefficient matrix. Compared with the solution of pre-storing rendering coefficients corresponding to multiple groups of head postures, the solution provided by this embodiment significantly reduces the amount of data stored in the headset, reducing the storage burden of the headset.

[0159] Furthermore, in some embodiments, the acquisition module 910 can also be used to determine the initial head posture of the user. On this basis, the acquisition module 910 can collect the head pitch angle and the head yaw angle as the head posture data when confirming that the user's head posture has changed relative to the initial head posture. In this way, the head posture data can be obtained more quickly, and the accuracy of the head posture data can be further ensured, providing the user with a better spatial audio experience.

[0160] On this basis, the determination module 920 can determine the spherical coordinate basis corresponding to the head pitch angle and the head yaw angle as the current spherical coordinate basis according to a plurality of pre-stored spherical coordinate basis.

[0161] Specifically, the determination module 920 can search for the head pitch angle and the head yaw angle in the pre-stored multiple pitch angles and multiple yaw angles; when it is determined that the head pitch angle is included in the multiple pitch angles and the head yaw angle is included in the multiple yaw angles, the spherical coordinate basis corresponding to the head pitch angle and the head yaw angle is directly determined as the current spherical coordinate basis; when it is determined that the current spherical coordinate basis cannot be directly determined in the multiple spherical coordinate bases, the current spherical coordinate basis is calculated by an interpolation algorithm. In this way, the limitations brought by the pre-determined discrete data can be effectively compensated, the rendering effect is more delicate and smooth, and the user can have a better auditory experience.

[0162] Preferably, in some embodiments, the earphone may include a left earphone and a right earphone, and the left earphone and the right earphone respectively have their own acquisition module, determination module and rendering module, so as to independently execute the audio processing method provided by any of the above embodiments.

[0163] Specifically, the left ear determination module of the left ear earphone is used to calculate the left ear rendering coefficient according to the head posture data, and the left ear rendering module is used to render the left ear audio data based on the left ear rendering coefficient to obtain the left ear spatial audio data; the right ear determination module of the right ear earphone is used to calculate the right ear rendering coefficient according to the head posture data, and the right ear rendering module is used to render the right ear audio data based on the right ear rendering coefficient to obtain the right ear spatial audio data.

[0164] Based on the audio processing method provided in this embodiment, the rendering processing of audio data can be implemented on the left earphone and the right earphone respectively, thereby improving the accuracy of spatial audio restoration and bringing a more realistic auditory experience to users.

[0165] It should be understood that the functions and technical effects of each module in the audio processing device 900 provided in the above embodiment can refer to the corresponding contents in the exemplary method and exemplary process, and will not be described one by one here.

[0166] Exemplary Devices

[0167] Fig.10 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device includes: a processor 1010; a memory 1020, the memory 1020 includes computer instructions stored thereon, and when the computer instructions are executed by the processor 1010, the processor 1010 executes the audio processing method provided by any of the above embodiments.

[0168] Fig.11 FIG. 1 is a schematic diagram of the structure of an earphone 1100 provided in an embodiment of the present application. Fig.11 As shown, the headset includes: a receiver 1110 for receiving audio data; a chip 1120 for executing the audio processing method provided in any of the above embodiments to obtain spatial audio data corresponding to the audio data; and a speaker 1130 for playing the spatial audio data.

[0169] Exemplary computer-readable storage media

[0170] Other embodiments of the present application further provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the audio processing method described in any of the above embodiments is implemented. It can be understood that the computer storage medium can be any tangible medium, such as a floppy disk, CD-ROM, DVD, hard disk drive or network medium.

[0171] The block diagrams of the devices, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. It will be appreciated by those skilled in the art that these devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "comprise", "include", "have", etc. are open words, to "including but not limited to", and can be used interchangeably with them, unless the context clearly indicates otherwise. The vocabulary "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0172] It should also be noted that in the apparatus, device and method of the present application, each module or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0173] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the above aspects, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0174] The above description is intended to illustrate and describe the technical solution of the present application. In addition, this description is not intended to limit the embodiments of the present application to the scope of the above disclosed forms. Although multiple exemplary aspects and embodiments have been discussed in the above content, those skilled in the art can easily obtain other variations, modifications, changes, additions and sub-combinations based on the above content.

[0175] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An audio processing method for headphones, characterized in that: include: Get the user's head posture data; Calculate the corresponding rendering coefficient in real time according to the head posture data; Rendering the audio data based on the rendering coefficients to obtain spatial audio data, The head posture data includes a head pitch angle and a head yaw angle, and the corresponding rendering coefficient is obtained by real-time calculation according to the head posture data, including: Calculating a current spherical coordinate basis according to the head pitch angle and the head yaw angle; The rendering coefficients are obtained by real-time calculation according to the current spherical coordinate basis and the predetermined basis coefficients. The current spherical coordinate basis includes a current first group of elements and a current second group of elements, and the current spherical coordinate basis is calculated according to the head pitch angle and the head yaw angle, including: Calculating the current first group of elements in real time according to the head pitch angle and the head yaw angle; Determine, from at least one second group of elements pre-stored in the headset, a second group of elements corresponding to the head pitch angle and the head roll angle as the current second group of elements; The current first group of elements is combined with the current second group of elements to obtain the current spherical coordinate basis.

2. The audio processing method according to claim 1, characterized in that: The step of calculating the current spherical coordinate basis according to the head pitch angle and the head yaw angle comprises: According to the head pitch angle and the head yaw angle, the current spherical coordinate basis is calculated using the following formula: Where Y is the spherical coordinate basis, θ is the head pitch angle, is the head roll angle, n and m together represent the order of the spherical coordinate basis and its corresponding basis coefficient, and P represents the Legendre polynomial.

3. The audio processing method according to claim 1, characterized in that: Also includes: determining an initial head pose of the user, Wherein, obtaining the user's head posture data includes: When the head posture of the user changes relative to the initial head posture, the head pitch angle and the head yaw angle are collected as the head posture data.

4. The audio processing method according to claim 1, characterized in that: The earphone comprises a left earphone and a right earphone, and the left earphone and the right earphone respectively and independently execute the audio processing method. in, The left ear earphone is used to calculate a left ear rendering coefficient according to the head posture data, and render the left ear audio data based on the left ear rendering coefficient to obtain left ear spatial audio data; The right ear earphone is used to calculate a right ear rendering coefficient according to the head posture data, and render the right ear audio data based on the right ear rendering coefficient to obtain right ear spatial audio data.

5. An audio processing device, characterized in that: include: An acquisition module is used to acquire the user's head posture data; A determination module, configured to calculate a corresponding rendering coefficient in real time according to the head posture data, wherein the head posture data includes a head pitch angle and a head yaw angle; A rendering module, used to render the audio data based on the rendering coefficient to obtain spatial audio data, Wherein, the determining module is further configured to: Calculating a current spherical coordinate basis according to the head pitch angle and the head yaw angle; The rendering coefficients are obtained by real-time calculation according to the current spherical coordinate basis and the predetermined basis coefficients, wherein the current spherical coordinate basis includes the current first group of elements and the current second group of elements, The determining module is further configured to: Calculating the current first group of elements in real time according to the head pitch angle and the head yaw angle; Determine, from at least one second group of elements pre-stored in the headset, a second group of elements corresponding to the head pitch angle and the head roll angle as the current second group of elements; The current first group of elements is combined with the current second group of elements to obtain the current spherical coordinate basis.

6. An electronic device comprising: processor; A memory, wherein the memory includes computer instructions stored thereon, and when the computer instructions are executed by the processor, the processor is caused to perform the audio processing method according to any one of claims 1 to 4.

7. A headset, characterized in that: include: A receiver for receiving audio data; A chip, configured to execute the audio processing method according to any one of claims 1 to 4, to obtain spatial audio data corresponding to the audio data; A speaker is used to play the spatial audio data.

Images