Audio data processing method, device, equipment and storage medium

Abstract

The present application provides an audio data processing method, device, equipment and storage medium, the method comprising: obtaining sound source position information; determining the target relative position between the virtual sound source and the head according to the sound source position information; when the target relative position is a preset position, determining the target enhanced head transfer function corresponding to the target relative position, wherein the target enhanced head transfer function is obtained by performing spectral enhancement processing on the first original head transfer function, and the first original head transfer function is used to characterize the phase and frequency response of the virtual sound source from the target relative position to the head; processing the audio data to be processed according to the target enhanced head transfer function to obtain the target position audio data. This technical solution can enhance the directionality of the target position audio data, so that the user can more easily identify the direction of the sound, produce a stereo surround sound effect, and bring a better spatial immersive experience to the user.

Description

Technical Field

[0001] The present application relates to the field of audio processing, and in particular to audio data processing methods, devices, equipment and storage media. Background Art

[0002] Current true wireless stereo (TWS) headphones generally play audio data based on dual-channel stereo processing technology, and listeners can only perceive the sound coming from the left and right ears when wearing headphones. Since dual channels lack uniform coverage of the entire listening area, and each channel has very small differences in sound pressure level and phase response in the listening area, listeners often find it difficult to distinguish the direction of the sound, and cannot experience realistic surround sound effects. Summary of the invention

[0003] The present application provides an audio data processing method, apparatus, device and storage medium to solve the technical problem that two-channel stereo processing technology cannot produce surround sound effects.

[0004] In a first aspect, a method for processing audio data is provided, comprising:

[0005] Get the location information of the sound source;

[0006] Determine the target relative position between the virtual sound source and the head according to the sound source position information;

[0007] When the target relative position is a preset position, determining a target enhanced head transfer function corresponding to the target relative position, wherein the target enhanced head transfer function is obtained by performing spectrum enhancement processing on a first original head transfer function, and the first original head transfer function is used to characterize a phase and frequency response of the virtual sound source transmitted from the target relative position to the head;

[0008] The audio data to be processed is processed according to the target enhancement head transfer function to obtain target orientation audio data.

[0009] In this technical solution, by obtaining the sound source position information, and determining the target relative position between the virtual sound source and the head according to the sound source position information, when the target relative position is a preset position, the target enhanced head transfer function corresponding to the target relative position is determined, and finally the audio data to be processed is processed according to the target enhanced head transfer function, thereby obtaining the target position audio data. Since the target enhanced head transfer function is obtained by performing spectral enhancement processing on the first original head transfer function that characterizes the phase and frequency response of the virtual sound source transferred from the target relative position to the head, when the target relative position is a preset position, the target enhanced head transfer function is used to process the audio data to be processed, which can enhance the directionality of the target position audio data, so that the user can more easily identify the direction of the sound, produce a stereo surround sound effect, and bring a better spatial immersive experience to the user.

[0010] In combination with the first aspect, in a possible implementation, determining the target enhanced head transfer function corresponding to the relative orientation of the target includes: calculating the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function, wherein the second original head transfer function is the original head transfer function of an azimuth point symmetrical to the azimuth point corresponding to the virtual sound source in space; performing spectrum enhancement processing on the first original head transfer function according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function to obtain the target enhanced head transfer function. By using the auditory excitation energy of the original head transfer functions of the two azimuth points in the symmetrical direction to modify the original head transfer function of one of the two azimuth points, the difference in sound perception in the symmetrical direction can be enhanced, thereby making the sense of direction of the audio data stronger and making it easier for the audience to distinguish the direction of the sound.

[0011] In combination with the first aspect, in a possible implementation, the calculation of the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function includes: calculating the critical band energy of the first original function, convolving the critical band energy of the first original function with the perceptible energy diffusion function to obtain the auditory excitation energy corresponding to the first original head transfer function, the perceptible energy diffusion function being used to represent the diffusion phenomenon of signal energy on the basilar membrane; calculating the critical band energy of the second original function, convolving the critical band energy of the second original function with the perceptible energy diffusion function to obtain the auditory excitation energy corresponding to the second original head transfer function. By convolving the critical band energy of the original head transfer function at two azimuth points in the symmetric direction with the perceptible energy diffusion function, the auditory excitation energy of the original head transfer function at two azimuth points in the symmetric direction can be obtained.

[0012] In combination with the first aspect, in a possible implementation, the spectrum enhancement processing of the first original head transfer function is performed according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function to obtain the target enhanced head transfer function, including: calculating a target weight coefficient according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function; and performing smooth weighted processing on the first original head transfer function according to the target weight coefficient to obtain the target enhanced head transfer function. By calculating the weight coefficient using the auditory excitation energy of the original head transfer functions at two azimuth points in the symmetric direction, and using the weight coefficient to weight one of the original head transfer functions, components such as the spectrum peak and minimum value can be made more prominent, thereby increasing the difference in excitation energy in the two symmetric directions.

[0013] In combination with the first aspect, in a possible implementation, after the audio data to be processed is processed according to the target enhancement head transfer function to obtain the target azimuth audio data, it also includes: obtaining the historical azimuth audio data corresponding to the target azimuth audio data, the historical azimuth audio data including at least one frame of azimuth audio data closest to the target azimuth audio data in time; overlapping and adding the historical azimuth audio data with the target azimuth audio data to obtain audio output data. By overlapping and adding the signals of adjacent frames, the audio signal at the angle switching point can sound smoother, improving the listening experience.

[0014] In combination with the first aspect, in a possible implementation, the processing of the audio data to be processed according to the target enhanced head transfer function to obtain the target azimuth audio data includes: convolving the target enhanced head transfer function with the audio data to be processed to obtain audio convolution data; and performing an inverse discrete Fourier transform on the audio convolution data to obtain the target azimuth audio data. By convolving the enhanced head transfer function with the audio data to be processed and performing an inverse discrete Fourier transform, a frequency domain audio signal with azimuth information can be obtained and converted into a time domain audio signal that can be output and played.

[0015] In combination with the first aspect, in a possible implementation, the method further includes: performing interpolation processing on an original head transfer function in an original head transfer function library to expand the original head transfer function in the original head transfer function library, wherein the original head transfer function in the original head transfer function library is obtained by measurement. By expanding the original head transfer function in the original head transfer function library, the original head transfer function of the unmeasured spatial point can be completed.

[0016] In a second aspect, an audio data processing device is provided, comprising:

[0017] A position acquisition module is used to obtain the sound source position information;

[0018] A position determination module, used to determine the target relative position between the virtual sound source and the head according to the sound source position information;

[0019] a function determination module, configured to determine a target enhanced head transfer function corresponding to the target relative position when the target relative position is a preset position, wherein the target enhanced head transfer function is obtained by performing spectrum enhancement processing on a first original head transfer function, and the first original head transfer function is used to characterize a phase and frequency response of the virtual sound source transferred from the target relative position to the head;

[0020] The audio data processing module is used to process the audio data to be processed according to the target enhancement head transfer function to obtain the target position audio data.

[0021] In a third aspect, an audio device is provided, comprising a memory and one or more processors, wherein the memory is connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the audio device implements the audio data processing method of the first aspect.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the audio data processing method of the first aspect.

[0023] The present application can achieve the following technical effects: since the target enhanced head transfer function is obtained by spectrally enhancing the first original head transfer function that characterizes the phase and frequency response of the virtual sound source transmitted from the target relative position to the head, when the target relative position is a preset position, the audio data to be processed is processed by the target enhanced head transfer function, which can enhance the directionality of the audio data at the target position, making it easier for the user to distinguish the direction of the sound, generating a stereo surround sound effect, and bringing the user a better spatial immersive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the vertical azimuth and horizontal azimuth provided in an embodiment of the present application;

[0026] Figure 3A schematic diagram showing the relationship between several position points in the space of the embodiment of the present application;

[0027] Figure 4 A flowchart of another audio data processing method provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the relative positions of several spatial position points provided in an embodiment of the present application;

[0029] Figure 6 is a structural schematic diagram of an audio data processing device provided in an embodiment of the present application;

[0030] Figure 7 It is a structural diagram of an audio device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0032] The technical solution of the present application can be applied to audio playback scenarios, and can be specifically applied to processing audio data to obtain directional sound, thereby producing surround sound effects. The technical solution of the present application can be applied to electronic playback devices for playing audio data, such as headphones, speakers, and terminals with audio players.

[0033] To facilitate understanding, some concepts involved in the technical solution of this application are first introduced.

[0034] 1. Head related transfer function (HRTF), also known as head transfer function (ATF), describes the transmission process of sound waves from the sound source to the ears, and is used to express the result of comprehensive filtering of sound waves by human physiological structures (such as head, auricle, and trunk). The physical process of sound waves emitted by the sound source and then scattered by the human physiological structure to reach the ears can be regarded as a linear time-invariant sound wave system. HRTF is the frequency domain transfer function of this sound wave system, which is defined as:

[0035]

[0036]

[0037] Among them, PL and PR are the complex sound pressures generated by the point sound source at the left and right ears of the listener respectively; Po is the complex sound pressure at the center of the head when the head does not exist; r is the distance from the sound source to the center of the head; θ is the horizontal azimuth angle of the sound source; is the vertical azimuth of the sound source; ω is the angular frequency of the sound wave; α is a set of parameters related to physiological structure and size. The transmission path of the sound wave to the center of the head is different, and the transmission medium through which the sound wave reaches the human ear is different, so the HRTF is different; when the center position of the head is fixed, the sound source is in different spatial positions, which corresponds to different transmission paths, and thus corresponds to different HRTFs; when the spatial position of the sound source remains unchanged, the center position of the head changes, which will also cause the transmission path to change, and thus will also correspond to different HRTFs; therefore, when the relative position between the sound source and the center of the head changes, the HRTF will change, and thus, the relative position between a sound source and the center of the head corresponds to a HRTF.

[0038] 2. Head-related impulse response (HRIR), also known as binaural impulse response, is the expression of HRTF in the time domain. It is a Fourier transform pair with the head transfer function and is expressed as:

[0039]

[0040]

[0041] Next, the technical solution of this application is introduced, see Figure 1 , Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application is provided. The method can be applied to the various electronic playback devices for playing audio data mentioned above, such as Figure 1 As shown, the method comprises the following steps:

[0042] S101, obtaining sound source location information.

[0043] The sound source position information is parameter information obtained by the head tracker to reflect the change of head movement posture. When there is a sound source in space, the parameter information obtained by the head tracker to reflect the change of head movement posture can also be used to indicate the relative angle in the relative position relationship between the sound source and the head. Specifically, Figure 2 As shown, the pitch motion of the head can be expressed as a rotation with the X-axis as the axis. The pitch motion of the head brings about the vertical azimuth angle between the sound source and the head. The left and right rotation of the head can be expressed as a rotation with the Y axis as the axis, which causes the horizontal azimuth angle θ between the sound source and the head to change; the left and right swing of the head can be expressed as a rotation with the Z axis as the axis, which causes the vertical azimuth angle θ between the sound source and the head to change. changes have occurred.

[0044] In a specific implementation, the head position can be tracked by a six-axis gyroscope, an inertial measurement unit, or a six-axis head tracking sensor to obtain the sound source position information. The head tracker can be built into the electronic playback device or can be connected to the electronic playback device as a peripheral device.

[0045] S102: Determine the target relative position between the virtual sound source and the head according to the sound source position information.

[0046] In the process of playing audio data, the sound source corresponding to the audio data is a virtual sound source, and the sound source position information can also be used to indicate the relative angle in the relative position relationship between the virtual sound source and the head. The target relative orientation between the virtual sound source and the head specifically includes the vertical azimuth angle between the virtual sound source and the head. and horizontal azimuth angle θ. The vertical azimuth angle between the virtual sound source and the head The horizontal azimuth angle θ can be calculated based on the position of the virtual sound source and the center of the head in space. Since the position of the center of the head in space changes with the movement of the head, the head movement causes the sound source position information to change. Therefore, by converting the sound source position information into position data, the vertical azimuth angle between the virtual sound source and the head can be obtained. and the horizontal azimuth angle θ.

[0047] S103: When the target relative orientation is a preset orientation, determine a target enhancement head transfer function corresponding to the target relative orientation.

[0048] Among them, the preset orientation refers to the orientation that easily causes auditory confusion to the audience. Since the ears are arranged horizontally, the audience can easily judge the sound source in the left and right directions. Due to symmetry, the time difference and intensity difference of the sound located in the front half plane and the symmetrically located in the back half plane in space reaching the human ear are the same. It is difficult for the audience to distinguish the orientation based on the auditory time difference and auditory intensity difference. The unprocessed head transfer function will cause the audience to confuse up and down and front and back, so that the orientation of the virtual sound source cannot be accurately distinguished. The position that is easy to confuse is called "confusion cone". Therefore, for the sound at the confusion cone, more complex components of spectral information are needed to correct it, that is, the difference between the spectra at the confusion cone needs to be increased. Exemplarily, the preset orientation can refer to the front in the front-to-back symmetrical direction, that is, when the virtual sound source is located in front of the head, the target enhanced head transfer function corresponding to the target relative orientation needs to be determined; the preset orientation can also refer to the top in the up-down symmetrical direction, that is, when the virtual sound source is located above the head, the target enhanced head transfer function corresponding to the target relative orientation needs to be determined.

[0049] In an embodiment of the present application, the audio data is processed by a target enhanced head transfer function to increase the difference between the spectra at the confusing cones. The target enhanced head transfer function is obtained by performing spectrum enhancement processing on the first original head transfer function, and the first original head transfer function is used to characterize the phase and frequency response of the virtual sound source transmitted from the target relative position to the head, that is, the first original head transfer function describes the physical process of the sound waves emitted by the virtual sound source passing through the human physiological structure to reach the ears without additional interference / processing. Since the relative position between a sound source and the center of the head in space corresponds to a head transfer function, when the target relative position is obtained, the head transfer function corresponding to the target relative position (i.e., the first original head transfer function) needs to be determined to characterize the result of filtering the sound waves emitted by the virtual sound source by the human physiological structure during the process of the virtual sound source being transmitted from the target relative position to the head and reaching the ears. Specifically, the first original head transfer function can be obtained from an original head transfer function library, wherein the original head transfer function library stores original head transfer functions corresponding to different relative orientations, and the original head transfer function refers to the process of filtering sound waves when they pass through the human physiological structure and reach the ears without additional interference / processing.

[0050] Performing spectrum enhancement processing on the first original head transfer function refers to processing the spectrum corresponding to the first original head transfer function by increasing the spectrum difference in symmetrical directions (such as front-to-back direction and up-down direction) so that the processed spectrum has enough spectrum information to support accurate sound source localization.

[0051] In some feasible implementations, the spectrum corresponding to the first original head function can be processed based on the energy ratio of the critical band (i.e., Bark domain) represented by the "Bark scale". Among them, the critical band is a professional term in hearing and psychology. The cochlea is the sensor organ of hearing in the inner ear. The critical band refers to the frequency bandwidth of the auditory filter generated by the cochlear structure, that is, the sound frequency band. In the critical band, the first single tone perception will be disturbed by the auditory masking of the second single tone. Researchers have found through research that the structure of the human ear will roughly resonate with 24 frequency points, so the sound signal will also present 24 critical bands in the frequency band.

[0052] When the target relative orientation is a preset orientation (such as the target relative orientation is forward or upward), the target enhanced head transfer function corresponding to the first original head transfer function may be determined through the following steps A1-A2:

[0053] Step A1: Calculate the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function.

[0054] Among them, the second original head transfer function is the original head transfer function of the azimuth point that is symmetrical with the azimuth point corresponding to the virtual sound source in space. Symmetry refers to symmetry on the horizontal plane or symmetry on the vertical plane, space refers to the virtual space where the virtual sound source and the head are located, and the azimuth point corresponding to the virtual sound source may refer to the position point of the virtual sound source in space. The azimuth point that is symmetrical to the azimuth point corresponding to the virtual sound source may refer to a position point that is symmetrical with the position point of the virtual sound source in space on the vertical plane; it may also refer to a position point that is symmetrical with the position point of the virtual sound source in space on the horizontal plane with the center of the head as the center of symmetry. For example, see Figure 3 , assuming that the position of the virtual sound source in space is Figure 3 The position point P1 in the image is located above and in front of the head. The position point of the center of the head in space is Figure 3 The position point P2 in the figure, then the azimuth point symmetrical to the virtual sound source can be Figure 3 The position point P3 in the figure is located below the head and is symmetrical with P1 in the vertical plane (ie, P1 and P3 are symmetrical in the vertical direction). The azimuth point symmetrical to the azimuth point corresponding to the virtual sound source can also be Figure 3 Position point P4 is located at the back of the head and is symmetrical with P1 on the horizontal plane (ie, P1 and P4 are symmetrical in the front-to-back direction).

[0055] Since the auditory excitation energy is used to reflect the frequency band energy diffusion phenomenon at different positions, by calculating the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function, it is convenient to change the spectrum information at the confusion cone, so as to increase the spectrum difference at the confusion cone.

[0056] Specifically, the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function can be calculated in the following manner: calculate the critical band energy of the first original function, and convolve the critical band energy of the first original function with the perceptible energy diffusion function to obtain the auditory excitation energy corresponding to the first original head transfer function; calculate the critical band energy of the second original function, and convolve the critical band energy of the second original function with the perceptible energy diffusion function to obtain the auditory excitation energy corresponding to the second original head transfer function.

[0057] The critical band energy is calculated as follows:

[0058] X a (z,i)=a0(z)X i (z, i) = a0(z)

[0059]

[0060] Among them, X a (z,i) is the critical band energy of a critical band, i represents the time frame, a0(z) represents the frequency-dependent attenuation coefficient of the sound wave, X(ω,i) represents the power spectrum of a critical band, ω zl and ω zh are the low-frequency and high-frequency boundaries of a critical frequency band Z.

[0061] By calculating the power spectra of the first primitive function and the second primitive function in each critical frequency band (i.e., 24 critical frequency bands) respectively according to the above formula 5, and summing the power spectra in each critical frequency band respectively, and multiplying them with the frequency-dependent attenuation coefficient of the sound wave, the critical frequency band energy of the first primitive function and the second primitive function can be obtained.

[0062] The perceptible energy diffusion function is used to represent the diffusion phenomenon of signal energy on the basilar membrane. The diffusion function equation is as follows:

[0063]

[0064] Among them, △z=zv refers to the difference between two critical frequency bands.

[0065] The calculation formula of excitation energy is as follows:

[0066]

[0067] After the critical band energies of the first primitive function and the second primitive function are obtained, the auditory excitation energies of the first primitive function and the second primitive function can be calculated according to the above formula 6 and formula 7.

[0068] Since the critical frequency band is the frequency spectrum corresponding to the frequency point where the human ear structure resonates, the auditory excitation energy of the original head transfer function of the two azimuth points in the symmetric direction is obtained by convolving the critical frequency band energy of the original head transfer function of the two azimuth points in the symmetric direction with the perceptible energy diffusion function, which helps to increase the frequency spectrum difference between the two azimuth points in the symmetric direction and thus obtain richer frequency spectrum information.

[0069] Step A2: performing spectrum enhancement processing on the first original head transfer function according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function to obtain a target enhanced head transfer function.

[0070] Here, performing spectral enhancement processing on the first original head transfer function according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function means correcting the spectral information of the first original head transfer function based on the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function, so as to amplify the difference between the spectrum corresponding to the first original head transfer function and the spectrum corresponding to the second original head transfer function, so as to maximize the difference in the spectrum at the confusion cone (here referring to the azimuth point corresponding to the virtual sound source and the azimuth point symmetrical to the azimuth point corresponding to the virtual sound source).

[0071] In a feasible implementation, a target weight coefficient can be calculated based on the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function. According to the target weight coefficient, the first original head transfer function is smoothed and weighted to obtain a target enhanced head transfer function.

[0072] Among them, the target weight coefficient is: rate(z,i)=X e1 (z,i) / X e2 (z, i); According to the target weight coefficient, the first original head transfer function is smoothed and weighted by using a first-order low-pass filter to filter and smooth the enhanced head transfer function between two adjacent frames to obtain a smoothed target enhanced head transfer function: HRTF2(f z ,i)=(1-α)×HRTF2(f z ,i-1)+α×HRTF1(f z ,i)×rate(z,i), where α is a constant that can be defined according to the degree of smoothing required, rate(z,i) is the target weight coefficient, and X e1 (z, i) is the auditory excitation energy corresponding to the first original head transfer function, X e2 (z, i) is the auditory excitation energy corresponding to the second original head transfer function, HRTF2(f z , i) is the target enhanced head transfer function, HRTF1(f z , i) is the first original head transfer function, f z is the frequency band of each critical band and i represents the index of the time frame.

[0073] By calculating the weight coefficient using the auditory excitation energy of the original head transfer function at the confusion cone and weighting the original head transfer function using the weight coefficient, components such as the spectrum peak and minimum value can be made more prominent, thereby increasing the spectrum difference at the confusion cone, making it easier for the audience to distinguish and thus more easily distinguish the direction of the virtual sound source.

[0074] In a specific implementation, after determining the target relative orientation between the virtual sound source and the head and determining that the target relative orientation is the preset orientation, the target enhanced head transfer function corresponding to the target relative orientation can be determined according to the method described in the above steps A1-A2. It is also possible to refer to the method described in the above steps A1-A2 for each original head transfer function corresponding to the preset orientation in the original head transfer function library, perform spectrum enhancement processing on each original head transfer function corresponding to the preset orientation in the original head transfer function library, and obtain enhanced head transfer functions corresponding to different preset orientations. Then, after determining the target relative orientation between the virtual sound source and the head and determining that the target relative orientation is the preset orientation, determine the enhanced head transfer function corresponding to the target relative orientation from the enhanced head transfer functions corresponding to different preset orientations to obtain the target enhanced head transfer function.

[0075] S104, processing the audio data to be processed according to the target enhancement head transfer function to obtain target orientation audio data.

[0076] The audio data to be processed refers to audio data without direction, the audio data to be processed may refer to mono audio data, or the audio data to be processed may refer to stereo audio data (including left channel audio data and right channel audio data). The audio data to be processed is processed according to the target enhanced head transfer function to obtain the target orientation audio data, which means combining the target head transfer function with the audio data to be processed so that the sound played based on the target orientation audio data is the sound emitted from the relative orientation of the target, that is, the target orientation audio data has a sense of spatial orientation.

[0077] Specifically, the audio data to be processed can be processed through the following steps B1-B2 to obtain the target direction audio data:

[0078] Step B1: convolve the target enhanced head transfer function with the audio data to be processed to obtain audio convolution data.

[0079] Since the head transfer function includes transfer functions corresponding to the left ear and the right ear, the target enhanced head transfer function is convolved with the audio data to be processed, including convolving the target enhanced head transfer function corresponding to the left ear with the audio data to be processed and convolving the target enhanced head transfer function corresponding to the right ear with the audio data to be processed. The convolution formula is as follows:

[0080]

[0081]

[0082] Wherein, S is the audio data to be processed. When the audio data to be processed is stereo audio data, S in Formula 8 is the left channel audio data, and S in Formula 9 is the right channel audio data; is the target enhanced head transfer function corresponding to the left ear, is the target enhanced head transfer function corresponding to the right ear.

[0083] By convolving the target enhanced head transfer function with the audio data to be processed, modulation of the audio data to be processed is achieved, so that the sound corresponding to the audio data to be processed has a sense of spatial orientation.

[0084] B2. Perform inverse discrete Fourier transform on the audio convolution data to obtain target direction audio data.

[0085] Here, performing an inverse discrete Fourier transform on the audio convolution data refers to converting the audio data in the frequency domain into audio data in the time domain, so that the target orientation audio data obtained by the transformation can be played by the electronic playback device. Among them, the audio convolution data can be converted into an audio impulse response with reference to the aforementioned formula 3 and formula 4 to obtain the target orientation audio data, and the target orientation audio data is audio data in the time domain. Among them, after obtaining the target orientation audio data, the electronic playback device can output the target orientation audio data in a time sequence to play the audio corresponding to the audio data to be processed.

[0086] By convolving the enhanced head transfer function with the audio data to be processed and performing an inverse discrete Fourier transform, a frequency domain audio signal with azimuth information can be obtained and converted into a time domain audio signal that can be output and played.

[0087] Optionally, when the target relative orientation is not a preset orientation (such as the target relative orientation is left, right, etc.), the audio data to be processed can be directly processed according to the first original head transfer function to obtain the target orientation audio data. Among them, the principle of obtaining the target orientation audio data by processing the audio data to be processed according to the first original head transfer function is the same as the principle of obtaining the target orientation audio data by processing the audio data to be processed according to the target enhanced head transfer function. The methods introduced in B1-B2 above can be referred to and will not be repeated here. Since the target orientation is not a preset orientation, it is not easy to cause auditory confusion to the audience. Therefore, directly processing the audio data to be processed according to the first original head transfer function can simplify the processing flow and realize rapid processing of the audio data to be processed.

[0088] In the above Figure 1In the corresponding technical solution, the target relative position between the virtual sound source and the head is determined based on the sound source position information, and then the target enhanced head transfer function corresponding to the target relative position is determined, and finally the audio data to be processed is processed according to the target enhanced head transfer function, thereby obtaining the target position audio data. Since the target enhanced head transfer function is obtained by performing spectral enhancement processing on the first original head transfer function that characterizes the phase and frequency response of the virtual sound source transferred from the target relative position to the head, when the target relative position is a preset position, the target enhanced head transfer function is used to process the audio data to be processed, which can enhance the directionality of the target position audio data, making it easier for users to identify the direction of the sound, produce stereo surround sound effects, and bring users a better spatial immersive experience.

[0089] See also Figure 4 , Figure 4 A flowchart of another audio data processing method provided in an embodiment of the present application is provided. The method can be applied to the various electronic playback devices for playing audio data mentioned above, such as Figure 4 As shown, the method comprises the following steps:

[0090] S201, obtaining sound source location information.

[0091] S202: Determine the target relative position between the virtual sound source and the head according to the sound source position information.

[0092] S203: When the target relative orientation is a preset orientation, determine a target enhancement head transfer function corresponding to the target relative orientation.

[0093] S204: Process the audio data to be processed according to the target enhancement head transfer function to obtain target orientation audio data.

[0094] The specific implementation of steps S201 to S204 may refer to the description of the aforementioned steps S101 to S104, which will not be repeated here.

[0095] S205: Acquire historical orientation audio data corresponding to the target orientation audio data.

[0096] The historical position audio data includes at least one frame of position audio data that is closest to the target position audio data in time, and the time sequence of the position audio data included in the historical position audio data is before the target position audio data. Since the target position audio data is audio data in the time domain, the historical position audio data corresponding to the target position audio data can be obtained through the time window.

[0097] Specifically, the time sequence corresponding to the previous HRIR of the first HRIR of the target orientation audio data can be taken as the last time sequence of the time window, and the HRIR corresponding to each time sequence in the time window is obtained, so as to obtain the historical orientation audio data corresponding to the target orientation audio data. In this case, the number of audio frames corresponding to the time sequence in the time window is equal to the number of audio frames contained in the historical orientation audio data; optionally, the time sequence corresponding to the last HRIR of the target orientation audio data can be taken as the last time sequence of the time window, and the HRIR corresponding to each time sequence in the time window is obtained, so as to obtain the historical orientation audio data and the target orientation audio data corresponding to the target orientation audio data. In this case, the number of audio frames corresponding to the time sequence in the time window is equal to the number of audio frames contained in the historical orientation audio data plus one.

[0098] S206: Overlap and add the historical orientation audio data corresponding to the target orientation audio data with the target orientation audio data to obtain audio output data.

[0099] Specifically, the historical orientation audio data corresponding to the target orientation audio data and the target orientation audio data can be averaged to overlap and add the historical orientation audio data corresponding to the target orientation audio data with the target orientation audio data, thereby obtaining audio output data, and the audio output data refers to the audio data output and played through the electronic playback device. Among them, the audio orientation data is composed of multiple HRIRs, and averaging the historical orientation audio data corresponding to the target orientation audio data and the target orientation audio data means averaging the HRIR corresponding to the historical orientation audio data corresponding to the target orientation audio data and the HRIR corresponding to the target orientation audio data. By averaging the historical orientation audio data corresponding to the target orientation audio data and the target orientation audio data, it is possible to prevent the phenomenon of signal jump discontinuity at the connection point when the angle is switched.

[0100] It should be noted that the above steps S205 to S206 can be executed each time the sound source position information is obtained, that is, in the process of the electronic playback device outputting and playing audio data, the above steps S201 to S206 are executed to output the played audio data. In this way, it can be ensured that the sound broadcast by the electronic playback device is always smooth; the above steps S205 to S206 can also be executed when the sound source position information changes significantly (that is, each time the angle information is updated), that is, in the process of the electronic device outputting and playing audio data, when the sound source position information does not change, the above steps S201 to S204 are executed, and when the sound source position information changes significantly, the above steps S201 to S206 are executed. In this way, some processing steps can be omitted and the output and playback efficiency of audio data can be improved.

[0101] In the above technical solution, after obtaining the target orientation audio data, the problem of sound amplitude jump caused by spatial angle switching can be solved by overlapping and adding the historical orientation audio data corresponding to the target orientation audio data with the target orientation audio data, thereby making the sound smoother and less abrupt.

[0102] Optionally, in some possible scenarios, after obtaining the original head transfer functions of multiple spatial positions by measurement and forming an original head transfer function library, the original head transfer functions in the original head transfer function library may be interpolated to expand the original head transfer functions in the original head transfer function library. The interpolation of the original head transfer functions in the original head transfer functions refers to obtaining the original head transfer function of the first spatial position point by using the head transfer functions corresponding to other position points (hereinafter referred to as the second spatial position points) around the first spatial position point that have been measured and have measured the head transfer functions, for a spatial position point that has not been measured (hereinafter referred to as the first spatial position point).

[0103] In a feasible implementation, the original head transfer function of the first spatial position point can be obtained by bilinear interpolation. Figure 5 , for the first spatial position point h1 that has not been measured, the head transfer function of the second spatial position points h2, h3, h4, and h5 around the first spatial position point h1 can be obtained, wherein the second spatial position points h2, h3, h4, and h5 are the spatial position points whose head transfer functions have been measured, and the second spatial position points h2, h3, h4, and h5 are the four closest spatial position points related to the spatial position point h1. The HRIRs of the first spatial position point h1 are:

[0104]

[0105] c θ =C θ / θ grid =θmodθ grid / θ grid Formula 11

[0106]

[0107] Among them, h2(k), h3(k), h4(k), and h5(k) are the HRIRs of the second spatial position points h2, h3, h4, and h5, respectively; C θ is the horizontal relative distance between the horizontal line of the first spatial position point h1 and the known point (i.e., the second spatial position point), is the vertical relative distance between the vertical line of the first spatial position point h1 and the known point (i.e., the second spatial position point), is the vertical azimuth step size, θ grid is the step size of the horizontal azimuth angle.

[0108] After determining the horizontal relative distance, vertical relative distance, vertical azimuth step size and horizontal azimuth step size between each second spatial position point and the first spatial position point, the impulse response of the first spatial position point can be calculated based on the head impulse response of each second spatial position point in combination with the above formulas 10 to 12, and then the original head transfer function of the first spatial position point can be obtained through discrete Fourier transform.

[0109] By expanding the original head transfer function in the original head transfer function library, the original head transfer function of the unmeasured spatial point can be completed, thereby ensuring the diversity of the sound source orientation.

[0110] When measuring the HRIRs of various spatial positions, in order to ensure accuracy, a 512-point head transfer function is usually taken for each measurement point (i.e., the spatial position point to be measured). In practical applications, in order to reduce the computational complexity of using discrete Fourier transform instead of linear convolution in the later stage, the transfer function needs to be reduced to 256 points. The head transfer function is equivalent to a finite impulse response (FIR) filter, and fitting a 512-point head transfer function to a 256-point filter is equivalent to a filter reduction operation.

[0111] Optionally, the original header transfer functions in the original header transfer function library may be subjected to order reduction processing to obtain each original header transfer function after order reduction.

[0112] Among them, the function freqz for obtaining the frequency response characteristics of the discrete system and the function infreqz for obtaining the discrete system function according to the frequency response characteristic curve can be used to complete the order reduction processing of each original head transfer function in the original head transfer function library.

[0113] Specifically, the coefficient H of the original head transfer function can be used as the coefficient b of the FIR filter, and the filter coefficient a is set to 1. Discrete Fourier transform is performed to obtain the frequency response characteristic parameters of the discrete system. The frequency response characteristic parameters include h and w. The frequency response curve of the head transfer function is drawn according to h and w. h contains the frequency response of N frequency equal points in the corresponding interval of the discrete system, N is a positive integer, and w is the value of N frequency equal points. Then, according to h and w, the desired target FIR filter order n (for example, n = 256) is set, and the inverse discrete Fourier transform is performed to obtain new FIR filter coefficients b_new, a_new, that is, the FIR filter coefficients b_new and a_new can form an n-order FIR filter. b_new is the n-point long head transfer function, and a_new = 1.

[0114] By reducing the order of the original head transfer functions in the original head transfer function library to obtain the reduced-order original head transfer functions, the order of the original head transfer functions in the original head transfer function library can be reduced to facilitate subsequent fast Fourier transform, reduce calculation complexity and improve calculation efficiency.

[0115] See also Figure 6 , Figure 6 Schematic diagram of the structure of an audio data processing device provided in an embodiment of the present application. Figure 6 As shown, the audio data processing device 30 includes:

[0116] A position acquisition module 301 is used to obtain the sound source position information;

[0117] A position determination module 302, used to determine a target relative position between the virtual sound source and the head according to the sound source position information;

[0118] a function determination module 303, configured to determine a target enhanced head transfer function corresponding to the target relative position when the target relative position is a preset position, wherein the target enhanced head transfer function is obtained by performing spectrum enhancement processing on a first original head transfer function, and the first original head transfer function is used to characterize a phase and frequency response of the virtual sound source transferred from the target relative position to the head;

[0119] The audio data processing module 304 is used to process the audio data to be processed according to the target enhanced head transfer function to obtain target direction audio data.

[0120] In a possible design, the function determination module 303 is specifically used to: calculate the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function, where the second original head transfer function is the original head transfer function of an azimuth point symmetrical to the azimuth point corresponding to the virtual sound source in space; perform spectral enhancement processing on the first original head transfer function according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function to obtain the target enhanced head transfer function.

[0121] In one possible design, the above-mentioned function determination module 303 is specifically used to: calculate the critical band energy of the first primitive function, convolve the critical band energy of the first primitive function with the perceptible energy diffusion function to obtain the auditory excitation energy corresponding to the first primitive head transfer function, and the perceptible energy diffusion function is used to represent the diffusion phenomenon of signal energy on the basilar membrane; calculate the critical band energy of the second primitive function, convolve the critical band energy of the second primitive function with the perceptible energy diffusion function to obtain the auditory excitation energy corresponding to the second primitive head transfer function.

[0122] In a possible design, the function determination module 303 is specifically used to: calculate a target weight coefficient based on the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function; and perform smooth weighted processing on the first original head transfer function according to the target weight coefficient to obtain the target enhanced head transfer function.

[0123] In a possible design, the audio data processing device 30 further includes: a historical data acquisition module 305, used to acquire historical orientation audio data corresponding to the target orientation audio data, wherein the historical orientation audio data includes at least one frame of orientation audio data that is closest in time to the target orientation audio data; and an overlay module 306, used to overlap and add the historical orientation audio data with the target orientation audio data to obtain audio output data.

[0124] In one possible design, the audio data processing module 304 is specifically used to: convolve the target enhancement head transfer function with the audio data to be processed to obtain audio convolution data; and perform an inverse discrete Fourier transform on the audio convolution data to obtain the target orientation audio data.

[0125] In a possible design, the audio data processing device 30 further includes an interpolation processing module 307, which is used to perform interpolation processing on the original head transfer function in the original head transfer function library to expand the original head transfer function in the original head transfer function library, where the original head transfer function in the original head transfer function library is obtained through measurement.

[0126] It should be noted that Figure 6 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0127] The above device obtains the sound source position information, determines the target relative position between the virtual sound source and the head according to the sound source position information, then determines the target enhanced head transfer function corresponding to the target relative position, and finally processes the audio data to be processed according to the target enhanced head transfer function, thereby obtaining the target position audio data. Since the target enhanced head transfer function is obtained by performing spectral enhancement processing on the first original head transfer function that characterizes the phase and frequency response of the virtual sound source transmitted from the target relative position to the head, when the target relative position is a preset position, the target enhanced head transfer function is used to process the audio data to be processed, which can enhance the directionality of the target position audio data, so that the user can more easily identify the direction of the sound, produce a stereo surround sound effect, and bring a better spatial immersive experience to the user.

[0128] See also Figure 7 , Figure 7 4 is a schematic diagram of the structure of an audio device provided in an embodiment of the present application, wherein the audio device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example, via a bus.

[0129] The processor 401 is configured to support the audio device 40 to perform the corresponding functions of the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0130] The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

[0131] The processor 401 may call the program code to perform the following operations:

[0132] Get the location information of the sound source;

[0133] Determine the target relative position between the virtual sound source and the head according to the sound source position information;

[0134] When the target relative position is a preset position, determining a target enhanced head transfer function corresponding to the target relative position, wherein the target enhanced head transfer function is obtained by performing spectrum enhancement processing on a first original head transfer function, and the first original head transfer function is used to characterize a phase and frequency response of the virtual sound source transmitted from the target relative position to the head;

[0135] The audio data to be processed is processed according to the target enhancement head transfer function to obtain target orientation audio data.

[0136] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0137] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0138] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An audio data processing method, characterized in that: include: Get the location information of the sound source; Determine the target relative position between the virtual sound source and the head according to the sound source position information; In the case where the target relative orientation is a preset orientation, determining a target enhanced head transfer function corresponding to the target relative orientation, including: calculating an auditory excitation energy corresponding to a first original head transfer function and an auditory excitation energy corresponding to a second original head transfer function, wherein the first original head transfer function is used to characterize a phase and frequency response of the virtual sound source transmitted from the target relative orientation to the head, and the second original head transfer function is an original head transfer function of an orientation point symmetrical to the orientation point corresponding to the virtual sound source in space; calculating a target weight coefficient according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function; and performing smooth weighted processing on the first original head transfer function according to the target weight coefficient to obtain the target enhanced head transfer function; The audio data to be processed is processed according to the target enhancement head transfer function to obtain target orientation audio data.

2. The method according to claim 1, characterized in that: The calculating the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function comprises: Calculating the critical band energy of the first primitive function, and convolving the critical band energy of the first primitive function with a perceptible energy diffusion function to obtain an auditory excitation energy corresponding to the first primitive head transfer function, wherein the perceptible energy diffusion function is used to represent a diffusion phenomenon of signal energy on the basilar membrane; The critical band energy of the second primitive function is calculated, and the critical band energy of the second primitive function is convolved with the perceptible energy spread function to obtain the auditory excitation energy corresponding to the second primitive head transfer function.

3. The method according to any one of claims 1 to 2, characterized in that: After the audio data to be processed is processed according to the target enhanced head transfer function to obtain the target position audio data, the method further includes: Acquire historical azimuth audio data corresponding to the target azimuth audio data, wherein the historical azimuth audio data includes at least one frame of azimuth audio data that is closest in time to the target azimuth audio data; The historical position audio data and the target position audio data are overlapped and added to obtain audio output data.

4. The method according to any one of claims 1 to 2, characterized in that: The step of processing the audio data to be processed according to the target enhanced head transfer function to obtain target position audio data includes: Convolving the target enhanced head transfer function with the audio data to be processed to obtain audio convolution data; Perform inverse discrete Fourier transform on the audio convolution data to obtain the target orientation audio data.

5. The method according to any one of claims 1-2, characterized in that: The method further comprises: An interpolation process is performed on an original head transfer function in an original head transfer function library to expand the original head transfer function in the original head transfer function library, wherein the original head transfer function in the original head transfer function library is obtained through measurement.

6. An audio data processing device, characterized in that: include: A position acquisition module is used to obtain the sound source position information; A position determination module, used to determine the target relative position between the virtual sound source and the head according to the sound source position information; A function determination module, for determining a target enhanced head transfer function corresponding to the target relative orientation when the target relative orientation is a preset orientation, comprising: calculating an auditory excitation energy corresponding to a first original head transfer function and an auditory excitation energy corresponding to a second original head transfer function, wherein the first original head transfer function is used to characterize a phase and frequency response of the virtual sound source transmitted from the target relative orientation to the head, and the second original head transfer function is an original head transfer function of an orientation point symmetrical to the orientation point corresponding to the virtual sound source in space; calculating a target weight coefficient according to the auditory excitation energy corresponding to the first original head transfer function and the auditory excitation energy corresponding to the second original head transfer function; and performing smooth weighted processing on the first original head transfer function according to the target weight coefficient to obtain the target enhanced head transfer function; The audio data processing module is used to process the audio data to be processed according to the target enhanced head transfer function to obtain target direction audio data.

7. An audio device, characterized in that: The method comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the audio device implements the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images