Spatial audio processing method and device, computer equipment and storage medium

Abstract

The invention discloses a spatial audio processing method and device, computer equipment and a storage medium, and the method comprises the steps: responding to an operation of a user for audio equipment, and initializing a gyroscope sensor in the audio equipment; starting a gyroscope sensor, obtaining head motion original data of a user, and then performing data preprocessing to obtain current head angle information of the user; determining a corresponding head-related transfer function based on the head angle information; performing conversion processing on the original audio data based on the head-related transfer function to obtain target audio data; and performing spatial reverberation simulation on the target audio data to obtain a corresponding spatial reverberation audio. According to the scheme, high-precision and high-real-time audio output can be provided while low resource occupation is kept, and more real and immersive auditory experience is provided for a user.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a spatial audio processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of multimedia technology, users have higher and higher requirements for audio experience, especially in the fields of virtual reality (VR), augmented reality (AR) and 3D games, where users expect to have a more realistic and immersive auditory experience. Spatial audio processing technology is one of the key technologies to achieve this goal. Its core lies in the ability to adjust audio output in real time according to the user's head movement and environmental changes to simulate the auditory effects in the real world.

[0003] Among the existing spatial audio processing technologies, head tracking technology is the key to achieving spatial sound effects. Head tracking technology detects the user's head movement and adjusts the audio output in real time to simulate the direction and distance of the sound. However, there are some technical challenges in the implementation of existing head tracking technology: gyroscope data usually contains DC deviation, resulting in inaccurate direction of audio output; since head-related transfer function data usually only corresponds to fixed angles, it is often necessary to increase the computational complexity for sound data at any angle; when simulating the reverberation effect in a real environment, traditional algorithms often require complex calculations and large memory, which limits their application on resource-constrained devices. It can be seen that the existing technology has problems such as insufficient accuracy, low real-time performance and high resource consumption. These limitations are particularly obvious on mobile devices and wearable devices, affecting the user experience.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0005] Based on this, the present application provides a spatial audio processing method, apparatus, computer device and storage medium, which can provide users with a more realistic and immersive auditory experience.

[0006] In a first aspect, an embodiment of the present application provides a spatial audio processing method, including:

[0007] In response to a user's operation on the audio device, initializing a gyro sensor in the audio device;

[0008] The gyro sensor is activated to obtain the raw data of the user's head movement and then perform data preprocessing to obtain the current head angle information of the user;

[0009] Based on the head angle information, determining a corresponding head-related transfer function;

[0010] Performing conversion processing on the original audio data based on the head-related transfer function to obtain target audio data;

[0011] Perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

[0012] Optionally, in some embodiments of the present application, the starting of the gyro sensor and the obtaining of raw head movement data of the user and performing data preprocessing to obtain current head angle information of the user include:

[0013] Performing DC deviation removal processing on the raw head motion data to obtain target head motion data;

[0014] Based on the target head motion data, the user's current head angle information is determined.

[0015] Optionally, in some embodiments of the present application, determining a corresponding head-related transfer function based on the head angle information includes:

[0016] Determine whether there is a corresponding head-related transfer function for the head angle information;

[0017] If it is determined that a corresponding head-related transfer function exists, determining the head-related transfer function as a calculation function for conversion processing;

[0018] If it is determined that there is no corresponding head-related transfer function, two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information are obtained, and the two adjacent head-related transfer functions are used as a calculation function for conversion processing.

[0019] Optionally, in some embodiments of the present application, if it is determined that there is no corresponding head-related transfer function, then converting the original audio data based on the head-related transfer function to obtain the target audio data includes:

[0020] Based on two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, the original audio data is converted and processed respectively to obtain two groups of audio data to be processed;

[0021] The two groups of audio data to be processed are linearly smoothed to obtain target audio data.

[0022] Optionally, in some embodiments of the present application, after performing linear smoothing processing on the two groups of audio data to be processed to obtain target audio data, the method further includes:

[0023] updating two adjacent angles of the head angle information in real time according to the user's head movement data;

[0024] The parameters of the linear smoothing process are dynamically adjusted based on transition speed requirement information, user feedback information or environment change information.

[0025] Optionally, in some embodiments of the present application, the performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio includes:

[0026] Performing near-end reverberation simulation processing on the target audio data through a preset set of non-recursive digital filters to obtain corresponding near-end reverberation audio;

[0027] Performing far-end reverberation simulation processing on the target audio data through a plurality of preset groups of recursive digital filters to obtain corresponding far-end reverberation audio;

[0028] Based on the near-end reverberation audio and the far-end reverberation audio, a spatial reverberation audio corresponding to the target audio data is obtained by combining.

[0029] Optionally, in some embodiments of the present application, the performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio further includes:

[0030] Determining a combination ratio corresponding to the near-end reverberation audio and the far-end reverberation audio according to a preset spatial perception parameter and a sound clarity parameter;

[0031] Dynamically adjusting the parameters of the non-recursive digital filter and the recursive digital filter according to the acquired user feedback information or environmental change information;

[0032] The output quality of the spatial reverberation audio is monitored in real time, and the parameters of the non-recursive digital filter and the recursive digital filter are adjusted according to the output quality.

[0033] In a second aspect, an embodiment of the present application further provides a spatial audio processing device, including:

[0034] An initialization module, configured to initialize a gyroscope sensor in the audio device in response to a user's operation on the audio device;

[0035] An acquisition module is used to start the gyro sensor, acquire the original data of the user's head movement, and then perform data preprocessing to obtain the user's current head angle information;

[0036] A determination module, configured to determine a corresponding head-related transfer function based on the head angle information;

[0037] A conversion module, used for converting the original audio data based on the head-related transfer function to obtain target audio data;

[0038] The simulation module is used to perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

[0039] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the spatial audio processing method described in the first aspect are implemented.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the spatial audio processing method as described in the first aspect are implemented.

[0041] The embodiment of the present application provides a spatial audio processing method, device, computer equipment and storage medium, and the spatial audio processing method includes: in response to the user's operation on the audio device, initializing the gyroscope sensor in the audio device; starting the gyroscope sensor, obtaining the user's head movement raw data and then performing data preprocessing to obtain the user's current head angle information; based on the head angle information, determining the corresponding head-related transfer function; based on the head-related transfer function, converting the original audio data to obtain the target audio data; simulating the spatial reverberation of the target audio data to obtain the corresponding spatial reverberation audio. In the spatial audio processing scheme provided by the present application, the accuracy and stability of the angle measurement are improved by removing the DC deviation of the original head movement data, and the original audio data is linearly smoothed by the head-related transfer function to enhance the smooth transition and naturalness of the sound field. Finally, by adjusting the combination ratio of near-end reverberation and far-end reverberation, as well as the parameters of FIR and IIR filters, different spatial sense effects are flexibly created, thereby improving the adaptability and flexibility of audio processing. It can be seen that the embodiment of the present application can provide high-precision and high-real-time audio output while maintaining low resource occupancy, providing users with a more realistic and immersive auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] in:

[0044] Figure 1 is a flowchart of a spatial audio processing method provided by an embodiment of the present application;

[0045] Figure 2is a schematic diagram of the structure of a spatial audio processing device provided in an embodiment of the present application;

[0046] Figure 3 It is a structural schematic diagram of the computer device provided by this application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] The present application provides a spatial audio processing method, apparatus, computer equipment and storage medium.

[0049] Among them, the spatial audio processing device provided by the present application can be integrated in a computer device, and the computer device can be directly or indirectly connected to the computer device through wired or wireless communication. The computer device can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker or smart watch, etc., but is not limited to this. In addition, the computer device can also be connected to a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This application does not limit this.

[0050] The present application discloses a spatial audio processing method, comprising: initializing a gyroscope sensor in an audio device in response to an operation of the user on the audio device; starting the gyroscope sensor, obtaining raw head movement data of the user, and then performing data preprocessing to obtain current head angle information of the user; determining a corresponding head-related transfer function based on the head angle information; converting and processing the raw audio data based on the head-related transfer function to obtain target audio data; and performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

[0051] See also Figure 1 , Figure 1 This is a schematic diagram of the flow of the spatial audio processing method provided in this application. The specific flow of the spatial audio processing method can be as follows:

[0052] S1. In response to a user's operation on the audio device, initializing a gyroscope sensor in the audio device;

[0053] Specifically, in step S1, in response to the user's operation on the audio device, the gyroscope sensor in the audio device is initialized. The purpose of this step is to capture the user's head movement and provide basic data for subsequent 3D sound effect processing.

[0054] S2. Start the gyroscope sensor, obtain the raw data of the user's head movement, and then perform data preprocessing to obtain the user's current head angle information;

[0055] Specifically, for step S2, the gyroscope sensor is started to obtain the raw data of the user's head movement, and the data is preprocessed to obtain the user's current head angle information. The data preprocessing may include removing DC bias, such as by a first-order high-pass IIR digital filter, to ensure the accuracy of the angle data. In the preprocessing step, in addition to removing DC bias, other digital filters (such as FIR or IIR) may be added to meet the requirements of a specific frequency response to further improve the accuracy of the angle data.

[0056] S3. Determine the corresponding head-related transfer function based on the head angle information;

[0057] Specifically, for step S3, based on the user's head angle information, the system determines the corresponding HRTF (head-related transfer function). HRTF is a key parameter that describes how the human ear perceives the position of sound in three-dimensional space. In addition, when the user's head angle changes, the two closest HRTFs will be used for linear smoothing to achieve a smooth transition of the sound and avoid auditory jumps.

[0058] S4. Converting the original audio data based on the head-related transfer function to obtain target audio data;

[0059] Specifically, in step S4, the original audio data is converted using the determined HRTF to generate target audio data. This step converts the planar sound into a 3D sound effect with a sense of space.

[0060] S5. Perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio;

[0061] Specifically, for step S5, the target audio data is subjected to spatial reverberation simulation to generate corresponding spatial reverberation audio, including simulation of near-end reverberation and far-end reverberation, and their proportional combination to create different spatial effects. In addition, the ratio of near-end reverberation and far-end reverberation, as well as the parameters of FIR and IIR filters can be dynamically adjusted according to user feedback or environmental changes to adapt to different auditory needs and environmental conditions. In addition, the output quality of the spatial reverberation audio can be monitored in real time, and the filter parameters can be adjusted according to the output quality to ensure that the user obtains the best hearing experience.

[0062] It can be seen that the spatial audio processing method provided in this embodiment, through precise head tracking and the application of HRTF, enables users to experience more accurate and natural spatial audio effects, thereby enhancing the sense of immersion in virtual reality and augmented reality applications; real-time updating of head angle information and dynamic adjustment of reverberation parameters enable audio output to better adapt to the user's real-time experience and provide personalized auditory enjoyment; by simplifying the reverberation algorithm and adopting efficient data processing technology, the computational complexity and memory requirements are reduced, making spatial audio processing more efficient and suitable for resource-constrained devices; by simulating the propagation and reflection of sound in space, a more realistic sound environment is provided, making the audio experience in remote communications, games, and multimedia applications richer and more realistic.

[0063] Optionally, in some embodiments, step S2 of "starting the gyro sensor, obtaining the raw data of the user's head movement, and performing data preprocessing to obtain the user's current head angle information" may specifically include:

[0064] S21. Perform DC deviation removal processing on the raw head motion data to obtain target head motion data;

[0065] S22. Determine the user's current head angle information based on the target head motion data.

[0066] Specifically, for step S2, in response to the user operating the audio device, after starting the gyroscope sensor, the user's head movement data (i.e., gyroscope data) is collected, and the collected raw head movement data is preprocessed to remove errors such as DC deviation to obtain more accurate head movement data. A first-order high-pass IIR digital filter is used to remove the DC deviation in the raw head movement data to ensure the accuracy of the data. Based on the preprocessed target head movement data, the user's current head angle information is determined.

[0067] Among them, this embodiment can adjust the parameters (such as cutoff frequency) of the IIR filter according to the actual application scenario and user needs to adapt to different dynamic range and response speed requirements. After the gyroscope is powered on, a specific startup strategy can be adopted, such as averaging the n-point data as the starting value of the IIR filter to shorten the stabilization time of the gyroscope power-on. In addition to the gyroscope data, other sensor data (such as accelerometers, magnetometers) can also be fused to improve the accuracy and robustness of head angle measurement. According to the user's head movement speed and acceleration, the parameters of the filter are dynamically adjusted to adapt to fast or slow head movements.

[0068] In a specific embodiment, due to the existence of a DC bias in the gyroscope data, if this bias is not removed, the ultimately calculated angle will have cumulative errors and cannot be accurate. Most current processing methods calculate the average value as the DC bias of the data, and then subtract this average value from the original data as the data with the bias removed. Its disadvantage is that if the window for calculating the average value is short, the average value will fluctuate inaccurately; if the window is long, the computational amount and memory occupation will be relatively large, and at the same time, the stabilization time at the start is also relatively long. The method proposed in this embodiment can achieve an accurate DC bias removal effect with a relatively small computational amount and a relatively large startup time. The specific steps are as follows: First, obtain the original gyroscope data, and then adopt a first-order high-pass IIR digital filter to remove the DC bias. Its frequency response function is: H(z-1)=(1–z -1 ) / (1–r z-1 ), where 0<r<1, which determines the cut-off frequency of the filter. Since the impulse response of this IIR filter is relatively long, its influence is that the accuracy of the angle takes some time to stabilize after the gyroscope is powered on. To solve this problem, after the gyroscope data at n points are averaged after power-on, it is used as the starting value of the IIR filter, thus greatly shortening the stabilization time after the gyroscope is powered on. If the application requirements necessitate restricting the data variation within a certain frequency range, then the processed gyroscope data is further added to a digital filter (FIR or IIR) to meet the requirements of the frequency response. Finally, the processed gyroscope data is converted into the head angle.

[0069] In this embodiment, by using a first-order high-pass IIR digital filter to remove the DC bias, the accuracy of the head angle measurement is improved, providing reliable basic data for subsequent 3D audio processing; by optimizing the startup strategy and parameters of the filter, the time from startup to stabilization of the system is reduced, and the response speed of the system is improved; by data fusion and dynamic filter adjustment, the adaptability to different motion states and environmental changes is enhanced, and the robustness of the system is improved; accurate head angle information brings a more natural and realistic 3D audio experience to users, enhancing the immersion in virtual reality and augmented reality applications; through precise data processing and filter optimization, the computational complexity and memory requirements are reduced, making spatial audio processing more efficient and applicable to resource-constrained devices.

[0070] Optionally, in some embodiments, step S3 "Based on the head angle information, determine the corresponding head-related transfer function" may specifically include:

[0071] S31. Determine whether there is a corresponding head-related transfer function for the head angle information;

[0072] S32. If it is determined that there is a corresponding head-related transfer function, then determine the head-related transfer function as the calculation function for the conversion process;

[0073] S33. If it is determined that there is no corresponding head-related transfer function, two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information are obtained, and the two adjacent head-related transfer functions are used as a calculation function for conversion processing.

[0074] Specifically, for step S3, since the head-related transfer function only corresponds to certain fixed angles, but any angle may appear in practice, in order to avoid the feeling of jumping when the sound field changes, the sound data at any angle must be smoothed. This embodiment proposes a simple calculation method that can smoothly transition from one angle to another so that the human auditory system does not feel the jumping phenomenon. The specific steps are as follows: first, obtain the user's current head angle information, and then determine whether the obtained head angle information has a directly corresponding HRTF (head-related transfer function). The head-related transfer function is used to simulate the physical effect of sound propagation in three-dimensional space, which is crucial to realizing 3D sound effects. If so, the corresponding head-related transfer function is directly used for calculation; if not, the two head-related transfer functions with the closest corresponding angles are used for calculation to obtain two sets of sound data, and then these two sets of data are used for linear smoothing.

[0075] In a specific embodiment, a comprehensive head-related transfer function database can be pre-built, containing head-related transfer function data at multiple angles to cover the user's possible head movement range. This database can be personalized according to different user characteristics (such as head shape, ear shape). For angles that do not directly correspond to head-related transfer functions, angle interpolation techniques such as linear interpolation or more advanced interpolation methods (such as cubic spline interpolation) can be used to obtain smoother HRTF transition effects. In some cases, the head-related transfer function can be calculated in real time, especially when the user's head movement is very fast or very slow, to adapt to these extreme situations. In addition, a user-customized head-related transfer function service can be provided, by allowing the user to perform a series of hearing tests to generate personalized head-related transfer function data to provide the best auditory experience.

[0076] By accurately matching or interpolating the head-related transfer function, the system of this embodiment can provide a more realistic and immersive 3D sound experience and enhance the spatial sense of the audio; by using smoothing processing technology, such as linear interpolation, it can ensure that the audio transitions naturally when the head angle changes, avoiding the auditory jump feeling; by calculating HRTF and user-customized HRTF in real time, the system can adapt to the personalized needs of different users and different usage environments; through effective HRTF database management and angle interpolation technology, the system can optimize the utilization of computing resources while maintaining high sound quality; by providing personalized HRTF and optimized audio processing, users can obtain a more comfortable and natural auditory experience, especially in virtual reality and augmented reality applications.

[0077] Optionally, in some embodiments, if it is determined that there is no corresponding head-related transfer function, step S4 "converting the original audio data based on the head-related transfer function to obtain target audio data" includes:

[0078] S41. Based on two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, the original audio data is converted and processed to obtain two sets of audio data to be processed;

[0079] S42. Perform linear smoothing processing on the two groups of audio data to be processed to obtain target audio data.

[0080] Specifically, when it is determined that there is no corresponding head-related transfer function (HRTF) for the user's head angle information, the method for generating the target audio data in step S4 specifically includes the following process: If there is no head-related transfer function directly corresponding to the user's head angle information, obtain the head-related transfer functions corresponding to the two angles adjacent to the angle. Based on the two adjacent head-related transfer functions, the original audio data is converted and processed respectively to obtain two groups of audio data to be processed. The two groups of audio data to be processed are linearly smoothed to obtain the final target audio data. The smoothing processing method is to assume that the angle A is to transition to the angle B within the time of n sampling points, and DA(i) and DB(i) are the sound data calculated by the head-related transfer functions corresponding to the angle A and the angle B, respectively, then the smoothed data is:

[0081] D(i)=(ni) / n*DA(i)+i / n*DB(i), where 0<=i <n

[0082] In the specific implementation process, in addition to linear smoothing, more advanced smoothing techniques such as nonlinear smoothing or smoothing algorithms based on machine learning can also be used to improve the sound quality and naturalness of the transition. It is also possible to update two adjacent angles of the head angle information in real time according to the user's head movement data, and dynamically adjust the parameters of the linear smoothing processing according to the transition speed requirement information, user feedback information or environmental change information. When processing audio data, not only the change in angle can be considered, but also the speed and acceleration of the head movement to achieve more accurate sound positioning. In addition, the selection of the head-related transfer function and the smoothing processing strategy can be adjusted according to different environments (such as indoors, outdoors) and different sound source characteristics (such as human voice, music).

[0083] This embodiment uses linear smoothing processing of adjacent head-related transfer functions to ensure that the conversion processing of audio data can still maintain sound quality and naturalness in the absence of a directly corresponding head-related transfer function; real-time updating and dynamic adjustment of smoothing processing parameters enable the audio output to better adapt to the user's real-time experience and provide a more personalized and comfortable auditory enjoyment; efficient data processing and smoothing technology reduces computational complexity and memory requirements, making spatial audio processing more efficient and suitable for resource-constrained devices; through precise head-related transfer function matching and smoothing processing, the system can provide a more realistic sound environment, making the audio experience in remote communications, games and multimedia applications richer and more realistic.

[0084] Optionally, in some embodiments, after step S42 of “performing linear smoothing processing on the two groups of audio data to be processed to obtain target audio data”, the method further includes:

[0085] S43. Update the two adjacent angles of the head angle information in real time according to the user's head movement data;

[0086] S44. Dynamically adjust the parameters of the linear smoothing process based on the transition speed requirement information, user feedback information or environmental change information.

[0087] Specifically, after linear smoothing is performed on two groups of audio data to be processed to obtain target audio data, this embodiment also provides a method for adjusting the parameters of the linear smoothing process. The specific process is as follows: based on the user's head movement data, two adjacent angles of the head angle information are updated in real time to ensure that the audio processing can keep up with the user's head movement and provide real-time spatial audio effects. The parameters of the linear smoothing process are dynamically adjusted based on transition speed requirement information, user feedback information or environmental change information. This can adapt to different usage scenarios and user preferences to optimize audio quality.

[0088] In practice, the smoothing parameters can be automatically adjusted based on the user's head movement pattern and environmental feedback by integrating adaptive algorithms to achieve a more natural sound transition. Machine learning technology can be used to learn user preferences and adjust audio processing strategies based on these preferences to provide a personalized auditory experience. Environmental perception technologies such as spatial audio analysis can be used to identify the user's environment and adjust HRTF and smoothing parameters accordingly to simulate a more realistic spatial sound effect.

[0089] This embodiment ensures that audio processing can keep up with the user's head movements and provide accurate spatial audio effects by updating head angle information in real time and dynamically adjusting smoothing processing parameters; by learning user preferences and adapting to environmental changes, the system can provide a more personalized auditory experience; it can adapt to different usage scenarios and user needs, improving the adaptability and flexibility of the system; through precise HRTF matching and smoothing processing, the system can provide a more realistic sound environment, making the audio experience in remote communications, games and multimedia applications richer and more realistic. Users can get a customized audio experience based on personal preferences and environmental changes, thereby improving overall satisfaction and comfort.

[0090] Optionally, in some embodiments, step S5 of “performing spatial reverberation simulation on target audio data to obtain corresponding spatial reverberation audio” may specifically include:

[0091] S51. Performing near-end reverberation simulation processing on the target audio data through a preset set of non-recursive digital filters to obtain corresponding near-end reverberation audio;

[0092] S52. Performing remote reverberation simulation processing on the target audio data through a preset plurality of recursive digital filters to obtain corresponding remote reverberation audio;

[0093] S53. Based on the near-end reverberation audio and the far-end reverberation audio, obtain a spatial reverberation audio corresponding to the target audio data by combining them.

[0094] Specifically, for step S5, the prior art requires a relatively complex algorithm to obtain a good reverberation effect, and also requires a large amount of memory. However, this embodiment divides the reverberation into near-end reverberation and far-end reverberation, and simulates them with a relatively simple algorithm. By combining the two in different proportions, different spatial effects can be created, that is, a relatively simple algorithm can be used to combine various reverberation effects. The specific process is as follows:

[0095] First, the target audio data is subjected to near-end reverberation simulation processing through a preset set of non-recursive digital filters (FIR filters) to obtain the corresponding near-end reverberation audio. Near-end reverberation is a one-time reflection of sound with a short delay, which simulates the early reflection of sound. These reflections usually reach the listener's ears soon after the sound is emitted, and have a direct impact on the spatial sense of the sound. Then, the target audio data is subjected to far-end reverberation simulation processing through a preset set of multiple recursive digital filters (IIR filters) to obtain the corresponding far-end reverberation audio. Far-end reverberation is a multiple sound reflection with a long delay, which simulates the late reflection of sound and the diffusion of sound. These reflections reach the listener's ears after a long path and multiple reflections, and have an important impact on the depth and spatial sense of sound. Finally, based on the near-end reverberation audio and the far-end reverberation audio, the spatial reverberation audio corresponding to the target audio data is obtained by combination. This combination can be adjusted according to different spatial sense parameters and sound clarity parameters, while changing the parameters of the above-mentioned FIR and IIR filters to achieve different auditory effects.

[0096] In the specific implementation process, a parametric reverberation model can be designed to adjust the parameters of the filter to simulate spaces of different sizes and shapes, such as small rooms, cathedrals, or outdoor environments. The reverberation parameters can also be dynamically adjusted according to the user's head movement and environmental changes to achieve more natural and real-time spatial sound changes. A feedback loop is established to adjust the reverberation effect based on user feedback to provide a more personalized listening experience. Combined with environmental simulation technology, such as the collection and analysis of environmental sounds, to enhance the realism of reverberation simulation.

[0097] This embodiment can provide a richer and more realistic sense of space and enhance the user's sense of immersion by simulating near-end and far-end reverberation; accurately simulate the propagation and reflection of sound in different environments to make the sound more natural and realistic; provide a more personalized auditory experience by dynamically adjusting reverberation parameters and user feedback loops; and optimize the use of computing resources while maintaining high sound quality by using a combination of FIR and IIR filters.

[0098] Optionally, in some embodiments, step S5 of “performing spatial reverberation simulation on target audio data to obtain corresponding spatial reverberation audio” may further specifically include:

[0099] S54. Determine the combination ratio of the near-end reverberation audio and the far-end reverberation audio according to the preset spatial perception parameters and sound clarity parameters;

[0100] S55. Dynamically adjust the parameters of the non-recursive digital filter and the recursive digital filter according to the obtained user feedback information or environmental change information;

[0101] S56. Monitor the output quality of the spatial reverberation audio in real time, and adjust the parameters of the non-recursive digital filter and the recursive digital filter according to the output quality.

[0102] Specifically, after obtaining the spatial reverberation audio, this embodiment further provides a method for further adjusting the combination ratio or filter parameters, and the specific process is as follows:

[0103] According to the preset spatial perception parameters and sound clarity parameters, the combination ratio of near-end reverberation audio and far-end reverberation audio is determined, so as to adjust the reverberation effect according to different application scenarios and user preferences. According to the user feedback information or environmental change information obtained, the parameters of the non-recursive digital filter (FIR) and the recursive digital filter (IIR) are dynamically adjusted to respond to external changes in real time and provide a more natural auditory experience. In addition, the output quality of the spatial reverberation audio can be monitored in real time, and the parameters of the FIR and IIR filters can be adjusted according to the output quality to ensure that the audio output always maintains high quality.

[0104] In a specific embodiment, an intelligent environmental analysis module can be integrated to automatically detect the user's environment (such as indoors, outdoors, noisy environments, etc.), and adjust the reverberation parameters accordingly to simulate the most realistic environmental sound effects. Machine learning algorithms are used to analyze user behaviors and preferences, and the reverberation settings can be automatically adjusted over time to better meet the user's personalized needs. In addition, visual and tactile feedback can be combined to provide a multimodal interactive experience to enhance the user's sense of immersion and spatial perception. Or develop adaptive sound quality enhancement technology to dynamically adjust the audio signal according to the user's hearing characteristics and ambient noise levels to provide the best listening experience.

[0105] By precisely controlling the reverberation ratio and dynamically adjusting the filter parameters, the system of this embodiment can provide a more realistic and high-quality spatial audio effect; it can make personalized adjustments based on user feedback and behavioral habits to provide a customized auditory experience; it can adapt to different environments and user needs, improving the adaptability and flexibility of the system; efficient reverberation simulation and real-time monitoring technology reduce computational complexity and memory requirements, making spatial audio processing more efficient and suitable for resource-constrained devices; users can obtain a customized audio experience based on personal preferences and environmental changes, thereby improving overall satisfaction and comfort.

[0106] The above completes the spatial audio processing flow of the embodiment of the present application.

[0107] As described above, the embodiment of the present application provides a spatial audio processing method, which initializes a gyroscope sensor in an audio device in response to a user's operation on the audio device; starts the gyroscope sensor, obtains the user's head movement raw data, and then performs data preprocessing to obtain the user's current head angle information; determines the corresponding head-related transfer function based on the head angle information; converts the raw audio data based on the head-related transfer function to obtain target audio data; performs spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio. In the spatial audio processing scheme provided in this embodiment, the accuracy and stability of angle measurement are improved by removing DC deviation from the raw head movement data, and the raw audio data is linearly smoothed by the head-related transfer function to enhance the smooth transition and naturalness of the sound field. Finally, by adjusting the combination ratio of near-end reverberation and far-end reverberation, as well as the parameters of FIR and IIR filters, different spatial effects can be flexibly created, thereby improving the adaptability and flexibility of audio processing.

[0108] In order to better implement the spatial audio processing method of the embodiment of the present application, the embodiment of the present application also provides a spatial audio processing device. The meanings of the terms are the same as those in the above-mentioned spatial audio processing method, and the specific implementation details can refer to the description in the method embodiment.

[0109] See also Figure 2 The present application embodiment provides a spatial audio processing device, which may include an initialization module 201, an acquisition module 202, a determination module 203, a conversion module 204 and a simulation module 205, as follows:

[0110] An initialization module 201, configured to initialize a gyroscope sensor in the audio device in response to a user's operation on the audio device;

[0111] The acquisition module 202 is used to start the gyro sensor, obtain the original data of the user's head movement, and then perform data preprocessing to obtain the user's current head angle information;

[0112] A determination module 203, used to determine a corresponding head-related transfer function based on the head angle information;

[0113] The conversion module 204 is used to convert the original audio data based on the head-related transfer function to obtain the target audio data;

[0114] The simulation module 205 is used to perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

[0115] Optionally, in some embodiments, the acquisition module 202 may specifically include:

[0116] A DC deviation unit is used to perform DC deviation removal processing on the raw head motion data to obtain target head motion data;

[0117] The angle unit is used to determine the user's current head angle information based on the target head motion data.

[0118] Optionally, in some embodiments, the determination module 203 may specifically include:

[0119] A judging unit, used for judging whether there is a corresponding head-related transfer function in the head angle information;

[0120] A first determining unit, configured to determine the head-related transfer function as a calculation function for conversion processing if it is determined that a corresponding head-related transfer function exists;

[0121] The second determining unit is configured to obtain two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information if it is determined that there is no corresponding head-related transfer function, and use the two adjacent head-related transfer functions as a calculation function for conversion processing.

[0122] Optionally, in some embodiments, the conversion module 204 may specifically include:

[0123] A conversion unit, configured to convert the original audio data based on two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, to obtain two groups of audio data to be processed;

[0124] The smoothing unit is used to perform linear smoothing processing on two groups of audio data to be processed to obtain target audio data.

[0125] Optionally, in some embodiments, the conversion module 204 may further include:

[0126] An angle updating unit, used for updating two adjacent angles of the head angle information in real time according to the user's head movement data;

[0127] The parameter adjustment unit is used to dynamically adjust the parameters of the linear smoothing process based on the transition speed requirement information, user feedback information or environment change information.

[0128] Optionally, in some embodiments, the simulation module 205 may specifically include:

[0129] A near-end simulation unit, used for performing near-end reverberation simulation processing on target audio data through a preset set of non-recursive digital filters to obtain corresponding near-end reverberation audio;

[0130] A remote simulation unit, used to perform remote reverberation simulation processing on target audio data through a plurality of preset groups of recursive digital filters to obtain corresponding remote reverberation audio;

[0131] The combining unit is used to combine the near-end reverberation audio and the far-end reverberation audio to obtain the spatial reverberation audio corresponding to the target audio data.

[0132] Optionally, in some embodiments, the simulation module 205 may further include:

[0133] A ratio determination unit, used to determine the combination ratio corresponding to the near-end reverberation audio and the far-end reverberation audio according to preset spatial perception parameters and sound clarity parameters;

[0134] A first filter adjustment unit, used to dynamically adjust parameters of a non-recursive digital filter and a recursive digital filter according to acquired user feedback information or environmental change information;

[0135] The second filter adjustment unit is used to monitor the output quality of the spatial reverberation audio in real time and adjust the parameters of the non-recursive digital filter and the recursive digital filter according to the output quality.

[0136] As described above, the present application provides a spatial audio processing device, wherein the initialization module 201 initializes the gyroscope sensor in the audio device in response to the user's operation on the audio device; the acquisition module 202 starts the gyroscope sensor, obtains the user's head movement raw data and then performs data preprocessing to obtain the user's current head angle information; the determination module 203 determines the corresponding head-related transfer function based on the head angle information; the conversion module 204 converts the raw audio data based on the head-related transfer function to obtain the target audio data; the simulation module 205 performs spatial reverberation simulation on the target audio data to obtain the corresponding spatial reverberation audio. In the spatial audio processing scheme provided by the present application, the accuracy and stability of the angle measurement are improved by removing the DC deviation of the raw head movement data, and the raw audio data is linearly smoothed by the head-related transfer function to enhance the smooth transition and naturalness of the sound field. Finally, by adjusting the combination ratio of the near-end reverberation and the far-end reverberation, as well as the parameters of the FIR and IIR filters, different spatial effects are flexibly created, thereby improving the adaptability and flexibility of audio processing.

[0137] In addition, the present application also provides a computer device, such as Figure 3 , which shows a schematic diagram of the structure of the computer device involved in the present application. Specifically, the computer device may include one or more processing core processors 301, one or more computer-readable storage media memories 302, a power supply 303, an input unit 304 and other components. Those skilled in the art can understand that Figure 3The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0138] The processor 301 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 302 and calling data stored in the memory 302, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0139] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0140] The computer device also includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators. The computer device can also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0141] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby realizing various functions, as follows:

[0142] In response to the user's operation on the audio device, the gyroscope sensor in the audio device is initialized; the gyroscope sensor is started, and the original data of the user's head movement is obtained and then pre-processed to obtain the user's current head angle information; based on the head angle information, the corresponding head-related transfer function is determined; based on the head-related transfer function, the original audio data is converted and processed to obtain target audio data; the target audio data is subjected to spatial reverberation simulation to obtain corresponding spatial reverberation audio.

[0143] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0144] The embodiment of the present application improves the accuracy and stability of angle measurement by removing DC deviation from the raw head movement data, and performs linear smoothing on the raw audio data through the head-related transfer function to enhance the smooth transition and naturalness of the sound field. Finally, by adjusting the combined ratio of near-end reverberation and far-end reverberation, as well as the parameters of FIR and IIR filters, different spatial effects can be flexibly created, thereby improving the adaptability and flexibility of audio processing.

[0145] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0146] To this end, the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the spatial audio processing methods provided in the present application. For example, the instructions can execute the following steps:

[0147] In response to the user's operation on the audio device, the gyroscope sensor in the audio device is initialized; the gyroscope sensor is started, and the original data of the user's head movement is obtained and then pre-processed to obtain the user's current head angle information; based on the head angle information, the corresponding head-related transfer function is determined; based on the head-related transfer function, the original audio data is converted and processed to obtain target audio data; the target audio data is subjected to spatial reverberation simulation to obtain corresponding spatial reverberation audio.

[0148] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0149] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0150] Since the instructions stored in the storage medium can execute the steps in any one of the spatial audio processing methods provided in the present application, the beneficial effects that can be achieved by any one of the spatial audio processing methods provided in the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0151] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0152] The above is a detailed introduction to a spatial audio processing method, device, computer equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A spatial audio processing method, characterized in that: include: In response to a user's operation on the audio device, initializing a gyro sensor in the audio device; The gyro sensor is activated to obtain the raw data of the user's head movement and then perform data preprocessing to obtain the current head angle information of the user; Based on the head angle information, determining a corresponding head-related transfer function; Performing conversion processing on the original audio data based on the head-related transfer function to obtain target audio data; Perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

2. The spatial audio processing method according to claim 1, characterized in that: The starting of the gyro sensor and the acquisition of the raw data of the user's head movement and the data preprocessing to obtain the current head angle information of the user include: Performing DC deviation removal processing on the raw head motion data to obtain target head motion data; Based on the target head motion data, the user's current head angle information is determined.

3. The spatial audio processing method according to claim 1, characterized in that: The step of determining a corresponding head-related transfer function based on the head angle information comprises: Determine whether there is a corresponding head-related transfer function for the head angle information; If it is determined that a corresponding head-related transfer function exists, determining the head-related transfer function as a calculation function for conversion processing; If it is determined that there is no corresponding head-related transfer function, two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information are obtained, and the two adjacent head-related transfer functions are used as a calculation function for conversion processing.

4. The spatial audio processing method according to claim 3, characterized in that: If it is determined that there is no corresponding head-related transfer function, the converting process is performed on the original audio data based on the head-related transfer function to obtain the target audio data, including: Based on two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, the original audio data is converted and processed respectively to obtain two groups of audio data to be processed; The two groups of audio data to be processed are linearly smoothed to obtain target audio data.

5. The spatial audio processing method according to claim 4, characterized in that: After the two groups of audio data to be processed are linearly smoothed to obtain target audio data, the method further includes: updating two adjacent angles of the head angle information in real time according to the user's head movement data; The parameters of the linear smoothing process are dynamically adjusted based on transition speed requirement information, user feedback information or environment change information.

6. The spatial audio processing method according to claim 1, characterized in that: The performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio includes: Performing near-end reverberation simulation processing on the target audio data through a preset set of non-recursive digital filters to obtain corresponding near-end reverberation audio; Performing far-end reverberation simulation processing on the target audio data through a plurality of preset groups of recursive digital filters to obtain corresponding far-end reverberation audio; Based on the near-end reverberation audio and the far-end reverberation audio, a spatial reverberation audio corresponding to the target audio data is obtained by combining.

7. The spatial audio processing method according to claim 6, characterized in that: The performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio further includes: Determining a combination ratio corresponding to the near-end reverberation audio and the far-end reverberation audio according to a preset spatial perception parameter and a sound clarity parameter; Dynamically adjusting the parameters of the non-recursive digital filter and the recursive digital filter according to the acquired user feedback information or environmental change information; The output quality of the spatial reverberation audio is monitored in real time, and the parameters of the non-recursive digital filter and the recursive digital filter are adjusted according to the output quality.

8. A spatial audio processing device, characterized in that: include: An initialization module, configured to initialize a gyroscope sensor in the audio device in response to a user's operation on the audio device; An acquisition module is used to start the gyro sensor, acquire the original data of the user's head movement, and then perform data preprocessing to obtain the user's current head angle information; A determination module, configured to determine a corresponding head-related transfer function based on the head angle information; A conversion module, used for converting the original audio data based on the head-related transfer function to obtain target audio data; The simulation module is used to perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the spatial audio processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the spatial audio processing method according to any one of claims 1 to 7 are implemented.

Images