A spatial audio processing method, apparatus, computer device and storage medium

Abstract

The application discloses a kind of spatial audio processing method, device, computer equipment and storage medium, the method includes: in response to the operation of user to audio device, initialization gyroscope sensor in audio device;Start gyroscope sensor, obtain the head movement original data of user after data pre-processing, obtain the current head angle information of user;Based on head angle information, determine corresponding head-related transfer function;Based on head-related transfer function, original audio data is converted and handled, and target audio data is obtained;Spatial reverberation simulation is carried out to target audio data, and corresponding spatial reverberation audio is obtained.This scheme can maintain low resource occupation while providing high-precision and high real-time audio output, to provide more real and immersive auditory experience for user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a spatial audio processing method and device, computer equipment and storage medium. BACKGROUND

[0002] With the rapid development of multimedia technology, users have higher and higher requirements for audio experience, especially in the fields of virtual reality (VR), augmented reality (AR) and 3D games, users expect to obtain more realistic and immersive auditory experience. Spatial audio processing technology, as one of the key technologies to achieve this goal, its core lies in being able to adjust the audio output in real time according to the user's head movement and environmental changes to simulate the auditory effect in the real world.

[0003] In the existing spatial audio processing technology, head tracking technology is the key to realizing spatial sound effect. Head tracking technology adjusts the audio output in real time by detecting the user's head movement to simulate the direction and distance of sound. However, there are some technical challenges in the implementation process of the existing head tracking technology: the gyroscope data usually contains direct current bias, resulting in inaccurate direction of audio output; since the head-related transfer function data usually only corresponds to a fixed angle, the calculation complexity needs to be increased for sound data of any angle; when simulating the reverberation effect in the real environment, the traditional algorithm often needs complex calculation and large memory, which limits its application on resource-constrained devices. It can be seen that the existing technology has the problems of insufficient accuracy, low real-time performance and high resource occupation. Especially on mobile devices and wearable devices, these limitations are particularly evident, affecting the user experience.

[0004] The foregoing narrative is to provide general background information and does not necessarily constitute the prior art. SUMMARY

[0005] Based on this, the present application provides a spatial audio processing method, device, computer equipment and storage medium, which can provide users with more realistic and immersive auditory experience.

[0006] In a first aspect, the embodiments of the present application provide a spatial audio processing method, comprising:

[0007] In response to the user's operation on the audio device, initializing the gyroscope sensor in the audio device;

[0008] Starting the gyroscope sensor, obtaining the user's head movement original data and performing data preprocessing to obtain the user's current head angle information;

[0009] Based on the head angle information, determining the corresponding head-related transfer function;

[0010] convert the original audio data based on the head-related transfer function to obtain target audio data;

[0011] simulate spatial reverberation for the target audio data to obtain corresponding spatial reverberation audio.

[0012] Optionally, in some embodiments of the present application, after the gyro sensor is started and the original head movement data of the user is obtained, data preprocessing is performed to obtain the current head angle information of the user, including:

[0013] performing direct current bias removal processing on the original head movement data to obtain target head movement data;

[0014] determine the current head angle information of the user based on the target head movement data.

[0015] Optionally, in some embodiments of the present application, determining the corresponding head-related transfer function based on the head angle information includes:

[0016] determine whether the head angle information exists corresponding head-related transfer function;

[0017] if it is determined that the corresponding head-related transfer function exists, determine the head-related transfer function as the calculation function of the conversion processing;

[0018] if it is determined that the corresponding head-related transfer function does not exist, obtain two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, and use the two adjacent head-related transfer functions as the calculation function of the conversion processing.

[0019] Optionally, in some embodiments of the present application, if it is determined that the corresponding head-related transfer function does not exist, the conversion processing of the original audio data based on the head-related transfer function to obtain target audio data includes:

[0020] convert the original audio data based on the two adjacent head-related transfer functions corresponding to the adjacent angles of the head angle information to obtain two groups of to-be-processed audio data;

[0021] perform linear smoothing processing on the two groups of to-be-processed audio data to obtain target audio data.

[0022] Optionally, in some embodiments of the present application, after the linear smoothing processing on the two groups of to-be-processed audio data to obtain target audio data, the method further includes:

[0023] real-time update two adjacent angles of the head angle information according to the head movement data of the user;

[0024] Based on the transition speed demand information, user feedback information or environmental change information, the parameters of the linear smoothing processing are dynamically adjusted.

[0025] Optionally, in some embodiments of the present application, the spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio includes:

[0026] The target audio data is processed by a preset set of non-recursive digital filters to obtain corresponding near-end reverberation audio;

[0027] The target audio data is processed by a preset set of recursive digital filters to obtain corresponding far-end reverberation audio;

[0028] Based on the near-end reverberation audio and the far-end reverberation audio, the spatial reverberation audio corresponding to the target audio data is obtained by combination.

[0029] Optionally, in some embodiments of the present application, the spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio further includes:

[0030] According to the preset spatial sense parameter and the sound clarity parameter, the combination ratio corresponding to the near-end reverberation audio and the far-end reverberation audio is determined;

[0031] According to the obtained user feedback information or environmental change information, the parameters of the non-recursive digital filter and the recursive digital filter are dynamically adjusted;

[0032] The output quality of the spatial reverberation audio is monitored in real time, and the parameters of the non-recursive digital filter and the recursive digital filter are adjusted according to the output quality.

[0033] In a second aspect, the embodiments of the present application also provide a spatial audio processing device, comprising:

[0034] An initialization module is configured to initialize a gyroscope sensor in an audio device in response to user operation on the audio device;

[0035] An acquisition module is configured to start the gyroscope sensor, acquire head movement original data of a user, and perform data preprocessing to obtain current head angle information of the user;

[0036] A determination module is configured to determine a corresponding head-related transfer function based on the head angle information;

[0037] A conversion module is configured to perform conversion processing on original audio data based on the head-related transfer function to obtain target audio data;

[0038] Analogize a spatial reverberation to the target audio data to obtain corresponding spatial reverberation audio.

[0039] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the spatial audio processing method according to the first aspect when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, storing a computer program, and the computer program implementing the steps of the spatial audio processing method according to the first aspect when executed by a processor.

[0041] The embodiment of the present application provides a spatial audio processing method, device, computer device and storage medium. The spatial audio processing method comprises the following steps: initializing a gyroscope sensor in an audio device in response to an operation of a user on the audio device; starting the gyroscope sensor, obtaining head movement original data of the user, and performing data preprocessing to obtain current head angle information of the user; determining a corresponding head-related transfer function based on the head angle information; performing conversion processing on original audio data based on the head-related transfer function to obtain target audio data; and performing spatial reverberation analogization on the target audio data to obtain corresponding spatial reverberation audio. In the spatial audio processing scheme provided by the present application, the accuracy and stability of angle measurement are improved by removing the direct current bias from the head movement original data, and the smooth transition and natural feeling of the sound field are enhanced by performing linear smoothing processing on the original audio data through the head-related transfer function. Finally, by adjusting the combination ratio of near-end reverberation and far-end reverberation, and the parameters of the FIR and IIR filters, different spatial effect effects can be flexibly created, and the adaptability and flexibility of audio processing are improved. It can be seen that the embodiment of the present application can provide high-precision and high-real-time audio output while maintaining low resource occupation, and provide more real and immersive auditory experience for users. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0043] Among them:

[0044] Figure 1 is a flowchart of the spatial audio processing method provided by the embodiment of the present application;

[0045] Figure 2is a structural schematic diagram of a spatial audio processing device provided by an embodiment of the present application.

[0046] Figure 3 is a structural schematic diagram of a computer device provided by the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0048] The present application provides a spatial audio processing method, device, computer equipment and storage medium.

[0049] The spatial audio processing device provided by the present application can be integrated in a computer device, which can be connected to the computer device directly or indirectly through wired or wireless communication. The computer device can be a smartphone, tablet computer, notebook computer, desktop computer, smart speaker or smart watch, but is not limited thereto. In addition, the computer device can also be connected to a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and basic cloud computing services such as big data and artificial intelligence platform, etc. The present application does not limit this.

[0050] The spatial audio processing method provided by the present application includes: initializing a gyroscope sensor in an audio device in response to a user's operation on the audio device; starting the gyroscope sensor, obtaining user head movement raw data, and performing data preprocessing to obtain the user's current head angle information; determining the corresponding head-related transfer function based on the head angle information; converting and processing the original audio data based on the head-related transfer function to obtain target audio data; and performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

[0051] Please refer to Figure 1 , Figure 1 is a flowchart of the spatial audio processing method provided by the present application. The specific process of the spatial audio processing method can be as follows:

[0052] S1. Initializing a gyroscope sensor in an audio device in response to a user's operation on the audio device;

[0053] Specifically, for step S1, in response to the user's operation on the audio device, the gyroscope sensor in the audio device is initialized, and the purpose of this step is to capture the user's head movement to provide basic data for subsequent 3D sound effect processing.

[0054] S2. Start the gyroscope sensor, obtain the user's head movement raw data, and perform data preprocessing to obtain the user's current head angle information;

[0055] Specifically, for step S2, the gyroscope sensor is started, the user's head movement raw data is obtained, and data preprocessing is performed to obtain the user's current head angle information. The data preprocessing can include removing the direct current bias, for example, through a first-order high-pass IIR digital filter to ensure the accuracy of the angle data. In the preprocessing step, in addition to removing the direct current bias, other digital filters (such as FIR or IIR) can be added to meet the requirements of specific frequency response, further improving the accuracy of the angle data.

[0056] S3. Determine the corresponding head-related transfer function based on the head angle information;

[0057] Specifically, for step S3, based on the user's head angle information, the system will determine the corresponding HRTF (Head-Related Transfer Function). HRTF is a key parameter that describes how the human ear perceives the location of sound in three-dimensional space. In addition, when the user's head angle changes, the closest two HRTFs are used for linear smoothing processing to achieve smooth transition of sound and avoid the feeling of auditory jump.

[0058] S4. Convert and process the original audio data based on the head-related transfer function to obtain target audio data;

[0059] Specifically, for step S4, the determined HRTF is used to convert and process the original audio data to generate target audio data. This step converts the plane sound into 3D sound effect with spatial sense.

[0060] S5. Spatial reverberation simulation is performed on the target audio data to obtain corresponding spatial reverberation audio;

[0061] Specifically, for step S5, spatial reverberation simulation is performed on the target audio data to generate corresponding spatial reverberation audio, including simulation of near-end reverberation and far-end reverberation, as well as their proportional combination to create different spatial effect. In addition, the proportion of near-end reverberation and far-end reverberation, as well as the parameters of FIR and IIR filters, can be dynamically adjusted according to user feedback or environmental changes to adapt to different auditory needs and environmental conditions. And it can also monitor the output quality of the spatial reverberation audio in real time, and adjust the filter parameters according to the output quality to ensure that the user obtains the best listening experience.

[0062] It can be seen that the spatial audio processing method provided in the embodiment can enable users to experience more accurate and natural spatial audio effects and enhance the sense of immersion in virtual reality and augmented reality applications through precise head tracking and application of HRTF. Real-time updating of head angle information and dynamic adjustment of reverberation parameters can make audio output better adapt to real-time experience of users and provide personalized auditory enjoyment. Simplification of the reverberation algorithm and adoption of efficient data processing technology can reduce computational complexity and memory requirements, making spatial audio processing more efficient and suitable for devices with limited resources. Simulation of sound propagation and reflection in space can provide a more realistic sound environment, making audio experience in remote communication, games, and multimedia applications more rich and realistic.

[0063] Optionally, in some embodiments, the step S2 of "starting the gyroscope sensor, obtaining the original head motion data of the user, and performing data preprocessing to obtain the current head angle information of the user" can specifically include:

[0064] S21. performing DC bias removal processing on the original head motion data to obtain target head motion data;

[0065] S22. determining the current head angle information of the user based on the target head motion data.

[0066] Specifically, for the step S2, in response to user operation of the audio device, after starting the gyroscope sensor, the user's head motion data (i.e., gyroscope data) is collected, and the collected original head motion data is preprocessed to remove errors such as DC bias, to obtain more accurate head motion data. A first-order high-pass IIR digital filter is used to remove the DC bias in the original head motion data, ensuring the accuracy of the data. Based on the preprocessed target head motion data, the current head angle information of the user is determined.

[0067] In the embodiment, the parameters (such as the cutoff frequency) of the IIR filter can be adjusted according to actual application scenarios and user needs to adapt to different dynamic range and response speed requirements. After the gyroscope is powered on, a specific startup strategy can be used, such as averaging n points of data as the starting value of the IIR filter, to shorten the stabilization time after the gyroscope is powered on. In addition to gyroscope data, other sensor data (such as an accelerometer and a magnetometer) can also be fused to improve the accuracy and robustness of head angle measurement. According to the head motion speed and acceleration of the user, the parameters of the filter are dynamically adjusted to adapt to fast or slow head motion.

[0068] In specific embodiments, due to the existence of direct current bias in the gyroscope data, if the bias is not removed, the final calculated angle will have cumulative errors and cannot be accurate. The current processing method is mostly to calculate the average value as the direct current bias of the data, and then subtract the average value from the original data as the data after removing the bias. The disadvantage is that if the window for calculating the average value is short, the average value will be inaccurate, and if the window is long, the calculation amount and memory occupation will be large, and the starting time will also be long. The method proposed in this embodiment can achieve the effect of accurately removing the direct current bias with small calculation amount and large starting time. The specific steps are as follows: first, obtain the original gyroscope data, then use a first-order high-pass IIR digital filter to remove the direct current bias, and the frequency response function is: H(z-1) = (1-z -1 ) / (1-r z-1 ), where 0 < r < 1, which determines the cutoff frequency of the filter. Because the impulse response of this IIR filter is long, it takes a certain time for the angle accuracy to stabilize after the gyroscope is powered on. To solve this problem, after the gyroscope is powered on, the n-point data of the gyroscope is averaged and used as the starting value of the IIR filter, thereby greatly shortening the stabilization time of the gyroscope after power-on. If the application requirement is necessary to limit the variation of the data within a certain frequency range, the processed gyroscope data is added to a digital filter (FIR or IIR) to meet the frequency response requirements. Finally, the processed gyroscope data is converted into head angle.

[0069] This embodiment uses a first-order high-pass IIR digital filter to remove the direct current bias, improves the accuracy of head angle measurement, and provides reliable basic data for subsequent 3D sound processing; by optimizing the starting strategy and parameters of the filter, the time from starting to stabilizing is reduced, and the response speed of the system is improved; through data fusion and dynamic filter adjustment, the adaptability to different motion states and environmental changes is enhanced, and the robustness of the system is improved; accurate head angle information provides users with a more natural and realistic 3D sound experience, and enhances the sense of immersion in virtual reality and augmented reality applications; through accurate data processing and filter optimization, the calculation complexity and memory requirement are reduced, making the spatial audio processing more efficient and suitable for resource-constrained devices.

[0070] Optionally, in some embodiments, step S3 "determining the corresponding head-related transfer function based on the head angle information" can specifically include:

[0071] S31. Determine whether the head angle information has a corresponding head-related transfer function;

[0072] S32. If it is determined that there is a corresponding head-related transfer function, determine the head-related transfer function as the calculation function of the conversion processing;

[0073] S33. If it is determined that there is no corresponding head-related transfer function, obtaining two adjacent head-related transfer functions corresponding to the adjacent angles of the head angle information, and using the two adjacent head-related transfer functions as the calculation function for the conversion processing.

[0074] Specifically, for step S3, since the head-related transfer function only corresponds to certain fixed angles, but in practice any angle can occur, in order to avoid the feeling of jumping in the sound field change, it is necessary to smooth the sound data at any angle. This embodiment proposes a simple calculation method, which can smoothly transition from one angle to another, so that the human auditory system does not feel the jumping phenomenon. The specific steps are as follows: first, obtain the current head angle information of the user, then determine whether the obtained head angle information has a direct corresponding HRTF (head-related transfer function). The head-related transfer function is used to simulate the physical effect of sound propagation in three-dimensional space, and is crucial for implementing 3D sound effects. If there is, the corresponding head-related transfer function is directly used for calculation; if not, the two head-related transfer functions corresponding to the closest angles are used for calculation, and two groups of sound data are obtained, then linear smoothing processing is performed on the two groups of data.

[0075] In a specific embodiment, a comprehensive head-related transfer function database can be constructed in advance, containing head-related transfer function data of multiple angles to cover the possible head movement range of the user. This database can be personalized according to different user characteristics (such as head shape, ear shape). For angles without a direct corresponding head-related transfer function, angle interpolation techniques such as linear interpolation or more advanced interpolation methods (such as cubic spline interpolation) can be used to obtain smoother HRTF transition effects. In some cases, head-related transfer functions can be calculated in real time, especially when the user's head movement is very fast or very slow, to adapt to these extreme situations. In addition, user customized head-related transfer function services can also be provided, by having the user perform a series of hearing tests to generate personalized head-related transfer function data to provide the best auditory experience.

[0076] This embodiment can provide a more realistic and immersive 3D sound experience by accurately matching or interpolating head-related transfer functions, enhancing the spatial sense of audio; using smoothing techniques such as linear interpolation can ensure smooth audio transitions when the head angle changes, avoiding the feeling of jumping in the auditory system; through real-time calculation of HRTF and user customization of HRTF, the system can adapt to the individual needs of different users and different use environments; through effective HRTF database management and angle interpolation techniques, the system can optimize the use of computing resources while maintaining high sound quality; by providing personalized HRTF and optimized audio processing, users can have a more comfortable and natural auditory experience, especially in virtual reality and augmented reality applications.

[0077] Optionally, in some embodiments, if it is determined that there is no corresponding head-related transfer function, then step S4 "transformation processing of the original audio data based on the head-related transfer function to obtain target audio data, includes":

[0078] S41. Based on the head angle information, two adjacent head-related transfer functions corresponding to adjacent angles are used to respectively transform process the original audio data to obtain two groups of to-be-processed audio data;

[0079] S42. Linear smoothing processing is performed on the two groups of to-be-processed audio data to obtain target audio data.

[0080] Specifically, when it is determined that there is no corresponding head-related transfer function (HRTF) for the user's head angle information, the way to generate target audio data in step S4 specifically includes the following process: if the user's head angle information does not have a directly corresponding head-related transfer function, the head-related transfer functions corresponding to two angles adjacent to the angle are obtained. Based on the two adjacent head-related transfer functions, the original audio data is respectively transformed to obtain two groups of to-be-processed audio data. Linear smoothing processing is performed on the two groups of to-be-processed audio data to obtain the final target audio data. The smoothing processing method is as follows: assuming that the angle A is to be transitioned to the angle B within n sampling points of time, and DA(i) and DB(i) are the sound data calculated by the head-related transfer functions corresponding to the angle A and the angle B, respectively, then the smoothed data is:

[0081] D(i) = (n-i) / n*DA(i) + i / n*DB(i), where 0<=i<n

[0082] In the specific implementation process, in addition to linear smoothing processing, more advanced smoothing techniques such as nonlinear smoothing or machine learning-based smoothing algorithms can also be used to improve sound quality and naturalness of transition. The two adjacent angles of the head angle information can also be updated in real time according to the user's head movement data, and the parameters of the linear smoothing processing can be dynamically adjusted according to the transition speed requirement information, user feedback information or environmental change information. When processing audio data, not only the change of angle, but also the speed and acceleration of head movement can be considered to achieve more accurate sound positioning. In addition, the selection of head-related transfer function and the smoothing processing strategy can also be adjusted according to different environments (such as indoor and outdoor) and different sound source characteristics (such as human voice and music).

[0083] By using linear smoothing processing of adjacent head-related transfer functions, the embodiment can ensure that the conversion processing of audio data can still maintain sound quality and naturalness without directly corresponding head-related transfer functions; real-time updating and dynamic adjustment of the parameters of smoothing processing make the audio output better adapt to the real-time experience of users, providing more personalized and comfortable auditory enjoyment; efficient data processing and smoothing technology reduces the computational complexity and memory requirements, making spatial audio processing more efficient and suitable for resource-constrained devices; through accurate head-related transfer function matching and smoothing processing, the system can provide a more realistic sound environment, making the audio experience in remote communication, games and multimedia applications more rich and realistic.

[0084] Optionally, in some embodiments, after step S42 "linear smoothing processing is performed on the two groups of audio data to be processed to obtain target audio data", it further includes:

[0085] S43. Real-time update two adjacent angles of head angle information according to user's head motion data;

[0086] S44. Dynamically adjust the parameters of linear smoothing processing based on transition speed requirement information, user feedback information or environmental change information.

[0087] Specifically, after linear smoothing processing is performed on the two groups of audio data to be processed to obtain target audio data, the embodiment further provides a method for adjusting the parameters of linear smoothing processing, and the specific process is as follows: according to the user's head motion data, real-time update two adjacent angles of head angle information, so as to ensure that the audio processing can follow the user's head action and provide real-time spatial audio effect. Dynamically adjust the parameters of linear smoothing processing based on transition speed requirement information, user feedback information or environmental change information. Thus, different use scenarios and user preferences can be adapted to optimize audio quality.

[0088] In actual process, the parameters of smoothing processing can be automatically adjusted according to the user's head motion mode and environmental feedback by integrating adaptive algorithm, so as to realize more natural sound transition. Through machine learning technology, user's preferences are learned, and audio processing strategy is adjusted according to these preferences to provide personalized auditory experience. And by using environmental perception technology, such as spatial audio analysis, the environment where the user is located is identified, and HRTF and smoothing parameters are adjusted accordingly to simulate more realistic spatial sound effect.

[0089] The embodiment ensures that the audio processing can follow the head movement of the user by updating the head angle information in real time and dynamically adjusting the smoothing processing parameters, and provides accurate spatial audio effects; the system can provide more personalized auditory experiences by learning the preferences of the user and adapting to environmental changes; the system can adapt to different use scenarios and user needs, improving the adaptability and flexibility of the system; the system can provide a more realistic sound environment through accurate HRTF matching and smoothing processing, making the audio experience in remote communication, games and multimedia applications more rich and realistic. The user can obtain customized audio experience according to personal preferences and environmental changes, thereby improving the overall satisfaction and comfort.

[0090] Optionally, in some embodiments, the step S5 "spatially reverberate the target audio data to obtain corresponding spatially reverberated audio" can specifically include:

[0091] S51. Perform near-end reverberation simulation processing on the target audio data through a preset set of non-recursive digital filters to obtain corresponding near-end reverberated audio;

[0092] S52. Perform far-end reverberation simulation processing on the target audio data through a preset plurality of sets of recursive digital filters to obtain corresponding far-end reverberated audio;

[0093] S53. Combine the near-end reverberated audio and the far-end reverberated audio to obtain spatially reverberated audio corresponding to the target audio data.

[0094] Specifically, for the step S5, the prior art needs a relatively complex algorithm to obtain a good reverberation effect, and also needs a large memory. The embodiment divides the reverberation into near-end reverberation and far-end reverberation, and simulates them through a relatively simple algorithm. Different spatial effects can be created by different combinations of the two, that is, various reverberation effects can be combined by a relatively simple algorithm. The specific process is as follows:

[0095] First, a set of non-recursive digital filters (FIR filters) is used to simulate the near-end reverberation of the target audio data, obtaining the corresponding near-end reverberation audio. Near-end reverberation is a one-time reflection of sound with short delay, simulating the early reflections of sound, which usually reach the listener's ear soon after the sound is emitted, having a direct impact on the spatial sense of the sound. Then, a set of recursive digital filters (IIR filters) is used to simulate the far-end reverberation of the target audio data, obtaining the corresponding far-end reverberation audio. Far-end reverberation is a multiple reflection of sound with long delay, simulating the late reflections and diffusion of sound, which reach the listener's ear after a long path and multiple reflections, having an important impact on the depth and spatial sense of the sound. Finally, based on the near-end reverberation audio and the far-end reverberation audio, the spatial reverberation audio corresponding to the target audio data is obtained by combining. This combination can be adjusted according to different spatial sense parameters and sound clarity parameters, while changing the parameters of the above FIR and IIR filters to achieve different auditory effects.

[0096] In the specific implementation process, a parameterized reverberation model can be designed to simulate different sizes and shapes of spaces, such as small rooms, large cathedrals, or outdoor environments, by adjusting the parameters of the filters. The reverberation parameters can also be dynamically adjusted according to the user's head movement and environmental changes to achieve more natural and real-time spatial sound changes. A feedback loop is established to adjust the reverberation effect according to the user's feedback to provide a more personalized auditory experience. Combined with environmental simulation techniques such as environmental sound collection and analysis, the realism of the reverberation simulation is enhanced.

[0097] This embodiment can provide a more rich and realistic spatial sense by simulating near-end and far-end reverberation, enhancing the user's sense of immersion; accurately simulating the propagation and reflection of sound in different environments, making the sound more natural and realistic; providing a more personalized auditory experience through dynamic adjustment of reverberation parameters and user feedback loops; and optimizing the use of computing resources while maintaining high sound quality by using a combination of FIR and IIR filters.

[0098] Optionally, in some embodiments, the step S5 "spatial reverberation simulation of the target audio data to obtain the corresponding spatial reverberation audio" can further include:

[0099] S54. According to the preset spatial sense parameters and sound clarity parameters, determine the combination ratio of the near-end reverberation audio and the far-end reverberation audio;

[0100] S55. According to the obtained user feedback information or environmental change information, dynamically adjust the parameters of the non-recursive digital filter and the recursive digital filter;

[0101] S56. Monitor the output quality of the spatial reverberation audio in real time, and adjust the parameters of the non-recursive digital filter and the recursive digital filter according to the output quality.

[0102] Specifically, after obtaining the spatial reverberation audio, the embodiment further provides a method for further adjusting the combination ratio or filter parameters, and the specific process is as follows:

[0103] According to the preset spatial sense parameter and sound clarity parameter, the combination ratio of the near-end reverberation audio and the far-end reverberation audio is determined, so as to adjust the reverberation effect according to different application scenarios and user preferences. According to the obtained user feedback information or environmental change information, the parameters of the non-recursive digital filter (FIR) and the recursive digital filter (IIR) are dynamically adjusted, which can respond to external changes in real time and provide a more natural auditory experience. In addition, the output quality of the spatial reverberation audio can be monitored in real time, and the parameters of the FIR and IIR filters are adjusted according to the output quality, so as to ensure that the audio output always maintains high quality.

[0104] In specific embodiments, an intelligent environment analysis module can also be integrated to automatically detect the environment in which the user is located (such as indoors, outdoors, noisy environment, etc.), and adjust the reverberation parameters accordingly to simulate the most realistic environmental sound effects. By using machine learning algorithms to analyze the user's behavior and preferences, the reverberation settings can be automatically adjusted over time to better meet the user's individual needs. In addition, visual and tactile feedback can be combined to provide a multi-modal interactive experience, enhancing the user's sense of immersion and spatial perception. Alternatively, adaptive sound quality enhancement technology can be developed to dynamically adjust audio signals according to the user's hearing characteristics and environmental noise levels to provide the best auditory experience.

[0105] The embodiment can provide more realistic and high-quality spatial audio effects by precisely controlling the reverberation ratio and dynamically adjusting the filter parameters; can provide customized auditory experiences by adjusting individual preferences and behavior habits; can adapt to different environments and user needs, improving the adaptability and flexibility of the system; efficient reverberation simulation and real-time monitoring technology reduces computational complexity and memory requirements, making spatial audio processing more efficient and suitable for resource-constrained devices; users can obtain customized audio experiences according to personal preferences and environmental changes, thereby improving overall satisfaction and comfort.

[0106] The above completes the spatial audio processing flow of the embodiment of the present application.

[0107] As described above, the embodiment of the present application provides a spatial audio processing method, in response to the operation of the user for the audio device, initializing the gyroscope sensor in the audio device; starting the gyroscope sensor, obtaining the head movement original data of the user and then performing data preprocessing to obtain the current head angle information of the user; determining the corresponding head-related transfer function based on the head angle information; converting and processing the original audio data based on the head-related transfer function to obtain target audio data; and performing spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio. In the spatial audio processing scheme provided in the embodiment, by removing the direct current bias of the head movement original data, the accuracy and stability of the angle measurement are improved, and by linearly smoothing the original audio data based on the head-related transfer function, the smooth transition and natural feeling of the sound field are enhanced, and finally by adjusting the combination ratio of the near-end reverberation and the far-end reverberation and the parameters of the FIR and IIR filters, different spatial effect effects are flexibly created, and the adaptability and flexibility of the audio processing are improved.

[0108] In order to better implement the spatial audio processing method of the embodiment of the present application, the embodiment of the present application further provides a spatial audio processing device. The meanings of the terms are the same as those in the above-mentioned spatial audio processing method, and the specific implementation details can be referred to the description in the method embodiment.

[0109] Please refer to Figure 2 The embodiment of the present application provides a spatial audio processing device, which can specifically include an initialization module 201, an acquisition module 202, a determination module 203, a conversion module 204 and a simulation module 205, and the details are as follows:

[0110] The initialization module 201 is configured to initialize the gyroscope sensor in the audio device in response to the operation of the user for the audio device.

[0111] The acquisition module 202 is configured to start the gyroscope sensor, acquire the head movement original data of the user, and then perform data preprocessing to obtain the current head angle information of the user.

[0112] The determination module 203 is configured to determine the corresponding head-related transfer function based on the head angle information.

[0113] The conversion module 204 is configured to convert and process the original audio data based on the head-related transfer function to obtain target audio data.

[0114] The simulation module 205 is configured to perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

[0115] Optionally, in some embodiments, the acquisition module 202 can specifically include:

[0116] A direct current bias unit is configured to perform direct current bias removal processing on the head movement original data to obtain target head movement data.

[0117] An angle unit is configured to determine current head angle information of the user based on the target head movement data.

[0118] Optionally, in some embodiments, the determining module 203 can specifically include:

[0119] A judging unit is configured to judge whether the head angle information exists corresponding head-related transfer function.

[0120] A first determining unit is configured to, if it is judged that the corresponding head-related transfer function exists, determine the head-related transfer function as the calculation function of the conversion processing.

[0121] A second determining unit is configured to, if it is judged that the corresponding head-related transfer function does not exist, obtain two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, and take the two adjacent head-related transfer functions as the calculation function of the conversion processing.

[0122] Optionally, in some embodiments, the conversion module 204 can specifically include:

[0123] A converting unit is configured to perform conversion processing on the original audio data based on the two adjacent head-related transfer functions corresponding to the adjacent angles of the head angle information, to obtain two groups of to-be-processed audio data.

[0124] A smoothing unit is configured to perform linear smoothing processing on the two groups of to-be-processed audio data to obtain target audio data.

[0125] Optionally, in some embodiments, the conversion module 204 can specifically further include:

[0126] An angle updating unit is configured to update the two adjacent angles of the head angle information in real time according to the head movement data of the user.

[0127] A parameter adjusting unit is configured to dynamically adjust parameters of the linear smoothing processing based on transition speed requirement information, user feedback information, or environment change information.

[0128] Optionally, in some embodiments, the simulation module 205 can specifically include:

[0129] A near-end simulation unit is configured to perform near-end reverberation simulation processing on the target audio data through a preset group of non-recursive digital filters to obtain corresponding near-end reverberation audio.

[0130] A far-end simulation unit is configured to perform far-end reverberation simulation processing on the target audio data through a preset plurality of groups of recursive digital filters to obtain corresponding far-end reverberation audio.

[0131] a combination unit, configured to combine the near-reverberation audio and the far-reverberation audio to obtain the spatial-reverberation audio corresponding to the target audio data.

[0132] Optionally, in some embodiments, the simulation module 205 can further include:

[0133] a proportion determination unit, configured to determine a combination proportion of the near-reverberation audio and the far-reverberation audio according to the preset spatial sense parameter and the sound clarity parameter;

[0134] a filter first adjustment unit, configured to dynamically adjust parameters of the non-recursive digital filter and the recursive digital filter according to the obtained user feedback information or environmental change information;

[0135] a filter second adjustment unit, configured to monitor an output quality of the spatial-reverberation audio in real time, and adjust the parameters of the non-recursive digital filter and the recursive digital filter according to the output quality.

[0136] As described above, the present application provides a spatial audio processing device, the initialization module 201 responds to the operation of the user for the audio device, initializes the gyroscope sensor in the audio device; the acquisition module 202 starts the gyroscope sensor, acquires the head movement original data of the user, and then performs data preprocessing to obtain the current head angle information of the user; the determination module 203 determines the corresponding head-related transfer function based on the head angle information; the conversion module 204 converts and processes the original audio data based on the head-related transfer function to obtain the target audio data; the simulation module 205 simulates the spatial reverberation of the target audio data to obtain the corresponding spatial reverberation audio. In the spatial audio processing scheme provided by the present application, the direct current bias removal processing is performed on the head movement original data, the accuracy and stability of the angle measurement are improved, and the linear smoothing processing is performed on the original audio data through the head-related transfer function, the smooth transition and natural feeling of the sound field are enhanced, and finally the combination proportion of the near-reverberation and the far-reverberation is adjusted, and the parameters of the FIR and IIR filters, so as to flexibly create different spatial sense effects, improve the adaptability and flexibility of the audio processing.

[0137] In addition, the present application also provides a computer device, as shown in Figure 3 The computer device can include a processor 301 with one or more processing cores, a memory 302 with one or more computer readable storage media, a power supply 303, and an input unit 304, etc. Those skilled in the art can understand that Figure 3The computer device structure shown in the figure is not a limitation of the computer device, and can include more or fewer components than shown, or combine certain components, or arrange different components. Among them:

[0138] The processor 301 is the control center of the computer device, connects various parts of the computer device through various interfaces and lines, executes various functions and processes data of the computer device by running or executing software programs and / or modules stored in the memory 302 and calling data stored in the memory 302, and thus monitors the computer device as a whole. Optionally, the processor 301 can include one or more processing cores; preferably, the processor 301 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 301.

[0139] The memory 302 can be used to store software programs and modules, and the processor 301 executes various functions and data processing by running the software programs and modules stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 302 can also include a memory controller to provide access for the processor 301 to the memory 302.

[0140] The computer device also includes a power supply 303 for powering various components, and preferably the power supply 303 can be logically connected to the processor 301 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 303 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power supply converter or inverter, a power supply state indicator, etc. Any component. The computer device can also include an input unit 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0141] Although not shown, the computer device can further include a display unit and the like, which will not be described herein. Specifically in the present embodiment, the processor 301 in the computer device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and run the application program stored in the memory 302 by the processor 301, thereby realizing various functions, as follows:

[0142] In response to the operation of the user on the audio device, the gyroscope sensor in the audio device is initialized; the gyroscope sensor is started to obtain the head movement raw data of the user, and the head movement raw data is preprocessed to obtain the current head angle information of the user; based on the head angle information, the corresponding head-related transfer function is determined; the original audio data is converted and processed based on the head-related transfer function to obtain target audio data; and the target audio data is subjected to spatial reverberation simulation to obtain corresponding spatial reverberation audio.

[0143] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described herein.

[0144] The embodiments of the present application improve the accuracy and stability of angle measurement by removing the direct current bias of the head movement raw data, and enhance the smooth transition and natural feeling of the sound field by linear smoothing the original audio data through the head-related transfer function. Finally, by adjusting the combination ratio of the near-end reverberation and the far-end reverberation, and the parameters of the FIR and IIR filters, different spatial effect effects can be flexibly created, and the adaptability and flexibility of the audio processing are improved.

[0145] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0146] To this end, the present application provides a storage medium having a plurality of instructions stored therein, which can be loaded by a processor to execute the steps in any of the spatial audio processing methods provided by the present application. For example, the instructions can execute the following steps:

[0147] In response to the operation of the user on the audio device, the gyroscope sensor in the audio device is initialized; the gyroscope sensor is started to obtain the head movement raw data of the user, and the head movement raw data is preprocessed to obtain the current head angle information of the user; based on the head angle information, the corresponding head-related transfer function is determined; the original audio data is converted and processed based on the head-related transfer function to obtain target audio data; and the target audio data is subjected to spatial reverberation simulation to obtain corresponding spatial reverberation audio.

[0148] The specific implementation of the above operations can refer to the foregoing embodiments, which will not be described here again.

[0149] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0150] Due to the instructions stored in the storage medium, the steps in any of the spatial audio processing methods provided in the present application can be executed, and thus the beneficial effects of any of the spatial audio processing methods provided in the present application can be achieved. Details are described in the foregoing embodiments, which will not be described here again.

[0151] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in any of the optional implementation manners.

[0152] The spatial audio processing method, device, computer device and storage medium provided in the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, the specific implementation manners and application range can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A spatial audio processing method, characterized by, The method comprises: in response to a user's operation on an audio device, initializing a gyroscope sensor in the audio device; starting the gyroscope sensor, obtaining user's head movement raw data, and performing data preprocessing to obtain the user's current head angle information; based on the head angle information, determining the corresponding head-related transfer function, including: judging whether the head angle information exists corresponding head-related transfer function; if it is judged that there is a corresponding head-related transfer function, the head-related transfer function is determined as the calculation function of the conversion processing; if it is judged that there is no corresponding head-related transfer function, two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information are obtained, and the two adjacent head-related transfer functions are used as the calculation function of the conversion processing; based on the head-related transfer function, the original audio data is converted and processed to obtain target audio data; spatial reverberation simulation is performed on the target audio data to obtain corresponding spatial reverberation audio.

2. A spatial audio processing method according to claim 1, wherein, The starting of the gyroscope sensor, the obtaining of the user's head movement raw data, and the data preprocessing to obtain the user's current head angle information, comprise: direct current bias removal processing is performed on the head movement raw data to obtain target head movement data; based on the target head movement data, the user's current head angle information is determined.

3. A spatial audio processing method according to claim 2, wherein, If it is judged that there is no corresponding head-related transfer function, the conversion processing of the original audio data based on the head-related transfer function to obtain the target audio data comprises: based on the two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, the original audio data is converted and processed to obtain two groups of audio data to be processed; linear smoothing processing is performed on the two groups of audio data to be processed to obtain target audio data.

4. A spatial audio processing method according to claim 3, wherein, After the linear smoothing processing of the two groups of audio data to be processed to obtain the target audio data, it further comprises: real-time updating two adjacent angles of the head angle information according to the user's head movement data; based on the transition speed requirement information, the user feedback information or the environmental change information, the parameters of the linear smoothing processing are dynamically adjusted.

5. A spatial audio processing method according to claim 1, wherein, The spatial reverberation simulation on the target audio data to obtain the corresponding spatial reverberation audio comprises: near-end reverberation simulation processing is performed on the target audio data through a preset group of non-recursive digital filters to obtain corresponding near-end reverberation audio; far-end reverberation simulation processing is performed on the target audio data through a plurality of groups of recursive digital filters to obtain corresponding far-end reverberation audio; based on the near-end reverberation audio and the far-end reverberation audio, the spatial reverberation audio corresponding to the target audio data is obtained by combination.

6. A spatial audio processing method according to claim 5, wherein, The spatial reverberation simulation on the target audio data to obtain the corresponding spatial reverberation audio further comprises: determining the combination ratio of the near-end reverberation audio and the far-end reverberation audio according to the preset spatial sense parameter and the sound clarity parameter; dynamically adjusting the parameters of the non-recursive digital filter and the recursive digital filter according to the obtained user feedback information or environmental change information; Real-time monitoring of the output quality of the spatial reverberation audio, and adjusting parameters of the non-recursive digital filter and the recursive digital filter according to the output quality.

7. A spatial audio processing apparatus, characterized by Comprise: An initialization module configured to initialize a gyroscope sensor in the audio device in response to a user operation on the audio device; An acquisition module configured to start the gyroscope sensor, acquire raw data of head movement of the user, and perform data preprocessing to obtain current head angle information of the user; A determination module configured to determine a corresponding head-related transfer function based on the head angle information, including: judging whether the head angle information exists a corresponding head-related transfer function; if it is judged that the corresponding head-related transfer function exists, determining the head-related transfer function as a calculation function of conversion processing; if it is judged that the corresponding head-related transfer function does not exist, acquiring two adjacent head-related transfer functions corresponding to adjacent angles of the head angle information, and taking the two adjacent head-related transfer functions as the calculation function of conversion processing; A conversion module configured to perform conversion processing on original audio data based on the head-related transfer function to obtain target audio data; An analog module configured to perform spatial reverberation simulation on the target audio data to obtain corresponding spatial reverberation audio.

8. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that, The processor executes the computer program to realize the steps of the spatial audio processing method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the spatial audio processing method in any one of claims 1 to 6.

Images