Spatial audio processing methods, devices, storage media, and computer program products

Abstract

This application discloses a spatial audio processing method, apparatus, storage medium, and computer program product, relating to the field of audio processing technology. The method is applied to a spatial audio device, which includes a main processor, an audio digital signal processor (ADSP), and a sensor hub. The method includes: transmitting audio stream data to be processed to the ADSP via the main processor; transmitting head motion sensor data to the ADSP via the sensor hub using a preset inter-processor communication method; converting the head motion sensor data into head posture data via the ADSP; and performing spatial audio rendering processing on the audio stream data to be processed based on the head posture data to obtain and output spatial audio stream data. This application reduces the transmission latency of head motion sensor data in spatial audio devices, improves the real-time performance of spatial audio rendering processing, and enhances the actual effect of spatial audio.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a spatial audio processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of virtual reality (VR), augmented reality (AR), and 360° audio technologies, spatial audio is becoming increasingly important in these technologies. Spatial audio technology can dynamically adjust the spatial positioning of audio signals based on the user's head movement and orientation, enhancing the immersive experience of the audio experience. However, existing VR / AR glasses, helmets, and other spatial audio devices often experience a certain delay when acquiring head motion sensor data for spatial audio rendering processing. This is especially true for spatial audio devices based on the Android operating system, where the transmission of head motion sensor data to the audio digital signal processor introduces a significant delay, affecting the real-time performance of spatial audio rendering processing. This results in spatial audio head motion delay and reduces the actual effect of the spatial audio.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a spatial audio processing method, a spatial audio processing model training method, an electronic device, a storage medium, and a computer program product, which aim to reduce the transmission delay of head motion sensor data in spatial audio devices, improve the real-time performance of spatial audio rendering and processing, and enhance the actual effect of spatial audio.

[0005] To achieve the above objectives, this application proposes a spatial audio processing method applied to a spatial audio device, the spatial audio device including a main processor, an audio digital signal processor, and a sensor hub, the spatial audio processing method comprising:

[0006] The main processor transmits the audio stream data to be processed to the audio digital signal processor.

[0007] The head motion sensor data is transmitted to the audio digital signal processor via the sensor hub using a preset inter-processor communication method.

[0008] The audio digital signal processor converts the head motion sensor data into head posture data, and performs spatial audio rendering processing on the audio stream data to be processed based on the head posture data to obtain spatial audio stream data and output it.

[0009] Optionally, the step of converting the head motion sensor data into head posture data using the audio digital signal processor, and performing spatial audio rendering processing on the audio stream data to be processed based on the head posture data to obtain and output spatial audio stream data includes:

[0010] The head motion sensor data is received by the head posture control module running in the audio digital signal processor, and the head motion sensor data is converted into head posture data.

[0011] The spatial audio stream data is obtained by performing spatial audio rendering processing on the audio stream data to be processed based on the head posture data by the spatial audio algorithm module running in the audio digital signal processor.

[0012] The spatial audio stream data is output by the audio back-end DSP driver module running in the audio digital signal processor.

[0013] Optionally, the step of performing spatial audio stream data by a spatial audio algorithm module running in the audio digital signal processor to perform spatial audio rendering processing on the audio stream data to be processed based on the head pose data includes:

[0014] The audio front-end stream processing module running in the audio digital signal processor receives the audio stream data to be processed transmitted by the main processor, and performs resampling and format conversion processing on the audio stream data to be processed.

[0015] The spatial audio stream data is obtained by performing spatial audio rendering on the audio stream data to be processed after resampling and format conversion based on the head posture data by the spatial audio algorithm module running in the audio digital signal processor.

[0016] Optionally, the operating system running on the main processor includes a framework layer, which includes an audio service module and a mixing module. The step of transmitting the audio stream data to be processed to the audio digital signal processor through the main processor includes:

[0017] The audio service module obtains the audio stream data to be processed output by the upper layer application and the spatial audio identifier corresponding to the audio stream data to be processed. The spatial audio identifier is used to identify the part of the audio stream data to be processed that needs to be spatially rendered.

[0018] The mixing module packages the audio stream data to be processed and the spatial audio identifier and outputs them to the audio digital signal processor.

[0019] The step of performing spatial audio rendering processing on the audio stream data to be processed based on the head posture data by the spatial audio algorithm module running in the audio digital signal processor to obtain spatial audio stream data includes:

[0020] The spatial audio algorithm module performs spatial audio rendering on the audio stream data that needs spatial audio rendering based on the head pose data and the spatial audio identifier. Then, it mixes the spatial audio stream data with the remaining audio stream data in the audio stream data to obtain spatial audio stream data.

[0021] Optionally, the spatial audio device further includes a codec, and the step of outputting the spatial audio stream data through an audio back-end DSP driver module running in the audio digital signal processor includes:

[0022] The spatial audio stream data is transmitted to the codec via the audio backend DSP driver module.

[0023] The spatial audio stream data is converted into an analog audio signal by the codec and then transmitted to the power amplifier in the spatial audio device. The power amplifier amplifies the analog audio signal and drives the speaker in the spatial audio device.

[0024] Alternatively, the spatial audio stream data can be converted into an analog audio signal by the codec and amplified before being output to headphones integrated into or connected to the spatial audio device.

[0025] Optionally, the spatial audio device further includes a head motion sensor connected to the sensor hub, and the spatial audio processing method further includes:

[0026] The head motion sensor data is periodically collected by the hub driver module running in the sensor hub, and the head motion sensor data is timestamped.

[0027] The step of performing spatial audio rendering processing on the audio stream data to be processed based on the head posture data by the spatial audio algorithm module running in the audio digital signal processor to obtain spatial audio stream data includes:

[0028] The spatial audio algorithm module aligns the head pose data and the audio stream data to be processed according to the timestamps corresponding to the head pose data and the timestamps corresponding to the audio stream data to be processed. Based on the head pose data, spatial audio rendering processing is performed on the aligned audio stream data to be processed to obtain spatial audio stream data.

[0029] Optionally, the inter-processor communication method is an SPI bus-based communication method, and the step of transmitting head motion sensor data to the audio digital signal processor through the sensor hub using a preset inter-processor communication method includes:

[0030] The head motion sensor data is transmitted to the audio digital signal processor via the sensor hub using the SPI bus.

[0031] The head motion sensor data is analyzed by the audio digital signal processor according to a preset communication protocol.

[0032] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the spatial audio processing method described above, or to implement the steps of the spatial audio processing model training method described above.

[0033] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the spatial audio processing method described above, or implements the steps of the spatial audio processing model training method described above.

[0034] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the spatial audio processing method described above, or implements the steps of the spatial audio processing model training method described above.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] This application proposes a spatial audio processing method for a spatial audio device, which includes a main processor, an audio digital signal processor (DSP), and a sensor hub. The method includes: transmitting audio stream data to be processed to the DSP via the main processor; transmitting head motion sensor data to the DSP via the sensor hub according to a preset inter-processor communication method; converting the sensor data into head posture data via the DSP; and performing spatial audio rendering processing on the audio stream to be processed based on this data before outputting the result. This application moves the spatial audio algorithm from the Android operating system framework to a dedicated digital signal processor (Audio DSP), making audio data processing and head motion sensor data processing more efficient. By running spatial audio rendering directly in the Audio DSP, the computational burden of the Android framework's intermediate layers and the main processor is avoided, thereby achieving lower latency. Head motion sensor data in the Sensor Hub (such as acceleration and angular velocity data of user head movement) will be sent directly to the Audio DSP for processing via IPC (Inter-Processor Communication). This process reduces the transmission path of head motion sensor data, improves the real-time performance of spatial audio processing, and reduces the time difference between audio and head movement, allowing users to instantly perceive changes in audio when they rapidly turn their heads, thus enhancing immersion. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic diagram of a traditional spatial audio processing framework for spatial audio equipment;

[0040] Figure 2 This is a flowchart illustrating the first embodiment of the spatial audio processing method of this application;

[0041] Figure 3 This is a flowchart illustrating the second embodiment of the spatial audio processing method of this application.

[0042] Figure 4 A schematic diagram of a spatial audio processing framework provided in one embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the spatial audio processing method in the embodiments of this application.

[0044] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0046] It should be noted that in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Surround sound technology plays a crucial role in spatial audio applications. By simulating sound propagation in the real world, it utilizes multiple sound sources in a specific layout to surround the user, creating a multi-dimensional sound space. Accurate and timely audio signal processing is essential when the user's head posture changes. As the user's head position changes, the spatial position of each channel's audio signal relative to the listener also changes. The audio output must be adjusted in real time to maintain the accuracy of sound localization and directionality. This requires the audio processing system to have a high degree of latency control to ensure that the spatial and directional perception of sound responds without lag when the user moves. To achieve this, the system needs to optimize the signal path, improve sampling and computation efficiency, and precisely allocate delays between each channel to avoid auditory misalignment or discomfort caused by delay differences.

[0049] Furthermore, reducing latency in head pose data is a key challenge in spatial audio, particularly in maintaining synchronization between different channels, especially when dealing with complex changes in the location of audio sources. For example, a user's head rotation or movement can cause simultaneous adjustments to the audio signals of multiple channels; excessive latency can negatively impact sound localization accuracy and immersion. Therefore, achieving low-latency audio rendering and processing in surround sound systems is crucial for enhancing the user experience, especially in real-time interactive environments such as virtual reality, augmented reality, or gaming applications.

[0050] In summary, by reducing the latency of head movement posture data, precise control over the spatial localization of sound can be achieved, ensuring the immediacy and smoothness of sound response to provide an immersive and realistic auditory experience.

[0051] With the rapid development of virtual reality, augmented reality, and 360° audio technologies, spatial audio is becoming increasingly important in these technologies. Spatial audio technology can dynamically adjust the spatial positioning of audio signals based on the user's head movement and orientation, enhancing the immersive experience of the audio experience. However, existing VR / AR glasses, helmets, and other spatial audio devices often experience a certain delay when acquiring head motion sensor data for spatial audio rendering processing. This is especially true for spatial audio devices based on the Android operating system, where the transmission of head motion sensor data to the audio digital signal processor introduces a significant delay, affecting the real-time performance of spatial audio rendering processing. This results in head motion delay in spatial audio, reducing the actual effect of the spatial audio.

[0052] To address these issues, researchers discovered that the root cause of the latency lay in the fact that head motion sensor data needed to be processed through the operating system framework layer, resulting in a lengthy transmission link. For example... Figure 1 As shown, sensor data from the spatial audio device based on the Android operating system is transmitted from the sensor driver module 401 through the sensor kernel driver module 302 of the Android KERNEL (Android operating system core layer), the sensor hardware abstraction layer (sensor-HAL) module 202 of the Android HAL (Android operating system hardware abstraction layer), and the sensor service module 105 of the Android Framework (Android operating system framework layer) to the head posture control module 104. This data is then converted into 6-DoF head posture data and transmitted to the spatial audio algorithm module 102 within the Android Framework. The spatial audio algorithm module 102 obtains the ordinary audio stream requiring spatial audio rendering from the audio service module 101, performs spatial audio rendering using the head posture data, and obtains the spatial audio stream. The mixing module 103 mixes the spatial audio stream with the ordinary audio stream (which does not require spatial audio rendering) obtained from the audio service module 101. Finally, the data is transmitted through the audio hardware abstraction layer (audio-HAL) module 201 of the Android HAL and the Android… KERNEL's ALSA (Linux kernel's default audio driver framework) audio kernel driver module 301 is transmitted to the Audio DSP (audio digital signal processor). Although some algorithms have been adopted to reduce latency, the problem of high audio latency still exists due to the long data transmission path and the reliance on the main processor for processing. This latency is particularly noticeable during rapid head movements or extreme head movements, which reduces the actual effect of spatial audio.

[0053] This application aims to solve the spatial audio head motion delay problem caused by the long data transmission path of head motion sensors and the reliance of the calculation process on the main processor (Android framework). Specifically, this application aims to solve the following technical problems: (1) High data transmission delay of head motion sensors: Head motion sensor data is usually transmitted from the bottom layer of the Android software stack (Sensor hub) to the spatial audio algorithm module of the Android Framework. There is a long data transmission and calculation delay during the processing; (2) Head motion sensor data and audio processing are out of sync: When the head rotates quickly, the audio processing of the head motion effect is transmitted from the upper layer of Android (Android Framework) to the speaker connected to the Audio DSP. The path is too long and the response is slow, which causes the audio to be out of sync with the head movement, affecting the user's immersion; (3) Low system performance: Relying on the Android main processor for head motion sensor data processing causes the main processor to be overburdened, reducing the system response speed and performance, especially when the Android main processor is under high load or multitasking.

[0054] To address the aforementioned technical problems, this application proposes a spatial audio processing method for a spatial audio device. The device includes a main processor, an audio digital signal processor (ADSP), and a sensor hub. The method includes: transmitting audio stream data to be processed to the ADSP via the main processor; transmitting head motion sensor data to the ADSP via the sensor hub according to a preset inter-processor communication method; converting the sensor data into head posture data via the ADSP; and performing spatial audio rendering processing on the audio stream based on this data before outputting the result. This application moves the spatial audio algorithm from the Android operating system framework to a dedicated digital signal processor (AudioDSP), making audio data processing and head motion sensor data processing more efficient. By running spatial audio rendering directly in the AudioDSP, the computational burden of the Android framework's intermediate layers and the main processor is avoided, resulting in lower latency. Head motion sensor data in the Sensor Hub (such as acceleration and angular velocity data of user head movement) will be sent directly to the Audio DSP for processing via IPC (Inter-Processor Communication). This process reduces the transmission path of head motion sensor data, improves the real-time performance of spatial audio processing, and reduces the time difference between audio and head movement, allowing users to instantly perceive changes in audio when they rapidly turn their heads, thus enhancing immersion.

[0055] The following presents a first embodiment of the spatial audio processing method of this application. The executing entity of this embodiment can be a spatial audio device, which refers to any electronic device that needs to implement spatial audio rendering processing functions, such as headphone devices, or head-mounted devices such as VR / AR glasses and helmets. In this embodiment, the spatial audio device includes a main processor, an audio digital signal processor (ADSP), and a sensor hub. The main processor refers to the central processing unit that runs the operating system and manages applications, such as a CPU (Central Processing Unit) or an MCU (Microcontroller Unit). If there are multiple MCUs in the spatial audio device, it is the main MCU, responsible for running the operating system, applications, managing the user interface, transmitting audio streams, etc. An audio digital signal processor is a digital signal processor specifically designed for processing audio signals. Its advantages include ultra-low power consumption and real-time performance, and it is specifically optimized for complex mathematical operations, making its audio stream processing efficiency far higher than that of a general-purpose main processor. A sensor hub is a coprocessor specifically designed to manage and process data from various sensors, such as gyroscopes, accelerometers, and magnetometers. It can be implemented using a DSP (Digital Signal Processor) or MCU (Microcontroller Unit). (See reference...) Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the spatial audio processing method of this application. In this embodiment, the spatial audio processing method includes steps S10 to S30:

[0056] Step S10: The main processor transmits the audio stream data to be processed to the audio digital signal processor.

[0057] The audio stream data to be processed can originate from upper-layer applications running on the main processor, such as audio stream data corresponding to game sound effects or system sound effects. The audio stream data to be processed can be single-channel or multi-channel; this embodiment does not impose any limitations. In this embodiment, the main processor transmits the audio stream data to an audio digital signal processor for spatial audio rendering processing, rather than performing spatial audio rendering processing within the operating system framework layer running on the main processor.

[0058] In one feasible implementation, the main processor can transfer the audio stream data to be processed to the buffer of the audio digital signal processor via memory mapping or DMA (Direct Memory Access).

[0059] Step S20: The head motion sensor data is transmitted to the audio digital signal processor via the sensor hub using a preset inter-processor communication method.

[0060] A sensor hub can collect data from head motion sensors at a fixed frequency (hereinafter referred to as head motion sensor data for distinction). Head motion sensors, such as gyroscopes and accelerometers, are sensors that can be used to collect data reflecting the user's head movements.

[0061] There are many ways to implement inter-processor communication (IPC), and this embodiment is not limited to any particular method. For example, it can be implemented using an SPI (Serial Peripheral Interface) bus or an I2C (Inter-Integrated Circuit) bus, or it can be implemented using a shared memory-based inter-processor communication method. For example, in one embodiment, the sensor hub can use the SPI bus to package and send head motion sensor data to the input buffer of the audio digital signal processor.

[0062] Step S30: The head motion sensor data is converted into head posture data by the audio digital signal processor, and spatial audio rendering processing is performed on the audio stream data to be processed based on the head posture data to obtain spatial audio stream data and output it.

[0063] In this embodiment, the conversion from head motion sensor data to head posture data, as well as the spatial audio rendering processing of the audio stream data to be processed, can be migrated from the main processor to the audio digital signal processor. The head posture data can be six-degree-of-freedom (6-DoF) head posture data, and the specific conversion method is not limited in this embodiment. Spatial audio rendering processing can be implemented using spatial audio algorithms, and the algorithm is not limited in this embodiment either.

[0064] In one embodiment, after receiving head motion sensor data, the audio digital signal processor can use quaternion or rotation matrix algorithms to convert the head motion sensor data into six-degree-of-freedom head pose data in three-dimensional space. At the same time, it reads the audio stream data to be processed from the buffer, aligns it with the timestamp, and performs spatial audio rendering processing on the current six-degree-of-freedom head pose data and the audio stream data to be processed in the corresponding time window. For example, it can use the head-related transfer function (HRTF) to adjust the sound source orientation, and finally output a spatial audio stream with spatial positioning information.

[0065] This embodiment moves the spatial audio algorithm from the Android operating system framework to a dedicated digital signal processor (Audio DSP), making audio data processing and head motion sensor data processing more efficient. By running spatial audio rendering directly in the Audio DSP, the computational burden of the Android framework's intermediate layers and the main processor is avoided, resulting in lower latency. Head motion sensor data from the Sensor Hub (such as acceleration and angular velocity data of the user's head movement) is sent directly to the Audio DSP for processing via IPC (Inter-Processor Communication). This process reduces the transmission path of head motion sensor data, improves the real-time performance of spatial audio processing, and reduces the time difference between audio and head movement, allowing users to instantly perceive changes in audio when rapidly turning their heads, thus enhancing immersion.

[0066] In one feasible implementation, the inter-processor communication method is an SPI bus-based communication method, and step S20 includes S201~S202:

[0067] Step S201: Transmit head motion sensor data to the audio digital signal processor via the sensor hub using the SPI bus.

[0068] SPI bus refers to a high-speed synchronous serial interface that uses full-duplex communication mode and supports real-time data transmission between master and slave devices. Specifically, it can be implemented using a four-wire interface, including clock line, master output / slave input line, master input / slave output line, and chip select line.

[0069] The preset communication protocol refers to the predefined data format and transmission rules, which can be implemented using a fixed-length data packet structure. The data packet contains sensor type identifiers, timestamps, and motion parameter fields to standardize data transmission format and improve parsing efficiency. The sensor hub can package head motion sensor data according to the preset communication protocol and transmit the head motion sensor data synchronously and serially to the audio digital signal processor via the SPI bus, utilizing the high-speed transmission characteristics of the SPI bus to reduce the time spent on data transmission at the physical layer.

[0070] Step S202: The head motion sensor data is parsed by the audio digital signal processor according to a preset communication protocol.

[0071] In a specific implementation, after receiving the data, the audio digital signal processor can extract the valid fields in the data packet according to the preset communication protocol. For example, it can distinguish different sensor sources by parsing the sensor type identifier in the header of the data packet and realize data alignment based on the timestamp field, thereby quickly completing the format conversion and content recognition of the sensor data.

[0072] This embodiment solves the latency problem of data transmission from head motion sensor to audio digital signal processor. By optimizing the transmission interface and data parsing process, the overall time from head motion sensor data acquisition to processing is significantly shortened, enabling the audio digital signal processor to acquire head posture change information in real time, ensuring the synchronization of spatial audio rendering processing with user head movement, thereby improving the real-time effect of spatial audio output.

[0073] Based on the first embodiment described above, a second embodiment of the spatial audio processing method of this application is proposed. In this embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. In this embodiment, as... Figure 3 As shown, step S30 includes S301 to S303:

[0074] Step S301: The head motion sensor data is received by the head posture control module running in the audio digital signal processor, and the head motion sensor data is converted into head posture data.

[0075] In this embodiment, the audio digital signal processor runs a head posture control module, a spatial audio algorithm module, and an audio back-end DSP driver module. These three modules form a closed-loop processing flow within the audio digital signal processor, reducing the transmission and processing latency of head motion sensor data by minimizing the number of cross-processor communications.

[0076] The head posture control module is responsible for receiving head motion sensor data, performing data conversion, and transmitting the converted head posture data to the spatial audio algorithm model. Specifically, the head posture control module is a dedicated processing unit used to convert raw head motion sensor data into three-dimensional head posture data. It can be implemented using Kalman filtering or quaternion transformation algorithms to eliminate sensor noise and calculate head posture data.

[0077] In this embodiment, the head posture control module directly receives raw head motion sensor data from the sensor hub and generates accurate head posture data through a built-in conversion algorithm. This avoids multiple transfers of data between the main processor and the audio digital signal processor, thereby reducing the transmission latency of head motion sensor data.

[0078] Step S302: The spatial audio algorithm module running in the audio digital signal processor performs spatial audio rendering processing on the audio stream data to be processed based on the head posture data to obtain spatial audio stream data.

[0079] The spatial audio algorithm module is responsible for performing spatial audio rendering processing on the audio stream data to be processed after acquiring the data transmitted from the main processor, using the head posture data transmitted from the head posture control module. Specifically, the spatial audio algorithm module is a computational unit that dynamically adjusts the spatial positioning of audio based on head posture data. It can be implemented using head-related transfer function algorithms or wave field synthesis algorithms to adjust sound field parameters in real time according to posture changes, thereby achieving spatial audio rendering processing.

[0080] Step S303: The spatial audio stream data is output through the audio back-end DSP driver module running in the audio digital signal processor.

[0081] The audio backend DSP driver module is responsible for managing the output routing and interface driving of the processed audio data. For example, it can be used to transmit the processed spatial audio stream to the codec, and output it after digital-to-analog conversion by the codec.

[0082] Compared to traditional solutions that require transmitting head motion sensor data to the main processor for attitude calculation before forwarding it to the audio digital signal processor for rendering, resulting in an excessively long data transmission path, this embodiment integrates attitude calculation functionality within the audio digital signal processor. This allows head motion sensor data to be directly transmitted to a dedicated processing module (head attitude control module) within the audio digital signal processor, eliminating intermediate processing steps in the main processor and shortening the end-to-end processing time from head motion sensor data to audio rendering.

[0083] In one feasible embodiment, step S302 includes S3021~S3022:

[0084] Step S3021: The audio stream data to be processed transmitted by the main processor is received by the audio front-end stream processing module running in the audio digital signal processor, and the audio stream data to be processed is resampled and format converted.

[0085] Step S3022: The spatial audio stream data is obtained by performing spatial audio rendering processing on the audio stream data to be processed after resampling and format conversion based on the head posture data by the spatial audio algorithm module running in the audio digital signal processor.

[0086] An audio front-end stream processing module can also run within the audio digital signal processor (DSP). This module is a dedicated data processing unit integrated within the DSP. Specifically, it enables continuous data reception through a configured circular buffer and interrupt mechanism. It can perform resampling and format conversion of audio stream data within the DSP, for example, converting the audio stream data to be processed into PCM format. Resampling refers to the process of converting audio stream data with different sampling rates to a uniform sampling rate, which can be achieved using polyphase filter banks or linear interpolation algorithms. When the sampling rate of the audio stream data output by the main processor does not match the computational requirements of the DSP, the audio front-end stream processing module automatically performs resampling operations. The resampled and format-converted audio stream can be directly stored in the DSP's local memory. The spatial audio algorithm module performs spatial audio rendering processing on the resampled and format-converted audio stream data based on head posture data, avoiding the time loss caused by cross-processor data format negotiation.

[0087] This embodiment effectively solves the processing delay problem caused by sampling rate differences during audio stream data transmission. By using a dedicated processing module integrated within the audio digital signal processor, it achieves seamless integration of resampling, data format conversion, and spatial audio rendering, eliminating the waiting time caused by cross-processor data transmission. This allows head posture data and audio stream data to maintain precise time synchronization, thereby improving the real-time performance of spatial audio rendering.

[0088] In one feasible implementation, the operating system running on the main processor includes a framework layer (e.g., the Android Framework), the framework layer including an audio service module and a mixing module, and step S20 includes S203~S204:

[0089] Step S203: Obtain the audio stream data to be processed output by the upper layer application and the spatial audio identifier corresponding to the audio stream data to be processed through the audio service module. The spatial audio identifier is used to identify the part of the audio stream data to be processed that needs to be spatially rendered.

[0090] Spatial audio identifiers are metadata tags used to mark data segments in the audio stream that require spatial rendering processing. They can be implemented using binary flags or timestamp range descriptors, and are used to quickly locate the target processing area in the data stream. The audio service module is the interface component in the operating system framework layer responsible for receiving audio data from the application layer. It can be implemented using Android AudioService or a similar service, and is used to extract the raw audio stream and its associated identifier information.

[0091] After the audio stream data output by the upper-layer application is captured by the audio service module, the spatial audio identifier it carries is extracted synchronously, for example, by parsing specific fields in the audio stream metadata.

[0092] Step S204: The audio stream data to be processed and the spatial audio identifier are packaged by the mixing module and output to the audio digital signal processor.

[0093] A mixing module is a logical unit that mixes multiple audio data streams, encapsulating the original audio streams to be processed with identification information into a unified data packet.

[0094] In one feasible implementation, step S302 includes S3023: the spatial audio algorithm module performs spatial audio rendering processing on the audio stream data that needs spatial audio rendering processing in the audio stream data to be processed according to the head posture data and the spatial audio identifier, and then mixes it with the remaining audio stream data in the audio stream data to be processed to obtain spatial audio stream data.

[0095] After the data packet is transmitted to the audio digital signal processor, the spatial audio algorithm module identifies the data segment to be processed based on the identification field. For example, by detecting the status of the identification bit frame by frame, the HRTF algorithm is called only for the audio frames marked as needing spatial rendering to perform orientation adjustment, and the unmarked parts are directly mixed with the processed spatial audio stream data for output.

[0096] This embodiment achieves refined processing of audio stream data. By using an identification mechanism to accurately filter the audio segments to be rendered, it reduces unnecessary computation and improves processing efficiency. At the same time, since only key audio data is rendered in real time, the time synchronization between head pose data and audio processing is enhanced, thereby improving the real-time performance of spatial audio output.

[0097] In one possible implementation, the spatial audio device further includes a codec, and step S303 includes S3031 and S3032, or includes S3031 and S3033:

[0098] Step S3031: The spatial audio stream data is transmitted to the codec via the audio back-end DSP driver module.

[0099] A codec is a hardware module that converts digital signals to analog signals. Specifically, it can be implemented using a chip that integrates digital-to-analog converter circuits and operational amplifiers to convert digital audio streams into analog electrical signals that can drive speakers or headphones.

[0100] After the spatial audio stream data is rendered, the audio digital signal processor transmits the spatial audio stream data to the codec's data input interface through its audio back-end DSP driver module.

[0101] In step S3032, the spatial audio stream data is converted into an analog audio signal by the codec and then transmitted to the power amplifier in the spatial audio device. The power amplifier amplifies the analog audio signal and drives the speaker in the spatial audio device.

[0102] A power amplifier (PA) is a circuit module used to enhance the voltage or current of an analog signal. It can be used to boost the low-power analog signal output from a codec to a level sufficient to drive a speaker diaphragm.

[0103] After receiving the spatial audio stream data, the codec uses its built-in digital-to-analog converter to convert it into an analog signal waveform. For scenarios requiring the driving of high-power loudspeakers, the analog signal is transmitted to a separate power amplifier for voltage amplification, thereby meeting the driving requirements of the loudspeaker unit.

[0104] Step S3033: The spatial audio stream data is converted into an analog audio signal by the codec and amplified before being output to headphones integrated into or connected to the spatial audio device.

[0105] For headphone output scenarios, the operational amplifier integrated within the codec directly adjusts the amplitude of the analog signal to conform to the headphone jack's driving standard. Both output paths share the same codec hardware resources, achieving compatibility with different playback devices by switching output ports.

[0106] In one feasible embodiment, the spatial audio device further includes a head motion sensor connected to the sensor hub, and the spatial audio processing method further includes step S40, periodically collecting head motion sensor data from the head motion sensor through a hub driver module running in the sensor hub, and marking the head motion sensor data with a timestamp.

[0107] A timestamp is a time sequence identifier that records the moment of data acquisition. Specifically, it can be implemented using time information generated by a system clock counter or a real-time clock module, and is used to accurately mark the generation time of sensor data and audio stream data.

[0108] The hub driver module refers to the firmware program running in the sensor hub. Specifically, it can use interrupt service routines or polling mechanisms to achieve periodic data acquisition. Its function is to actively control the data acquisition frequency of the head motion sensor and add time information to avoid time errors caused by transmission delays when passively receiving data.

[0109] The hub driver module triggers data acquisition from the head motion sensor at fixed intervals, such as acquiring gyroscope and accelerometer data every 10 milliseconds, and embeds the timestamp of the acquisition time into the data packet.

[0110] In one feasible implementation, the steps include S302 and S3024, whereby the spatial audio algorithm module aligns the head pose data and the audio stream data to be processed according to the timestamp corresponding to the head pose data and the timestamp corresponding to the audio stream data to be processed, and performs spatial audio rendering processing on the aligned audio stream data to be processed based on the head pose data to obtain spatial audio stream data.

[0111] Dynamic alignment refers to matching the temporal relationship between different data sources based on timestamps. Specifically, it can be achieved using interpolation algorithms or buffer management mechanisms. By compensating for delay differences in the transmission link, it ensures that the head pose data and the audio stream data to be processed remain synchronized.

[0112] After receiving timestamped head pose data, the spatial audio algorithm module matches it with the timestamps of the audio stream data to be processed, for example, by finding data frames within the same time window for alignment. When timestamp discrepancies exist, a linear interpolation algorithm can be used to generate intermediate head pose data, or the processing delay of the audio stream data to be processed can be adjusted to ensure that changes in head movement correspond in real time to the audio rendering effect.

[0113] In this embodiment, the spatial audio rendering process can dynamically adjust the sound field positioning based on precisely aligned timing information, thereby improving the real-time matching accuracy between the sound source location and the user's head movement in an immersive audio experience.

[0114] In one feasible implementation, such as Figure 4 The diagram illustrates a spatial audio processing framework in a spatial audio device. It should be noted that... Figure 4 Compared to spatial audio processing frameworks Figure 1 Regarding the framework, the position and function of some modules have been adjusted. The following are the adjustments made to the following modules: Figure 4 The following is an explanation of each module:

[0115] 1. Figure 4 The audio service module 101, audio hardware abstraction layer module 201, and ALSA audio kernel driver module 301 in the system can adopt the same... Figure 1 The same module implementation as the audio service module 101, audio hardware abstraction layer module 201, and ALSA audio kernel driver module 301;

[0116] 2. Figure 4 Mixing module 103 and Figure 1 The mixing module 103 in the middle has different functions. Figure 4 The mixing module 103 is used to package the ordinary audio stream (i.e., the audio stream data to be processed) and spatial audio identifier transmitted by the audio service module 101 and output them to the audio digital signal processor (Audio DSP).

[0117] 3. Audio front-end stream processing module 501: Ordinary audio streams are transmitted from the audio service module 101 of the Android Framework through the mixing module 103, the audio-HAL module 201, and the ALSA audio kernel driver module 301 to the audio front-end stream processing module 501 in the Audio DSP. This module performs audio preprocessing operations such as resampling and PCM format conversion on the audio stream, and then sends it to the spatial audio algorithm module 102.

[0118] 4. Sensor driver module 401: Periodically collects head motion sensor data, establishes communication with the Audio DSP, and transmits the head motion sensor data to the head posture control module 104 in the Audio DSP.

[0119] 5. Head posture control module 104: Receives head motion sensor data, converts the head motion sensor data into 6-DOF head posture data, and sends the head posture data to the spatial audio algorithm module 102.

[0120] 6. Spatial audio algorithm module 102: Receives head pose data and ordinary audio stream data, performs spatial audio rendering on the ordinary audio stream, and sends the spatial audio stream to the audio backend driver module 502 (specifically, the audio backend DSP driver module).

[0121] 7. Audio back-end driver module 502: Receives spatial audio stream and sends the spatial audio stream to the PA (power amplifier) ​​or devices such as speakers or headphones connected to the codec.

[0122] The spatial audio processing scheme based on this spatial audio processing framework can achieve the following technical effects:

[0123] Significantly reduced head movement latency: By moving the spatial audio algorithm from the Android framework to the audio digital signal processor and directly transmitting head motion sensor data to the audio digital signal processor for processing, audio latency caused by head movement is effectively reduced, resulting in shorter audio response time and a more real-time audio experience.

[0124] Improved audio synchronization and accuracy: By optimizing data stream transmission and processing paths, a high degree of synchronization between head motion sensor data and audio output is achieved. When head movements occur, the spatial positioning and directional performance of the audio more accurately match the user's head position, avoiding a decrease in immersion caused by audio misalignment or delay, thereby enhancing the user's immersive experience.

[0125] Reduced system load and optimized performance: By shifting spatial audio algorithm calculations from the Android framework to the audio digital signal processor (DSP), the computational burden on the main processor is reduced, improving the overall system efficiency. The DSP, as a dedicated hardware processing unit, can process audio data and head motion sensor data more efficiently, resulting in a smoother audio experience and reduced power consumption and latency.

[0126] Enhanced user immersion: By synchronizing audio and head movements in real time and with precision, the audio experience in virtual reality (VR) and augmented reality (AR) is significantly improved. When users experience rapid head movements or vigorous activity, the audio digital signal processor can run spatial audio algorithms with shorter latency, allowing them to perceive audio changes more naturally and achieve a more immersive and realistic auditory experience, thus enhancing immersion and interactivity.

[0127] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the spatial audio processing method or spatial audio processing model training method described above.

[0128] The following is for reference. Figure 5 This document illustrates a structural diagram of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0129] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0130] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0131] The electronic device provided in this application adopts the spatial audio processing method or spatial audio processing model training method in the above embodiments. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as the beneficial effects of the spatial audio processing method or spatial audio processing model training method provided in the above embodiments. Moreover, other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, and will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0134] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the spatial audio processing method or spatial audio processing model training method described above.

[0135] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0136] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0137] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the functions defined in the methods of the embodiments disclosed in this application.

[0138] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0141] The readable storage medium provided in this application embodiment is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-described spatial audio processing method or spatial audio processing model training method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the spatial audio processing method or spatial audio processing model training method provided in the above-described embodiments, and will not be repeated here.

[0142] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the spatial audio processing method or spatial audio processing model training method described above.

[0143] Compared with the prior art, the beneficial effects of the computer program product provided in this application embodiment are the same as the beneficial effects of the spatial audio processing method or spatial audio processing model training method provided in the above embodiments, and will not be repeated here.

[0144] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A spatial audio processing method, characterized by, The spatial audio processing method is applied to a spatial audio device including a main processor, an audio digital signal processor and a sensor hub, and the spatial audio processing method includes: transmitting, by the main processor, to-be-processed audio stream data to the audio digital signal processor; transmitting, by the sensor hub, head motion sensor data to the audio digital signal processor in a preset inter-processor communication mode; receiving, by a head pose control module running in the audio digital signal processor, the head motion sensor data and converting the head motion sensor data into head pose data; performing, by a spatial audio algorithm module running in the audio digital signal processor, spatial audio rendering processing on the to-be-processed audio stream data according to the head pose data to obtain spatial audio stream data; outputting, by an audio back-end DSP driving module running in the audio digital signal processor, the spatial audio stream data; an operating system running in the main processor includes a framework layer, the framework layer includes an audio service module and a mix module, and the step of transmitting, by the main processor, to-be-processed audio stream data to the audio digital signal processor includes: obtaining, by the audio service module, to-be-processed audio stream data output by an upper-layer application and a spatial audio identifier corresponding to the to-be-processed audio stream data, wherein the spatial audio identifier is used to identify a part of the to-be-processed audio stream data that needs to be subjected to spatial audio rendering processing; encapsulating, by the mix module, the to-be-processed audio stream data and the spatial audio identifier into a unified data packet and then outputting the unified data packet to the audio digital signal processor; the step of performing, by the spatial audio algorithm module running in the audio digital signal processor, spatial audio rendering processing on the to-be-processed audio stream data according to the head pose data to obtain spatial audio stream data includes: performing, by the spatial audio algorithm module, spatial audio rendering processing on audio stream data in the to-be-processed audio stream data that needs to be subjected to spatial audio rendering processing according to the head pose data and the spatial audio identifier, and then mixing the audio stream data with the rest of the to-be-processed audio stream data to obtain spatial audio stream data, wherein the spatial audio algorithm module identifies the audio stream data in the to-be-processed audio stream data that needs to be subjected to spatial audio rendering processing according to the spatial audio identifier.

2. The spatial audio processing method of claim 1, wherein, the step of performing, by the spatial audio algorithm module running in the audio digital signal processor, spatial audio rendering processing on the to-be-processed audio stream data according to the head pose data to obtain spatial audio stream data includes: receiving, by an audio front-end stream processing module running in the audio digital signal processor, the to-be-processed audio stream data transmitted by the main processor and performing resampling and format conversion processing on the to-be-processed audio stream data; The spatial audio stream data is obtained by performing spatial audio rendering processing on the resampled and format-converted audio stream data according to the head pose data by a spatial audio algorithm module running in the audio digital signal processor.

3. The spatial audio processing method of claim 1, wherein, The step of outputting the spatial audio stream data by the audio backend DSP driving module running in the audio digital signal processor includes: transmitting the spatial audio stream data to the codec by the audio backend DSP driving module; transmitting the spatial audio stream data converted into analog audio signals by the codec to a power amplifier in the spatial audio device, performing power amplification on the analog audio signals by the power amplifier, and driving a loudspeaker in the spatial audio device by the power amplifier; or, converting the spatial audio stream data into analog audio signals by the codec, amplifying the analog audio signals, and outputting the amplified analog audio signals to earphones integrated with or externally connected to the spatial audio device.

4. The spatial audio processing method of claim 1, wherein, The spatial audio device further includes a head motion sensor connected to the sensor hub, and the spatial audio processing method further includes: periodically collecting head motion sensor data of the head motion sensor by a hub driving module running in the sensor hub, and marking a timestamp for the head motion sensor data; The step of obtaining the spatial audio stream data by performing spatial audio rendering processing on the audio stream data to be processed according to the head pose data by a spatial audio algorithm module running in the audio digital signal processor includes: aligning the head pose data and the audio stream data to be processed according to the timestamp corresponding to the head pose data and the timestamp corresponding to the audio stream data to be processed by the spatial audio algorithm module, performing spatial audio rendering processing on the aligned audio stream data to be processed based on the head pose data, and obtaining the spatial audio stream data.

5. A spatial audio processing method according to any one of claims 1 to 4, wherein, The inter-processor communication mode is a communication mode based on an SPI bus, and the step of transmitting the head motion sensor data to the audio digital signal processor by the sensor hub in a preset inter-processor communication mode includes: transmitting the head motion sensor data to the audio digital signal processor based on an SPI bus by the sensor hub; parsing the head motion sensor data according to a preset communication protocol by the audio digital signal processor.

6. An electronic device, comprising: The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the spatial audio processing method according to any one of claims 1 to 5.

7. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the spatial audio processing method according to any one of claims 1 to 5.

8. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by a processor to implement the steps of the spatial audio processing method according to any one of claims 1 to 5.

Images