Audio processing method, chip, and electronic device

Abstract

The present application relates to the technical field of electronics, and provides an audio processing method, a chip, and an electronic device, which resolve, to some extent, the problem of low head motion data transmission and processing of an electronic device. The method is applied to a terminal device, the terminal device is communicationally connected to a wearable device, and the terminal device comprises a main processor and an auxiliary processor. The method comprises: the main processor decodes source audio data to obtain decoded audio data; the main processor transmits the audio data to the auxiliary processor; the auxiliary processor obtains inertial sensing data of the wearable device; and the auxiliary processor renders the audio data on the basis of the inertial sensing data, to obtain spatial audio data.

Description

Audio processing method, chip and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on November 2, 2023, with application number 202311452641.6 and application name “An audio processing method, chip and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to an audio processing method, chip, and electronic device. Background Art

[0003] With the rapid development of wearable devices, product functions are becoming increasingly rich, such as active noise reduction, call noise reduction, spatial audio, and wireless charging. Among them, the spatial audio function, also known as the 3D audio effect function, can capture the user's head movements in real time through the wearable device, and then dynamically track the sound field based on the captured head movements, creating an audio effect that surrounds the head, achieving an immersive listening experience.

[0004] The implementation of spatial audio functionality relies on the electronic device's rapid transmission and processing of head motion data. If this is slow, the sound will only change after the user's head has already changed relative to the space, resulting in a delayed response. This affects the user's immersive experience and reduces the user's enjoyment of spatial audio.

[0005] Summary of the Invention

[0006] The present application provides an audio processing method, chip, and electronic device, which to a certain extent solve the problem of slow transmission and processing of head motion data by electronic devices, and improve the user experience of spatial audio functions.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, the present application provides an audio processing method, which is applied to a terminal device, which is communicatively connected to a wearable device, and the terminal device includes a main processor and an auxiliary processor. The method includes: the main processor decodes the source audio data to obtain decoded audio data; the main processor transmits the audio data to the auxiliary processor; the auxiliary processor obtains inertial sensor data of the wearable device; the auxiliary processor renders the audio data according to the inertial sensor data to obtain spatial audio data.

[0009] Based on the audio processing method provided in the embodiment of the present application, in the process of processing audio data, the auxiliary processor renders the audio data according to the inertial sensor data of the wearable device obtained to obtain spatial audio data, effectively avoiding the slow transmission and processing of audio data caused by the main processor needing to process too much business data. By processing the audio data through the auxiliary processor, the processing speed of the electronic device for inertial sensor data (i.e., the user's head motion data) is improved, and "sound follows the head movement" is quickly realized, effectively solving the problem of slow transmission and processing of head motion data by the electronic device.

[0010] Furthermore, using the auxiliary processor to process audio data can also address the issue of excessive power consumption by the main processor due to the large amount of data it must process. It can also further reduce the main processor's continuous utilization, freeing up its computing power, preventing the main processor from preempting it, and further minimizing the impact on audio data transmission and processing. For example, if a request to process other business data with a higher priority than the audio data is received while the main processor is processing audio data, the main processor may prioritize processing the other business data, thereby affecting the main processor's ability to transmit and process audio data.

[0011] In the embodiment of the present application, the main processor may include a CPU; the auxiliary processor may include a DSP.

[0012] For example, terminal devices may include mobile phones, PCs, smart phones, netbooks, tablet computers, smart cameras, PDAs, smart TVs, etc. Wearable devices include but are not limited to headphones, smart glasses, or smart helmets.

[0013] In one example, the source audio data may be music audio data, video audio data, call audio data, etc. Furthermore, the source of the source audio data may include audio data played from an application pre-installed on the terminal device, or may include audio data played from a recorded audio source file, which is not limited in this embodiment of the present application.

[0014] In one possible implementation, the inertial sensing data may include the translation distance and angular velocity data of the wearable device; it may also include the translation distance, acceleration data, and angular velocity data of the wearable device, etc. Among them, the inertial sensing data can be obtained by an IMU set in the wearable device. For example, a preset number of IMUs can be set in the wearable device, and the IMU may include an accelerometer sensor and a gyroscope sensor to obtain the inertial sensing data (i.e., IMU data) of the wearable device by using the accelerometer sensor and the gyroscope sensor. In this possible implementation, the number of IMUs set in the wearable device may vary depending on the type of chip set in the wearable device. The embodiments of the present application do not limit the specific content of the inertial sensing data, the number of IMUs set in the wearable device, and the type and number of specific sensors set in the IMU.

[0015] In a possible implementation of the first aspect, the terminal device further includes a communication module, and the method further includes:

[0016] The communication module receives the inertial sensor data transmitted by the wearable device through a first protocol, where the first protocol is a unidirectional transmission protocol;

[0017] The auxiliary processor obtains the inertial sensing data of the wearable device, including: the auxiliary processor obtains the inertial sensing data through the communication module.

[0018] It should be understood that the communication module receives the inertial sensing data transmitted by the wearable device through the first protocol, that is, the wearable device transmits the inertial sensing data to the communication module through the first protocol.

[0019] In the embodiment of the present application, the first protocol may include the HID protocol.

[0020] For example, the communication module may be a Bluetooth communication module, which may specifically be a Bluetooth control chip BTC.

[0021] Based on the above possible implementation methods, compared to transmitting the inertial sensor data to the main processor for processing the inertial sensor data by the main processor, the communication module receives the inertial sensor data transmitted by the wearable device through a one-way transmission protocol, so that the communication module only needs to receive the inertial sensor data sent by the wearable device, and there is no need to reply corresponding response information to the wearable device whether the inertial sensor data transmitted by the wearable device is received. This can effectively reduce the transmission path of the inertial sensor data and further reduce the transmission delay of the audio data, so that the user can hear smoother spatial audio, thereby improving the user's experience of the spatial audio function and improving the user's auditory experience.

[0022] In addition, through the above-mentioned possible implementation methods, not only the speed of transmitting inertial sensor data from the wearable device to the communication module is accelerated, but also the main processor (or auxiliary processor) is avoided from having to reply to the wearable device with response information corresponding to the received inertial sensor data, causing the main processor (or auxiliary processor) to be in an awake state all the time, further reducing the utilization rate of the main processor (or auxiliary processor) and reducing the power consumption of the main processor (or auxiliary processor).

[0023] In one possible implementation of the first aspect, the auxiliary processor acquiring inertial sensor data via the communication module includes: the auxiliary processor receiving the inertial sensor data transmitted by the communication module. In other words, the auxiliary processor directly receives the inertial sensor data transmitted by the communication module. In an actual design, a corresponding transmission module may also be provided in the auxiliary processor to utilize the transmission module to receive the inertial sensor data transmitted by the Bluetooth communication module.

[0024] In a possible implementation of the first aspect, the terminal device further includes a data transmission module, and the auxiliary processor obtains the inertial sensing data through the communication module, including:

[0025] The data transmission module receives the inertial sensor data transmitted by the communication module;

[0026] The auxiliary processor obtains inertial sensor data through the data transmission module.

[0027] Exemplarily, the data transmission module may be provided in a Sensor Hub, and the Sensor Hub may be used to transmit the inertial sensor data transmitted by the communication module to the auxiliary processor.

[0028] Based on the above possible implementation methods, not only can the transmission path for the auxiliary processor to obtain inertial sensor data be enriched, but the transmission of inertial sensor data through the data transmission module other than the auxiliary processor in the terminal device can further accelerate the transmission speed of the inertial sensor data, accelerate the processing efficiency of the auxiliary processor for inertial sensor data, and further reduce the transmission and processing delay of the audio data.

[0029] In a possible implementation of the first aspect, the method further includes:

[0030] The auxiliary processor encodes the spatial audio data to obtain encoded spatial audio data;

[0031] The auxiliary processor transmits the encoded spatial audio data to the wearable device through the communication module.

[0032] In a possible implementation of the first aspect, the first protocol includes a HID protocol.

[0033] In a possible implementation of the first aspect, the auxiliary processor includes a DSP chip.

[0034] In a second aspect, an embodiment of the present application provides a chip, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory, so that the chip executes the method executed by the auxiliary processor in any possible implementation of the first aspect.

[0035] In a third aspect, an embodiment of the present application provides a chip system, which includes: a main processor and an auxiliary processor, and the chip system is used to call a program from a memory so that a device equipped with the chip system executes the method described in any possible implementation of the first aspect.

[0036] In a fourth aspect, the present application provides an electronic device comprising: a main processor and an auxiliary processor, wherein the electronic device is configured to execute a computer program or instruction stored in a memory so that the electronic device executes the method described in any possible implementation of the first aspect.

[0037] In a possible implementation of the fourth aspect, the main processor is configured to perform the following steps: decoding the source audio data to obtain decoded audio data; transmitting the audio data to the auxiliary processor;

[0038] The auxiliary processor is used to perform the following steps: obtaining inertial sensor data of the wearable device; rendering and processing the audio data according to the inertial sensor data to obtain spatial audio data.

[0039] In a fifth aspect, the present application provides a communication system, which includes: a terminal device and / or a wearable device.

[0040] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method in any possible implementation of the first aspect.

[0041] In a seventh aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method in any possible implementation of the first aspect.

[0042] The technical effects of the second to seventh aspects provided in this application can refer to the technical effects of the various possible implementation methods of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a schematic diagram of a scenario corresponding to an audio processing method provided in an embodiment of the present application.

[0044] FIG2 is a schematic diagram of a scenario in which a user hears spatial audio data, provided by an embodiment of the present application.

[0045] FIG3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0046] FIG4 is a schematic diagram of the hardware structure of an earphone provided in an embodiment of the present application.

[0047] FIG5 is a schematic diagram of a system architecture of a terminal device and headphones provided in an embodiment of the present application.

[0048] FIG6 is a flow chart of an audio processing method provided in an embodiment of the present application.

[0049] FIG7 is a structural diagram of an inertial sensing data flow provided in an embodiment of the present application.

[0050] FIG8 is a structural diagram of another inertial sensing data flow provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings and related embodiments in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a kind of "or" relationship.

[0052] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0053] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] It can be understood that the audio processing method provided in the embodiment of the present application can be applied to the following scenarios: the terminal device is communicatively connected to the wearable device, the wearable device is worn on the user's head, the wearable device can be used to collect the user's head rotation data, and send the collected user's head rotation data to the terminal device through the communication connection, the terminal device uses the received user's head rotation data to dynamically track the sound field to obtain spatial audio data, and then transmits the spatial audio data to the wearable device through the communication connection, so that the user can listen to the spatial audio data through the wearable device.

[0055] As shown in Figure 1, a schematic diagram of a scenario corresponding to the audio processing method provided in an embodiment of the present application is shown in Figure 1. Assume that the terminal device is a mobile phone 100, the wearable device is a true wireless stereo (TWS) headset 200, and the mobile phone 100 is communicatively connected to the TWS headset 200. In order to enable the user to listen to audio with a good sense of space and direction through the TWS headset 200, the mobile phone 100 can capture the user's head rotation data through the WS headset 200 worn on the head by the user, and then process the head rotation data to achieve dynamic tracking of the sound field, creating an audio effect of sound surrounding the head, and achieving an immersive listening experience, see Figure 2. In other words, if the sound source position is fixed, then when the user's head turns, the sound effects heard by the two ears should be different.

[0056] For example, a user wearing TWS headphones is enjoying a musical instrument. Assuming the user can determine that the instrument is directly in front of them, the volume of the sound heard by both ears is roughly the same. If the user turns their head to the right, then corresponding to the real auditory environment, the volume of the sound heard by the user's left ear should be louder than that heard by the right ear because the user's left ear is closer to the instrument.

[0057] In some embodiments, the audio processing method provided in the embodiments of the present application can be applied to scenarios in which the terminal device and the wearable device are communicated and connected, as well as to other scenarios. As an example and not a limitation, assuming that the wearable device (such as smart glasses) itself has the function of processing the user's head rotation data by the terminal device, the wearable device can directly use the user's head rotation data collected by itself to dynamically track the sound field and play the spatial audio data after dynamic tracking processing. For example, the terminal device can collect its own inertial sensor data to realize dynamic tracking processing of the sound field, and play the spatial audio data after dynamic tracking processing through the built-in speaker of the terminal device. Alternatively, the terminal device sends the audio data after dynamic tracking processing to an audio playback device (for example, a speaker) to use the audio playback device to play the spatial audio data after dynamic tracking processing, so that the audio heard by the user has a better sense of space, etc. The embodiments of the present application do not impose any restrictions on the application scenarios of the audio processing method.

[0058] It should be understood that "head rotation data", "inertial sensor data" and "IMU data" all refer to the same content. For the convenience of description, one of the descriptions is used in different application scenarios, and the three can be interchangeable.

[0059] Based on the above possible application scenarios, as examples and not limitations, terminal devices may include but are not limited to personal computers (PCs), smart phones, netbooks, tablet computers, smart cameras, wearable devices, PDAs, smart TVs, personal digital assistants (PDAs), portable multimedia players (PMPs), projection devices, smart screen devices, augmented reality (AR) / virtual reality (VR) devices, mixed reality (MR) devices, in-vehicle devices, smart screens, cloud servers, televisions, or somatosensory game consoles in human-computer interaction scenarios. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0060] For example, the wearable device can be any one of headphones, smart glasses, smart helmets, AR devices, VR devices, and MR devices. Depending on the actual application scenario, the wearable device can be any of the above-mentioned terminal devices. Of course, the wearable device or terminal device can also be other devices based on future technologies. The embodiments of this application do not impose any restrictions on the specific type of wearable device.

[0061] FIG3 is a schematic diagram of the structure of an electronic device 300 provided in the present application. The electronic device 300 may include the terminal device in the above embodiment. Referring to FIG3 , the electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 331, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395. The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0062] It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0063] For example, when the electronic device 300 is a mobile phone or a tablet computer, it may include all the components shown in the figure, or may include only some of the components shown in the figure.

[0064] The processor 310 may include a main processor and one or more auxiliary processors. For example, the main processor may be a central processing unit (CPU), and at least one auxiliary processor may be a digital signal processor (DSP). In other possible implementations, the processor 310 may further include other processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0065] The controller may be the nerve center and command center of the electronic device 300. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0066] Processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 310 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 310. If processor 310 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 310 latency, and thus improves system efficiency.

[0067] In some embodiments, the processor 310 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0068] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 310 may include multiple I2C busses. The processor 310 may be coupled to the touch sensor 380K, the charger, the flash, the camera 393, and the like via different I2C bus interfaces. For example, the processor 310 may be coupled to the touch sensor 380K via the I2C interface, enabling communication between the processor 310 and the touch sensor 380K via the I2C bus interface, thereby enabling the touch function of the electronic device 300.

[0069] The I2S interface can be used for audio communication. In some embodiments, the processor 310 can include multiple I2S buses. The processor 310 can be coupled to the audio module 370 via the I2S bus to enable communication between the processor 310 and the audio module 370. In some embodiments, the audio module 370 can transmit audio signals to the wireless communication module 360 ​​via the I2S interface.

[0070] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 370 and the wireless communication module 360 ​​can be coupled via a PCM bus interface.

[0071] In some embodiments, the audio module 370 may also transmit audio signals to the wireless communication module 360 ​​via a PCM interface. Both the I2S interface and the PCM interface may be used for audio communication.

[0072] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between parallel communication and serial communication.

[0073] In some embodiments, a UART interface is typically used to connect the processor 310 to the wireless communication module 360. For example, the processor 310 communicates with the Bluetooth module in the wireless communication module 360 ​​via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 370 can transmit audio signals to the wireless communication module 360 ​​via the UART interface, enabling the playback of music via Bluetooth headphones.

[0074] The MIPI interface can be used to connect the processor 310 to peripheral devices such as the display screen 394 and the camera 393. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 310 and the camera 393 communicate via the CSI interface to implement the camera function of the electronic device 300. The processor 310 and the display screen 394 communicate via the DSI interface to implement the display function of the electronic device 300.

[0075] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 310 to the camera 393, the display 394, the wireless communication module 360, the audio module 370, the sensor module 380, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0076] USB interface 330 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. USB interface 330 can be used to connect a charger to charge electronic device 300, or to transfer data between electronic device 300 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0077] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0078] The charging management module 340 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 can receive charging input from the wired charger via the USB interface 330. In some wireless charging embodiments, the charging management module 340 can receive wireless charging input via the wireless charging coil of the electronic device 300. While charging the battery 342, the charging management module 340 can also provide power to the electronic device via the power management module 341.

[0079] The power management module 341 is used to connect the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340 and provides power to the processor 310, the internal memory 331, the external memory interface 320, the display 394, the camera 393, and the wireless communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance), etc.

[0080] In some other embodiments, the power management module 341 may also be provided in the processor 310. In some other embodiments, the power management module 341 and the charging management module 340 may also be provided in the same device.

[0081] The wireless communication function of the electronic device 300 can be implemented through the antenna 1, the antenna 2, the mobile communication module 350, the wireless communication module 360, the modem processor and the baseband processor.

[0082] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0083] The mobile communication module 350 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the electronic device 300. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), and the like. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 350 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 1.

[0084] In some embodiments, at least some functional modules of the mobile communication module 350 may be provided in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 may be provided in the same device as at least some functional modules of the processor 310.

[0085] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 370A, the receiver 370B, etc.) or displays an image or video through the display screen 394. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 310 and be provided in the same device as the mobile communication module 350 or other functional modules.

[0086] The wireless communication module 360 ​​can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 300. The wireless communication module 360 ​​can be one or more devices that integrate at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310. The wireless communication module 360 ​​can also receive the signal to be sent from the processor 310, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0087] In some embodiments, antenna 1 of electronic device 300 is coupled to mobile communication module 350, and antenna 2 is coupled to wireless communication module 360, so that electronic device 300 can communicate with a network and other devices via wireless communication technology. Wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0088] Electronic device 300 implements display functionality through a GPU, display screen 394, and an application processor. A GPU is a microprocessor for image processing that connects display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or modify display information.

[0089] The display screen 394 is used to display images, videos, etc. For example, the first lighting pattern and the second lighting pattern in the embodiment of the present application. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include one or N display screens 394, where N is a positive integer greater than one.

[0090] The electronic device 300 can realize the shooting function through the ISP, camera 393, video codec, GPU, display screen 394 and application processor.

[0091] The ISP processes data fed back by camera 393. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 393.

[0092] The camera 393 is used to capture still images or videos. The object generates an optical image through the lens and is projected onto the photosensitive element. The focal length of the lens can be used to indicate the camera's field of view. The smaller the focal length of the lens, the larger the lens's field of view. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB, YUV, etc.

[0093] In the present application, the electronic device 300 may include cameras 393 with two or more focal lengths.

[0094] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 300 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0095] Video codecs are used to compress or decompress digital video. Electronic device 300 may support one or more video codecs. This allows electronic device 300 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, and MPEG4.

[0096] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in electronic device 300, such as image recognition, face recognition, speech recognition, and text comprehension.

[0097] In an embodiment of the present application, the NPU or other processors can be used to perform operations such as analyzing and processing images in the video stored in the electronic device 300.

[0098] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 via the external memory interface 320 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0099] The internal memory 331 can be used to store computer executable program code, which includes instructions. The processor 310 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the internal memory 331. The internal memory 331 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.). The data storage area may store data created during the use of the electronic device 300 (such as audio data, a phone book, etc.).

[0100] In addition, the internal memory 331 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0101] The electronic device 300 can implement audio functions through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, the headphone jack 370D, and the application processor.

[0102] The audio module 370 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio input into digital audio signals. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be provided in the processor 310, or some functional modules of the audio module 370 can be provided in the processor 310.

[0103] Speaker 370A, also called a "horn," is used to convert audio electrical signals into sound signals. Electronic device 300 can listen to music or listen to hands-free calls through speaker 370A. For example, the speaker can play the comparison and analysis results provided in the embodiments of the present application.

[0104] The receiver 370B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 300 receives a call or a voice message, the user can place the receiver 370B close to the ear to hear the voice.

[0105] Microphone 370C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 370C to input the sound signal into the microphone 370C. The electronic device 300 can be provided with at least one microphone 370C. In other embodiments, the electronic device 300 can be provided with two microphones 370C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 300 can also be provided with three, four or more microphones 370C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0106] The headphone jack 370D is used to connect a wired headphone. The headphone jack 370D can be a USB interface 330 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0107] The pressure sensor 380A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 380A can be set on the display screen 394. There are many types of pressure sensors 380A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates with conductive material. When force acts on the pressure sensor 380A, the capacitance between the electrodes changes. The electronic device 300 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 394, the electronic device 300 detects the intensity of the touch operation based on the pressure sensor 380A. The electronic device 300 can also calculate the position of the touch based on the detection signal of the pressure sensor 380A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0108] The gyroscope sensor 380B can be used to determine the motion posture of the electronic device 300. In some embodiments, the angular velocity of the electronic device 300 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 380B. The gyroscope sensor 380B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 380B detects the angle of the electronic device 300 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 300 through reverse movement to achieve anti-shake. The gyroscope sensor 380B can also be used for navigation and somatosensory game scenes.

[0109] The air pressure sensor 380C is used to measure air pressure. In some embodiments, the electronic device 300 calculates the altitude using the air pressure value measured by the air pressure sensor 380C to assist in positioning and navigation.

[0110] The magnetic sensor 380D includes a Hall sensor. The electronic device 300 can use the magnetic sensor 380D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 300 is a flip phone, the electronic device 300 can detect the opening and closing of the flip cover using the magnetic sensor 380D. Based on the detected opening and closing status of the leather case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0111] Accelerometer 380E can detect the magnitude of acceleration of electronic device 300 in all directions (generally three axes). When electronic device 300 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0112] The distance sensor 380F is used to measure distance. The electronic device 300 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 300 can use the distance sensor 380F to measure distance to achieve fast focusing.

[0113] The proximity light sensor 380G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 300 emits infrared light outward through the light emitting diode. The electronic device 300 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 300. When insufficient reflected light is detected, the electronic device 300 can determine that there is no object near the electronic device 300. The electronic device 300 can use the proximity light sensor 380G to detect when the user holds the electronic device 300 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 380G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0114] Ambient light sensor 380L is used to sense ambient light brightness. Electronic device 300 can adaptively adjust the brightness of display screen 394 based on the perceived ambient light. Ambient light sensor 380L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 380L can also work with proximity light sensor 380G to detect whether electronic device 300 is in a pocket to prevent accidental touches.

[0115] The fingerprint sensor 380H is used to collect fingerprints. The electronic device 300 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0116] The temperature sensor 380J is used to detect temperature. In some embodiments, the electronic device 300 uses the temperature detected by the temperature sensor 380J to implement a temperature processing strategy. For example, when the temperature reported by the temperature sensor 380J exceeds a threshold, the electronic device 300 reduces the performance of the processor located near the temperature sensor 380J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 300 heats the battery 342 to prevent the electronic device 300 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 300 boosts the output voltage of the battery 342 to prevent abnormal shutdown due to low temperature.

[0117] The touch sensor 380K is also called a "touch panel." The touch sensor 380K can be disposed on the display screen 394. The touch sensor 380K and the display screen 394 form a touch screen, also called a "touch screen." The touch sensor 380K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 394. In other embodiments, the touch sensor 380K can also be disposed on the surface of the electronic device 300, at a location different from that of the display screen 394.

[0118] Bone conduction sensor 380M can acquire vibration signals. In some embodiments, bone conduction sensor 380M can acquire vibration signals from vibrating bones in the human body. Bone conduction sensor 380M can also contact the human pulse to receive blood pressure signals.

[0119] In some embodiments, the bone conduction sensor 380M can also be installed in headphones, forming a bone conduction headset. The audio module 370 can parse the vibration signal of the vocal bone obtained by the bone conduction sensor 380M into a voice signal, thus implementing a voice function. The application processor can parse the heart rate information based on the blood pressure pulse signal obtained by the bone conduction sensor 380M, thus implementing a heart rate detection function.

[0120] The buttons 390 include a power button, a volume button, and the like. The buttons 390 may be mechanical buttons or touch buttons. The electronic device 300 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 300.

[0121] Motor 391 can generate vibration prompts. Motor 391 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 394, motor 391 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0122] Indicator 392 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0123] The SIM card interface 395 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 300 by inserting it into or removing it from the SIM card interface 395. The electronic device 300 can support 3 or N SIM card interfaces, where N is a positive integer greater than 3. The SIM card interface 395 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 395 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 395 can also be compatible with different types of SIM cards. The SIM card interface 395 can also be compatible with external memory cards. The electronic device 300 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 300 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 300 and cannot be separated from the electronic device 300.

[0124] For example, as shown in FIG4 , a hardware structure diagram of a wearable device 400 provided in an embodiment of the present application is shown. Referring to FIG4 , the wearable device 400 may include one or more processors 410, one or more memories 420, a communication interface 430, an audio acquisition circuit, and an audio playback circuit. The audio acquisition circuit may further include at least one microphone 440 and an analog-to-digital converter (ADC) 450. The audio playback circuit may further include a speaker 460 and a digital-to-analog converter (DAC).

[0125] The wearable device 400 may further include one or more sensors 480, such as an inertial measurement unit (IMU), a proximity sensor, a motion sensor, etc. These hardware components may communicate on one or more communication buses.

[0126] In the embodiment of the present application, the IMU can be used to measure the motion posture of the wearable device 400. For example, the IMU can be used to determine the translation distance data, angular velocity data, and acceleration data when the user wears the wearable device 400. The IMU can be provided with a gyroscope sensor and an acceleration sensor.

[0127] The processor 410 is the control center of the wearable device 400. The processor may also be referred to as a control unit, a controller, a microcontroller, or another appropriate term. The processor 410 uses various interfaces and lines to connect the various components of the wearable device 400. In a possible embodiment, the processor 410 may also include one or more processing cores. In a possible embodiment, a main control unit and a signal processing module may be integrated into the processor 410. The main control unit (MCU) is used to receive data collected by the sensor 480 or monitoring signals from the signal processing module or control signals from a terminal device (such as a mobile phone), and finally control the wearable device 400 through comprehensive judgment and decision-making.

[0128] The memory 420 can be coupled to the processor 410 or connected to the processor 410 via a bus to store various software programs and / or multiple sets of instructions and data. The memory 420 can also store a communication program that can be used to communicate with the terminal. In one example, the memory 420 can also store data / program instructions, and the processor 410 can be used to call and execute the data / program instructions in the memory 420. Exemplarily, the memory 420 can store multiple sets of noise reduction parameters. Optionally, the memory 420 can be a memory external to the MCU or a storage unit built into the MCU.

[0129] The communication interface 430 is used to communicate with the terminal, and the communication method can be wired or wireless. When the communication method is wired communication, the communication interface 430 can be connected to the terminal device via a cable. When the communication method is wireless communication, the communication interface 430 is used to receive and send radio frequency signals. The wireless communication method supported by it can be, for example, Bluetooth communication, wireless fidelity (Wifi) communication, infrared communication, near field communication technology (NFC), or cellular 2 / 3 / 4 / 5 generation (2G / 3G / 4G / 5G) communication, etc., such as at least one of the communication methods.

[0130] The microphone 440 can be used to collect sound signals (or audio signals, which are analog signals), and the analog-to-digital converter 450 is used to convert the analog signals collected by the microphone 440 into digital signals, and send the digital signals to the processor 410 for processing. In a specific embodiment, the digital signals can be sent to the signal processing module for processing, and the signal processing module can transmit the processed signals (such as mixed audio signals) to the digital-to-analog converter 470. The digital-to-analog converter 470 can convert the received signals into analog signals, and then transmit them to the speaker 460. The speaker 460 is used to play according to the analog signal, so that the user can hear the sound.

[0131] In an embodiment of the present application, the communication interface 430 can be used to transmit the translation distance, acceleration data, and / or angular velocity data detected by the IMU to a terminal device, so that the terminal device can use the translation distance, acceleration data, and / or angular velocity data to determine the IMU data. Furthermore, the communication interface 430 can also be used to receive spatial audio data transmitted by the terminal device. This spatial audio data can be understood as audio data with a certain sense of space and direction after being rendered by the terminal device.

[0132] It is understood that the wearable device 400 may be a headset, which may also be referred to as an earbud, headset, walkman, audio player, media player, head-mounted receiver, earpiece device, or other appropriate term. The specific type of headset may be a TWS headset, which is not limited in the embodiments of the present application.

[0133] It should be noted that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the wearable device 400. In other embodiments of the present application, the wearable device 400 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of hardware and software.

[0134] To better understand the embodiments of the present application, the following uses a mobile phone as a terminal device and a headset as a wearable device as an example, and illustrates the system architecture for data exchange between the terminal device and the wearable device in conjunction with Figure 5. Figure 5 is a schematic diagram of a system architecture for data exchange between a mobile phone and a headset provided in an embodiment of the present application. Referring to Figure 5, the system architecture may include a mobile phone and a headset, and a communication connection between the mobile phone and the headset.

[0135] The uplink data in Figure 5 is the IMU data collected by the IMU in the headset. IMU data can also be called the user's head rotation data or the inertial sensor data of the headset; the downlink data is the audio data transmitted and processed in the mobile phone; the IMU data channel refers to the transmission channel used in the process of transmitting the IMU data collected by the IMU in the headset to the mobile phone.

[0136] Referring to Figure 5 , a layered architecture can be displayed in a mobile phone. From bottom to top, it can be divided into the physical layer, hardware abstraction layer (HAL), framework layer, and application layer. The layers communicate with each other via software interfaces. For example, the layers communicate with each other via JNI (Java Native Interface). It is easy to understand that this layered architecture can also include other layers depending on the actual scenario, and this embodiment of the application will not be described in detail.

[0137] Among them, the physical layer can be equipped with a variety of hardware devices, such as a Bluetooth controller (BTC). The BTC can be used to receive IMU data collected by the IMU transmitted by the headset, and forward the received IMU data to the HAL layer so that the application in the HAL layer can transmit and process the received IMU.

[0138] For example, a BTC protocol stack composed of multiple protocols may also be embedded in the BTC. After receiving the IMU data transmitted by the headset, the BTC may encapsulate the received IMU data using a protocol in the BTC protocol stack to transmit the encapsulated IMU data to the HAL layer.

[0139] It should be noted that BTC typically uses the Serial Port Profile (SPP) protocol to receive IMU data transmitted by headphones. In practical applications of the SPP protocol, a corresponding application can be configured in the HAL layer so that after receiving IMU data transmitted by the headphones, the BTC sends a response message corresponding to the received IMU data to the headphones, that is, sends corresponding confirmation information to the headphones, thereby ensuring the reliability of IMU data transmission.

[0140] For example, assuming that the headset transmits about 20 IMU data to the BTC within 200 seconds, then based on the SPP protocol, after BTC receives each of the 20 IMU data transmitted by the headset, the corresponding application in the HAL layer will send confirmation information corresponding to each IMU data to the headset. That is to say, after BTC receives 20 IMU data, the corresponding application in the HAL layer will send 20 confirmation messages to the headset.

[0141] The BTC protocol stack in BTC can be used not only to transmit IMU data from the headphones to the HAL layer, but also to transmit spatial audio data from the HAL layer to the headphones. This involves decapsulating the spatial audio data using a protocol in the BTC protocol stack, transmitting the decapsulated spatial audio data to the headphones. The spatial audio data received by the headphones is then transmitted to the speakers via a digital audio interface (e.g., an SPK interface) for playback. It should be understood that spatial audio data is audio data with a good sense of space and direction obtained after rendering by the mobile phone.

[0142] In one possible implementation, the physical layer may also include devices such as a battery and a camera (not shown in FIG5 ), which is not limited in the embodiments of the present application.

[0143] The purpose of the HAL layer is to abstract the hardware and provide a unified interface for querying hardware devices for upper-layer applications, or to provide data storage services or data processing services for upper-layer applications.

[0144] In an example, the HAL layer may include: an encoding module and a Bluetooth host (BTH) module.

[0145] Among them, the encoding module can be used to receive, process and forward the spatial audio data transmitted to the encoding module by the upper layer (rendering module in Figure 5). Processing refers to the encoding module encoding the spatial audio data to obtain the encoded spatial audio data. Among them, the encoding process can be understood as converting the audio format of the spatial audio data transmitted by the upper layer so that the converted spatial audio data conforms to the audio format of the audio data received by the next module (i.e., the BTH module) in terms of audio format. In specific implementations, the encoding module can implement the above functions by various encoders.

[0146] It should be understood that since the data received, processed and forwarded in the encoding module is audio data, in actual implementation, the Bluetooth audio transmission model protocol (Advanced Audio Distribution Profile, A2DP) protocol can be used to receive, process and forward the audio data.

[0147] Exemplarily, the audio data transmitted from the upper layer to the encoding module can be encoded and processed into audio data in any audio format such as advanced audio coding (AAC), sub-band coding (SBC), and low-latency hi-definition audio codec (LHDC).

[0148] In this example, the BTH module can not only receive and forward IMU data transmitted by the headset via BTC, that is, receive IMU data transmitted by the headset via BTC, transmit response information corresponding to the received IMU data to the headset, and forward the received IMU data to the upper-layer application. The BTH module can also be used to receive and forward audio data encoded by the encoding module, that is, the BTH module can also be used to receive audio data encoded by the encoding module and forward the received encoded audio data to the headset.

[0149] In a possible implementation, the BTH module may include a HOST protocol stack, and the protocol in the HOST protocol stack may be used to implement Bluetooth connection and control with the headset.

[0150] In a possible implementation, the HAL layer may further be provided with other modules, which is not limited in the embodiments of the present application.

[0151] The framework layer provides user program frameworks and capability frameworks in multiple languages, such as Java, C, C++, and JS, for application development in the application layer above it, two user interface (UI) frameworks (including a Java UI framework for the Java language and a JS UI framework for the JS language), and a multi-language framework application programming interface (API) open to the outside world for various software and hardware services.

[0152] In the audio processing process, the framework layer may include a decoding module and a rendering module, etc., wherein the decoding module can be used to receive the source audio data transmitted by the application layer, such as a video playback application, wherein the source audio data refers to audio data that has not been rendered; the decoding module can also be used to convert the audio format of the received source audio data. In other words, the decoding module can decode the received source audio data into audio data in the audio format required by the next module (such as a rendering module). Exemplarily, assuming that the audio format of the audio data required by the next module is audio data of independent channels, and the audio format of the source audio data received by the decoding module is stereo, then the source audio data in stereo format can be transmitted to the decoding module for decoding processing to obtain audio data with two channels after decoding processing. In a specific implementation, the decoding module can implement the above functions by various decoders.

[0153] The rendering module in the framework layer is used not only to receive and process the audio data transmitted by the decoding module, but also to receive IMU data forwarded by the application layer above, such as the audio manager application, and to render the decoded audio data based on the received IMU data to obtain rendered spatial audio data. The rendering module can also be used to transmit the rendered spatial audio data to the HAL layer (the encoding module in Figure 5).

[0154] In a possible implementation, the framework layer may also include: a window manager, a content provider, a resource manager, a view system, and a notification manager (not shown in FIG5 ), which is not limited in the embodiments of the present application.

[0155] The application layer may include a series of application programs, such as an audio manager application and a video player application.

[0156] The audio manager app can receive, process, and forward IMU data transmitted by headphones via BTC and BTH. The video player app can not only control the playback and pause of source audio data, but also set up interfaces or controls corresponding to the audio data in the playback space to enable the transmission and processing of audio data.

[0157] It is not difficult to understand that the audio manager application and the video playback application can both be system applications in the mobile phone or applications provided by a third party, and this is not limited in the embodiments of the present application.

[0158] It is understandable that the application layer may also include: music, weather, calendar, themes, mailbox, social, application download, payment, contacts, my device, album, mobile phone manager, settings, mall, connection, browser, camera, game, navigation, shopping, address book, phone, information and headphone settings and other applications (not shown in Figure 5), which are not limited in the embodiments of the present application.

[0159] It is worth noting that the implementation of the corresponding functions of each module set in the application layer, framework layer and hardware abstraction layer is inseparable from the control and execution of the main processor (i.e., CPU) in the mobile phone, and the transmission and processing of other business data in the mobile phone also require the calculation and processing of the main processor. Therefore, the system architecture for data interaction between the mobile phone and the headset described in Figure 5 is slow to transmit and process audio data, and cannot quickly achieve the effect of "sound follows the head", which affects the user's auditory experience.

[0160] Therefore, in response to the problem that current electronic devices have slow transmission and processing of head motion data, the present application provides an audio processing method. In the process of processing audio data, the main processor only needs to transmit the audio data to the auxiliary processor, and the auxiliary processor processes the audio data. This effectively avoids the slow transmission and processing of audio data caused by the main processor needing to process too much business data, further improves the processing speed of electronic devices for head motion data (i.e., inertial sensor data), quickly realizes "sound follows head movement", and to a certain extent solves the problem of slow transmission and processing of head motion data by electronic devices.

[0161] The audio processing method provided in this application is exemplarily described below with reference to specific embodiments.

[0162] FIG6 is a flow chart of an audio processing method provided by the present application. Referring to FIG6 , the method is applied to a terminal device, which is in communication with a wearable device. The terminal device includes a main processor and a secondary processor, wherein the main processor may include a CPU and the secondary processor may include a DSP chip. Referring to FIG6 , the method includes:

[0163] S601: The main processor decodes the source audio data to obtain decoded audio data.

[0164] It should be understood that in the embodiment of the present application, the main processor may include a CPU, which is mainly used to decode the source audio data and transmit the decoded audio data to the auxiliary processor.

[0165] In one possible example, the source audio data may be music audio data, video audio data, call audio data, etc. Furthermore, the source of the source audio data may include audio data played by an application pre-installed in the terminal device (such as a video application or a music application); or may include audio data after a recorded audio source file is played back, which is not limited in this embodiment of the present application.

[0166] The decoding process of the source audio data is essentially the process of using the main processor to decode the digital audio signal and restore it to an analog audio signal. In other words, the decoding process of the source audio data is actually the process of converting the audio format of the source audio data into the audio format of the audio data required by the auxiliary processor. For example, assuming that the source audio data is audio data in a 5.1-channel format (or a 7.1-channel format), the main processor can be used to decode the audio data in the 5.1-channel format (or the 7.1-channel format) into 6 (or 8) channels of audio data, where the 6 (or 8) channels of audio data are the audio data decoded and processed by the main processor.

[0167] S602: The main processor transmits the audio data to the auxiliary processor.

[0168] It should be understood that the auxiliary processor may include a DSP chip. Compared to the main processor, in the embodiment of the present application, the main function of the auxiliary processor is to receive the audio data transmitted by the main processor and render the audio data. It is understandable that in the embodiment of the present application, the main processor does not need to render the audio data after decoding, but only needs to transmit the decoded audio data to the auxiliary processor, which effectively avoids the problem of slow transmission and processing of audio data due to the main processor in the terminal device needing to process too much business data, further speeds up the processing speed of audio data, and improves the user's auditory experience.

[0169] After the main processor decodes the source audio data, it can transmit the audio data obtained after the decoding process to the auxiliary processor so that the auxiliary processor can execute the subsequent step S603 to achieve rendering processing on the decoded audio data.

[0170] S603: The auxiliary processor obtains inertial sensor data of the wearable device.

[0171] In an embodiment of the present application, after receiving the audio data transmitted by the main processor, the auxiliary processor can obtain the inertial sensor data of the wearable device, and use the obtained inertial sensor data of the wearable device to render and process the audio data to obtain spatial audio data with better spatial and directional sense.

[0172] It should be understood that inertial sensing data may also be referred to as IMU data, and IMU data may be used to represent changes in the movement (eg, posture) of the user's head.

[0173] In one possible example, the IMU data may include six-degree-of-freedom data, namely: the translation distance of the wearable device on the X-axis, Y-axis, and Z-axis, and the angle values ​​of the wearable device's rotation around the X-axis, Y-axis, and Z-axis; the IMU data may also include the translation distance of the wearable device on the X-axis, Y-axis, and Z-axis, the angle values ​​of the wearable device's rotation around the X-axis, Y-axis, and Z-axis, and the acceleration values ​​of the wearable device on the X-axis, Y-axis, and Z-axis. This application is not limited to this.

[0174] For example, a preset number (e.g., 6) of IMUs can be set in the wearable device to collect IMU data of the wearable device. It should be understood that the number of IMUs set in the wearable device can correspond to the type of chip set in the wearable device, or can be user-defined.

[0175] It is not difficult to understand that, depending on the specific content of the IMU data, different numbers and / or different types of sensors can be set in the IMU. As an example and not a limitation, an accelerometer sensor and a gyroscope sensor can be set in the IMU, and an inclinometer, accelerometer, gyroscope, and magnetometer can also be set in the IMU. The embodiments of this application do not impose any restrictions on the number of IMUs set in the wearable device, the number of sensors set in the IMU, and the specific types of sensors.

[0176] In a possible implementation, the terminal device may include a communication module, which receives inertial sensing data transmitted by the wearable device through a first protocol, wherein the first protocol may be a unidirectional transmission protocol.

[0177] It should be understood that in the above possible implementations, the specific type of the communication module may be related to the communication connection method between the terminal device and the wearable device. Correspondingly, the first protocol may also be a unidirectional transmission protocol related to the communication connection method between the terminal device and the wearable device.

[0178] Exemplarily, when the communication connection mode between the terminal device and the wearable device is a Bluetooth connection, the communication module may be a Bluetooth communication module, and correspondingly, the first protocol may be a human interface device (HID) protocol.

[0179] It is not difficult to understand that when the terminal device and the wearable device are connected through other communication methods, the communication module can be a communication module corresponding to the other communication method, and the first protocol can also be other unidirectional transmission protocols corresponding to the other communication method. This application does not limit this.

[0180] Based on the above possible implementation methods, the communication module receives the inertial sensor data transmitted by the wearable device through a one-way transmission protocol, which enables the communication module to only receive the inertial sensor data sent by the wearable device, without having to reply to the wearable device with corresponding response information regarding whether the inertial sensor data transmitted by the wearable device is received. This not only speeds up the speed at which the wearable device transmits the inertial sensor data to the communication module, but also avoids the need for the main processor (or auxiliary processor) to reply to the wearable device with response information corresponding to the received inertial sensor data, causing the main processor (or auxiliary processor) to be in an awake state all the time, further reducing the utilization rate of the main processor (or auxiliary processor) and reducing the power consumption of the main processor (or auxiliary processor).

[0181] In some embodiments, the auxiliary processor can directly receive the inertial sensor data transmitted by the communication module; or a corresponding data transmission module can be set in the terminal device so that the data transmission module receives the inertial sensor data transmitted by the communication module, and then the auxiliary processor obtains the inertial sensor data through the data transmission module.

[0182] Exemplarily, the data transmission module may be provided in a Sensor Hub, and the Sensor Hub may be used to transmit the inertial sensor data transmitted by the communication module to the auxiliary processor.

[0183] Based on the two possible implementations described above, the transmission paths for the auxiliary processor to obtain inertial sensor data can be enriched. Furthermore, using a data transmission module independent of the auxiliary processor to transmit inertial sensor data can further accelerate the transmission speed of inertial sensor data to the auxiliary processor, increase the auxiliary processor's processing efficiency of inertial sensor data, and further reduce the transmission and processing latency of audio data.

[0184] S604: The auxiliary processor renders the audio data according to the inertial sensor data to obtain spatial audio data.

[0185] It should be understood that after the auxiliary processor obtains the inertial sensor data of the wearable device, it can use the inertial sensor data to render the decoded audio data to obtain spatial audio data with better spatial and directional sense.

[0186] Exemplarily, during the actual rendering process, the auxiliary processor may adopt at least one of the following algorithms to obtain spatial audio data: head related transfer function (HRTF), a positioning method based on controllable beamforming with maximum output power, a positioning method based on high-resolution spectrum estimation, a positioning method based on time difference of arrival (TDOA) estimation, and a method based on machine learning, etc.

[0187] In one possible implementation, after acquiring the spatial audio data, an auxiliary processor may be used to encode the spatial audio data to obtain encoded spatial audio data. Subsequently, the auxiliary processor transmits the encoded spatial audio data to the wearable device through the communication module, thereby allowing the user to listen to spatial audio data with a better sense of space and direction through the wearable device.

[0188] It should be understood that the encoding process corresponds to the decoding process. The encoding process is essentially the process of using the auxiliary processor to convert the analog audio signal into a digital audio signal. In other words, it is the process of converting the rendered spatial audio data into an audio format that can be received by the wearable device.

[0189] It is not difficult to understand that in actual applications, after the wearable device receives the encoded spatial audio data transmitted by the auxiliary processor through the communication module, a corresponding decoder or decoding module can be set in the wearable device to decode the spatial audio data transmitted to the wearable device, and obtain the decoded spatial audio data, which is convenient for playback through the speakers set in the wearable device.

[0190] The following takes the main processor as a CPU, the auxiliary processor as a DSP chip, the terminal device as a mobile phone, and the wearable device as a Bluetooth headset as an example, and combines the inertial sensor data (i.e., IMU data below) shown in Figure 7 to illustrate the audio processing method provided in the embodiment of the present application. Referring to Figure 7, the mobile phone and the Bluetooth headset are connected via Bluetooth communication. The layered architecture can still be displayed in the mobile phone. The specific layered architecture in the mobile phone can refer to the relevant description in the embodiment described in Figure 5 above, which will not be repeated here. Referring to Figure 7, the method may include:

[0191] In step 701, the Bluetooth headset transmits the collected IMU data to the DSP chip via BTC.

[0192] It should be understood that at least one IMU can be set in the Bluetooth headset to collect IMU data. The specific settings of the IMU can refer to the relevant description in the embodiment shown in Figure 6 above, and will not be repeated here.

[0193] Exemplarily, the DSP chip may include an IMU data transmission module, an encoding module and a rendering module, wherein the IMU data transmission module can be used to receive IMU data transmitted by BTC and transmit the received IMU data to the rendering module; the encoding module can be used to encode the spatial audio data output by the rendering module to transmit the encoded spatial audio data to the Bluetooth headset via BTC.

[0194] The rendering module can be used to receive audio data transmitted by the CPU after being decoded by the decoding module and IMU data transmitted by the IMU data transmission module, and render the decoded audio data based on the received IMU data to obtain rendered spatial audio data. Compared with the decoded audio data, the spatial audio data has a greater sense of space and direction. The rendering module can also be used to transmit the rendered spatial audio data to the encoding module so that the encoding module can encode the spatial audio data.

[0195] In a possible implementation, the Bluetooth headset transmits IMU data to the DSP via a first protocol, where the first protocol may be a unidirectional Bluetooth protocol.

[0196] In some embodiments, the first protocol may be a HID protocol. It should be understood that the HID protocol is a one-way Bluetooth protocol. In actual applications, after the Bluetooth headset transmits IMU data to the DSP via the HID protocol, the DSP does not need to send response information corresponding to the received IMU data to the Bluetooth headset. That is, in the process of the DSP receiving the IMU data sent by the Bluetooth headset via the HID protocol, the DSP is mainly used for the IMU data transmitted by the Bluetooth headset via BTC. After the DSP receives the IMU data, it does not need to send corresponding confirmation information to the Bluetooth headset for each piece of IMU data received. In this way, since the DSP does not need to send corresponding confirmation information to the Bluetooth headset, the power consumption of the DSP is further reduced, the transmission speed of the IMU data is accelerated, and the delay in audio data transmission and processing is reduced.

[0197] Step 702: The CPU decodes the source audio data to obtain decoded audio data.

[0198] It should be understood that the CPU, as the main processor, can realize the transmission and processing of data between the application layer, framework layer and hardware abstraction layer in the mobile phone by running the application program.

[0199] In some examples, the application layer of the mobile phone may include a video player application, which may be used to control the playback and pause of source audio data. Of course, in other possible implementations, the application layer of the mobile phone may also include other applications, such as a music application.

[0200] In some examples, the framework layer of a mobile phone may include a decoding module that can be used to receive source audio data transmitted by a video playback application in the application layer of the mobile phone, decode the received source audio data to obtain decoded audio data, and then forward the decoded audio data to the audio data transmission module.

[0201] In some examples, the hardware abstraction layer of the mobile phone may include an audio data transmission module, which can be used to receive decoded audio data transmitted by the decoding module; and can also be used to transmit the received audio data to the DSP.

[0202] Step 703: The CPU transmits the decoded audio data to the DSP.

[0203] It should be understood that in order to reduce the energy consumption of the CPU and further release the computing power of the CPU, in an embodiment of the present application, the CPU is mainly used to decode the source audio data and transmit the decoded audio data to the DSP, so as to use the DSP to realize the rendering of the decoded audio data.

[0204] In step 704 , the DSP renders the audio data transmitted by the CPU according to the received IMU data to obtain spatial audio data.

[0205] It should be understood that the DSP may include an IMU data transmission module and a rendering module. The IMU data transmission module can be used to receive and forward IMU data transmitted by the Bluetooth headset via BTC; the rendering module can be used to render and process the IMU data and audio data to obtain spatial audio data. The specific process of the DSP rendering audio data based on the IMU data can be found in the corresponding description of the embodiment shown in Figure 6 above and will not be repeated here.

[0206] After the DSP renders the received IMU data and audio data, it may also transmit the rendered spatial audio data to the Bluetooth headset. Therefore, after step 704, the method may further include:

[0207] Step 705: The DSP transmits the spatial audio data to the Bluetooth headset via the BTC.

[0208] It should be understood that in the embodiment of the present application, the DSP may also include an encoding module; the rendering module in the DSP can transmit the rendered spatial audio data to the encoding module; the encoding module can be used to receive the spatial audio data transmitted by the rendering module, and can also be used to encode the spatial audio data to obtain the encoded spatial audio data; it can also be used to forward the encoded spatial audio data to the Bluetooth headset through the BTC, so that the Bluetooth headset receives the encoded spatial audio data.

[0209] In one example, the encoding module can use the A2DP protocol in the BTC protocol stack to forward the encoded spatial audio data to the Bluetooth headset. It should be understood that using the A2DP protocol can effectively improve the transmission reliability of the encoded spatial audio data and reduce energy consumption of the mobile phone and Bluetooth headset.

[0210] Step 706: The Bluetooth headset receives the spatial audio data transmitted by the DSP.

[0211] It should be understood that the specific process of the Bluetooth headset receiving the spatial audio data transmitted by the DSP can be referred to the relevant description in the embodiment shown in FIG6 , which will not be repeated here.

[0212] In another possible implementation, the IMU data transmission module in the DSP in the embodiment shown in FIG7 can also be decoupled from the DSP to reduce the processing steps of the DSP and further reduce the power consumption of the DSP. For example, the IMU data transmission module can be set in the sensor hub (SensorHub) to use the IMU data transmission module set in the SensorHub to receive and transmit the IMU data transmitted by the Bluetooth headset via BTC. As shown in FIG8, a structural diagram of another inertial sensor data flow provided in an embodiment of the present application is shown. Referring to FIG8, the method can also include:

[0213] In step 801, the Bluetooth headset transmits the collected IMU data to the DSP via BTC.

[0214] Step 802: The CPU decodes the source audio data to obtain decoded audio data.

[0215] Step 803: The CPU transmits the decoded audio data to the DSP.

[0216] Step 804: The Bluetooth headset transmits IMU data to the sensor hub (SensorHub) via BTC.

[0217] For example, the Sensor Hub may include an IMU data transmission module, which may be used to receive and forward IMU data transmitted by the Bluetooth headset via BTC. Furthermore, the IMU data transmission module may transmit the received IMU data to the DSP.

[0218] Step 805: SensorHub transmits IMU data to DSP.

[0219] It should be understood that in this embodiment, the DSP may include a rendering module, which can be used to receive IMU data transmitted by the Sensor Hub and also to receive decoded audio data transmitted by the CPU. In this way, the DSP can directly receive IMU data and decoded audio data from the Bluetooth headset, and use the IMU data to render the audio data, thereby quickly obtaining rendered spatial audio data.

[0220] In step 806 , the DSP renders the audio data transmitted by the CPU according to the received IMU data to obtain spatial audio data.

[0221] Step 807: The DSP transmits the spatial audio data to the Bluetooth headset via the BTC.

[0222] Step 808: The Bluetooth headset receives the spatial audio data transmitted by the DSP.

[0223] It should be understood that the above steps 801 to 803 and steps 806 to 808 can refer to the relevant descriptions of steps 701 to 706 in the embodiment described in FIG7 , and are not repeated here.

[0224] Based on the above implementation, the transmission path of IMU data can be enriched. Compared with the transmission of IMU data by DSP, the above transmission of IMU data through SensorHub can not only reduce the processing steps of DSP and further reduce the power consumption of DSP, but also speed up the processing speed of IMU data and audio data, quickly obtain the rendered spatial audio data, and effectively reduce the processing delay of audio data.

[0225] It should be noted that, taking the terminal device as a mobile phone and the wearable device as a TWS Bluetooth headset as an example, when the mobile phone is in the same operating state, experiments are carried out using 7.1-channel source audio data based on the embodiments shown in Figure 5 and 7, respectively. It is found that after the 7.1-channel source audio data is processed using the embodiment shown in Figure 5, the power consumption of the mobile phone increases by 63 milliamperes (mA); compared with the embodiment shown in Figure 5, after the 7.1-channel source audio data is processed using the embodiment shown in Figure 7, the power consumption of the mobile phone can be reduced by 30mA. It can be seen that the audio processing method provided in the embodiment of the present application can effectively reduce the overall power consumption of the terminal device.

[0226] In addition, still taking the terminal device as a mobile phone and the wearable device as a TWS Bluetooth headset as an example, the delay test is performed based on the embodiment shown in Figure 5 and the embodiment shown in Figure 7, respectively. It can be obtained that: using the embodiment shown in Figure 5, after the user wears the TWS Bluetooth headset, the time consumed from the user's head starting to rotate until the corresponding spatial audio data is played in the TWS Bluetooth headset is about 215 milliseconds (ms); while the time consumed by the embodiment shown in Figure 7 is about 185ms. Compared with the embodiment shown in Figure 5, the audio processing method provided in the embodiment of the present application can effectively reduce the delay of about 30ms. It can be seen that the audio processing method provided in the embodiment of the present application can effectively reduce the transmission and processing delay of audio data.

[0227] In summary, the audio processing method provided in the embodiment of the present application renders and processes the audio data according to the inertial sensor data of the wearable device obtained by the auxiliary processor to obtain spatial audio data, effectively avoiding the situation where the transmission and processing of audio data is slow due to the excessive business data that the main processor needs to process. The processing of audio data by the auxiliary processor not only improves the processing speed of the electronic device for inertial sensor data and quickly realizes "sound follows head movement"; but also, compared with transmitting the inertial sensor data to the main processor for processing the inertial sensor data by the main processor, it can effectively reduce the transmission path of the inertial sensor data and further reduce the transmission delay of the audio data, thereby solving the problem of slow transmission and processing of head motion data by the electronic device, allowing users to hear smoother spatial audio, improving the user experience of the spatial audio function, and improving the user's listening experience.

[0228] Furthermore, using the auxiliary processor to process audio data can also address the issue of excessive power consumption by the main processor due to the large amount of data it must process. It can also further reduce the main processor's continuous utilization, freeing up its computing power, preventing the main processor from preempting it, and further minimizing the impact on audio data transmission and processing. For example, if a request to process other business data with a higher priority than the audio data is received while the main processor is processing audio data, the main processor may prioritize processing the other business data, thereby affecting the main processor's ability to transmit and process audio data.

[0229] It should be understood that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the order of execution of the named or numbered process steps can also be changed according to the technical objectives to be achieved, as long as the same or similar technical effects can be achieved.

[0230] Based on the audio processing methods provided in the above embodiments, the embodiments of the present application also provide the following content:

[0231] An embodiment of the present application provides an electronic device, which includes a main processor and a secondary processor, and is configured to execute the audio processing method shown in the above embodiments. For example, the structure of the electronic device can be as shown in FIG3 above.

[0232] An embodiment of the present application provides a computer program product, which includes a program. When the program is executed by an electronic device, the electronic device implements the audio processing method shown in the above embodiments.

[0233] An embodiment of the present application also provides a communication system, which includes: the above-mentioned terminal device and / or the above-mentioned wearable device.

[0234] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the audio processing method shown in the above embodiments is implemented.

[0235] An embodiment of the present application provides a chip, which includes a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory, so that the chip executes the steps performed by the auxiliary processor in the audio processing method shown in each of the above embodiments.

[0236] An embodiment of the present application provides a chip system, which includes: a main processor and an auxiliary processor. The chip system is used to call a program from a memory so that a device equipped with the chip system executes the audio processing method shown in the above embodiments.

[0237] It should be understood that the main processor and / or auxiliary processor mentioned in the embodiments of the present application may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0238] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0239] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application. In the above embodiments, the description of each embodiment has its own focus. For the parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0240] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0241] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0243] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to a large-screen device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0245] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An audio processing method, characterized in that: The method is applied to a terminal device, the terminal device is communicatively connected with a wearable device, the terminal device includes a main processor and an auxiliary processor, and the method includes: The main processor decodes the source audio data to obtain decoded audio data; The main processor transmits the audio data to the auxiliary processor; The auxiliary processor obtains inertial sensing data of the wearable device; The auxiliary processor performs rendering processing on the audio data according to the inertial sensor data to obtain spatial audio data.

2. The method according to claim 1, characterized in that The terminal device further includes a communication module, and the method further includes: The communication module receives the inertial sensing data transmitted by the wearable device through a first protocol, where the first protocol is a unidirectional transmission protocol; The auxiliary processor obtains inertial sensing data of the wearable device, including: The auxiliary processor obtains the inertial sensing data through the communication module.

3. The method according to claim 2, characterized in that The auxiliary processor obtains the inertial sensing data through the communication module, including: The auxiliary processor receives the inertial sensing data transmitted by the communication module.

4. The method according to claim 2, characterized in that: The terminal device further includes a data transmission module, and the auxiliary processor obtains the inertial sensor data through the communication module, including: The data transmission module receives the inertial sensing data transmitted by the communication module; The auxiliary processor obtains the inertial sensing data through the data transmission module.

5. The method according to any one of claims 2 to 4, characterized in that The method further comprises: The auxiliary processor encodes the spatial audio data to obtain the encoded spatial audio data; The auxiliary processor transmits the encoded spatial audio data to the wearable device through the communication module.

6. The method according to any one of claims 2 to 5, characterized in that The first protocol includes a Human Interface Device (HID) protocol.

7. The method according to any one of claims 1 to 6, characterized in that The auxiliary processor includes a digital signal processing DSP chip.

8. A chip, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory, so that the chip executes the method executed by the auxiliary processor in any one of claims 1 to 7.

9. A chip system, characterized in that: include: A main processor and an auxiliary processor, the chip system is used to call a program from a memory so that a device equipped with the chip system executes a method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A main processor and an auxiliary processor, the electronic device is used to execute a computer program or instruction stored in a memory, so that the electronic device implements the method as claimed in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images