Audio processing method, chip and electronic equipment

Abstract

The invention provides an audio processing method, a chip and electronic equipment, relates to the technical field of electronics, and solves the problem that the transmission and processing process of head motion data by the electronic equipment is slow to a certain extent. The method is applied to a terminal device, the terminal device is in communication connection with a wearable device, the terminal device comprises a main processor and an auxiliary processor, and the method comprises the following steps: the main processor decodes source audio data to obtain decoded audio data; the main processor transmits the audio data to the auxiliary processor; the auxiliary processor obtains inertial sensing data of the wearable device; and the auxiliary processor performs rendering processing on the audio data according to the inertial sensing data to obtain spatial audio data.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an audio processing method, a chip and an electronic device. Background Art

[0002] With the rapid development of wearable devices, the functions of products are becoming more and more abundant, such as active noise reduction, call noise reduction, spatial audio, wireless charging, etc. Among them, the spatial audio function can also be called the 3D audio effect function, which can capture the user's head movements in real time through the wearable device, and then realize the dynamic tracking of the sound field according to the captured user's head movements, creating an audio effect of sound surrounding the head, achieving an immersive listening experience.

[0003] The realization of spatial audio function depends on the rapid transmission and processing of head motion data by electronic devices. If the transmission and processing of head motion data by electronic devices is slow, the user's head will have changed relative to the spatial position, and then the sound will change laggingly according to the change of the head's relative spatial position, affecting the user's immersive experience and reducing the user's experience of spatial audio function. Summary of the invention

[0004] The present application provides an audio processing method, chip and electronic device, which to a certain extent solve the problem of slow transmission and processing of head motion data by electronic devices, and improve the user experience of spatial audio functions.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides an audio processing method, which is applied to a terminal device, which is communicatively connected to a wearable device, and the terminal device includes a main processor and an auxiliary processor. The method includes: the main processor decodes the source audio data to obtain decoded audio data; the main processor transmits the audio data to the auxiliary processor; the auxiliary processor obtains inertial sensor data of the wearable device; the auxiliary processor renders the audio data according to the inertial sensor data to obtain spatial audio data.

[0007] Based on the audio processing method provided in the embodiment of the present application, in the process of processing audio data, the auxiliary processor renders the audio data according to the inertial sensor data of the wearable device obtained to obtain spatial audio data, which effectively avoids the slow transmission and processing of audio data caused by the main processor needing to process too much business data. The processing speed of the electronic device for inertial sensor data (i.e., the user's head motion data) is improved by processing the audio data through the auxiliary processor, and "sound follows the head movement" is quickly realized, which effectively solves the problem of slow transmission and processing of head motion data by the electronic device.

[0008] In addition, using the auxiliary processor to process audio data can also solve the problem of excessive power consumption of the main processor due to the large amount of data that the main processor needs to process; it can also further reduce the continuous usage rate of the main processor, release the computing power of the main processor, avoid the situation where the main processor is preempted, and further reduce the impact on the transmission and processing of audio data. For example, when the main processor is processing audio data, there is a request to process other business data with a higher priority than audio data. The main processor may give priority to processing other business data with a higher priority, which in turn affects the transmission and processing of audio data by the main processor.

[0009] In the embodiment of the present application, the main processor may include a CPU; the auxiliary processor may include a DSP.

[0010] Exemplarily, the terminal device may include a mobile phone, a PC, a smart phone, a netbook, a tablet computer, a smart camera, a PDA, a smart TV, etc. Wearable devices include but are not limited to headphones, smart glasses or smart helmets, etc.

[0011] In one example, the source audio data may be music audio data, video audio data, call audio data, etc. Further, the source of the source audio data may include audio data played in an application pre-installed in the terminal device; it may also include audio data after a playback operation is performed on a recorded audio source file, and the present application embodiment does not impose any limitation on this.

[0012] In a possible implementation, the inertial sensing data may include the translation distance and angular velocity data of the wearable device; it may also include the translation distance, acceleration data, and angular velocity data of the wearable device, etc. Among them, the inertial sensing data can be obtained by an IMU set in the wearable device. For example, a preset number of IMUs can be set in the wearable device, and the IMU may include an accelerometer sensor and a gyroscope sensor to obtain the inertial sensing data (i.e., IMU data) of the wearable device by using the accelerometer sensor and the gyroscope sensor. In this possible implementation, the number of IMUs set in the wearable device may vary depending on the type of chip set in the wearable device. The embodiments of the present application do not limit the specific content of the inertial sensing data, the number of IMUs set in the wearable device, and the type and number of specific sensors set in the IMU.

[0013] In a possible implementation of the first aspect, the terminal device further includes a communication module, and the method further includes:

[0014] The communication module receives the inertial sensor data transmitted by the wearable device through a first protocol, where the first protocol is a unidirectional transmission protocol;

[0015] The auxiliary processor obtains the inertial sensing data of the wearable device, including: the auxiliary processor obtains the inertial sensing data through the communication module.

[0016] It should be understood that the communication module receives the inertial sensing data transmitted by the wearable device through the first protocol, that is, the wearable device transmits the inertial sensing data to the communication module through the first protocol.

[0017] In the embodiment of the present application, the first protocol may include a HID protocol.

[0018] Exemplarily, the communication module may be a Bluetooth communication module, which may specifically be a Bluetooth control chip BTC.

[0019] Based on the above possible implementation methods, compared to transmitting the inertial sensor data to the main processor and using the main processor to process the inertial sensor data, the communication module receives the inertial sensor data transmitted by the wearable device through a unidirectional transmission protocol, so that the communication module only needs to receive the inertial sensor data sent by the wearable device, and there is no need to reply to the wearable device with corresponding response information whether the inertial sensor data transmitted by the wearable device is received. This can effectively reduce the transmission path of the inertial sensor data, further reduce the transmission delay of the audio data, so that the user can hear smoother spatial audio, improve the user's experience of the spatial audio function, and improve the user's auditory experience.

[0020] In addition, through the above-mentioned possible implementation methods, not only the speed of transmitting inertial sensor data from the wearable device to the communication module is accelerated, but also the main processor (or auxiliary processor) is avoided from needing to reply to the wearable device with response information corresponding to the received inertial sensor data, so that the main processor (or auxiliary processor) is always in an awake state, further reducing the utilization rate of the main processor (or auxiliary processor) and reducing the power consumption of the main processor (or auxiliary processor).

[0021] In a possible implementation of the first aspect, the auxiliary processor acquires the inertial sensor data through the communication module, including: the auxiliary processor receives the inertial sensor data transmitted by the communication module. That is, the auxiliary processor directly receives the inertial sensor data transmitted by the communication module. In actual design, a corresponding transmission module may also be provided in the auxiliary processor to use the transmission module to receive the inertial sensor data transmitted by the Bluetooth communication module.

[0022] In a possible implementation manner of the first aspect, the terminal device further includes a data transmission module, and the auxiliary processor acquires the inertial sensor data through the communication module, including:

[0023] The data transmission module receives the inertial sensor data transmitted by the communication module;

[0024] The auxiliary processor obtains inertial sensor data through the data transmission module.

[0025] Exemplarily, the data transmission module may be provided in a Sensor Hub, and the Sensor Hub may be used to transmit the inertial sensor data transmitted by the communication module to the auxiliary processor.

[0026] Based on the above possible implementation methods, not only can the transmission path for the auxiliary processor to obtain inertial sensor data be enriched, but also the transmission speed of the inertial sensor data can be further accelerated by transmitting the inertial sensor data through a data transmission module other than the auxiliary processor in the terminal device, thereby accelerating the processing efficiency of the inertial sensor data by the auxiliary processor and further reducing the transmission processing delay of the audio data.

[0027] In a possible implementation of the first aspect, the method further includes:

[0028] The auxiliary processor encodes the spatial audio data to obtain encoded spatial audio data;

[0029] The auxiliary processor transmits the encoded spatial audio data to the wearable device through the communication module.

[0030] In a possible implementation of the first aspect, the first protocol includes a HID protocol.

[0031] In a possible implementation of the first aspect, the auxiliary processor includes a DSP chip.

[0032] In a second aspect, an embodiment of the present application provides a chip, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, so that the chip executes the method executed by the auxiliary processor in any possible implementation of the first aspect.

[0033] In a third aspect, an embodiment of the present application provides a chip system, the chip system comprising: a main processor and an auxiliary processor, the chip system being used to call a program from a memory so that a device equipped with the chip system executes the method described in any possible implementation of the first aspect.

[0034] In a fourth aspect, the present application provides an electronic device, comprising: a main processor and an auxiliary processor, wherein the electronic device is used to execute a computer program or instruction stored in a memory, so that the electronic device executes the method described in any possible implementation manner of the first aspect.

[0035] In a possible implementation of the fourth aspect, the main processor is used to perform the following steps: decoding the source audio data to obtain decoded audio data; transmitting the audio data to the auxiliary processor;

[0036] The auxiliary processor is used to perform the following steps: obtaining inertial sensor data of the wearable device; rendering and processing the audio data according to the inertial sensor data to obtain spatial audio data.

[0037] In a fifth aspect, the present application provides a communication system, which includes: a terminal device and / or a wearable device.

[0038] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in any possible implementation manner of the first aspect is implemented.

[0039] In a seventh aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method in any possible implementation manner of the first aspect.

[0040] The technical effects of the second to seventh aspects provided in the present application can refer to the technical effects of the various possible implementation methods of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1A schematic diagram of a scenario corresponding to an audio processing method provided in an embodiment of the present application.

[0042] Figure 2 A schematic diagram of a scenario in which a user hears spatial audio data provided in an embodiment of the present application.

[0043] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0044] Figure 4 A schematic diagram of the hardware structure of an earphone provided in an embodiment of the present application.

[0045] Figure 5 A schematic diagram of the system architecture of a terminal device and headphones provided in an embodiment of the present application.

[0046] Figure 6 A flowchart of an audio processing method provided in an embodiment of the present application.

[0047] Figure 7 A structural schematic diagram of an inertial sensor data flow provided in an embodiment of the present application.

[0048] Figure 8 A structural schematic diagram of another inertial sensor data flow provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings and related embodiments in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a "or" relationship.

[0050] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0051] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0052] It can be understood that the audio processing method provided in the embodiment of the present application can be applied to the following scenarios: the terminal device is communicatively connected with the wearable device, the wearable device is worn on the user's head, the wearable device can be used to collect the user's head rotation data, and send the collected user's head rotation data to the terminal device through the communication connection, the terminal device uses the received user's head rotation data to dynamically track the sound field to obtain spatial audio data, and then transmits the spatial audio data to the wearable device through the communication connection, so that the user can listen to the spatial audio data through the wearable device.

[0053] like Figure 1 FIG. 1 is a schematic diagram of a scene corresponding to an audio processing method provided in an embodiment of the present application, see Figure 1 , assuming that the terminal device is a mobile phone 100, the wearable device is a true wireless stereo (TWS) headset 200, and the mobile phone 100 is connected to the TWS headset 200 for communication. In order to enable the user to listen to audio with a good sense of space and direction through the TWS headset 200, the mobile phone 100 can capture the user's head rotation data through the WS headset 200 worn on the head of the user, and then process the head rotation data to achieve dynamic tracking of the sound field, creating an audio effect of sound surrounding the head, achieving an immersive listening experience, see Figure 2That is to say, if the sound source is fixed, the sound effects heard by the two ears should be different when the user's head turns.

[0054] For example, a user is wearing TWS headphones and enjoying a musical instrument. Assuming that the user can determine that the instrument is located directly in front of him, during this process, the volume of the sound heard by the user's two ears is roughly the same. If the user turns his head to the right, then corresponding to the real auditory environment, since the user's left ear is closer to the location of the instrument, the volume of the sound heard by the left ear should be louder than the volume of the sound heard by the right ear.

[0055] In some embodiments, the audio processing method provided in the embodiments of the present application can be applied to other scenarios in addition to the scenario in which the terminal device and the wearable device are communicated and connected. As an example and not a limitation, assuming that the wearable device (such as smart glasses) itself has the function of processing the user's head rotation data by the terminal device, the wearable device can directly use the user's head rotation data collected by itself to dynamically track the sound field, and play the spatial audio data after dynamic tracking processing. For example, the terminal device can collect its own inertial sensor data to realize dynamic tracking processing of the sound field, and play the spatial audio data after dynamic tracking processing through the built-in speaker of the terminal device. Alternatively, the terminal device sends the audio data after dynamic tracking processing to an audio playback device (for example, a speaker) to use the audio playback device to play the spatial audio data after dynamic tracking processing, so that the audio heard by the user has a better sense of space, etc. The embodiments of the present application do not impose any restrictions on the application scenarios of the audio processing method.

[0056] It should be understood that "head rotation data", "inertial sensor data" and "IMU data" all refer to the same content. For the convenience of description, one of the descriptions is used in different application scenarios, and the three can be interchangeable.

[0057] Based on the above possible application scenarios, as examples and not limitations, the terminal device may include but is not limited to a personal computer (PC), a smart phone, a netbook, a tablet computer, a smart camera, a wearable device, a PDA, a smart TV, a personal digital assistant (PDA), a portable multimedia player (PMP), a projection device, a smart screen device, an augmented reality (AR) / virtual reality (VR) device, a mixed reality (MR) device, a vehicle-mounted device, a smart screen, a cloud server, a television or a somatosensory game console in a human-computer interaction scenario, etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0058] Exemplarily, the wearable device may be any one of headphones, smart glasses, smart helmets, AR devices, VR devices, and MR devices. According to the actual application scenario, the wearable device may be a device in the above-mentioned terminal devices. Of course, the wearable device or terminal device may also be other devices for future technologies. The embodiments of the present application do not impose any restrictions on the specific types of wearable devices.

[0059] like Figure 3 FIG. 3 is a schematic diagram of the structure of an electronic device 300 provided by the present application. The electronic device 300 may include the terminal device in the above embodiment, see Figure 3 The electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 331, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc. The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0060] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0061] For example, when the electronic device 300 is a mobile phone or a tablet computer, it may include all the components shown in the figure, or may include only some of the components shown in the figure.

[0062] The processor 310 may include a main processor and one or more auxiliary processors. For example, the main processor may be a central processing unit (CPU), and at least one auxiliary processor may be a digital signal processor (DSP), etc. In other possible implementations, the processor 310 may also include other processing units, such as an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0063] The controller may be the nerve center and command center of the electronic device 300. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 310 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 310 may include multiple groups of I2C buses. The processor 310 may be coupled to the touch sensor 380K, the charger, the flash, the camera 393, etc. through different I2C bus interfaces. For example: the processor 310 may be coupled to the touch sensor 380K through the I2C interface, so that the processor 310 communicates with the touch sensor 380K through the I2C bus interface, thereby realizing the touch function of the electronic device 300.

[0067] The I2S interface can be used for audio communication. In some embodiments, the processor 310 can include multiple groups of I2S buses. The processor 310 can be coupled to the audio module 370 via the I2S bus to achieve communication between the processor 310 and the audio module 370. In some embodiments, the audio module 370 can transmit audio signals to the wireless communication module 360 ​​via the I2S interface.

[0068] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 370 and the wireless communication module 360 ​​can be coupled via a PCM bus interface.

[0069] In some embodiments, the audio module 370 may also transmit audio signals to the wireless communication module 360 ​​via the PCM interface. Both the I2S interface and the PCM interface may be used for audio communication.

[0070] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between parallel communication and parallel communication.

[0071] In some embodiments, the UART interface is generally used to connect the processor 310 and the wireless communication module 360. For example, the processor 310 communicates with the Bluetooth module in the wireless communication module 360 ​​through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 370 can transmit audio signals to the wireless communication module 360 ​​through the UART interface to implement the function of playing music through a Bluetooth headset.

[0072] The MIPI interface can be used to connect the processor 310 with peripheral devices such as the display screen 394 and the camera 393. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 310 and the camera 393 communicate via the CSI interface to realize the shooting function of the electronic device 300. The processor 310 and the display screen 394 communicate via the DSI interface to realize the display function of the electronic device 300.

[0073] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 310 with the camera 393, the display screen 394, the wireless communication module 360, the audio module 370, the sensor module 380, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0074] The USB interface 330 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 330 can be used to connect a charger to charge the electronic device 300, and can also be used to transfer data between the electronic device 300 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0075] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0076] The charging management module 340 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 may receive charging input from a wired charger through the USB interface 330. In some wireless charging embodiments, the charging management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 300. While the charging management module 340 is charging the battery 342, it may also power the electronic device through the power management module 341.

[0077] The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, and supplies power to the processor 310, the internal memory 331, the external memory interface 320, the display screen 394, the camera 393, and the wireless communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle number, and battery health status (leakage, impedance).

[0078] In some other embodiments, the power management module 341 may also be disposed in the processor 310. In some other embodiments, the power management module 341 and the charging management module 340 may also be disposed in the same device.

[0079] The wireless communication function of the electronic device 300 can be implemented through the antenna 1, the antenna 2, the mobile communication module 350, the wireless communication module 360, the modem processor and the baseband processor.

[0080] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0081] The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 300. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0082] In some embodiments, at least some functional modules of the mobile communication module 350 may be disposed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be disposed in the same device.

[0083] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 370A, a receiver 370B, etc.), or displays an image or video through a display screen 394. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 310 and be set in the same device as the mobile communication module 350 or other functional modules.

[0084] The wireless communication module 360 ​​can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 300. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 310. The wireless communication module 360 ​​can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0085] In some embodiments, the antenna 1 of the electronic device 300 is coupled to the mobile communication module 350, and the antenna 2 is coupled to the wireless communication module 360, so that the electronic device 300 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0086] The electronic device 300 implements the display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0087] The display screen 394 is used to display images, videos, etc. For example, the first lighting diagram and the second lighting diagram in the embodiment of the present application. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0088] The electronic device 300 can realize the shooting function through ISP, camera 393, video codec, GPU, display screen 394 and application processor.

[0089] The ISP is used to process the data fed back by the camera 393. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 393.

[0090] Camera 393 is used to capture still images or videos. The object generates an optical image through the lens and is projected onto the photosensitive element. The focal length of the lens can be used to indicate the field of view of the camera. The smaller the focal length of the lens, the larger the field of view of the lens. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV and other formats.

[0091] In the present application, the electronic device 300 may include cameras 393 with 2 or more focal lengths.

[0092] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 300 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0093] Video codecs are used to compress or decompress digital videos. The electronic device 300 may support one or more video codecs. Thus, the electronic device 300 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, MPEG4, etc.

[0094] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 300 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0095] In an embodiment of the present application, the NPU or other processors may be used to perform operations such as analyzing and processing images in a video stored in the electronic device 300.

[0096] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0097] The internal memory 331 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 310 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the internal memory 331. The internal memory 331 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 300 (such as audio data, a phone book, etc.).

[0098] In addition, the internal memory 331 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0099] The electronic device 300 can implement audio functions through an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, and an application processor.

[0100] The audio module 370 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be arranged in the processor 310, or some functional modules of the audio module 370 can be arranged in the processor 310.

[0101] The speaker 370A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 300 can listen to music or listen to hands-free calls through the speaker 370A. For example, the speaker can play the comparison analysis results provided in the embodiment of the present application.

[0102] The receiver 370B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 300 receives a call or voice message, the voice can be received by placing the receiver 370B close to the human ear.

[0103] Microphone 370C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 370C to input the sound signal into the microphone 370C. The electronic device 300 can be provided with at least one microphone 370C. In other embodiments, the electronic device 300 can be provided with two microphones 370C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 300 can also be provided with three, four or more microphones 370C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0104] The earphone interface 370D is used to connect a wired earphone and can be a USB interface 330 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0105] The pressure sensor 380A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 380A can be set on the display screen 394. There are many types of pressure sensors 380A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 380A, the capacitance between the electrodes changes. The electronic device 300 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 394, the electronic device 300 detects the intensity of the touch operation based on the pressure sensor 380A. The electronic device 300 can also calculate the position of the touch based on the detection signal of the pressure sensor 380A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0106] The gyro sensor 380B can be used to determine the motion posture of the electronic device 300. In some embodiments, the angular velocity of the electronic device 300 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 380B. The gyro sensor 380B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 380B detects the angle of the electronic device 300 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 300 through reverse movement to achieve anti-shake. The gyro sensor 380B can also be used for navigation and somatosensory game scenes.

[0107] The air pressure sensor 380C is used to measure air pressure. In some embodiments, the electronic device 300 calculates the altitude through the air pressure value measured by the air pressure sensor 380C to assist in positioning and navigation.

[0108] The magnetic sensor 380D includes a Hall sensor. The electronic device 300 can use the magnetic sensor 380D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 300 is a flip phone, the electronic device 300 can detect the opening and closing of the flip cover according to the magnetic sensor 380D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0109] The acceleration sensor 380E can detect the magnitude of the acceleration of the electronic device 300 in all directions (generally three axes). When the electronic device 300 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0110] The distance sensor 380F is used to measure the distance. The electronic device 300 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 300 can use the distance sensor 380F to measure the distance to achieve fast focusing.

[0111] The proximity light sensor 380G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 300 emits infrared light outward through the light emitting diode. The electronic device 300 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 300. When insufficient reflected light is detected, the electronic device 300 can determine that there is no object near the electronic device 300. The electronic device 300 can use the proximity light sensor 380G to detect that the user holds the electronic device 300 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 380G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0112] The ambient light sensor 380L is used to sense the ambient light brightness. The electronic device 300 can adaptively adjust the brightness of the display screen 394 according to the perceived ambient light brightness. The ambient light sensor 380L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 380L can also cooperate with the proximity light sensor 380G to detect whether the electronic device 300 is in a pocket to prevent accidental touch.

[0113] The fingerprint sensor 380H is used to collect fingerprints. The electronic device 300 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0114] The temperature sensor 380J is used to detect temperature. In some embodiments, the electronic device 300 uses the temperature detected by the temperature sensor 380J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 380J exceeds a threshold, the electronic device 300 reduces the performance of the processor located near the temperature sensor 380J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 300 heats the battery 342 to avoid abnormal shutdown of the electronic device 300 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 300 performs a boost on the output voltage of the battery 342 to avoid abnormal shutdown caused by low temperature.

[0115] The touch sensor 380K is also called a "touch panel". The touch sensor 380K can be set on the display screen 394, and the touch sensor 380K and the display screen 394 form a touch screen, also called a "touch screen". The touch sensor 380K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 394. In other embodiments, the touch sensor 380K can also be set on the surface of the electronic device 300, which is different from the position of the display screen 394.

[0116] The bone conduction sensor 380M can obtain vibration signals. In some embodiments, the bone conduction sensor 380M can obtain vibration signals of vibrating bones of the human body. The bone conduction sensor 380M can also contact the human body's pulse to receive blood pressure beating signals.

[0117] In some embodiments, the bone conduction sensor 380M can also be set in the earphone to form a bone conduction earphone. The audio module 370 can parse the voice signal based on the vibration signal of the vocal bone obtained by the bone conduction sensor 380M to realize the voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 380M to realize the heart rate detection function.

[0118] The key 390 includes a power key, a volume key, etc. The key 390 may be a mechanical key or a touch key. The electronic device 300 may receive key input and generate key signal input related to user settings and function control of the electronic device 300.

[0119] Motor 391 can generate vibration prompts. Motor 391 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 394, motor 391 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0120] Indicator 392 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0121] The SIM card interface 395 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 300 by inserting it into the SIM card interface 395 or pulling it out from the SIM card interface 395. The electronic device 300 can support 3 or N SIM card interfaces, where N is a positive integer greater than 3. The SIM card interface 395 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 395 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 395 can also be compatible with different types of SIM cards. The SIM card interface 395 can also be compatible with external memory cards. The electronic device 300 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 300 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 300 and cannot be separated from the electronic device 300.

[0122] For example, Figure 4 FIG. 4 is a schematic diagram of the hardware structure of a wearable device 400 provided in an embodiment of the present application. Figure 4 The wearable device 400 may include one or more processors 410, one or more memories 420, a communication interface 430, an audio acquisition circuit, and an audio playback circuit. The audio acquisition circuit may further include at least one microphone 440 and an analog-to-digital converter (ADC) 450. The audio playback circuit may further include a speaker 460 and a digital-to-analog converter (DAC).

[0123] The wearable device 400 may further include one or more sensors 480, such as an inertial measurement unit (IMU), a proximity sensor, a motion sensor, etc. The above hardware components may communicate on one or more communication buses.

[0124] In the embodiment of the present application, the IMU can be used to measure the motion posture of the wearable device 400. For example, the IMU can be used to determine the translation distance data, angular velocity data and acceleration data when the user wears the wearable device 400. Among them, the IMU can be provided with a gyroscope sensor and an acceleration sensor.

[0125] The processor 410 is the control center of the wearable device 400. The processor may also be referred to as a control unit, a controller, a microcontroller, or another suitable term. The processor 410 uses various interfaces and lines to connect the various components of the wearable device 400. In a possible embodiment, the processor 410 may also include one or more processing cores. In a possible embodiment, a main control unit and a signal processing module may be integrated in the processor 410. The main control unit (MCU) is used to receive data collected by the sensor 480 or a monitoring signal from the signal processing module or a control signal from a terminal device (such as a mobile phone), and finally controls the wearable device 400 through comprehensive judgment and decision-making.

[0126] The memory 420 may be coupled to the processor 410, or connected to the processor 410 via a bus, and is used to store various software programs and / or multiple sets of instructions and data. The memory 420 may also store a communication program, which may be used to communicate with the terminal. In one example, the memory 420 may also store data / program instructions, and the processor 410 may be used to call and execute the data / program instructions in the memory 420. Exemplarily, multiple sets of noise reduction parameters may be stored in the memory 420. Optionally, the memory 420 may be a memory external to the MCU, or may be a storage unit provided by the MCU.

[0127] The communication interface 430 is used to communicate with the terminal, and the communication mode can be a wired mode or a wireless mode. When the communication mode is wired communication, the communication interface 430 can be connected to the terminal device through a cable. When the communication mode is wireless communication, the communication interface 430 is used to receive and send radio frequency signals, and the wireless communication mode supported by it can be, for example, Bluetooth communication, wireless fidelity (wireless-fidelity, Wifi) communication, infrared communication, near field communication technology (near field communication, NFC), or cellular 2 / 3 / 4 / 5 generation (2 / 3 / 4 / 5 generation, 2G / 3G / 4G / 5G) communication and other communication modes. At least one of the communication modes.

[0128] Microphone 440 can be used to collect sound signals (or audio signals, which are analog signals), and analog-to-digital converter 450 is used to convert the analog signals collected by microphone 440 into digital signals, and send the digital signals to processor 410 for processing. In a specific embodiment, the digital signals can be sent to a signal processing module for processing, and the signal processing module can transmit the processed signals (such as mixed audio signals) to digital-to-analog converter 470. Digital-to-analog converter 470 can convert the received signals into analog signals, and then transmit them to speaker 460. Speaker 460 is used to play according to the analog signals so that the user can hear the sound.

[0129] In the embodiment of the present application, the communication interface 430 can be used to send the translation distance, acceleration data and / or angular velocity data detected by the IMU to the terminal device, so that the terminal device can use the translation distance, acceleration data and / or angular velocity data to determine the IMU data. In addition, the communication interface 430 can also be used to receive spatial audio data sent by the terminal device, which can be understood as audio data with a certain sense of space and direction after being rendered by the terminal device.

[0130] It is understandable that the wearable device 400 may be a headset, wherein the headset may also be referred to as an earplug, a headset, a walkman, an audio player, a media player, a head-mounted receiver, a handset device, or another suitable term. The specific type of the headset may be a TWS headset, which is not limited in the embodiments of the present application.

[0131] It should be noted that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the wearable device 400. In other embodiments of the present application, the wearable device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of hardware and software.

[0132] To better understand the embodiments of the present application, the following will take a mobile phone as a terminal device and a headset as a wearable device as an example. Figure 5 This paper introduces the system architecture of data interaction between terminal devices and wearable devices. Figure 5 FIG. 1 is a schematic diagram of a system architecture for data interaction between a mobile phone and a headset provided in an embodiment of the present application. Figure 5 The system architecture may include a mobile phone and a headset, and the mobile phone and the headset are communicatively connected.

[0133] Figure 5The uplink data refers to the IMU data collected by the IMU in the headset. The IMU data can also be called the user's head rotation data or the inertial sensor data of the headset; the downlink data refers to the audio data transmitted and processed in the mobile phone; the IMU data channel refers to the transmission channel used in the process of transmitting the IMU data collected by the IMU in the headset to the mobile phone.

[0134] See also Figure 5 , a layered architecture can be displayed in a mobile phone, which can be divided into a physical layer, a hardware abstraction layer (HAL), a framework layer, and an application layer from bottom to top. The layers communicate with each other through a software interface. Exemplarily, the layers communicate with each other through JNI (java native interface). It is not difficult to understand that the layered architecture can also include other layers according to the actual scenario, and the embodiments of the present application will not be repeated here.

[0135] Among them, a variety of hardware devices can be set in the physical layer, such as a bluetooth controller (BTC). The BTC can be used to receive the IMU data collected by the IMU transmitted by the headset, and forward the received IMU data to the HAL layer, so that the application in the HAL layer can transmit and process the received IMU.

[0136] Exemplarily, a BTC protocol stack composed of multiple protocols may also be embedded in BTC. After receiving the IMU data transmitted by the headset, BTC may encapsulate the received IMU data using a certain protocol in the BTC protocol stack to transmit the encapsulated IMU data to the HAL layer.

[0137] It should be noted that BTC can usually use the serial port profile (SPP) protocol to receive IMU data transmitted by the headset. In the actual application of the SPP protocol, a corresponding application can be set in the HAL layer so that after BTC receives the IMU data transmitted by the headset, it sends response information corresponding to the received IMU data to the headset, that is, sends corresponding confirmation information to the headset, thereby ensuring the reliability of IMU data transmission.

[0138] For example, assuming that the headset transmits about 20 IMU data to BTC within 200 seconds, then based on the SPP protocol, after BTC receives each of the 20 IMU data transmitted by the headset, the corresponding application in the HAL layer sends confirmation information corresponding to each IMU data to the headset. That is to say, after BTC receives 20 IMU data, the corresponding application in the HAL layer will send 20 confirmation messages to the headset.

[0139] Not only can the BTC protocol stack in BTC be used to transmit IMU data from the headset to the HAL layer, but the BTC protocol stack in BTC can also be used to transmit spatial audio data from the HAL layer to the headset, that is, a certain protocol in the BTC protocol stack is used to decapsulate the spatial audio data to transmit the decapsulated spatial audio data to the headset, and then the spatial audio data received by the headset is transmitted to the speaker via a digital audio interface (such as an SPK interface) for playback. It should be understood that spatial audio data is audio data with a good sense of space and direction obtained after rendering by the mobile phone.

[0140] In a possible implementation, the physical layer may also include devices such as batteries and cameras ( Figure 5 (not shown in the figure), which is not limited in the embodiments of the present application.

[0141] The purpose of the HAL layer is to abstract the hardware. It can provide a unified interface for querying hardware devices for upper-level applications, or provide data storage services or data processing services for upper-level applications.

[0142] In one example, the HAL layer may include: an encoding module and a bluetooth host (BTH) module.

[0143] Among them, the encoding module can be used to receive, process and forward the upper layer ( Figure 5 The processing refers to the encoding module encoding the spatial audio data to obtain the encoded spatial audio data. The encoding process can be understood as converting the audio format of the spatial audio data transmitted by the upper layer so that the converted spatial audio data conforms to the audio format of the audio data received by the next module (i.e., the BTH module) in the audio format. In a specific implementation, the encoding module can implement the above functions by various encoders.

[0144] It should be understood that since the data received, processed and forwarded in the encoding module is audio data, in actual implementation, the Bluetooth audio transmission model protocol (advanced audio distribution profile, A2DP) protocol can be used to receive, process and forward the audio data.

[0145] Exemplarily, the audio data transmitted from the upper layer to the encoding module can be encoded and processed into audio data in any audio format such as advanced audio coding (AAC), sub-band coding (SBC), and low-latency hi-definition audio codec (LHDC).

[0146] In this example, the BTH module can not only be used to receive and forward the IMU data transmitted by the headset through BTC, that is, receive the IMU data transmitted by the headset through BTC, transmit the response information corresponding to the received IMU data to the headset, and forward the received IMU data to the upper layer application. The BTH module can also be used to receive and forward the audio data encoded by the encoding module, that is, the BTH module can also be used to receive the audio data encoded by the encoding module, and forward the received encoded audio data to the headset.

[0147] In a possible implementation, the BTH module may include a HOST protocol stack, and the protocol in the HOST protocol stack may be used to implement Bluetooth connection and control with the headset.

[0148] In a possible implementation, the HAL layer may also be provided with other modules, which is not limited in the embodiments of the present application.

[0149] The framework layer provides user program frameworks and capability frameworks in multiple languages ​​such as Java, C, C++, and JS, two user interface (UI) frameworks (including a Java UI framework for the Java language and a JS UI framework for the JS language), and a multi-language framework application programming interface (API) for various software and hardware services to develop applications for the application layer above it.

[0150] In the audio processing process, the framework layer may include a decoding module and a rendering module, etc., wherein the decoding module can be used to receive the source audio data transmitted by the application layer, such as a video playback application, wherein the source audio data refers to audio data that has not been rendered; the decoding module can also be used to convert the audio format of the received source audio data. In other words, the decoding module can decode the received source audio data into audio data in the audio format required by the next module (such as a rendering module). Exemplary, assuming that the audio format of the audio data required by the next module is audio data of independent channels, and the audio format of the source audio data received by the decoding module is stereo, then the source audio data in stereo format can be transmitted to the decoding module for decoding processing, and the audio data with two channels after decoding processing is obtained. In a specific implementation, the decoding module can be implemented by various decoders.

[0151] The rendering module in the framework layer can not only be used to receive and process the audio data transmitted by the decoding module, but also to receive the IMU data forwarded by the upper application layer, such as the audio manager application, to render the decoded audio data according to the received IMU data to obtain the rendered spatial audio data. The rendering module can also be used to transmit the rendered spatial audio data to the HAL layer ( Figure 5 Module in the encoding).

[0152] In a possible implementation, the framework layer may also include: a window manager, a content provider, a resource manager, a view system, and a notification manager, etc. ( Figure 5 (not shown in the figure), which is not limited in the embodiments of the present application.

[0153] The application layer may include a series of application programs, such as an audio manager application and a video player application.

[0154] Among them, the audio manager application can be used to receive, process and forward the IMU data transmitted by the headset through BTC and BTH. The video playback application can not only be used to control the playback and pause of the source audio data, but also to set the interface or control corresponding to the audio data of the playback space to start the transmission and processing of the audio data.

[0155] It is not difficult to understand that the audio manager application and the video playback application can both be system applications in the mobile phone or applications provided by a third party, which is not limited in the embodiments of the present application.

[0156] It is understandable that the application layer may also include: music, weather, calendar, theme, mailbox, social, application download, payment, contact, my device, album, mobile manager, settings, mall, connection, browser, camera, game, navigation, shopping, address book, phone, information and headset settings and other applications ( Figure 5 (not shown in the figure), which is not limited in the embodiments of the present application.

[0157] It is worth noting that the realization of the corresponding functions of each module set in the application layer, framework layer and hardware abstraction layer is inseparable from the control and execution of the main processor (i.e. CPU) in the mobile phone, and the transmission and processing of other business data in the mobile phone also require the calculation and processing of the main processor. Figure 5 The system architecture for data interaction between the mobile phone and the headset described in the text is slow in transmitting and processing audio data, and cannot quickly achieve the effect of "sound follows the head", thus affecting the user's auditory experience.

[0158] Therefore, in response to the problem that current electronic devices have slow transmission and processing of head motion data, the present application provides an audio processing method. In the process of processing audio data, the main processor only needs to transmit the audio data to the auxiliary processor, and the auxiliary processor processes the audio data, which effectively avoids the slow transmission and processing of audio data caused by too much business data that the main processor needs to process, further improves the processing speed of electronic devices for head motion data (i.e., inertial sensor data), quickly realizes "sound follows head movement", and to a certain extent solves the problem of slow transmission and processing of head motion data by electronic devices.

[0159] The audio processing method provided in this application is exemplarily described below in conjunction with specific embodiments.

[0160] like Figure 6 FIG. 1 is a flow chart of an audio processing method provided by the present application. Figure 6 The method is applied to a terminal device, the terminal device is connected to the wearable device for communication, and the terminal device includes a main processor and an auxiliary processor, wherein the main processor may include a CPU and the auxiliary processor may include a DSP chip. Figure 6 , the method comprising:

[0161] S601: The main processor decodes the source audio data to obtain decoded audio data.

[0162] It should be understood that in the embodiment of the present application, the main processor may include a CPU, which is mainly used to decode the source audio data and transmit the decoded audio data to the auxiliary processor.

[0163] In a possible example, the source audio data may be music audio data, video audio data, call audio data, etc. Further, the source of the source audio data may include audio data played in an application program (such as a video application or a music application) pre-installed in the terminal device; or may include audio data after a playback operation is performed on a recorded audio source file, which is not limited in any way in the embodiments of the present application.

[0164] The decoding process of the source audio data is essentially a process of using the main processor to decode the digital audio signal and restore it into an analog audio signal. In other words, the decoding process of the source audio data is actually to convert the audio format of the source audio data into the audio format of the audio data required by the auxiliary processor. For example, assuming that the source audio data is audio data in a 5.1-channel format (or a 7.1-channel format), the main processor can be used to decode the audio data in the 5.1-channel format (or a 7.1-channel format) into 6 (or 8) channels of audio data, where the 6 (or 8) channels of audio data are the audio data decoded and processed by the main processor.

[0165] S602: The main processor transmits the audio data to the auxiliary processor.

[0166] It should be understood that the auxiliary processor may include a DSP chip. Compared to the main processor, in the embodiment of the present application, the main function of the auxiliary processor is to receive the audio data transmitted by the main processor and render the audio data. It is understandable that in the embodiment of the present application, the main processor does not need to render the audio data after decoding, and only needs to transmit the audio data after decoding to the auxiliary processor, which effectively avoids the problem of slow transmission and processing of audio data due to the main processor in the terminal device needing to process too much business data, further speeds up the processing speed of audio data, and improves the user's auditory experience.

[0167] After the main processor decodes the source audio data, it can transmit the audio data obtained after the decoding process to the auxiliary processor, so that the auxiliary processor executes the subsequent step S603 to achieve rendering processing of the decoded audio data.

[0168] S603: The auxiliary processor obtains inertial sensor data of the wearable device.

[0169] In an embodiment of the present application, after receiving the audio data transmitted by the main processor, the auxiliary processor can obtain the inertial sensor data of the wearable device, and use the acquired inertial sensor data of the wearable device to render the audio data to obtain spatial audio data with better sense of space and direction.

[0170] It should be understood that inertial sensing data may also be referred to as IMU data, and IMU data may be used to indicate changes in the movement (eg, posture) of the user's head.

[0171] In a possible example, the IMU data may include six-degree-of-freedom data, namely: the translation distance of the wearable device on the X-axis, Y-axis and Z-axis, and the angle value of the wearable device rotating around the X-axis, Y-axis and Z-axis; the IMU data may also include the translation distance of the wearable device on the X-axis, Y-axis and Z-axis, the angle value of the wearable device rotating around the X-axis, Y-axis and Z-axis, and the acceleration value of the wearable device on the X-axis, Y-axis and Z-axis. This application does not limit this.

[0172] Exemplarily, a preset number (e.g., 6) of IMUs may be set in the wearable device to collect IMU data of the wearable device. It should be understood that the number of IMUs set in the wearable device may correspond to the type of chip set in the wearable device, or may be user-defined.

[0173] It is not difficult to understand that different numbers and / or different types of sensors can be set in the IMU according to the specific content of the IMU data. As an example and not a limitation, an accelerometer sensor and a gyroscope sensor can be set in the IMU, and sensors such as an inclinometer, an accelerometer, a gyroscope, and a magnetometer can also be set in the IMU. The embodiment of the present application does not impose any restrictions on the number of IMUs set in the wearable device, the number of sensors set in the IMU, and the specific types of sensors.

[0174] In a possible implementation, the terminal device may include a communication module, which receives inertial sensing data transmitted by the wearable device through a first protocol, wherein the first protocol may be a unidirectional transmission protocol.

[0175] It should be understood that in the above possible implementations, the specific type of the communication module may be related to the communication connection mode between the terminal device and the wearable device. Correspondingly, the first protocol may also be a unidirectional transmission protocol related to the communication connection mode between the terminal device and the wearable device.

[0176] Exemplarily, when the communication connection mode between the terminal device and the wearable device is a Bluetooth connection, the communication module may be a Bluetooth communication module, and correspondingly, the first protocol may be a human interface device (HID) protocol.

[0177] It is not difficult to understand that when the terminal device and the wearable device are connected to each other through other communication methods, the communication module can be a communication module corresponding to the other communication methods, and the first protocol can also be other unidirectional transmission protocols corresponding to the other communication methods. This application does not limit this.

[0178] Based on the above possible implementation methods, the communication module receives the inertial sensor data transmitted by the wearable device through a unidirectional transmission protocol, so that the communication module only needs to receive the inertial sensor data sent by the wearable device, and there is no need to reply corresponding response information to the wearable device based on whether the inertial sensor data transmitted by the wearable device is received. This not only speeds up the speed at which the wearable device transmits the inertial sensor data to the communication module, but also avoids the need for the main processor (or auxiliary processor) to reply to the wearable device with response information corresponding to the received inertial sensor data, causing the main processor (or auxiliary processor) to be in an awake state all the time, further reducing the utilization rate of the main processor (or auxiliary processor) and reducing the power consumption of the main processor (or auxiliary processor).

[0179] In some embodiments, the auxiliary processor can directly receive the inertial sensor data transmitted by the communication module; or a corresponding data transmission module can be set in the terminal device so that the data transmission module receives the inertial sensor data transmitted by the communication module, and then the auxiliary processor obtains the inertial sensor data through the data transmission module.

[0180] Exemplarily, the data transmission module may be provided in a Sensor Hub, and the Sensor Hub may be used to transmit the inertial sensor data transmitted by the communication module to the auxiliary processor.

[0181] Based on the above two possible implementations, the transmission path for the auxiliary processor to obtain inertial sensor data can be enriched. In addition, using a data transmission module independent of the auxiliary processor to transmit inertial sensor data can further speed up the transmission speed of inertial sensor data to the auxiliary processor, speed up the processing efficiency of inertial sensor data by the auxiliary processor, and further reduce the transmission processing delay of audio data.

[0182] S604: The auxiliary processor renders the audio data according to the inertial sensor data to obtain spatial audio data.

[0183] It should be understood that after the auxiliary processor obtains the inertial sensor data of the wearable device, it can use the inertial sensor data to render the decoded audio data to obtain spatial audio data with a better sense of space and direction.

[0184] Exemplarily, in the actual rendering process, the auxiliary processor can adopt at least one of the following algorithms to obtain spatial audio data: head related transfer function (HRTF), a positioning method based on controllable beamforming with maximum output power, a positioning method based on high-resolution spectrum estimation, a positioning method based on time delay difference of arrival (TDOA) estimation, and a method based on machine learning, etc.

[0185] In a possible implementation, after acquiring the spatial audio data, the auxiliary processor can also be used to encode the spatial audio data to obtain the encoded spatial audio data; thereafter, the auxiliary processor transmits the encoded spatial audio data to the wearable device through the communication module, so that the user can listen to the spatial audio data with better spatial sense and direction through the wearable device.

[0186] It should be understood that the encoding process corresponds to the decoding process, and the encoding process is essentially the process of using the auxiliary processor to convert the analog audio signal into a digital audio signal, that is, the process of converting the spatial audio data after rendering into the audio format of the audio data that the wearable device can receive.

[0187] It is not difficult to understand that in actual applications, after the wearable device receives the encoded spatial audio data transmitted by the auxiliary processor through the communication module, a corresponding decoder or decoding module can be set in the wearable device to decode the spatial audio data transmitted to the wearable device, and obtain the decoded spatial audio data, which is convenient for playing through the speakers set in the wearable device.

[0188] The following example takes the main processor as CPU, the auxiliary processor as DSP chip, the terminal device as mobile phone, and the wearable device as Bluetooth headset as an example. Figure 7 The inertial sensor data (hereinafter referred to as IMU data) shown in the figure is used to exemplarily illustrate the audio processing method provided in the embodiment of the present application. Figure 7 The mobile phone and the Bluetooth headset communicate via Bluetooth. The layered architecture can still be displayed in the mobile phone. The specific layered architecture in the mobile phone can refer to the aforementioned Figure 5 The relevant descriptions in the above embodiments will not be repeated here. Figure 7 , the method may include:

[0189] Step 701: The Bluetooth headset transmits the collected IMU data to the DSP chip via BTC.

[0190] It should be understood that at least one IMU can be set in the Bluetooth headset to collect IMU data. The specific settings of the IMU can be found in the above Figure 6 The relevant description in the illustrated embodiment will not be repeated here.

[0191] Exemplarily, the DSP chip may include an IMU data transmission module, an encoding module and a rendering module, wherein the IMU data transmission module can be used to receive the IMU data transmitted by the BTC and transmit the received IMU data to the rendering module; the encoding module can be used to encode the spatial audio data output by the rendering module, so as to transmit the encoded spatial audio data to the Bluetooth headset via the BTC.

[0192] The rendering module can be used to receive the audio data transmitted by the CPU after being decoded by the decoding module and the IMU data transmitted by the IMU data transmission module, and render the decoded audio data according to the received IMU data, thereby obtaining the rendered spatial audio data, wherein the spatial audio data has a greater sense of space and direction than the decoded audio data. The rendering module can also be used to transmit the rendered spatial audio data to the encoding module so that the encoding module can encode the spatial audio data.

[0193] In a possible implementation, the Bluetooth headset transmits the IMU data to the DSP via a first protocol, wherein the first protocol may be a unidirectional transmission Bluetooth protocol.

[0194] In some embodiments, the first protocol may be a HID protocol. It should be understood that the HID protocol is a one-way Bluetooth protocol. In actual applications, after the Bluetooth headset transmits IMU data to the DSP through the HID protocol, the DSP does not need to send a response message corresponding to the received IMU data to the Bluetooth headset. That is to say, in the process of the DSP receiving the IMU data sent by the Bluetooth headset through the HID protocol, the DSP is mainly used for the IMU data transmitted by the Bluetooth headset through the BTC. After the DSP receives the IMU data, it is not necessary to send corresponding confirmation information to the Bluetooth headset for each piece of IMU data received. In this way, since the DSP does not need to send corresponding confirmation information to the Bluetooth headset, the power consumption of the DSP is further reduced, the transmission speed of the IMU data is accelerated, and the delay in audio data transmission and processing is reduced.

[0195] Step 702: The CPU decodes the source audio data to obtain decoded audio data.

[0196] It should be understood that the CPU, as the main processor, can realize the transmission and processing of data between the application layer, framework layer and hardware abstraction layer in the mobile phone by running the application program.

[0197] In some examples, the application layer of the mobile phone may include a video player application, which may be used to control the play and pause of the source audio data. Of course, in other possible implementations, the application layer of the mobile phone may also include other applications, such as music applications.

[0198] In some examples, the framework layer of the mobile phone may include a decoding module, which can be used to receive source audio data transmitted by a video playback application in the application layer of the mobile phone; it can also be used to decode the received source audio data to obtain decoded audio data. Afterwards, it can also be used to forward the decoded audio data to the audio data transmission module.

[0199] In some examples, the hardware abstraction layer of the mobile phone may include an audio data transmission module, which can be used to receive the decoded audio data transmitted by the decoding module; and can also be used to transmit the received audio data to the DSP.

[0200] Step 703: The CPU transmits the decoded audio data to the DSP.

[0201] It should be understood that in order to reduce the energy consumption of the CPU and further release the computing power of the CPU, in an embodiment of the present application, the CPU is mainly used to decode the source audio data and transmit the decoded audio data to the DSP, so as to use the DSP to realize the rendering of the decoded audio data.

[0202] Step 704: The DSP renders the audio data transmitted by the CPU according to the received IMU data to obtain spatial audio data.

[0203] It should be understood that the DSP may include an IMU data transmission module and a rendering module, among which the IMU data transmission module may be used to receive and forward the IMU data transmitted by the Bluetooth headset via BTC; the rendering module may be used to render the IMU data and audio data to obtain spatial audio data. The specific process of DSP rendering audio data according to IMU data can be found in the above Figure 6 The corresponding description in the illustrated embodiment will not be repeated here.

[0204] After the DSP renders the received IMU data and audio data, it can also transmit the rendered spatial audio data to the Bluetooth headset. Therefore, after the above step 704, the method can also include:

[0205] Step 705: DSP transmits spatial audio data to the Bluetooth headset via BTC.

[0206] It should be understood that in the embodiment of the present application, the DSP may also include a coding module; the rendering module in the DSP can transmit the rendered spatial audio data to the coding module; the coding module can be used to receive the spatial audio data transmitted by the rendering module, and can also be used to encode the spatial audio data to obtain the encoded spatial audio data; it can also be used to forward the encoded spatial audio data to the Bluetooth headset through the BTC, so that the Bluetooth headset receives the encoded spatial audio data.

[0207] In one example, the encoding module can use the A2DP protocol in the BTC protocol stack to forward the encoded spatial audio data to the Bluetooth headset. It should be understood that the use of the A2DP protocol can effectively improve the transmission reliability of the encoded spatial audio data and reduce the energy consumption of the mobile phone and the Bluetooth headset.

[0208] Step 706: The Bluetooth headset receives the spatial audio data transmitted by the DSP.

[0209] It should be understood that the specific process of the Bluetooth headset receiving the spatial audio data transmitted by the DSP can be found in Figure 6 The relevant description in the illustrated embodiment will not be repeated here.

[0210] In another possible implementation, Figure 7 In the embodiment shown, the IMU data transmission module in the DSP is decoupled from the DSP, reducing the processing steps of the DSP and further reducing the power consumption of the DSP. For example, the IMU data transmission module can be set in the sensor hub (SensorHub) to use the IMU data transmission module set in the SensorHub to receive and transmit the IMU data transmitted by the Bluetooth headset through the BTC. Figure 8 FIG. 1 is a schematic diagram of another structure of inertial sensor data flow provided in an embodiment of the present application, see Figure 8 , the method may further include:

[0211] Step 801: The Bluetooth headset transmits the collected IMU data to the DSP via BTC.

[0212] Step 802: The CPU decodes the source audio data to obtain decoded audio data.

[0213] Step 803: The CPU transmits the decoded audio data to the DSP.

[0214] Step 804: The Bluetooth headset transmits IMU data to the sensor hub (SensorHub) via BTC.

[0215] Exemplarily, the SensorHub may include an IMU data transmission module, which may be used to receive and forward IMU data transmitted by the Bluetooth headset via BTC. Further, the IMU data transmission module may transmit the received IMU data to the DSP.

[0216] Step 805, SensorHub transmits IMU data to DSP.

[0217] It should be understood that in this embodiment, the DSP may include a rendering module, wherein the rendering module can be used to receive the IMU data transmitted by the SensorHub; it can also be used to receive the decoded audio data transmitted by the CPU. In this way, the DSP can directly receive the IMU data and the decoded audio data of the Bluetooth headset, and use the IMU data to render the audio data, thereby quickly obtaining the rendered spatial audio data.

[0218] Step 806: The DSP renders the audio data transmitted by the CPU according to the received IMU data to obtain spatial audio data.

[0219] Step 807: DSP transmits spatial audio data to the Bluetooth headset via BTC.

[0220] Step 808: The Bluetooth headset receives the spatial audio data transmitted by the DSP.

[0221] It should be understood that the above steps 801 to 803 and steps 806 to 808 can refer to Figure 7 The relevant descriptions of step 701 to step 706 in the embodiment are not repeated here.

[0222] Based on the above implementation, the transmission path of IMU data can be enriched, and compared with the transmission of IMU data by DSP, the transmission of IMU data through SensorHub can not only reduce the processing steps of DSP and further reduce the power consumption of DSP, but also speed up the processing speed of IMU data and audio data, quickly obtain the rendered spatial audio data, and effectively reduce the processing delay of audio data.

[0223] It should be noted that, taking the terminal device as a mobile phone and the wearable device as a TWS Bluetooth headset as an example, when the mobile phone is in the same operating state, the source audio data of 7.1 channels is based on Figure 5 The embodiments shown and Figure 7 The embodiment shown in the experiment was adopted Figure 5 After the embodiment shown processes the 7.1-channel source audio data, the power consumption of the mobile phone increases by 63 milliamps (mA); Figure 5 The embodiment shown adopts Figure 7 After the embodiment shown processes the 7.1-channel source audio data, the power consumption of the mobile phone can be reduced by 30mA. It can be seen that the audio processing method provided in the embodiment of the present application can effectively reduce the overall power consumption of the terminal device.

[0224] In addition, taking the terminal device as a mobile phone and the wearable device as a TWS Bluetooth headset as an example, the Figure 5 The embodiments shown and Figure 7 The delay test of the embodiment shown in the figure can be obtained: Figure 5 In the embodiment shown, after the user wears the TWS Bluetooth headset, the time consumed from the user's head starting to rotate until the corresponding spatial audio data is played in the TWS Bluetooth headset is about 215 milliseconds (ms); Figure 7 The time consumed by the embodiment shown is about 185ms. Figure 5 Compared with the embodiment shown, the audio processing method provided in the embodiment of the present application can effectively reduce the delay of about 30ms. It can be seen that the audio processing method provided in the embodiment of the present application can effectively reduce the transmission and processing delay of audio data.

[0225] In summary, the audio processing method provided in the embodiment of the present application, in which the auxiliary processor renders the audio data according to the inertial sensor data of the wearable device obtained to obtain spatial audio data, effectively avoids the situation where the transmission and processing of the audio data is slow due to the excessive business data that the main processor needs to process. The processing of the audio data by the auxiliary processor not only improves the processing speed of the electronic device for the inertial sensor data, and quickly realizes the "sound follows the head movement"; but also, compared with transmitting the inertial sensor data to the main processor for processing the inertial sensor data by the main processor, it can effectively reduce the transmission path of the inertial sensor data, and further reduce the transmission delay of the audio data, thereby solving the problem of slow transmission and processing of head motion data by the electronic device, so that the user can hear smoother spatial audio, improve the user experience of the spatial audio function, and improve the user's auditory experience.

[0226] In addition, using the auxiliary processor to process audio data can also solve the problem of excessive power consumption of the main processor due to the large amount of data that the main processor needs to process; it can also further reduce the continuous usage rate of the main processor, release the computing power of the main processor, avoid the situation where the main processor is preempted, and further reduce the impact on the transmission and processing of audio data. For example, when the main processor is processing audio data, there is a request to process other business data with a higher priority than audio data. The main processor may give priority to processing other business data with a higher priority, which in turn affects the transmission and processing of audio data by the main processor.

[0227] It should be understood that the order of execution of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the named or numbered process steps can also be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0228] Based on the audio processing methods provided in the above embodiments, the embodiments of the present application also provide the following contents:

[0229] The present application embodiment provides an electronic device, the electronic device includes a main processor and an auxiliary processor, and the electronic device is configured to execute the audio processing method shown in the above embodiments. Figure 3 The structure shown.

[0230] An embodiment of the present application provides a computer program product, which includes a program. When the program is executed by an electronic device, the electronic device implements the audio processing method shown in the above embodiments.

[0231] An embodiment of the present application also provides a communication system, which includes: the above-mentioned terminal device and / or the above-mentioned wearable device.

[0232] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the audio processing method shown in the above embodiments is implemented.

[0233] An embodiment of the present application provides a chip, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory, so that the chip executes the steps performed by the auxiliary processor in the audio processing method shown in the above-mentioned embodiments.

[0234] An embodiment of the present application provides a chip system, which includes: a main processor and an auxiliary processor. The chip system is used to call a program from a memory so that a device equipped with the chip system executes the audio processing method shown in the above embodiments.

[0235] It should be understood that the main processor and / or auxiliary processor mentioned in the embodiments of the present application may be a CPU, or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0236] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (doubledatarate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus RAM (DR RAM).

[0237] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, refer to the relevant description of other embodiments.

[0238] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0239] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to a large-screen device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0243] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: The method is applied to a terminal device, the terminal device is communicatively connected with a wearable device, the terminal device includes a main processor and an auxiliary processor, and the method includes: The main processor decodes the source audio data to obtain decoded audio data; The main processor transmits the audio data to the auxiliary processor; The auxiliary processor obtains inertial sensing data of the wearable device; The auxiliary processor performs rendering processing on the audio data according to the inertial sensor data to obtain spatial audio data.

2. The method according to claim 1, characterized in that The terminal device further includes a communication module, and the method further includes: The communication module receives the inertial sensing data transmitted by the wearable device through a first protocol, where the first protocol is a unidirectional transmission protocol; The auxiliary processor obtains inertial sensing data of the wearable device, including: The auxiliary processor obtains the inertial sensing data through the communication module.

3. The method according to claim 2, characterized in that The auxiliary processor obtains the inertial sensing data through the communication module, including: The auxiliary processor receives the inertial sensing data transmitted by the communication module.

4. The method according to claim 2, characterized in that: The terminal device further includes a data transmission module, and the auxiliary processor obtains the inertial sensor data through the communication module, including: The data transmission module receives the inertial sensing data transmitted by the communication module; The auxiliary processor obtains the inertial sensing data through the data transmission module.

5. The method according to any one of claims 2 to 4, characterized in that The method further comprises: The auxiliary processor encodes the spatial audio data to obtain the encoded spatial audio data; The auxiliary processor transmits the encoded spatial audio data to the wearable device through the communication module.

6. The method according to any one of claims 2 to 5, characterized in that The first protocol includes a Human Interface Device (HID) protocol.

7. The method according to any one of claims 1 to 6, characterized in that The auxiliary processor includes a digital signal processing DSP chip.

8. A chip, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory, so that the chip executes the method executed by the auxiliary processor in any one of claims 1 to 7.

9. A chip system, characterized in that: include: A main processor and an auxiliary processor, the chip system is used to call a program from a memory so that a device equipped with the chip system executes a method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A main processor and an auxiliary processor, the electronic device is used to execute a computer program or instruction stored in a memory, so that the electronic device implements the method as claimed in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images