Audio processing method and electronic device

Abstract

This application provides an audio processing method and an electronic device. The electronic device includes a digital signal processing (DSP) chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. In this method, the DSP chip can sequentially process, encode, and package the first audio data using the first audio processing unit, the first encoding / decoding unit, and the first packet processing unit to obtain fourth audio data, which is then sent to the device node. The first application can obtain the fourth audio data from the device node and send it using the transceiver unit. This method can be applied to VoIP calls, reducing VoIP call latency and improving VoIP call efficiency.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to an audio processing method and an electronic device. Background Technology

[0002] Voice over Internet Protocol (VoIP) calls are real-time communication technologies that utilize the internet. VoIP calls allow communication via applications on electronic devices and are relatively inexpensive compared to other call methods, making them widely used. However, current VoIP calls suffer from high latency and low efficiency. Summary of the Invention

[0003] This application provides an audio processing method and an electronic device for reducing VoIP call latency and improving VoIP call efficiency.

[0004] In a first aspect, embodiments of this application provide a control method applied to an electronic device. The electronic device includes a digital signal processing (DSP) chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. The method includes: the DSP chip acquiring first audio data; the DSP chip performing audio processing on the first audio data through the first audio processing unit to obtain second audio data; encoding the second audio data through the first encoding / decoding unit to obtain third audio data; and packaging the third audio data through the first packet processing unit to obtain fourth audio data. The audio processing includes at least one of the following: acoustic echo cancellation, background noise suppression, and automatic gain control; the DSP chip sending the fourth audio data to the device node; the first application acquiring the fourth audio data from the device node through the transceiver unit; and the first application sending the fourth audio data through the transceiver unit.

[0005] This method utilizes a DSP chip with mature and comprehensive audio processing algorithms to complete audio processing tasks during audio transmission. On one hand, it leverages the existing audio processing capabilities of the DSP chip to quickly and efficiently complete audio processing tasks. On the other hand, it enables communication between the DSP chip and the application through device nodes, reducing latency in audio processing at the application layer and HAL layer of the electronic device, lowering the computational power consumption of the AP system (large core), and thus reducing the power consumption of the large core. Simultaneously, it reduces the number of buffers for audio data processing at the application framework layer and HAL layer, thereby reducing the demand for large core storage space. This method can be applied to VoIP call processes, reducing VoIP call latency and improving VoIP call efficiency. Furthermore, when applied to VoIP call processes, the application primarily forwards audio data, and the main audio processing algorithms during audio transmission can be centralized in the DSP chip. This ensures that the entire audio transmission link essentially uses only one set of audio processing algorithms from the DSP chip, avoiding redundancy in audio processing algorithms, improving audio transmission efficiency, reducing application power consumption, and enhancing overall application performance.

[0006] In one possible design, the second audio processing unit is located in the application framework layer of the electronic device, the third audio processing unit is located in the hardware abstraction layer of the electronic device, and the fourth audio data does not pass through the second audio processing unit and the third audio processing unit.

[0007] In this method, audio data is not processed in the application framework layer and hardware abstraction layer, which simplifies the process that traditional VoIP applications and DSP chips need to interact with the application framework layer and hardware abstraction layer in a complex manner, thereby improving processing efficiency.

[0008] In one possible design, the fourth audio data does not pass through the fourth audio processing unit, the second encoding / decoding unit, and the second packet processing unit.

[0009] In this method, audio data is not processed in the application, which reduces the application's resource consumption, thereby reducing application power consumption and improving application performance.

[0010] In one possible design, the device node is located in the kernel layer of the electronic device.

[0011] In this method, bidirectional communication between the DSP chip and the AP side is achieved through device nodes in the kernel layer, which can reduce the dependence on the application framework layer and hardware abstraction layer and help improve processing efficiency.

[0012] In one possible design, the AP system further includes a first control information processing unit and a second control information processing unit, wherein the first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device; the method further includes: the first application generating first control information for controlling the DSP chip to perform a first operation; the first application sending the first control information to the device node; the DSP chip obtaining the first control information from the device node; and the DSP chip performing the first operation according to the first control information; wherein the first control information does not pass through the first control information processing unit and the second control information processing unit.

[0013] In this method, control information can be transmitted through device nodes, so that the transmission of control information does not depend on the application framework layer and hardware abstraction layer, thereby improving the efficiency of control information transmission.

[0014] In one possible design, the DSP chip sends the fourth audio data to the device node, including: the DSP chip sending first data including the fourth audio data and first identification information to the device node; wherein the first identification information is used to indicate the data type of the fourth audio data; the first application obtaining the fourth audio data from the device node through the transceiver unit includes: the first application obtaining the first data from the device node through the transceiver unit and determining the fourth audio data based on the first data; the first application sending the first control information to the device node includes: the first application sending second data including the first control information and second identification information to the device node; wherein the second identification information is used to indicate the data type of the first control information; the DSP chip obtaining the first control information from the device node includes: the DSP chip obtaining the second data from the device node and determining the first control information based on the second data.

[0015] This method distinguishes audio data from control information through different identification information, enabling the transmission of audio data and control information in electronic devices through only one device node, making it highly practical.

[0016] In one possible design, the data type of the fourth audio data is an audio type or an uplink audio type, and the data type of the first control information is a control type or an uplink control type.

[0017] In one possible design, the device node corresponds to the data type of the fourth audio data; the AP system further includes a first control information processing unit and a second control information processing unit, the first control information processing unit being located in the application framework layer of the electronic device, and the second control information processing unit being located in the hardware abstraction layer of the electronic device; the method further includes: the first application generating first control information for controlling the DSP chip to perform a first operation; the first application sending the first control information to the device node corresponding to the data type of the first control information; wherein the device node corresponding to the data type of the first control information is located in the kernel layer of the electronic device; the DSP chip obtaining the first control information from the device node corresponding to the data type of the first control information; the DSP chip performing the first operation according to the first control information; wherein the first control information does not pass through the first control information processing unit and the second control information processing unit.

[0018] This method enables the transmission of control information through device nodes, allowing the transmission to be independent of the application framework layer and hardware abstraction layer, thereby improving the efficiency of control information transmission. Furthermore, different device nodes can be used to transmit audio data and control information separately, ensuring that different types of data have independent data transmission channels and further improving data transmission efficiency.

[0019] Secondly, embodiments of this application provide a control method applied to an electronic device. The electronic device includes a digital signal processing (DSP) chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. The method includes: the first application receiving first audio data through the transceiver unit; the first application sending the first audio data to the device node through the transceiver unit; the DSP chip obtaining the first audio data from the device node; the DSP chip unpacking the first audio data through the first packet processing unit to obtain a second audio data; decoding the second audio data through the first encoding / decoding unit to obtain a third audio data; and performing audio processing on the third audio data through the first audio processing unit to obtain fourth audio data. The audio processing includes at least one of the following: acoustic echo cancellation, background noise suppression, and automatic gain control.

[0020] This method utilizes a DSP chip with mature and comprehensive audio processing algorithms to complete audio processing tasks during audio transmission. On one hand, it leverages the existing audio processing capabilities of the DSP chip to quickly and efficiently complete audio processing tasks. On the other hand, it enables communication between the DSP chip and the application through device nodes, reducing latency in audio processing at the application layer and HAL layer of the electronic device, lowering the computational power consumption of the AP system (large core), and thus reducing the power consumption of the large core. Simultaneously, it reduces the number of buffers for audio data processing at the application framework layer and HAL layer, thereby reducing the demand for large core storage space. This method can be applied to VoIP call processes, reducing VoIP call latency and improving VoIP call efficiency. Furthermore, when applied to VoIP call processes, the application primarily forwards audio data, and the main audio processing algorithms during audio transmission can be centralized in the DSP chip. This ensures that the entire audio transmission link essentially uses only one set of audio processing algorithms from the DSP chip, avoiding redundancy in audio processing algorithms, improving audio transmission efficiency, reducing application power consumption, and enhancing overall application performance.

[0021] In one possible design, the second audio processing unit is located in the application framework layer of the electronic device, the third audio processing unit is located in the hardware abstraction layer of the electronic device, and the first audio data does not pass through the second audio processing unit and the third audio processing unit.

[0022] In one possible design, the first audio data does not pass through the fourth audio processing unit, the second encoding / decoding unit, and the second packet processing unit.

[0023] In one possible design, the DSP chip further includes a first anti-jitter unit, and the first application further includes a second anti-jitter unit; after the DSP chip performs unpacking processing on the first audio data through the first packet processing unit to obtain the second audio, and before decoding the second audio through the first encoding / decoding unit to obtain the third audio, the method further includes:

[0024] The second audio is processed to reduce jitter using the first anti-jitter unit;

[0025] The first audio data does not pass through the second anti-jitter unit.

[0026] In one possible design, the device node is located in the kernel layer of the electronic device.

[0027] In one possible design, the AP system further includes a first control information processing unit and a second control information processing unit, wherein the first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device; the method further includes:

[0028] The first application generates first control information for controlling the DSP chip to perform a first operation;

[0029] The first application sends the first control information to the device node;

[0030] The DSP chip obtains the first control information from the device node;

[0031] The DSP chip executes the first operation according to the first control information;

[0032] The first control information does not pass through the first control information processing unit and the second control information processing unit.

[0033] In one possible design, the first application sends the first audio data to the device node through the transceiver unit, including:

[0034] The first application sends first data, including the first audio data and first identification information, to the device node through the transceiver unit; wherein, the first identification information is used to indicate the data type of the first audio data;

[0035] The DSP chip obtains the first audio data from the device node, including:

[0036] The DSP chip obtains the first data from the device node and determines the first audio data based on the first data;

[0037] The first application sends the first control information to the device node, including:

[0038] The first application sends second data, including the first control information and the second identification information, to the device node; wherein the second identification information is used to indicate the data type of the first control information;

[0039] The DSP chip obtains the first control information from the device node, including:

[0040] The DSP chip obtains the second data from the device node and determines the first control information based on the second data.

[0041] In one possible design, the data type of the first audio data is an audio type or an uplink audio type, and the data type of the first control information is a control type or an uplink control type.

[0042] In one possible design, the device node corresponds to the data type of the first audio data; the AP system further includes a first control information processing unit and a second control information processing unit, the first control information processing unit being located in the application framework layer of the electronic device, and the second control information processing unit being located in the hardware abstraction layer of the electronic device; the method further includes:

[0043] The first application generates first control information for controlling the DSP chip to perform a first operation;

[0044] The first application sends the first control information to a device node corresponding to the data type of the first control information; wherein the device node corresponding to the data type of the first control information is located in the kernel layer of the electronic device;

[0045] The DSP chip obtains the first control information from the device node corresponding to the data type of the first control information;

[0046] The DSP chip executes the first operation according to the first control information;

[0047] The first control information does not pass through the first control information processing unit and the second control information processing unit.

[0048] Thirdly, this application provides an electronic device including a memory and one or more processors; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by one or more processors, the electronic device causes the electronic device to perform the method described in the first aspect or any possible design of the first aspect, or to perform the method described in the second aspect or any possible design of the second aspect.

[0049] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect or any possible design of the first aspect, or to perform the method described in the second aspect or any possible design of the second aspect.

[0050] Fifthly, this application provides a computer program product comprising a computer program or instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect or any possible design of the first aspect, or to perform the method described in the second aspect or any possible design of the second aspect.

[0051] For the beneficial effects described in the second to fifth aspects above, please refer to the description of the beneficial effects in the first aspect above, which will not be repeated here. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of a VoIP call process;

[0053] Figure 2 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application;

[0054] Figure 3 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0055] Figure 4 This application provides a schematic diagram of the architecture of an audio processing system.

[0056] Figure 5 A schematic diagram illustrating an audio transmission method provided in an embodiment of this application;

[0057] Figure 6 A schematic diagram illustrating an audio receiving method provided in an embodiment of this application;

[0058] Figure 7 A schematic diagram illustrating an audio processing method provided in an embodiment of this application;

[0059] Figure 8 A schematic diagram illustrating an audio processing method provided in an embodiment of this application;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0062] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0063] For ease of understanding, exemplary descriptions of concepts related to this application are provided for reference.

[0064] 1) Electronic devices can be devices with communication functions. In some embodiments of this application, electronic devices can be portable devices, such as mobile phones, tablets, wearable devices with wireless communication functions (e.g., watches, bracelets, etc.), vehicle-mounted terminal devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart home devices (e.g., smart TVs, smart speakers, etc.), smart robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, and flying devices (e.g., smart robots, drones, airplanes), etc.

[0065] In some embodiments of this application, the electronic device may also be a portable terminal device that includes other functions. Exemplary embodiments of the portable terminal device include, but are not limited to, carrying... Alternatively, it could be a portable terminal device with another operating system. The aforementioned portable terminal device could also be other portable terminal devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of this application, the aforementioned electronic device may not be a portable terminal device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0066] 2) Voice over Internet Protocol (VoIP) calling is a voice communication technology that uses the Internet Protocol (IP) to achieve voice calls and multimedia conferencing, i.e., communication via the Internet. The basic principle of VoIP calling includes: the sending end encodes, compresses, and packages audio data into audio data packets, which are then sent to the receiving end via the network; the receiving end unpacks, decompresses, and decodes the audio data packets from the sending end to restore the original audio signal, thus achieving the effect of transmitting audio over the Internet. VoIP calling can also be called or understood as IP-based voice transmission, IP telephony, Internet telephony, broadband telephony, broadband phone service, etc.

[0067] 3) 3A audio processing, or simply 3A, is a collective term for three audio algorithms: acoustic echo cancellation (AEC), acoustic noise suppression (ANS), and automatic gain control (AGC).

[0068] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0069] In the era of rich media information, audio and video have become important ways for people to obtain, publish, and exchange information. Due to the upgrading of service experience demands, the iteration of audio and video transmission technologies, and the improvement of network infrastructure, people's requirements for the latency of audio and video services have become increasingly stringent, which has spurred the booming growth of the real-time audio and video industry. Currently, real-time audio and video services such as audio and video telephony (e.g., MeeTime), online conferencing, interactive live streaming / classrooms, and remote work / medical services are gradually penetrating all aspects of people's lives.

[0070] One widely used real-time audio service is Voice over Internet Protocol (VoIP) calling. VoIP allows calls to be made over the network through applications on electronic devices, and its cost is relatively low compared to other call methods, hence its widespread use. However, VoIP calls generally have higher latency, resulting in lower call efficiency. This is mainly because the VoIP call process is time-consuming at the application framework layer and hardware abstraction layer (HAL). A detailed explanation follows with accompanying diagrams.

[0071] Figure 1 This is a diagram illustrating the audio processing flow of a VoIP call. Figure 1As shown, in the audio processing flow of a VoIP call, for the uplink audio link (i.e., the link for sending audio), the audio processing flow mainly includes: the audio acquisition device (e.g., microphone) acquires audio data and sends the acquired audio data to the digital signal processing (DSP) chip. The DSP chip directly sends the received audio data to the HAL layer on the application processor (AP) side. After the HAL layer performs 3A audio processing on the audio data, it sends the processed audio data to the corresponding VoIP application through the application framework layer. Typically, there is also corresponding audio processing in the VoIP application, such as further 3A enhancement processing. The VoIP application can also encode the processed audio data, package it using the real-time transport protocol (RTP), and call the corresponding data packet sending interface to send the packaged audio data to the modem (or modem chip). The modem can then send the audio data out. For the downlink audio link (i.e., the link for receiving audio), the audio processing flow mainly includes: the modem can receive audio data packets and send the received audio data packets to the VoIP application for unpacking. VoIP applications can perform anti-jitter, decoding, and software 3A processing on the unpacked audio data to obtain the final audio data. After further processing by the application framework layer and HAL layer, the VoIP application sends the audio data to the DSP chip. The DSP chip can then transmit the audio data to an audio playback device (such as a microphone) for playback.

[0072] In the aforementioned VoIP call audio processing flow, the application framework layer and HAL layer involve numerous audio processing steps (such as 3A audio processing), leading to significant latency. Furthermore, the VoIP call audio processing flow involves multiple processing modules; data interaction between these modules and data caching and copying operations within the modules also contribute to substantial latency and power consumption. Additionally, the VoIP application handles a significant amount of audio processing, increasing its power consumption and potentially degrading its performance.

[0073] Based on the above problems, in order to reduce the latency and power consumption of VoIP calls, this application provides an audio processing method and electronic device. This solution can transfer the audio processing process of the application framework layer and HAL layer, and even the VoIP application, on the Android platform to the DSP chip, thereby leveraging the audio processing capabilities of the DSP chip to achieve fast and efficient audio processing, and thus reduce the latency and power consumption of VoIP calls.

[0074] See below. Figure 2 The structure of the electronic device to which the method provided in the embodiments of this application is applicable will be described.

[0075] like Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc.

[0076] The sensor module 180 may include a gyroscope sensor, an accelerometer, a proximity sensor, a fingerprint sensor, a touch sensor, a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, a barometric pressure sensor, a bone conduction sensor, etc.

[0077] Understandable, Figure 2 The electronic device 100 shown is merely an example and does not constitute a limitation on the electronic device. Furthermore, the electronic device may have more than [specific features]. Figure 2 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 2 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0078] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the electronic device 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0079] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0080] The audio processing method provided in this application embodiment can be executed by the processor 110 controlling or calling other components. For example, it can call the processing program of this application embodiment stored in the internal memory 121, or call the processing program of this application embodiment stored in a third-party device through the external memory interface 120 to control the wireless communication module 160 to perform data communication with other devices, thereby improving the intelligence and convenience of the electronic device 100 and enhancing the user experience. The processor 110 may include different devices. For example, when integrating a CPU and a GPU, the CPU and GPU can cooperate to execute the audio processing method provided in this application embodiment. For example, some algorithms in the audio processing method are executed by the CPU, and other algorithms are executed by the GPU to obtain faster processing efficiency.

[0081] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays screens 194, where N is a positive integer greater than 1. Display screen 194 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs). For example, display screen 194 can display photos, videos, web pages, or documents. As another example, display screen 194 can display the user control interface described in the embodiments of this application (e.g., a projection mode selection interface, a projection device selection interface, etc.).

[0082] In this embodiment of the application, the display screen 194 can be a single flexible display screen, or it can be a splicing display screen composed of two rigid screens and a flexible screen located between the two rigid screens.

[0083] Camera 193 (a front-facing camera or a rear-facing camera, or a single camera that can function as both) is used to capture still images or videos. Typically, camera 193 may include a photosensitive element such as a lens assembly and an image sensor. The lens assembly includes multiple lenses (convex or concave lenses) for collecting light signals reflected from the object being photographed and transmitting these signals to the image sensor. The image sensor generates a raw image of the object being photographed based on the light signals.

[0084] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area can store the operating system, application program code (such as the functions corresponding to the scheme in this application), etc. The data storage area can store data created during the use of the electronic device 100.

[0085] The internal memory 121 may also store one or more computer programs corresponding to the algorithm of this application. The one or more computer programs are stored in the internal memory 121 and configured to be executed by one or more processors 110. The one or more computer programs include instructions that can be used to perform the various steps in the following embodiments.

[0086] In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0087] Of course, the algorithm code of the embodiment of this application can also be stored in external memory. In this case, the processor 110 can run the algorithm code of the embodiment of this application stored in external memory through the external memory interface 120.

[0088] The sensor module 180 may include a gyroscope sensor, an accelerometer sensor, a proximity sensor, a fingerprint sensor, a touch sensor, etc.

[0089] A touch sensor, also known as a "touch panel," can be located on the display screen 194. The touch sensor and display screen 194 together form a touch display screen, also called a "touch screen." The touch sensor detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor may also be located on the surface of the electronic device 100, in a different position than the display screen 194.

[0090] For example, the display screen 194 of the electronic device 100 displays a main interface, which includes icons for multiple applications (such as a camera application). For instance, a user can tap the camera application icon on the main interface using a touch sensor, triggering the processor 110 to launch the camera application and open the camera 193. The display screen 194 then displays the camera application's interface, such as a viewfinder.

[0091] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0093] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device. In this embodiment, the mobile communication module 150 can also be used for information interaction with other devices.

[0094] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0095] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (WiFi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2. In this embodiment, the wireless communication module 160 is used to establish connections with other electronic devices for data interaction. Alternatively, the wireless communication module 160 can be used to access access point devices, send control commands to other electronic devices, or receive data from other electronic devices.

[0096] In addition, the electronic device 100 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor, such as music playback and recording. The electronic device 100 can receive input from buttons 190, generating key signal inputs related to user settings and function control. The electronic device 100 can use a motor 191 to generate vibration alerts (such as vibration alerts for incoming calls). The indicator 192 in the electronic device 100 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. The SIM card interface 195 in the electronic device 100 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100.

[0097] It should be understood that, in practical applications, electronic device 100 may include more than Figure 2The number of more or fewer components shown is not limited in the embodiments of this application. The illustrated electronic device 100 is merely an example, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0098] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. A layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, ... Figure 3 As shown, the software architecture can be divided into four layers, from top to bottom: the application layer, the application framework layer (framework, FWK), the runtime and system libraries, and the (Linux) kernel layer.

[0099] The application layer is the top layer of the operating system and includes native operating system applications such as camera, gallery, calendar, Bluetooth, music, video, and messaging applications, as well as third-party applications. The applications discussed in this application embodiment are referred to as applications (APPs), which are software programs capable of performing one or more specific functions. Typically, multiple applications can be installed on an electronic device, such as camera applications and email applications. The applications mentioned below can be system applications pre-installed on the electronic device at the factory, or third-party applications downloaded by the user from the network or obtained from other electronic devices during the use of the electronic device.

[0100] Of course, for developers, they can write applications and install them into this layer. In one possible implementation, the application can be developed using the Java language, by calling the application programming interface (API) provided by the application framework layer. Developers can then interact with the underlying operating system (such as the kernel layer) through the application framework to develop their own applications.

[0101] The application framework layer provides the API and programming framework for the application layer. It can include predefined functions. The application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0102] The window manager is used to manage window programs. The window manager can obtain the screen size, determine if a status bar is present, lock the screen (or display), and capture the screen, among other things.

[0103] Content providers are used to store and retrieve data, and make that data accessible to applications. This data may include files (e.g., documents, videos, images, audio), text, and other information.

[0104] A view system includes visual controls, such as controls that display text, images, documents, and other content. View systems can be used to build applications. An interface in a display window can consist of one or more views. For example, a display interface including a text message notification icon could include a view that displays text and a view that displays images.

[0105] The phone manager provides communication functionality for electronic devices. The notification manager allows applications to display notification information in the status bar; it can be used to convey informative messages and can disappear automatically after a short pause without user interaction.

[0106] The runtime includes the core libraries and the virtual machine. The runtime is responsible for system scheduling and management.

[0107] The system's core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part is the system's core library. The application layer and application framework layer run in a virtual machine. Taking Java as an example, the virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0108] The system library can include multiple functional modules. Examples include: a surface manager, a media library, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), and an image processing library. The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.564, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics.

[0109] The kernel layer provides core system services for the operating system, such as security, memory management, process management, network protocol stack, and driver models, all of which are implemented based on the kernel layer. The kernel layer also serves as an abstraction layer between the hardware and software stacks. This layer contains many drivers related to electronic devices, including: display drivers; keyboard drivers as input devices; Flash drivers for memory-based devices; camera drivers; audio drivers; Bluetooth drivers; and WiFi drivers.

[0110] It is important to understand that the functional services described above are just an example. In practical applications, electronic devices may be divided into more or fewer functional services based on other factors, or the functions of each service may be divided in other ways, or they may not be divided into functional services but work as a whole.

[0111] The solutions provided in the embodiments of this application will be described in detail below.

[0112] The solution provided in this application can be applied to audio transmission and reception scenarios, such as VoIP call scenarios. For example, VoIP call scenarios may include MeeTime calls, voice calls via social media software, etc., and this application does not impose specific limitations.

[0113] Figure 4 This is a schematic diagram of the architecture of an audio processing system provided in an embodiment of this application. This system can be deployed in an electronic device. Figure 4 As shown, the audio processing system includes at least a VoIP application, a DSP chip, and device nodes deployed in the kernel layer of electronic devices.

[0114] VoIP applications can be applications installed on electronic devices for making VoIP calls. Device nodes can act as data transmission channels between the DSP chip and the VoIP application. Specifically, device nodes can send audio data from the DSP chip to the VoIP application, or vice versa.

[0115] In some embodiments of this application, a device node may include two data queues, which can be used to transmit uplink data streams and downlink data streams, respectively. The uplink data stream can be a data stream from a DSP chip to a VoIP application, and the downlink data stream can be a data stream from a VoIP application to a DSP chip.

[0116] In some embodiments of this application, the device node can also be used to transmit control information. Optionally, the control information can be control information for controlling the start and stop of DSP chip acquisition and playback. As an optional implementation, the audio processing system may include multiple device nodes, wherein different data nodes are used to transmit different types of data. For example, the audio processing system may include two device nodes, one of which can be used to transmit audio data streams, and the other device node can be used to transmit control information streams (or control data streams). As another optional implementation, the audio processing system may include one device node. This device node can transmit both audio data streams and control information streams. The header of the audio data and / or control information includes identification information for identifying the data type (e.g., it may include uplink or downlink data types, audio data types or control data types, etc.), and the components in the audio processing system can identify different types of data based on the identification information.

[0117] For example, such as Figure 4 As shown in the downlink control link, VoIP applications can control the DSP chip by sending control information to the DSP chip through the device node.

[0118] Optionally, the audio processing system may also include at least one of the following: a communication unit, an audio acquisition device, and an audio playback device.

[0119] In some embodiments of this application, the audio acquisition device, DSP chip, device node, VoIP application, and communication unit can form an uplink audio link (i.e., a link for sending audio, also known as an audio transmission link), which can be used to send out the audio acquired by the audio acquisition device. The communication unit, VoIP application, device node, DSP chip, and audio playback device can form a downlink audio link (i.e., a link for receiving audio, also known as an audio reception link), which can be used to receive and play audio.

[0120] In this embodiment, the audio acquisition device can be used to acquire audio from the environment, such as user audio, and send the acquired audio data to the DSP chip. For example, the audio acquisition device can be a microphone, etc.

[0121] The data transmitted on the uplink audio link can be called uplink data, and the data transmitted on the downlink audio link can be called downlink data.

[0122] In this embodiment, the audio playback device can be used to receive audio data from the DSP chip and play the corresponding audio. For example, the audio playback device can be a speaker, headphones, etc.

[0123] In some embodiments of this application, the communication unit can be used to send or receive audio data. For example, the communication unit can be a modem, a WiFi chip, or other devices that can send and receive data over a network.

[0124] In the first possible solution, for the uplink audio link, the DSP chip can be used to receive audio data from the audio acquisition device and send the received audio data to the device node. The device node can be used to send the audio data from the DSP chip to the VoIP application. The VoIP application can be used to receive the audio data from the device node, perform audio processing, encoding, and packetization on the received audio data, and send the processed audio data (data packet format) to the communication unit. The audio processing performed by the VoIP application may include 3A enhancement processing, etc. The communication unit can be used to send the audio data from the VoIP application out over the network. For the downlink audio link, the communication unit can be used to receive audio data over the network and send the received audio data to the VoIP application. The VoIP application can be used to receive the audio data from the communication unit, perform unpacking, jitter reduction, decoding, and audio processing on the received audio data, and send the processed audio data to the device node. The device node can be used to send the audio data from the VoIP application to the DSP chip. The DSP chip can be used to send the audio data to the audio playback device for playback.

[0125] In the second possible solution, such as Figure 4 As shown, for the uplink audio link, the DSP chip can receive audio data from the audio acquisition device, perform audio processing, encoding, and packaging on the received audio data, and send the processed audio data (data packet format) to the device node. The device node can then send the audio data from the DSP chip to the VoIP application. The VoIP application can then send the audio data from the device node to the communication unit. The communication unit can then transmit the audio data from the VoIP application over the network. For the downlink audio link, the communication unit can receive audio data over the network and send the received audio data to the VoIP application. The VoIP application can receive audio data (data packet format) from the communication unit and send the received audio data to the device node. Optionally, the VoIP application can receive audio data from the communication unit through a forwarding service and send the received audio data to the device node. The device node can then send the audio data from the VoIP application to the DSP chip. DSP chips can be used to unpack, de-jitter, decode, and process received audio data, and then send the processed audio data to an audio playback device for playback.

[0126] Optional, such as Figure 4 As shown in the diagram, in the above scheme, when a VoIP application sends data to a communication unit, the data can be transparently transmitted to the electronic device's kernel layer through the application framework layer and HAL layer. The data can then be passed to the communication unit through the IP stack, smart packet engine (SPE), and IP filtering (IPF) services (used for filtering uplink and downlink data and inter-kernel data transfer) within the kernel layer. When the communication unit sends data to a VoIP application, the data can be transmitted to the VoIP application through the IPF, SPE, and IP stack within the kernel layer, as well as through the transparent transmission via the HAL layer and application framework layer.

[0127] In some embodiments of this application, Figure 4 The VoIP application in the audio processing system shown can be deployed in the application layer of the electronic device, and the device node can be deployed in the kernel layer of the electronic device. The data of the VoIP application can be passed through the application framework layer and HAL layer of the electronic device to the device node, and the data of the device node can be passed through the HAL layer and application framework layer of the electronic device to the VoIP application.

[0128] It is important to understand that the system architecture of the audio processing system described above is just an example. In practical applications, the audio processing system may include more or fewer functional modules (or services), or the functions of each module may be divided in other ways, or it may not be divided into functional modules but work as a whole.

[0129] Based on the above solutions, this application provides an audio transmission method that can be applied to the aforementioned audio processing system, such as... Figure 5 As shown, the execution flow of this method may include:

[0130] S501: An audio acquisition device in an electronic device acquires audio data and encodes the acquired audio data into a PCM audio data stream.

[0131] In one alternative implementation, the audio acquisition device may use an audio encoder to encode the acquired audio data into a pulse code modulation (PCM) audio data stream.

[0132] S502: The audio acquisition device sends the PCM audio data stream to the DSP chip in the electronic device.

[0133] S503: The DSP chip performs 3A processing, encoding, and packaging on the PCM audio data stream to obtain audio data packets.

[0134] Specifically, the DSP chip performs 3A processing on the PCM audio data stream, which can be at least one of the 3A processing methods. For example, the DSP chip can perform AEC processing on the PCM audio data stream.

[0135] In some embodiments of this application, after the DSP chip performs 3A processing on the PCM audio data stream, it can encode the obtained audio data stream into a format suitable for transmission over a network (such as Opus, Advanced Audio Coding (AAC) format, etc.), and then package the encoded audio data stream to obtain an audio data packet.

[0136] Optionally, the DSP chip can compress the audio data stream before encoding it, and then encode the compressed audio data stream.

[0137] In one alternative implementation, the DSP chip can package the encoded audio data stream into RTP packets or real-time transport control protocol (RTCP) packets.

[0138] S504: The DSP chip sends audio data packets to the device node in the kernel layer of the electronic device.

[0139] S505: VoIP applications in electronic devices obtain audio data packets from device nodes.

[0140] S506: VoIP applications send audio data packets to the communication unit of electronic devices.

[0141] S507: The communication unit sends audio data packets.

[0142] Optionally, the communication unit can send audio data packets to a server or other electronic devices via a network.

[0143] Based on the above solutions, this application also provides an audio receiving method, which can be applied to the aforementioned audio processing system, such as... Figure 6 As shown, the execution flow of this method may include:

[0144] S601: The communication unit of the electronic device receives audio data packets.

[0145] Optionally, the communication unit can receive audio data packets from a server or other electronic devices via a network.

[0146] S602: The communication unit sends audio data packets to the VoIP application in the electronic device.

[0147] S603: VoIP applications send audio data packets to device nodes in the kernel layer of electronic devices.

[0148] S604: The DSP chip in the electronic device obtains audio data packets from the device node.

[0149] S605: The DSP chip unpacks audio data packets to obtain an audio data stream, decodes the audio data stream into a PCM audio data stream, and performs 3A processing on the PCM audio data stream.

[0150] Specifically, when the audio data packet is in RTP format, the DSP chip can disassemble the audio data packet based on the RTP protocol; when the audio data packet is in RTCP format, the DSP chip can disassemble the audio data packet based on the RTCP protocol.

[0151] In some embodiments of this application, after the DSP chip unpacks the audio data packets to obtain the audio data stream, before decoding the audio data stream into a PCM audio data stream, the AJB algorithm can be used to perform anti-jitter processing on the audio data stream.

[0152] In one alternative implementation, the DSP chip can use an audio decoder to decode the audio data stream into a PCM audio data stream.

[0153] The DSP chip performs 3A processing on the PCM audio data stream, specifically at least one of the 3A processing methods. For example, the DSP chip could perform AEC processing on the PCM audio data stream.

[0154] Optionally, when the audio data stream is compressed, the DSP chip can decompress it before decoding and other processing.

[0155] S606: The DSP chip sends the PCM audio data stream processed by 3A to the audio playback device in the electronic device.

[0156] S607: The audio playback device plays the received PCM audio data stream.

[0157] The above Figure 5 or Figure 6In the method described, in addition to the transmission of audio data, control of the DSP chip is also required. Since existing VoIP calls on the Android platform are based on an audio framework, and the VoIP uplink and downlink audio links provided in the above embodiments of this application bypass the application layer and HAL layer, the audio framework cannot be used, necessitating the design of a new control channel. Therefore, as described in the above method, the electronic device can create at least one device node in the kernel layer for transmitting audio data and control information between the DSP chip and the VoIP application. At least one device node can serve as a transmission channel for uplink and downlink audio data and / or uplink and downlink control information, wherein each device node can be used to transmit at least one type of data. When a device node is used to transmit multiple types of data, different types of data can be distinguished by identification information added to the data header to identify the data type. For uplink data (including uplink audio data and / or uplink control information), the DSP chip can send the uplink data to the device node in the at least one device node used for transmitting uplink data, and the VoIP application can read the uplink data from that device node. For downlink data (including downlink audio data and / or downlink control information), VoIP applications can send downlink data to at least one device node used for transmitting downlink data, and the DSP chip can read the downlink data from the device node.

[0158] For example, in one possible implementation, the electronic device can create a first device node for transmitting uplink and downlink audio data and a second device node for transmitting uplink and downlink control information in the kernel layer. Figure 5 or Figure 6 The device node in the illustrated method can be a first device node. Furthermore, when a VoIP application controls the start and stop of the DSP chip's acquisition and playback (i.e., acquisition and playback), it can send the corresponding control information to a second device node. The DSP chip can then read the control information from the second device node and process it accordingly. In another possible implementation, the electronic device can create a device node in the kernel layer for transmitting uplink and downlink audio data and uplink and downlink control information. Figure 5 or Figure 6The device node in the illustrated method can be this device node. In this scenario, uplink and downlink audio data and uplink and downlink control information can be distinguished by the identifier header. That is, the electronic device can add identifier information to the header of audio data and control information to indicate the data type, thereby distinguishing different types of data. The identifier information corresponding to uplink audio data, downlink audio data, uplink control information, and downlink control information is different. For example, the identifier information corresponding to uplink audio data (e.g., audio data sent from the DSP chip to the device node and from the device node to the VoIP application) can be 0xDF50. The identifier information corresponding to downlink audio data (e.g., audio data sent from the VoIP application to the device node and from the device node to the DSP chip) is 0xDF51. The identifier information corresponding to uplink control information (e.g., control information sent from the DSP chip to the device node and from the device node to the VoIP application) can be 0xDF52. The identifier information corresponding to downlink control information (e.g., control information sent from the VoIP application to the device node and from the device node to the DSP chip) is 0xDF53. Based on this method, each module or device in the electronic device (e.g., VoIP application, DSP chip, device node, etc.) can accurately identify different types of data. In another possible implementation, the electronic device can create a first device node for transmitting uplink audio data, a second device node for transmitting downlink audio data, a third device node for transmitting uplink control information, and a fourth device node for transmitting downlink control information in the kernel layer. Figure 5 The device node in the method shown can be the first device node, as described above. Figure 6 The device node in the method shown can be a second device node. Furthermore, when VoIP applications control the start and stop of DSP chip acquisition and broadcasting, they can send corresponding control information to a fourth device node. The DSP chip can then read the control information from the fourth device node and process it accordingly.

[0159] The above method enables bidirectional communication between the DSP chip and the AP side through device nodes. VoIP applications and DSP chips can directly read and write data from the device nodes, thus simplifying the complex interaction process that traditional VoIP applications and DSP chips need to perform with the application framework layer and HAL layer, thereby improving processing efficiency.

[0160] Considering the increasing maturity of 3A processing, encoding / decoding, and AJB algorithms in current DSP chips, and the fact that a significant portion of VoIP call latency on the Android platform originates from audio processing at the application framework layer and HAL layer, the above method does not utilize the application framework layer and HAL layer for audio processing. Instead, it leverages the DSP chip to complete audio processing tasks in either the VoIP uplink or downlink audio link. This approach leverages the existing audio processing capabilities of the DSP chip to quickly and efficiently complete audio processing tasks, while simultaneously reducing processing latency at the application layer and HAL layer, thus lowering the computational power consumption on the AP side (large core) and consequently reducing the power consumption of the large core. It also reduces the number of buffers used for audio stream processing at the application framework layer and HAL layer, thereby reducing the demand for large core storage space. Experiments have verified that, without reducing the mean opinion score (MOS), the average end-to-end latency of VoIP calls implemented based on the above scheme can be reduced by more than 250 milliseconds (i.e., more than 50%) under different network quality conditions, which is noticeably perceptible to users.

[0161] Furthermore, the above method removes the 3A processing, encoding and decoding, and other processing schemes performed in VoIP applications in existing solutions. Therefore, VoIP applications are mainly used for forwarding audio data streams, and the main audio processing algorithms in the uplink and downlink audio links are concentrated in the DSP chip. This ensures that there is basically only one set of audio processing algorithms in the DSP chip on the entire link, which can improve the efficiency of audio transmission and reception, reduce the power consumption of VoIP applications, and improve the overall performance of VoIP applications.

[0162] It should be noted that the specific implementation processes provided in the above embodiments are merely illustrative examples of the applicable method processes in the embodiments of this application. The execution order of each step can be adjusted according to actual needs, and other steps can be added or some steps can be reduced. The specific implementation methods of each step can be referred to the relevant descriptions in the foregoing embodiments, and will not be detailed here.

[0163] Based on the above embodiments and the same technical concept, this application also provides an audio processing method, which can be applied to an electronic device. The electronic device includes a DSP chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. The second audio processing unit is located in the application framework layer of the electronic device, the third audio processing unit is located in the hardware abstraction layer of the electronic device, and the device node is located in the kernel layer of the electronic device.

[0164] In the above scheme, the first audio processing unit, the second audio processing unit, the third audio processing unit, and the fourth audio processing unit can have audio processing capabilities, such as supporting 3A processing and anti-jitter processing of audio, etc., which will not be listed one by one in this embodiment. The first encoding / decoding unit and the second encoding / decoding unit can have audio encoding and audio decoding functions. The first packet processing unit and the second packet processing unit can have data packing and unpacking functions.

[0165] In one example, the AP system can be Figure 4 The system on the AP side shown; the first application can be Figure 4 In the VoIP application shown, the transceiver unit can be... Figure 4 The forwarding unit shown in the VoIP application; the DSP chip can be Figure 4 The DSP chip shown; the second audio processing unit may include Figure 4 The third audio processing unit may include the audio tracks, audio frameworks, and other functional services shown. Figure 4 The 3A, sound card driver and other functional services shown are illustrated.

[0166] like Figure 7 As shown, the audio processing method may include:

[0167] S701: The DSP chip acquires the first audio data.

[0168] Optionally, the first audio data may be audio data collected by the audio acquisition device in the electronic device described in the foregoing embodiments.

[0169] S702: The DSP chip processes the first audio data through the first audio processing unit to obtain the second audio data, encodes the second audio data through the first encoding / decoding unit to obtain the third audio data, and packages the third audio data through the first packet processing unit to obtain the fourth audio data; wherein, the audio processing includes at least one of the following: acoustic echo cancellation, background noise suppression, and automatic gain control.

[0170] Optionally, the fourth audio data may be an audio data packet processed by the DSP chip in the electronic device described in the foregoing embodiments.

[0171] S703: The DSP chip sends the fourth audio data to the device node.

[0172] S704: The first application obtains the fourth audio data from the device node through the transceiver unit.

[0173] S705: The first application sends fourth audio data through the transceiver unit.

[0174] Optionally, the electronic device may further include a communication unit. The first application can send the fourth audio data via the transceiver unit by sending the fourth audio data to the communication unit of the electronic device. The communication unit can then send the fourth audio data to other electronic devices. Specific details can be found in the descriptions in the foregoing embodiments and will not be repeated here.

[0175] In the above scheme, the fourth audio data does not pass through the second and third audio processing units during transmission. Therefore, the latency caused by audio processing at the application card frame layer and HAL layer can be avoided.

[0176] In the above scheme, the fourth audio data does not pass through the fourth audio processing unit, the second encoding / decoding unit, and the second packet processing unit. Therefore, the latency caused by audio processing in the first application can be avoided, while reducing the power consumption of the first application and improving the overall performance of the first application.

[0177] In the above solution, the audio processing of audio data is mainly completed in the DSP chip. It can use the audio processing capabilities of the current DSP chip itself to complete the main audio processing tasks in the audio transmission and reception process of electronic devices, thereby improving the overall processing efficiency.

[0178] In some embodiments of this application, the device node described in the above method may correspond to the data type of the fourth audio data; optionally, the data type of the fourth audio data may be an audio type or an uplink audio type. The AP system may also include a first control information processing unit and a second control information processing unit, wherein the first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device; the first control information processing unit and the second control information processing unit may be used to process and transmit control information.

[0179] For example, the first control information processing unit may include Figure 4 The audio frame shown.

[0180] The AP system may further include a device node corresponding to the data type of the first control information. This device node functions similarly to the device node described in the above embodiments, except that it transmits a different data type. The audio processing method may further include: a first application generating first control information for controlling a DSP chip to perform a first operation; the first application sending the first control information to the device node corresponding to the data type of the first control information; wherein the device node corresponding to the data type of the first control information is located in the kernel layer of the electronic device; the DSP chip obtaining the first control information from the device node corresponding to the data type of the first control information; and the DSP chip performing the first operation according to the first control information; wherein the first control information does not pass through a first control information processing unit or a second control information processing unit. Optionally, the data type of the first control information may be a control type or an uplink control type.

[0181] For example, the first operation can be the start / stop operation of seeding and broadcasting as described in the foregoing embodiments.

[0182] In other embodiments of this application, the AP system may further include a first control information processing unit and a second control information processing unit. The first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device. The first control information processing unit and the second control information processing unit can be used to process and transmit control information.

[0183] For example, the first control information processing unit may include Figure 4 The audio frame shown.

[0184] The above audio processing method may further include: a first application generating first control information for controlling a DSP chip to perform a first operation;

[0185] The first application sends the first control information to the device node; the DSP chip obtains the first control information from the device node; the DSP chip executes the first operation according to the first control information; wherein the first control information does not pass through the first control information processing unit and the second control information processing unit.

[0186] For example, the first operation can be the start / stop operation of seeding and broadcasting as described in the foregoing embodiments.

[0187] In some embodiments of this application, based on the above method, the DSP chip can send the fourth audio data to the device node by sending first data including the fourth audio data and first identification information to the device node; wherein, the first identification information is used to indicate the data type of the fourth audio data. The first application can send the first control information to the device node by sending second data including the first control information and second identification information to the device node; wherein, the second identification information is used to indicate the data type of the first control information. Optionally, the data type of the fourth audio data can be an audio type or an uplink audio type, and the data type of the first control information can be a control type or an uplink control type. Based on this method, the device node can distinguish between uplink audio data and uplink control information through the identification information.

[0188] Based on the above method, the first application can obtain the fourth audio data from the device node through the transceiver unit in the following ways: the first application obtains the first data from the device node through the transceiver unit, and determines the fourth audio data based on the first data. The DSP chip can obtain the first control information from the device node in the following ways: the DSP chip obtains the second data from the device node, and determines the first control information based on the second data.

[0189] The functional services in the electronic device provided in the above embodiments can be referred to in the foregoing embodiments. Figure 1 or Figure 4 For the relevant explanations in the above, the specific implementation methods corresponding to the above methods can be referred to the description of the audio sending method provided in the foregoing embodiments, which will not be repeated here.

[0190] Based on the above embodiments and the same technical concept, this application also provides an audio processing method, which can be applied to an electronic device. The electronic device includes a DSP chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. The second audio processing unit is located in the application framework layer of the electronic device, the third audio processing unit is located in the hardware abstraction layer of the electronic device, and the device node is located in the kernel layer of the electronic device.

[0191] In the above scheme, the first audio processing unit, the second audio processing unit, the third audio processing unit, and the fourth audio processing unit can have audio processing capabilities, such as supporting 3A processing and anti-jitter processing of audio, etc., which will not be listed one by one in this embodiment. The first encoding / decoding unit and the second encoding / decoding unit can have audio encoding and audio decoding functions. The first packet processing unit and the second packet processing unit can have data packing and unpacking functions.

[0192] In one example, the AP system can be Figure 4 The system on the AP side shown; the first application can be Figure 4 In the VoIP application shown, the transceiver unit can be... Figure 4 The forwarding unit shown in the VoIP application; the DSP chip can be Figure 4 The DSP chip shown; the second audio processing unit may include Figure 4 The third audio processing unit may include the audio tracks, audio frameworks, and other functional services shown. Figure 4 The 3A, sound card driver and other functional services shown are illustrated.

[0193] like Figure 8 As shown, the audio processing method may include:

[0194] S801: The first application receives the first audio data through the transceiver unit.

[0195] Optionally, the electronic device may further include a communication unit, which can be used to receive audio data packets from other electronic devices. Refer to the description in the foregoing embodiments for details, which will not be repeated here. The first audio data may be an audio data packet received by the communication unit in the electronic device described in the foregoing embodiments. The first application may receive the first audio data through the transceiver unit by receiving the first audio data from the communication unit through the transceiver unit.

[0196] S802: The first application sends the first audio data to the device node through the transceiver unit.

[0197] S803: The DSP chip obtains the first audio data from the device node.

[0198] S804: The DSP chip depackets the first audio data through the first packet processing unit to obtain the second audio data, decodes the second audio data through the first encoding / decoding unit to obtain the third audio data, and processes the third audio data through the first audio processing unit to obtain the fourth audio data; wherein, the audio processing includes at least one of the following: acoustic echo cancellation, background noise suppression, and automatic gain control.

[0199] Optionally, after the DSP chip obtains the fourth audio data, it can send the fourth audio data to an audio playback device in the electronic device, and the audio playback device can play the fourth audio data.

[0200] In the above scheme, the first audio data does not pass through the second and third audio processing units during transmission. Therefore, the latency caused by audio processing at the application card frame layer and HAL layer can be avoided.

[0201] In the above scheme, the first audio data does not pass through the fourth audio processing unit, the second encoding / decoding unit, and the second packet processing unit. Therefore, the latency caused by audio processing in the first application can be avoided, while the power consumption of the first application can be reduced, and the overall performance of the first application can be improved.

[0202] In the above solution, the audio processing of audio data is mainly completed in the DSP chip. It can use the audio processing capabilities of the current DSP chip itself to complete the main audio processing tasks in the audio transmission and reception process of electronic devices, thereby improving the overall processing efficiency.

[0203] In some embodiments of this application, the device node described in the above method may correspond to the data type of the first audio data; optionally, the data type of the fourth audio data may be an audio type or an uplink audio type. The AP system may also include a first control information processing unit and a second control information processing unit, wherein the first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device; the first control information processing unit and the second control information processing unit may be used to process and transmit control information.

[0204] For example, the first control information processing unit may include Figure 4 The audio frame shown.

[0205] The AP system may further include a device node corresponding to the data type of the first control information. This device node functions similarly to the device node described in the above embodiments, except that it transmits a different data type. The audio processing method may further include: a first application generating first control information for controlling a DSP chip to perform a first operation; the first application sending the first control information to the device node corresponding to the data type of the first control information; wherein the device node corresponding to the data type of the first control information is located in the kernel layer of the electronic device; the DSP chip obtaining the first control information from the device node corresponding to the data type of the first control information; and the DSP chip performing the first operation according to the first control information; wherein the first control information does not pass through a first control information processing unit or a second control information processing unit.

[0206] For example, the first operation can be the start / stop operation of seeding and broadcasting as described in the foregoing embodiments.

[0207] In other embodiments of this application, the AP system may further include a first control information processing unit and a second control information processing unit. The first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device. The first control information processing unit and the second control information processing unit can be used to process and transmit control information.

[0208] For example, the first control information processing unit may include Figure 4 The audio frame shown.

[0209] The above audio processing method may further include: a first application generating first control information for controlling a DSP chip to perform a first operation; the first application sending the first control information to a device node; the DSP chip obtaining the first control information from the device node; and the DSP chip performing the first operation according to the first control information; wherein the first control information does not pass through a first control information processing unit and a second control information processing unit.

[0210] For example, the first operation can be the start / stop operation of seeding and broadcasting as described in the foregoing embodiments.

[0211] In some embodiments of this application, based on the above method, the first application can send first audio data to the device node via the transceiver unit by sending first data including first audio data and first identification information to the device node via the transceiver unit; wherein, the first identification information is used to indicate the data type of the first audio data. The first application can send first control information to the device node by sending second data including first control information and second identification information to the device node; wherein, the second identification information is used to indicate the data type of the first control information. The data type of the first audio data can be an audio type or an uplink audio type, and the data type of the first control information can be a control type or an uplink control type. Based on this method, the device node can distinguish between downlink audio data and downlink control information through the identification information.

[0212] Based on the above method, the DSP chip can obtain the first audio data from the device node in the following ways: the DSP chip obtains the first data from the device node and determines the first audio data based on the first data. The DSP chip can also obtain the first control information from the device node in the following ways: the DSP chip obtains the second data from the device node and determines the first control information based on the second data.

[0213] The functional services in the electronic device provided in the above embodiments can be referred to in the foregoing embodiments. Figure 1 or Figure 4 For the relevant explanations in the above, the specific implementation methods corresponding to the above methods can be referred to the description of the audio receiving method provided in the foregoing embodiments, which will not be repeated here.

[0214] Based on the above embodiments and the same technical concept, this application also provides an electronic device for implementing the audio processing method provided in this application. Figure 9As shown, the electronic device 900 may include: a memory 901, one or more processors 902, and one or more computer programs (not shown). These devices may be coupled via one or more communication buses 903. Optionally, the electronic device 900 may also include a display screen 904.

[0215] The memory 901 stores one or more computer programs (code), and the one or more computer programs include computer instructions; one or more processors 902 call the computer instructions stored in the memory 901, causing the electronic device 900 to execute the audio processing method provided in the embodiments of this application.

[0216] In a specific implementation, memory 901 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 901 may store an operating system (hereinafter referred to as the system), such as embedded operating systems like Android, iOS, Windows, or Linux. Memory 901 can be used to store implementation programs of the embodiments of this application. Memory 901 may also store network communication programs, which can be used to communicate with one or more additional devices, one or more user devices, or one or more network devices.

[0217] One or more processors 902 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs according to the present application.

[0218] Display screen 904 is used to display application interfaces and related user interfaces, such as the application interface of a VoIP application.

[0219] It should be noted that, Figure 9 This is merely one implementation of the electronic device 900 provided in this application embodiment. In practical applications, the electronic device 900 may include more or fewer components, as detailed in the following references. Figure 2 The specific structure and description shown are not limited here.

[0220] Based on the above embodiments and the same technical concept, this application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method provided in the above embodiments.

[0221] Based on the above embodiments and the same technical concept, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are run on an electronic device, the electronic device performs the method provided in the above embodiments.

[0222] Based on the above embodiments and the same technical concept, this application also provides a chip system, which includes a processor and a memory, wherein the memory stores instructions; when the instructions are executed by the processor, they can implement the methods provided in the above embodiments. This chip system can be composed of chips, or it can include chips and other discrete devices.

[0223] The methods provided in this application can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs), or semiconductor media (e.g., SSDs), etc.

[0224] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An audio processing method applied to an electronic device, characterized in that, The electronic device includes a digital signal processing (DSP) chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. The method includes: The DSP chip acquires the first audio data; The DSP chip processes the first audio data through the first audio processing unit to obtain second audio data, encodes the second audio data through the first encoding / decoding unit to obtain third audio data, and packages the third audio data through the first packet processing unit to obtain fourth audio data; wherein, the audio processing includes at least one of the following: acoustic echo cancellation, background noise suppression, and automatic gain control; The DSP chip sends the fourth audio data to the device node; The first application obtains the fourth audio data from the device node through the transceiver unit; The first application sends the fourth audio data through the transceiver unit.

2. The method as described in claim 1, characterized in that, The second audio processing unit is located in the application framework layer of the electronic device, the third audio processing unit is located in the hardware abstraction layer of the electronic device, and the fourth audio data does not pass through the second audio processing unit and the third audio processing unit.

3. The method as described in claim 1 or 2, characterized in that, The fourth audio data does not pass through the fourth audio processing unit, the second encoding / decoding unit, and the second packet processing unit.

4. The method according to any one of claims 1 to 3, characterized in that, The device node is located in the kernel layer of the electronic device.

5. The method according to any one of claims 1 to 4, characterized in that, The AP system further includes a first control information processing unit and a second control information processing unit, wherein the first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device; the method further includes: The first application generates first control information for controlling the DSP chip to perform a first operation; The first application sends the first control information to the device node; The DSP chip obtains the first control information from the device node; The DSP chip executes the first operation according to the first control information; The first control information does not pass through the first control information processing unit and the second control information processing unit.

6. The method as described in claim 5, characterized in that, The DSP chip sends the fourth audio data to the device node, including: The DSP chip sends first data, including the fourth audio data and first identification information, to the device node; wherein, the first identification information is used to indicate the data type of the fourth audio data; The first application obtains the fourth audio data from the device node through the transceiver unit, including: The first application obtains the first data from the device node through the transceiver unit, and determines the fourth audio data based on the first data; The first application sends the first control information to the device node, including: The first application sends second data, including the first control information and the second identification information, to the device node; wherein the second identification information is used to indicate the data type of the first control information; The DSP chip obtains the first control information from the device node, including: The DSP chip obtains the second data from the device node and determines the first control information based on the second data.

7. The method as described in claim 6, characterized in that, The data type of the fourth audio data is an audio type or an uplink audio type, and the data type of the first control information is a control type or an uplink control type.

8. The method according to any one of claims 1 to 4, characterized in that, The device node corresponds to the data type of the fourth audio data; the AP system further includes a first control information processing unit and a second control information processing unit, the first control information processing unit being located in the application framework layer of the electronic device, and the second control information processing unit being located in the hardware abstraction layer of the electronic device; the method further includes: The first application generates first control information for controlling the DSP chip to perform a first operation; The first application sends the first control information to a device node corresponding to the data type of the first control information; wherein the device node corresponding to the data type of the first control information is located in the kernel layer of the electronic device; The DSP chip obtains the first control information from the device node corresponding to the data type of the first control information; The DSP chip executes the first operation according to the first control information; The first control information does not pass through the first control information processing unit and the second control information processing unit.

9. An audio processing method applied to an electronic device, characterized in that, The electronic device includes a digital signal processing (DSP) chip and an application processing (AP) system. The DSP chip includes a first audio processing unit, a first encoding / decoding unit, and a first packet processing unit. The AP system includes a second audio processing unit, a third audio processing unit, a first application, and a device node. The first application includes a fourth audio processing unit, a second encoding / decoding unit, a second packet processing unit, and a transceiver unit. The method includes: The first application receives first audio data through the transceiver unit; The first application sends the first audio data to the device node through the transceiver unit; The DSP chip obtains the first audio data from the device node; The DSP chip unpacks the first audio data using a first packet processing unit to obtain a second audio data, decodes the second audio data using a first encoding / decoding unit to obtain a third audio data, and processes the third audio data using a first audio processing unit to obtain a fourth audio data; wherein, the audio processing includes at least one of the following: acoustic echo cancellation, background noise suppression, and automatic gain control.

10. The method as described in claim 9, characterized in that, The second audio processing unit is located in the application framework layer of the electronic device, and the third audio processing unit is located in the hardware abstraction layer of the electronic device. The first audio data does not pass through the second audio processing unit and the third audio processing unit.

11. The method as described in claim 9 or 10, characterized in that, The first audio data does not pass through the fourth audio processing unit, the second encoding / decoding unit, and the second packet processing unit.

12. The method according to any one of claims 9 to 11, characterized in that, The DSP chip further includes a first anti-jitter unit, and the first application further includes a second anti-jitter unit; after the DSP chip performs unpacking processing on the first audio data through the first packet processing unit to obtain the second audio, and before decoding the second audio through the first encoding / decoding unit to obtain the third audio, the method further includes: The second audio is processed to reduce jitter using the first anti-jitter unit; The first audio data does not pass through the second anti-jitter unit.

13. The method according to any one of claims 9 to 12, characterized in that, The device node is located in the kernel layer of the electronic device.

14. The method according to any one of claims 9 to 13, characterized in that, The AP system further includes a first control information processing unit and a second control information processing unit, wherein the first control information processing unit is located in the application framework layer of the electronic device, and the second control information processing unit is located in the hardware abstraction layer of the electronic device; the method further includes: The first application generates first control information for controlling the DSP chip to perform a first operation; The first application sends the first control information to the device node; The DSP chip obtains the first control information from the device node; The DSP chip executes the first operation according to the first control information; The first control information does not pass through the first control information processing unit and the second control information processing unit.

15. The method as described in claim 14, characterized in that, The first application sends the first audio data to the device node through the transceiver unit, including: The first application sends first data, including the first audio data and first identification information, to the device node through the transceiver unit; wherein, the first identification information is used to indicate the data type of the first audio data; The DSP chip obtains the first audio data from the device node, including: The DSP chip obtains the first data from the device node and determines the first audio data based on the first data; The first application sends the first control information to the device node, including: The first application sends second data, including the first control information and the second identification information, to the device node; wherein the second identification information is used to indicate the data type of the first control information; The DSP chip obtains the first control information from the device node, including: The DSP chip obtains the second data from the device node and determines the first control information based on the second data.

16. The method as described in claim 15, characterized in that, The data type of the first audio data is audio type or uplink audio type, and the data type of the first control information is control type or uplink control type.

17. The method according to any one of claims 9 to 13, characterized in that, The device node corresponds to the data type of the first audio data; the AP system further includes a first control information processing unit and a second control information processing unit, the first control information processing unit being located in the application framework layer of the electronic device, and the second control information processing unit being located in the hardware abstraction layer of the electronic device; the method further includes: The first application generates first control information for controlling the DSP chip to perform a first operation; The first application sends the first control information to a device node corresponding to the data type of the first control information; wherein the device node corresponding to the data type of the first control information is located in the kernel layer of the electronic device; The DSP chip obtains the first control information from the device node corresponding to the data type of the first control information; The DSP chip executes the first operation according to the first control information; The first control information does not pass through the first control information processing unit and the second control information processing unit.

18. An electronic device, characterized in that, The electronic device includes a display screen, a memory, and one or more processors; The memory is used to store computer program code, which includes computer instructions; when the computer instructions are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1 to 8, or performs the method as described in any one of claims 9 to 17.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 8, or the method as described in any one of claims 9 to 17.

Images