Method of processing a sound signal and electronic device

Abstract

Embodiments of the present application provide a method for processing a sound signal and an electronic device, the method comprising: a digital signal processor (DSP) outputting a second sound signal by preprocessing a first sound signal; the DSP transmitting the second sound signal to a neural network processor (NPU) through a central processing unit (CPU); receiving a third sound signal output by the NPU, wherein the third sound signal is a sound signal obtained by the NPU performing deep neural network processing on the second sound signal; and the DSP outputting a target sound signal by post-processing the third sound signal. Through the method and the electronic device, the DSP and the NPU in the terminal device can establish a transmission channel through the CPU, so that the terminal device can utilize the computing power resources of the DSP and the NPU, can improve the running performance of the device, and due to the calling of the deep neural network processing capability of the NPU, the processing capability of the device for the sound signal can be improved.

Description

Technical Field

[0001] This application relates to the field of processing sound signals, and more specifically, to a method and electronic device for processing sound signals. Background Technology

[0002] In voice call scenarios, people have high requirements for the sound quality of voice calls. Current sound processing methods mainly include two types: traditional signal processing methods and deep learning processing methods. Among them, traditional signal processing methods use signal modeling, and the models built are relatively simple. Moreover, some of the parameters on which the model is based are assumed parameters. When the assumed parameters are not actually true, it may lead to poor processing results. Compared with traditional signal processing methods, deep learning processing methods use more complex models. These models are learned and trained based on a large amount of signal data, and the sound processing effect is better. However, the processing process requires a lot of computing resources. Furthermore, for terminal devices, current technology makes it difficult to utilize the computing resources of the NPU, making it difficult to apply deep learning processing methods to terminal devices. Summary of the Invention

[0003] This application provides a method and electronic device for processing audio signals. Through this method and electronic device, the DSP and NPU in the terminal device can establish a transmission channel through the CPU, enabling the terminal device to utilize the computing resources of the DSP and NPU, thereby improving the device's operating performance. Furthermore, by utilizing the deep neural network processing capabilities of the NPU, the device's ability to process audio signals can be enhanced.

[0004] In a first aspect, a method for processing sound signals is provided, the method comprising: a digital signal processor (DSP) preprocessing a first sound signal to output a second sound signal; the DSP transmitting the second sound signal to a neural network processor (NPU) via a central processing unit (CPU); receiving a third sound signal output by the NPU, wherein the third sound signal is a sound signal obtained by the NPU performing deep neural network processing on the second sound signal; and the DSP postprocessing the third sound signal to output a target sound signal.

[0005] In some embodiments, the preprocessing includes one or more of echo cancellation, noise reduction, gain, and frequency response control.

[0006] In some embodiments, the deep neural network processing includes residual echo cancellation and / or noise reduction.

[0007] In some embodiments, the post-processing includes gain and frequency response control.

[0008] In this embodiment, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal device to utilize the computing resources of the DSP and NPU. For example, preprocessing and postprocessing can be performed on the DSP, while deep neural network processing can be performed on the NPU, which can improve the device's operating performance. Furthermore, by utilizing the deep neural network processing capabilities of the NPU, the device's ability to process audio signals can be improved, resulting in better audio signal processing performance.

[0009] In conjunction with the first aspect, in one possible implementation, the DSP transmits the second audio signal to the NPU via the CPU, including:

[0010] The DSP transmits the second audio signal to the CPU via shared memory, so that the CPU can transmit the second audio signal to the NPU by calling the NPU. The shared memory is shared by the DSP and the CPU.

[0011] In this embodiment, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal device to utilize the computing resources of the DSP and NPU, and further enables the terminal device to use the deep neural network processing capabilities of the NPU. In other words, it enables the deployment of the deep neural network processing model of the NPU on the terminal device, which can greatly improve the sound processing capabilities of the terminal device.

[0012] In conjunction with the first aspect, in one possible implementation, the method further includes: when the DSP receives the first audio signal of the (n+i)th frame, the DSP outputs the target audio signal of the i-th frame; when the DSP receives the first audio signal of the (n+i+1)th frame, the DSP outputs the target audio signal of the (i+1)th frame; where n is a positive integer less than or equal to 10, i is a natural number, i=1,2,3,...

[0013] In this embodiment, the DSP receives the first audio signal and outputs the target audio signal with a time delay. That is, the deep neural network processing of the NPU and the preprocessing of the DSP can be performed synchronously. However, for the same frame of audio signal, the deep neural network processing performed by the NPU is asynchronous to the preprocessing performed by the DSP. In this way, after the DSP processes the first audio signal of the current frame, it does not need to wait for the third audio signal corresponding to the current frame to be output by the NPU. Instead, it continues to process the first audio signal of the next frame. Within the time delay frame range, the DSP will receive the third audio signal corresponding to the current frame output by the NPU. The DSP only needs to perform post-processing and output the corresponding third audio signal after the time delay frame is reached. In this way, the audio signal can be continuously output even when the NPU buffer is insufficient, thereby meeting the requirements of continuity and real-time performance of audio output.

[0014] In a second aspect, a method for processing sound signals is provided, the method comprising: a neural network processor (NPU) receiving a second sound signal via a central processing unit (CPU), the second sound signal being a sound signal obtained by a digital signal processor (DSP) through preprocessing a first sound signal; the NPU performing deep neural network processing on the second sound signal to output a third sound signal; and the NPU transmitting the third sound signal to the DSP via the CPU.

[0015] In some embodiments, the preprocessing includes one or more of echo cancellation, noise reduction, gain, and frequency response control.

[0016] In some embodiments, the deep neural network processing includes residual echo cancellation and / or noise reduction.

[0017] In this embodiment, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal device to utilize the computing resources of the DSP and NPU. For example, preprocessing and postprocessing can be performed on the DSP, while deep neural network processing can be performed on the NPU, which can improve the device's operating performance. Furthermore, by utilizing the deep neural network processing capabilities of the NPU, the device's ability to process audio signals can be improved, resulting in better audio signal processing performance.

[0018] In conjunction with the second aspect, in one possible implementation, the NPU transmits the third audio signal to the DSP via the CPU, including: the NPU transmits the third audio signal to the DSP via the CPU and shared memory, wherein the shared memory is shared memory between the DSP and the CPU.

[0019] In this embodiment of the application, the DSP and NPU in the terminal system device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal system device to utilize the computing resources of both the DSP and NPU simultaneously, thereby enabling the terminal device to use the deep neural network processing capabilities of the NPU. In other words, it enables the deployment of the deep neural network processing model of the NPU on the terminal device, which can greatly improve the sound processing capabilities of the terminal device.

[0020] Thirdly, a method for processing audio signals is provided, the method comprising: when the screen of a first device displays a call interface, displaying a first prompt on the screen of the first device, the first prompt being used to prompt a user to confirm whether to enable intelligent noise reduction mode; or when the first device is in a call state, displaying a first notification in the notification bar of the first device, the first notification being used to prompt a user to confirm whether to enable intelligent noise reduction mode; in response to the user confirming the enabling of intelligent noise reduction mode, the first device performing the method of the first aspect or any possible implementation thereof, or performing the method of the second aspect or any possible implementation thereof.

[0021] In this embodiment of the application, a method for triggering deep processing of audio signals is provided to the user. When the user uses the call function of the terminal device, he / she can decide whether to enable the intelligent noise reduction mode according to the actual needs. When the user determines that the intelligent noise reduction mode needs to be enabled, the terminal device uses the computing power resources of DSP and NPU to perform deep processing of audio signals and output them.

[0022] Fourthly, an electronic device is provided, comprising a digital signal processor (DSP), a central processing unit (CPU), and a neural network processor (NPU), wherein: the DSP is used to preprocess a first audio signal to output a second audio signal; the DSP is also used to transmit the second audio signal to the NPU via the CPU; the NPU is used to perform deep neural network processing on the second audio signal to output a third audio signal; the NPU is also used to transmit the third audio signal to the DSP via the CPU; and the DSP is also used to post-process the third audio signal to output a target audio signal.

[0023] In some embodiments, the preprocessing includes one or more of echo cancellation, noise reduction, gain, and frequency response control.

[0024] In some embodiments, the deep neural network processing includes residual echo cancellation and / or noise reduction.

[0025] In some embodiments, the post-processing includes gain and frequency response control.

[0026] In this embodiment, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal device to utilize the computing resources of the DSP and NPU. For example, preprocessing and postprocessing can be performed on the DSP, while deep neural network processing can be performed on the NPU, which can improve the device's operating performance. Furthermore, by utilizing the deep neural network processing capabilities of the NPU, the device's ability to process audio signals can be improved, resulting in better audio signal processing performance.

[0027] In conjunction with the fourth aspect, in one possible implementation, the DSP is specifically used to: transmit the second audio signal to the CPU via shared memory, wherein the shared memory is shared by the DSP and the CPU; the CPU is used to: transmit the second audio signal to the NPU by calling the NPU.

[0028] In this embodiment, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal device to utilize the computing resources of the DSP and NPU, and further enables the terminal device to use the deep neural network processing capabilities of the NPU. In other words, it enables the deployment of the deep neural network processing model of the NPU on the terminal device, which can greatly improve the sound processing capabilities of the terminal device.

[0029] In conjunction with the fourth aspect, in one possible implementation, the DSP is specifically used to: output the target sound signal of the i-th frame when the first sound signal of the (n+i)-th frame is received; and output the target sound signal of the (i+1)-th frame when the first sound signal of the (n+i+1)-th frame is received; where n is a positive integer less than or equal to 10, i is a natural number, i=1,2,3,...

[0030] In this embodiment, the DSP receives the first audio signal and outputs the target audio signal with a time delay. That is, the deep neural network processing of the NPU and the preprocessing of the DSP can be performed synchronously. However, for the same frame of audio signal, the deep neural network processing performed by the NPU is asynchronous to the preprocessing performed by the DSP. In this way, after the DSP processes the first audio signal of the current frame, it does not need to wait for the third audio signal corresponding to the current frame to be output by the NPU. Instead, it continues to process the first audio signal of the next frame. Within the time delay frame range, the DSP will receive the third audio signal corresponding to the current frame output by the NPU. The DSP only needs to perform post-processing and output the corresponding third audio signal after the time delay frame is reached. In this way, the audio signal can be continuously output even when the NPU buffer is insufficient, thereby meeting the requirements of continuity and real-time performance of audio output.

[0031] Fifthly, an electronic device is provided, comprising: a display module, configured to display a first prompt on the screen of the electronic device when a call interface is displayed on the screen of the electronic device, the first prompt being used to prompt a user to confirm whether to enable intelligent noise reduction mode, or to display a first notification in the notification bar of the first device when the electronic device is in a call state, the first notification being used to prompt a user to confirm whether to enable intelligent noise reduction mode; and an execution module, configured to execute the method in the first aspect or any possible implementation thereof, or execute the method in the second aspect or any possible implementation thereof, in response to the user confirming the enabling of intelligent noise reduction mode.

[0032] In this embodiment of the application, when a user uses the call function of the terminal device, they can decide whether to enable the intelligent noise reduction mode according to their actual needs. When the user determines that the intelligent noise reduction mode needs to be enabled, the terminal device uses the computing power resources of the DSP and NPU to perform in-depth processing on the sound signal and output it.

[0033] Sixthly, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the method in the first aspect or any possible implementation of the first aspect, or to implement the method in the second aspect or any possible implementation of the second aspect, or to implement the method in the third aspect.

[0034] In a seventh aspect, a chip is provided, wherein instructions are stored therein, which, when executed on a device, cause the chip to perform the method of the first aspect or any possible implementation thereof, or cause the chip to perform the method of the second aspect or any possible implementation thereof, or perform the method of the third aspect thereof.

[0035] Eighthly, a computer-readable storage medium is provided that stores a computer program or instructions, which, when executed, implement the method of the first aspect or any possible implementation thereof, or implement the method of the second aspect or any possible implementation thereof, or implement the method of the third aspect. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0037] Figure 2 This is a software structure block diagram of the electronic device provided in the embodiments of this application;

[0038] Figure 3 This is a schematic diagram of a system architecture for processing sound signals provided in an embodiment of this application;

[0039] Figure 4 This is a schematic flowchart illustrating a method for processing sound signals provided in an embodiment of this application;

[0040] Figure 5 This is a schematic flowchart illustrating another method for processing sound signals provided in an embodiment of this application;

[0041] Figure 6 This is a schematic flowchart illustrating another method for processing sound signals provided in an embodiment of this application;

[0042] Figure 7 This is a schematic flowchart illustrating another method for processing sound signals provided in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram of a device for processing sound signals provided in an embodiment of this application;

[0044] Figure 9 This is a schematic diagram of an interface for enabling intelligent noise reduction mode provided in an embodiment of this application;

[0045] Figure 10 This is a schematic diagram of another interface for enabling intelligent noise reduction mode provided in the embodiments of this application;

[0046] Figure 11 This is a schematic diagram of another interface for enabling intelligent noise reduction mode provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments.

[0048] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

[0049] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0050] The methods provided in this application can be applied to electronic devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0051] For example, Figure 1 A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0052] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0053] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0054] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0055] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0056] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0057] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0058] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0059] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0060] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0061] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0062] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0063] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0064] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0065] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0066] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0067] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0068] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0069] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0070] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0071] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0072] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0073] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0075] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0076] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0077] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0078] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0079] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0080] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0081] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0082] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0083] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0084] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0085] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0086] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0087] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0088] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0089] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0090] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0091] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0092] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0093] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and is applicable to screen orientation switching, pedometers, and other applications.

[0094] A distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, during a shooting scene, the electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0095] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0096] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0097] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0098] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0099] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0100] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals based on the vibration signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information based on the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0101] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0102] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0103] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0104] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an embedded SIM (eSIM) card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0105] It should be understood that the phone cards in the embodiments of this application include, but are not limited to, SIM cards, eSIM cards, universal subscriber identity modules (USIM), universal integrated circuit cards (UICC), etc.

[0106] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0107] Figure 2 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0108] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0109] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0110] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0111] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0112] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0113] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0114] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection, hang-up, etc.).

[0115] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0116] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0117] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0118] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0119] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0120] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0121] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0122] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0123] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0124] A 2D graphics engine is a graphics engine for 2D drawing.

[0125] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0126] It should be understood that the technical solutions in the embodiments of this application can be used in systems such as Android, iOS, and HarmonyOS.

[0127] The technical solutions of this application can be applied to electronic devices with voice call functionality. For example, they can be applied to mobile phones, foldable screens, tablets, and other electronic devices. The main application scenario is for scenarios involving voice calls via electronic devices, such as video calls and answering phone calls. Specifically, they can be used for noise reduction in voice calls.

[0128] The electronic devices in the embodiments of this application can be televisions, desktop computers, laptops, or portable electronic devices such as mobile phones, tablets, cameras, camcorders, video recorders, smartwatches, smart bracelets, or other electronic devices with voice call functions, electronic devices in 5G networks, or electronic devices in future evolved public land mobile networks (PLMNs), etc. This application does not limit them.

[0129] In voice call scenarios, people have high requirements for the sound quality of voice calls. Current sound processing methods mainly include two types: traditional signal processing methods and deep learning processing methods. Among them, traditional signal processing methods use signal modeling, and the models built are relatively simple. Moreover, some of the parameters on which the model is based are assumed parameters. When the assumed parameters are not actually true, it may lead to poor processing results. Compared with traditional signal processing methods, deep learning processing methods use more complex models. These models are learned and trained based on a large amount of signal data, and the sound processing effect is better. However, the processing process requires a lot of computing resources. Furthermore, for terminal devices, current technology makes it difficult to utilize the computing resources of the NPU, making it difficult to apply deep learning processing methods to terminal devices.

[0130] In view of this, this application provides a method and electronic device for processing audio signals. In this method, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling signal exchange between the DSP and NPU. This allows the terminal device to utilize the computing resources of the DSP and NPU. For example, preprocessing and post-processing can be performed on the DSP, while deep neural network processing can be performed on the NPU. This provides the system with abundant computing resources, and because the deep neural network processing capabilities of the NPU are utilized, the terminal device achieves better audio signal processing results.

[0131] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0132] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0133] To better understand the solution of this application, the relevant terms involved in the embodiments of this application will be introduced below.

[0134] 1. Digital Signal Processor (DSP)

[0135] A DSP is a unique type of microprocessor that processes large amounts of information using digital signals. Its working principle involves receiving analog signals, converting them into digital signals of 0s and 1s, modifying, deleting, and amplifying these digital signals, and then interpreting the digital data back into analog data or the actual environment format in other system chips. It not only possesses programmability but also boasts a real-time operating speed of tens of millions of complex instructions per second, far exceeding that of general-purpose microprocessors, making it an increasingly important computer chip in the digital electronic world.

[0136] 2. Neural network processing units (NPU)

[0137] Neural Processing Units (NPUs) employ a "data-driven parallel computing" architecture, making them particularly adept at processing massive amounts of multimedia data, such as video and images.

[0138] 3. Central processing unit (CPU)

[0139] As the core of a computer system for computation and control, the CPU is the final execution unit for information processing and program execution.

[0140] For example, Figure 3 A schematic diagram of a system architecture for processing sound signals according to an embodiment of this application is shown. Figure 3As shown, the system architecture includes a codec (coeddecoder), a modem, a digital signal processor (DSP), a central processing unit (CPU), and neural network processing units (NPUs). The DSP communicates with the CPU via shared memory, and the CPU can schedule the NPU to perform relevant processing operations and enable signal exchange between the CPU and the NPU. Specifically:

[0141] A codec (coedder) is used to convert analog audio signals into digital audio signals before audio signal processing, so that the audio signals can be transmitted between DSP, CPU, and NPU, and can also be processed.

[0142] A modem is used to convert processed digital audio signals into analog audio signals for further output.

[0143] In one example, taking the uplink as an example, the audio signal stream is mic-codec-DSP-CPU-NPU-DSP-modem. Here, the mic is the analog audio signal. After the codec converts the analog audio signal into a digital audio signal, the DSP, CPU, and NPU process this digital audio signal. The processed digital audio signal is then modulated back into an analog audio signal by the modem before being output. In other words, the codec receives the analog audio signal, converts it into a digital audio signal, and enables the audio signal to be transmitted between the DSP, CPU, and NPU in digital signal transmission form.

[0144] The DSP and NPU use the CPU as a communication medium to work together to process the audio signal converted by the encoder and decoder, thereby obtaining the target audio signal.

[0145] In one embodiment, the processing of the audio signal converted by the encoder-decoder may include preprocessing, deep neural network processing (which may also be described as AI model processing), and postprocessing. The preprocessing is performed by a DSP and may include one or more of echo cancellation, noise reduction, gain, and frequency response control. The deep neural network processing is performed by an NPU and may include noise reduction processing and residual echo suppression processing. The postprocessing is performed by a DSP and may include gain and frequency response control.

[0146] The DSP transmits the target audio signal to the modem, which converts it from a digital audio signal to an analog audio signal, thus obtaining the target audio signal in analog format.

[0147] In this embodiment of the application, the transmission and processing of sound signals can be the transmission and processing of sound data.

[0148] In this embodiment, the digital signal processor (DSP) and the neural network processor (NN) establish a transmission channel between the DSP and the NN through the central processing unit (CPU), enabling signal exchange between them. This allows the system to utilize the computing resources of the DSP and NPU. For example, preprocessing and postprocessing can be performed on the DSP, while deep neural network processing can be performed on the NPU. This provides the system with abundant computing resources, and the sound signal processing effect is better due to the utilization of the NPU's deep neural network processing capabilities.

[0149] It should be understood that in this embodiment, the CPU establishes a connection between the DSP and the NPU. The NPU's deep neural network processing of the audio signal relies on the established speech deep learning model. The solution in this embodiment enables the speech deep learning model to be implemented on terminal products such as mobile phones and tablets, thereby improving the voice call quality of mobile phones, tablets and other terminal devices.

[0150] For example, Figure 4 A schematic flowchart of a method 400 for processing sound signals according to an embodiment of this application is shown. Figure 4 As shown, the method 400 includes:

[0151] The codec converts the received first sound (analog signal format) into a first sound signal (digital signal format) and transmits it to the DSP.

[0152] The first audio signal may include a multi-microphone signal (mic signal) and may also include multiple retrieval signals (ref signals).

[0153] In one example, the first audio signal is a 3-mic signal input + a 2-ref signal.

[0154] Among them, mic signal refers to the sound signal collected by the microphone, multi-mic signal refers to the sound signal collected by multiple microphones, and the mic signal can be converted into a digital signal by the codec; ref signal refers to the audio signal played by the downstream speaker, which is a digital signal before being converted by the modem.

[0155] S401: After receiving the first audio signal, the DSP preprocesses the first audio signal to obtain the second audio signal.

[0156] In some embodiments, the DSP preprocessing of the first audio signal includes: the DSP performing echo cancellation on the first audio signal, specifically, the DSP performing filter-based echo cancellation on the first audio signal; it may also include: the DSP performing noise reduction on the first audio signal; and it may also include: the DSP performing gain and frequency response control on the first audio signal.

[0157] In one implementation, the DSP includes a first processing module for processing a first audio signal to obtain a second audio signal.

[0158] S402: The DSP transmits the second audio signal to the NPU via the CPU.

[0159] In one implementation, the DSP transmits the second audio signal to the CPU via shared memory with the CPU, and the CPU further calls the NPU to perform deep neural network processing on the second audio signal.

[0160] S403: The NPU performs deep neural network processing on the second audio signal to obtain the third audio signal.

[0161] In some embodiments, the NPU performing deep neural network processing on the second audio signal includes: the NPU performing noise reduction on the first audio signal; it may also include: the NPU performing residual echo suppression on the first audio signal.

[0162] In one implementation, the NPU has a second processing module that processes the second audio signal to obtain a third audio signal.

[0163] In one example, the second processing module is a speech deep learning model.

[0164] S404: The NPU transmits the third audio signal to the DSP via the CPU.

[0165] In one implementation, the NPU transmits a third audio signal to the CPU, and the DSP obtains the third audio signal through shared memory with the CPU.

[0166] S405: The DSP performs post-processing on the third audio signal to obtain the target audio signal, and then transmits it to the modem.

[0167] In some embodiments, the DSP post-processing of the third audio signal includes: the DSP performing gain and frequency response control on the third audio signal.

[0168] In one implementation, the DSP includes a third processing module for processing the third audio signal to obtain the target audio signal.

[0169] A modem converts the received target audio signal (digital signal format) into the target audio signal (analog signal format).

[0170] It should be understood that the transmission and processing of sound signals described in the embodiments of this application can be the transmission and processing of sound signals, the transmission and processing of sound-related data, or the transmission and processing of sound signals and sound-related data.

[0171] In this embodiment, the DSP and NPU in the terminal device establish a transmission channel through the CPU, enabling the DSP and NPU to exchange signals and data. This allows the terminal device to utilize the computing resources of the DSP and NPU. For example, preprocessing and postprocessing can be performed on the DSP, while deep neural network processing can be performed on the NPU. In this way, the device has abundant computing resources, and because the deep neural network processing capability of the NPU is utilized, the terminal device processes audio signals better.

[0172] For example, in Figure 4 Based on the illustrated embodiment, Figure 5 A schematic flowchart illustrating another method for processing sound signals provided in an embodiment of this application is shown. Figure 5 As shown, in this method:

[0173] The DSP and NPU establish a transmission channel through the CPU, enabling the mutual transmission of signals and data between the DSP and NPU. Specifically, the DSP establishes a connection with the CPU through shared memory, and the CPU interacts with the NPU by calling the NPU.

[0174] The DSP is used to preprocess the received first audio signal to obtain the second audio signal.

[0175] Among them, the process of preprocessing the first sound signal is as follows: Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0176] The NPU is used to process the second audio signal using a deep neural network to obtain the third audio signal.

[0177] Among them, the process of performing deep neural network processing on the second sound signal is as follows: Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0178] The DSP is also used to post-process the third audio signal to obtain the target audio signal.

[0179] Regarding the post-processing of the third audio signal, in Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0180] In some embodiments, the transmission of sound signals can be performed in frames. For example, the DSP receives the first frame of the first sound signal, preprocesses the first frame of the first sound signal to obtain the corresponding first frame of the second sound signal; the DSP receives the second frame of the first sound signal, preprocesses the second frame of the first sound signal to obtain the corresponding second frame of the second sound signal; the NPU performs deep neural network processing on the first frame of the second sound signal and the second frame of the second sound signal to obtain the corresponding first frame of the third sound signal and the second frame of the third sound signal; the DSP performs post-processing on the first frame of the third sound signal and the second frame of the third sound signal to obtain the first frame of the target sound signal and the second frame of the target sound signal.

[0181] Should Figure 5 In the embodiment shown, the DSP's processing of the first audio signal and the processing of the third audio signal have an n-frame delay. This can also be understood as the DSP's processing of the first audio signal and the NPU's processing of the second audio signal being performed synchronously, but the audio signals corresponding to different frames are being processed simultaneously. The number of frames corresponding to the second audio signal processed by the NPU is later than the number of frames corresponding to the first audio signal processed by the DSP.

[0182] In some embodiments, the DSP's processing of the first audio signal and the processing of the third audio signal have an n-frame delay, which is ultimately manifested as follows: the terminal device sequentially receives the first frame of the first audio signal, the second frame of the first audio signal, ...; after the terminal device receives the (n+1)th frame of the first audio signal, the terminal device outputs the first frame of the target audio signal corresponding to the first frame of the first audio signal; after the terminal device receives the (n+2)th frame of the first audio signal, the terminal device outputs the second frame of the target audio signal corresponding to the second frame of the first audio signal; and so on. That is, the input and output of the terminal device have an n-frame delay, and the output process of the target audio signal of the terminal device has a certain degree of real-time performance and continuity.

[0183] In some embodiments, n takes the value of any integer between 1 and 10.

[0184] In some embodiments, the processing delay of the first audio signal and the processing delay of the third audio signal by the DSP can be less than n frames, but the DSP receiving the first audio signal and the DSP outputting the target audio signal have a delay of n frames.

[0185] It should be understood that the transmission and processing of sound signals described in the embodiments of this application can be the transmission and processing of sound signals, the transmission and processing of sound-related data, or the transmission and processing of sound signals and sound-related data.

[0186] In this embodiment, the NPU's processing of the second audio signal has a time delay relative to the DSP's processing of the first audio signal (or it can be understood as the DSP receiving the first audio signal and the DSP outputting the target audio signal having a time delay). In other words, the NPU's processing is asynchronous relative to the DSP's processing. Thus, after the DSP processes the first audio signal of a frame, it does not need to wait for the NPU to output the corresponding third audio signal of that frame. Instead, it continues to process the first audio signal of the next frame. After reaching the number of time delay frames, it obtains the third audio signal of the corresponding frame output by the NPU, processes it, and outputs it. In this way, even when the NPU buffer is insufficient, it can continuously output audio signals, thereby meeting the requirements of continuity and real-time performance of audio output.

[0187] To more clearly illustrate the latency characteristics of DSP processing and NPU processing, let's take an example with a latency of 3 frames (i.e., n=3). Figure 6 A schematic flowchart of another method 600 for processing sound signals provided in an embodiment of this application is shown. Figure 6 As shown, the method 600 includes:

[0188] S601: The DSP preprocesses the first audio signal of frame 0 to obtain the second audio signal of frame 0.

[0189] In this case, the first sound signal in frame 0 is not a continuously input sound signal, but a preset sound signal, such as a preset sound signal that is converged according to an algorithm.

[0190] S602: The DSP preprocesses the received first frame of the first audio signal to obtain the second frame of the first audio signal.

[0191] Among them, the process of DSP preprocessing the first audio signal is as follows: Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0192] S603: The DSP transmits the pre-processed second audio signal of frame 0 and the second audio signal of frame 1 to the NPU.

[0193] Among them, the method of DSP transmitting the second audio signal to NPU is as follows: Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0194] S604: The NPU performs deep neural network processing on the second audio signal of frame 0 and the second audio signal of frame 1 respectively to obtain the third audio signal of frame 0 and the third audio signal of frame 1.

[0195] Among them, the process of NPU performing deep neural network processing on the second audio signal is as follows: Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0196] S605: The DSP preprocesses the received second frame first audio signal to obtain the second frame second audio signal.

[0197] S607: The DSP transmits the pre-processed second frame second audio signal and the third frame second audio signal to the NPU.

[0198] S608: The NPU performs deep neural network processing on the second sound signal of the second frame and the second sound signal of the third frame respectively to obtain the third sound signal of the second frame and the third sound signal of the third frame.

[0199] S609: The DSP receives the third audio signal of frame 0 and the third audio signal of frame 1 transmitted by the NPU, and performs post-processing on the third audio signal of frame 0 and the third audio signal of frame 1 to obtain the target audio signal of frame 0 and the target audio signal of frame 1, so that the DSP outputs the target audio signal of frame 0 after receiving the first audio signal of frame 3.

[0200] Among them, the process of DSP post-processing the third audio signal is as follows: Figure 4 The embodiments shown are described in detail, and for the sake of brevity, they will not be repeated here.

[0201] S610: The DSP preprocesses the received 4th frame first audio signal to obtain the 4th frame second audio signal, and the DSP outputs the 1st frame target audio signal after receiving the 4th frame first audio signal.

[0202] S611: The DSP preprocesses the received first audio signal of the 5th frame to obtain the second audio signal of the 5th frame.

[0203] S612: The DSP transmits the pre-processed second audio signal of frame 4 and second audio signal of frame 5 to the NPU.

[0204] S613: The NPU performs deep neural network processing on the second audio signal of the 4th frame and the second audio signal of the 5th frame respectively to obtain the third audio signal of the 4th frame and the third audio signal of the 5th frame.

[0205] S614: The DSP receives the second frame third audio signal and the third frame third audio signal transmitted by the NPU, and performs post-processing on the second frame third audio signal and the third frame third audio signal to obtain the second frame target audio signal and the third frame target audio signal, so that the DSP outputs the second frame target audio signal after receiving the fifth frame first audio signal.

[0206] S615: The DSP preprocesses the received first audio signal of the 6th frame to obtain the second audio signal of the 6th frame.

[0207] S616: The DSP preprocesses the received first audio signal of the 7th frame to obtain the second audio signal of the 7th frame.

[0208] S617: The DSP transmits the pre-processed second audio signal of frame 6 and second audio signal of frame 7 to the NPU.

[0209] S618: The NPU performs deep neural network processing on the second audio signal of the 6th frame and the second audio signal of the 7th frame respectively to obtain the third audio signal of the 6th frame and the third audio signal of the 7th frame.

[0210] S619: The DSP receives the third audio signal of the 4th frame and the third audio signal of the 5th frame transmitted by the NPU, and performs post-processing on the third audio signal of the 4th frame and the third audio signal of the 5th frame to obtain the target audio signal of the 4th frame and the target audio signal of the 5th frame, so that the DSP outputs the target audio signal of the 4th frame after receiving the first audio signal of the 7th frame.

[0211] …(and so on)

[0212] It should be understood that the order of the steps described above in the embodiments of this application is not limited by the step numbers. The step numbers are only introduced to clearly describe the embodiments of this application.

[0213] It should also be understood that when the DSP processes the two consecutive frames of the second audio signal, it is also simultaneously processing the newly received first audio signal. For example, during the execution of S604, the DSP is simultaneously executing step S605 and subsequent steps. After the DSP processes the two consecutive frames of the second audio signal to obtain two consecutive frames of the third audio signal, it sends the third audio signal to the DSP. The DSP processes the two consecutive frames of the third audio signal to obtain two consecutive frames of the target audio signal and outputs the target audio signal at the corresponding output time. For example, when the time delay is 3 frames, after the DSP receives the (i+3)th frame of the first audio signal, the DSP outputs the i-th frame of the target audio signal.

[0214] exist Figure 6 Based on the illustrated embodiment, exemplarily, taking a delay of 3 frames (i.e., n=3) as an example, Figure 7 A schematic flowchart of another method 700 for processing sound signals provided in an embodiment of this application is shown. Figure 7 As shown, the method 700 includes:

[0215] (1) The DSP continuously receives a first sound signal of length Xms, wherein the first sound signal of length Xms can be understood as a frame of the first sound signal described in the above embodiment; and the DSP preprocesses each first sound signal of length Xms after receiving it to obtain a second sound signal of length Xms.

[0216] In some embodiments, X can be any value from 8 to 20, for example, it can be 10.

[0217] (2) After the DSP outputs two second sound signals with a length of Xms, it transmits the two output second sound signals with a length of Xms to the NPU through the CPU.

[0218] (3) The NPU performs deep neural network processing on the two second sound signals with a length of Xms output by the DSP to obtain two third sound signals with a length of Xms (i.e. Figure 7 The NPU inference steps shown in the figure transmit the two third audio signals of length Xms to the DSP via the CPU.

[0219] In other words, compared to the DSP's processing of the first audio signal, the NPU's processing of the second audio signal has a time delay.

[0220] (4) After receiving two third sound signals with a length of Xms, the DSP performs post-processing to obtain two target sound signals with a length of Xms and outputs them.

[0221] To ensure the continuity and real-time performance of the target audio signal output, the DSP's preprocessing of the first audio signal and post-processing of the third audio signal have a time delay. For example, in this embodiment, the DSP's preprocessing of the first audio signal and post-processing of the third audio signal have a 3-frame time delay. Figure 7For example, the DSP will only perform post-processing and output of the first third audio signal when it preprocesses the fourth first audio signal; the DSP will only perform post-processing and output of the second third audio signal when it preprocesses the fifth first audio signal; the DSP will only perform post-processing and output of the third third audio signal when it preprocesses the sixth first audio signal; the DSP will only perform post-processing and output of the fourth third audio signal when it preprocesses the seventh first audio signal; and the DSP will only perform post-processing and output of the fifth third audio signal when it preprocesses the eighth first audio signal.

[0222] In this process, the DSP processes data once every Xms. Here, 2Xms means that the DSP transmits the data to the NPU for inference after accumulating two frames of the first audio signal (i.e., the NPU processes two frames of the second audio signal to obtain two frames of the third audio signal). As long as the inference is completed within 2Xms, the return is sufficient. This increases the inference time of the NPU. When the NPU returns the third audio signal to the DSP, it also returns two frames of the third audio signal (i.e., 2Xms of data).

[0223] For example, Figure 8 A schematic diagram of an apparatus for processing sound signals provided in an embodiment of this application is shown.

[0224] like Figure 8 As shown, the DSP and CPU communicate via shared memory. The CPU can call the NPU to communicate with the NPU. The DSP includes a first processing module 810 and a third processing module 830, and the NPU includes a second processing module 820. The first processing module 810 processes the first sound signal to obtain a second sound signal; the second processing module 820 processes the second sound signal to obtain a third sound signal; and the third processing module 830 processes the third sound signal to obtain a target sound signal.

[0225] In another implementation, the DSP includes only a first processing module 810, which can also be used to process a third audio signal to obtain a target audio signal.

[0226] In some embodiments, the second processing module 820 may be a speech deep learning model.

[0227] The interpretation of the first sound signal, the second sound signal, the third sound signal, and the target sound signal is as follows: Figure 4 The embodiments shown have been described in detail, and for the sake of brevity, they will not be repeated here.

[0228] For example, Figure 9 This illustration shows a schematic diagram of an interface for enabling intelligent noise reduction mode according to an embodiment of this application.

[0229] like Figure 9 As shown, in one implementation, when device 900 receives an incoming call notification, the displayed incoming call interface is as follows: Figure 9 As shown in (a) of the interface, a pop-up window 901 can be displayed. This pop-up window 901 is used to prompt the user whether to enable the intelligent noise reduction mode. When the user clicks the "Yes" control in the pop-up window, the device 900 enables the intelligent noise reduction mode. Thus, when the user answers the call, the voice output by the device 900 will be the one processed as described above. Figure 4 The illustrated embodiments to Figure 7 The method in any of the embodiments shown is used to process the data; when the user clicks the "No" control in the pop-up window, the device 900 does not enable the intelligent noise reduction mode.

[0230] In another implementation, when device 900 receives an incoming call notification, pop-up 901 is not displayed on the incoming call notification screen, but rather on the call screen after the user confirms answering the call (e.g., ...). Figure 9 As shown in (b) of the diagram, in this call interface, the pop-up 901 is used to prompt the user whether to enable the intelligent noise reduction mode. When the user clicks the "Yes" control in the pop-up, the device 900 enables the intelligent noise reduction mode. In this way, the voice output by the device 900 is the voice processed as described above. Figure 4 The illustrated embodiments to Figure 7 The method in any of the embodiments shown is used to process the data; when the user clicks the "No" control in the pop-up window, the device 900 does not enable the intelligent noise reduction mode.

[0231] In some embodiments, the pop-up 901 is only displayed on the incoming call screen or call screen of the device 900 during the first few calls (e.g., the first 3, the first 5, etc.).

[0232] In another implementation, pop-up 901 is not displayed on either the incoming call interface or the call interface. In this case, during the call, the user can click the "More" control in the call interface to view the undisplayed function controls. These undisplayed function controls include control 902 for turning the intelligent noise reduction mode on and off. When the user clicks control 902, device 900 activates the intelligent noise reduction mode. Thus, the voice output by device 900 is the one processed as described above. Figure 4 The illustrated embodiments to Figure 7 The method of any of the embodiments shown is used to process the data; when the user wants to turn off the smart noise reduction mode, they can click the control 902 again.

[0233] It should be understood that the embodiments of this application only take the scenario of the device receiving an incoming call as an example. The solution of the embodiments of this application can also be applied to the scenario of the device making a call, as well as to any voice call scenario such as video calls and online conferences.

[0234] For example, Figure 10 This illustration shows another interface diagram for enabling intelligent noise reduction mode provided in an embodiment of this application.

[0235] like Figure 10 As shown, in one implementation, when the electronic device 1000 receives an incoming call notification, the displayed incoming call interface is as follows: Figure 10 As shown in (a), when the user answers an incoming call (e.g., via...), Figure 10 The operation of sliding to answer an incoming call is implemented on the interface shown in (a) of the electronic device 1000, and the display interface of the electronic device 1000 switches to the interface of the call in progress (e.g., Figure 10 As shown in (b) of the diagram, a pop-up window 1001 can be displayed in the call interface. This pop-up window 1001 is used to prompt the user whether to enable the intelligent noise reduction mode. The user can enable the intelligent noise reduction mode by clicking anywhere in the pop-up window. When the user enables the intelligent noise reduction mode, the voice output by the device 1000 is the voice processed as described above. Figure 4 The illustrated embodiments to Figure 7 The pop-up window 1001 may include display content or controls such as "intelligent noise reduction mode", "enables improved voice call quality after activation", "click anywhere to activate intelligent noise reduction mode" or "click here to activate intelligent noise reduction mode".

[0236] When the user clicks on the pop-up 1001, the pop-up 1001 is canceled.

[0237] In some embodiments, the pop-up 1001 can be displayed on the interface of the electronic device 1000 throughout the entire call; the pop-up 1001 can also be displayed at the moment of entering the call interface and automatically canceled after a certain period of time.

[0238] In some embodiments, the pop-up 1001 is displayed during the first m calls of the electronic device 1000. If the user does not click on the pop-up 1001 during the m calls, the pop-up 1001 will no longer be displayed in the call interface after the mth call.

[0239] It should be understood that the embodiments of this application only take the scenario of the device receiving an incoming call as an example. The solution of the embodiments of this application can also be applied to the scenario of the device making a call, and can also be applied to any voice call scenario such as video calls and online conferences.

[0240] For example, Figure 11This illustration shows another interface diagram for enabling intelligent noise reduction mode provided in an embodiment of this application.

[0241] like Figure 11 As shown, in one implementation, the current display interface of the electronic device 1100 is a call interface (e.g., Figure 11 As shown in (a)), when the user wants to activate the intelligent noise reduction mode, the user can view the notification bar of the electronic device 1100 by performing a swipe-down operation on the screen of the electronic device 1100 (e.g., ...). Figure 11 As shown in (b) of the diagram, the notification bar displays multiple notification items. These items may include, for example, notification item 1101 and notification item 1102. Notification item 1101 is used to notify the user that they are currently in a call. Notification item 1101 can also display information related to the current call, such as the caller and call duration. Notification item 1101 can also include multiple controls, such as a mute control, a hang-up control, and a speaker control. Notification item 1102 is used to prompt the user whether to enable the intelligent noise reduction mode. The user can enable the intelligent noise reduction mode by clicking any position on notification item 1102. When the user enables the intelligent noise reduction mode, the voice output by device 1100 is the voice processed by the above-mentioned noise reduction mode. Figure 4 The illustrated embodiments to Figure 7 The notification item 1102, processed by the method of any embodiment shown in the examples, may include display content or controls such as "intelligent noise reduction mode," "enables improved voice call quality when enabled," "click anywhere to enable intelligent noise reduction mode," or "click here to enable intelligent noise reduction mode." When the user clicks on the notification item 1102, the notification item 1102 is de-displayed.

[0242] In some embodiments, the notification item 1102 may be displayed in the notification bar of the electronic device 1100 throughout the entire call; the notification item 1102 may also be displayed in the notification bar of the electronic device 1100 when the call interface is first entered, and automatically canceled after a certain period of display.

[0243] In some embodiments, the notification item 1102 is displayed in the notification bar of the electronic device 1100 during the first m calls of the electronic device 1000. If the user does not click on the notification item 1102 during the m calls, the notification item 1102 will no longer be displayed in the notification bar of the electronic device 1100 during subsequent calls.

[0244] In some embodiments, the above Figure 10 The illustrated embodiments and Figure 11 The illustrated embodiment is a combination implementation.

[0245] In one example, when the user is Figure 10 When the intelligent noise reduction mode is selected in the interface shown in (b), Figure 11 The interface shown in (b) will not display notification item 1102, but it can display a notification item to remind the user that the current mode is smart noise cancellation, and can also provide the user with a control to turn off smart noise cancellation at any time.

[0246] In yet another example, if the user is Figure 10 In the interface shown in (b), the intelligent noise reduction mode is not selected. After a certain duration of the call, the pop-up 1001 will no longer appear on the call interface, meaning the display interface of the electronic device will switch to the mode shown in (b). Figure 11 In the interface shown in (a), if the user wants to enable the intelligent noise reduction mode, the user can... Figure 11 The method provided in the illustrated embodiment enables the intelligent noise reduction mode.

[0247] It should be understood that the embodiments of this application only take the scenario of the device receiving an incoming call as an example. The solution of the embodiments of this application can also be applied to the scenario of the device making a call, as well as to any voice call scenario such as video calls and online conferences.

[0248] One or more modules or units described herein can be implemented in software, hardware, or a combination of both. When any of the above modules or units are implemented in software, the software exists as computer program instructions and is stored in memory. A processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores within the processor for executing software instructions to perform calculations or processing, it may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0249] When the modules or units described herein are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of software.

[0250] When the modules or units described herein are implemented using software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0251] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0252] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0253] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0254] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0256] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0257] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing sound signals, characterized in that, The method includes: The digital signal processor (DSP) preprocesses the first audio signal and outputs the second audio signal. The DSP transmits the second sound signal to the neural network processor (NPU) through the central processing unit (CPU); and receives the third sound signal output by the NPU, wherein the third sound signal is a sound signal obtained by the NPU through deep neural network processing of the second sound signal; The DSP outputs the target sound signal by post-processing the third sound signal; The DSP transmits the second audio signal to the NPU via the CPU, including: The DSP transmits the second audio signal to the CPU via shared memory, so that the CPU can transmit the second audio signal to the NPU by calling the NPU. The shared memory is the shared memory of the DSP and the CPU.

2. The method according to claim 1, characterized in that, The preprocessing includes one or more of echo cancellation, noise reduction, gain and frequency response control.

3. The method according to claim 1 or 2, characterized in that, The deep neural network processing includes residual echo cancellation and / or noise reduction.

4. The method according to claim 1 or 2, characterized in that, The post-processing includes gain and frequency response control.

5. The method according to claim 1 or 2, characterized in that, The method further includes: When the DSP receives the first sound signal of the (n+i)th frame, the DSP outputs the target sound signal of the i-th frame; When the DSP receives the first sound signal of the (n+i+1)th frame, the DSP outputs the target sound signal of the (i+1)th frame. Where n is a positive integer less than or equal to 10, and i is a natural number, i = 1, 2, 3, ...

6. A method for processing sound signals, characterized in that, The method includes: The neural network processor (NPU) receives a second audio signal through the central processing unit (CPU). The second audio signal is an audio signal obtained by the digital signal processor (DSP) through preprocessing of the first audio signal. The NPU processes the second sound signal using a deep neural network to output a third sound signal; The NPU transmits the third audio signal to the DSP through the CPU; The NPU transmits the third audio signal to the DSP via the CPU, including: The NPU transmits the third audio signal to the DSP through the CPU and shared memory, wherein the shared memory is the shared memory of the DSP and the CPU.

7. The method according to claim 6, characterized in that, The preprocessing includes one or more of echo cancellation, noise reduction, gain control, and frequency response control.

8. The method according to claim 6 or 7, characterized in that, The deep neural network processing includes residual echo cancellation and / or noise reduction.

9. A method for processing sound signals, characterized in that, The method includes: When the screen of the first device displays the call interface, a first prompt is displayed on the screen of the first device. The first prompt is used to prompt the user to confirm whether to enable the intelligent noise reduction mode. Or when the first device is in a call state, a first notification is displayed in the notification bar of the first device. The first notification is used to prompt the user to confirm whether to enable the intelligent noise reduction mode. In response to the user's confirmation of enabling the intelligent noise reduction mode, the first device performs the method as described in any one of claims 1 to 5, or the method as described in any one of claims 6 to 8.

10. An electronic device, characterized in that, The electronic device includes a digital signal processor (DSP), a central processing unit (CPU), and a neural network processor (NPU), wherein: The DSP is used to preprocess the first sound signal and output the second sound signal; The DSP is also used to transmit the second audio signal to the NPU through the CPU; The NPU is used to output a third sound signal by performing deep neural network processing on the second sound signal; The NPU is also used to transmit the third audio signal to the DSP via the CPU; The DSP is also used to output the target sound signal by post-processing the third sound signal; The DSP is specifically used for: The second audio signal is transmitted to the CPU via shared memory, which is the shared memory of the DSP and the CPU; The CPU is used for: The second audio signal is transmitted to the NPU by calling the NPU.

11. The electronic device according to claim 10, characterized in that, The preprocessing includes one or more of echo cancellation, noise reduction, gain control, and frequency response control.

12. The electronic device according to claim 10 or 11, characterized in that, The deep neural network processing includes residual echo cancellation and / or noise reduction.

13. The electronic device according to claim 10 or 11, characterized in that, The post-processing includes gain and frequency response control.

14. The electronic device according to claim 10 or 11, characterized in that, The DSP is specifically used for: Upon receiving the first sound signal of the (n+i)th frame, output the target sound signal of the i-th frame; Upon receiving the first sound signal of the (n+i+1)th frame, output the target sound signal of the (i+1)th frame; Where n is a positive integer less than or equal to 10, and i is a natural number, i = 1, 2, 3, ...

15. An electronic device, characterized in that, The electronic device includes: The display module is used to display a first prompt on the screen of the electronic device when the call interface is displayed on the screen of the electronic device, the first prompt being used to prompt the user to confirm whether to enable the intelligent noise reduction mode, or to display a first notification in the notification bar of the electronic device when the electronic device is in a call state, the first notification being used to prompt the user to confirm whether to enable the intelligent noise reduction mode. An execution module is configured to, in response to a user's confirmation of enabling the intelligent noise reduction mode, perform the method as described in any one of claims 1 to 5, or perform the method as described in any one of claims 6 to 8.

16. An electronic device, characterized in that, include: One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 5, or the method of any one of claims 6 to 8, or the method of claim 9.

17. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 5, or the method as described in any one of claims 6 to 8, or the method as described in claim 9.

18. A chip, characterized in that, The chip stores instructions that, when executed, implement the method as described in any one of claims 1 to 5, or the method as described in any one of claims 6 to 8, or the method as described in claim 9.

Images