Audio processing method and electronic device
By obtaining the audio response and ear canal characteristics of the speaker output, combining the target sound effect, using transfer functions and neural networks to process audio, the problem of high cost and poor effect of hardware improvement in the prior art is solved, and better listening is achieved Sound experience and sound quality.
Patent Information
- Application Number
- CN202311136573.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-04-18
AI Technical Summary
The prior art improves transient response by changing the hardware structure of electronic devices, which is costly and ineffective, affecting the user's listening experience.
By obtaining the audio response output by the speaker and the user's ear canal characteristics, combining the target sound effects, using transfer functions and neural networks to process the original audio, generate target audio that meets the user's needs, and improve the user's listening experience.
Without changing the hardware structure, the user's listening experience is significantly improved, the sound quality of audio playback is improved, the cost is reduced, and the audio fidelity is enhanced.
Smart Images

Figure CN117177135B_ABST
Abstract
Description
[0001] This application is a divisional application. The application number of the original application is 202310415787.7, and the original application date is April 18, 2023. The entire contents of the original application are incorporated into this application by reference. Technical Field
[0002] The embodiments of the present application relate to the field of audio processing, and in particular to an audio processing method and electronic device. Background Art
[0003] The transient response of an electronic device refers to the ability of the electronic device to accurately reproduce transient audio changes. The transient response affects the sound quality of the audio played by the electronic device, thereby affecting the user's listening experience. In the prior art, the transient response of the electronic device is improved by changing the hardware structure of the electronic device. However, improving the transient response of the electronic device by changing the hardware structure of the electronic device is not only costly, but also has a poor effect on improving the user's listening experience. Summary of the invention
[0004] The present application provides an audio processing method and an electronic device, which are used to improve the user's listening experience when the electronic device plays audio by improving the transient response in the acoustic system formed when the user wears the electronic device based on the original hardware of the electronic device.
[0005] In order to achieve the above objectives, this application adopts the following technical solutions:
[0006] In a first aspect, an audio processing method is provided, the method comprising: first, obtaining a response for indicating that audio played by a speaker of an audio playback device is received by the audio playback device after being propagated through an acoustic system, and a target sound effect selected by a user, the target sound effect being the effect that the user expects to achieve when the audio is played, and the acoustic system is formed after the user wears the audio playback device; then, according to the first transfer function, obtaining a second transfer function, the second transfer function can be used to indicate the response of audio played by a speaker of the audio playback device being propagated through the above-mentioned acoustic system to the position of the user's eardrum; finally, according to the second transfer function, the original audio and the target sound effect, obtaining the target audio to be played.
[0007] When the user wears an audio player to listen to audio, the user's ear canal and the audio player form an acoustic system. Using an audio processing method provided by the present application, first, obtain the response of the audio played by the speaker of the audio player device after being transmitted by the acoustic system and received by the audio player device, that is, the first transfer function, which can be obtained by measurement. Then, according to the first transfer function and the mapping relationship between the first transfer function and the second transfer function, determine the second transfer function that cannot be obtained by measurement but can be used to represent the user's real listening experience (the second transfer function is also the transient response function of the audio player device in the acoustic system). According to the second transfer function and the target sound effect selected by the user, the target audio obtained after processing the original audio can meet the user's listening needs, so that the user has a better listening experience in the process of wearing the audio player device to listen to audio. It can be seen that the hardware structure of the audio player device is not changed in the present application, but the original audio is processed by software in combination with the user's ear canal characteristics, the wearing state of the user when wearing the audio player device, and the target sound effect selected by the user to obtain the target audio that meets the user's listening needs.
[0008] In a possible implementation manner of the first aspect, the first transfer function is a preset transfer function, or the first transfer function is determined by an audio playback device.
[0009] Since the first transfer function refers to the corresponding relationship between the audio output by the speaker of the audio playback device and the audio received by the audio playback device after being propagated by the acoustic system, and the audio playback device receives the audio output by the speaker through the built-in feedback microphone, the hardware structure of the audio playback device will affect the source of obtaining the first transfer function. Specifically, if the audio playback device includes a feedback microphone, the first transfer function can be determined by the audio playback device itself; if the audio playback device does not include a feedback microphone, the preset transfer function can be determined as the first transfer function.
[0010] In another possible implementation of the first aspect, the first transfer function may be determined by an electronic device other than the audio playback device. In this case, obtaining the first transfer function includes: first, the electronic device obtains the audio played by the speaker of the audio playback device and the audio received by the feedback microphone in the audio playback device; then, the electronic device obtains the first transfer function according to the audio played by the speaker of the audio playback device and the audio received by the feedback microphone of the audio playback device.
[0011] If the audio playback device does not have data processing capabilities but includes a feedback microphone, the audio playback device can send the audio played by the speaker and the audio received by the feedback microphone to an electronic device with data processing capabilities, and obtain the first transfer function after processing by the electronic device.
[0012] In another possible implementation of the first aspect, the target audio to be played is obtained according to the second transfer function, the original audio and the target sound effect, including: first, according to the second transfer function and the target sound effect, a target filter or a target neural network is obtained for processing the original audio; then, the original audio is processed by the target filter or the target neural network to obtain the target audio.
[0013] The second transfer function can be used to indicate the real listening experience of the user when wearing the audio playback device to listen to the audio. The target sound effect is the effect that the user expects to achieve when the audio is played, and a sound effect can be represented by a transient response function. Therefore, a target filter or a target neural network that can process the second transfer function into a transient response function corresponding to the target sound effect can process the original audio into the target audio that meets the user's listening needs.
[0014] In a second aspect, the present application provides an electronic device, comprising a processor and a memory, wherein instructions are stored in the memory, and when the processor executes the instructions, the method described in the first aspect and any embodiment thereof is executed.
[0015] According to a third aspect, a computer storage medium is provided, comprising instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to the first aspect and any embodiment thereof.
[0016] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed on the electronic device, the electronic device executes the method as described in the first aspect and any embodiment thereof.
[0017] In a fifth aspect, a chip system is provided, the chip system including a processor for supporting an electronic device to implement the functions involved in the first aspect and any of its embodiments. In one possible design, the electronic device also includes an interface circuit, which can be used to receive signals from other devices (such as a memory) or send signals to other devices (such as a communication interface). The chip system may include a chip and may also include other discrete devices.
[0018] The technical effects of the second to fifth aspects refer to the technical effects of the first aspect and any of its embodiments, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A schematic diagram of the structure of an audio playback system provided in an embodiment of the present application;
[0020] Figure 2 A schematic diagram of the structure of a first electronic device provided in an embodiment of the present application;
[0021] Figure 3 A schematic diagram of a software architecture running on a first electronic device provided in an embodiment of the present application;
[0022] Figure 4 A schematic diagram of the structure of a second electronic device provided in an embodiment of the present application;
[0023] Figure 5 A waveform diagram of output audio of a speaker of a second electronic device provided in an embodiment of the present application;
[0024] Figure 6 A schematic diagram of a waveform diagram of a first transient response function, a waveform diagram of input audio, and a waveform diagram of output audio of a speaker of a second electronic device provided in an embodiment of the present application;
[0025] Figure 7 A waveform diagram of a second transient response function of a speaker of a second electronic device and a schematic diagram of a waveform diagram of output audio provided in an embodiment of the present application;
[0026] Figure 8 A schematic diagram of the structure of an acoustic system provided in an embodiment of the present application;
[0027] Fig. 9 One of the interface schematic diagrams of a first electronic device provided in an embodiment of the present application;
[0028] Fig.10 A flowchart of an audio processing method provided in an embodiment of the present application;
[0029] Fig.11 A second schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;
[0030] Fig.12 A schematic diagram of the framework structure of an adaptive filter provided in an embodiment of the present application;
[0031] Fig.13 A third schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;
[0032] Fig.14 A fourth schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;
[0033] Fig.15A fifth schematic diagram of an interface of a first electronic device provided in an embodiment of the present application;
[0034] Fig.16 A schematic diagram of the structure of a first neural network provided in an embodiment of the present application;
[0035] Fig.17 A schematic diagram of a transient response function of a target sound effect obtained according to a second transfer function provided in an embodiment of the present application;
[0036] Fig.18 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] The terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.
[0038] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.
[0039] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.
[0040] First, some concepts involved in this application are described.
[0041] The transfer function refers to the relationship between the input and output of an object with linear characteristics, and is usually represented by a function (the ratio of the Laplace transform of the output waveform to the Laplace transform of the input waveform).
[0042] Transient response refers to the process of the system (or equipment) output changing from the initial state to the stable state under the action of a typical signal input. Transient response is also called dynamic response or transition process or transient response. A system (or equipment) with good transient response should respond immediately when a signal comes, and stop abruptly when the signal stops, without any delay. In other words, the better the transient response of the system (or equipment), the more fidelity the audio output of the system (or equipment). Therefore, by improving the transient response of the system (or equipment), the quality of the audio output of the system (or equipment) can be improved.
[0043] An audio processing method provided in the embodiment of the present application can be applied to Figure 1 The audio playback system shown in FIG. Figure 1 As shown, the audio playback system may include: a first electronic device 101 and a second electronic device 102, wherein the first electronic device 101 is in communication connection with the second electronic device 102. A user wears the second electronic device 102 to listen to audio.
[0044] In one embodiment, the first electronic device 101 is used to process the pre-stored original audio to obtain the target audio to be played that meets the user's listening needs, and then send the target audio to the second electronic device 102. The second electronic device 102 is used to play the target audio. In other words, the first electronic device 101 can be an audio processing device, and the second electronic device 102 can be an audio playing device.
[0045] In another embodiment, the first electronic device 101 is used to send the original audio to the second electronic device. The second electronic device 102 is used to process the original audio, obtain the target audio to be played that meets the user's listening needs, and play the target audio. In other words, the second electronic device 102 can be both an audio processing device and an audio playing device.
[0046] Optionally, the audio playback system involved in the embodiment of the present application may also include a second electronic device 102. In this case, the second electronic device pre-stores the original audio. The second electronic device 102 is used to process the original audio, obtain the target audio to be played that meets the user's listening needs, and play the target audio. In other words, the second electronic device 102 is both an audio processing device and an audio playback device.
[0047] The first electronic device involved in the embodiment of the present application may be a device with communication function and data processing function, and the first electronic device may be mobile or fixed. The first electronic device may be deployed on land (e.g., indoors or outdoors, handheld or vehicle-mounted, etc.), on water (e.g., ships, etc.), or in the air (e.g., airplanes, balloons, etc.). The first electronic device may be referred to as user equipment (UE), access terminal, terminal unit, subscriber unit, terminal station, mobile station (MS), mobile station, terminal agent or terminal device, etc. For example, the first electronic device may be a mobile phone, a tablet computer, a laptop computer, etc. The embodiment of the present application does not limit the specific type and structure of the first electronic device. A possible structure of the first electronic device is described below.
[0048] For example, taking the first electronic device as a mobile phone, Figure 2FIG. 1 shows a possible structure of the first electronic device 101. Figure 2 As shown, the first electronic device 101 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a power management module 240, a battery 241, a wireless charging coil 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.
[0049] Among them, the sensor module 280 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.
[0050] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the first electronic device 101. In other embodiments of the present application, the first electronic device 101 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0051] The processor 210 may include one or more processing units, for example, the processor 210 may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), an application processor (AP), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a baseband processor, and a neural network processor (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. For example, the processor 210 may be an application processor AP. Alternatively, the above-mentioned processor 210 may be integrated in a system on chip (SoC). Alternatively, the above-mentioned processor 210 may be integrated in an integrated circuit (IC) chip. The processor 210 may include an analog front end (AFE) and a micro-controller unit (MCU) in an IC chip.
[0052] The controller may be the nerve center and command center of the first electronic device 101. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0053] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.
[0054] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.
[0055] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the first electronic device 101. In other embodiments of the present application, the first electronic device 101 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0056] The power management module 240 is used to receive charging input from a charger. The charger may be a wireless charger (such as a wireless charging base of the first electronic device 101 or other device that can wirelessly charge the first electronic device 101), or a wired charger. For example, the power management module 240 may receive charging input from a wired charger through the USB interface 230. The power management module 240 may receive wireless charging input through the wireless charging coil 242 of the electronic device.
[0057] The power management module 240 can charge the battery 241 and also power the electronic device. The power management module 240 receives input from the battery 241 and powers the processor 210, the internal memory 221, the external memory interface 220, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 240 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance) of the battery 241. In some other embodiments, the power management module 240 can also be set in the processor 210.
[0058] The wireless communication function of the first electronic device 101 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, the baseband processor, and the like.
[0059] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first electronic device 101 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0060] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the first electronic device 101. The wireless communication module 260 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), frequency modulation (FM), nearfield communication technology (NFC), infrared technology (IR), etc., applied to the first electronic device 101. In some embodiments, the antenna 1 of the first electronic device 101 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the first electronic device 101 can communicate with the network and other devices through wireless communication technology.
[0061] The first electronic device 101 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.
[0062] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the first electronic device 101 may include 1 or N display screens 294, where N is a positive integer greater than 1.
[0063] The first electronic device 101 can realize the shooting function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, and an application processor. The ISP is used to process the data fed back by the camera 293. In some embodiments, the ISP can be set in the camera 293. The camera 293 is used to capture a static image or a video. In some embodiments, the electronic device may include 1 or N cameras 293, where N is a positive integer greater than 1. Exemplarily, the camera of the embodiment of the present application includes a wide-angle camera and a main camera.
[0064] The external memory interface 220 can be used to connect an external memory card, such as a micro SanDisk (Micro SD) card, to expand the storage capacity of the first electronic device 101. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos are stored in the external memory card.
[0065] The internal memory 221 may be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the first electronic device 101 by running the instructions stored in the internal memory 221. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0066] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0067] The first electronic device 101 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D and the application processor.
[0068] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio input. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210. The speaker 270A, also known as the "speaker", is used to convert the audio electrical signal into a sound signal. The receiver 270B, also known as the "earpiece", is used to convert the audio electrical signal into a sound signal. The microphone 270C, also known as the "microphone", is used to convert the sound signal into an electrical signal. The first electronic device 101 can be provided with at least one microphone 270C. The headphone interface 270D is used to connect a wired headset. The headphone interface 270D can be a USB interface 230, or it can be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0069] The button 290 includes a power button, a volume button, etc. The button 290 may be a mechanical button. It may also be a touch button. The first electronic device 101 may receive a button input and generate a key signal input related to the user settings and function control of the first electronic device 101. The motor 291 may generate a vibration prompt. The motor 291 may be used for an incoming call vibration prompt or for touch vibration feedback. The indicator 292 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect a SIM card. The SIM card may be inserted into the SIM card interface 295 or pulled out from the SIM card interface 295 to achieve contact and separation with the first electronic device 101. The first electronic device 101 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 may support a Nano SIM card, a Micro SIM card, a SIM card, etc. In some embodiments, the first electronic device 101 uses an embedded SIM (eSIM) card. The eSIM card can be embedded in the first electronic device 101 and cannot be separated from the first electronic device 101.
[0070] The processor 210 executes the audio processing method provided in the embodiment of the present application by executing the program and instructions stored in the internal memory 221. The program run by the processor 210 can be based on an operating system, such as Android. Apple (iOS) Windows Etc. Figure 3 As shown, the program running on the processor 210 is based on Android For example, the programs run by the processor 210 are layered according to their functions, and may include an application layer, a framework layer, and a kernel layer.
[0071] The application layer may include an audio processing program. The audio processing program is used to process the original audio to obtain the target audio to be played. The target audio is the audio that meets the user's listening needs.
[0072] The framework layer is used to provide system resource services and application programming interfaces (APIs) to applications in the application layer. For example, the framework layer provides a communication API to the application layer.
[0073] The kernel layer includes the operating system (OS) kernel, which is used to manage the system's processes, memory, drivers, file systems, and network systems.
[0074] The second electronic device 102 involved in the embodiment of the present application may be a terminal device with relatively simple functions and configurations compared to the first electronic device 101. For example, the second electronic device 102 may be a device with an audio playback function, such as a headset, a virtual reality (VR) device, or an augmented reality (AR) device.
[0075] In one embodiment, when the second electronic device 102 is a headset, since the data processing capability of the headset is limited or the headset does not have data processing capability, the target audio to be played can be determined by the first electronic device 101 and sent to the second electronic device 102. The second electronic device 102 is used to play the target audio. The headset can be a Bluetooth headset (such as a true wireless stereo (TWS) headset), a wired headset, etc.
[0076] In another embodiment, when the second electronic device 102 is a virtual reality device or an augmented reality device with strong data processing capabilities, the second electronic device 102 is used to determine the target audio and play the target audio.
[0077] For example, taking the second electronic device as a Bluetooth headset, Figure 4 FIG. 1 shows a possible structure of the second electronic device 102. Figure 4As shown, the second electronic device 102 may include: at least one processor 401, at least one memory 402, a wireless communication module 403, an audio module 404, a power module 405, an input / output interface 406, and a sensor 407. The processor 401 may include one or more interfaces for connecting with other components of the Bluetooth headset. The Bluetooth headset may also be stored in a headset storage box. Figure 4 Give a detailed introduction to the various components of Bluetooth headsets.
[0078] Among them, the sensor 407 may include a distance sensor, a proximity light sensor, a bone conduction sensor, a touch sensor, etc. The Bluetooth headset can be used to determine whether the Bluetooth headset is worn by the user through the distance sensor and the proximity light sensor. The Bluetooth headset can obtain the vibration signal of the vibrating bone of the human vocal part through the bone conduction sensor, parse the voice signal, realize the voice function, and thus receive the user's voice command, etc. The Bluetooth headset can detect the user's touch operation through the touch sensor. The touch operation may include the user's single click, double click, multiple clicks, long press, heavy pressure and other operations.
[0079] The wireless communication module 403 can be used to support data exchange between the Bluetooth headset and other electronic devices or headset boxes through wireless communication technology. The wireless communication module 403 may also include an antenna. The wireless communication module 403 receives electromagnetic waves via the antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 401. The wireless communication module 403 can also receive the signal to be sent from the processor 401, modulate and amplify it, and then convert it into electromagnetic waves for radiation through the antenna.
[0080] The audio module 404 can be used to manage audio data and implement the input and output of audio data by the Bluetooth headset. For example, the audio module 404 can obtain audio data from the wireless communication module 403, or transmit audio data to the wireless communication module 403, to implement functions such as playing music through the Bluetooth headset. The audio module 404 may include a speaker (or earpiece, receiver) component for outputting audio data. The speaker can be used to convert audio electrical signals into sound signals and play them.
[0081] In one embodiment, the audio module 404 may also include a feedback microphone (or microphone, speaker), and a microphone sound receiving circuit matched with the feedback microphone, etc. The feedback microphone may be used to convert a sound signal into an audio electrical signal.
[0082] The power module 405 can be used to provide system power for the Bluetooth headset, power each module of the Bluetooth headset, and support the Bluetooth headset to receive charging input, etc. The power module 405 may include a power management unit (PMU) and a battery. Among them, the power management unit can receive external charging input; transform the electrical signal input by the charging circuit and provide it to the battery for charging, and can also transform the electrical signal provided by the battery and provide it to other modules such as the audio module 404 and the wireless communication module 403; and prevent the battery from being overcharged, over-discharged, short-circuited or overcurrent, etc. In some embodiments, the power module 405 may also include a wireless charging coil for wirelessly charging the Bluetooth headset. In addition, the power management unit can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc.
[0083] Multiple input / output interfaces 406 can be used to provide a wired connection for charging or communication between the Bluetooth headset and the headset box. In some embodiments, the input / output interface can be a USB interface. In other embodiments, the input / output interface 406 can be a headset electrical connector. When the Bluetooth headset is placed in the headset box, the Bluetooth headset can establish an electrical connection with the electrical connector in the headset box through the headset electrical connector, thereby charging the battery in the Bluetooth headset. In other embodiments, after the electrical connection is established, the Bluetooth headset can also communicate data with the headset box, for example, it can receive a pairing instruction from the headset box.
[0084] The memory 402 can be used to store program codes, such as program codes for physically connecting the Bluetooth headset with multiple electronic devices, connecting with electronic devices in accordance with business regulations, processing audio services of electronic devices (such as music playing, answering / making calls, etc.), charging the Bluetooth headset, and wirelessly pairing the Bluetooth headset with other electronic devices. For example, when a user wears a Bluetooth headset, the program codes for determining the first transfer function between the speaker and the feedback microphone of the Bluetooth headset, determining the transient response function (i.e., the second transfer function) of the Bluetooth headset in the above-mentioned acoustic system, and determining the target audio are used. The memory 402 can also be used to store other information, such as the priority of electronic devices.
[0085] The processor 401 can be used to execute the program code stored in the memory 402, and replace the relevant modules to realize the functions of the Bluetooth headset. For example, when the user wears the Bluetooth headset, the processor determines the first transfer function between the speaker and the microphone based on the audio data output by the speaker of the Bluetooth headset and the audio data collected by the feedback microphone of the Bluetooth headset. For another example, the transient response function (i.e., the second transfer function) of the Bluetooth headset in the above-mentioned acoustic system is determined, and the target audio is determined.
[0086] It is understandable that the structure shown in the embodiment of the present application does not constitute a specific limitation on the Bluetooth headset. Figure 4 More or fewer components shown may be combined with two or more components, or may have different component configurations. For example, the outer surface of the Bluetooth headset may also include a button 408, an indicator light (which may indicate battery level, incoming / outgoing call, pairing mode, etc.), a display screen (which may prompt the user with relevant information), a dust screen (which may be used in conjunction with a handset), and other components. The button 408 may be a physical button or a touch button (used in conjunction with a touch sensor), etc., which is used to trigger operations such as power on, power off, pause, play, record, start pairing, and reset.
[0087] During the process of the second electronic device playing the input audio: Figure 5 As shown, the amplitude of the output audio of the speaker of the second electronic device has a process of building up from 0 to the maximum value; when the second electronic device stops playing the input audio, the diaphragm of the speaker of the second electronic device will not stop vibrating immediately, resulting in that the amplitude of the output audio of the speaker of the second electronic device will not decay immediately. Therefore, there will be a tail in the process of the amplitude of the output audio from the maximum value to 0. If the time of the building process of the output audio and the time of the tail are long, it is considered that the transient response of the speaker of the second electronic device is poor. If the transient response of the speaker of the second electronic device is poor, it will cause the speaker of the second electronic device to be difficult to accurately reproduce the input audio. In other words, if the transient response of the speaker of the second electronic device is poor, the overlap between the waveform of the output audio and the waveform of the input audio will be low. On the contrary, the overlap between the waveform of the output audio and the waveform of the input audio is high.
[0088] Combine the following Figure 6 and Figure 7 , briefly describing the relationship between the waveform graph of the transient response function of the speaker of the second electronic device, the waveform graph of the input audio, and the waveform graph of the output audio.
[0089] Exemplarily, assuming that the input audio is a “clean” audio, this “clean” audio can be represented by an impulse signal δ(n) in the time domain; the first transient response function of the speaker of the second electronic device is h1(n). Figure 6 δ(n) in
[0090] h1(n) y1(n)A shows the waveform of the input audio. Figure 6 B in FIG. 1 shows a waveform diagram of a first transient response function of a loudspeaker of an audio playback device. Figure 6 C in Figure 1 shows the waveform of the output audio. Figure 6 The waveform diagram in A to Figure 6It can be seen from the waveform diagram in C that the waveform diagram of the output audio of the speaker of the second electronic device is the same as the waveform diagram of the first transient response function of the speaker of the second electronic device, that is, y1(n)=h1(n), but the waveform diagram of the output audio of the speaker of the second electronic device is different from the waveform diagram of the input audio. Therefore, the output audio of the speaker of the second electronic device can be described by the transient response function of the speaker of the second electronic device.
[0091] When the input audio is continuous audio, the transient response of the speaker of the second electronic device has a greater impact on the output audio of the speaker of the second electronic device. Specifically, the output audio of the speaker of the second electronic device is not only related to the input audio of the speaker of the second electronic device at the current moment, but also to the input audio of the speaker of the second electronic device in the previous period of time.
[0092] Exemplarily, assuming that the input audio played by the audio playback device is an impulse signal δ(n) in the time domain, the second transient response function of the speaker of the audio playback device is δ(n-τ). Figure 7 A in FIG. 1 shows a waveform diagram of a second transient response function δ(n-τ) of a speaker of a second electronic device under an ideal condition. Figure 7 B in FIG. 1 shows the waveform of the output audio y2(n). Figure 6 The waveform diagram in A, Figure 7 The waveform diagram in A in , and Figure 7 From the waveform diagram of B in , we can know that the waveform diagram of the output audio of the speaker of the second electronic device is the same as the waveform diagram of the second transient response function of the speaker of the second electronic device, that is, y2(n)=δ(n-τ); and the waveform diagram of the output audio of the speaker of the second electronic device is the same as the waveform diagram of the input audio of the speaker of the second electronic device. It can be seen that the second transient response function δ(n-τ) is the transient response function of the ideal speaker.
[0093] In summary, the output audio of the speaker of the second electronic device is related to the transient response function of the speaker of the second electronic device. Therefore, by improving the transient response function of the speaker of the second electronic device, the performance of the speaker of the second electronic device can be improved, thereby improving the quality of the output audio of the second electronic device, and then improving the user's listening experience. For example, if you want to make the output audio of the second electronic device consistent with the input audio of the second electronic device, that is, to achieve audio output with high-fidelity sound effects, you can adjust the transient response function of the speaker of the second electronic device to δ(n-τ).
[0094] When the user wears the second electronic device to listen to audio, Figure 8As shown, the second electronic device and the user's ear canal form an acoustic system (or acoustic environment). The output audio (i.e., the first audio) output by the speaker of the second electronic device is the input signal of the acoustic system, and the audio received at the user's eardrum position is the audio heard by the user. The audio heard by the user is an output signal of the acoustic system. Therefore, the transient response function of the second electronic device in the acoustic system will affect the user's listening experience. The transient response function of the acoustic system is related to the characteristics of the user's ear canal, and the wearing state of the user when wearing the second electronic device. Therefore, on the basis of the original hardware of the second electronic device, in order to improve the user's listening experience, it is also necessary to consider the characteristics of the user's ear canal, and the wearing state of the user when wearing the second electronic device.
[0095] It should be noted that the ear canal characteristics may include: ear canal size, ear canal shape, eardrum acoustic resistance, etc., which can reflect the physiological acoustic characteristics of the human ear. The wearing state refers to the tightness of the second electronic device when the user wears it.
[0096] Based on this, an embodiment of the present application provides an audio processing method. When a user wears a second electronic device to listen to audio, the first electronic device (or the second electronic device) processes the original audio on the basis of the original hardware of the second electronic device, combined with the user's ear canal characteristics, the user's wearing status, and the target sound effects selected by the user, to obtain target audio that can meet the user's listening needs, thereby allowing the user to have a better listening experience while wearing the second electronic device to listen to audio.
[0097] Exemplarily, it is assumed that the audio processing method is executed by a first electronic device, and the first electronic device is a mobile phone. Fig. 9 As shown in A in FIG. 1 , the setting interface of the mobile phone includes a sound and vibration control 901. In response to the user clicking the sound and vibration control 901, as shown in FIG. Fig. 9 As shown in B, the mobile phone jumps from the setting interface to the sound and vibration interface 902. The sound and vibration interface 902 includes an audio processing control 903. In response to the user clicking the audio processing control 903, the mobile phone turns on the audio processing function, starts running the audio processing program, and executes the audio processing method provided in the embodiment of the present application.
[0098] Specifically, Fig.10 As shown, the following description is made by taking the first electronic device as an example in which the first electronic device is used to provide the original audio, and obtains the target audio to be played according to the original audio, and the second electronic device is used to play the target audio. The audio processing method may include:
[0099] S1001: A first electronic device obtains a first transfer function and a target sound effect.
[0100] The first transfer function is used to indicate the response of the audio played by the speaker of the second electronic device after being transmitted by the acoustic system and received by the second electronic device. The acoustic system is related to the characteristics of the user's ear canal and the wearing state of the user when wearing the second electronic device. The target sound effect is the sound effect selected by the user, which is the effect that the user expects to achieve when the audio is played.
[0101] The following describes the methods of obtaining the first transfer function and the target sound effect.
[0102] like Figure 8 As shown, when the user wears the second electronic device, the second electronic device and the user's ear canal form an acoustic system (or acoustic environment). The first transfer function can be the corresponding relationship between the input signal and the first output signal of the acoustic system. The input signal can be the first audio played by the speaker of the second electronic device, and the first output signal can be the second audio received by the second electronic device after the first audio is transmitted through the acoustic system.
[0103] The second electronic device can receive the second audio through a built-in feedback microphone, but not every second electronic device includes a feedback microphone. Of course, not every second electronic device has a data processing function. Therefore, how to obtain the first transfer function is related to whether the second electronic device includes a feedback microphone and whether the second electronic device has a data processing function. The following are examples.
[0104] In one embodiment, the second electronic device includes a built-in feedback microphone, and the second electronic device has a data processing function. At this time, the first audio played by the speaker of the second electronic device will be transmitted to the feedback microphone of the second electronic device through the above-mentioned acoustic system, so that the feedback microphone of the second electronic device receives the second audio. The second electronic device processes the first audio and the second audio to obtain a first transfer function. Then, the second electronic device sends the first transfer function to the first electronic device. That is to say, when the second electronic device includes a feedback microphone and the second electronic device has a data processing function, the first transfer function can be determined by the second electronic device, and the first electronic device can receive the first transfer function sent by the second electronic device.
[0105] Of course, the second electronic device may also send the first audio and the second audio to the first electronic device. Then, the first electronic device processes the first audio and the second audio to directly obtain the first transfer function.
[0106] The first audio may be: an in-ear prompt tone pre-stored in the second electronic device, a downlink audio or a test audio sent from the first electronic device to the second electronic device.
[0107] Exemplarily, it is assumed that the first electronic device is a mobile phone and the first audio is a test audio. Fig.11 As shown in A in FIG. 1 , the main interface 1101 of the mobile phone displays the application icon 1102 for settings. In response to the user clicking the application icon 1102 for settings, as shown in FIG. Fig.11 As shown in B in FIG. 1 , the mobile phone jumps from the main interface 1101 to the setting interface 1103. The setting interface 1103 displays the switch control 1104 of the ear canal adaptation. In response to the user clicking the switch control 1104 of the ear canal adaptation, as shown in FIG. Fig.11 As shown in C, the ear canal adaptation function is switched to the on state, and at the same time, the mobile phone sends a test audio to the second electronic device.
[0108] Optionally, an adaptive filter may be pre-stored in the second electronic device. The adaptive filter may include: an adjustable filter with an adjustable filter coefficient, and an adaptive filtering algorithm for adjusting the filter coefficient of the adjustable filter. The second electronic device obtains a target filter coefficient of the adjustable filter through the adjustable filter, the adaptive filtering algorithm, the first audio, and the second audio, thereby obtaining the first transfer function.
[0109] For example, Fig.12 A schematic diagram of the framework structure of an adaptive filter is shown in FIG. Fig.12 As shown, the signals involved in the adaptive filter include: an input signal (first audio), an expected signal, and an output signal (second audio). In the process of determining the first transfer function: first, the second electronic device determines the difference between the expected signal and the second audio corresponding to the first audio as an error signal. Again, the second electronic device adjusts the filter coefficient of the adjustable filter according to the first audio, the error signal, the adjustable filter, and the adaptive filtering algorithm until the error signal is minimized. When the error signal is minimized, the target filter coefficient of the adjustable filter can be obtained, thereby obtaining the target filter. The target filter is the first transfer function.
[0110] In actual application, once the input signal changes, the adjustable filter can automatically track the changes of the input signal, thereby automatically adjusting its own filter coefficient until the target filter coefficient is obtained, thereby realizing an adaptive process.
[0111] The above-mentioned adaptive filtering algorithms may include: least mean square algorithm (least mean square, LMS), normalized least mean square algorithm (normalized least mean square, NLMS), recursive least squares algorithm (recursive least square, RLS), but are not limited thereto.
[0112] Exemplarily, if the adaptive filtering algorithm is a minimum mean square algorithm, the filter coefficient of the adjustable filter can be determined by the following formula (1):
[0113]
[0114] Where μ is the step size; W o is the filter coefficient of the adjustable filter at the current iteration number; W o-1 is the filter coefficient of the adjustable filter in the previous iteration; S f is the frequency domain sequence of the first audio; f is a frequency domain sequence of the second audio; is the conjugate function of the frequency domain sequence of the first audio; O is the number of iterations.
[0115] Of course, the adaptive filter may also be pre-stored in the first electronic device. After the first electronic device receives the first audio and the second audio sent by the second electronic device, the first electronic device may also directly obtain the first transfer function using the above process.
[0116] In summary, since the ear canal characteristics and wearing states of different users are different, the acoustic systems of different users are also different. That is, the first transfer functions of different users are different. When the second electronic device includes a feedback microphone, the second electronic device (or the first electronic device) can obtain the first transfer functions of different users, that is, it can automatically identify the ear canal characteristics of the user and determine the wearing state of the user when wearing the second electronic device. In this way, for different users, the target audio that meets the listening needs of different users can be "customized", thereby improving the user's listening experience. Moreover, in the actual application process, as long as the speaker of the second electronic device plays the first audio, the first electronic device can obtain a first transfer function. In this way, the second electronic device (or the first electronic device) can make timely adjustments in the process of obtaining the target audio. Based on this, any target audio played can meet the user's listening needs and further improve the user's listening experience.
[0117] In another embodiment, the second electronic device does not include a feedback microphone. If the second electronic device does not include a feedback microphone, the second electronic device cannot obtain the second audio, and the second electronic device cannot obtain the first transfer function. In this case, the first transfer function is a preset transfer function.
[0118] The preset transfer function may be a transfer function obtained by wearing the second electronic device including the feedback microphone in a standard wearing state of the artificial ear or artificial head in a test environment, and adopting the above-mentioned method of determining the first transfer function. The specific process may refer to the above process, which will not be described in detail here.
[0119] It should be noted that the artificial head is a device based on ergonomic design, which includes an artificial ear. The artificial ear is a simulation device that simulates the physical characteristics of the human ear, and the artificial ear has the same acoustic characteristics as a real human ear. In the embodiment of the present application, the artificial ear has a standard ear canal (i.e., a target ear canal). The standard ear canal can be designed by the tester after analyzing a large amount of ear canal characteristic sample data. The standard wearing state refers to the state of proper tightness when wearing the second electronic device.
[0120] Optionally, the first electronic device may determine the configuration parameters of the second electronic device according to the model and type of the second electronic device, thereby determining whether the second electronic device includes a feedback microphone, and further determining whether to determine the preset function as the first transfer function. If the first electronic device determines that the second electronic device does not include a feedback microphone, the first electronic device directly determines the preset transfer function as the first transfer function. The following briefly introduces how the first electronic device determines whether the second electronic device includes a feedback microphone.
[0121] In one embodiment, the first electronic device may pre-store configuration parameters of multiple models and types of second electronic devices. In actual application, the first electronic device may obtain the model and type of the second electronic device in response to the user selecting the second electronic device that the user is wearing from the pre-stored multiple second electronic devices. The first electronic device determines whether the model and type of the second electronic device includes a feedback microphone based on the configuration parameters of the model and type of the second electronic device.
[0122] For example, it is assumed that the first electronic device is a mobile phone and the second electronic device is a headset. Fig.13 As shown, the mobile phone displays an adapter headset model interface 1301. The adapter headset model interface 1301 includes two sections: headset type and headset model. In the headset type section, a variety of headset types are displayed; in the headset model section, a variety of headset models are displayed. First, in response to the user's selection of the headset type and headset model, the mobile phone device obtains the model and type of the headset. Then, the mobile phone searches for the configuration parameters of the headset according to the model and type, thereby determining whether the headset of the model and type includes a feedback microphone.
[0123] In another embodiment, the second electronic devices pre-stored in the first electronic device do not include the second electronic device worn by the user. First, the first electronic device can obtain the model and type of the second electronic device in response to the user inputting the model and type of the second electronic device worn by the user in the first electronic device. Then, the first electronic device can search for the configuration parameters of the second electronic device of the model and type through the Internet, thereby determining whether the second electronic device includes a feedback microphone.
[0124] For example, it is assumed that the first electronic device is a mobile phone. Fig.13 ,like Fig.14 As shown in A in FIG. 1 , the mobile phone's headset model interface 1301 further includes: a first add control 1401 and a second add control 1402. The first add control 1401 is located in the headset type section, and the second add control 1402 is located in the headset model section. In response to the user clicking the first add control 1401, as shown in FIG. Fig.14 As shown in B in FIG. 1 , the mobile phone's adapter headset model interface 1301 may display a first drop-down list control 1403. In response to the user clicking the first drop-down list control 1403, as shown in FIG. Fig.14 As shown in C in FIG. 1 , the model section of the headset in the headset model interface 1301 displays a headset type list 1405. The headset type list includes controls for multiple types of headsets (such as general neck-hanging type). In response to the user clicking on the general neck-hanging type control, such as Fig.14 As shown in D in FIG. 1 , the model section of the headset in the adapter headset model interface 1301 displays a general neck-hanging selection control 1407, and the general neck-hanging selection control 1407 displays a selected state. In response to the user clicking the second add control 1402, as shown in FIG. Fig.14 As shown in B in FIG. 1 , the mobile phone's compatible headset model interface 1301 may display a second drop-down list control 1404. In response to the user clicking the second drop-down list control 1404, as shown in FIG. Fig.14 As shown in C in FIG. 1 , the model section of the headset in the headset model adaptation interface 1301 displays a headset model list 1406. The headset model list includes controls for multiple models of headsets (such as Type A Youth Edition). In response to the user clicking on the control for Type A Youth Edition, Fig.14 As shown in D in FIG. 1 , the model section of the headset in the headset model interface 1301 displays a selection control 1408 for the type A youth version, and the selection control 1408 for the type A youth version displays a selected state. Finally, the mobile phone obtains the configuration parameters of the headset from the Internet according to the model and type of the headset added by the user, thereby determining whether the model and type of the headset includes a feedback microphone.
[0125] In another embodiment, when the first electronic device is connected to the second electronic device, the second electronic device will actively send its own model and type to the first electronic device. Based on the received model and type, the first electronic device first searches the first electronic device to see whether the second electronic device is pre-stored. If the first electronic device finds the second electronic device of the model and type, it directly determines whether the second electronic device includes a feedback microphone based on the configuration parameters of the second electronic device. If the first electronic device does not find the second electronic device of the model and type, the first electronic device can obtain the configuration parameters of the second electronic device through the Internet to determine whether the second electronic device includes a feedback microphone.
[0126] In another embodiment, if the first electronic device cannot obtain the model and type of the second electronic device, and the first electronic device does not receive the first audio and the second audio sent by the second electronic device within a preset time period (such as 3 seconds after the first electronic device and the second electronic device are connected), or does not receive the first transfer function sent by the second electronic device, the first electronic device determines that the second electronic device does not include a feedback microphone.
[0127] The first electronic device involved in the embodiment of the present application may have a plurality of sound effects pre-stored therein. The first electronic device may obtain a target sound effect in response to a selection operation of a user on the first electronic device. The sound effects pre-stored therein may include: high-fidelity sound effects, stage sound effects (or live sound effects), and electronic sound effects, but are not limited thereto.
[0128] For example, it is assumed that the first electronic device is a mobile phone. Fig.15 As shown, the mobile phone displays a sound effect selection interface 1501. The sound effect selection interface includes: a high-fidelity sound effect selection control 1502, a stage sound effect selection control 1503, and an electronic sound effect selection control 1504. In response to the user clicking the high-fidelity sound effect selection control 1502, the mobile phone determines the high-fidelity sound effect as the target sound effect.
[0129] It should be noted that the sound effect selection interface and the sound and vibration interface may be associated with each other. For example, when the user triggers the audio processing control, the first electronic device may jump from the sound and vibration interface to the sound effect selection interface.
[0130] S1002: The first electronic device obtains a second transfer function according to the first transfer function.
[0131] The second transfer function may be the correspondence between the input signal and the second output signal of the acoustic system, that is, the second transfer function is the transient response function of the second electronic device in the acoustic system. The second output signal may be the third audio received at the eardrum position of the user, that is, the audio actually heard by the user. Therefore, the second transfer function may reflect the actual listening experience of the user when hearing the audio played by the speaker of the second electronic device.
[0132] Among them, since the third audio frequency cannot be measured, the second transfer function cannot be determined by the method of determining the first transfer function. However, since the first transfer function and the second transfer function are transfer functions at different measurement positions in the same acoustic system, the two are interrelated, and therefore, the second transfer function can be determined by the first transfer function.
[0133] The first electronic device has a first neural network model pre-stored therein. The first neural network model is used to indicate a mapping relationship between a first transfer function and a second transfer function. The first electronic device inputs the first transfer function into the first neural network model, and after being processed by the first neural network model, the first electronic device obtains the second transfer function.
[0134] Optional, such as Fig.16 As shown, the first neural network model involved in the embodiment of the present application may include: an input layer, at least one hidden layer and an output layer according to the flow direction of data. Among them, the output layer can generally be a fully connected layer (fully connected layer, FC), and the hidden layer can be a convolutional layer of a convolutional neural network (convolutional neural network, CNN), a long short term memory (long short term memory, LSTM) layer, etc., or a combination of these layers, so as to achieve different functions. The first neural network model can perform linear operations on data and network parameters, and realize nonlinear operations through activation functions. In addition, in order to simplify the neural network model, the operation results of the linear operation output will also be taken as the maximum value, minimum value, average value and pooling operations.
[0135] Among them, the linear operation in the first neural network model can be used d =aG e +b to indicate G d represents the output (i.e., the second transfer function), G e represents the input (i.e., the first transfer function), a represents the weight, and b represents the bias. e It can be called general matrix multiplication (GEMM), and the network parameters of the first neural network include weight a and bias b.
[0136] A commonly used activation function is the rectified linear units (ReLU). The role of the activation function is to add nonlinear elements to the first neural network model so that the first neural network model can better solve complex problems. If the activation function is not used and only linear transformation is performed, then even if there are more layers, it is only equivalent to a combination of multiple linear equations. The entire first neural network model is only equivalent to a linear regression model, and its ability to solve complex problems is limited. The activation function can achieve nonlinear transformation, allowing the first neural network model to learn and perform more complex tasks.
[0137] Pooling, also known as spatial pooling, is a method used in CNN to extract features. It aggregates and statistically processes different features to obtain a relatively lower dimension while avoiding overfitting. Pooling can reduce the dimensions of each feature while retaining most of the important information.
[0138] Optionally, the training process of the first neural network model is as follows:
[0139] First, collect a large number of first sample data sets. Specifically, recruit a large number of volunteers. First, make each volunteer wear headphones (the headphones include a feedback microphone, such as true wireless stereo (TWS) headphones) repeatedly for many times, and install a probe microphone on the eardrum of each volunteer. Then, play the first test audio through the speaker of the headphones. The first test audio is received by the feedback microphone of the headphones through the ear canal of the volunteer to obtain the second test audio. The first test audio is received by the probe microphone through the ear canal of the volunteer to obtain the third test audio. Finally, based on the first test audio and the second test audio, obtain the first sample transfer function According to the first test audio and the third test audio, the second sample transfer function is obtained Where i is the volunteer number, i=1, ...I, I is a positive integer greater than 1; j is the wearing number, j=1, ...J, J is a positive integer greater than 1. Based on this, multiple groups of first sample data sets for training the first neural network model can be obtained.
[0140] Among them, the ear canals of the volunteers include ear canals of different sizes, ear canals of different shapes, ear canals with different eardrum sound resistance, normal ear canals, pathological ear canals, etc. In other words, the multiple sets of first sample data sets collected contain a variety of ear canal characteristics. At the same time, the wearing state of the volunteers each time they wear the headphones can also be different, so that the trained first neural network model is more comprehensive and has a higher prediction accuracy.
[0141] Second, first, a set of first sample data sets Input the neural network model to be trained to obtain the second transfer function of the actual output Then, determine the second sample transfer function With the second transfer function The error is used to adjust the weights between the hidden layers of the neural network model to be trained. This process is repeated for each first sample data set until the error converges to the control range for all first sample data sets. In this way, a trained first neural network model can be obtained.
[0142] S1003: The first electronic device obtains target audio to be played according to the second transfer function, the original audio, and the target sound effect.
[0143] Optionally, the process of the first electronic device obtaining the target audio may include: first, the first electronic device obtains a target filter or a target neural network for processing the original audio according to the second transfer function and the target sound effect. Then, the first electronic device processes the original audio through the target filter or the target neural network to obtain the target audio to be played.
[0144] The second transfer function can be used to indicate the real listening experience of the user when wearing the second electronic device to listen to the audio, the target sound effect is the effect that the user expects to achieve when the audio is played, and a sound effect can be represented by a transient response function. Therefore, the target filter or target neural network that can process the second transfer function into the target sound effect can process the original audio into the target audio that meets the user's listening needs.
[0145] Combination Figure 6 In B, such as Fig.17 As shown, when the target sound effect is high-fidelity sound effect, the second transfer function G d The time domain series g d After being processed by the target filter or the target neural network, the transient response function h2(n) of the target sound effect is obtained. It should be noted that the parameter τ in the transient response function h2(n) of the target sound effect needs to be calculated according to g d The time delay is set so that the above acoustic system satisfies causality.
[0146] Optionally, the process of first electronically determining the target filter is as follows:
[0147] The first electronic device may have multiple filters pre-stored therein. One filter may process the second transfer function into a transient response function corresponding to a sound effect. When the first electronic device obtains the target sound effect, the first electronic device may determine the target filter according to the correspondence between the sound effect and the filter.
[0148] Among them, filters can be divided into finite impulse response (FIR) filters (or non-recursive filters) and infinite impulse response (IIR) filters (or recursive filters) according to their types.
[0149] The following is a brief description of how to determine each filter in each type:
[0150] Exemplarily, assuming that the filter is a FIR filter, first, the order N of the FIR filter is set; then, the frequency response function of the FIR filter can be determined by the following formula (2).
[0151]
[0152]
[0153] Where w1 is the time domain sequence of the frequency response function of the FIR filter; g d is the time domain sequence of the second transfer function (G d is the frequency domain sequence of the second transfer function, g d With G d is a Fourier transform pair); * represents convolution operation; ||·|| 2 represents the square of the 2-norm; h3(n) is the time domain sequence of the transient response function of the target sound effect (for example, when the target sound effect is a high-fidelity sound effect, h3(n)=δ(n)).
[0154] For example, the FIR filter can be expressed by the following formula (3):
[0155]
[0156] Among them, W k is the frequency domain sequence of the frequency response function of the FIR filter; G d is the frequency domain sequence of the second transfer function; is the conjugate function of the frequency domain sequence of the second transfer function; H is the frequency domain sequence of the transient response function of the target sound effect.
[0157] Exemplarily, it is assumed that the filter is an IIR filter. The frequency response function of the IIR filter can be determined by the frequency response function of the FIR filter. Specifically, the frequency response function of the IIR filter can be determined by the following formula (4).
[0158]
[0159] Where w1 is the time domain sequence of the frequency response function of the FIR filter; f(c, d) is the time domain sequence of the frequency response function of the IIR filter; c and d are coefficients; ||·|| 2 Represents the square of the 2-norm.
[0160] Optionally, the process of first electronically determining the target neural network is as follows:
[0161] A plurality of second neural network models may be pre-stored in the first electronic device. The second neural network model is used to indicate the mapping relationship between the second transfer function and the transient response function of the sound effect, that is, a second neural network model can process the second transfer function into a transient response function corresponding to a sound effect. When the first electronic device acquires the target sound effect, the first electronic device can determine the target neural network model based on the matching relationship between the sound effect and the second neural network model.
[0162] Among them, the structure of the second neural network model can be the same as the structure of the first neural network model, and the embodiment of the present application is not limited to this.
[0163] S1004: The first electronic device sends target audio to the second electronic device.
[0164] The first electronic device sends the processed target audio to the second electronic device, and the user can listen to the audio that meets his / her listening needs by wearing the second electronic device.
[0165] It should be noted that if the above process of obtaining the target audio to be played is executed by the second electronic device, there is no need to execute S1004.
[0166] S1005: The second electronic device plays the target audio.
[0167] The target audio played by the second electronic device is obtained according to the listening needs of the user, and thus the listening experience of the user when wearing the second electronic device to listen to the audio can be improved.
[0168] In summary, in the audio processing method provided by the embodiment of the present application, first, through the first transfer function that can be determined by measurement and is used to indicate the response of the second electronic device received by the second electronic device after the audio played by the speaker of the second electronic device is propagated through the acoustic system when the user wears the second electronic device, it is determined that the transient response function (i.e., the second transfer function) of the second electronic device in the acoustic system cannot be determined by measurement, but can indicate the user wearing the second electronic device, and then, the target sound effect selected by the user and the second transfer function are combined to obtain the target filter or target neural network model for processing the original audio. Then, the original audio is processed by the target filter or the target neural network model to obtain the target audio that meets the user's listening needs. Since the second transfer function can represent the real listening experience when the user wears the second electronic device, and the target sound effect is the effect achieved when the user expects the audio to be played, therefore, the target filter or the target neural network that can process the second transfer function into the target sound effect can process the original audio into the target audio that meets the user's listening needs.
[0169] In addition, the audio processing method provided in the embodiment of the present application not only optimizes the amplitude-frequency response of the original audio, but also optimizes the amplitude and phase of the original audio, so that the quality of the obtained target audio is better.
[0170] like Fig.18 As shown, the embodiment of the present application also provides a chip system. The chip system 1800 includes at least one processor 1801 and at least one interface circuit 1802. The at least one processor 1801 and the at least one interface circuit 1802 can be interconnected through a line. The processor 1801 is used to support the electronic device to implement each step in the above method embodiment, such as Fig.10 In the method shown, at least one interface circuit 1802 may be used to receive signals from other devices (eg, memory) or to send signals to other devices (eg, communication interface). The chip system may include a chip and may also include other discrete devices.
[0171] The present application also provides a computer storage medium, which includes instructions. When the instructions are executed on the electronic device, the electronic device executes each step in the method embodiment, such as executing Fig.10 The method shown.
[0172] The present application also provides a computer program product including instructions. When the instructions are executed on the electronic device, the electronic device executes each step in the method embodiment, such as executing Fig.10 The method shown.
[0173] Regarding the technical effects of the chip system, computer storage medium, and computer program product, refer to the technical effects of the previous method embodiments.
[0174] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0175] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0177] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0178] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0179] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.
[0180] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer storage medium, or transmitted from one computer storage medium to another computer storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0181] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An audio processing method, It is characterized in that Applied to a first electronic device, the first electronic device is connected to a second electronic device; an ear canal of a user and the second electronic device form an acoustic system, the method comprising: The first electronic device displays a setting interface, the setting interface includes a first control, and the first control is used to turn on or off the ear canal adaptation function; In response to the activation operation of the first control, the first electronic device acquires a first transfer function; wherein the first transfer function is used to indicate a response of the audio played by the speaker of the second electronic device and received by the second electronic device after being propagated by the acoustic system; the first transfer function is a preset transfer function, or the first transfer function is obtained based on the audio played by the speaker of the second electronic device and the audio received by the feedback microphone of the second electronic device; The first electronic device displays a sound effect selection interface, wherein the sound effect selection interface includes at least one sound effect; The first electronic device obtains a second transfer function based on the first transfer function in response to a selection operation of a target sound effect among the at least one sound effect, and processes the original audio according to the second transfer function and the target sound effect to obtain the target audio to be played; wherein the second transfer function is used to indicate the response of the audio played by the speaker of the second electronic device worn by the user, which is transmitted to the eardrum position of the user through the acoustic system.
2. The method according to claim 1, It is characterized in that In response to the opening operation of the first control, the first electronic device obtains a first transfer function, including: In response to an opening operation of the first control, the first electronic device sends a downlink audio or a test audio to the second electronic device; The first electronic device receives a first audio and a second audio sent by the second electronic device, wherein the first audio is audio played by a speaker of the second electronic device according to the downlink audio or the test audio, and the second audio is audio received by a feedback microphone of the second electronic device; The first electronic device obtains the first transfer function according to the first audio and the second audio.
3. The method according to claim 2, It is characterized in that The method further comprises: The first electronic device processes the first transfer function according to a mapping relationship between the first transfer function and the second transfer function to obtain the second transfer function.
4. The method according to claim 1, It is characterized in that In response to the opening operation of the first control, the first electronic device acquires the first transfer function, including: In response to an opening operation of the first control, the first electronic device sends a downlink audio or a test audio to the second electronic device; The first electronic device receives a first transfer function sent by the second electronic device; the first transfer function is obtained by the second electronic device based on first audio and second audio, the first audio is the audio played by the speaker of the second electronic device based on the downlink audio or the test audio, and the second audio is the audio received by the feedback microphone of the second electronic device.
5. The method according to claim 4, It is characterized in that The method further comprises: The first electronic device receives the second transfer function sent by the second electronic device; the second transfer function is obtained by the second electronic device processing the first transfer function according to a mapping relationship between the first transfer function and the second transfer function.
6. The method according to any one of claims 2 to 5, It is characterized in that The method further comprises: The first electronic device acquires a configuration parameter of the second electronic device; the configuration parameter is used to indicate whether the second electronic device includes the feedback microphone; The first electronic device determines whether the second electronic device includes the feedback microphone according to configuration parameters of the second electronic device.
7. The method according to claim 6, It is characterized in that In response to the opening operation of the first control, the first electronic device sends a downlink audio or a test audio to the second electronic device worn by the user, including: The first electronic device determines that the second electronic device worn by the user includes the feedback microphone, and the first electronic device sends downlink audio or test audio to the second electronic device worn by the user in response to the activation operation of the first control.
8. The method according to claim 6, It is characterized in that The first electronic device acquiring the configuration parameters of the second electronic device includes: The first electronic device displays a selection interface, wherein the selection interface includes the model and type of the second electronic device; The first electronic device acquires configuration parameters of the second electronic device in response to a selection operation on the model and type of the second electronic device.
9. The method according to claim 6, It is characterized in that The first electronic device acquiring the configuration parameters of the second electronic device includes: The first electronic device displays a selection interface, wherein the selection interface includes a first adding control and a second adding control; The first electronic device acquires the type of the second electronic device in response to a triggering operation on the first adding control; The first electronic device acquires the model of the second electronic device in response to the triggering operation on the second adding control; The first electronic device obtains configuration parameters of the second electronic device according to the type and model of the second electronic device.
10. The method according to claim 6, It is characterized in that The first electronic device acquiring the configuration parameters of the second electronic device includes: The first electronic device receives the model and type sent by the second electronic device; The first electronic device obtains configuration parameters of the second electronic device according to the type and model of the second electronic device.
11. The method according to any one of claims 1 to 5, It is characterized in that The step of processing the original audio according to the second transfer function and the target sound effect to obtain the target audio to be played includes: The first electronic device obtains a target filter or a target neural network model for processing the original audio according to the second transfer function and the target sound effect; The first electronic device processes the original audio through the target filter or the target neural network model to obtain the target audio.
12. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: The first electronic device sends the target audio to the second electronic device to play the target audio.
13. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: The first electronic device sends the target sound effect, the second transfer function and the original audio to the second electronic device; the second electronic device is used to process the original audio according to the second transfer function and the target sound effect to obtain the target audio to be played.
14. An audio processing method, It is characterized in that Applied to an electronic device, after a user wears the electronic device, the electronic device and the ear canal of the user form an acoustic system; the method comprises: The electronic device acquires a first transfer function and a target sound effect; wherein the first transfer function is used to indicate a response of the audio played by the speaker of the electronic device and received by the electronic device after being propagated by the acoustic system; the first transfer function is a preset transfer function, or the first transfer function is obtained based on the audio played by the speaker of the electronic device and the audio received by the feedback microphone of the electronic device; A second transfer function is obtained based on the first transfer function; wherein the second transfer function is used to indicate a response of the audio played by the speaker of the electronic device being transmitted to the eardrum position of the user through the acoustic system; The original audio is processed according to the second transfer function and the target sound effect to obtain the target audio to be played.
15. The method according to claim 14, It is characterized in that The electronic device acquires a first transfer function, including: The electronic device acquires a first audio and a second audio, wherein the first audio is audio played by a speaker of the electronic device, and the second audio is audio received by a feedback microphone of the electronic device; The electronic device obtains the first transfer function according to the first audio and the second audio.
16. The method according to claim 14 or 15, It is characterized in that The method further comprises: The electronic device processes the first transfer function according to a mapping relationship between the first transfer function and the second transfer function to obtain the second transfer function.
17. The method according to claim 14 or 15, It is characterized in that The method further comprises: The electronic device obtains a configuration parameter of the electronic device worn by the user; a configuration parameter of the electronic device is used to indicate whether the electronic device includes the feedback microphone; The electronic device determines whether the electronic device worn by the user includes the feedback microphone according to configuration parameters of the electronic device worn by the user.
18. The method according to claim 17, It is characterized in that The electronic device acquires the first audio and the second audio, including: In a case where it is determined that the electronic device worn by the user includes the feedback microphone, the electronic device acquires the first audio and the second audio.
19. The method according to claim 14 or 15, It is characterized in that The step of processing the original audio according to the second transfer function and the target sound effect to obtain the target audio to be played includes: The electronic device obtains a target filter or a target neural network model for processing the original audio according to the second transfer function and the target sound effect; The electronic device processes the original audio through the target filter or the target neural network model to obtain the target audio.
20. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, the method according to any one of claims 1 to 13 or any one of claims 14 to 19 is executed.
21. A computer-readable storage medium, It is characterized in that The method comprises instructions which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 13 or any one of claims 14 to 19.
