Audio data processing method, electronic equipment and storage medium

Abstract

The invention provides an audio data processing method, electronic equipment and a storage medium, and relates to the technical field of intelligent terminals. According to the audio data processing method provided by the invention, the global variable corresponding to the application identifier of each application can be created for each application in a scene that audio acquisition tasks of a plurality of applications run concurrently, and the audio parameters associated with the audio types of the applications are added to the corresponding global variables; the problem that parameters are covered due to the fact that multiple applications share one global variable is avoided. When a certain application is switched from the background to the foreground for operation, the corresponding global variable can be called according to the application identifier of the switched application, and then the audio data transmitted by the audio driver are processed according to the parameters in the global variable, so that the collected audio data are processed according to the audio parameters of the switched application, and the audio data processing efficiency is improved. The problem of silent recording in the prior art can be solved, and the user experience is improved.

Description

[0001] This application is a divisional application of the invention patent application with the application number 202311694470.8, the application date of December 12, 2023, and the invention creation name of "Audio Data Processing Method, Electronic Device and Storage Medium", which was submitted to the China National Patent Office. Technical Field

[0002] This application relates to the technical field of intelligent terminals, and in particular, to an audio data processing method, an electronic device, and a storage medium. Background Art

[0003] With the development of terminal technology, users' functional requirements for electronic devices are becoming more and more diverse. To meet users' recording needs for sound, most electronic devices support audio collection functions, such as functions like recording, video recording, or live streaming.

[0004] Currently, when a user uses a mobile phone with the digital audio zoom technology (Audiozoom) function, they first start a recording application to record, then switch the recording application to the background of the mobile phone to keep it running, and run a video recording or live streaming application in the foreground of the mobile phone. After the video recording or live streaming ends, switch the recording application running in the background to the foreground until the recording application finishes recording. There may be a problem of silent recording in this process. Summary of the Invention

[0005] To solve the above technical problems, this application provides an audio data processing method, an electronic device, and a storage medium. When the recording application is switched back to the foreground from the background after video recording or live streaming, a global variable corresponding to the application identifier of the recording application is called to process the audio data collected by the recording mic path, thereby solving the problem of silent recording in the background art.

[0006] In a first aspect, an embodiment of this application provides an audio data processing method, which is applied to an electronic device. The method includes: in response to a first operation on a first application, collecting first audio data through a first mic path; determining a first global variable corresponding to the first application, and processing the first audio data according to the parameters in the first global variable to obtain a first target audio corresponding to the first application; after switching the first application to the background, in response to a second operation on a second application, collecting second audio data through a second mic path; determining a second global variable corresponding to the second application, and processing the second audio data according to the parameters in the second global variable to obtain a second target audio corresponding to the second application; in response to a third operation to switch the first application back to the foreground, continuing to collect third audio data through the first mic path, and processing the third audio data according to the parameters in the first global variable to obtain a third target audio corresponding to the first application.

[0007] Among them, the first operation is an operation for starting the first application to collect audio, and the second operation is an operation for starting the second application to collect audio. The operation of collecting audio is like a recording application recording, a video recording application recording video, or a live broadcast application conducting a live broadcast. The first mic path is different from the second mic path. The first mic path includes a first mic and a second mic, and the second mic path includes a first mic, a second mic, and a third mic. For example, the first mic is a top microphone, the second mic is a bottom microphone, and the third mic is a back microphone. The first application is a recording application, and the second application is a video recording application or a live broadcast application.

[0008] In this way, in a scenario where audio collection tasks of multiple applications run concurrently, a global variable corresponding to the application identifier of each application can be created, and the audio parameters associated with its audio type are added to the corresponding global variable, avoiding the problem that parameters are overwritten due to multiple applications sharing a global variable. When switching a certain application from the background to the foreground, the corresponding global variable can be called according to the application identifier of the switched application. Furthermore, according to the parameters in the global variable, the audio data transmitted by the audio driver is processed to solve the problem of silent recording in the existing technology, ensure the processing effect of audio data under different applications, and improve the user experience.

[0009] According to the first aspect, determining the first global variable corresponding to the first application includes: creating a first global variable corresponding to the application identifier of the first application; adding the audio parameters associated with the audio type of the first application to the first global variable.

[0010] Among them, the audio type is a parameter related to the recording algorithm. The audio parameters corresponding to different audio types of the same application are different, or the numerical values of the audio parameters corresponding to different audio types of the same application are different.

[0011] In this way, a global variable corresponding to the application identifier of each application is created, and the audio parameters associated with its audio type are added to the corresponding global variable, facilitating subsequent applications to directly call the corresponding global variable to process audio data according to the application identifier of the application after switching from the background to the foreground, thereby improving the processing efficiency and effect of audio data.

[0012] According to the first aspect, or any one of the implementation manners of the above first aspect, after processing the third audio data according to the parameters in the first global variable to obtain the third target audio corresponding to the first application, the method further includes: in response to a fourth operation on the first application, closing the first application and the first mic path, and deleting the first global variable.

[0013] In this way, after the audio acquisition task of the application ends, by deleting the corresponding global variables, the memory of the global variables is released, the available memory space of the electronic device is increased, and further, the running fluency of the electronic device is improved, and the battery life of the electronic device is improved.

[0014] According to the first aspect, or any one of the implementation manners of the first aspect above, after switching the first application to run in the background and before responding to the second operation on the second application, the method further includes: acquiring fourth audio data through the first mic path; processing the fourth audio data according to the parameters in the first global variable to obtain fourth target audio corresponding to the first application.

[0015] In this way, it can be ensured that the audio acquisition tasks of the applications running in the background are continuously executed.

[0016] According to the first aspect, or any one of the implementation manners of the first aspect above, after processing the second audio data according to the parameters in the second global variable to obtain second target audio corresponding to the second application, the method further includes: responding to the fifth operation on the second application, closing the second application and the second mic path, and deleting the second global variable.

[0017] According to the first aspect, or any one of the implementation manners of the first aspect above, determining the second global variable corresponding to the second application includes: creating a second global variable corresponding to the application identifier of the second application; adding audio parameters associated with the audio type of the second application to the second global variable.

[0018] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, where one or more computer programs are stored on the memory, and when the computer programs are executed by one or more processors, the electronic device executes the audio data processing method of the first aspect and any one of the first aspect.

[0019] The second aspect and any one of the implementation manners of the second aspect respectively correspond to the first aspect and any one of the implementation manners of the first aspect. The technical effects corresponding to the second aspect and any one of the implementation manners of the second aspect can refer to the technical effects corresponding to the first aspect and any one of the implementation manners of the first aspect above, which will not be elaborated here.

[0020] In a third aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium includes a computer program, and when the computer program runs on an electronic device, the electronic device executes the audio data processing method of the first aspect and any one of the first aspect.

[0021] The third aspect and any implementation manner of the third aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the third aspect and any implementation manner of the third aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated herein.

[0022] Fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when run, causes a computer to execute the audio data processing method as described in the first aspect or any item in the first aspect.

[0023] The fourth aspect and any implementation manner of the fourth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated herein.

[0024] Fifth aspect, the present application provides a chip, which includes a processing circuit and transceiver pins. Among them, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the audio data processing method as described in the first aspect or any item in the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal.

[0025] The fifth aspect and any implementation manner of the fifth aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the fifth aspect and any implementation manner of the fifth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect above, which will not be elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figures 1a - 1d It is a schematic diagram of an exemplary application scenario;

[0027] Figure 1e It is a flowchart of an exemplary current solution;

[0028] Figure 1f It is a schematic diagram of audio data obtained by a recording application under an exemplary current solution;

[0029] Figure 1g It is a schematic diagram of an exemplary current solution;

[0030] Figure 2 It is a schematic diagram of the hardware structure of an exemplary electronic device;

[0031] Figure 3 It is a schematic diagram of the software structure of an exemplary electronic device;

[0032] Figure 4 Schematic diagram of module interaction shown by way of example;

[0033] Figure 5 Flowchart of the recording application from starting to ending recording shown by way of example;

[0034] Figures 6a - 6b Flowchart of the concurrent running scenario of the recording application and the video recording application shown by way of example;

[0035] Figure 7a Schematic diagram of this solution shown by way of example;

[0036] Figure 7b Schematic diagram of the audio data obtained by the recording application under this solution shown by way of example;

[0037] Figure 8 Flowchart of the audio data processing method shown by way of example. Detailed implementation manners

[0038] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0039] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0040] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe a specific order of the target objects.

[0041] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0042] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0043] Figures 1a - 1d FIG. is a schematic diagram of an application scenario provided for the embodiments of the present application. Before introducing the embodiments of the present application, first based on Figures 1a - 1d the application scenario of the embodiments of the present application will be described. Refer to Figure 1a in (1), taking a mobile phone as an example of an electronic device, the interface is the mobile phone desktop. Figure 1a (1) in FIG. shows the main display interface 101 of the mobile phone. The main display interface 101 includes a plurality of application icons. For example, a clock application icon, a calendar application icon, a gallery application icon, a memo application icon, a file management icon, an email application icon, a music application icon, a calculator application icon, a video application icon, a sports health application icon, a weather application icon, a browser application icon, a settings icon, a recorder application icon 1011, a video recorder application icon 1012, a camera icon, a contacts icon, a call application icon, a messages application icon, etc.

[0044] The user can click on the application icon to start the application they want to use. Refer to Figure 1a in (1), when the user clicks on the recorder application icon 1011, the mobile phone responds to this operation of the user to start the recording application and enters the recording interface 102 of the recording application. Refer to Figure 1a in (2), after the mobile phone starts the recording application, it immediately starts the audio acquisition task of recording. On this interface, the user can click on the return control 1021 to return to the main display interface 101 of the mobile phone; the user can also click on the mark control 1022 in the lower left corner of the interface to mark the recording at a certain moment; the user can also click on the end control 1023 in the middle of the interface to end the audio acquisition task of recording; the user can also click on the pause control 1024 in the lower right corner of the interface to pause the audio acquisition task of recording. The user can return to the main display interface 101 of the mobile phone by swiping their fingertip upward from the bottom of the interface to switch the recording application to run in the background. Refer to Figure 1b in (1), the user can continue to select the application they want to use on the main display interface 101 of the mobile phone. At this time, the recording application is still running in the background of the mobile phone.

[0045] Refer to Figure 1b in (1), the recording application is still recording in the background. When the user clicks on the video recorder application icon 1012 on the main display interface 101 again, the mobile phone responds to this operation of the user to start the video recorder application, and the mobile phone enters the video recorder application interface. As Figure 1bAs shown in (2), at this time, the picture displayed in the video recording interface 103 is captured by the rear camera of the mobile phone, and the mobile phone also collects the surrounding sounds through the relevant microphones configured by itself. In the video recording interface 103, the user can click the flip camera control 1031 to make the video recording interface 103 display the picture captured by the front camera of the mobile phone; the user can also click the end control 1032 in the video recording interface 103 to end the video recording task of the video recorder application.

[0046] See Figure 1c As shown in (1), after the user clicks the end control 1032, a video recording saving window 1033 pops up in the interface 103. The user can click the saving date control 1034 to modify the saving time; the user can also click the deletion control 1035 to delete the video recording data; the user can also click the confirmation control 1036 to save the video recording data. If the user wants to return to the recording interface 102 of the recording application, they need to enter the main display interface 101 or the background display interface 104 of the mobile phone. See Figure 1c As shown in (2), in the background display interface 104 of the mobile phone, the user can browse the application programs mounted in the background of the mobile phone by swiping left and right; the user can also enter the display interface of the application program by clicking the background interface of the application program to use the functions of the application program; the user can also end the mobile phone from mounting the application program by swiping up and down the background interface of the application program. Continue to see Figure 1c In the background display interface 104 shown in (2), the floating window 1041 is the background interface of other background applications, the floating window 1042 is the recording background interface of the recorder, and the floating window 1043 is the background interface of the video recorder application. The user can click the clear background control 1044 to clear all the application programs mounted in the background of the mobile phone. The user can enter the recording interface 102 of the recording program in the background display interface 104 by clicking the floating window 1042 to switch the recording application from the background of the mobile phone back to the foreground of the mobile phone for use.

[0047] See Figure 1d As shown in (1), after the user enters the recording interface 102 by clicking the floating window 1042, if they want to end the recording task, they need to click the end control 1023 in the recording interface 102. When the user clicks the end control 1023, a pop-up window 1025 will appear in the recording interface 102. See Figure 1d As shown in (2), the user can modify the saving time of this section of recording data by clicking on the time in the saving date control 1026; the user can also delete this section of recording data by clicking the deletion control 1027; the user can also click the confirmation control 1028 to confirm the saving of this section of recording data.

[0048] When the user uses a mobile phone with the Audiozoom noise reduction function, currently, only when the application starts, the audio service in the application framework layer sends the relevant parameters of the application to the algorithm module, and the algorithm module processes the audio data collected during the operation of the application according to these parameters. Figures 1a to 1d In the application scenario shown, if the user keeps the recording application recording in the background of the mobile phone, uses the mobile phone to record video for a period of time and then ends the video recording, and then switches the recording application back to the foreground, at this time, the audio service will no longer re-send the relevant parameters of the recording application to the algorithm module, and the algorithm module can only continue to use the relevant parameters of the video recording application to process the audio data collected after the recording application is switched back to the foreground. However, the relevant parameters of the video recording application may involve an audio noise reduction algorithm. For example, when the video recording application focuses on obtaining the sound emitted by a distant sound source, the Audiozoom noise reduction algorithm will treat the sound emitted by a nearby sound source as noise and filter it out to highlight the sound emitted by the distant sound source. And recording is often used to collect nearby human voices. In this way, this audio noise reduction algorithm may misidentify the audio data collected after the recording application is switched back to the foreground as noise and filter it out, resulting in no sound in the audio data collected after the recording application is switched back to the foreground.

[0049] Specifically, see Figure 1e In the existing solution, for the process from the recording application starting to record, to the recording application switching to the background and the video recording application running in the foreground for video recording, and then to the video recording application ending the video recording and the recording application being switched to the foreground for operation, the following steps are included:

[0050] S101, In response to the user's start recording operation, the recording application sends a start recording request to the audio service.

[0051] S102, The audio service sends the audio type 1 of the recording application to the audio HAL.

[0052] S103, The audio HAL opens the recording mic path through the audio driver.

[0053] S104, The audio HAL sends the audio type 1 to the algorithm module.

[0054] S105, The algorithm module adds the parameter 1 associated with the audio type 1 to the global variable.

[0055] S106, The algorithm module receives the audio data collected by the audio driver through the recording mic path.

[0056] S107, The algorithm module processes the audio data collected from the recording mic path according to the parameter 1 in the global variable.

[0057] S108, In response to the user's switch to the background operation, the recording application sends a message to the audio service to switch to running in the background.

[0058] S109, The algorithm module receives the audio data collected by the audio driver through the recording mic path.

[0059] S110, The algorithm module processes the audio data collected by the recording mic path according to Parameter 1 in the global variable.

[0060] S111, In response to the user's start video recording operation, the video recording application sends a start video recording request to the audio service.

[0061] S112, The audio service sends the audio type 2 of the video recording application to the audio HAL.

[0062] S113, The audio HAL opens the video recording mic path through the audio driver.

[0063] Exemplarily, since the recording mic path is in an open state when the recording application runs in the background. The recording mic path includes a top microphone and a bottom microphone, and the video recording mic path includes a top microphone, a bottom microphone, and a back microphone. Therefore, here, only based on the open state of the recording mic path, and then opening the back microphone can open the video recording mic path.

[0064] S114, The audio HAL sends the audio type 2 to the algorithm module.

[0065] S115, The algorithm module updates the global variable with Parameter 2 associated with the audio type 2.

[0066] Here, the result of the update is that Parameter 2 overwrites Parameter 1, resulting in the erasure of Parameter 1, and only Parameter 2 remains in the global variable.

[0067] S116, The algorithm module receives the audio data collected by the audio driver through the video recording mic path.

[0068] S117, The algorithm module processes the audio data collected by the video recording mic path according to Parameter 2 in the global variable.

[0069] It should be noted that during the process of collecting audio by running two applications in the foreground and background, the algorithm module only processes the audio data collected by the mic path corresponding to the foreground running application, and stores the processed audio data in the file specified by the foreground running application. Since the algorithm module does not store the audio data required by the background application in the file specified by the background application, the background application cannot obtain the corresponding audio data during this period. Therefore, during the process of recording in the background running recording application and recording video in the foreground running video recording application, there will be a problem of silent recording in the recording application.

[0070] S118, In response to the user's end video recording operation, the video recording application sends an end video recording request to the audio service.

[0071] S119, the audio service sends an end video recording identifier to the audio HAL.

[0072] S120, the audio HAL closes the video recording mic path through the audio driver.

[0073] Exemplarily, when closing the video recording mic path, the top microphone, the bottom microphone, and the back microphone will all be closed. Therefore, closing the video recording mic path at this time will also cause the recording mic path to be closed together.

[0074] S121, in response to the user's operation of switching to the foreground, the recording application sends a message of switching to the foreground running to the audio service.

[0075] S122, the audio service sends a continue recording identifier to the audio HAL.

[0076] S123, the audio HAL opens the recording mic path through the audio driver.

[0077] S124, the algorithm module receives the audio data collected by the audio driver through the recording mic path.

[0078] S125, the algorithm module processes the audio data collected by the recording mic path according to Parameter 2 in the global variable.

[0079] It should be noted that for Figure 1e the solution shown only represents the solution used by the applicant of this application and does not represent the solution publicly disclosed in the prior art.

[0080] As Figure 1e shown, after the recording application switches to the foreground running, since the audio service will no longer re-issue Audio Type 1 of the recording application, the algorithm module will process the audio data collected by the recording mic (Microphone) path according to Parameter 2 in the global variable. However, Parameter 2 corresponding to the audio data collected by the video recording application involves a noise reduction function. For example, when the video recording application focuses on collecting the sound emitted by a distant sound source, the Audiozoom noise reduction algorithm will treat the sound emitted by a nearby sound source as noise and filter it out. After switching the recording application back to the foreground running, if the recording application collects nearby human voices, they will also be treated as noise by the Audiozoom noise reduction algorithm, resulting in no sound in the audio data collected after the recording application switches back to the foreground. As Figure 1f shown, when the recording application plays the audio data it obtains, it can be known from the content format of the audio bar that in the current solution, there is only sound in the stage when the recording application records in the foreground and the stage when the recording application switches to the background. In the stage when the recording application remains in the background and the video recording application records in the foreground, as well as in the stage from the end of video recording to the end of recording after switching the recording application back to the foreground, there is no sound.

[0081] Briefly speaking, as Figure 1g shown, in the scenario of concurrent application, when each App is opened, the parameters sent by the App are loaded into the global variables, thus overwriting the parameters of the previous App. When switching the App to the foreground for running, due to the device level reason, the parameters of the switched App will not be sent again, resulting in the switched App still using the parameters finally loaded into the global variables. Therefore, there will be no sound in the audio data collected after the recording application is switched back to the foreground, and the user experience is poor.

[0082] Based on this, the present application provides an audio data processing method, an electronic device and a storage medium. In the scenario where the audio collection tasks of multiple applications run concurrently, global variables corresponding to the application identifiers of each application can be created, and the audio parameters associated with the audio types thereof are added to the corresponding global variables, avoiding the problem that the parameters are overwritten due to multiple applications sharing a global variable. When switching a certain application from the background to the foreground for running, the corresponding global variable can be called according to the application identifier of the switched application. Furthermore, according to the parameters in the global variable, the audio data transmitted by the audio driver is processed, so as to realize processing the collected audio data according to the audio parameters of the switched application, ensuring the processing effect of the audio data and improving the user experience.

[0083] Figure 2 shows a schematic structural diagram of the electronic device 100. It should be understood that Figure 2 the shown electronic device 100 is only an example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 2 The various components shown in can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0084] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0085] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0086] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0087] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0088] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0089] The USB interface 130 is an interface compliant with the USB standard specification, which may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transmission between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0090] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative descriptions and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0091] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger may be a wireless charger or a wired charger. As Figure 2 shown, in some embodiments of wired charging, the charging management module 140 may receive the charging input from the wired charger 201 through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 may receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0092] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0093] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0094] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: the antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0095] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through the antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0096] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0097] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive the signals to be transmitted from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 and radiate them out.

[0098] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0099] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0100] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0101] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0102] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0103] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0104] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency, the digital signal processor is used to perform Fourier transform on the frequency energy, etc.

[0105] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0106] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0107] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0108] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0109] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, etc. For example, music playback, recording, etc. Among them, the microphone 170C includes but is not limited to the top microphone and bottom microphone on the front of the electronic device, and the back microphone on the back of the electronic device. Different microphones can be combined with each other to form different mic paths. For example, the top microphone and the bottom microphone are combined to form a recording mic path, and the top microphone, bottom microphone, and back microphone are combined to form a video recording mic path.

[0110] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0111] The pressure sensor is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be disposed on the display screen 194. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view short messages is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0112] The touch sensor, also known as the "touch panel". The touch sensor can be disposed on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also known as the "touch screen". The touch sensor is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194.

[0113] The keys 190 include a power-on key, volume keys, etc. The keys 190 can be mechanical keys. They can also be touch keys. The electronic device 100 can receive key inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0114] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. For touch operations on different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, receiving messages, alarms, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0115] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0116] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 is exemplarily described.

[0117] Figure 3 It is the software structure block diagram of the electronic device 100 in the embodiments of this application.

[0118] The layered architecture of the electronic device 100 divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel layer. It can be understood that Figure 3 the layers in the software structure and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which are not limited in this application.

[0119] The application layer may include a series of application packages. As Figure 3 shown, the application packages may include applications such as recording, video recording, memo, camera, gallery, WLAN, and Bluetooth. The application packages may also include applications such as call, calendar, map, navigation, music, video, and short message.

[0120] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer.

[0121] As Figure 3As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, an audio service, a notification manager, etc.

[0122] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0123] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0124] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0125] The audio service can provide audio-related services to users, such as triggering audio acquisition and processing operations under the invocation of an application. For example, after the audio service receives a start recording request sent by a recording application, the audio service sends the audio type and application identifier of the recording application to the audio HAL in the HAL layer; after the audio service receives an end recording request sent by the recording application, the audio service sends an end recording identifier to the audio HAL in the HAL layer. Among them, the application identifier may be the serial number of the application. After the mobile phone downloads and installs the application, it can automatically assign a serial number to the installed application, that is, the application identifier. For example, using the key value to represent the application identifier, after the mobile phone downloads and installs the recording application and the video recording application, the key value assigned to the recording application is 1, and the key value assigned to the video recording application is 2. The audio type is the usage scenario or mode of the application. Each application may have one or more audio types. For example, the recording application's conference recording, interview recording, voice mode, 3D recording, or music mode, etc., and the video recording application's front video recording, rear video recording, or dual-view video recording, etc. The end recording identifier is used to instruct the audio HAL to close the recording mic path and instruct the algorithm module to release the memory of the global variable corresponding to the application identifier of the recording application.

[0126] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0127] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including connection, hangup, etc.).

[0128] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background running application, or a notification that appears on the screen in the form of a dialogue window. For example, it can prompt text information in the status bar, emit a prompt tone, vibrate the electronic device, blink the indicator light, etc.

[0129] HAL is an interface layer located between the operating system kernel and the hardware circuit. HAL includes but is not limited to: audio HAL and algorithm module. Among them, the audio HAL is a part of the operating system of the electronic device responsible for the audio hardware abstraction layer, which provides an interface between the application and the audio hardware. The application can access and control the functions of the audio hardware by using the audio HAL, such as audio input, output, encoding, decoding, and mixing, etc. In this application, the audio HAL can also be used to receive the audio type and application identifier of a certain application sent by the audio service in the application framework layer, and open the corresponding mic path through the audio driver according to the audio type or application identifier; the audio HAL is also used to start the algorithm module and send the audio type and application identifier of the application to the algorithm module. The algorithm module is used to determine whether there is a global variable corresponding to the application identifier. If it exists, the global variable is directly called; if not, a global variable corresponding to the application identifier is created, and the audio parameters associated with the audio type are added to the global variable, so that after receiving the audio data sent by the audio driver, the audio data is processed according to the parameters in the global variable. Among them, the audio parameters can be related parameters of algorithms such as automatic gain control (AGC) algorithm, active noise control (ANC) algorithm, acoustic echo cancellation (AEC) algorithm, acoustic feed back cancellation (AFC) algorithm, dereverberation algorithm, etc. The audio parameters or the values of the audio parameters associated with each audio type are different.

[0130] The algorithm module is also configured with a recording algorithm, which can use the recording algorithm to perform gain and noise reduction on the audio data. The recording algorithm refers to an algorithm that processes audio data to achieve effects such as gain, noise reduction, and echo cancellation of audio data, also known as an audio algorithm. The recording algorithm can process a certain aspect of the audio data stream, such as gain control algorithm, noise reduction algorithm, echo cancellation algorithm, acoustic feedback cancellation algorithm, dereverberation algorithm, etc.

[0131] It should be noted that Figure 3 Describing the audio HAL and the algorithm module as two independent modules, it can be understood that the algorithm module is independent of the audio HAL. The audio HAL calls the algorithm module to process audio data and then transmits the processed audio data to the application layer. In some embodiments, the algorithm module can also be regarded as a sub-module of the audio HAL. The audio HAL calls this sub-module to obtain the processed audio data and then transmits the processed audio data to the application layer. The embodiments of the present application do not make any limitations in this regard.

[0132] The kernel layer is a layer between the hardware and the above software layer. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver. Among them, the hardware may include devices such as a camera, a display screen, a microphone, a processor, and a memory. The audio driver is used to drive the microphone to work to collect the sounds emitted by each sound source in the surrounding environment. The sound source can be a person, an animal, or various electronic devices with a sound output function, etc.

[0133] It can be understood that Figure 3 The layers in the shown software structure and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components. The present application does not make any limitations.

[0134] It can be understood that in order for the electronic device to implement the audio data processing method in the embodiments of the present application, it includes the corresponding hardware and / or software modules for executing each function. Combining the algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described function for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0135] In one example, take the recording application starting to record as an example. As Figure 4As shown, in response to the user's operation of starting to record, the recording application sends a start recording request to the audio service. In response to the start recording request, the audio service sends the audio type and application identifier of the recording application to the audio HAL. After receiving the audio type and application identifier of the recording application, the audio HAL opens the recording mic path according to the audio type of the recording application, and at the same time starts the algorithm module, and sends the audio type and application identifier of the recording application to the algorithm module. The algorithm module determines whether there is a global variable corresponding to the application identifier. If it exists, it directly calls the global variable. If it does not exist, it creates a global variable corresponding to the application identifier, and adds the audio parameters associated with the audio type to the created global variable. After the algorithm module receives the audio data collected by the audio driver through the recording mic path, it processes the audio data according to the audio parameters in the global variable. The algorithm module can save the processed audio data in the file specified by the recording application for the recording application to play; or, when the recording application plays the collected audio data, the algorithm module sends the processed audio data to the audio service, and the audio service transmits it to the recording application. Among them, the algorithm module can save the processed audio data to the specified file while processing the audio data, or after all the audio data is processed, save all the processed audio data to the specified file.

[0136] Based on the above embodiments, the audio data processing method of the above embodiments will be described below through two specific application scenarios.

[0137] In one application scenario, still taking the recording application starting to record to ending recording as an example, referring to Figure 5 , the process of the recording application starting to record to ending recording includes:

[0138] S501, in response to the user's operation of starting to record, the recording application sends a start recording request to the audio service.

[0139] Exemplarily, in response to the user's operation of starting to record, for example Figure 1a in (1) click on the recorder application icon 1011 to enter the recording interface 102 of the recording application and start the recording operation, the recording application sends a start recording request to the audio service.

[0140] S502, the audio service sends the first audio type and the first application identifier of the recording application to the audio HAL.

[0141] Exemplarily, after receiving a start recording request, the audio service sends the first audio type and the first application identifier of the recording application to the audio HAL. For example, the audio service sends the first start input session start inputsession1 to the audio HAL, and the start input session includes the first audio type algo_set_parameter1 and the first application identifier algo_scene_id = 1. Among them, the first audio type can be one of conference recording, interview recording, voice mode, 3D recording, or music mode, and the first application identifier is the serial number of the recording application. For example, when using a key value to represent the application identifier, the first application identifier is the key value equal to 1.

[0142] S503, the audio HAL opens the recording mic path through the audio driver according to the first audio type.

[0143] Exemplarily, the recording mic path is used to collect the audio data required by the recording application. The recording mic path may include the top microphone and the bottom microphone on the front of the mobile phone. The audio HAL can open the top microphone and the bottom microphone on the front of the mobile phone through the recording mic path to open the recording mic path.

[0144] S504, the audio HAL sends the first audio type and the first application identifier to the algorithm module.

[0145] S505, the algorithm module creates a first global variable corresponding to the first application identifier and adds the audio parameters associated with the first audio type to the first global variable.

[0146] Exemplarily, after receiving the first application identifier, the algorithm module first determines whether there is a first global variable corresponding to the first application identifier. If it exists, the first global variable can be directly called; if it does not exist, a first global variable corresponding to the first application identifier needs to be created. The algorithm module is configured with the association relationship between the audio type and the audio parameters. According to the association relationship between the audio type and the audio parameters, the audio parameters associated with the first audio type can be determined and added to the created first global variable. The association relationship between the audio type and the audio parameters, for example, the conference recording, interview recording, voice mode, 3D recording, and music mode of the recording application, are all associated with the parameters of the equalizer, including: frequency band, gain, quality factor Q value, and filter type, etc. Among them, the frequency band refers to the range of frequencies, such as low, low-middle, high-middle, and high frequencies, etc.; the gain refers to the degree of amplification or attenuation of the signal; the quality factor Q value is used to describe the bandwidth of the frequency band; the filter type usually has different options such as high-pass filter and low-pass filter. The values of the audio parameters associated with different audio types of the same application are different, and one audio type can be associated with the audio parameters of one or more recording algorithms.

[0147] S506, the algorithm module obtains the audio data collected through the recording mic path sent by the audio driver.

[0148] Exemplarily, the audio driver can collect the audio data of the surrounding environment through the recording mic path. For example, the audio driver respectively obtains the first input audio data of the top microphone and the bottom microphone on the front of the mobile phone, and sends the first input audio data as the audio data collected through the recording mic path to the algorithm module.

[0149] S507, the algorithm module processes the audio data collected through the recording mic path according to the parameters in the first global variable.

[0150] Exemplarily, the algorithm module can configure the recording algorithm used according to the parameters in the first global variable, and use the configured recording algorithm to process the audio data collected through the recording mic path. For example, according to the proportional coefficient parameter, integration time parameter, and differential time parameter in the first global variable, configure the gain control algorithm, and use the configured gain control algorithm to process the audio data collected through the recording mic path. Another example is to configure the reverberation removal algorithm according to the Weighted Prediction Error, multichannel linear prediction (MCLP), and inverse filtering in the first global variable, and use the configured reverberation removal algorithm to process the audio data collected through the recording mic path.

[0151] Here, the algorithm module can save the processed audio data to the file specified by the recording application while processing the audio data, or after processing all the audio data from the start of recording to the end of recording, save all the processed audio data to the file specified by the recording application at once.

[0152] S508, the recording application sends an end recording request to the audio service.

[0153] Among them, the end recording request is used to notify the audio service of the message that the recording application ends the recording.

[0154] Exemplarily, in response to the user's end recording operation, for example Figure 1d in (1) the operation of clicking the end control 1023, the recording application sends an end recording request to the audio service.

[0155] S509, the audio service sends an end recording flag to the audio HAL.

[0156] Exemplarily, after receiving an end recording request, the audio service sends an end recording identifier to the audio HAL. The end recording identifier is used to instruct the audio HAL to close the recording mic path and to instruct the algorithm module to release the memory of the first global variable.

[0157] S510, the audio HAL closes the recording mic path through the audio driver according to the end recording identifier.

[0158] Exemplarily, after receiving the end recording identifier, the audio HAL closes the bottom microphone and the top microphone on the front of the mobile phone through the audio driver to close the recording mic path.

[0159] S511, the audio HAL sends the end recording identifier to the algorithm module.

[0160] S512, the algorithm module deletes the first global variable according to the end recording identifier.

[0161] Exemplarily, after the recording application ends recording, the algorithm module can release the memory of the first global variable by deleting the first global variable, reduce the memory occupancy rate of the mobile phone, and thereby improve the running speed of the mobile phone.

[0162] It should be noted that in some other application scenarios, such as the scenario where the user uses certain applications for video recording or live streaming, the process is similar to the process of the above recording scenario, only the applications, application identifiers, and audio types involved are different, and will not be described in detail here one by one.

[0163] In this embodiment, the algorithm module can create a global variable corresponding to the application identifier for each application, and add the audio parameters associated with its audio type to the global variable, avoiding the problem that the parameters are overwritten due to multiple applications sharing a global variable.

[0164] The above example shows the process of an audio acquisition task of an application from start to end. Next, the scenario where the audio acquisition tasks of two applications run concurrently to the end will be introduced. As for the concurrent scenarios of three applications or more than three applications, they are similar to the concurrent scenario of two applications, and will not be introduced in detail in this application one by one.

[0165] In another application scenario, in the scenario where the audio acquisition tasks of the recording application and the video recording application run concurrently to the end, as Figure 6a shown, the process of the recording application starting to record, until the recording application switches to the background and the video recording application runs in the foreground to record, includes:

[0166] S601, the recording application sends a start recording request to the audio service.

[0167] S602, The audio service sends the first audio type and the first application identifier of the recording application to the audio HAL.

[0168] S603, The audio HAL opens the recording mic path through the audio driver according to the first audio type.

[0169] S604, The audio HAL sends the first audio type and the first application identifier to the algorithm module.

[0170] S605, The algorithm module creates a first global variable corresponding to the first application identifier and adds the audio parameters associated with the first audio type to the first global variable.

[0171] S606, The algorithm module obtains the audio data collected through the recording mic path sent by the audio driver.

[0172] S607, The algorithm module processes the audio data collected through the recording mic path according to the parameters in the first global variable.

[0173] Among them, for the implementation methods of S601 to S607, reference can be made to the detailed descriptions of S501 to S507 above.

[0174] S608, The recording application sends a message to the audio service to switch to running in the background.

[0175] Exemplarily, in response to the user's operation of switching to the background, the recording application sends a message to the audio service to switch to running in the background to notify the audio service of the message to switch the recording application to running in the background.

[0176] S609, The algorithm module obtains the audio data collected through the recording mic path sent by the audio driver.

[0177] S610, The algorithm module processes the audio data collected through the recording mic path according to the parameters in the first global variable.

[0178] It should be noted that during the process that the recording application keeps recording in the background and no other applications with audio collection functions are running in the foreground, the algorithm module will continuously obtain the audio data collected through the recording mic path and process the audio data collected through the recording mic path according to the parameters in the first global variable. Among them, applications with audio collection functions include recording applications, video recording applications, live broadcast applications, etc.

[0179] S611, The video recording application sends a start video recording request to the audio service.

[0180] Exemplarily, in response to the user's operation of starting video recording, the video recording application sends a start video recording request to the audio service.

[0181] It should be noted that during the process of the recording application running in the background for recording and the video application running in the foreground for video recording, the algorithm module only processes the audio data collected by the mic path corresponding to the video application during this period, resulting in the recording application being unable to obtain the corresponding audio data, that is, the recording application will record silently during this period.

[0182] S612, the audio service sends the second audio type and the second application identifier of the video application to the audio HAL.

[0183] Exemplarily, after receiving the start video recording request, the audio service sends the second audio type and the second application identifier of the video application to the audio HAL. The audio service sends a second start input session start inputsession2 to the audio HAL, and the start input session includes a second audio type algo_set_parameter2 and a second application identifier algo_scene_id = 2. The second audio type can be one of front video recording, rear video recording, or dual-view video recording, and the second application identifier is the serial number of the video application. For example, when using a key value to represent the application identifier, the second application identifier is the key value equal to 2.

[0184] S613, the audio HAL opens the video recording mic path through the audio driver according to the second audio type.

[0185] Exemplarily, the video recording mic path is used to collect the audio data required by the video application. The video recording mic path includes the top microphone and the bottom microphone on the front of the mobile phone, and the rear microphone on the back of the mobile phone. The audio HAL can open the top microphone and the bottom microphone on the front of the mobile phone, and the rear microphone on the back of the mobile phone through the audio driver to open the video recording mic path. At this time, since the recording application is running in the background, the top microphone and the bottom microphone corresponding to the recording mic path are in the open state. When opening the video recording mic path through the audio driver, only the rear microphone on the back of the mobile phone needs to be opened on the basis of the recording mic path being open. It can be understood that the recording mic path and the rear microphone can be combined into the video recording mic path.

[0186] S614, the audio HAL sends the second audio type and the second application identifier to the algorithm module.

[0187] S615, the algorithm module creates a second global variable corresponding to the second application identifier and adds the audio parameters associated with the second audio type to the second global variable.

[0188] Exemplarily, after receiving the second application identifier, the algorithm module first needs to determine whether there is a second global variable corresponding to the second application identifier. If it exists, the second global variable can be directly called; if not, a second global variable corresponding to the second application identifier needs to be created. The algorithm module is configured with the association relationship between the audio type and the audio parameters, and can determine the audio parameters associated with the second audio type according to the association relationship between the audio type and the audio parameters, and add the audio parameters associated with the second audio type to the created second global variable. For example, the front video recording, rear video recording, or dual-view video recording of the video recording application is associated with the parameters of the noise reduction algorithm, including: filter type, filter order, filter coefficient, sampling frequency, and frame length, etc.

[0189] S616. The algorithm module obtains the audio data collected through the video recording mic path sent by the audio driver.

[0190] Exemplarily, the audio driver can collect the audio data of the surrounding environment through the video recording mic path. For example, the audio driver respectively obtains the second input audio data of the top microphone and the bottom microphone on the front of the mobile phone, and the back microphone on the back of the mobile phone, and sends the second input audio data as the audio data collected through the video recording mic path to the algorithm module.

[0191] S617. The algorithm module processes the audio data collected through the video recording mic path according to the parameters in the second global variable.

[0192] Exemplarily, the algorithm module can configure the recording algorithm used according to the parameters in the second global variable, and use the configured recording algorithm to process the audio data collected through the video recording mic path. For example, according to the parameters such as filter type, filter order, filter coefficient, sampling frequency, and frame length in the second global variable, configure the noise reduction algorithm, and use the configured noise reduction algorithm to process the audio data collected through the video recording mic path.

[0193] Continue to refer to Figure 6b the process shown. The process of the video recording application ending the video recording and then switching the recording application to the foreground until the recording ends includes:

[0194] S618. The video recording application sends an end video recording request to the audio service.

[0195] Exemplarily, in response to the user's operation of ending video recording or in response to the user switching the video recording application to run in the background, the video recording application sends an end video recording request to the audio service. The end video recording request is used to notify the audio service of the message that the video recording application ends video recording. Here, during the process of the video recording application recording video, if the video recording application is switched to run in the background, the mobile phone will automatically determine to end video recording. Therefore, if it is detected that the user switches the video recording application to the background, the video recording application sends an end video recording request to the audio service.

[0196] S619, the audio service sends an end video recording flag to the audio HAL.

[0197] Exemplarily, after receiving the end video recording request, the audio service sends an end video recording flag to the audio HAL. The end video recording flag is used to instruct the audio HAL to close the video recording mic path and to instruct the algorithm module to release the memory of the second global variable.

[0198] S620, the audio HAL closes the video recording mic path through the audio driver according to the end video recording flag.

[0199] Exemplarily, after receiving the end video recording flag, the audio HAL closes the bottom microphone and the top microphone on the front of the mobile phone, and the back microphone on the back of the mobile phone through the audio driver to close the recording mic path. It should be noted that the recording mic path includes the bottom microphone and the top microphone on the front of the mobile phone. At this time, when closing the video recording mic path, the recording mic path will also be closed together.

[0200] S621, the audio HAL sends an end video recording flag to the algorithm module.

[0201] S622, the algorithm module deletes the second global variable according to the end video recording flag.

[0202] Exemplarily, the algorithm module can release the memory of the second global variable by deleting the second global variable, increase the available space memory of the mobile phone, and further improve the running fluency of the mobile phone.

[0203] S623, the recording application sends a message of switching to the foreground to the audio service.

[0204] Exemplarily, in response to the user's operation of switching to the foreground, the recording application sends a message of switching to the foreground to the audio service to notify the audio service of the message that the recording application switches from the background to the foreground.

[0205] S624, the audio service sends a first application identifier to the audio HAL.

[0206] Exemplarily, after receiving the message to switch to the foreground operation sent by the recording application, the audio service sends the first application identifier of the recording application to the audio HAL.

[0207] S625, according to the first application identifier, the audio HAL opens the recording mic path through the audio driver.

[0208] Exemplarily, since the recording mic path is also closed when the video recording mic path is closed, the audio HAL needs to reopen the recording mic path through the audio driver according to the first application identifier. The audio HAL can search for the path opening record related to the recording application according to the first application identifier, determine the recording mic path last used by the recording application from the path opening record, and then the audio HAL opens the recording mic path through the audio driver.

[0209] S626, the audio HAL sends the first application identifier to the algorithm module.

[0210] S627, the audio driver sends the audio data collected through the recording mic path to the algorithm module.

[0211] S628, the algorithm module calls the first global variable corresponding to the first application identifier; processes the audio data collected through the recording mic path according to the parameters in the first global variable.

[0212] Here, since the first global variable corresponding to the first application identifier has been created when the first application starts recording for the first time, when the first application switches back to the foreground operation, there is no need to recreate it again, and the first global variable can be directly called according to the identifier of the first application.

[0213] S629, the recording application sends an end recording request to the audio service.

[0214] Exemplarily, in response to the user's end recording operation, the recording application sends an end recording request to the audio service.

[0215] S630, the audio service sends an end recording identifier to the audio HAL.

[0216] S631, according to the end recording identifier, the audio HAL closes the recording mic path through the audio driver.

[0217] S632, the audio HAL sends the end recording identifier to the algorithm module.

[0218] S633, the algorithm module deletes the first global variable according to the end recording identifier.

[0219] Among them, for S629 to S633, refer to the detailed description of S508 to S512 above.

[0220] Briefly speaking, as Figure 7a shown, in the solution of this application, when each App (Application) is opened, the audio service transmits the audio type and application identifier of the App to the algorithm module through the audio HAL. The algorithm module determines whether there is a global variable corresponding to the application identifier. If it exists, the global variable can be directly called and used; if not, a global variable corresponding to the application identifier needs to be created, and the audio parameters associated with the audio type are added to the created global variable. In the concurrent scenario of multiple applications, such as Figure 7a shown in the scenario where four applications, App1 to App4, run concurrently, when a background App is switched to the foreground, the audio service transmits the application identifier of the switched App to the algorithm module through the audio HAL. The algorithm module can directly call the global variable corresponding to the application identifier according to the application identifier of the switched App, and then, according to the parameters in the global variable, process the audio data transmitted by the audio driver. This avoids the problem that the parameters in the global variable are overwritten in the concurrent scenario of multiple applications, and can implement processing the collected audio data according to the audio parameters of the switched application, ensuring the processing effect of the audio data and improving the user experience.

[0221] Figures 1a to 1d In the application scenario shown, when the recording application plays the obtained audio data using the solution of this application, the content format of the audio bar is as Figure 7b shown. Compared with the audio bar (such as Figure 1f ) when playing audio data in the current solution, this application can ensure that after the video recording ends, when the recording application is switched back to the foreground and runs until the recording ends, there is sound.

[0222] The following specifically describes the process of the electronic device executing the audio data processing method in conjunction with the accompanying drawings. Figure 8 For the flowchart of an exemplary audio data processing method, in this example, the electronic device is described by taking a mobile phone as an example.

[0223] S801, the audio HAL obtains the first audio type and the first application identifier of the first application.

[0224] Among them, the first application is an application with audio collection function, such as an application with recording function, video recording function or live broadcast function. When the first application is a recording application, the first audio type is one of conference recording, interview recording, voice mode, 3D recording and music mode; when the first application is a video recording application, the first audio type is one of front video recording, rear video recording or dual-view video recording. Among them, some instructions or parameters can be used to identify the audio type. For example, IMEDIA_RECORD_3D_REC is identified as 3D recording, IMEDIA_RECORD_VR_FRONT is identified as front video recording, IMEDIA_RECORD_DOUBLE_VR is identified as dual-view video recording, IMEDIA_RECORD_VR_BACK is identified as rear video recording, IMEDIA_RECORD_LIVE_TELECAST is identified as live broadcast, IMEDIA_RECORD_MULMIC_EAR is identified as multi-microphone headset mode, IMEDIA_RECORD_KARAOKE is identified as Karaoke, etc.

[0225] Exemplarily, in response to the user's operation of starting the audio collection task of the first application, the first application sends a first audio collection request to the audio service. In response to the first audio collection request, the audio service sends the first audio type and the first application identifier of the first application to the audio HAL, so that the audio HAL can obtain the first audio type and the first application identifier. Among them, the audio collection task can be recording, video recording or live broadcast.

[0226] S802, the audio HAL opens the first mic path corresponding to the first audio type, the algorithm module creates the first global variable corresponding to the first application identifier, and adds the audio parameters associated with the first audio type to the first global variable.

[0227] For example, when the first application is a recording application, the first mic path corresponding to the first audio type is the recording mic path; when the first application is a video recording application, the first mic path corresponding to the first audio type is the video recording mic path.

[0228] In an implementable manner, after the audio HAL opens the first mic path according to the first audio type, it sends the first audio type and the first application identifier to the algorithm module. The algorithm module determines whether there is a global variable corresponding to the first application identifier. If it exists, it directly calls the global variable corresponding to the first application identifier; if it does not exist, it creates the first global variable corresponding to the first application identifier and adds the audio parameters associated with the first audio type to the first global variable.

[0229] S803, the algorithm module processes the audio data collected by the first mic path according to the parameters in the first global variable.

[0230] Exemplarily, the algorithm module receives the audio data collected through the first mic path sent by the audio driver, and processes the audio data collected through the first mic path according to the parameters in the first global variable. When the first application is a recording application, the implementation of this step can refer to the description for S507; when the first application is a video recording application, the implementation of this step can refer to the description for S617.

[0231] S804. The audio service determines whether the audio collection task of the first application has ended. If so, S805 is executed; if not, S806 is executed.

[0232] S805. The audio HAL closes the first mic path, and the algorithm module deletes the first global variable, and then it ends.

[0233] Exemplarily, in response to the user's operation to end the audio collection task of the first application, the first application sends a first end collection request to the audio service. After receiving the first end collection request, the audio service sends a first end collection flag to the audio HAL. The audio HAL closes the first mic path according to the first end collection flag, and sends the first end collection flag to the algorithm module. The algorithm module deletes the first global variable according to the first end collection flag.

[0234] S806. The audio service determines whether to switch the first application to run in the background and run the audio collection task of the second application in the foreground. If so, S807 is executed; if not, it returns to execute S803.

[0235] Among them, the second application is an application with an audio collection function, and the second application and the first application are two different applications.

[0236] S807. The audio HAL obtains the second audio type and the second application identifier of the second application.

[0237] Exemplarily, in response to the user's operation to run the audio collection task of the second application in the foreground, the second application sends a second audio collection request to the audio service. In response to the second audio collection request, the audio service sends the second audio type and the second application identifier of the second application to the audio HAL, so that the audio HAL obtains the second audio type and the second application identifier.

[0238] S808. The audio HAL opens the second mic path corresponding to the second audio type, and the algorithm module creates a second global variable corresponding to the second application identifier, and adds the audio parameters associated with the second audio type to the second global variable.

[0239] Exemplarily, after the audio HAL opens the second mic path according to the second audio type, it sends the second audio type and the second application identifier to the algorithm module. The algorithm module determines whether there is a global variable corresponding to the second application identifier. If there is, it directly calls the global variable corresponding to the second application identifier; if not, it creates a second global variable corresponding to the second application identifier and adds the audio parameters associated with the second audio type to the second global variable.

[0240] S809. The algorithm module processes the audio data collected by the second mic path according to the parameters in the second global variable.

[0241] Exemplarily, the algorithm module configures the corresponding recording algorithm according to the parameters in the second global variable and processes the audio data collected by the second mic path by using the configured recording algorithm. For example, when the first application is a recording application, the implementation of this step can refer to the description of S507; when the first application is a video recording application, the implementation of this step can refer to the description of S617.

[0242] S810. The audio service determines whether the audio capture task of the second application is completed. If so, it executes S811; if not, it returns to execute S809.

[0243] S811. The audio HAL closes the second mic path, and the algorithm module deletes the second global variable.

[0244] S812. The audio service determines whether to switch the audio capture task of the first application to the foreground for running. If so, it executes S813; if not, it ends.

[0245] S813. The audio HAL opens the first mic path according to the first application identifier; the algorithm module calls the first global variable, and then executes S803.

[0246] Exemplarily, the audio HAL searches for the path opening record related to the first application according to the first application identifier, determines the first mic path last used by the first application from the path opening record related to the first application, and further, the audio HAL opens the first mic path through the audio driver.

[0247] This embodiment also provides a computer storage medium. Computer instructions are stored in the computer storage medium. When the computer instructions run on an electronic device, the electronic device is enabled to execute the above relevant method steps to implement the audio data processing method in the above embodiment.

[0248] This embodiment also provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above relevant steps to implement the audio data processing method in the above embodiment.

[0249] In addition, an embodiment of the present application further provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer-executable instructions. When the device runs, the processor may execute the computer-executable instructions stored in the memory, so that the chip executes the audio data processing method in each of the above method embodiments.

[0250] Among them, the electronic device (such as a mobile phone, etc.), computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0251] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0252] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0253] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in each of the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of each embodiment of the present application.

Claims

1. An audio data processing method, characterized in that, Applied to an electronic device, the method includes: In response to a first operation input by a user, a first application obtains a first target audio at a first moment; In response to a first switching operation input by the user, the first application switches to running in the background at a second moment, and the second moment is later than the first moment; In response to a second operation input by the user, a second application obtains a second target audio at a third moment, and the third moment is later than the second moment; In response to a first end operation input by the user, the second application ends the operation of obtaining the target audio at a fourth moment and outputs a first output audio, and the first output audio includes the second target audio, and the fourth moment is later than the third moment; In response to a third operation input by the user, the first application switches back to the foreground at a fifth moment and obtains a third target audio, and the fifth moment is later than the fourth moment; In response to a second end operation input by the user, the first application ends the operation of obtaining the target audio at a sixth moment and outputs a second output audio, and the second output audio includes the first target audio, the third target audio, and a fourth target audio; Wherein, the first target audio is the target audio obtained by the first application from the first moment to the second moment; the fourth target audio is the target audio obtained by the first application from the second moment to the third moment; the second target audio is the target audio obtained by the second application from the third moment to the fourth moment; the third target audio is the target audio obtained by the first application from the fifth moment to the sixth moment, and the sixth moment is later than the fifth moment.

2. The method according to claim 1, characterized in that, The electronic device includes at least three microphones; the at least three microphones include a first microphone, a second microphone, and a third microphone. The first microphone is located at the top of the electronic device, the second microphone is located at the bottom of the electronic device, and the third microphone is located at the back of the electronic device; Wherein, the top of the electronic device is the plane formed by the short axis and the thickness of the electronic device, the bottom of the electronic device is parallel and opposite to the top of the electronic device, and the back of the electronic device is parallel to the display screen of the electronic device and intersects with the top and the bottom of the electronic device.

3. The method according to claim 2, characterized in that, In response to a first operation input by the user, the first application obtains a first target audio at a first moment, including: In response to a first operation input by the user, the first application instructs the hardware abstraction layer HAL of the electronic device to turn on the first microphone and the second microphone at a first moment, so that the first microphone and the second microphone form a first mic path to collect first audio data, and the HAL processes the first audio data to obtain the first target audio; The first application obtains the first target audio.

4. The method according to claim 3, characterized in that, The step of, in response to a second operation input by the user, the second application obtains a second target audio at a third moment, includes: When the first microphone and the second microphone are turned on, in response to a second operation input by the user, the second application instructs the HAL to turn on the third microphone at the third moment, so that the second microphone path composed of the first microphone, the second microphone, and the third microphone collects second audio data, and the HAL processes the second audio data to obtain the second target audio; The second application obtains the second target audio.

5. The method according to any one of claims 2 to 4, characterized in that, In response to a first end operation input by the user, the second application ends the operation of obtaining the target audio at the fourth moment, including: In response to a first end operation input by the user, the second application instructs the hardware abstraction layer HAL of the electronic device to turn off the first microphone, the second microphone, and the third microphone at the fourth moment; In response to a third operation input by the user, the first application switches back to the foreground and obtains the third target audio at the fifth moment, including: In response to a third operation input by the user, the first application switches to the foreground; When the first application switches to the foreground, the first application instructs the HAL to turn on the first microphone and the second microphone, so that the first microphone path composed of the first microphone and the second microphone collects third audio data, and the HAL processes the third audio data to obtain the third target audio; The first application obtains the third target audio.

6. The method according to claim 3, characterized in that, The HAL is used for: Create a first global variable corresponding to the application identifier of the first application; Add audio parameters associated with the audio type of the first application to the first global variable; When the HAL obtains the first audio data, the HAL is also used for: Process the first audio data according to the parameters in the first global variable to obtain the first target audio.

7. The method according to claim 4, characterized in that, The HAL is also used for: Create a second global variable corresponding to the application identifier of the second application; Add audio parameters associated with the audio type of the second application to the second global variable; When the HAL obtains the second audio data, the HAL is also used for: Process the second audio data according to the parameters in the second global variable to obtain the second target audio.

8. The method according to claim 7, wherein When the second application instructs the HAL to turn off the first microphone, the second microphone, and the third microphone, the HAL is also used for: Delete the second global variable.

9. The method according to claim 6, wherein In response to a second end operation input by the user, the first application ends the operation of obtaining the target audio at the sixth moment, including: In response to a second end operation input by the user, the first application instructs the HAL to turn off the first microphone and the second microphone at the sixth moment; When the HAL turns off the first microphone and the second microphone, the HAL is also used for: Delete the first global variable.

10. The method according to any one of claims 1 to 4, 6 to 9, wherein The first application is a recording application, and the second application is a video recording application or a live broadcast application.

11. An electronic device, wherein Including: One or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the audio data processing method according to any one of claims 1-10.

12. A computer-readable storage medium, comprising a computer program, wherein When the computer program runs on an electronic device, cause the electronic device to perform the audio data processing method according to any one of claims 1-10.

Images