Audio processing method, apparatus, medium, and terminal device

Abstract

The application discloses an audio processing method, device, medium and terminal equipment, the audio processing method can be applied to the terminal equipment, the layered system of the terminal equipment includes an abstract layer and an audio drive layer, the method comprises the following steps: the abstract layer obtains the first audio data stream of the audio file to be played at the first moment, and caches to the cache space of the abstract layer; the abstract layer decodes and post-processes the first audio data stream by starting a thread, and obtains the first decoded data stream; the abstract layer transmits the first decoded data stream to the audio drive layer, and caches to the cache space of the audio drive layer. The method can realize the function of the analog hardware data signal processor by means of software, realize the decoding and post-processing of the audio data stream, so that the hardware cost is saved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an audio processing method, apparatus, medium, and terminal device. Background Technology

[0002] Currently, Android's playback architecture primarily employs two modes. The first mode, with low power consumption, involves the application processor reading the file, demultiplexing and decoding it, and then transmitting the decoded data through the audio hardware abstraction layer and audio driver layer to the audio signal processing chip. Finally, the audio output module outputs the audio signal. The second mode, with high power consumption, typically requires the playback device to have its own audio signal processing chip. The application processor only demultiplexes the data without decoding or post-processing; it directly transmits the demultiplexed data through the audio hardware abstraction layer and audio driver layer to the audio signal processing chip, which then decodes, processes, and outputs the audio data. This mode allows the application processor to sleep for extended periods, as most of the audio playback is handled by the low-power audio signal processing chip, achieving power savings. However, this mode has the disadvantage of higher hardware costs, requiring a dedicated low-power audio signal processing chip.

[0003] Therefore, it is necessary to improve the software system of playback devices in order to save hardware costs as much as possible while saving power consumption. Summary of the Invention

[0004] This invention provides an audio processing method, apparatus, medium, and terminal device to minimize hardware costs while saving power consumption.

[0005] In a first aspect, embodiments of the present invention provide an audio processing method applicable to a terminal device. The terminal device's layered system includes an audio hardware abstraction layer and an audio driver layer. The method includes: the audio hardware abstraction layer acquiring a first audio data stream of an audio file to be played at a first moment and caching it in its cache space; the audio hardware abstraction layer decoding and post-processing the first audio data stream by starting a thread to obtain a first decoded data stream; and the audio hardware abstraction layer transmitting the first decoded data stream to the audio driver layer and caching it in its cache space.

[0006] The beneficial effects of the audio processing method provided in this invention are as follows: On the one hand, decoding and post-processing are both completed in a single thread within the audio hardware abstraction layer, reducing the complex playback process in the upper-layer audio playback module of Android, such as thread interaction and process interaction, and reducing the instruction consumption of the application processor; on the other hand, this method can realize the decoding and post-processing of audio data streams by starting a thread in the audio hardware abstraction layer to simulate the function of a hardware data signal processor, which can solve the audio playback power consumption problem of Android smartphones and save hardware costs.

[0007] In one possible implementation, the above method further includes: the audio output module of the terminal device outputs the audio signal corresponding to the first decoded data stream.

[0008] In one possible implementation, the above method further includes:

[0009] When the first decoded data stream in the cache space is output by the audio output module of the terminal device, the audio hardware abstraction layer returns at a second moment to retrieve the second audio data stream of the audio file to be played and caches it in the cache space of the audio hardware abstraction layer. The audio hardware abstraction layer decodes and post-processes the second audio data stream through the thread to obtain the second decoded data stream. The audio hardware abstraction layer transmits the second decoded data stream to the audio driver layer and caches it in the cache space of the audio driver layer. The audio output module of the terminal device outputs the second decoded data stream in the cache space. In this method, after the upper layer fills the cache space of the audio hardware abstraction layer with data, the upper layer software module is in a stopped waiting state. Since the cache space of the audio hardware abstraction layer contains a lot of data, the upper layer module will only be notified to continue filling data after the data in the cache space of the audio hardware abstraction layer is exhausted, thus reducing thread interaction and process interaction.

[0010] In one possible implementation, the above method further includes an audio output module comprising a first audio output path and a second audio output path; the audio output module also includes a mixing module for mixing the sounds output from the first and second audio output paths respectively. Additionally, the second output path includes a volume gain module for controlling the volume. This eliminates the need for a dedicated audio data signal processing chip; only a second audio output path needs to be added to the audio output module, significantly reducing costs.

[0011] In one possible implementation, the above method is performed on the premise that the terminal device is operating in a low-power playback mode. When the terminal device is operating in a low-power playback mode, the application processor can perform demultiplexing, decoding, and post-processing.

[0012] Secondly, embodiments of the present invention also provide an audio processing apparatus, wherein the audio processing apparatus comprises modules / units that execute any possible design method of the first aspect described above. These modules / units can be implemented in hardware or by hardware executing corresponding software.

[0013] Thirdly, embodiments of the present invention provide a terminal device, including a processor and a memory. The memory stores one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the terminal device is able to implement any of the possible design methods described in the first aspect.

[0014] Fourthly, this invention also provides a computer-readable storage medium comprising a computer program that, when run on a terminal device, causes the terminal device to perform any of the possible designs of the first aspect described above.

[0015] Fifthly, embodiments of the present invention also provide a method comprising a computer program product, which, when the computer program product is run on a terminal, causes the terminal device to execute any of the possible designs of the first aspect described above.

[0016] For the beneficial effects of the second to fifth aspects mentioned above, please refer to the description in the first aspect mentioned above. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the software architecture of a terminal device provided in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of an audio processing method provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of an audio processing device provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the embodiments of the present invention, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the present invention. As used in the specification and appended claims of the present invention, the singular expressions “a,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present invention, “at least one” and “one or more” refer to one or more (including two). The term “and / or” is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0023] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the invention. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," and "in still other embodiments" appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized. The term "connection" includes both direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0024] In embodiments of the present invention, "exemplarily" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design described as "exemplarily" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0025] like Figure 1 As shown, the audio processing method provided in this application embodiment can be applied to, for example... Figure 1 The terminal device shown, Figure 1 A hardware configuration block diagram of the terminal device 100 is shown.

[0026] In some embodiments, the terminal device 100 includes at least one of a tuner 110, a communicator 120, a data acquisition unit 130, an external device interface 140, a controller 150, a display 160, an audio output interface 170, a memory, a power supply, and a user interface.

[0027] In some embodiments, the display 160 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a controller, and a user control interface, etc.

[0028] In some embodiments, the display 160 may be at least one of a liquid crystal display, an organic light-emitting diode (OLED), and a projection display, and may also be a projection device and a projection screen.

[0029] In some embodiments, the tuner 110 receives broadcast television signals via wired or wireless means and demodulates audio and video signals from a plurality of wireless or wired broadcast television signals.

[0030] In some embodiments, the communicator 120 is a component used to communicate with external devices or servers according to various communication protocol types. In one possible embodiment, the communicator may include at least one of a wireless fidelity (Wi-Fi) module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chips or near-field communication protocol chips, as well as an infrared receiver. The terminal device 100 can establish the transmission and reception of control signals and data signals with other devices through the communicator 120.

[0031] In some embodiments, the collector 130 is used to collect signals from the external environment or to interact with the outside world. In one possible embodiment, the collector 130 includes a light receiver and a sensor for collecting ambient light intensity; or the collector 130 includes an image collector for collecting external environmental scenes, user attributes, or user interaction gestures; or the collector 130 includes a sound collector for receiving external sounds.

[0032] In some embodiments, the external device interface 140 may include, but is not limited to, one or more interfaces such as: a high-definition multimedia interface, an analog or high-definition component input interface, a composite video input interface, and a universal serial bus (USB) input interface. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0033] In some embodiments, the controller 150 controls the operation of the terminal device 100 and responds to user operations through various software control programs stored in memory. The controller 150 controls the overall operation of the terminal device 100. In one possible embodiment, in response to receiving a user command for selecting an object to be displayed on the display 160, the controller 150 can perform operations related to the object selected by the user command.

[0034] In some embodiments, the object can be any of the optional objects. In one possible embodiment, the object can be a hyperlink, an icon, or other operable control. Operations related to the selected object include: displaying links to hyperlinked pages, documents, images, etc., or performing operations corresponding to the program associated with the icon.

[0035] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), random access memory (RAM), read-only memory (ROM), a first to an nth interface for input / output, a communication bus, etc.

[0036] A central processing unit (CPU) is used to execute operating system and application instructions stored in memory, as well as various interactive instructions received from external input, to execute various applications, data, and content in order to ultimately display and play various audio and video content. A CPU may include multiple processors. In one possible embodiment, the CPU includes a main processor and one or more subprocessors.

[0037] In some embodiments, a graphics processor is used to generate various graphical objects. In one possible embodiment, the various graphical objects include at least one of icons, operation menus, and user-input command-displayed graphics. The graphics processor includes an arithmetic logic unit (ALU) that performs calculations based on various user-input interactive commands and displays various objects according to display attributes; it also includes a renderer that renders the various objects obtained based on the ALU, and the rendered objects are used to display on a display.

[0038] In some embodiments, a video processor is configured to receive an external video signal and perform at least one of the following video processing operations according to a standard encoding and decoding protocol of the input signal: decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc., to obtain a signal that can be directly displayed or played on a terminal device 100.

[0039] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, and a display formatting module. The demultiplexing module demultiplexes the input audio and video data streams. The video decoding module processes the demultiplexed video signal, including decoding and scaling. The image compositing module overlays and blends a graphical user interface (GUI) signal generated by a graphics generator based on user input or its own generation with the scaled video image to generate a displayable image signal. The frame rate conversion module converts the input video frame rate. The display formatting module modifies the received frame rate-converted video output signal to conform to a display format. In one possible embodiment, the display formatting module outputs RGB data signals.

[0040] In some embodiments, an audio processor is configured to receive external audio signals, perform decompression and decoding according to a standard codec protocol of the input signal, and at least one of the following processes: noise reduction, digital-to-analog conversion, and amplification, to obtain a sound signal that can be played in a speaker.

[0041] In some embodiments, a user can input user commands through a graphical user interface displayed on the display 160, and the user input interface receives the user input commands through the graphical user interface. Alternatively, a user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0042] In some embodiments, a user interface is the medium through which an application or operating system interacts and exchanges information with a user, converting information between its internal form and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of a terminal device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, and navigation bars.

[0043] In some embodiments, the user interface 180 is an interface that can be used to receive control input. In one possible embodiment, the user interface 180 may be a physical button on the terminal device body.

[0044] In one possible implementation, the terminal device 100 can be any one of a mobile phone, tablet computer, handheld computer, personal computer (PC), cellular phone, personal digital assistant (PDA), wearable device, smart home device, in-vehicle computer, game console, or augmented reality (AR) / virtual reality (VR) device. It is worth noting that this embodiment does not impose specific limitations on the specific device form of the terminal device 100.

[0045] like Figure 2 As shown in the embodiments of this application, a software system architecture suitable for the aforementioned terminal device is provided. During audio playback, in low-power playback mode, the audio playback module reads the audio file and demultiplexes it. The demultiplexed file is then transmitted to the audio execution layer (AudioFlinger), which is responsible for managing input / output stream devices and processing and transmitting audio stream data. The audio execution layer (AudioFlinger) transmits the demultiplexed file to the audio hardware abstraction layer (Audio HAL), which is responsible for interacting with the audio hardware device and is directly invoked by the audio execution layer (AudioFlinger). The audio hardware abstraction layer caches the data in a large stream buffer. It starts a thread to perform decoding and post-processing, then puts the data into the driver's large PCM buffer and moves the data from the driver's large PCM buffer to the audio output module. The audio output module includes a first audio output path and a second audio output path; the audio output module also includes a mixing module, which mixes the sounds output from the first and second audio output paths respectively. In addition, the second output path includes a volume gain module, which is used to control the volume.

[0046] In addition, during audio playback, in high-power playback mode, the audio playback module reads the audio file for demultiplexing, then demultiplexes, decodes, and recomputes the file. After that, the audio execution layer mixes the audio, and then the data passes through the audio hardware abstraction layer and the audio driver layer before being transmitted to the audio output module.

[0047] based on Figure 1 The hardware structure and software system architecture of the terminal device 100 shown in this application embodiment are illustrated in the flowchart of an audio processing method. Figure 3As shown, the process of this method can be executed by terminal device 100, and the method includes the following steps:

[0048] S301, the audio hardware abstraction layer obtains the first audio data stream of the audio file to be played at the first moment and caches it in the cache space of the audio hardware abstraction layer.

[0049] S302, the audio hardware abstraction layer decodes and post-processes the first audio data stream by starting a thread to obtain the first decoded data stream;

[0050] S303, the audio hardware abstraction layer transmits the first decoded data stream to the audio driver layer and caches it in the cache space of the audio driver layer.

[0051] It is worth noting that when the terminal device is working in low-power playback mode, the terminal device can use the above method to simulate the function of a hardware data signal processor through software, thereby realizing the decoding and post-processing of audio data streams and saving hardware costs.

[0052] Specifically, in the above audio processing flow, the upper-layer application sends the demultiplexed data to the audio hardware abstraction layer. The audio hardware abstraction layer stores the demultiplexed data in its storage space (largestream buffer) (60Kbytes of data). The audio hardware abstraction layer starts a thread to perform decoding and post-processing, and then puts the data into the driver layer's storage space (Large PCM Buffer). Finally, the data in the driver layer's storage space (Large PCM Buffer) is moved to the audio output module.

[0053] Furthermore, this embodiment also improves the hardware structure of the playback device. The method further includes the terminal device's audio output module outputting the audio signal corresponding to the first decoded data stream. The audio output module includes a first audio output path and a second audio output path; the audio output module also includes a mixer module, which is used to mix the sounds output from the first audio output path and the second audio output path respectively. In addition, the second output path includes a digital gain module, which is used to control the volume. This eliminates the need for a dedicated audio data signal processing chip; only the second audio output path needs to be added to the audio output module, significantly reducing costs.

[0054] In one possible embodiment, the method further includes the following steps: when the first decoded data stream in the cache space is output by the audio output module of the terminal device, the audio hardware abstraction layer returns at a second moment to obtain the second audio data stream of the audio file to be played and caches it in the cache space of the audio hardware abstraction layer; the audio hardware abstraction layer decodes and post-processes the second audio data stream through the thread to obtain the second decoded data stream; the audio hardware abstraction layer transmits the second decoded data stream to the audio driver layer and caches it in the cache space of the audio driver layer; the audio output module of the terminal device outputs the second decoded data stream in the cache space. In this method, after the upper layer fills the data into the cache space of the audio hardware abstraction layer, the upper layer software module is in a stopped waiting state. Since the cache space of the audio hardware abstraction layer contains a lot of data, the upper layer module is notified to continue filling data only after the data in the cache space of the audio hardware abstraction layer is exhausted, reducing thread interaction and process interaction. In addition, according to actual testing, the low-power playback mode provided in this embodiment saves about 10mA (3.8V) of power consumption compared with the high-power playback mode.

[0055] In other embodiments of this application, a terminal device 100 is disclosed, such as... Figure 4 As shown, the terminal device 400 may include: one or more processors 401; a memory 402; a display 403; one or more application programs (not shown); and one or more computer programs 404. These devices can be connected via one or more communication buses 405. The one or more computer programs 404 are stored in the memory 402 and configured to be executed by the one or more processors 401. The one or more computer programs 404 include instructions that can be used to perform actions such as... Figure 3 Each step in the corresponding embodiment.

[0056] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0057] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0058] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0059] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An audio processing method applied to a terminal device, wherein the layered system of the terminal device includes an audio hardware abstraction layer and an audio driver layer, characterized in that, The method includes: The audio hardware abstraction layer acquires the first audio data stream of the audio file to be played at the first moment and caches it in the cache space of the audio hardware abstraction layer; The audio hardware abstraction layer decodes and post-processes the first audio data stream by starting a thread to obtain the first decoded data stream; The audio hardware abstraction layer transmits the first decoded data stream to the audio driver layer and caches it in the cache space of the audio driver layer; the terminal device operates in low-power playback mode; When the first decoded data stream in the buffer space is output by the audio output module of the terminal device, the audio hardware abstraction layer returns at a second moment to obtain the second audio data stream of the audio file to be played, and caches it in the buffer space of the audio hardware abstraction layer; the audio hardware abstraction layer decodes and post-processes the second audio data stream through the thread to obtain the second decoded data stream; the audio hardware abstraction layer transmits the second decoded data stream to the audio driver layer and caches it in the buffer space of the audio driver layer; the audio output module of the terminal device outputs the second decoded data stream in the buffer space.

2. The method according to claim 1, characterized in that, Also includes: The audio output module of the terminal device outputs the audio signal corresponding to the first decoded data stream.

3. The method according to claim 2, characterized in that, The audio output module includes a first audio output path and a second audio output path; the audio output module also includes a mixing module, which is used to mix the sounds output from the first audio output path and the second audio output path respectively.

4. The method according to claim 3, characterized in that, The second audio output path includes a volume gain module, which is used to control the volume.

5. An audio processing device, characterized in that, Applied to terminal devices, the device includes an audio hardware abstraction layer and an audio driver layer: The audio hardware abstraction layer is used to acquire the first audio data stream of the audio file to be played at the first moment and cache it in the cache space of the audio hardware abstraction layer; the first audio data stream is decoded and post-processed by a newly started thread to obtain the first decoded data stream; The audio hardware abstraction layer is also used to transmit the first decoded data stream to the audio driver layer and cache it in the cache space of the audio driver layer; The terminal device operates in low-power playback mode; When the first decoded data stream in the cache space is played by the audio player of the terminal device, the audio hardware abstraction layer is also used to obtain the second audio data stream of the audio file to be played at the second moment and cache it in the cache space of the audio hardware abstraction layer. The audio hardware abstraction layer is also used to decode and post-process the second audio data stream through the thread to obtain a second decoded data stream; The audio hardware abstraction layer transmits the second decoded data stream to the audio driver layer and caches it in the cache space of the audio driver layer. The second decoded data stream in the cache space is played by the audio output module of the terminal device.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

7. A terminal device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, causes the processor to perform the method of any one of claims 1 to 4.

Images