Audio playing method and device and storage medium

Abstract

The invention relates to an audio playing method and device and a storage medium. The method comprises the following steps: in response to a voice call performed by a first application, starting a second application which is used for providing call background sound for the first application; a first audio file is obtained based on the first application, a second audio file is obtained based on the second application, the first audio file is an audio file of a user call, and the second audio file is an audio file of call background sound; performing sound mixing processing on the first audio file and the second audio file to obtain a target audio file; and performing a voice call based on an internet protocol based on the target audio file. By means of the method, the requirement of the user for playing the call background sound in the VoIP scene is met, and then the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing, and in particular to an audio playback method, device and storage medium. Background Art

[0002] Call background sound refers to the relevant audio played by the user during a voice call so that the played audio can be used as the background audio of the user's voice call, thereby providing protection for the user's call scenario.

[0003] However, at present, call background sound can only be used in system call scenarios and cannot be applied to social software with voice call functions, resulting in a single application scenario and affecting user experience. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an audio playback method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an audio playback method, comprising: in response to a first application making a voice call, enabling a second application, the second application being used to provide call background sound for the first application; obtaining a first audio file based on the first application and obtaining a second audio file based on the second application, the first audio file being an audio file of the user's call, and the second audio file being an audio file of the call background sound; mixing the first audio file with the second audio file to obtain a target audio file; and making an Internet Protocol-based voice call based on the target audio file.

[0006] In one embodiment, obtaining the second audio file based on the second application includes: obtaining the second audio file based on an audio data stream; or, obtaining the second audio file based on a first thread, wherein the first thread is used to send the second audio file to a recording thread; or, obtaining the second audio file based on a second thread, wherein the second thread is used to send the second audio file to the recording thread and send other audio files to an audio processor, wherein the other audio files are audio files other than the second audio file.

[0007] In another embodiment, obtaining the second audio file based on the audio data stream includes: configuring a target identifier for the second audio file; based on the target identifier, selecting a corresponding audio data stream and a sending path for the audio data stream; and sending the audio data stream to an audio processor based on the sending path of the audio stream to obtain the second audio file.

[0008] In another embodiment, obtaining the second audio file based on the first thread includes: creating a first thread and a first cache area, wherein the first cache area is an internal cache area of ​​the first thread; selecting a designated audio file in the second application as the second audio file; sending the second audio file to the first cache area so that the first thread carries the second audio file; sending the first thread to the cache area of ​​a recording thread; and obtaining the second audio file in the cache area of ​​the recording thread.

[0009] In another embodiment, obtaining the second audio file based on the second thread includes: creating a second cache area, the second cache area is an internal cache area of ​​the second thread; selecting a specified audio file in the second application as the second audio file; sending the second audio file to the second cache area; multiplexing the second thread to send the second audio file to the cache area of ​​the recording thread, and deleting the second audio file inside the second cache area; obtaining the second audio file in the cache area of ​​the recording thread.

[0010] In another embodiment, obtaining the target audio file includes any one of the following methods: mixing the first audio file and the second audio file in an audio processor to obtain the target audio file; mixing the first audio file and the second audio file in a recording thread to obtain the target audio file.

[0011] In yet another embodiment, the obtaining of the first audio file based on the first application includes: obtaining an initial audio file of a user call based on the first application; and performing denoising on the initial audio file to obtain the first audio file.

[0012] According to a second aspect of an embodiment of the present disclosure, an audio playback device is provided, including: a processing unit, for enabling a second application in response to a voice call made by a first application, the second application being used to provide call background sound for the first application; obtaining a first audio file based on the first application and obtaining a second audio file based on the second application, the first audio file being an audio file of a user call, and the second audio file being an audio file of the call background sound; mixing the first audio file and the second audio file to obtain a target audio file; and conducting an Internet Protocol-based voice call based on the target audio file.

[0013] In one embodiment, the processing unit obtains the second audio file based on the second application in the following manner: or, obtains the second audio file based on an audio data stream; or, obtains the second audio file based on a first thread, wherein the first thread is used to send the second audio file to a recording thread; or, obtains the second audio file based on a second thread, wherein the second thread is used to send the second audio file to the recording thread and send other audio files to the audio processor, wherein the other audio files are audio files other than the second audio file.

[0014] In another embodiment, the processing unit obtains the second audio file based on the audio data stream, including: configuring a target identifier for the second audio file; based on the target identifier, selecting a corresponding audio data stream and a sending path for the audio data stream; and sending the audio data stream to an audio processor based on the sending path of the audio stream to obtain the second audio file.

[0015] In another embodiment, the processing unit obtains the second audio file based on the first thread, including: creating a first thread and creating a first cache area, the first cache area being an internal cache area of ​​the first thread; selecting a designated audio file in the second application as the second audio file; sending the second audio file to the first cache area so that the first thread carries the second audio file; sending the first thread to the cache area of ​​the recording thread; and obtaining the second audio file in the cache area of ​​the recording thread.

[0016] In another embodiment, the processing unit obtains the second audio file based on the second thread, including: creating a second cache area, the second cache area is an internal cache area of ​​the second thread; selecting a specified audio file in the second application as the second audio file; sending the second audio file to the second cache area; multiplexing the second thread to send the second audio file to the cache area of ​​the recording thread, and deleting the second audio file inside the second cache area; and obtaining the second audio file in the cache area of ​​the recording thread.

[0017] In another embodiment, the processing unit obtains the target audio file in any one of the following ways: mixing the first audio file and the second audio file in an audio processor to obtain the target audio file; mixing the first audio file and the second audio file in a recording thread to obtain the target audio file.

[0018] In another embodiment, the processing unit obtains the first audio file based on the first application in the following manner, including: obtaining an initial audio file of a user call based on the first application; and performing denoising on the initial audio file to obtain the first audio file.

[0019] According to a third aspect of an embodiment of the present disclosure, an audio playback device is provided, comprising: a processor; and a memory for storing processor executable commands; wherein the processor is configured to: execute the audio playback method described in the first aspect or any one implementation of the first aspect.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a device, the device is enabled to execute the audio playback method described in the first aspect or any one of the implementations of the first aspect.

[0021] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by obtaining the user's audio file from the application for voice call, mixing it with the call background audio file, and then sending it to the voice call object through the application for voice call, the user's need for call background sound playback is met, thereby improving the user experience.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] Figure 1 The present invention is a flowchart of an audio method according to an exemplary embodiment.

[0025] Figure 2 The figure is a flow chart of a method for obtaining a second audio file according to an exemplary embodiment.

[0026] Figure 3 The present invention is a flow chart of a method for acquiring a second audio file based on an audio data stream according to an exemplary embodiment.

[0027] Figure 4 The present invention is a flow chart showing a method for obtaining a second audio file based on a first thread according to an exemplary embodiment.

[0028] Figure 5 The present invention is a flow chart showing a method for obtaining a second audio file based on a first thread according to an exemplary embodiment.

[0029] Figure 6 The figure is a flow chart of a method for acquiring a target audio file according to an exemplary embodiment.

[0030] Figure 7 This is a schematic diagram of an application framework of a method for obtaining a first audio file according to an exemplary embodiment.

[0031] Fig. 8A The present invention is a schematic diagram of an application framework of an audio playback method according to an exemplary embodiment.

[0032] Figure 8B The present invention is a schematic diagram of an application framework of an audio playback method according to an exemplary embodiment.

[0033] Figure 8C The present invention is a schematic diagram of an application framework of an audio playback method according to an exemplary embodiment.

[0034] Fig. 9 The figure is a block diagram of an audio playback device according to an exemplary embodiment.

[0035] Fig.10 The invention is a block diagram of a device for audio playback according to an exemplary embodiment.

[0036] Fig.11 The invention is a block diagram of a device for audio playback according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0038] The audio playback method involved in the embodiments of the present disclosure is applied to a terminal. In some embodiments, the terminal includes, for example, a mobile phone, a wearable device, an Internet of Things device, a car with a communication function, a smart car, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in a smart grid, a wireless terminal device in transportation safety, a wireless terminal device in a smart city, and at least one of a wireless terminal device in a smart home, but is not limited thereto.

[0039] It should be noted that, in order to facilitate understanding of the technical solution, the following embodiments are described using a terminal in the form of a smart phone and a network in the form of a cell.

[0040] In related technologies, with the advancement of communication technology and the development of terminal technology, call background sound technology has gradually entered the lives of users. Call background sound technology is a technology used during a call, which can add a kind of background noise or music. It can be used for a variety of purposes, such as setting background sound in a confidential meeting to blur the content of the conversation and prevent eavesdropping, or playing soft music during a call to increase the pleasure of the call.

[0041] For example, after the user selects the corresponding background sound in the call background sound software, the call background sound will be sent to the terminal of the call partner through the subscriber identity module (SIM) to play the call background sound. Therefore, in the related art, the use of call background sound is limited to the scenario where the user makes a system call.

[0042] With the development of communication technology, the scenario of users making network voice calls through social software has gradually increased. However, the relevant social software cannot provide users with the corresponding function of playing call background sounds. For example, social software such as "A Messenger" and "B Chat" can both provide voice call functions. However, since the above social software conducts voice calls based on the Internet Protocol (IP), that is, VoIP (Voice over Internet Protocol), it is not a system call scenario. VoIP transmits communication voice data based on IP, while system calls transmit voice data based on SIM cards. Therefore, the background sound call technology in the related art cannot be directly applied to the VoIP scenario.

[0043] Based on the above problems, the present disclosure proposes an audio playback method. By mixing the background sound audio file and the audio file of the user's call, a target file is obtained. And the target call software that conducts calls based on IP is called for audio playback, enabling users to play call background sounds in the VoIP scenario, thereby providing scenario protection for the calls of users in the VoIP scenario and further enhancing the user experience.

[0044] It should be noted that for the convenience of understanding the embodiments of the present disclosure, some terms in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0045] VoIP: VoIP is the abbreviation of "Voice over Internet Protocol", which is a technology that converts analog voice signals into digital signals and transmits them over the Internet. This technology enables people to make telephone calls through specific devices or software (such as Internet phones or VoIP applications) where there is an Internet connection, without the need for traditional telephone lines. The Chinese name of VoIP is "Voice / Audio over Network Protocol" or more commonly "Internet phone" or "IP phone". This technology can not only reduce the costs of long-distance and international calls, but also provide advanced functions such as voicemail and call forwarding. At the same time, VoIP also supports functions such as video calls and multi-person conferences, so it has been widely used in many fields of modern society.

[0046] Mixing: For example, in the Android audio architecture, mixing is a key link, which involves the process of combining multiple audio sources (or audio tracks) into a single audio stream. Specifically, it includes the following steps:

[0047] 1) Selection of audio sources; 2) Reading of audio data; 3) Volume control; 4) Audio mixing; 5) Output.

[0048] Audio data stream: also known as audio stream or audio streaming, is a continuous audio signal that is encoded into a string of digital data and sent to users through the network. Any audio file on the Internet, such as online music, radio, podcasts, music on the web, audio parts of voice calls or video conferences, can be considered as an audio data stream. The main feature of audio data stream is that users can start listening without waiting for the entire file to be loaded. This means that you can play audio while downloading. When you listen to online radio or watch online live broadcasts, you are receiving an audio data stream. Audio data streams are usually compressed to reduce their size, so they need to be decoded before playback. The decoded audio signal is then converted into sound output. This process is usually completed by related audio playback software or hardware devices.

[0049] The Audio Digital Signal Processor (ADSP) is a low-power digital signal processor optimized for multimedia and signal processing, with multi-threading and multi-core capabilities. In the ADSP framework, the processor works with other hardware (such as codecs, ADCs and DACs) and the Android OS (operating system) to implement various audio processing functions, such as sound effect processing, noise reduction, echo cancellation, speech recognition, and voice enhancement.

[0050] For ease of understanding, in some embodiments, the “audio digital signal processor ADSP” will be referred to as the “audio processor” for short.

[0051] Recording thread: In the Android audio architecture, the recording thread is a thread dedicated to handling audio recording operations. Its main responsibility is to obtain audio data from audio hardware (for example, microphone) or software, and then transmit the data to applications or other audio components.

[0052] Framework: In the Android audio architecture, "framework" usually refers to the AndroidFramework layer, which is an important part of the Android system architecture. The Android Framework provides various APIs (application programming interfaces) for application developers to use to implement various functions, including audio playback and recording.

[0053] AudioFlinger: It is an audio system service in the Android operating system, responsible for managing and processing all audio resources. AudioFlinger is the session layer in the OSI model and is a bridge between the hardware abstraction layer and the application layer. It can handle various audio data formats and audio effects, and can manage the audio output and input of the device. It can be regarded as an audio "service" that can provide functions such as audio playback and audio recording.

[0054] Audio Calibration Database (ACDB): In audio systems, ACDB is usually used to store audio path configuration and calibration data for specific audio devices. These data are developed during the device manufacturing process to optimize audio performance and achieve a high-quality audio experience.

[0055] Key-Value Manager (KV Manager): It is responsible for managing key-value data. Key-Value is a very basic data storage structure, in which the "key" is a unique identifier used to find the corresponding "value". The "key-value" data storage structure is very useful in many scenarios, such as cache systems, configuration management, NoSQL databases, etc. In these systems, the "KV Manager" is the component responsible for operating key-value data (such as inserting, deleting, searching, etc.). The key-value manager in the embodiment of the present disclosure is mainly aimed at the cache system of audio data and the configuration management of audio data.

[0056] MediaPlayer: refers to a class of audio files, used to play audio and video in various formats. Applications can control the playback, pause, and stop of audio through the MediaPlayer API.

[0057] Audio track: refers to a class of audio files, used to directly play PCM audio. Compared with MediaPlayer, AudioTrack provides more low-level control and can be used in applications that require fine-grained control of audio playback.

[0058] Figure 1 is a flowchart of an audio method according to an exemplary embodiment. Figure 1 As shown, the audio method is used in a terminal and includes the following steps.

[0059] In step S11 , in response to the first application making a voice call, the second application is activated.

[0060] In step S12, a first audio file is obtained, and a second audio file is obtained.

[0061] In step S13, the first audio file and the second audio file are mixed to obtain a target audio file.

[0062] In step S14, a voice call based on the Internet protocol is performed based on the target audio file.

[0063] In some embodiments, in response to the first application conducting a voice over Internet Protocol call, the second application is enabled.

[0064] In some embodiments, the first application is an application capable of conducting voice calls based on IP.

[0065] Optionally, the first application is a reference that supports VoIP calls.

[0066] In some embodiments, the first audio file is obtained based on the first application.

[0067] In some embodiments, the first audio file is an audio file of a user's voice call.

[0068] Optionally, the first audio file is the audio emitted by the user during a voice call.

[0069] Exemplarily, the first audio file is the user's speaking sound, breathing sound, etc. during a voice call.

[0070] In some embodiments, obtaining the first audio file based on the first application includes: the first application obtaining the first audio file by controlling an audio acquisition device.

[0071] Exemplarily, the first application collects the audio emitted by the user during a voice call by controlling a microphone of a preset value of the terminal.

[0072] In some embodiments, the second application is used to provide call background sound for the first application.

[0073] In some embodiments, the second audio file is obtained based on the second application.

[0074] In some embodiments, the second audio file is an audio file of call background sound.

[0075] Optionally, the second audio file is the call background sound configured for the user during a voice call.

[0076] Exemplarily, the second audio file is a pre-set scene audio (eg, subway background sound, classroom background sound, etc.) or music audio (eg, song background sound, etc.).

[0077] In some embodiments, the second audio file may be obtained based on at least one of the following methods, including:

[0078] Based on pre-configured audio file libraries, based on network databases, based on user-defined editing, etc.

[0079] In some embodiments, sending the target voice call file to the voice call object can also be understood as the voice call object obtaining the target voice call file.

[0080] The audio playback method provided in the embodiment of the present disclosure obtains the user's audio file from the IP-based voice call application, mixes it with the call background audio file, and then sends it to the voice call object through the IP-based voice call application, thereby meeting the user's demand for call background sound playback in the VoIP scenario, thereby improving the user experience.

[0081] In some embodiments, the method further comprises performing audio denoising on at least one of the following:

[0082] The first audio file is subjected to denoising, the second audio file is subjected to denoising, and the target audio file is subjected to denoising.

[0083] In some embodiments, after acquiring the first audio file and before mixing the first audio file with the second audio file, the method further includes: performing denoising on the first audio file.

[0084] In some embodiments, after acquiring the first audio and the second audio, and before mixing the first audio file with the second audio file, the method further includes: performing denoising on the first audio and the second audio, respectively.

[0085] In some embodiments, after acquiring the second audio file and before mixing the first audio file with the second audio file, the method further includes: performing denoising on the first audio file.

[0086] In some embodiments, after acquiring the target audio and before sending the voice call object, the method further includes: performing denoising on the target audio.

[0087] In some embodiments, the denoising process includes at least one of the following methods: denoising based on spectrum subtraction, denoising based on Wiener filtering, denoising based on video mask, denoising based on artificial neural network, etc.

[0088] The audio playback method provided in the embodiment of the present disclosure can reduce the noise of the target audio by performing noise reduction processing on the specified audio, thereby improving the user experience.

[0089] Figure 2 is a flow chart of a method for obtaining a second audio file according to an exemplary embodiment. Figure 2 As shown, the method includes the following steps.

[0090] In step S21, a second audio file is acquired based on the audio data stream.

[0091] In step S22, a second audio file is obtained based on the first thread.

[0092] In step S23, a second audio file is obtained based on the second thread.

[0093] In some embodiments, the audio data stream may be output by the second application.

[0094] In some embodiments, the first thread is used to send the second audio file to the recording thread.

[0095] In some embodiments, the second thread is used to send the second audio file to the recording thread and to send other audio files to the audio processor.

[0096] In some embodiments, the other audio files are audio files other than the second audio file.

[0097] In some embodiments, the above steps S21 to S23 may be implemented independently of each other.

[0098] Figure 3 is a flow chart of a method for obtaining a second audio file based on an audio data stream according to an exemplary embodiment. Figure 3 As shown, the method includes the following steps.

[0099] In step S31, a target identifier is configured for the second audio file.

[0100] In step S32, a corresponding audio data stream and a first sending path are selected.

[0101] In step S33, a second audio file is obtained.

[0102] In some embodiments, the object identifier is used to indicate data characteristics of the second audio file.

[0103] Exemplarily, the target identifier is used to indicate that the second audio file is an audio file used as background sound for a call.

[0104] In some embodiments, the target identifier is configured by the framework layer.

[0105] Optionally, the target identifier is configured by the framework layer based on the received indication information, and the indication information is notified to the framework layer by the first application.

[0106] Optionally, the framework sends an identifier of a specified audio data stream type, ie, a target identifier, to the Hardware Abstraction Layer (HAL) based on the specified information.

[0107] In some embodiments, the audio hardware abstraction layer (audio HAL) creates a corresponding audio data stream according to the target identifier, and sends information related to the audio data stream to the audio processor to establish a transmission path in the audio processor.

[0108] In some embodiments, the audio data stream is sent to the audio processor based on the first sending path to obtain the second audio file.

[0109] In some embodiments, the first audio file is mixed with the second audio file in an audio processor to obtain a target audio file.

[0110] The audio playback method provided in the embodiment of the present disclosure can meet the user's demand for call background sound playback in a VoIP scenario by mixing the first audio file and the second audio file obtained based on the audio data stream in the audio processor to obtain a target audio file, thereby improving the user experience. In addition, the mixing process is directly performed based on the audio processor capability, without the need to establish a new thread or reuse an old thread, thus saving processor resource usage.

[0111] Figure 4 is a flow chart of a method for obtaining a second audio file based on a first thread according to an exemplary embodiment. Figure 4 As shown, the method includes the following steps.

[0112] In step S41, a first thread is created and a first buffer area is created.

[0113] In step S42, a designated audio file is selected in the second application as the second audio file.

[0114] In step S43, the second audio file is sent to the first buffer area.

[0115] In step S44, the first thread is sent to the buffer area of ​​the recording thread.

[0116] In step S45, a second audio file is obtained.

[0117] In some embodiments, the first thread is a second audio file dedicated thread.

[0118] In some embodiments, the first thread is configured not to communicate with the audio manager.

[0119] In some embodiments, the first cache area is an internal cache area of ​​the first thread.

[0120] In some embodiments, the first buffer is used to store the first audio file so that the first thread can carry the second audio file.

[0121] In some embodiments, the method further includes: creating a shared buffer (ring buffer) in AudioFlinger.

[0122] In some embodiments, the shared buffer area is used to store the second audio file to be sent when the first thread sends the second audio file to the buffer area of ​​the recording thread.

[0123] It should be understood that the buffer area of ​​the recording thread mentioned here can be understood as a buffer area used only to store the second audio file.

[0124] It should be understood that the recording thread can be configured with multiple buffer areas for storing the first audio file, the second audio file and the target audio file respectively.

[0125] In some embodiments, the shared buffer may send the first audio file to the buffer of the recording thread.

[0126] In some embodiments, the method further includes: performing denoising processing on the second audio file stored in the first buffer area.

[0127] In some embodiments, the recording thread is used to obtain a first audio file.

[0128] In some embodiments, the recording thread obtains the first audio file based on the audio processor.

[0129] In some embodiments, the first audio file and the second audio file are mixed in a recording thread to obtain a target audio file.

[0130] The audio playback method provided in the embodiment of the present disclosure can meet the user's call background sound playback requirements in the VoIP scenario by mixing the first audio file and the second audio file obtained based on the audio data stream in the audio processor to obtain the target audio file, thereby improving the user experience. In addition, a separate thread is opened for the call background audio file to ensure the accuracy of file transmission.

[0131] Figure 5 is a flow chart of a method for obtaining a second audio file based on a first thread according to an exemplary embodiment. Figure 5 As shown, the method includes the following steps.

[0132] In step S51, a second buffer area is created.

[0133] In step S52, a designated audio file is selected in the second application as the second audio file.

[0134] In step S53, the second audio file is sent to the second buffer area.

[0135] In step S54, the second thread is multiplexed to send the second audio file to the buffer area of ​​the recording thread.

[0136] In step S55, a second audio file is obtained.

[0137] In some embodiments, the second cache area is a cache area inside the second thread.

[0138] In some embodiments, the second thread is an existing thread in the current audio framework.

[0139] In some embodiments, the second thread is arranged in a communication relationship with the audio processor.

[0140] In some embodiments, the second thread may be a mixer thread.

[0141] In some embodiments, the second thread is further provided with a third buffer area for storing other audio files.

[0142] In some embodiments, other audio files may be understood as audio files other than the first audio file and the second audio file.

[0143] In some embodiments, the second thread sends other audio files stored in the third buffer to the audio processor.

[0144] Exemplarily, when the second thread is a mixing thread and is configured as a second buffer and a third buffer, the second application sends the second audio file to the second buffer, and other applications (applications to which other audio files belong) send other audio files to the third buffer.

[0145] The second audio file in the second buffer area is sent to the buffer area in the recording thread, and the second buffer area is cleared.

[0146] The cleared second buffer area and the third buffer area are mixed.

[0147] It can be understood that the second buffer after zero is mixed with the third buffer, and the obtained audio is the other audio in the third buffer (without change), and the mixed audio is sent to the audio processor.

[0148] In some embodiments, the method further includes: creating a shared buffer (ring buffer) in AudioFlinger.

[0149] In some embodiments, the shared buffer area is used to store the second audio file to be sent when the first thread sends the second audio file to the buffer area of ​​the recording thread.

[0150] It should be understood that for the mixing process of the first audio and the second audio in the recording thread, please refer to Figure 4 The description of the related embodiments in the embodiments will not be repeated here one by one.

[0151] The audio playback method provided in the embodiment of the present disclosure can meet the user's demand for call background sound playback in a VoIP scenario by mixing the first audio file and the second audio file obtained based on the audio data stream in the audio processor to obtain a target audio file, thereby improving the user experience. In addition, the mixing process is directly performed based on the audio processor capability, without the need to establish a new thread, thus saving processor resource usage.

[0152] Figure 6 is a flow chart of a method for obtaining a target audio file according to an exemplary embodiment. Figure 6 As shown, the method includes.

[0153] In step S61, a first audio file and a second audio file are obtained.

[0154] In step S62, the target audio file is obtained in the audio processor.

[0155] In step S63, the target audio file is obtained in the recording thread.

[0156] In some embodiments, obtaining the target audio file in the audio processor includes: mixing the first audio file and the second audio file in the audio processor to obtain the target audio file.

[0157] Exemplarily, mixing a first audio file with a second audio file in an audio processor to obtain a target audio file includes the following steps.

[0158] Obtain the first audio file and the second audio file;

[0159] Calling a mixer set in the audio processor to read the first audio file and the second audio file;

[0160] The first audio file and the second audio file are mixed to obtain a target audio file.

[0161] In some embodiments, obtaining a target audio file in a recording thread includes: mixing a first audio file with a second audio file in the recording thread to obtain the target audio file.

[0162] Exemplarily, mixing a first audio file with a second audio file in a recording thread to obtain a target audio file includes the following steps.

[0163] Obtain the first audio file and the second audio file;

[0164] The first audio file and the second audio file are mixed in a recording buffer area preset in the recording thread to obtain a target audio file.

[0165] It can be understood that the action of performing the mixing process can be performed by setting a sub-thread for performing the mixing process in the current recording thread. Figure 7 is a schematic diagram of an application framework of a method for obtaining a first audio file according to an exemplary embodiment. Figure 7 As shown, the steps include.

[0166] In step S71, an initial audio file of a user call is obtained based on the first application.

[0167] In step S72, the initial audio file is subjected to denoising to obtain a first audio file.

[0168] The audio playback method provided in the embodiment of the present disclosure can reduce interference factors in the sample to be mixed by denoising the acquired user's voice before mixing, thereby improving the mixing effect.

[0169] In some embodiments, the initial audio file includes user call audio and other interference factor audio.

[0170] It can be understood that the initial audio file is subjected to denoising, that is, other interfering audio factors are removed.

[0171] It is understandable that if no denoising process is performed (for example, the audio influence of the interference factor in the currently acquired initial audio file is relatively small), the initial audio file is determined as the first audio file.

[0172] Fig. 8A is a schematic diagram of an application framework of an audio playback method according to an exemplary embodiment. Fig. 8A Shown, incl.

[0173] The framework includes: a second application, an audio architecture system, an ADSP, a recording unit and a first application.

[0174] In some embodiments, the first application is an IP call-based application.

[0175] In some embodiments, the first application creates an upstream or downstream track.

[0176] It should be understood that the "uplink" here refers to the path through which the ADSP sends audio files to the software application; the "downlink" here refers to the path through which the software application sends audio files to the ADSP.

[0177] In some embodiments, the second application is used to provide an audio file for background sound of the call.

[0178] In some embodiments, the category of the audio file provided by the second application may be: Audio Track or MediaPlayer category.

[0179] In some embodiments, the second application notifies the audio framework system (Framework) of the information that needs to be looped back (loopback) by sending parameters, that is, notifying the Framework of the audio file that needs to be used as the background sound of the call.

[0180] In some embodiments, the parameter is an identifier.

[0181] In some embodiments, the sent parameters include at least one of the following:

[0182] Select output, select flag, select the type of audio data stream (pal_stream_type), select the key value manager (KV Manager) and select the audio calibration database (ACBD).

[0183] In some embodiments, the Framework sends an identifier specifying the type of audio data stream to the audio hardware abstraction layer (audio hal) based on the parameter.

[0184] Exemplarily, the flag is as follows: (AUDIO_OUTPUT_FLAG_VOIP_INCALL_MUSIC).

[0185] In some embodiments, the audio hal creates a corresponding audio data stream according to the identifier, sends information related to the audio data stream to the ADSP, and finally establishes a path in the ADSP.

[0186] Exemplarily, the access path is represented as follows: Stream_RX:voip_incall_music.

[0187] In some embodiments, the ADSP mixes the first audio file and the second audio file, and transmits the mixed data as uplink data to the first application, and sends it to the other party through the network.

[0188] Figure 8Bis a schematic diagram of an application framework of an audio playback method according to an exemplary embodiment. Figure 8B Shown, incl.

[0189] The framework includes: a second application, AudioFlinger, ADSP and a first application.

[0190] In some embodiments, in AudioFlinger, a thread for upstream injection, ie, a first thread (loopback Thread), is established based on inheriting the playbackthread class.

[0191] In some embodiments, establishing the first thread includes at least one of the following:

[0192] In AudioFlinger, audio processing, sampling rate processing, and sampling bit depth processing are performed on the first thread.

[0193] An identifier is set for the second audio file.

[0194] For example, when playing the background sound of a call, set parameter to audiotrack and set loopback=true.

[0195] In some embodiments, in response to obtaining the set identifier, a target identifier is set for the second audio file.

[0196] Exemplarily, when loopback=true is received, the flag: AUDIO_OUTPUT_FLAG_LOOPBACK is set for the audiotrack.

[0197] If the identifier is identified as a target identifier, a first thread is created, and the first thread has no communication relationship with the hal layer.

[0198] For example, when openOutput is called, if flag is AUDIO_OUTPUT_FLAG_LOOPBACK, loopbackThread is created and output=null is set (no path is opened at the hal layer).

[0199] In some embodiments, a shared buffer is established in AudioFlinger.

[0200] In some embodiments, the second audio file carried by the first thread is sent to the shared buffer area.

[0201] Exemplarily, when thread_loop of loopbackThread writes data, the data is stored in ringbuffer.

[0202] In some embodiments, the first application mixes the shared buffer data into the uplink buffer (ie, the output buffer) during uplink recording.

[0203] In some embodiments, the recording thread stores the acquired first audio file into an input buffer.

[0204] Figure 8C is a schematic diagram of an application framework of an audio playback method according to an exemplary embodiment. Figure 8C Shown, incl.

[0205] The framework includes: a second application, other applications, AudioFlinger, ADSP and a first application.

[0206] In some embodiments, a shared buffer is created within AudioFlinger.

[0207] In some embodiments, when downlink data is written into the mixing thread, it is identified whether the current audio file has a second audio file.

[0208] In some embodiments, if there is a second audio file when the downlink data is written into the mixing thread, the second audio file is stored in the second buffer area, and the audio file in the second buffer area is sent to the shared buffer area.

[0209] For example, when writing downlink data, it is determined whether the current audio track is loopback. If so, the buffer is written into the ringbuffer. And the background sound track data is cleared to 0 so that mixing with the track of the music APP does not affect the playback of other apps.

[0210] It is understandable that the various embodiments of the present disclosure may be implemented separately or in combination. For example, the audio playback schemes corresponding to obtaining the second audio file based on the audio data stream, obtaining the second audio file based on the first thread, and obtaining the second audio file based on the second thread may be implemented separately or together. The present disclosure only lists the independent implementation schemes of the various embodiments, and the implementation methods of the combined application of the various embodiments should not be considered to exceed the scope of the technical solutions of the embodiments of the present disclosure.

[0211] Based on the same concept, an embodiment of the present disclosure also provides an audio playback device.

[0212] It is understandable that, in order to realize the above functions, the audio playback device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0213] Fig. 9 FIG. 1 is a block diagram of an audio playback device 100 according to an exemplary embodiment. Fig. 9 , the device includes a processing unit 101.

[0214] The processing unit 101 is used to enable a second application in response to a voice call made by a first application, where the second application is used to provide background sound for the call for the first application; obtain a first audio file based on the first application and obtain a second audio file based on the second application, where the first audio file is an audio file of the user's call and the second audio file is an audio file of the background sound for the call; mix the first audio file with the second audio file to obtain a target audio file; and conduct an Internet Protocol-based voice call based on the target audio file.

[0215] In one implementation of the disclosed embodiment, the processing unit 101 obtains the second audio file based on the second application in the following manner: or, obtains the second audio file based on the audio data stream; or, obtains the second audio file based on the first thread, and the first thread is used to send the second audio file to the recording thread; or, obtains the second audio file based on the second thread, and the second thread is used to send the second audio file to the recording thread and send other audio files to the audio processor, where the other audio files are audio files other than the second audio file.

[0216] In one implementation of the embodiments of the present disclosure, the processing unit 101 obtains a second audio file based on the audio data stream, including: configuring a target identifier for the second audio file; based on the target identifier, selecting a corresponding audio data stream and a sending path for the audio data stream; and sending the audio data stream to an audio processor based on the sending path of the audio stream to obtain the second audio file.

[0217] In one implementation of the disclosed embodiment, the processing unit 101 obtains the second audio file based on the first thread, including: creating the first thread and creating a first cache area, the first cache area being an internal cache area of ​​the first thread; selecting a specified audio file in the second application as the second audio file; sending the second audio file to the first cache area so that the first thread carries the second audio file; sending the first thread to the cache area of ​​the recording thread; and obtaining the second audio file in the cache area of ​​the recording thread.

[0218] In one implementation of the disclosed embodiment, the processing unit 101 obtains a second audio file based on a second thread, including: creating a second cache area, the second cache being an internal cache area of ​​the second thread; selecting a specified audio file as the second audio file in the second application; sending the second audio file to the second cache area; multiplexing the second thread to send the second audio file to the cache area of ​​the recording thread, and deleting the second audio file within the second cache area; and obtaining the second audio file in the cache area of ​​the recording thread.

[0219] In one implementation of the embodiment of the present disclosure, the processing unit 101 obtains the target audio file in any of the following ways: mixing the first audio file with the second audio file in the audio processor to obtain the target audio file; mixing the first audio file with the second audio file in the recording thread to obtain the target audio file.

[0220] In one implementation of the embodiments of the present disclosure, the processing unit 101 obtains a first audio file based on the first application in the following manner, including: obtaining an initial audio file of a user call based on the first application; and denoising the initial audio file to obtain a first audio file that only includes the user call audio.

[0221] Fig.10 2 is a block diagram of an apparatus 200 for audio playback according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0222] Reference Fig.10 The audio playback device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0223] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0224] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0225] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0226] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0227] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0228] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0229] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0230] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0231] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0232] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0233] Fig.11 3 is a block diagram of a device 300 for audio playback according to an exemplary embodiment. For example, the device 300 may be provided as a server. Fig.11 The apparatus 300 includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332 for storing instructions executable by the processing component 322, such as an application. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute instructions to perform the above-mentioned screen display method.

[0234] The device 300 may also include a power supply component 326 configured to perform power management of the device 300, a wired or wireless network interface 350 configured to connect the device 300 to a network, and an input / output (I / O) interface 358. The device 300 may operate based on an operating system stored in the memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0235] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0236] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0237] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0238] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0239] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0240] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0241] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An audio playback method, characterized in that: include: In response to the first application making a voice call, enabling a second application, wherein the second application is used to provide call background sound for the first application; Acquire a first audio file based on the first application and acquire a second audio file based on the second application, the first audio file being an audio file of a user call, and the second audio file being an audio file of call background sound; Mixing the first audio file with the second audio file to obtain a target audio file; A voice over Internet Protocol (VoIP) call is performed based on the target audio file.

2. The audio playback method according to claim 1, characterized in that: The acquiring the second audio file based on the second application includes: Acquire the second audio file based on the audio data stream; or, The second audio file is obtained based on the first thread, and the first thread is used to send the second audio file to the recording thread; or The second audio file is obtained based on the second thread, and the second thread is used to send the second audio file to the recording thread and send other audio files to the audio processor, where the other audio files are audio files other than the second audio file.

3. The audio playback method according to claim 2, characterized in that: The acquiring the second audio file based on the audio data stream comprises: configuring a target identifier for the second audio file; Based on the target identifier, selecting a corresponding audio data stream and a sending path of the audio data stream; The audio data stream is sent to an audio processor based on a sending path of the audio stream to obtain the second audio file.

4. The audio playback method according to claim 2, characterized in that: The acquiring the second audio file based on the first thread includes: Creating a first thread and a first cache area, where the first cache area is an internal cache area of ​​the first thread; Selecting a specified audio file in the second application as the second audio file; Sending the second audio file to the first buffer area, so that the first thread carries the second audio file; Sending the first thread to the buffer area of ​​the recording thread; In the buffer area of ​​the recording thread, the second audio file is obtained.

5. The audio playback method according to claim 2, characterized in that: The obtaining the second audio file based on the second thread includes: Creating a second cache area, where the second cache area is an internal cache area of ​​the second thread; Selecting a specified audio file in the second application as the second audio file; Sending the second audio file to the second buffer area; Reusing the second thread to send the second audio file to the buffer area of ​​the recording thread, and deleting the second audio file in the second buffer area; In the buffer area of ​​the recording thread, the second audio file is obtained.

6. The audio playback method according to claim 1, characterized in that: The method of obtaining the target audio file includes any of the following methods: Mixing the first audio file and the second audio file in an audio processor to obtain the target audio file; The first audio file and the second audio file are mixed in the recording thread to obtain the target audio file.

7. The audio playback method according to claim 1, characterized in that: The acquiring the first audio file based on the first application includes: Acquire an initial audio file of the user's call based on the first application; The initial audio file is subjected to denoising processing to obtain a first audio file.

8. An audio playback device, characterized in that: include: A processing unit, configured to enable a second application in response to the first application making a voice call, wherein the second application is configured to provide a call background sound for the first application; The processing unit is further configured to obtain a first audio file based on the first application and obtain a second audio file based on the second application, wherein the first audio file is an audio file of a user call and the second audio file is an audio file of a call background sound; The processing unit is further used to mix the first audio file and the second audio file to obtain a target audio file; The processing unit is further configured to conduct a voice call based on Internet Protocol based on the target audio file.

9. An audio playback device, characterized in that: include: a memory for storing processor executable instructions; Wherein, the processor is configured to: execute the audio playback method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the audio playback method according to any one of claims 1 to 7.

Images