Audio processing method, system and device and storage medium

Abstract

The embodiment of the invention discloses an audio processing method, system and device and a storage medium, and aims at solving the problem that in an existing multi-screen collaborative scene, audio playing sounds of a sending end and a receiving end are the same. The method comprises the following steps: caching audio data corresponding to a plurality of applications running at a sending end; in response to a first operation of setting a sending end playing application, playing the cached audio data of the sending end playing application at the sending end; in response to a second operation of setting the playing application of the receiving end, sending the cached audio data of the playing application of the receiving end to the receiving end; wherein the sending end playing application is an application determined to play sound at the sending end in the multiple applications, and the receiving end playing application is a screen projection application determined to play sound at the receiving end in the multiple applications.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an audio processing method, system, device and storage medium. Background Art

[0002] At present, multi-device interconnection has a wide range of application scenarios, such as live broadcasts by game anchors, reverse game control, AR (Augmented Reality) split machines, user entertainment projection on large screens, and other scenarios. Taking mobile devices as an example, the mobile phone can be used as a host to project an application A running on the mobile phone to other devices (such as computers, monitors, TVs, etc.) for display. In this way, users can also control the projected application A through peripherals (such as Bluetooth-connected mouse and keyboard, or the computer's own mouse and keyboard), and can control application B on the mobile phone at the same time. However, the existing technology can only play system sounds when running multiple applications, that is, the sounds of all applications running on the mobile phone are mixed, and then played simultaneously at the sending and receiving ends, resulting in different applications displayed on the sending and receiving ends but the same sounds, and a poor user experience. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an audio processing method, system, device and storage medium to solve the problem that the audio playback sounds at the sending end and the receiving end are the same in the existing multi-screen collaboration scenario.

[0004] The embodiments of the present application also provide another audio processing method, device and storage medium to solve the problem that in the existing screen recording scenario, the sound of the screen recording application cannot be selected, which may cause the sound of other applications to be mixed.

[0005] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an audio processing method is provided, which is applied to a sending end of screen projection, comprising:

[0007] caching audio data corresponding to multiple applications running on the sending end;

[0008] In response to a first operation of setting a transmitting end playing application, playing the cached audio data of the transmitting end playing application at the transmitting end; wherein the transmitting end playing application is an application determined among the multiple applications to play sound at the transmitting end;

[0009] In response to the second operation of setting the receiving end playback application, the cached audio data of the receiving end playback application is sent to the receiving end; wherein the receiving end playback application is a screen projection application determined among the multiple applications to play sound at the receiving end.

[0010] In a second aspect, an audio processing system is provided, including a screen projection transmitting end and a receiving end, wherein:

[0011] The transmitting end is used to cache audio data corresponding to multiple applications running on the transmitting end; in response to a first operation of setting a transmitting end playback application, the cached audio data of the transmitting end playback application is played at the transmitting end; in response to a second operation of setting a receiving end playback application, the cached audio data of the receiving end playback application is sent to the receiving end; wherein the transmitting end playback application is an application determined among the multiple applications to play sound at the transmitting end, and the receiving end playback application is a screen projection application determined among the multiple applications to play sound at the receiving end;

[0012] The receiving end is used to play the audio data received by the receiving end playback application at the receiving end.

[0013] In a third aspect, an audio processing method is provided, which is applied to a terminal device, comprising:

[0014] caching audio data corresponding to multiple applications running on the terminal device;

[0015] In response to the fifth operation of setting the screen recording application, the cached audio data of the screen recording application is sent to the client for screen recording for storage; wherein the screen recording application is the application determined for screen recording among the multiple applications.

[0016] In a fourth aspect, an electronic device is provided, including:

[0017] processor;

[0018] a memory for storing instructions executable by the processor;

[0019] The processor is configured to execute the instructions to implement the method as described in the first aspect or the third aspect.

[0020] According to a fifth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method described in the first aspect or the third aspect.

[0021] The audio processing solution provided in the embodiment of the present application is to solve the problem that in a multi-screen collaboration scenario, the sending end of the projection screen mixes the sounds of all applications, and only the mixed system sounds can be played at the sending end and the receiving end. Before the audio of multiple applications running on the sending end is mixed, the audio data corresponding to the multiple applications are cached. According to user settings, the sounds of different applications can be dynamically played at the sending end and the receiving end, thereby realizing the projection sound separation function and greatly improving the user experience.

[0022] Another audio processing solution provided by an embodiment of the present application is to solve the problem that the sound of the screen recording application cannot be selected in the screen recording scenario, which may cause the sound of other applications to be mixed. Before the audio of multiple applications running on the terminal device is mixed, the audio data corresponding to multiple applications are cached. According to user settings, the cached audio data of the screen recording application can be sent to the client used for screen recording for storage, thereby realizing the function of selecting the sound of the screen recording application and greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A schematic diagram of an implementation environment applicable to an audio processing method provided by an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the functional architecture of a screen projection sending end provided for one embodiment of the present application;

[0026] Figure 3 A flowchart of an audio processing method provided for one embodiment of the present application;

[0027] Figure 4a A schematic diagram of a user setting a transmitting end and a receiving end to play an application provided in an embodiment of the present application;

[0028] Figure 4b A schematic diagram of sound separation between a transmitting end and a receiving end provided for an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of an audio processing system provided for one embodiment of the present application;

[0030] Figure 6 A flowchart of another audio processing method provided for one embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application.

[0033] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0034] As mentioned above, in existing multi-screen collaboration scenarios, there is a problem that the audio playback sounds at the sending end and the receiving end are the same.

[0035] With the development of game live broadcast and the diversification of entertainment scenes, users use mobile phones to cast screens in more and more scenarios, and derived functions such as extended screen casting and reverse control. Extended screen casting refers to casting application A running on the mobile phone to the remote end, and application B can be run on the mobile phone at the same time. The remote end can be a computer, monitor, TV and other devices. At present, mobile phones based on the Android operating system, referred to as Android phones, can achieve extended screen casting through the VirtualDisplay solution. Reverse control refers to controlling the application of the screen casting through peripherals, so that the sending and receiving ends of the screen casting can control different applications. For example, when casting the mobile phone to the PC (Personal Computer) end, the application A on the mobile phone can be controlled by the mouse and keyboard on the PC end; similarly, for example, when casting application A running on the mobile phone to a TV or monitor, the mouse, keyboard, handle or other peripherals can be connected via Bluetooth to control the projected application A. It can be understood that in the above-mentioned multi-screen collaborative scenario, the sending end of the screen casting refers to the mobile phone end, and the receiving end of the screen casting refers to the remote end (computer, monitor, TV, etc.).

[0036] The inventor discovered during the invention process that as mobile phones have more and more functions and stronger performance, mobile phone hostization is a trend. Hostization requires enriching the usage scenarios of mobile phones, and multi-screen collaboration is an important direction. Improving the user experience in multi-screen collaboration scenarios has become a top priority. At the receiving and sending ends of the screen projection, it is necessary to focus on improving the picture and sound control, as well as the control experience at both ends. Therefore, in the existing multi-screen collaboration scenarios, the problem of the same audio playback sound at the sending and receiving ends needs to be solved urgently. In addition, in the subsequent development direction of multi-screen collaboration, the receiving end can support two or more windows, and different applications can be displayed separately through different windows, which is convenient for user operation. The user can control which window plays the sound, which is also an important function. In addition, the control of the sound of different screen projection applications is also an important function.

[0037] Based on the problems and development trends in the above-mentioned multi-screen collaboration scenarios, the embodiments of the present application propose a technical concept of sound separation. Before the sounds of multiple applications are mixed, the audio data corresponding to multiple applications are cached, so that the sending end and the receiving end can play the sounds of different applications. Based on this technical concept, the embodiments of the present application provide an audio processing method. For multi-screen collaboration scenarios, users can select applications that need to play sounds at the sending end and the receiving end, and then perform secondary diversion of the audio data of multiple applications, send the cached audio data of the receiving end playing the application to the receiving end, and play the cached audio data of the sending end playing the application at the sending end.

[0038] The implementation environment of the audio processing method provided in the embodiment of the present application is as follows: Figure 1 As shown, it includes a screen projection transmitter 101 and a receiver 102, and data can be transmitted between the transmitter 101 and the receiver 102 through a wired network or a wireless network. The applicable hardware of the screen projection transmitter 101 includes a smart phone (such as an Android phone), or a tablet computer, a laptop computer, etc.; the applicable hardware of the screen projection receiver 102 includes a computer, a monitor, a TV, other smart phones, AR glasses, etc. The receiver 102 is connected to peripherals such as a keyboard, a mouse, a handle, etc., and can perform extended screen projection control.

[0039] Below, we take an Android phone as an example to introduce the functional architecture of the screen projection sender. Android phones can use the VirtualDisplay solution to achieve extended screen projection. Please refer to Figure 2 ,in:

[0040] In order to realize screen projection, you can install a screen projection tool APP on your Android phone. Based on user needs, you can also install various applications on your Android phone, such as game APPs, music APPs, video APPs, social APPs, etc. Figure 2In the figure, application A, application B, application C, etc. are used for schematic illustration. Each application can be used as a screen projection application. A screen projection application refers to an application that is projected to a receiving end. All applications are located in the application layer.

[0041] Screen projection tool APP: It is an application for realizing screen projection, providing functions such as wireless screen projection device search, code scanning screen projection, picture quality and audio settings, etc. In the embodiment of the present application, the sound separation and volume of each application can be controlled.

[0042] VirtualDisplay: Virtual display, screen projection transmits VirtualDisplay data to the receiving end for display. In the case of multi-window screen projection, the top application of each window stack can be obtained through VirtualDisplay.

[0043] MediaProjectionManager: Media image manager, through which the screen capture function is implemented. In the embodiment of the present application, multi-window screen projection can be supported.

[0044] AudioManager: Audio manager, through this service to control volume and other functions.

[0045] AudioService: Audio service, through which various basic functions of controlling audio are implemented.

[0046] AudioRecord: Audio recording. This interface is used to obtain the audio stream required for screen projection.

[0047] AudioPolicyService: Audio policy service, which is mainly responsible for audio policy-related, volume adjustment, device selection, audio channel selection and other functions.

[0048] AudioFlinger: Audio strategy execution, mainly responsible for audio stream device management, audio stream data processing and transmission, volume calculation, sampling, mixing, sound effects, etc.

[0049] AudioMixer: audio mixing, implements audio mixing operations, sets corresponding parameters for each channel of audio source data for mixing. In the embodiment of the present application, before audio mixing, the audio data of each application needs to be cached.

[0050] AudioTrack: audio track. Each application corresponds to an AudioTrack. In the implementation of this application, the audio stream of the application is obtained through AudioTrack.

[0051] MixerThread: Mixer Thread is the core part of audio output in the Android operating system. All audio needs to be mixed by MixerThread before being output to the audio device. In the embodiment of the present application, sound separation is achieved by controlling in MixerThread.

[0052] HAL: Hardware Abstraction Layer. The audio HAL associates AudioFlinger / AudioPolicyService with the hardware to implement audio playback and mic (microphone) reception.

[0053] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0054] Please refer to Figure 3 , which is a flowchart of an audio processing method provided by an embodiment of the present application, is applied to a sending end of screen projection, and the method may include:

[0055] S301. Cache the audio data corresponding to multiple applications running on the sending end of the screen projection.

[0056] In order to achieve screen projection, a screen projection tool APP can be pre-installed on the sending end of the screen projection. After the screen projection tool APP is installed, the client for screen projection can be run on the sending end, and the user can initiate the operation of projecting one or more applications on the user interface provided by the client. It can be understood that in the embodiment of the present application, the screen projection tool APP and the screen projection application are different applications. The screen projection tool APP is an application used to achieve screen projection, and the screen projection application refers to the application that is projected to the receiving end.

[0057] Specifically, during the screen projection process, the screen projection tool APP can obtain the Display managed by the operating system through the DisplayManagerService, and can list the application stack in the Display to obtain the applications running on the sending end, and then obtain the audio data corresponding to each application for caching.

[0058] In an optional implementation, the audio data includes an audio stream, and the step S301 of caching the audio data corresponding to the multiple applications running on the sending end may specifically include:

[0059] Through the application's AudioTrack, the audio streams in the AudioTrack corresponding to different applications are cached in a buffer associated with the application package name of the application before the system mixing process.

[0060] Each application corresponds to its own AudioTrack, which includes the corresponding application's own sound (ie, audio stream). Before AudioFlinger mixes the sound, the audio streams in the AudioTrack corresponding to multiple applications are written to the buffer associated with the application package name of the application.

[0061] In the multi-screen collaboration scenario, before the audio of multiple applications is mixed, the audio stream in the Audio Track corresponding to each application is buffered, so that users can select the application that needs to play sound at the sending and receiving ends. By secondary diversion of multiple audio streams, the sound separation between the receiving and sending ends can be achieved.

[0062] In an optional implementation, the audio data may further include volume data. In order to control the playback volume of each application, in addition to caching the audio streams of multiple applications, trackVolume (track volume variable) may also be cached. In step S201, the audio data corresponding to the multiple applications running on the sending end are cached, and the following may also be included:

[0063] The trackVolume in the AudioTrack corresponding to different applications is cached in the buffer associated with the application package name of the application before the system mixing process; wherein the trackVolume is used to control the volume of the application during playback through the saved volume data.

[0064] The AudioTrack corresponding to the application also includes the volume data corresponding to the window playing the sound, which is recorded in trackVolume, and the value range is generally 0 to 1.0. In addition to writing the audio stream in the AudioTrack corresponding to the application to the buffer associated with the application package name, the trackVolume in the AudioTrack corresponding to the application can also be written to the buffer associated with the application package name.

[0065] Specifically, the volume data may be set by the system or by the user. If the volume data is set by the system, it includes: setting the volume data by the operating system according to the user's habits or the sound environment when the application is playing.

[0066] There are two ways to assign volume data: one is that trackVolume is set by the user; the other is that trackVolume is set by the system. The operating system can make intelligent settings based on the user's common setting habits, or the external sound environment when the application is playing. For example, the foreground application sound is set to 0.7, and the background service application sound is set to 0.3.

[0067] In an embodiment of the present application, among the multiple applications running on the sending end, the application that plays sound on the sending end is called the sending end playback application, and the screen projection application that plays sound on the receiving end is called the receiving end playback application. The screen projection application refers to the application that is projected to the receiving end.

[0068] The AudioTrack corresponding to the application also includes displayId (display identifier), which is different on the sender and receiver. For example, the default display on the sender is displayId=0; when the receiver supports multiple windows, each window is a VirtualDisplay, and different VirtualDisplays have different displayIds. For example, the receiver displays two windows, displayId=1 and displayId=2.

[0069] The user can flexibly set the sending end playback application and the receiving end playback application. The user can set whether the application plays sound in the screen projection tool APP, and can also set the volume. Among them, audioPlayback is true for playback, and audioPlayback is false for non-playing; trackVolume saves the volume data of the application playback. The relevant setting information is saved in the AudioTrack corresponding to the application.

[0070] It can be understood that the application displayed on the sending end and with audioPlayback being true is the sending end playback application, and the screen projection application displayed on the receiving end and with audioPlayback being true is the receiving end playback application.

[0071] For example, please refer to Figure 4a , through user settings, the applications displayed on the sending end (displayId=0) include application A, application B and application C; the applications displayed on the receiving end (displayId=1 or 2) include application D, application E, application F and application G. Since the receiving end supports multiple windows, application D and application E are displayed on the receiving end screen projection interface 1 (displayId=1), and application F and application G are displayed on the receiving end screen projection interface 2 (displayId=2). Among application A, application B and application C displayed on the sending end, since the audioPlayback of application A is true, the application played on the sending end is application A; among application D, application E, application F and application G displayed on the receiving end, since the audioPlayback of application D, application E and application F is true, the application played on the receiving end is application D, application E and application F.

[0072] S302: In response to a first operation of setting a transmitting end playing application, the cached audio data of the transmitting end playing application is played at the transmitting end.

[0073] In an optional implementation, the transmitting end playback application may include at least two, and step S302 plays the cached audio data of the transmitting end playback application at the transmitting end, which may specifically include:

[0074] In the case that the transmitting end playback applications include at least two, at least two audio data corresponding to the at least two transmitting end playback applications are audio mixed and then played at the transmitting end.

[0075] S303: In response to the second operation of setting the receiving end playing application, the cached audio data of the receiving end playing application is sent to the receiving end.

[0076] In an optional implementation, the receiving end playback application may include at least two, and the buffered audio data of the receiving end playback application is sent to the receiving end in step S303, which may be implemented in any of the following two ways:

[0077] Method 1: Mixing and playing at the receiving end, that is, when the receiving end includes at least two playing applications, the sending end mixes at least two audio data corresponding to the at least two receiving end playing applications and sends them to the receiving end; in this way, the mixed sound of at least two receiving end playing applications will be played at the receiving end.

[0078] Method 2: Select playback at the receiving end, that is, when the receiving end includes at least two playback applications, the sending end directly sends at least two audio data corresponding to the at least two receiving end playback applications to the receiving end; in this way, the user can further choose at the receiving end whether to play the sound of a certain application or the mixed sound of some applications in the at least two receiving end playback applications, or to play the mixed sound of at least two receiving end playback applications, and the receiving end performs corresponding sound playback based on the user settings.

[0079] Specifically, the sending end may send the audio data corresponding to the playing application of the receiving end to the receiving end through the LocalSocket communication mode.

[0080] Since the user may set multiple sending and receiving end playback applications, resampling and mixing are performed in the MixerThread of AudioFlinger. The sounds of at least two applications can be mixed together. The class that mixes them together is called AudioMixer. Next, AudioMixer is used to distinguish displayId for sound distribution, thereby realizing screen projection sound separation, and then playing the sound from the sending and receiving ends respectively.

[0081] For example, please refer to Figure 4b The audio data of application A, application D, application E and application F are resampled and mixed in MixerThread, and the sounds of the four applications are mixed together. The class that mixes them together is AudioMixer. AudioMixer distinguishes displayId for sound distribution, and plays the audio data of application A (displayId=0) at the sending end, and the audio data of application D, application E and application F are sent to the receiving end. In this example, since the trackVolume of application F is 0, the sounds of application D and application E are played by the receiving end.

[0082] In the case of supporting multi-window screen projection, each window can play the sound of the application, which may cause sound confusion. In the embodiment of the present application, the following two methods can be used to control which window plays the sound. The first method is that the user specifies which window plays the sound; the second method is that the window that the user operates plays the sound. Furthermore, in the case where each window includes at least two screen projection applications, it is possible to further control which application's sound is played, such as identifying the top application in the VirtualDisplay corresponding to the window, and then playing the sound of the top application; of course, it is also possible to control the sound of all applications in the window, or the sound of a certain application, or the sound of some applications. When multiple windows are projected, the user experience can be improved through window audio control.

[0083] In an optional implementation, the receiving end may support at least two windows, and different windows display different applications respectively. In this case, the second operation of setting the receiving end to play the application may specifically include an operation of pre-specifying a window for playing sound, or an operation of selecting a window currently playing sound; accordingly, sending the cached audio data of the receiving end playing application to the receiving end in step S303 may specifically include:

[0084] In a case where the receiving end supports at least two windows, determining a target window for playing the sound from the at least two windows based on the second operation;

[0085] The target screen projection application in the target window is used as the receiving end playback application, and the audio data corresponding to the target screen projection application in the target window is sent to the receiving end. The target screen projection application is the top application in the stack, or all applications, or a specified application in the target window; in this way, the sound of the target screen projection application in the target window will be played at the receiving end.

[0086] When casting multiple windows, there is also a need to play the sound of an application together. For example, if a certain application is cast to multiple receiving devices, each receiving device needs to play the sound of the same casting application. Therefore, in multi-window casting, background applications or shared applications need to be identified, and the above applications are also listed in the Display playback sound setting list to allow users to select.

[0087] In an optional implementation, while achieving screen projection sound separation, the volume of different applications can also be controlled. For example, when the user is projecting the screen, he chooses to play the sound of the game APP and the sound of the music player at the same time. The user can flexibly adjust the volume of the above two applications according to his own preferences to improve the user experience. When caching the audio data corresponding to the application, the volume data set by the user is saved in trackVolume, and then the volume of the application when playing can be controlled by modifying trackVolume. Specifically, the user can also adjust the playback volume of any sending end playback application or the sending end playback application at any time according to actual needs. Accordingly, the method also includes:

[0088] In response to a third operation of adjusting the playback volume of any transmitter playback application, modifying the volume data of trackVolume in the buffer associated with the corresponding transmitter playback application;

[0089] In response to a fourth operation of adjusting the playback volume of any receiving-end playback application, the volume data of trackVolume in the buffer associated with the corresponding receiving-end playback application is modified.

[0090] In an optional implementation, the audio processing method may further include:

[0091] When the receiving end has a voice input device, the sending end receives the audio data of any screen projection application transmitted by the receiving end as the input audio of the corresponding screen projection application; wherein, the audio data of the screen projection application is obtained by the receiving end through the voice input device.

[0092] In an embodiment of the present application, when the receiving end has a mic, the receiving end can be supported to make calls directly, and the audio data received by the mic can be transmitted to the sending end as the input audio of the screen projection application, thereby realizing multiple audio transmission methods in the multi-screen collaborative scenario and enriching the sound separation scenario.

[0093] In an optional implementation, in a multi-screen collaboration scenario, there may be a screen recording scenario at the same time, that is, a user initiates a screen recording operation for a certain application at the sending end. In order to realize the function of selecting the sound of the screen recording application, the audio processing method may also include:

[0094] In response to the fifth operation of setting the screen recording application, the cached audio data of the screen recording application is sent to the client for screen recording for storage; wherein the screen recording application is an application determined for screen recording from multiple applications running on the sending end.

[0095] In order to realize screen recording, it is necessary to pre-install the screen recording tool APP on the sending end. After the screen recording tool APP is installed, the client for screen recording can be run on the sending end, and the user can initiate the operation of recording the screen of a certain application on the user interface provided by the client. It can be understood that in the embodiment of the present application, the screen recording tool APP and the screen recording application are different applications. The screen recording tool APP is an application for realizing screen recording, and the screen recording application is the application being recorded.

[0096] The audio processing method provided in the embodiment of the present application is to solve the problem that in a multi-screen collaborative scenario, the sending end of the projection screen mixes the sounds of all applications, and only the mixed system sounds can be played at the sending end and the receiving end. Before the audio of multiple applications running on the sending end is mixed, the audio data corresponding to the multiple applications are cached. According to user settings, the sounds of different applications can be dynamically played at the sending end and the receiving end, thereby realizing the projection sound separation function and greatly improving the user experience.

[0097] Furthermore, after achieving screen projection and sound separation, the volume of different applications can be controlled; and, when the receiving end supports multi-window screen projection, it can be controlled which windows and the sounds of which applications in the windows are played; the function of selectable sound of screen recording applications can also be realized.

[0098] The audio processing method provided in the embodiment of the present application can enhance the user experience of smartphones in scenarios such as extended screen projection, multi-screen interaction, AR split machines, screen recording, etc., truly achieve multi-person entertainment and multi-scene entertainment, and realize the hostization of smartphones.

[0099] In addition, Figure 3 Corresponding to the audio processing method shown, the embodiment of the present application also provides an audio processing system, Figure 5 is a structural diagram of an audio processing system 500 provided in an embodiment of the present application, including a screen projection transmitting end 501 and a receiving end 502, wherein:

[0100] The transmitting end 501 is used to cache the audio data corresponding to the multiple applications running on the transmitting end 501; in response to the first operation of setting the transmitting end to play the application, the cached audio data of the transmitting end to play the application is played on the transmitting end 501; in response to the second operation of setting the receiving end to play the application, the cached audio data of the receiving end to play the application is sent to the receiving end 502; wherein the transmitting end playing application is an application determined among the multiple applications to play the sound on the transmitting end 501, and the receiving end playing application is a screen projection application determined among the multiple applications to play the sound on the receiving end 502;

[0101] The receiving end 502 is used to play the audio data received by the receiving end playing application at the receiving end.

[0102] In an optional implementation, the transmitter 501 is specifically configured to cache the audio streams in the AudioTrack corresponding to different applications in a buffer register associated with the application package name of the application before the system mixing process through the AudioTrack of the application.

[0103] The sending end 501 is also used to cache the track volume variables in the AudioTrack corresponding to different applications in a buffer register associated with the application package name of the application before the system mixing processing; wherein the track volume variable is used to control the volume of the application during playback through the saved volume data, and the volume data is set by the system or the user.

[0104] The transmitting end 501 is also used to modify the volume data of the track volume variable in the register associated with the corresponding transmitting-end playback application in response to the third operation of adjusting the playback volume of any transmitting-end playback application; and to modify the volume data of the track volume variable in the register associated with the corresponding receiving-end playback application in response to the fourth operation of adjusting the playback volume of any receiving-end playback application.

[0105] In an optional implementation, the transmitter 501 is specifically configured to, when the transmitter includes at least two playback applications, perform audio mixing on at least two audio data corresponding to the at least two playback applications and then play the audio data on the transmitter.

[0106] In an optional implementation, the sending end 501 is specifically used to, when the receiving end playback application includes at least two, perform audio mixing on at least two audio data corresponding to the at least two receiving end playback applications, and then send the audio mixing to the receiving end for playback at the receiving end; or, directly send at least two audio data corresponding to the at least two receiving end playback applications to the receiving end for selection and playback at the receiving end.

[0107] In an optional implementation, the sending end 501 is specifically used to determine a target window for playing sound from at least two windows based on a second operation when the receiving end supports at least two windows; the second operation includes an operation of pre-specifying a window for playing sound, or an operation of selecting a window currently playing sound; the target projection application in the target window is used as the receiving end playback application, and the audio data corresponding to the target projection application in the target window is sent to the receiving end for playback at the receiving end, wherein the target projection application is the top application in the stack, or all applications, or a specified application in the target window.

[0108] The sending end 501 is specifically used to send the audio data corresponding to the receiving end playback application to the receiving end through the local socket communication mode.

[0109] In an optional implementation, the sending end 501 is also used to receive the audio data of any screen projection application transmitted by the receiving end 502 as the input audio of the corresponding screen projection application when the receiving end has a voice input device; wherein the audio data of the screen projection application is obtained by the receiving end through the voice input device.

[0110] In an optional implementation, the sending end 501 is also used to respond to the fifth operation of setting the screen recording application, and send the cached audio data of the screen recording application to the client used for screen recording for storage; wherein the screen recording application is an application for screen recording determined among multiple applications running on the sending end.

[0111] The audio processing system provided in the embodiment of the present application can dynamically play the sounds of different applications at the sending and receiving ends according to user settings, realize the screen projection sound separation function, and can also control the volume of different applications. When the receiving end supports multi-window screen projection, it can control which windows and which applications in the windows play the sounds, thereby greatly improving the user experience.

[0112] Obviously, the transmitting end 501 in the audio processing system of the embodiment of the present application can be used as the above Figure 3 The execution subject of the audio processing method shown in FIG. Figure 3 The functions realized are as follows. Since the principles are the same, they will not be described here.

[0113] During the invention process, the inventor discovered that in existing screen recording scenarios, the sound of the screen recording application cannot be selected, which may cause the sound of other applications to be mixed. The above-mentioned technical concept of sound separation can also be used to realize the function of selecting the sound of the screen recording application.

[0114] Based on the same technical concept, the present application embodiment provides another audio processing method, such as Figure 6 As shown, it is applied to a terminal device and includes the following steps:

[0115] S601: Cache audio data corresponding to multiple applications running on a terminal device.

[0116] In order to realize screen recording, a screen recording tool APP can be pre-installed on the terminal device. After the screen recording tool APP is installed, the client for screen recording can be run on the terminal device, and the user can initiate the operation of recording a certain application on the user interface provided by the client. It can be understood that in the embodiment of the present application, the screen recording tool APP and the screen recording application are different applications. The screen recording tool APP is an application for realizing screen recording, and the screen recording application is the application to be recorded. The terminal device can be a smart phone (such as Android), a tablet computer, a laptop computer, etc.

[0117] For multiple applications running on the terminal device, the audio streams in the AudioTrack corresponding to different applications can be cached in the buffer associated with the application package name of the application before the system mixing processing through the application's AudioTrack. The trackVolume in the AudioTrack corresponding to different applications can also be cached in the buffer associated with the application package name of the application before the system mixing processing.

[0118] S602. In response to the fifth operation of setting a screen recording application, the cached audio data of the screen recording application is sent to a client for screen recording for storage, wherein the screen recording application is an application determined for screen recording from among multiple applications running on the terminal device.

[0119] Another audio processing method provided in an embodiment of the present application is to solve the problem that the sound of the screen recording application cannot be selected in the screen recording scenario, which may cause the sound of other applications to be mixed. Before the audio of multiple applications running on the terminal device is mixed, the audio data corresponding to the multiple applications are cached. According to the user settings, the cached audio data of the screen recording application can be sent to the client used for screen recording for storage, thereby realizing the function of selecting the sound of the screen recording application and greatly improving the user experience.

[0120] Alternatively, if Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned audio processing method are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0121] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0122] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0123] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0124] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned audio processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0125] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0126] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned audio processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0127] The embodiments of the present application can be applied to a variety of multi-device collaboration or interconnection scenarios: including: collaborative interconnection between mobile phones and laptops / tablet computers; collaborative interconnection between mobile phones and AR glasses or wearable devices; collaborative interconnection between mobile terminals and smart TVs / displays; collaborative interconnection between mobile phones, or tablet computers and car entertainment systems; collaborative interconnection between mobile terminals and smart conference systems, etc. . Thereby meeting the needs of users in various scenarios such as smart homes, smart offices, and smart travel. For example, when extending the screen projection, any electronic device is used as the screen projection transmitter, and the computer, TV, display or other Android device is used as the receiving end, and the connection of peripherals (such as keyboards, mice, handles, etc.) can be used for extended screen projection control. When the AR split machine is projecting, the AR glasses are connected to the Android device for 3D projection, and the mobile phone runs local applications. When interacting with multiple screens, the Android device realizes multi-screen interaction and performs sound control in the multi-window interface. When recording the screen, sound separation technology can also be used to realize the screen recording sound selection function.

[0128] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0130] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio processing method, characterized in that: The sending end used for screen projection includes: caching audio data corresponding to multiple applications running on the sending end; In response to a first operation of setting a transmitting end playing application, playing the cached audio data of the transmitting end playing application at the transmitting end; wherein the transmitting end playing application is an application determined among the multiple applications to play sound at the transmitting end; In response to the second operation of setting the receiving end playback application, the cached audio data of the receiving end playback application is sent to the receiving end; wherein the receiving end playback application is a screen projection application determined among the multiple applications to play sound at the receiving end.

2. The method according to claim 1, characterized in that The caching of audio data corresponding to the multiple applications running on the sending end includes: Through the audio track AudioTrack of the application, the audio streams in the audio tracks corresponding to different applications are cached in a buffer register associated with the application package name of the application program before the system mixing process.

3. The method according to claim 2, characterized in that The step of caching the audio data corresponding to the multiple applications running on the transmitting end further includes: The track volume variables in the audio tracks corresponding to different applications are cached in a buffer register associated with the application package name of the application before system mixing processing; wherein the track volume variables are used to control the volume of the application during playback through the saved volume data, and the volume data is set by the system or the user.

4. The method according to claim 3, characterized in that The volume data is set by the system, including: setting the volume data by the operating system according to user habits or the sound environment when the application is played.

5. The method according to claim 3, characterized in that: The method further comprises: In response to a third operation of adjusting the playback volume of any transmitter playback application, modifying the volume data of the track volume variable in the buffer register associated with the corresponding transmitter playback application; In response to a fourth operation of adjusting the playback volume of any receiving-end playback application, the volume data of the track volume variable in the buffer register associated with the corresponding receiving-end playback application is modified.

6. The method according to claim 1, characterized in that The step of playing the cached audio data of the transmitting end playing application at the transmitting end includes: In the case that the transmitting end playback applications include at least two, at least two audio data corresponding to the at least two transmitting end playback applications are audio mixed and then played at the transmitting end.

7. The method according to claim 1, characterized in that The step of sending the cached audio data of the receiving end playback application to the receiving end includes: In the case where the receiving end playback applications include at least two, at least two audio data corresponding to the at least two receiving end playback applications are mixed and then sent to the receiving end for playback at the receiving end; or, at least two audio data corresponding to the at least two receiving end playback applications are directly sent to the receiving end for selection and playback at the receiving end.

8. The method according to claim 1, characterized in that: In response to the second operation of setting the receiving end playing application, sending the cached audio data of the receiving end playing application to the receiving end includes: In the case where the receiving end supports at least two windows, determining a target window for playing the sound from the at least two windows based on the second operation; the second operation includes an operation of pre-designating a window for playing the sound, or an operation of selecting a window currently playing the sound; The target screen projection application in the target window is used as the receiving end playback application, and the audio data corresponding to the target screen projection application in the target window is sent to the receiving end for playback at the receiving end. The target screen projection application is the top application in the target window, or all applications, or a specified application.

9. The method according to claim 1, 7 or 8, characterized in that: The step of sending the cached audio data of the receiving end playback application to the receiving end includes: The audio data corresponding to the receiving end playback application is sent to the receiving end through a local socket communication method.

10. The method according to claim 1, characterized in that The method further comprises: In the case where the receiving end has a voice input device, the sending end receives the audio data of any screen projection application transmitted by the receiving end as the input audio of the corresponding screen projection application; wherein the audio data of the screen projection application is obtained by the receiving end through the voice input device.

11. The method according to claim 1, characterized in that Also includes: In response to the fifth operation of setting the screen recording application, the cached audio data of the screen recording application is sent to the client for screen recording for storage; wherein the screen recording application is the application determined for screen recording among the multiple applications.

12. An audio processing system, characterized in that: It includes the sending and receiving ends of screen projection, including: The transmitting end is used to cache audio data corresponding to multiple applications running on the transmitting end; in response to a first operation of setting a transmitting end playback application, the cached audio data of the transmitting end playback application is played at the transmitting end; in response to a second operation of setting a receiving end playback application, the cached audio data of the receiving end playback application is sent to the receiving end; wherein the transmitting end playback application is an application determined among the multiple applications to play sound at the transmitting end, and the receiving end playback application is a screen projection application determined among the multiple applications to play sound at the receiving end; The receiving end is used to play the audio data received by the receiving end playback application at the receiving end.

13. An audio processing method, characterized in that: Applied to terminal equipment, including: caching audio data corresponding to multiple applications running on the terminal device; In response to the fifth operation of setting the screen recording application, the cached audio data of the screen recording application is sent to the client for screen recording for storage; wherein the screen recording application is the application determined for screen recording among the multiple applications.

14. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 11, or to implement the method according to claim 13.

15. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 11, or to implement the method as claimed in claim 13.

Images