Audio playing method, electronic equipment and storage medium

Abstract

The invention discloses an audio playing method, electronic equipment and a storage medium, and relates to the technical field of terminals. The method comprises the following steps: starting a plurality of applications and carrying out audio playing; determining a watching area when the user watches the screen of the electronic equipment; performing a first optimization operation on an audio stream of a first application corresponding to the gazing area, and performing a second optimization operation on audio streams of other applications except the first application in the plurality of applications, the first optimization operation comprising one or more of the following items: adjusting a loudspeaker playing the audio stream, increasing the volume, and performing optimization by using a sound effect optimization algorithm; the second optimization operation comprises one or more of the following items: adjusting a loudspeaker playing the audio stream, reducing the volume and stopping using a sound effect optimization algorithm; and playing the optimized audio streams of the plurality of applications. According to the method, a user attention selection mechanism is introduced, and the user does not need to carry out additional adjustment operation, so that the concentration degree and the experience quality of the user are not influenced, and the intelligence during audio adjustment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal, and in particular, to an audio playing method, an electronic device and a storage medium. BACKGROUND

[0002] At present, on electronic devices such as mobile phones and tablet computers, some applications in use may cause the electronic device to play audio or process audio. These applications include, but are not limited to, telephone applications, game applications, music playing applications, video playing applications, voice recognition applications, etc. Multiple speakers can be provided on the electronic device to enable the electronic device to support multiple applications running in different windows and playing audio at the same time, and to enable the electronic device to realize a stereo sound field effect of multiple sound sources by using multiple speakers when multiple sound sources are included in a single application.

[0003] When the applications concurrently play audio, the electronic device can only adjust the volume of the audio streams of the multiple applications by type. When the audio streams of the multiple applications are of the same type, the electronic device cannot adjust the audio stream of a single application. When the audio streams of the multiple applications are of different types, the electronic device needs to adjust the audio streams respectively. When a video played by a video application includes multiple sound sources, the electronic device can only adjust the volume of the audio stream of the application as a whole and cannot adjust the audio stream of a single sound source. In summary, the current audio playing method has poor intelligence and is inconvenient for a user to manage the audio streams. SUMMARY

[0004] To solve the above problems, the present application provides an audio playing method, an electronic device and a storage medium, which improves the intelligence of audio adjustment and facilitates a user to manage the audio streams.

[0005] In a first aspect, the present application provides an audio playing method, which includes: starting multiple applications to concurrently play audio; determining a gaze area when a user gazes at a screen of an electronic device; performing a first optimization operation on an audio stream of a first application corresponding to the gaze area and performing a second optimization operation on audio streams of other applications in the multiple applications except the first application, wherein the first optimization operation includes one or more of the following: adjusting a speaker playing the audio stream, increasing the volume, and optimizing using an audio effect optimization algorithm; the second optimization operation includes one or more of the following: adjusting a speaker playing the audio stream, decreasing the volume, and stopping using the audio effect optimization algorithm; and playing the optimized audio streams of the multiple applications.

[0006] The implementation introduces a user attention selection mechanism, that is, the gaze area when the user gazes at the screen of the electronic device is determined, the application currently focused by the user is determined, and different optimization operations are automatically performed on the application focused by the user and the application not focused by the user, without the need for the user to manually perform additional adjustment operations, so as to not affect the concentration and quality of experience of the user. Moreover, the user can more quickly and conveniently manage the audio streams of multiple applications, and the intelligence in audio adjustment is improved.

[0007] In a possible implementation, the gaze area when the user gazes at the screen of the electronic device is determined, and specifically includes: obtaining an image including the eyes of the user; detecting eye features in the image; and when it is determined according to the eye features that the user gazes at a first area of the screen for more than a first length of time, determining the first area as the gaze area.

[0008] In this implementation, the gaze area can be determined by means of the existing front camera of the electronic device, without the need for using other additional hardware, thereby reducing the cost.

[0009] In a possible implementation, the first optimization operation is performed on the audio stream of the first application corresponding to the gaze area, and specifically includes: determining the first application corresponding to the gaze area; determining the audio stream of the first application; and performing the first optimization operation on the audio stream of the first application.

[0010] In a possible implementation, the first application corresponding to the gaze area is determined, and specifically includes: determining, according to current running states of multiple applications, display areas occupied by each application of the multiple applications on the screen; and matching the gaze area with the display areas occupied by each application on the screen to determine the first application.

[0011] In a possible implementation, the gaze area is matched with the display areas occupied by each application on the screen to determine the first application, and specifically includes: when the gaze area includes a first display area, or the first display area includes the gaze area, or an overlap ratio of the gaze area and the first display area is greater than a preset ratio, determining the application corresponding to the first display area as the first application.

[0012] The preset ratio can be greater than 50%, and thus a unique first application can be determined.

[0013] In a possible implementation, the first optimization operation includes adjusting a loudspeaker playing the audio stream, and the first optimization operation performed on the audio stream of the first application specifically includes: determining a loudspeaker playing the audio stream of the first application, and determining a newly added loudspeaker playing the audio stream of the first application according to a correspondence between the display area of the first application and a selection priority of the loudspeaker.

[0014] By increasing the number of loudspeakers playing the audio stream of the first application, the audio playing effect of the first application can be improved.

[0015] In a possible implementation, the first optimization operation comprises adjusting the loudspeakers playing the audio stream and optimizing using a sound effect optimization algorithm, and the first optimization operation on the audio stream of the first application specifically comprises:

[0016] When the first application is a video playing application, the characteristic parameters of each sound source object in the video are determined, the characteristic parameters comprising one or more of the following: the type of the sound source object, the number of the sound source object, the position of the sound source object, and the distance information between the sound source objects; the first sound source object to which the user gazes is determined according to the gaze area, the first sound source object being a sound-emitting sound source object; the audio stream of the first application is optimized using a sound effect optimization algorithm and the characteristic parameters of the first sound source object, the sound effect optimization algorithm being used to replay the first sound source object to improve the matching degree between the position of the replayed sound in one or more loudspeakers and the position of the first sound source object in the video, and to improve the accuracy of the audio spatial position of the first sound source object and to enhance the sound field effect.

[0017] In this implementation, when the first application to which the user gazes is a video playing application and the video played by the video playing application includes multiple sound source objects, the first sound source object to which the user gazes can be further determined, and the spatial sound field effect of the first sound source object can be enhanced, thereby improving the intelligence in audio playing and management and improving the listening experience of the user.

[0018] In a possible implementation, the first optimization operation on the audio stream of the first application further comprises: optimizing the audio stream of the first application using a sound effect optimization algorithm and the characteristic parameters of other sound source objects, the sound effect optimization algorithm being used to replay the other sound source objects to reduce the matching degree between the position of the replayed sound in one or more loudspeakers and the position of the other sound source objects in the video, and to reduce the accuracy of the audio spatial position of the other sound source objects and to reduce the sound field effect of the other sound source objects, the other sound source objects being all sound source objects in the video except the first sound source object.

[0019] In this implementation, when the first application to which the user gazes is a video playing application and the video played by the video playing application includes multiple sound source objects, the spatial sound field effect of the other sound source objects to which the user does not gaze can be reduced, thereby improving the intelligence in audio playing and management and further improving the listening experience of the user. In addition, the calculation overhead in optimizing the spatial sound field effect of the other sound source objects can be reduced, thereby reducing the power consumption of the device.

[0020] In a possible implementation, the method further includes: mixing the audio stream of the first application after the first optimization operation and the audio stream of the other applications after the second optimization operation, and creating multi-channel audio data through a time division multiplexing (TDM) interface; and playing the audio data of each channel by a corresponding speaker.

[0021] In a second aspect, the present application provides an audio playing method, which includes: starting a video playing application to play a video and audio corresponding to the video; determining characteristic parameters of each sound source object in the video, the characteristic parameters including a display area of the sound source object and one or more of the following: a type of the sound source object, a number of the sound source object, and distance information between the sound source objects; determining a gaze area when a user gazes at a screen of the electronic device; determining a first sound source object gazed at by the user according to the gaze area and the display area of each sound source object, the first sound source object being a sound-emitting sound source object; enhancing a spatial sound field effect of the first sound source object by using an audio effect optimization algorithm and the characteristic parameters of the first sound source object, and reducing spatial sound field effects of other sound source objects except the first sound source object by using the audio effect optimization algorithm and the characteristic parameters of the other sound source objects; and playing an audio stream of the video after optimization.

[0022] In this implementation, when the video played by the video playing software includes multiple sound source objects, the sound source object currently focused on by the user can be determined according to the gaze area when the user gazes at the screen of the electronic device, the spatial sound field effect of the sound source object focused on by the user is enhanced, and the spatial sound field effects of other sound source objects not focused on by the user are reduced at the same time, without the need for the user to perform additional adjustment operations or settings, so that the concentration and quality of experience of the user are not reduced, and the intelligence in audio adjustment is improved, thereby improving the listening experience of the user.

[0023] In a possible implementation, the enhancing of the spatial sound field effect of the first sound source object by using the audio effect optimization algorithm and the characteristic parameters of the first sound source object includes: improving a matching degree between a position of the first sound source object in the video and a position of a sound played in one or more speakers by using the audio effect optimization algorithm and the characteristic parameters of the first sound source object; and improving an accuracy of an audio spatial position of the first sound source object and enhancing a sound field effect of the first sound source object by using the audio effect optimization algorithm and the characteristic parameters of the first sound source object.

[0024] In this implementation, the spatial sound field effect of the first sound source object focused on by the user is enhanced, the intelligence in audio playing and management is improved, and the listening experience of the user is improved.

[0025] In one possible implementation, the spatial sound field effect of other sound source objects (excluding the first sound source object) is reduced by using an audio effect optimization algorithm and feature parameters of other sound source objects. Specifically, this includes: reducing the matching degree between the location of the sound reproduced in one or more speakers and the location of other sound source objects in the video by using an audio effect optimization algorithm and feature parameters of other sound source objects; reducing the accuracy of the audio spatial location of other sound source objects by using an audio effect optimization algorithm and feature parameters of other sound source objects; and reducing the sound field effect of other sound source objects.

[0026] This implementation reduces the spatial sound field effect of other sound source objects not being observed by the user, improves the intelligence of audio playback and management, and further enhances the user's listening experience. Furthermore, it reduces the computational overhead of optimizing the spatial sound field effect of other sound source objects, thereby reducing device power consumption.

[0027] In one possible implementation, determining the gaze region when a user gazes at the screen of an electronic device specifically includes: acquiring an image including the user's eyes; detecting eye features in the image; and determining the second region as the gaze region when the user gazes at the screen for more than a second time length based on the eye features.

[0028] In one possible implementation, determining the gaze region when a user gazes at the screen of an electronic device specifically includes: acquiring an image including the user's eyes; detecting eye features in the image; and when, based on the eye features, it is determined that the user has gazed at the screen for more than a third time period, and continues to gaze at a second region of the screen for more than a fourth time period, defining the second region as the gaze region.

[0029] In one possible implementation, the area where each sound source object is located includes a second display area. Based on the gaze area and the position of each sound source object, the first sound source object that the user is looking at is determined. Specifically, when the gaze area includes the second display area, or the second display area includes the gaze area, or the overlap ratio between the gaze area and the second display area is greater than a preset ratio, the sound source object corresponding to the second display area is determined as the first sound source object.

[0030] Thirdly, this application provides an electronic device including at least one processor, which is configured to execute computer programs or instructions to implement the audio playback method provided by the first aspect and any implementation thereof, and / or to implement the audio playback method provided by the second aspect and any implementation thereof.

[0031] Fourthly, this application also provides a storage medium storing a computer program, which, when executed, implements the audio playback method provided by the first aspect and any implementation thereof, and / or implements the audio playback method provided by the second aspect and any implementation thereof. Attached Figure Description

[0032] Figure 1 A schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0033] Figure 2 A schematic diagram of the volume adjustment interface provided in an embodiment of this application;

[0034] Figure 3 A flowchart illustrating an audio playback method provided in this application embodiment;

[0035] Figure 4 Scenario illustration provided for embodiments of this application Figure 1 ;

[0036] Figure 5 Scenario illustration provided for embodiments of this application Figure 2 ;

[0037] Figure 6 Scenario illustration provided for embodiments of this application Figure 3 ;

[0038] Figure 7 A flowchart illustrating another audio playback method provided in this application embodiment;

[0039] Figure 8 Scenario illustration provided for embodiments of this application Figure 4 ;

[0040] Figure 9 A flowchart illustrating yet another audio playback method provided in this application embodiment;

[0041] Figure 10 Scenario illustration provided for embodiments of this application Figure 5 ;

[0042] Figure 11 A flowchart illustrating another audio playback method provided in this application embodiment;

[0043] Figure 12 Scenario illustration provided for embodiments of this application Figure 6 ;

[0044] Figure 13 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the solution of this application, the application scenario of the technical solution of this application will be described first below.

[0046] The software structure of the electronic device using the technical solution of this application will be described first below.

[0047] See Figure 1 This figure is a schematic diagram of the software structure of the electronic device provided in an embodiment of this application.

[0048] It should be understood that different electronic devices can have different software systems. Optionally, the software system of an electronic device can adopt different architectures such as layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, for example, the Android system with a layered architecture, and the Harmony OS system with a microkernel architecture.

[0049] This application uses a layered architecture as an example to illustrate the software structure of an electronic device.

[0050] The application layer can include a series of application packages. Some applications (Apps) may cause the electronic device to play or process audio during use. These applications include, but are not limited to, calling applications, game applications, music playback applications, video playback applications, and voice recognition applications.

[0051] Electronic devices can simultaneously play audio from different applications based on an audio system. In one possible implementation, the external playback interface of this audio system is an audio track. Each application creates a corresponding audio track at the application framework layer, and each audio track can further call the audio system engine (Audio Flinger). When an application outputs an audio stream, the electronic device can determine the audio stream type based on the usage of the audio stream and output the decoded audio stream to the Audio Flinger through the corresponding audio track of the application. The Audio Flinger can load the volume corresponding to that audio stream type, and the phone's speaker can play the application's audio based on the volume loaded by the Audio Flinger.

[0052] In one possible implementation scenario, multiple different apps on an electronic device may simultaneously output their respective audio streams. Correspondingly, the phone's audio system can also play audio based on the multiple audio streams output by the different applications.

[0053] The audio processing algorithm in the application framework layer outputs the processed audio data to the Audio Hardware Abstraction Layer (Audio HAL). The Audio HAL can then determine the output audio hardware device based on the received audio data. Furthermore, the Audio HAL can access the audio driver corresponding to that audio hardware device.

[0054] The audio mixer module of the audio digital signal processor (ADSP) of electronic devices can mix multiple different audio streams, drive the smart power amplifier (Smart PA) based on the mixed audio data, and drive the corresponding speaker to play the audio.

[0055] A loudspeaker, also known as a "speaker," is used to convert audio electrical signals into sound signals. Electronic devices can use loudspeakers to play audio aloud. This application does not limit the specific number of loudspeakers installed on an electronic device. In some embodiments, multiple loudspeakers can be installed on an electronic device to achieve a stereo effect in the audio playback, thus improving the user experience.

[0056] In one possible implementation, the user can adjust the playback volume of the electronic device while it is playing audio. It should be understood that in this system, volume adjustment is primarily based on the audio stream type.

[0057] For example, a user can adjust the volume of different types of audio streams separately. Specifically, if an electronic device plays multiple different types of audio streams simultaneously, the user can adjust the volume of one type of audio stream without affecting the volume of the other types of audio streams.

[0058] To illustrate, audio streams can currently include the following different types: call stream, system stream, ringtone stream, media stream, alert stream, notification stream, and keystroke stream.

[0059] Different types of audio streams can correspond to a variety of possible use cases for electronic devices, or include a variety of possible sound sources. In other words, each type of audio stream can originate from multiple different applications, meaning that the sound output by multiple different applications can correspond to one type of audio stream.

[0060] For example, the audio output of applications such as music apps, video apps, and game apps is classified as "media streams"; when a user answers a phone call, the voice of the other user played through the phone's earpiece can be classified as "call streams," and so on.

[0061] See Figure 2 The figure is a schematic diagram of the volume adjustment interface provided in an embodiment of this application.

[0062] Users can access the volume adjustment interface through the "Sound and Vibration" option in the system settings of their electronic devices.

[0063] The volume adjustment interface 10 illustrates the interface for adjusting the volume of different types of audio streams.

[0064] For example, on the volume adjustment interface 10, users can set the playback volume for each audio stream by sliding the volume bar corresponding to each audio stream. For instance, for "media stream," if the user sets the volume of "media stream" to the level shown by volume bar 11, then playing songs, playing audio corresponding to videos, or playing audio while playing games will all have the same playback volume corresponding to volume bar 11.

[0065] In addition, for the applications corresponding to "Media Streaming," you can also set independent sound effects used by the application. These independent sound effect settings can be configured within the application or in the system settings interface. Figure 2 (Not shown in the image) Independent sound effects may include one or more of the following: stereo, Dolby sound, high-fidelity (Hi-Fi) sound, digital theatre system (DTS) sound, etc.

[0066] The above adjustment methods suffer from poor intelligence, as detailed below.

[0067] 1. When an electronic device plays multiple audio streams of the same type simultaneously, the user often cannot independently adjust the volume of a single audio stream, or in other words, the user cannot independently adjust the volume of any one of the multiple audio streams of the same type. For example, if a user launches a music player app to play music while simultaneously launching a video player app to play a video and its corresponding audio stream, both apps' audio streams are media streams, and their volumes cannot be adjusted separately; they can only be adjusted synchronously.

[0068] 2. When multiple applications have different audio stream types, users need to adjust them separately, for example, in Figure 2 The volume adjustment interface shown in 10 can be adjusted individually, and users may need to switch interfaces. Especially in scenarios with multiple audio applications running concurrently, users may need to frequently switch between different applications, and the audio stream types of different applications may be different. If the volume and sound effects need to be reset every time an application is switched, it will make the operation cumbersome.

[0069] 3. When a single application contains multiple sound sources, the electronic device can only adjust the overall audio stream volume of that application, and cannot adjust the volume or sound effects of individual sound sources. For example, when a video application plays a video containing multiple sound sources, it can only adjust the overall audio stream corresponding to that video, and cannot adjust the volume or sound effects of each individual sound source.

[0070] To address the above issues, this application provides an audio playback method, electronic device, and storage medium. When multiple applications are launched concurrently for audio playback, or when a video playback application is launched to play a video containing multiple sound source objects, the method can determine the user's gaze area when looking at the screen of the electronic device and use the application or sound source object corresponding to the gaze area as the optimization target. Therefore, when the audio streams of multiple applications are of the same type, the method can optimize the audio stream of the application that the user is interested in. When the video played by the video application contains multiple sound sources, the method can optimize the spatial sound field effect of the sound source object that the user is interested in. No additional adjustment operation is required from the user, which improves the intelligence of audio adjustment and facilitates the user's management of audio streams.

[0071] The implementation of the technical solution of this application will be described below with reference to the accompanying drawings.

[0072] The terms "first" and "second" used in this application description are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0073] See Figure 3 The figure is a flowchart of an audio playback method provided in an embodiment of this application.

[0074] The following details each step included in this method.

[0075] S11: Launch multiple applications to play audio concurrently.

[0076] In this embodiment, the electronic device running the application is not specifically limited. The electronic device can be a mobile phone, tablet computer, laptop computer, etc. When the electronic device is a mobile phone, the mobile phone can be a non-foldable screen phone or a foldable screen phone.

[0077] See also Figure 4 and Figure 5 .in, Figure 4 Scenario illustration provided for embodiments of this application Figure 1 ; Figure 5 Scenario illustration provided for embodiments of this application Figure 2 .

[0078] Electronic devices can run multiple applications in parallel, and these applications can play audio concurrently. This application does not specifically limit the type of audio streams corresponding to the multiple applications. The types of audio streams corresponding to the multiple applications can be the same or different.

[0079] For example, Figure 4 The electronic device is a foldable phone, with the calling app and music playback app running in parallel windows. The audio stream for the calling app is a call stream, while the audio stream for the music playback app is a media stream. In this scenario, the user can set the volume of the audio stream for the calling app separately, and the volume of the audio stream for the music playback app separately.

[0080] For example, Figure 5 The electronic device is a foldable phone, with game and video playback applications running in parallel windows. The audio streams for both applications are media streams. In this scenario, the volume of the audio streams for the game and video playback applications can only be adjusted synchronously.

[0081] See also Figure 1 When multiple applications concurrently play audio, the Framework layer identifies the number of audio source channels and selects the playback path. In one possible implementation, the playback path is selected as offload, where decoding is performed by the ADSP, and volume and sound effect control are also implemented by the ADSP. After selecting the offload path, a new multi-channel playback path is created, and the audio effect processing module of the Framework layer can resample and determine the sound effects for each audio stream separately.

[0082] When multiple applications play audio concurrently, the Framework layer changes one or more parameters of the audio stream, such as the sampling rate, number of channels, and bit depth, through resampling.

[0083] Sampling rate refers to the number of times an audio signal is sampled per second. Changing the sampling rate can affect audio quality and file size. Number of channels refers to the number of channels in the audio stream, such as mono, stereo, 5.1 channel, etc. By changing the number of channels, you can switch from stereo to mono to reduce file size, or convert the multi-channel layout to suit different playback systems. Bit depth refers to the number of bits used to represent each audio sample, also known as quantization precision. Increasing the bit depth can improve the dynamic range and signal-to-noise ratio of the audio and reduce quantization noise, but it will increase the file size.

[0084] The Framework layer can also adjust the amplitude of each audio waveform, that is, change the bit depth parameter value of the sampling point, through the volume processing module, based on the volume value set by the user on the electronic device.

[0085] S12: Determine the area of ​​the user's gaze when looking at the screen of the electronic device.

[0086] In this embodiment, the electronic device can determine the area of ​​the screen that the user is looking at, and determine the application or sound source that the user is currently focusing on based on the area of ​​the screen.

[0087] Electronic devices can determine the gaze area through eye-tracking technology. This application does not limit the specific eye-tracking method. For example, the pupil-corneal reflection method, retinal imaging method, method of calculating the visual center after modeling the eye, retinal reflection light intensity detection method, corneal reflection light intensity detection method, etc. can be used. These will be described below.

[0088] The pupillary-corneal reflex method tracks eye movements by connecting the center of the cornea and the center of the pupil. It acquires images of the eye using a front-facing camera on an electronic device, extracts the pupil contour from these images to calculate the pupil center, and then infers the displacement of eye movement in each frame. This, combined with an eye model, maps the gaze point onto a display screen, thus determining the gaze area. The error between the actual observed point and the estimated observation can be calibrated using a convolutional neural network (CNN) model.

[0089] Retinal imaging refers to eye tracking by calculating changes in retinal images using patterns formed by irregular capillaries, fovea, and other physiological structures on the retina.

[0090] The two methods described above mainly acquire images, including those of the eyes, through a camera. The electronic device then processes the images, extracts eye features, and calculates and determines the gaze area based on those features.

[0091] The method of calculating the visual center after modeling the eye mainly uses structured light or light field cameras to reconstruct the eyeball into a three-dimensional model in real time. By calculating the three-dimensional model of the eyeball, the direction of eye movement is obtained, thereby realizing eye tracking. The visual center can be calculated by reconstructing the eye into a three-dimensional model based on infrared cameras, depth cameras, etc.

[0092] The retinal reflection light intensity detection method refers to the process where a laser passes through the cornea, pupil, and lens and finally hits the retina. The retina reflects the incident laser light, and an external photosensor determines the direction of eye movement by detecting the intensity of the reflected light from the retina.

[0093] The principle of corneal reflectance intensity detection is as follows: the outer wall of the cornea is relatively smooth, so when a beam of light is emitted from one side of the eye onto the cornea, it will be reflected. When the light hits the center of the cornea (the highest point), the intensity of light reflection is the greatest. Therefore, the direction of eye movement is determined by detecting the intensity of the reflected light from the cornea using a photosensor.

[0094] Both of the above methods can obtain the visual center of the eye by capturing the intensity of the light reflected from the eye using one or more photosensitive sensor elements.

[0095] Considering that in practical applications, electronic devices such as mobile phones and tablets generally have front-facing cameras, in order to reduce the need for additional hardware and avoid increasing costs, the fixation area can be determined by using methods such as pupil-corneal reflex, retinal imaging, or by calculating the visual center after modeling the eye.

[0096] Furthermore, for scenarios where applications run in multiple windows, since it is only necessary to determine roughly which window the gaze area is located in, the accuracy requirement for the gaze area can be lower. Therefore, the recognition accuracy of the scheme that uses images to determine the user's gaze area can meet the usage requirements.

[0097] This application does not limit the specific size and shape of the gaze region. In one possible implementation, the gaze region can be an area comprising a certain number of pixels, and can be defined as a circle, square, or rectangle. For example... Figure 4 As shown, the gaze area S2 is located in the lyrics section of a music playback application, for example... Figure 5 As shown, the gaze area S3 is located on the person in the video being played by the video playback software.

[0098] The gaze area reflects the user's current focus. Typically, the application corresponding to the user's gaze area is also the application the user is currently focusing on, and the audio stream corresponding to that application is also the audio stream the user wants to ensure a high level of audio experience.

[0099] The application corresponding to the gaze area can be displayed in full-screen mode. In this case, the application is usually a video playback application. When there are multiple sound source objects in the video, the sound source object corresponding to the gaze area is the sound source object that the user is paying attention to.

[0100] The application corresponding to the gaze area can also be displayed in a window, such as... Figure 4 and Figure 5 As shown, the application corresponding to the gaze area is the first application, and the audio stream of the first application is the audio stream to be optimized.

[0101] In addition, the application corresponding to the gaze area can also be in a dynamic capsule state, as explained below with reference to the attached diagram.

[0102] See Figure 6 This figure is a schematic diagram of a scenario provided in an embodiment of this application. Figure 3 .

[0103] Figure 6 In the scenario shown, the video playback software and instant messaging software run in separate windows. The audio stream for the video playback software is a media stream, while the instant messaging software is not currently playing audio. Furthermore, the electronic device's "Capsule" is located at the top of the screen, displayed in a capsule shape, showing ongoing tasks for easy viewing of real-time status, execution of quick actions, or rapid access to applications. The Capsule can collapse into a capsule shape. It can also expand into a larger form, such as... Figure 6 The expanded shape is shown. At this point, the application corresponding to the gaze area is the first application, and the audio stream of the first application is the audio stream to be optimized.

[0104] S13: Perform a first optimization operation on the audio stream of the first application corresponding to the gaze region, and perform a second optimization operation on the audio streams of other applications besides the first application among multiple applications.

[0105] Since the first application corresponding to the gaze area is the application the user is currently focusing on, it is necessary to ensure the playback effect of the audio stream of the first application. The solution of this application performs a first optimization operation on the audio stream of the first application to improve the playback effect of the audio stream of the first application.

[0106] Specifically, the first optimization operation may include one or more of the following: adjusting the speaker playing the audio stream, increasing the volume, and using an audio optimization algorithm for optimization.

[0107] In this context, adjusting the speaker playing the audio stream refers to adding or changing the speaker playing the audio of the first application according to a predetermined adjustment strategy.

[0108] by Figure 4 Taking the scenario shown as an example, the electronic device includes four speakers, namely SPK1, SPK2, SPK3 and SPK4.

[0109] In one possible implementation, when the user is not looking at the screen, SPK1 and SPK3 can be used to play the audio stream of a calling application, while SPK2 and SPK4 can be used to play the audio stream of a music playback application. When the user's gaze area S2 corresponds to the music playback application, the electronic device can adjust the speakers playing the music playback application's audio stream, for example, adjusting SPK1, SPK2, SPK3, and SPK4 to all play the music playback application's audio stream. It is understood that SPK1 and SPK3 can simultaneously continue playing the audio stream of the calling application.

[0110] Increasing the volume refers to increasing the volume of the audio played by the speaker for the first application according to a predetermined volume adjustment strategy. Continuing with the example above, when the user's gaze area S2 corresponds to the music playback application, the first application is the music playback application. For SPK2 and SPK4, the volume can be increased from the first volume to the second volume according to the volume adjustment strategy. For SPK1 and SPK3, since no audio from the first application has been played previously, the initial volume for the first application can be considered 0. When SPK1 and SPK3 are added to play the audio from the first application, the volume can be increased from zero to the third volume according to the volume adjustment strategy. The third volume may or may not be equal to the second volume; this embodiment does not specifically limit this.

[0111] Taking mobile phones as an example, their maximum volume can reach 115-120 dB. Normally, the human ear has a relatively poor ability to distinguish weak sounds, and a difference of 3-4 dB is usually perceptible. When the volume reaches approximately 20-30 dB, a change of about 1 dB can generally be detected. In practical applications, the volume of various audio streams played by users on electronic devices is usually greater than 30 dB. Therefore, when the volume increase is greater than or equal to 1 dB, users can generally perceive the change in volume.

[0112] In this embodiment of the application, for the first application that the user is focused on, the volume of the audio stream of the first application can be increased by 1-3dB.

[0113] In one possible implementation, see [link to previous section] Figure 4 When the user is not looking at the screen, the audio streams from the call application and the music playback application are mixed and then played together by SPK1, SPK2, SPK3, and SPK4. When the user's gaze area S2 corresponds to the music playback application, the playback volume of the music playback application's audio stream can be increased by 1-3dB, and then mixed with the call application's audio stream before being played by SPK1, SPK2, SPK3, and SPK4. At this time, the user can intuitively feel the increase in the volume of the music played on the four speakers. The specific increase can be related to the initial volume of the audio streams being played. In one possible implementation, the higher the initial volume, the greater the increase in volume; in another possible implementation, the lower the initial volume, the greater the increase in volume.

[0114] Optimization using audio effect optimization algorithms refers to enabling audio effect algorithms to process the audio stream to alter the user's auditory perception. Audio effect algorithms can be provided by the electronic device manufacturer, the application developer, or the operating system developer of the electronic device; this application embodiment does not impose specific limitations. Common audio effect optimization algorithms include, but are not limited to: converting multi-channel audio to stereo two-channel audio, dynamic range adjustment, echo cancellation, noise reduction, bass enhancement, ambient reverberation, and spatial sound effects.

[0115] For audio streams from other applications that the user is not paying attention to, that is, audio streams from other applications besides the first application, a second optimization operation is performed. This second optimization operation reduces the playback effect of audio streams from other applications, thereby reducing the interference that audio streams from other applications cause to the user listening to the audio of the first application, so that the user can clearly hear the audio corresponding to the first application.

[0116] The second optimization operation includes one or more of the following: adjusting the speaker playing the audio stream, reducing the volume, and stopping the use of the sound optimization algorithm.

[0117] The second optimization operation, which includes adjusting the speaker playing the audio stream, refers to changing the speaker playing audio from other applications according to a predetermined adjustment strategy, which generally reduces the number of speakers.

[0118] by Figure 4 Taking the scenario shown as an example, when the user is not looking at the screen, the audio streams from the call application and the music playback application are mixed and played together by SPK1, SPK2, SPK3, and SPK4. When the user's gaze area S1 corresponds to the call application, the call application becomes the first application. At this time, a second optimization operation is performed on the music playback application. For example, the speakers playing the music playback application can be adjusted to SPK2 and SPK4, that is, the number of speakers playing the music playback application is reduced.

[0119] Lowering the volume refers to reducing the volume of audio from applications other than the first application played by the speaker, according to a predetermined volume adjustment strategy. Continuing with the example above, when the user's gaze area S1 corresponds to the call application, for SPK2 and SPK4, the volume of the music playback application can be lowered from the fourth volume to the third volume according to the volume adjustment strategy.

[0120] In this embodiment of the application, for applications that the user is not paying attention to, the volume of the corresponding application can be reduced by 1-3dB so that the user can clearly feel the difference in volume.

[0121] In one possible implementation, see [link to previous section] Figure 4When the user is not looking at the screen, the audio streams from the call application and the music playback application are mixed and then played together by SPK1, SPK2, SPK3, and SPK4. When the user's gaze area S1 corresponds to the call application, the playback volume of the call application's audio stream can be increased by 1-3dB, and the playback volume of the music playback application's audio stream can be decreased by 1-3dB. Then, the audio streams from the call application and the music playback application are mixed and played by SPK1, SPK2, SPK3, and SPK4. At this time, the speaker playing the music playback application's audio is not adjusted, but the user can intuitively feel the decrease in the volume of the music played on the four speakers. The specific decrease can be related to the initial volume of the audio stream. In one possible implementation, the higher the initial volume, the greater the decrease in volume; in another possible implementation, the lower the initial volume, the greater the decrease in volume.

[0122] Stopping the use of audio optimization algorithms means ceasing the processing of audio streams from applications other than the first application using audio optimization algorithms, thereby altering the user's auditory perception when hearing audio from other applications. See the example above for further details. Figure 4 When the user's gaze area S1 corresponds to the call application, the audio effect optimization algorithm can be stopped to optimize the audio stream corresponding to the music playback application. For example, noise reduction, bass enhancement, and ambient reverberation optimizations can be stopped for the audio stream corresponding to the music playback application.

[0123] It is understood that in this application, the operations performed on the audio streams of the first application and other applications besides the first application can all be regarded as optimization operations, the difference being that the specific optimization directions are opposite.

[0124] S14: Play optimized audio streams from multiple applications.

[0125] For multiple optimized audio streams, audio data can be obtained through mixing. The mixed audio data drives the Smart PA, which in turn drives the corresponding speaker to play the audio.

[0126] This application utilizes a combination of a Smart PA and a speaker to effectively improve a range of performance characteristics of electronic devices, including loudness and sound quality. The Smart PA, essentially a power amplifier, possesses feedback functionality compared to ordinary power amplifiers. This allows it to specifically enhance the dynamic range of the sound output, dynamically track the speaker's state, and sense changes in its environment, adapting accordingly. In one possible implementation, the speaker's characteristics are related to its frequency / impedance curve. The Smart PA can measure the output voltage and current in real time to calculate the speaker's frequency / impedance curve. Based on a pre-established algorithm and preset algorithm parameters, the Smart PA can calculate the speaker's current amplitude. By calculating the input signal, it can also predict the speaker's future amplitude, thereby effectively adjusting the speaker's performance, such as increasing volume or improving sound quality.

[0127] In summary, the technical solution provided in this application introduces a user attention selection mechanism. This mechanism determines the user's currently focused application by identifying the area of ​​focus when the user is looking at the screen of the electronic device. Different optimization operations are automatically performed on the applications the user is focused on and those the user is not, without requiring additional user adjustments. Therefore, it does not affect the user's focus or experience quality. Furthermore, it allows users to manage audio streams from multiple applications more quickly and conveniently, improving the intelligence of audio adjustment.

[0128] The following section will explain the specific implementation method.

[0129] See Figure 7 The figure is a flowchart of another audio playback method provided in an embodiment of this application.

[0130] The method specifically includes the following steps:

[0131] S21: Launch multiple applications to play audio concurrently.

[0132] This application does not specifically limit the type of audio streams corresponding to multiple applications. The types of audio streams corresponding to multiple applications can be the same or different.

[0133] For specific scenarios, please refer to Figures 4-6 As shown, it will not be elaborated further here.

[0134] S22: Obtain an image including the user's eyes.

[0135] Electronic devices can determine the area of ​​the screen that a user is looking at, and based on that area, determine the application or sound source that the user is currently focusing on.

[0136] In this embodiment, the electronic device includes a front-facing camera located on one side of the screen. The electronic device can acquire images in front of the screen through the front-facing camera. When the user is looking at the screen and the user's face is within the field of view of the front-facing camera, the front-facing camera can acquire images including the user's eyes.

[0137] S23: Detect eye features in the image.

[0138] The processor of an electronic device can extract eye features from acquired image data, and these eye features are used to determine the user's gaze area. This application does not limit the eye-tracking technology used in the electronic device; for example, it can use methods such as pupil-corneal reflection, retinal imaging, or methods that calculate the visual center after modeling the eye, which require a front-facing camera and do not require additional hardware, thus reducing costs.

[0139] S24: When it is determined from eye characteristics that the user is gazing at a first area of ​​the screen for more than a first time period, the first area is defined as the gazing area.

[0140] This application does not limit the first time length in its embodiments; for example, it can be set to 10-30 seconds. The specific value of the first time length can be a configured system parameter, or it can be actively configured by the user in the system settings. In other embodiments, the first time length can also be dynamically adjusted according to the current system time, the specific application being launched, etc.

[0141] The embodiments of this application do not limit the specific size and shape of the gaze region. The gaze region can be an area including a certain number of pixels, and the gaze region can be defined as a circle, square, or rectangle, etc.

[0142] It is understandable that the above Figures 4-6 The gaze areas S1-S4 shown are for ease of understanding only. In actual applications, the gaze area markers will not appear on the screen of electronic devices, that is, no specific lines or diagrams will appear on the screen to indicate the current user's gaze area.

[0143] S25: Determine the first application corresponding to the gaze area.

[0144] Once the gaze area is determined, the processor further determines the first application corresponding to the gaze area, which is the application that the user's eyes are currently looking at.

[0145] The processor of an electronic device can determine the display area occupied by each application on the screen based on the current running status of multiple applications.

[0146] byFigure 4 For example, in this case, the display area occupied by the call application is the left side of the electronic device's screen. The pixel coordinates of this display area can be represented as X1-X2, Y1-Y2. Here, X1 is the x-coordinate of the leftmost pixel, X2 is the x-coordinate of the rightmost pixel, Y1 is the y-coordinate of the bottommost pixel, and Y2 is the y-coordinate of the topmost pixel. Pixels whose coordinates satisfy the above range are used to display the call application. It is understandable that the application's display area can be represented in other ways, but the idea and principle are similar to the example above, and will not be elaborated further here.

[0147] The gaze area can be represented in the same way as the display area.

[0148] For example, the pixel coordinate range of the gaze region can be represented as X3-X4, Y3-Y4. Here, X3 is the x-coordinate of the leftmost pixel in the gaze region, X4 is the x-coordinate of the rightmost pixel in the gaze region, Y3 is the y-coordinate of the bottommost pixel in the gaze region, and Y4 is the y-coordinate of the topmost pixel in the gaze region.

[0149] Understandably, for ease of judgment, in practical applications, the gaze area is usually chosen to be circular or rectangular. In this case, the processor can easily calculate the area of ​​the gaze area based on the range of pixel coordinates.

[0150] The processor matches the gaze area with the display area occupied by each application on the screen to determine the first application. In other words, the processor determines which application's display area the gaze area is in, or which application's display area the gaze area overlaps with the largest.

[0151] The processor can determine the display area occupied by each of the multiple applications on the screen based on their current running states. For example, when multiple applications are running in separate windows, the processor can determine the display area corresponding to the application based on the display window it is in. Then, it matches the gaze area with the display area occupied by each application on the screen to identify the first application.

[0152] Specifically, when the gaze area includes the first display area, or the first display area includes the gaze area, or the overlap ratio between the gaze area and the first display area is greater than a preset ratio, the application corresponding to the first display area is determined as the first application.

[0153] Examples are given below.

[0154] For example, see Figure 5The user's gaze area is S3, and the first display area of ​​the video playback application is the lower half of the foldable screen, denoted by A. At this time, the first display area A includes the gaze area S3, so the video playback application corresponding to the first display area is considered to be the first application.

[0155] For example, see Figure 6 The user's gaze area is S4, and the dynamic capsule used to display music playback application information is in an unfolded state. At this time, the first display area of ​​the music playback application is B in the figure. If the overlap ratio between the gaze area S4 and the first display area B is greater than a preset ratio, then the music playback application corresponding to the first display area is determined as the first application. This application embodiment does not specifically limit the preset ratio. In practical applications, in order to determine a unique first application, the value of the preset ratio can be greater than 50%, for example, it can be set to 60%.

[0156] For example, see Figure 8 The scene shown is a schematic diagram. Figure 4 The user's gaze area is S5. The dynamic capsule used to display music playback application information is in an expanded state. At this time, the first display area of ​​the music playback application is C in the figure. Since the gaze area S5 includes the first display area C, it is considered that the music playback application corresponding to the first display area C is the first application.

[0157] S26: Determine the audio stream for the first application.

[0158] Once the first application is determined, the processor of the electronic device determines the audio stream corresponding to the first application, and at the same time, the processor also determines the audio streams of other applications besides the first application.

[0159] S27: Perform the first optimization operation on the audio stream of the first application.

[0160] The first application corresponding to the gaze area is the application the user is currently focusing on; therefore, it is necessary to ensure the playback effect of the audio stream of the first application. The solution in this application embodiment performs a first optimization operation on the audio stream of the first application to improve its playback effect.

[0161] Specifically, the first optimization operation may include one or more of the following: adjusting the speaker playing the audio stream, increasing the volume, and using an audio optimization algorithm for optimization.

[0162] Adjusting the speaker playing the audio stream refers to adding or changing the speaker playing the audio of the first application according to a predetermined adjustment strategy.

[0163] Specifically, you can first determine the speaker currently playing the audio stream of the first application, and then determine the additional speaker to play the audio stream of the first application based on the correspondence between the display area of ​​the first application and the speaker selection priority.

[0164] by Figure 4 For example, assuming the first application is a music player, before the user looks at the first application, the speakers playing the music are SPK2 and SPK4. After the user looks at the first application, the display area of ​​the first application is on the right side of the screen, so the speakers SPK1 and SPK3 can be added to be enabled. SPK1 and SPK3 can have the same enabling priority, meaning that SPK1, SPK2, SPK3, and SPK4 will play music together; or SPK1 and SPK3 can have different enabling priorities, for example, SPK3 can be enabled first, then SPK2, SPK3, and SPK4 will play music together.

[0165] Increasing the volume refers to increasing the volume of the audio played by the speaker for the first application according to a predetermined volume adjustment strategy. In this embodiment, for the first application that the user is interested in, its corresponding volume can be increased by 1-3dB so that the user can clearly perceive the difference in volume.

[0166] Optimization using audio effect algorithms refers to enabling audio effect algorithms to process the audio stream and alter the user's auditory experience. Audio effect optimization algorithms include, but are not limited to: converting multi-channel audio to stereo, dynamic range adjustment, echo cancellation, noise reduction, bass enhancement, ambient reverberation, and spatial sound effects.

[0167] S28: Perform a second optimization operation on the audio streams of the other applications besides the first application among multiple applications.

[0168] For audio streams from other applications not being listened to by the user—that is, audio streams from applications other than the first application—a second optimization operation is performed. This second optimization reduces the playback quality of the audio streams from other applications, minimizing interference with the user's listening to the audio from the first application, thus allowing the user to clearly hear the audio from the first application. Simultaneously, it also reduces the power consumption of the electronic device when playing audio from other applications, optimizing the device's battery life.

[0169] The second optimization operation includes one or more of the following: adjusting the speaker playing the audio stream, reducing the volume, and stopping the use of the sound optimization algorithm.

[0170] The second optimization operation includes adjusting the speaker playing the audio stream, which means changing the speaker playing the audio of other applications according to a predetermined adjustment strategy. Generally, the number of speakers will be reduced so that the difference between the audio streams of the first application and other applications is obvious when playing.

[0171] by Figure 4 For example, when the user is not looking at the screen, the audio streams from the call application and the music playback application are mixed and played by SPK1, SPK2, SPK3, and SPK4. When the user's gaze area S1 corresponds to the call application, the call application is the primary application, and the speaker playing the music playback application's audio stream can be adjusted. When disabling speakers, the speaker deactivation priority can be determined according to the correspondence between the display area and the speaker deactivation priority. For example, the upper speakers SPK1 and SPK2 can be deactivated first, or the lower speakers SPK3 and SPK4 can be deactivated first.

[0172] In another possible implementation, the number of speakers disabled is related to the length of time the user gazes at the first application. For example, the longer the user gazes at the first application, the fewer speakers are used to play audio streams from other applications. Continuing... Figure 4 For example, when the user gazes at the call app for a certain duration, SPK1 stops playing the audio stream from the music player app. At this time, SPK2, SPK3, and SPK4 all play the audio stream from the music player app. When the user gazes at the call app for a certain duration, SPK3 stops playing the audio stream from the music player app. At this time, SPK2 and SPK4 all play the audio stream from the music player app. When the user gazes at the call app for a certain duration, SPK2 stops playing the audio stream from the music player app. At this time, only SPK4 plays the audio stream from the music player app. It is understood that the above is only an example and does not constitute a limitation on the order in which SPKs are deactivated. For example, the deactivation order can also be replaced with SPK1, SPK2, SPK3, or SPK3, SPK1, SPK2, etc.

[0173] Power consumption can be reduced by decreasing the number of speakers used to play audio streams from other applications.

[0174] Lowering the volume refers to reducing the volume of audio from applications other than the primary application played by the speaker, according to a pre-determined volume adjustment strategy. In this embodiment, for applications not of interest to the user, their corresponding volume can be lowered by 1-3 dB so that the user can clearly perceive the difference in volume. By reducing the volume of audio streams from other applications, power consumption can be reduced.

[0175] Stopping the use of audio optimization algorithms means ceasing the processing of audio streams from applications other than the primary application using audio optimization algorithms, thereby altering the user's auditory experience when hearing audio from other applications. For example... Figure 9 In this process, when the user's gaze area is S1, optimizations such as noise reduction, bass enhancement, and ambient reverberation for the audio stream corresponding to the music playback application are stopped. By disabling these optimization algorithms, the power consumption of the electronic device's processor can be reduced.

[0176] S29: Mix the audio stream of the first application after the first optimization operation and the audio streams of other applications after the second optimization operation, and create multi-channel audio data through the interface.

[0177] The systems of electronic devices such as mobile phones and tablets support multi-channel audio interfaces. The hardware platform can use time division multiplexing (TDM) to create and distribute multi-channel audio data.

[0178] S30: Audio data for each channel is played by a corresponding speaker.

[0179] This completes the optimization of audio streams for multiple applications.

[0180] In summary, the technical solution provided in this application introduces a user attention selection mechanism. By identifying the user's gaze area when looking at the screen of an electronic device, the mechanism determines the application the user is currently focusing on. The audio stream corresponding to this application is then used as the optimization target. Different optimization operations are automatically performed on applications the user is focusing on and those the user is not, without requiring additional user adjustments. Therefore, the adjustment process does not affect the user's focus or experience quality. Furthermore, it allows users to manage audio streams from multiple applications more quickly and conveniently, improving the intelligence of audio adjustment.

[0181] The following explains how to implement audio playback when multiple applications are launched concurrently, and the video playback application that the user is interested in plays a video containing multiple sound source objects.

[0182] See Figure 7 The figure is a flowchart of another audio playback method provided in the embodiments of this application.

[0183] The method specifically includes the following steps:

[0184] S31: Launch multiple applications to play audio concurrently.

[0185] S32: Obtain an image including the user's eyes.

[0186] S33: Detect eye features in an image.

[0187] S34: When it is determined from eye characteristics that the user is gazing at a first area of ​​the screen for more than a first time period, the first area is defined as the gazing area.

[0188] S35: Determine the first application corresponding to the gaze area.

[0189] S36: Determine the audio stream for the first application.

[0190] For explanations of S31-S36 above, please refer to the corresponding documents. Figure 10 S21-S26 in the present application will not be described again here. The difference is that the first application identified in the present application is a video playback application, and the video being played includes multiple sound source objects.

[0191] S37: Determine the characteristic parameters of each sound source object in the video.

[0192] The first optimization actions for video playback applications include adjusting the speaker playing the audio stream, increasing the volume, and using one or more of the following: optimization using audio effect optimization algorithms.

[0193] Adjusting the speaker playing the audio stream refers to adding or changing the speaker playing the audio of the first application according to a predetermined adjustment strategy.

[0194] For example, with Figure 5 The scene shown is a schematic diagram. Figure 10 For example, before the user looks at the video playback application, the speakers playing the video are SPK2 and SPK4. When the user looks at area S6, the electronic device can control the speakers playing the video to be SPK1, SPK2, SPK3, and SPK4. By increasing the number of speakers, the playback effect is improved.

[0195] Increasing the volume refers to increasing the volume of audio played by the speaker in a video playback application according to a predetermined volume adjustment strategy. In this embodiment, the corresponding volume can be increased by 1-3dB so that the user can clearly perceive the difference in volume.

[0196] The following explains how to optimize the audio stream of a video playback application using a sound effect optimization algorithm.

[0197] First, determine the characteristic parameters of each sound source object in the video.

[0198] A sound source object refers to an object in the video that has the ability to produce sound. Its characteristic parameters are the attribute information of each sound source object. It's understandable that a sound source object, although appearing in the video, may not actually produce sound; therefore, the audio stream may not contain the audio data corresponding to that sound source object. For example, a video may contain two sound source objects, person 1 and person 2, but only person 1 may produce sound, while person 2 may remain silent. Therefore, it is necessary to further determine the sound source object producing sound based on the audio stream.

[0199] The feature parameters include one or more of the following: the type of sound source object, the number of sound source objects, the location of the sound source objects, and the distance information between the sound source objects.

[0200] The types of sound source objects can be people, animals, machinery and equipment (such as vehicles, airplanes, computers, mobile phones, etc.), etc., and this application embodiment does not make specific limitations.

[0201] The location of the sound source object can include its position in the picture, or its relative position in the spatial sound field when it emits sound.

[0202] The distance between sound source objects can include the relative distance between sound source objects in the picture, or the relative distance between each sound source object in the spatial sound field when the sound source object emits sound.

[0203] In practical applications, the above characteristic parameters of the sound source object can be determined by multimodal video image analysis technology. This technology can use recognition algorithms and artificial intelligence models to determine the above characteristic parameters.

[0204] Different characteristic parameters of the sound source object lead to different sound effect optimization algorithms used when optimizing the sound source object, or different parameters of the effective optimization algorithm used.

[0205] For example, when the sound source is a human voice, algorithms optimized for human voices can be used, or noise reduction algorithms can be employed to make the voice clearer. Furthermore, when the sound source is a person of different genders, the specific parameters of the algorithm used to optimize the human voice can differ.

[0206] Once the feature parameters of each sound source object in the video are determined, the sound source object can be detected and matched based on the audio stream. Specifically, the detection algorithm and the matching algorithm can be implemented by different algorithms, or they can be combined into a single algorithm, which can be called an audio algorithm.

[0207] The detection algorithm can detect the sound source object corresponding to the audio stream, and the matching algorithm is used to match the sound source object detected by the detection algorithm with the sound source object in the video frame determined by the multimodal video image analysis technology, thereby determining the sound source object.

[0208] For example, the detection algorithm identifies Person 1 as the sound source in the current audio stream, while multimodal video image analysis determines that the sound source in the video frame can include Person 1 and Person 2. The matching algorithm then matches the detection results from both the detection algorithm and the multimodal video image analysis, ultimately determining that Person 1 is the actual sound source in the current video. It's important to note that for Person 2, since they are not speaking, the detection algorithm detects no audio data matching Person 2's vocal characteristics in the audio stream, thus determining that Person 2 in the video frame is not speaking.

[0209] S38: Determine the first sound source object that the user is looking at based on the gaze area.

[0210] When the user's gaze area includes the location area of ​​the first sound source object, or the location area of ​​the first sound source object includes the user's gaze area, or the overlap ratio between the user's gaze area and the location area of ​​the first sound source object is greater than a preset ratio, the sound source object corresponding to the location area of ​​the first sound source object is determined as the first sound source object.

[0211] This application does not specifically limit the preset ratio. In practical applications, in order to identify a unique sound source and avoid misjudgment, the preset ratio is generally greater than 50%, such as 51% or 60%.

[0212] See also Figure 10 The first display area of ​​the video playback application includes the user's gaze area S6, therefore the first application is determined to be a video playback application. At this time, the overlap ratio between S6 and the first sound source object position area of ​​the sound source object 61 on the right is greater than a preset ratio, therefore the sound source object 61 is determined to be the first sound source object.

[0213] Understandably, the positional information of each sound source object, as a feature parameter, has already been obtained in S37. The first sound source object that the user focuses on should be the sound source object that is emitting sound. If the user focuses on a non-sounding sound source object, then there is no need to enhance the spatial sound field effect for each sound source object.

[0214] S39: Optimize the audio stream of the first application using the sound effect optimization algorithm and the feature parameters of the first sound source object.

[0215] The audio optimization algorithm is used to reproduce the first sound source object in order to improve the matching degree between the location of the sound reproduced in one or more speakers and the location of the first sound source object in the video.

[0216] The audio optimization algorithm is also used to improve the accuracy of the audio spatial location of the first sound source object and enhance the sound field effect, thereby improving the user's auditory experience. Furthermore, by combining the design of distant and close-up views in the video image, the sound field effect can be reconstructed to make the sound field effect closer to the video demonstration.

[0217] The audio spatial orientation of the first sound source object is the spatial position of the first sound source object in the spatial environment as reflected by information such as distance, pitch angle, and horizontal angle. The solution of this application combines traditional sound object reproduction technology with psychoacoustic spatial sound field perception, which can improve the accuracy of human ear orientation perception.

[0218] The sound effect optimization algorithm can also perform bass enhancement, sound dynamic range adjustment, etc. on the first sound source object, but the embodiments of this application do not make specific limitations.

[0219] S40: Optimize the audio stream of the first application using sound effect optimization algorithms and feature parameters of other sound source objects.

[0220] In this embodiment, the spatial sound field effect of the first sound source object is enhanced by optimizing its sound effects. Furthermore, the spatial sound field effect of other sound source objects besides the first sound source object can be reduced by optimizing them. For example, currently... Figure 11 The sound source object 62 in the image is also a sound source object that produces sound, but the user is not currently looking at the sound source object 62, so the sound effect of the sound source object 62 can be reduced.

[0221] The optimization direction for other sound source objects is the opposite of that for the first sound source object. That is, the audio optimization algorithm is used to reproduce the other sound source objects to reduce the matching degree between the location of the sound reproduced from one or more speakers and the location of the other sound source objects in the video. The audio optimization algorithm can also be used to reduce the accuracy of the audio spatial location of other sound source objects, as well as to reduce the sound field effect of other sound source objects.

[0222] Other sound source objects are all sound source objects in the video except for the first sound source object.

[0223] By optimizing other sound source objects, the sound effects of the primary sound source object can be enhanced in the auditory experience, improving the user's auditory experience when focusing on the primary sound source object. Furthermore, it can reduce the computational overhead of optimizing the spatial sound field effects of other sound source objects, thereby reducing device power consumption.

[0224] S41: Mix the audio stream of the first application after the first optimization operation and the audio streams of other applications after the second optimization operation, and create multi-channel audio data through the interface.

[0225] For audio streams from other applications not being listened to by the user—that is, audio streams from applications other than the first application—a second optimization operation is performed. This second optimization reduces the playback quality of the audio streams from other applications, minimizing interference with the user's listening to the audio from the first application, thus allowing the user to clearly hear the audio from the first application. Simultaneously, it also reduces the power consumption of the electronic device when playing audio from other applications, optimizing the device's battery life.

[0226] The second optimization operation includes one or more of the following: adjusting the speaker playing the audio stream, reducing the volume, and stopping the use of the sound optimization algorithm.

[0227] The systems of electronic devices such as mobile phones and tablets support multi-channel audio interfaces, and the hardware platform can use time-division multiplexing to create and distribute multi-channel audio data.

[0228] S42: The audio data for each channel is played by a corresponding speaker.

[0229] The above steps in the embodiments of this application are for illustrative purposes only and do not constitute a limitation on the technical solution of this application. In practical applications, those skilled in the art can adjust the order of the above steps according to the actual situation, such as changing the order of S39 and S40.

[0230] In summary, the solution provided in this application can determine the application the user is currently focusing on based on the area the user is looking at when viewing the screen of an electronic device. The audio stream corresponding to this application is then used as the optimization target, and different optimization operations are automatically performed on applications the user is focusing on and those the user is not, without requiring additional adjustments from the user. Therefore, it does not affect the user's focus or experience quality. Furthermore, it allows users to manage the audio streams of multiple applications more quickly and conveniently, improving the intelligence of audio adjustment. In addition, when the first application is a video playback application, and the video played by the application includes multiple sound source objects, the system can further identify the first sound source object the user is looking at and automatically perform different optimization operations on the first sound source object and other sound source objects the user is not looking at, further improving the intelligence of audio playback and management, and further enhancing the user's listening experience.

[0231] The above embodiments illustrate optimization methods when multiple applications are launched concurrently for audio playback. The following describes the implementation method when the video playback software enabled by the electronic device includes multiple sound source objects and no other applications are enabled concurrently for audio playback.

[0232] See Figure 12 The figure is a flowchart of another audio playback method provided in an embodiment of this application.

[0233] The method specifically includes the following steps:

[0234] S51: Launch the video playback application to play the video and its corresponding audio.

[0235] At this point, the electronic device can detect that a video playback application is running in the foreground window, and that the video playback application needs to play audio.

[0236] In one possible implementation, the audio stream can first have the number of audio source channels identified by the Framework layer, and a playback link can be selected. For example, if the playback link is selected as offload, decoding is performed by the ADSP, and volume and sound effects control are also implemented by the ADSP. After selecting the offload link, a new multi-channel playback link is created, and the sound effects processing module of the Framework layer can resample and determine the sound effects for each audio stream separately.

[0237] S52: Obtain an image including the user's eyes.

[0238] The electronic device can determine the area of ​​the screen the user is looking at, and based on that area, determine the sound source the user is currently focusing on. In this embodiment, the electronic device includes a front-facing camera located on one side of the screen. The electronic device can acquire images in front of the screen through the front-facing camera. When the user is looking at the screen and their face is within the field of view of the front-facing camera, the camera can acquire images including the user's eyes.

[0239] S53: Detect eye features in an image.

[0240] The processor of an electronic device can extract eye features from acquired image data, and these eye features are used to determine the user's gaze area. This application does not limit the eye-tracking technology used in the electronic device; for example, it can use methods such as pupil-corneal reflection, retinal imaging, or methods that calculate the visual center after modeling the eye, which require a front-facing camera and do not require additional hardware, thus reducing costs.

[0241] S54: When it is determined from eye characteristics that the user is gazing at a second area of ​​the screen for more than a second time length, the second area is defined as the gazing area.

[0242] This application does not limit the second time length in its embodiments; for example, it can be set to 10-30 seconds. The specific value of the second time length can be a pre-configured system parameter, or it can be actively configured by the user in the system settings. In other embodiments, the second time length can also be dynamically adjusted according to the current system time, the specific application being launched, etc.

[0243] The embodiments of this application do not limit the specific size and shape of the gaze region. The gaze region can be an area including a certain number of pixels, and the gaze region can be defined as a circle, square, or rectangle, etc.

[0244] S55: Determine the characteristic parameters of each sound source object in the video.

[0245] See Figure 6 This figure is a schematic diagram of a scenario provided in an embodiment of this application. Figure 12 .

[0246] A sound source object refers to an object in the video that has the ability to produce sound. Its characteristic parameters are the attribute information of each sound source object. It's understandable that a sound source object, although appearing in the video, may not actually produce sound; therefore, the audio stream may not contain the audio data corresponding to that sound source object. For example, Figure 12 The video frame shown includes two sound source objects, namely sound source object 71 and sound source object 72. Sound source object 71 emits sound, while sound source object 72 does not emit sound. Therefore, there is no audio data corresponding to sound source object 72 in the audio stream.

[0247] In this embodiment of the application, it is also necessary to further determine the sound source object that produces the sound from each sound source object based on the audio stream.

[0248] The feature parameters include one or more of the following: the type of sound source object, the number of sound source objects, the location of the sound source objects, and the distance information between the sound source objects.

[0249] The types of sound source objects can be people, animals, machinery and equipment (such as vehicles, airplanes, computers, mobile phones, etc.), etc., and this application embodiment does not make specific limitations.

[0250] The location of the sound source object can include its position in the picture, or its relative position in the spatial sound field when it emits sound.

[0251] The distance between sound source objects can include the relative distance between sound source objects in the picture, or the relative distance between each sound source object in the spatial sound field when the sound source object emits sound.

[0252] The above-mentioned characteristic parameters of the sound source object can be determined by multimodal video image analysis technology, which can use recognition algorithms and artificial intelligence models to determine the above characteristic parameters.

[0253] Different characteristic parameters of the sound source object lead to different sound effect optimization algorithms used when optimizing the sound source object, or different parameters of the effective optimization algorithm used.

[0254] For example, when the sound source is a human voice, algorithms optimized for human voices can be used, or noise reduction algorithms can be employed to make the voice clearer. Furthermore, when the sound source is a person of different genders, the specific parameters of the algorithm used to optimize the human voice can differ.

[0255] Once the characteristic parameters of each sound source object in the video are determined, the sound source object that is emitting sound can be detected and matched based on the audio stream.

[0256] The functions of detection and matching algorithms can be implemented by different algorithms, or they can be combined and implemented by a single algorithm. The combined algorithm can be called an audio algorithm.

[0257] The detection algorithm can detect the sound source object corresponding to the audio stream. The matching algorithm is used to match the sound source object detected by the detection algorithm with the sound source object in the video frame determined by multimodal video image analysis technology, thereby determining the location information of the sound source object.

[0258] The detection algorithm identifies female character 1 as a sound source in the current audio stream, while multimodal video image analysis determines that sound source objects in the video frame can include sound source object 71 and sound source object 72. The matching algorithm matches the detection results from both the detection algorithm and the multimodal video image analysis, ultimately identifying sound source object 72 as the actual sound source object in the current video frame.

[0259] It should be noted that, for sound source object 71, since it does not produce sound, the detection algorithm detects that there is no audio data in the audio stream that matches the sound production characteristics of sound source object 71, and thus determines that sound source object 71 in the video frame does not produce sound.

[0260] S56: Determine the first sound source object that the user is looking at based on the gaze area and the display area of ​​each sound source object.

[0261] When the user's gaze area includes the location area of ​​the first sound source object, or the location area of ​​the first sound source object includes the user's gaze area, or the overlap ratio between the user's gaze area and the location area of ​​the first sound source object is greater than a preset ratio, the sound source object corresponding to the location area of ​​the first sound source object is determined as the first sound source object.

[0262] This application does not specifically limit the preset ratio. In practical applications, in order to identify a unique sound source and avoid misjudgment, the preset ratio is generally greater than 50%, such as 51% or 60%.

[0263] See also Figure 12 The overlap ratio between the user's gaze area S6 and the first sound source object location area of ​​the sound source object 71 on the right is greater than the preset ratio, therefore the sound source object 71 is determined to be the first sound source object.

[0264] It is understandable that the location information of each sound source object, as a feature parameter, has already been obtained in S55.

[0265] In this embodiment, the first sound source object that the user is focusing on is a sound source object that emits sound. If the user is focusing on a non-sounding sound source object, then there is no need to enhance the spatial sound field effect for each sound source object.

[0266] S57: Enhance the spatial sound field effect of the first sound source object by using sound effect optimization algorithms and feature parameters of the first sound source object.

[0267] Audio optimization algorithms can enhance the spatial sound field effect of the first sound source object. Specifically, the audio optimization algorithm is used to reproduce the first sound source object to improve the matching degree between the location of the sound reproduced from one or more speakers and the position of the first sound source object in the video.

[0268] The audio optimization algorithm is also used to improve the accuracy of the audio spatial location of the first sound source object and enhance the sound field effect, thereby improving the user's auditory experience. Furthermore, by combining the design of distant and close-up views in the video image, the sound field effect can be reconstructed to make the sound field effect closer to the video demonstration.

[0269] The audio spatial orientation of the first sound source object is the spatial position of the first sound source object in the spatial environment as reflected by information such as distance, pitch angle, and horizontal angle. The solution of this application combines traditional sound object reproduction technology with psychoacoustic spatial sound field perception, which can improve the accuracy of human ear orientation perception.

[0270] The sound effect optimization algorithm can also perform bass enhancement, sound dynamic range adjustment, etc. on the first sound source object, but the embodiments of this application do not make specific limitations.

[0271] S58: Using sound effect optimization algorithms and characteristic parameters of other sound source objects besides the first sound source object, reduce the spatial sound field effect of other sound source objects besides the first sound source object.

[0272] In this embodiment, the spatial sound field effect of the first sound source object is enhanced by optimizing its sound effects. Furthermore, the spatial sound field effect of other sound source objects besides the first sound source object can be reduced by optimizing them. For example, currently... Figure 13 The sound source object 72 in the image is also a sound source object that produces sound, but the user is not currently looking at the sound source object 72, so the sound effect of the sound source object 72 can be reduced.

[0273] Other sound source objects are all sound source objects in the video except for the first sound source object.

[0274] By optimizing other sound source objects, the sound effects of the primary sound source object can be enhanced in the auditory experience, improving the user's auditory experience when focusing on the primary sound source object.

[0275] Specifically, when using audio optimization algorithms to optimize other sound source objects, the audio optimization algorithms and the feature parameters of other sound source objects can be used to reduce the matching degree between the location of the sound reproduced in one or more speakers and the location of other sound source objects in the video.

[0276] Furthermore, by utilizing sound effect optimization algorithms and the characteristic parameters of other sound source objects, the accuracy of the audio spatial orientation of other sound source objects can be reduced, as well as the sound field effect of other sound source objects can be reduced.

[0277] By reducing the spatial sound field effect of other sound source objects, the computational overhead of optimizing the spatial sound field effect of other sound source objects can be reduced, thereby reducing device power consumption. In addition, it can enhance the sound effect of the primary sound source object in the auditory experience, improving the user's auditory experience when focusing on the primary sound source object.

[0278] S59: Play the audio stream corresponding to the optimized video.

[0279] By processing different sound source objects in different ways, an optimized audio stream corresponding to the video is obtained, and then an electronic device drives one or more speakers to play the optimized audio stream.

[0280] The above steps in the embodiments of this application are for illustrative purposes only and do not constitute a limitation on the technical solution of this application. In practical applications, those skilled in the art can adjust the order of the above steps according to the actual situation. For example, the order of S57 and S58 can be swapped, or S55 can be performed before S52.

[0281] In summary, by utilizing the solution provided in this application embodiment, when the video played by the video playback software includes multiple sound source objects, the sound source object currently of the user's attention can be determined based on the user's gaze area when looking at the screen of the electronic device. The spatial sound field effect of the sound source object of the user's attention can be enhanced, while the spatial sound field effect of other sound source objects that the user is not paying attention to can be reduced. No additional adjustment or settings are required from the user, so it will not reduce the user's focus or experience quality, and the intelligence of audio adjustment is improved, thus also improving the user's listening experience.

[0282] Based on the audio playback method provided in the above embodiments, this application also provides an electronic device with audio playback capability, which will be described in detail below with reference to the accompanying drawings.

[0283] See ​ This figure is a schematic diagram of an electronic device provided in an embodiment of this application.

[0284] This application does not specifically limit the type of electronic device 100. The electronic device 100 may be a foldable screen phone, a non-foldable screen phone, a laptop computer, a tablet computer, an augmented reality (AR) device, a virtual reality (VR) device, or an in-vehicle device, etc.

[0285] The illustrated electronic device includes: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0286] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0287] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Processor 110 can utilize audio optimization algorithms to optimize the audio stream of the user's primary application and the spatial sound field effects of the sound source objects of interest to the user.

[0288] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0289] Electronic device 100 can achieve its shooting function through an image signal processor (ISP), a camera 193, a video codec, a GPU, a display screen 194, and an application processor. The ISP processes data fed back from the camera 193. The camera 193 captures still images or videos. An object passes through a lens to generate an optical image, which is projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats.

[0290] In this embodiment of the application, the electronic device 100 may include multiple cameras, including at least one front-facing camera, for acquiring images of the user's eyes and thereby determining the user's gaze area.

[0291] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn. Electronic devices can use NPUs to achieve relatively accurate localization of the gaze area. When the user's primary application is video playback, the NPU can use multimodal video image analysis technology to determine the feature parameters of sound source objects in the video image, use detection algorithms to detect the sound source objects corresponding to the audio stream, and use matching algorithms to determine the sound source objects in the video frame.

[0292] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. Audio module 170 may also include Smart PA.

[0293] Using the electronic device provided in this application embodiment, when multiple applications are launched and playing audio concurrently, the system identifies the user's gaze area when looking at the screen of the electronic device, determines the application the user is currently focusing on, and uses the audio stream corresponding to that application as the optimization target. Different optimization operations are automatically performed on the applications the user is focusing on and those the user is not focusing on. When the application the user is focusing on is a video playback application, and the video played by the video playback application includes multiple sound source objects, the system can further identify the first sound source object the user is focusing on, and automatically perform different optimization operations on the first sound source object and other sound source objects the user is not focusing on. When the electronic device only uses the video playback application to play audio, the system can determine the sound source object the user is currently focusing on based on the user's gaze area when looking at the screen of the electronic device, enhance the spatial sound field effect of the sound source object the user is focusing on, and simultaneously reduce the spatial sound field effect of other sound source objects the user is not focusing on.

[0294] In summary, this electronic device enables users to manage the spatial sound field effects of multiple application audio streams or multiple sound source objects more quickly and conveniently, improving the intelligence of audio adjustment.

[0295] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0296] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An audio playback method, characterized by, The method comprises: starting a plurality of applications to play audio concurrently; determining a gaze area when a user gazes at a screen of an electronic device; performing a first optimization operation on an audio stream of a first application corresponding to the gaze area, and performing a second optimization operation on an audio stream of other applications in the plurality of applications except the first application, wherein the first optimization operation comprises one or more of the following: adjusting a speaker playing the audio stream, increasing a volume, and optimizing using an audio effect optimization algorithm; and the second optimization operation comprises one or more of the following: adjusting a speaker playing the audio stream, decreasing a volume, and stopping using an audio effect optimization algorithm; playing the audio streams of the plurality of applications after optimization.

2. The method of claim 1, wherein, The determination of the gaze area when the user gazes at the screen of the electronic device specifically comprises: obtaining an image including the user's eyes; detecting eye features in the image; when it is determined according to the eye features that the user gazes at a first area of the screen for more than a first length of time, determining the first area as the gaze area.

3. The method of claim 1, wherein, The first optimization operation on the audio stream of the first application corresponding to the gaze area specifically comprises: determining the first application corresponding to the gaze area; determining the audio stream of the first application; performing the first optimization operation on the audio stream of the first application.

4. The method of claim 3, wherein, The determination of the first application corresponding to the gaze area specifically comprises: determining display areas occupied by each of the plurality of applications on the screen according to current running states of the plurality of applications; matching the gaze area with the display areas occupied by each of the applications on the screen to determine the first application.

5. The method of claim 4, wherein, The matching of the gaze area with the display areas occupied by each of the applications on the screen to determine the first application specifically comprises: when the gaze area includes a first display area, or the first display area includes the gaze area, or an overlap ratio of the gaze area and the first display area is greater than a preset ratio, determining an application corresponding to the first display area as the first application.

6. The method of claim 4, wherein, The first optimization operation comprises adjusting a speaker playing the audio stream, and the first optimization operation on the audio stream of the first application specifically comprises: determining a speaker playing the audio stream of the first application, and determining a newly added speaker playing the audio stream of the first application according to a correspondence between a display area of the first application and a speaker selection priority.

7. The method of claim 4, wherein, The first optimization operation comprises adjusting a speaker playing the audio stream and optimizing using an audio effect optimization algorithm, and the first optimization operation on the audio stream of the first application specifically comprises: when the first application is a video playing application, determining characteristic parameters of each sound source object in a video, the characteristic parameters comprising one or more of the following: a type of the sound source object, a number of the sound source objects, a position of the sound source object, distance information between the sound source objects; determining a first sound source object gazed at by the user according to the gaze area, the first sound source object being a sound-emitting sound source object. The audio stream of the first application is optimized by using an audio effect optimization algorithm and characteristic parameters of the first sound source object, the audio effect optimization algorithm being used to play back the first sound source object to improve the matching degree between the position of the first sound source object in the video and the position of the sound played back by one or more loudspeakers, and the audio effect optimization algorithm being used to improve the accuracy of the audio spatial position of the first sound source object and enhance the sound field effect of the first sound source object.

8. The method of claim 7, wherein, The first optimization operation on the audio stream of the first application further includes: The audio stream of the first application is optimized by using the audio effect optimization algorithm and characteristic parameters of other sound source objects, the audio effect optimization algorithm being used to play back the other sound source objects to reduce the matching degree between the position of the other sound source objects in the video and the position of the sound played back by one or more loudspeakers, and the audio effect optimization algorithm being used to reduce the accuracy of the audio spatial position of the other sound source objects and reduce the sound field effect of the other sound source objects, the other sound source objects being all sound source objects in the video except the first sound source object.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The audio stream of the first application after the first optimization operation and the audio stream of the other applications after the second optimization operation are mixed, and multi-channel audio data is created through a time division multiplexing (TDM) interface. The audio data of each channel is played back by a corresponding loudspeaker.

10. An audio playing method, characterized in that, The method includes: A video playback application plays a video and audio corresponding to the video; Characteristic parameters of sound source objects in the video are determined, the characteristic parameters including a display area of a sound source object and one or more of the following: a type of the sound source object, a number of the sound source objects, and distance information between the sound source objects; A gaze area when a user gazes at a screen of an electronic device is determined; A first sound source object gazed at by the user is determined according to the gaze area and the display area of each sound source object, the first sound source object being a sound-emitting sound source object; The spatial sound field effect of the first sound source object is enhanced by using an audio effect optimization algorithm and the characteristic parameters of the first sound source object, and the spatial sound field effect of other sound source objects except the first sound source object is reduced by using the audio effect optimization algorithm and the characteristic parameters of the other sound source objects; The audio corresponding to the video after optimization is played back.

11. The method of claim 10, wherein, The spatial sound field effect of the first sound source object is enhanced by using an audio effect optimization algorithm and the characteristic parameters of the first sound source object, and the spatial sound field effect of other sound source objects except the first sound source object is reduced by using the audio effect optimization algorithm and the characteristic parameters of the other sound source objects; The spatial sound field effect of the first sound source object is enhanced by using an audio effect optimization algorithm and the characteristic parameters of the first sound source object, and the spatial sound field effect of other sound source objects except the first sound source object is reduced by using the audio effect optimization algorithm and the characteristic parameters of the other sound source objects; The matching degree between the position of the first sound source object in the video and the position of the sound played back by one or more loudspeakers is improved by using an audio effect optimization algorithm and the characteristic parameters of the first sound source object; The accuracy of the audio spatial position of the first sound source object is improved and the sound field effect of the first sound source object is enhanced by using an audio effect optimization algorithm and the characteristic parameters of the first sound source object.

12. The method of claim 10, wherein, The sound effect optimization algorithm and the characteristic parameters of the other sound source objects except the first sound source object are used to reduce the spatial sound field effect of the other sound source objects except the first sound source object, specifically including: The sound effect optimization algorithm and the characteristic parameters of the other sound source objects are used to reduce the matching degree between the position of the other sound source objects in the video and the position of the sound played by one or more of the loudspeakers. The sound effect optimization algorithm and the characteristic parameters of the other sound source objects are used to reduce the accuracy of the audio spatial position of the other sound source objects and reduce the sound field effect of the other sound source objects.

13. The method of claim 10, wherein, The gaze area when the user gazes at the screen of the electronic device is determined, specifically including: An image including the eyes of the user is acquired. Eye features in the image are detected. When it is determined according to the eye features that the user gazes at a second area of the screen for more than a second time length, the second area is determined as the gaze area.

14. The method of claim 10, wherein, The gaze area when the user gazes at the screen of the electronic device is determined, specifically including: An image including the eyes of the user is acquired. Eye features in the image are detected. When it is determined according to the eye features that the user gazes at a second area of the screen for more than a second time length, the second area is determined as the gaze area.

15. The method according to claim 13 or 14, characterized in that, The second display area is included in the area where each sound source object is located, and the first sound source object gazed at by the user is determined according to the gaze area and the position of each sound source object, specifically including: When the gaze area includes the second display area, or the second display area includes the gaze area, or the overlap ratio of the gaze area and the second display area is greater than a preset ratio, the sound source object corresponding to the second display area is determined as the first sound source object.

16. An electronic device, comprising: The electronic device includes at least one processor, and the at least one processor is used to execute computer programs or instructions to implement the audio playing method in any one of claims 1-9 and / or the audio playing method in any one of claims 10-15.

17. A storage medium, characterized by The storage medium has a computer program stored thereon, and the computer program is executed to implement the audio playing method in any one of claims 1-9 and / or the audio playing method in any one of claims 10-15.

Images