Display device and multi-window control method

Abstract

The invention discloses display equipment and a multi-window control method, and relates to the field of display equipment, the display equipment comprises a display, an audio output interface and a controller, and the controller is in communication connection with the display and the audio output interface; wherein the audio output interface is configured to be in communication connection with a plurality of target audio devices. The method comprises the steps of outputting audio data of corresponding windows based on target audio equipment corresponding to each window under multi-window display; in response to a hiding instruction for the first window, adjusting the target audio equipment corresponding to the first window to mute output, and maintaining normal output of the target audio equipment corresponding to the second window; when the hiding instruction for the first window is received, the first window is adjusted to be output in a mute mode, it is avoided that the audio data corresponding to the hidden window continues to be played to distract attention of a user and disturb attention of the user to other windows, normal output of the target audio equipment corresponding to the second window is maintained, continuity and effectiveness of audio playing are ensured, and user experience is improved. And the user can audio and video more smoothly.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular, to a display device and a multi-window control method. Background Art

[0002] With the continuous development of terminal technologies, display devices are more and more widely used. For example, a display device supports simultaneous presentation of multiple windows, that is, multiple windows are displayed in a display interface, and the content presented in each window is different, so as to meet the viewing needs of different users simultaneously.

[0003] Taking a smart TV as an example, during the use of the smart TV, mutual switching between multiple windows and other interfaces will be performed according to user needs. For example, switching from multiple windows to full screen, or from full screen to multiple windows; since the requirements for audio devices are different under different interface displays, therefore, in the scenario of switching between multiple windows and other interfaces, a control method applied to multi-window switching is urgently needed. Summary of the Invention

[0004] The present application provides a display device and a multi-window control method, so that a stable audio-visual effect is maintained during the process of a user switching back and forth between different display interfaces.

[0005] In a first aspect, a display device is provided, including a display, an audio output interface, and a controller, where the controller is communicatively connected to the display and the audio output interface; wherein, the audio output interface is configured to be communicatively connected to a plurality of target audio devices, and the target audio devices are used for outputting audio data;

[0006] The controller is configured to perform the following process:

[0007] Under multi-window display, based on the target audio devices corresponding to each window, output the audio data of the corresponding window; in response to a hide instruction for a first window, adjust the target audio device corresponding to the first window to mute output, and maintain the normal output of the target audio device corresponding to a second window, where the first window and the second window are windows under multi-window display.

[0008] Under multi-window display, outputting corresponding audio data based on different target audio devices corresponding to each window can achieve accurate matching of audio devices and windows, and meet the user's needs for different audio device outputs for different windows.

[0009] When a hide instruction for a first window is received, adjust the corresponding target audio device to mute output, so as to avoid the audio of the hidden window interfering with the user's audio-visual experience of the content of the currently focused window; and since the hidden window is no longer in the user's field of view, the continuous playback of its corresponding audio data will distract the user's attention.

[0010] Maintaining the normal output of the target audio device corresponding to the second window can ensure that the user can focus on the visible content of the second window, guarantee the coherence and effectiveness of audio playback, and enable the user to enjoy a smoother audio-visual experience.

[0011] In a possible implementation, when the controller adjusts the target audio device corresponding to the first window to mute output and maintains the normal output of the target audio device corresponding to the second window, it is further configured to: perform a hiding operation on the first window; adjust the second display parameters corresponding to the second window to obtain adjusted second target parameters, where the second target parameters include the display position and the display area size.

[0012] In a multi-window display, responding to user demands to hide one window or a certain window can release system resources, optimize the display layout, and prevent interference with the user's attention to the content of other windows.

[0013] In a possible implementation, when the controller performs a hiding operation on the first window, it is configured to: adjust the first display parameters of the first window to obtain adjusted first target parameters, where the display area size in the first target parameters is smaller than the display area size in the first display parameters; based on the display area size and the display position in the first target parameters, display the adjusted first window.

[0014] It should be understood that adjusting the display parameters of the first window to make the display area size smaller and displaying the window based on the new size and position can optimize the multi-window layout.

[0015] In a possible implementation, when the controller displays the adjusted first window based on the display area size and the display position in the first target parameters, it is configured to: when the display position in the first target parameters is a first position outside the visible area corresponding to the display, display the adjusted first window, where the display size area in the first target parameters is less than or equal to a first area, and the first area is the area between the first position and the visible area.

[0016] In a possible implementation, when the controller displays the adjusted first window based on the display area size and the display position in the first target parameters, it is configured to: when the display position in the first target parameters is a second position corresponding to an interface control, display the adjusted first window, where the display size area in the first target parameters is smaller than the area corresponding to the interface control.

[0017] In a possible implementation, when the controller displays the adjusted first window based on the display area size and the display position in the first target parameters, it is configured to: when the display position in the first target parameters is a third position in a second layer, display the adjusted first window, where the second layer is a layer under the first layer where the second window is located.

[0018] By one of the above methods, the first window is hidden and displayed, which facilitates the user to focus more on the content in other windows (such as the second window), and the user cannot feel the existence of the first window at all. This can not only optimize the system performance, save memory, but also create an immersive experience and achieve seamless transition. At the same time, it is convenient for background tasks to run and for system maintenance and update.

[0019] In a possible implementation, the controller adjusts the second display parameters of the second window and is also configured to: when the number of windows involved in the second window is greater than 1, adjust the second display parameters of each window in the second window; arrange each window in the second window based on the adjusted second display parameters of each window in the second window.

[0020] In a possible implementation, the controller adjusts the second display parameters of the second window and is also configured to: when the number of windows involved in the second window is equal to 1, adjust the second display parameters of the second window to full-screen display; display the second window in full screen.

[0021] After the first window is hidden and the target audio device corresponding to the first window is muted, in order to improve the audio-visual experience of the user after the first window is hidden, the second display parameters corresponding to the second window are adjusted, and the adjusted second window is displayed to enhance the visual experience of the user in this scenario.

[0022] In a possible implementation, when the controller displays the second window in full screen, it is also configured to: in response to a recovery instruction for multi-window display, adjust the output of the target audio device of the first window from muted output to normal output; adjust the first target parameters of the first window and the second target parameters of the second window to obtain an updated session list, where the session list is used to record the attribute information of each terminal device casting the screen to the display device; based on the attribute information in the updated session list, display the first window and the second window simultaneously.

[0023] Under multi-window display, when the window is restored in response to user needs, the user can continue the previous operations, and the system quickly restores the window state, ensuring the continuity of operations and maintaining the convenience of multi-task processing.

[0024] In a possible implementation, after the controller outputs the audio data of the corresponding window based on the target audio device corresponding to each window, it is also configured to: in response to a disconnection instruction for the first window, disconnect the screen mirroring connection corresponding to the first window and delete the attribute information corresponding to the first window in the session list, where the session list is used to record the attribute information of each terminal device casting the screen to the display device; maintain the display interfaces of the first window and the second window.

[0025] Under multi-window display, in response to a disconnection instruction for the first window, the screen mirroring content of one of the windows is disconnected, meeting the user's needs. At the same time, network and system resources can be released, device performance can be optimized, the display layout can be made more reasonable, the viewing and interaction experience of other windows can be improved, and connection management is facilitated, enhancing the stability of device operation.

[0026] In a possible implementation manner, after the controller outputs the audio data of the corresponding window based on the target audio device corresponding to each window, it is further configured to: in response to a full-screen instruction for the first window, maintain the normal output of the target audio device of the first window, and adjust the first window to full-screen display; disconnect the screen mirroring connection corresponding to the second window, and delete the attribute information corresponding to the second window in the session list, where the session list is used to record the attribute information of each terminal device mirroring to the display device.

[0027] Under multi-window display, when one window is set to full-screen display, the screen mirroring content of another window will be disconnected, which can concentrate system resources to render the full-screen window, improving its picture quality and running smoothness; at the same time, the display layout is optimized to avoid interference from other screen mirroring content and audio data output, allowing the user to focus on the full-screen window and obtain a more immersive viewing experience.

[0028] In a second aspect, a multi-window control method is provided, which is applied to the display device in any one of the first aspects. The multi-window control method includes:

[0029] Under multi-window display, based on the target audio device corresponding to each window, output the audio data of the corresponding window;

[0030] In response to a hide instruction for the first window, adjust the target audio device corresponding to the first window to mute output, and maintain the normal output of the target audio device corresponding to the second window, where the first window and the second window are windows under multi-window display.

[0031] In a third aspect, a multi-window control device is provided, including units for executing any one of the multi-window control methods in the second aspect. The device can be a display device or a chip within the display device.

[0032] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is run by the multi-window control device, the multi-window control device is enabled to execute any one of the multi-window control methods in the second aspect.

[0033] In a fifth aspect, a computer program product is provided. The computer program product includes: a computer program, and when the computer program is run by the multi-window control device, the multi-window control device is enabled to execute any one of the multi-window control methods in the second aspect.

[0034] It can be understood that for the beneficial effects of the second to fifth aspects described above, reference can be made to the relevant descriptions in the first aspect, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 FIG. shows a schematic diagram of an operation scenario between a display device and a control device;

[0036] Figure 2 FIG. shows a schematic diagram of audio mixing control in a smart TV;

[0037] Figure 3 FIG. shows a configuration block diagram of a display device 200 provided by an embodiment of the present application;

[0038] Figure 4 FIG. shows a system architecture diagram of a display device 200 provided by an embodiment of the present application;

[0039] Figure 5 FIG. shows a schematic diagram of a multi-window display application scenario provided by an embodiment of the present application;

[0040] Figure 6 FIG. shows a schematic flow diagram of an audio control method provided by an embodiment of the present application;

[0041] Figure 7 FIG. shows a schematic diagram of a multi-window display application scenario provided by an embodiment of the present application;

[0042] Figure 8 FIG. shows an example diagram of audio control in a smart TV provided by an embodiment of the present application;

[0043] Figure 9 FIG. shows a schematic flow diagram of another audio control method provided by an embodiment of the present application;

[0044] Figure 10 FIG. shows a schematic diagram of another multi-window display application scenario provided by an embodiment of the present application;

[0045] Figure 11 FIG. shows a system architecture diagram of another display device provided by an embodiment of the present application;

[0046] Figure 12 FIG. shows a schematic flow diagram of another audio control method provided by an embodiment of the present application;

[0047] Figure 13 FIG. shows an example diagram of a channel between a target window and a preset audio device provided by an embodiment of the present application;

[0048] Figure 14Shows a schematic diagram of another multi-window display application scenario provided by an embodiment of the present application;

[0049] Figure 15 Shows a schematic flowchart of a multi-window control method provided by an embodiment of the present application;

[0050] Figure 16 Shows an application scenario diagram of window hiding control in a multi-window provided by an embodiment of the present application;

[0051] Figure 17 Shows another application scenario diagram of window hiding control in a multi-window provided by an embodiment of the present application;

[0052] Figure 18 Shows another application scenario diagram of window hiding control in a multi-window provided by an embodiment of the present application;

[0053] Figure 19 Shows another application scenario diagram of window hiding control in a multi-window provided by an embodiment of the present application;

[0054] Figure 20 Shows another application scenario diagram of window hiding control in a multi-window provided by an embodiment of the present application;

[0055] Figure 21 Shows another application scenario diagram of window hiding control in a multi-window provided by an embodiment of the present application;

[0056] Figure 22 Shows a schematic flowchart of another multi-window control method provided by an embodiment of the present application;

[0057] Figure 23 Shows an application scenario diagram of full-screen window restoration control provided by an embodiment of the present application;

[0058] Figure 24 Shows an application scenario diagram of disconnection control under a multi-window provided by an embodiment of the present application;

[0059] Figure 25 Shows an application scenario diagram of full-screen control under a multi-window provided by an embodiment of the present application. Detailed implementation manners

[0060] The following will describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; "and / or" herein is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0061] Hereinafter, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", and "third" may explicitly or implicitly include one or more of such features.

[0062] The reference to "an embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, the statements "in an embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in the specification of the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. Also, the terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0063] To better understand the embodiments of the present application, the following explains the terms or concepts that may be involved in the embodiments.

[0064] 1. System prompt tone

[0065] The system prompt tone refers to the sound signal emitted by the operating system or application program of the display device to convey specific system events, operation states, or information to the user. The system prompt tone includes operation prompt tones, connection prompt tones, and notification prompt tones.

[0066] Operation prompt tone: When the user uses the remote control or the body buttons to perform operations such as power on / off, channel switching, volume adjustment, input signal switching, etc., the operation prompt tone will sound. Usually, it is a short sound effect, such as "beep", "boop", "click", etc., or a simple melody, such as the startup music.

[0067] Connection-related prompt sound: When the TV is connected to external devices such as speakers and game consoles through interfaces such as HDMI and USB, a prompt sound will be emitted to indicate that the device has been successfully connected or recognized. It may be a relatively loud sound, such as "ding dong", to prompt the user that they can start using the external device.

[0068] Notification prompt sound: When there is a software update available for the TV system, a prompt sound will be emitted to remind the user. It may be a relatively special sound effect, such as "dingling", to attract the user's attention and let the user know that there is a new system version available for update to obtain better functions and performance.

[0069] 2. Built-in speaker

[0070] A speaker, also known as a "loudspeaker", is a transducer that converts electrical signals into sound signals and is often used in electronic devices to play sound.

[0071] In the embodiments of the present application, the speaker is integrated inside the display device, so this speaker is also called a built-in speaker.

[0072] In common traditional application scenarios, the display device outputs audio data through a single audio device (such as a built-in speaker or an external audio output device). For example, when watching TV programs in the living room at home or conducting video conferencing demonstrations in the office, a single audio device is often used to output audio data.

[0073] The display device can have various implementation forms. For example, a TV (which can also be called a smart TV, intelligent screen, or large-screen device), a tablet computer, a projection device, an electronic bulletin board, a mobile phone, a laptop computer, a netbook, a personal digital assistant (PDA), a vehicle-mounted device, etc. The audio data in the display device is processed and converted into an audio signal, which is transmitted to the built-in speaker, or transmitted to a corresponding external audio output device such as a speaker or a headphone through different connection methods such as Bluetooth, a 3.5mm audio cable, an HDMI cable, a coaxial cable, a USB cable, or an optical fiber, so as to realize the playback output of audio and meet people's reception requirements for sound in various scenarios.

[0074] Exemplarily, as a common display device, a smart TV often outputs audio data through a single audio device and meets the diverse audio-visual needs of users through different audio devices.

[0075] For example, an audio device can use a built-in speaker, which is convenient and fast, and has good compatibility with smart TVs. That is to say, there is no need for additional and complex pairing, connection and other operations. As long as the TV is turned on, users can quickly enter the state of watching programs or videos, which is very convenient to use. Moreover, the built-in speaker is part of the overall TV system, highly integrated with the TV's hardware and software systems, and can perfectly cooperate with the TV's screen display, operation interface, etc. in terms of audio output, without compatibility problems, ensuring the synchronization and stability of sound and picture.

[0076] For another example, the audio device can also choose a Bluetooth speaker and establish a connection between the Bluetooth speaker and the smart TV through Bluetooth wireless technology. When the smart TV plays video resources, the Bluetooth speaker creates an immersive surround sound effect. When the smart TV plays music resources, the Bluetooth speaker can provide richer and fuller sound, such as a more mellow bass and a more powerful shock effect.

[0077] Figure 1 The figure shows a schematic diagram of an operation scenario between a display device and a control device. As Figure 1 shown, the user can operate the display device 200 through a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop, etc.) or a control device 100 (such as a remote control, etc.). For example, the user can input user instructions through the buttons on the remote control, voice input, control panel input, etc. to control the display device 200.

[0078] In some embodiments, the display device 200 can be controlled by obtaining the user's touch instructions or gesture instructions, receiving the user's voice instructions, and so on.

[0079] In some embodiments, the display device 200 also communicates with the server 400 for data. The display device 200 is connected through a local area network (LAN), a wireless local area network (WLAN) and other networks, and the server 400 can provide various contents and interactions to the display device 200. The server 400 can be a cluster or multiple clusters, and can include one or more types of servers.

[0080] In a variety of application scenarios, the display device will receive audio data from different sources. Taking a smart TV as an example, while playing a TV program, it will also synchronously receive system prompt sounds such as operation prompt sounds. When the user switches channels, adjusts the volume, etc., the operation prompt sound will sound and coexist with the audio of the TV program. When the user is playing a game, the background music of the game itself and various sound effects in the game are constantly coming out. At the same time, the sound of the user's voice chat through the network will also be added, and a variety of sound effects are intertwined.

[0081] To meet the requirement of simultaneous playback of multiple sound sources and adapt to different application scenarios such as watching movies, daily operations, gaming, and voice chatting, a display device usually performs audio mixing processing internally, mixes different audio data, and outputs the mixed audio data through a single audio device.

[0082] The target chip in a smart TV is a chip that can be used for audio data processing. The mixing processing of different audio data is achieved through the target chip. The target chip can refer to the main control chip in the display device, which is usually responsible for functions such as processing video and audio data, image rendering, data reception, and decoding. The target chip can also be an audio processing chip that assists the main control chip.

[0083] Figure 2 Fig. shows a schematic diagram of mixing control in a smart TV. It is a schematic diagram of mixing control of the target chip when the smart TV outputs audio data through a single audio device. For the convenience of description and display, only the part used for audio mixing processing is shown in the target chip 10. Among them, the pins for receiving audio data are CH1, CH2, CH3, CH4, CH5, etc. Each pin receives audio data in the form of stereo channels.

[0084] In the embodiment of the present application, CH1-CH4 are used to receive multi-channel (Multi-Channel, mch) audio data, such as 5.1-channel audio data and 7.1-channel audio data. CH5 is used to receive stereo (2-Channel, 2ch) audio data.

[0085] It should be understood that the pins in different target chips may be different. Therefore, the interface for audio input in the target chip can also be different from the pin distribution in this embodiment. The pins in the target chip in the embodiment of the present application are not limited.

[0086] Therefore, in common traditional application scenarios, to meet the requirement of simultaneous playback of multiple sound sources in various application scenarios, as Figure 2 shown in (a) of, the target chip 10 is controlled to achieve the mixing processing of the audio data A received by CH1-CH4 and the audio data B received by CH5, mix different audio data, and output the mixed audio data through the audio device 20. Here, the audio device 20 refers to a single audio device.

[0087] It should be understood that in the target chip 10, there may be more pins as the interface for audio input.

[0088] Specifically, as Figure 2As shown in (b) in [description], the audio data A received by CH1-CH4 and the audio data B received by CH5 are mixed, and the mixed audio data is output through the built-in speaker 21. As Figure 2 As shown in (c) in [description], the audio data A received by CH1-CH4 and the audio data B received by CH5 are mixed, and the mixed audio data is output through the external audio output device, the Bluetooth speaker 22.

[0089] In some embodiments, the mixing control of the above target chip 10 can be performed and implemented by an audio mixer (Mixer) in the target chip. Among them, a Mixer is a device or software tool for mixing, adjusting, and processing multiple audio signals. It can combine audio signals from different sound sources (such as microphones, musical instruments, audio players, etc.), and adjust parameters such as the volume, tone color, and phase of each signal, and finally output a mixed audio signal.

[0090] With the continuous development of terminal technology, the application of display devices is becoming more and more extensive. For example, a display device supports simultaneous presentation of multiple windows, that is, multiple windows are displayed in a display interface, and the content presented in each window is different, so as to meet the viewing needs of different users at the same time. However, in the multi-window display mode, the audio data of different windows is still mixed through the mixing process in the above technology. Specifically, the audio data corresponding to one window in the multi-windows and the audio data corresponding to other windows are mixed, and the mixed audio data of multiple windows is played simultaneously, resulting in confusion for users in terms of hearing and reducing the audio-visual experience of users.

[0091] Therefore, in the related art, only the audio data corresponding to one window in the multi-windows is retained for output, and the audio data corresponding to this window is played through the built-in speaker or an external audio output device, while the audio data corresponding to other windows is muted. However, the above method of only playing the audio data of a single window, although effectively reducing the mutual interference of the audio data between windows, makes users unable to hear the audio data of other windows, reducing the audio-visual experience of users in the multi-window display.

[0092] In view of this, an embodiment of the present application provides a display device and an audio control method. The audio control method is applied to the display device, which includes a display, an audio output interface, and a controller. The controller is communicatively connected to the display and the audio output interface. The audio output interface is configured to be communicatively connected to a target audio device, and the target audio device includes multiple audio devices for audio data output. The method further includes: under multi-window display, first stop the mixing process of each audio data; then, according to the determined target relationship, establish a channel for audio data transmission between the target window and the preset audio devices (i.e., the first preset audio device and the second preset audio device) to implement the transmission of audio data of different windows to different audio devices, thereby improving the audio-visual experience of the user under multi-window display.

[0093] The following describes the display device to which the audio control method provided by the embodiment of the present application is applied with reference to the accompanying drawings. Figure 3 Fig. shows a configuration block diagram of a display device 200 provided by an embodiment of the present application; as Figure 3 shown, the display device 200 includes a tuner-demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0094] It should be noted that Figure 3 the structure shown does not specifically limit the display device 200. In some other embodiments of the present application, the display device 200 may include more or fewer components than Figure 3 shown, or the display device 200 may include Figure 3 a combination of some of the components shown, or the display device 200 may include Figure 3 sub-components of some of the components shown. Figure 3 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0095] The display 260 includes a display screen component for presenting a picture and a driving component for driving image display, which is used to receive an image signal output from the controller and display video content, image content, a menu control interface component, and a user control UI interface.

[0096] The communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example: the communicator may include a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near-field communication protocol chips, as well as an infrared receiver.

[0097] For example, communicate with external audio output devices such as Bluetooth speakers and Bluetooth headsets through the Bluetooth module.

[0098] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0099] In some embodiments, the controller may include some or all of the following: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a random access memory (RAM), a read-only memory (ROM), a first interface to an nth interface for input / output, and a communication bus. Among them, different processors may be independent devices or integrated devices.

[0100] The audio processor is a device for processing and optimizing audio data and can perform functions such as mixing, equalizing, compressing, delaying, and noise reduction.

[0101] Before multi-window display, the audio processor can be used for mixing audio data. Multiple different audio data are mixed or superimposed through a mixer to form a new mixed audio data.

[0102] The audio output interface 270 can be a built-in speaker in the display device 200 or an external audio output device connected to the display device 200. The external audio output device is a device connected to the display device through different connection methods such as Bluetooth, USB cable, coaxial cable, etc., such as a Bluetooth speaker, a Bluetooth headset, a digital audio player, etc.

[0103] In some embodiments, the display device 200 may also be provided with an external audio output terminal, such as a 3.5 mm audio jack, an optical fiber audio interface, a coaxial audio interface, a USB interface, etc. The external audio output device can be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0104] An operating system runs on the display device 200, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system, etc. Application programs can be installed and run on the operating system.

[0105] The operating system of the display device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android operating system as an example to exemplify the system architecture of the display device 200.

[0106] It should be noted that although the embodiment of the present application is described using the Android operating system as an example, its basic principles are also applicable to display devices 200 of other operating systems.

[0107] Figure 4 A system architecture diagram of a display device 200 provided in an embodiment of the present application is shown. Figure 4 As shown, the system architecture of the display device 200 can be divided into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom, respectively, the application layer (Applications) layer (referred to as "application layer"), the application framework (Application Framework) layer (referred to as "application framework layer"), the Android runtime (Android runtime) and the system library layer (referred to as "system runtime layer"), the hardware abstraction layer (HAL) and the Linux kernel layer.

[0108] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0109] Among them, in the embodiment of the present application, the multi-window application running in the application layer can be used as a functional module in the operating system, integrated into the bottom layer of the operating system, and can also exist as an independent application; the multi-window application has its own independent running logic and interface.

[0110] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Applications can access system resources and obtain system services during execution through the API interface.

[0111] like Figure 4As shown in the figure, in the embodiment of the present application, the application framework layer includes managers, etc. Among them, the managers include an Audio Manager for managing and controlling audio functions, mainly for volume control (volume type, volume size, etc.), audio mode (output path of audio data, etc.), and audio device management (status of audio devices, such as connection, etc.).

[0112] The managers also include some or all of the following modules: an Activity Manager for interacting with all activities running in the system; a Location Manager for providing access to the system location service for system services or applications; a Package Manager for retrieving various information related to application packages currently installed on the device; a Notification Manager for controlling the display and clearing of notification messages; a Window Manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0113] In some embodiments, the application framework layer further includes server components responsible for processing and managing screen mirroring related functions. For example, a CastServer for performing screen mirroring listening, interruption, and management, etc.

[0114] After the application program involving screen mirroring is started or the smart TV is powered on, the CastServer will perform initialization preparations, such as starting to listen for the mirror or pushing the corresponding protocol service.

[0115] During the screen mirroring process, a session is first established between the client (such as a mobile phone, tablet, or other terminal devices) and the server (smart TV). The session contains various key information related to screen mirroring. For example, a screen mirroring identifier, screen mirroring type (such as mirroring, URL, etc.), information about the screen mirroring media assets (such as video stream width, video stream height, video name, URL address for pushing the video, metadata in the address, etc.).

[0116] After the session is established, the CastServer starts and registers a series of screen mirroring related listening functions. Specifically, it will monitor the connection and disconnection status of the session between the client and the server in real time, and also closely monitor changes in playback information and status, etc.

[0117] Thereafter, the encoded data is sent to the CastServer through the network. After receiving this data, the CastServer will perform a decoding operation and transmit the decoded audio and video data to the display screen, audio device, etc. of the display device, thereby achieving playback.

[0118] In some embodiments, the system runtime library layer provides support for the upper layer, i.e., the application framework layer. When the application framework layer is used, the Android operating system runs the C / C++ libraries included in the system runtime library layer to implement the functions to be achieved by the application framework layer.

[0119] The hardware abstraction layer is located above the Linux kernel layer. Its main function is to provide a unified and abstract hardware access interface for the upper-layer software; different hardware devices of the Android operating system (such as speakers, headphones, etc. produced by different manufacturers) can transmit audio data to the corresponding audio devices for playback through their respective corresponding HAL interfaces.

[0120] Among them, the primary HAL is responsible for managing and controlling key hardware such as cameras, sensors, audio devices, display devices, etc. For example, in terms of audio processing, the audio part in the primary HAL provides a unified audio device access interface for the system's audio services.

[0121] In the hardware abstraction layer, the audio path (AudioPatch) is a concept used to describe the transmission path of audio data in the system. It involves the complete connection link from the audio source (such as the audio output of an application) to the audio destination (such as speakers, headphones, etc.).

[0122] In some embodiments, the Linux kernel layer is the layer between hardware and software. As Figure 4 shown, the kernel layer includes audio drivers, display drivers, Bluetooth drivers, and power drivers, and may also include some or all of the following: camera drivers, WIFI drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), etc.

[0123] The execution subject of the audio control method provided in the embodiments of the present application can be the above-mentioned display device (such as a smart TV, etc.), or a functional module or functional entity in the display device that can implement the functions of the method, and this solution can be implemented through hardware and / or software, and in a combined manner, which can be specifically determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0124] Next, the embodiments of the present application will be illustrated by the following multiple exemplary embodiments in conjunction with the accompanying drawings. The methods in the following embodiments can all be implemented in the display device with the above-mentioned hardware structure and system architecture. The hardware structure diagram of the display device can be as Figure 3 shown, and the system architecture diagram of the display device can be as Figure 4 shown, but the embodiments of the present application are not limited to this.

[0125] For ease of explanation, in the embodiments of the present application, the display device is taken as a smart TV as an example. The smart TV can simultaneously display multiple windows (i.e., multi-windows), which provides convenience for users to handle multiple tasks simultaneously. The multi-window display can simultaneously display two windows, three windows, four windows, or more windows. And the multi-windows can be presented in one or a combination of display modes such as tiling, cascading, side-by-side, maximizing, and thumbnail.

[0126] Figure 5 FIG. shows a schematic diagram of a multi-window display application scenario provided by an embodiment of the present application. When the display 260 in the smart TV performs multi-window display, there are various display forms. It can be as shown in Figure 5 (a) therein, where window 31 and window 32 are tiled, window 31 and window 32 are evenly distributed on the display, and the window sizes are the same and do not overlap; it can be as shown in Figure 5 (b) therein, where window 33 and window 34 are side-by-side, the window sizes are different and do not overlap; it can also be as shown in Figure 5 (c) therein, where window 35 is full-screen, and window 36 is displayed on window 35 in the form of a small window or a floating window, and window 35 and window 36 overlap.

[0127] It should be understood that generally, the content presented in each window is different, and the form of the content can be multimedia resources (i.e., media assets) such as videos, pictures, music, files, games, etc. In the embodiments of the present application, when the media assets presented in at least two windows involve the output of audio data, the corresponding audio data output provides a more intuitive and immersive audio-visual experience for users. For example, Figure 5 (a) therein, window 31 plays video A, window 32 plays video B, and the smart TV can play the audio data corresponding to video A and the audio data corresponding to video B; for another example, Figure 5 (c) therein, window 35 plays video C, window 36 plays music A, and the smart TV can play the audio data corresponding to video C and the audio data corresponding to music A.

[0128] For ease of further understanding of the technical solutions in some embodiments of the present application, the technical solutions of the display device and the audio control method, and how the technical solutions solve the above technical problems will be described in detail below in combination with some specific embodiments and the accompanying drawings. The embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments.

[0129] The following will be combined with Figures 6 to 14 to describe in detail the audio control method provided by the embodiments of the present application. Figure 6The figure shows a schematic flowchart of an audio control method provided by an embodiment of the present application. This method is applied to a display device. Taking a smart TV as an example, as Figure 6 shown, the audio control method provided by the embodiment of the present application includes the following steps:

[0130] S510. In response to a first operation, enter an interface for multi-window display.

[0131] Among them, the first operation is used to trigger multi-window display and can be a start instruction for multi-windows.

[0132] In some embodiments, multi-window presentation is implemented in a single application. With the help of an application that supports multi-window display, multiple windows are displayed. For example, multiple windows play TV programs of different channels, or different media resources presented by screen mirroring or other means; in another embodiment, multi-window presentation is jointly implemented by multiple applications. For example, one window plays a movie with a video software, and another window plays short videos with a short video software.

[0133] It should be understood that during the use of a smart TV, there may be various forms of presenting multi-windows, and the ways to trigger multi-window display are also diverse, involving multiple channels and conditions. Therefore, the embodiment of the present application does not limit the first operation for triggering multi-window presentation.

[0134] Figure 7 The figure shows a schematic diagram of an application scenario of multi-window display provided by the embodiment of the present application. The application scenario of the embodiment of the present application is described. Taking the case where multi-windows are presented by a single application as an example, for the sake of convenience of description, in the embodiment of the present application, this single application is a split-screen application, which is not limited thereto.

[0135] Exemplarily, in response to a trigger instruction for long pressing the "main menu" key on the remote control, enter the display interface of the service center, as Figure 7 shown in (a); in the display interface of the service center, when the "split-screen movie viewing" or "split-screen screen mirroring" control is triggered, the multi-window display corresponding to the split-screen application can be entered; for example, when the "split-screen screen mirroring" control is triggered, enter the interface for multi-window display, as Figure 7 shown in (b). This interface is the first interface of the split-screen application; Figure 7 The multi-windows in (b) are used to play media resources screen mirrored from different terminal devices (such as mobile phones, tablets, etc.).

[0136] S520. Under multi-window display, stop the mixing process of the first audio data and the second audio data.

[0137] The first audio data and the second audio data are different audio data. For example, the number of channels of the first audio data is multi-channel, and the number of channels of the second audio data is stereo.

[0138] Exemplarily, for Figure 2 the target chip 10 in, CH1-CH4 are used to receive multi-channel audio data, that is, the first audio data; CH5 is used to receive two-channel audio data, that is, the second audio data. Stop the mixing process of the first audio data and the second audio data, that is to say, control the target chip 10 to stop the mixing process of the first audio data received by CH1-CH4 and the second audio data received by CH5.

[0139] It should be noted that in the embodiments of the present application, under multi-window display, stopping the mixing process of the first audio data and the second audio data indicates that the smart TV is in a state of supporting multiple audio data to be output through different audio devices.

[0140] It should be understood that in order to output the audio data corresponding to different windows in the embodiments of the present application through different audio devices, the smart TV needs to support multiple audio devices at the same time. The multiple audio devices may refer to a built-in speaker and at least one external audio output device, or multiple external audio output devices. Among them, the audio output device may be a Bluetooth speaker, a Bluetooth headset, etc.

[0141] It should also be understood that in the case where the multiple audio devices are implemented by multiple external audio output devices, the built-in speaker in the smart TV can still maintain the ability to output normal audio data.

[0142] Figure 8 Shows an example diagram of audio control in the smart TV provided by the embodiments of the present application. The audio devices include a built-in speaker and a Bluetooth headset. Among them, CH1-CH4 are used to receive the first audio data; CH5 is used to receive the second audio data.

[0143] The smart TV is in a state of supporting multiple audio data to be output through different audio devices. In this example, the audio devices currently supported by the smart TV are a built-in speaker and a Bluetooth headset.

[0144] It should be understood that in some embodiments, before entering multi-window display, the smart TV already supports multiple audio devices, such as a built-in speaker and a Bluetooth headset, and only outputs audio data through a single audio device. For example, if the audio data is output through the built-in speaker, the Bluetooth headset is muted. If the audio data is output through the Bluetooth headset, the built-in speaker is muted.

[0145] Therefore, under multi-window display, it is necessary to stop the mixing process of the first audio data and the second audio data, and it is also necessary to unmute some of the audio devices.

[0146] In one case, such as Figure 8As shown in (a) of [reference], for the first audio data received by CH1 - CH4, it is output through the Bluetooth headset; for the second audio data received by CH5, it is output through the built - in speaker. Among them, the mixing process of the first audio data received by CH1 - CH4 and the second audio data received by CH5 is stopped.

[0147] In another case, as Figure 8 shown in (b) of [reference], for the first audio data received by CH1 - CH4, it is output through the built - in speaker; for the second audio data received by CH5, it is output through the Bluetooth headset. Among them, the mixing process of the first audio data received by CH1 - CH4 and the second audio data received by CH5 is stopped.

[0148] That is to say, under multi - window display, there are various situations where the audio data corresponding to each window is output through different audio devices. Therefore, it is necessary to further determine whether the audio data (such as the first audio data) corresponding to a window that is being played or to be played is output from the built - in speaker or from the Bluetooth headset.

[0149] S530. Determine the correspondence between each target window and the target audio device to obtain the target relationship.

[0150] It should be understood that the audio data corresponding to the target window refers to the audio data being played by the target window or the audio data to be played by the target window. Different target window states result in different references to the corresponding audio data.

[0151] Under multi - window display, each target window includes the first window and the second window.

[0152] It should be understood that the number of the first windows and the number of the second windows can be one or more. For example, the number of the first windows is 1 and the number of the second windows is 1; the number of the first windows is 1 and the number of the second windows is 3; the number of the first windows is 2 and the number of the second windows is 2.

[0153] In some embodiments, the target window further includes a third window and even more.

[0154] As Figure 7 shown in (b) of [reference], the first interface of the split - screen application is the main interface for two - way screen mirroring. Among them, the first window is the left window 41 and the second window is the right window 42. In this example, the media resources played by the left window 41 and the right window 42 are presented through the screen mirroring technology. Among them, both the left window and the right window display information such as the guiding picture, the Wi - Fi name accessed by the smart TV, and the name of the smart TV, which is convenient for users to perform screen mirroring.

[0155] It should be understood that screen mirroring technology includes wired screen mirroring and wireless screen mirroring. Among them, wireless screen mirroring includes Wi-Fi screen mirroring. Wi-Fi screen mirroring is based on the Wi-Fi network to achieve the transmission of pictures and sounds between devices. For example, mirror screen mirroring and DLNA screen mirroring.

[0156] Among them, DLNA screen mirroring is a function implemented according to the Digital Living Network Alliance (DLNA) technology. The media assets on the terminal device are pushed to the smart TV in the form of a Uniform Resource Locator (URL) for playback.

[0157] The target audio device refers to an audio device that is communicatively connected to the smart TV through an audio output interface. The target audio device is used for audio data output. There are multiple target audio devices, including a first target audio device and a second target audio device. For example, Figure 8 the audio device shown in (a) in includes a built-in speaker and a Bluetooth headset.

[0158] In some embodiments, the target audio device further includes a third target audio device, and even more.

[0159] The target relationship is one of the following corresponding relationships: First, the corresponding relationship between the first window and the first target audio device, and the second window and the second target audio device; Second, the corresponding relationship between the first window and the second target audio device, and the second window and the first target audio device.

[0160] When the target audio device includes a built-in speaker and a Bluetooth headset, Figure 7 in the example shown in (b) in, the target relationship is one of the following corresponding relationships: First, the corresponding relationship between the left window and the built-in speaker, and the right window and the Bluetooth headset; Second, the corresponding relationship between the left window and the Bluetooth headset, and the right window and the built-in speaker.

[0161] It should be noted that the determination of the corresponding relationship (i.e., the target relationship) between the target window and the target audio device depends on the analysis of the number of channels of the audio data corresponding to the target window.

[0162] For example, when the number of channels of the audio data corresponding to the left window is multi-channel, the audio data corresponding to the left window belongs to the first audio data. In the case where the audio data (i.e., the first audio data) corresponding to the left window is output through the built-in speaker, combined with Figure 8 shown in (b) in, it is determined that the target relationship is the corresponding relationship between the left window and the built-in speaker, and the right window and the Bluetooth headset; In the case where the audio data (i.e., the first audio data) corresponding to the left window is output through the Bluetooth headset, combined with Figure 8As shown in (a) therein, determine the target relationship as the corresponding relationship between the left window and the Bluetooth headset, and between the right window and the built-in speaker.

[0163] For another example, when the number of channels of the audio data corresponding to the left window is stereo, the audio data corresponding to the left window belongs to the second audio data. In the case where the audio data (i.e., the second audio data) is output through the built-in speaker in the left window, in combination with Figure 8 As shown in (a) therein, determine the target relationship as the corresponding relationship between the left window and the built-in speaker, and between the right window and the Bluetooth headset; in the case where the audio data (i.e., the second audio data) is output through the Bluetooth headset in the left window, in combination with Figure 8 As shown in (b) therein, determine the target relationship as the corresponding relationship between the left window and the Bluetooth headset, and between the right window and the built-in speaker.

[0164] S540. According to the target relationship, respectively establish the audio output channels between each target window and the target audio device.

[0165] It should be understood that the determination of the target relationship depends on the analysis of the number of channels of the audio data corresponding to the target window. Therefore, the number of channels of the audio data corresponding to the first window and the number of channels of the audio data corresponding to the second window are implied in the determined target relationship.

[0166] That is to say, according to the target relationship, and the number of channels of the audio data corresponding to the first window and the number of channels of the audio data corresponding to the second window in the target relationship, establish the audio output channels between each target window and the target audio device.

[0167] For example, establish the audio output channel between the first window and the first target audio device, and the audio output channel between the second window and the second target audio device, or establish the audio output channel between the first window and the second target audio device, and the audio output channel between the second window and the first target audio device.

[0168] In some embodiments, the first preset audio device is the first target audio data or the second target audio data, and the second preset audio device is a target audio device different from the first preset audio device. That is to say, if the first preset audio device is the first target audio data, then the second preset audio device is the second target audio data; if the first preset audio device is the second target audio data, then the second preset audio device is the first target audio data.

[0169] That is to say, according to the target relationship, and the number of channels of the audio data corresponding to the first window and the number of channels of the audio data corresponding to the second window in the target relationship, establish the first audio output channel between the first window and the first preset audio device, and the second audio output channel between the second window and the second preset audio device.

[0170] For example, when the number of channels of the audio data corresponding to the first window is multi-channel (i.e., the audio data corresponding to the first window belongs to the first audio data), the audio data corresponding to the first window is transmitted to the first pin for receiving the first audio data, and the audio data corresponding to the second window is transmitted to the second pin for receiving the second audio data (the number of channels is stereo). An audio output channel for the audio data of the first window to be transmitted to the first preset audio device through the first pin and an audio output channel for the audio data of the second window to be transmitted to the second preset audio device through the second pin are established.

[0171] For another example, when the number of channels of the audio data corresponding to the first window is stereo (i.e., the audio data corresponding to the first window belongs to the second audio data), the audio data corresponding to the first window is transmitted to the second pin for receiving the second audio data, and the audio data corresponding to the second window is transmitted to the first pin for receiving the first audio data (the number of channels is multi-channel). An audio output channel for the audio data of the first window to be transmitted to the first preset audio device through the second pin and an audio output channel for the audio data of the second window to be transmitted to the second preset audio device through the first pin are established.

[0172] For each window, a channel between the window and the corresponding audio device is established so that the audio data corresponding to the window can be output through the corresponding audio device; for example, in the example of (b) in Figure 7 the left window 41 is the first window and the right window 42 is the second window; when the number of channels of the audio data corresponding to the first window (i.e., the left window 41) is multi-channel, the audio data corresponding to the left window 41 (i.e., the first audio data) is transmitted to the CH1-CH4 pins shown in (a) in Figure 8 and shown in (b) in Figure 8 and the audio data corresponding to the right window 42 (i.e., the second audio data) is transmitted to the CH5 pin shown in (a) in Figure 8 and shown in (b) in Figure 8

[0173] When the first preset audio device is a built-in speaker and the second preset audio device is a Bluetooth speaker, referring to (b) in Figure 8 control the first audio data received by the CH1-CH4 pins to be transmitted to the built-in speaker, and the second audio data received by the CH5 pin to be transmitted to the Bluetooth headset.

[0174] That is to say, establishing a first audio output channel between the left window 41 and the built-in speaker and a second audio output channel between the right window 42 and the Bluetooth headset can bring a more flexible and personalized audio experience to the user.

[0175] In one scenario, different usage requirements of users are met. For example, user A can simultaneously enjoy external audio through the built-in speaker in the left window to meet the sharing or convenient listening needs in an open environment; use a Bluetooth headset in the right window to achieve private listening and avoid disturbing others, greatly improving the freedom of audio use and scene adaptability.

[0176] In another scenario, the usage requirements of different users are met. For example, when user B watches a video in the left window, the video sound will be played through the built-in speaker for convenient direct listening; when user C watches a video in the right window, the video sound will be played through a Bluetooth headset to achieve private audio playback; and user B and user C do not interfere with each other and watch the video as needed.

[0177] In the embodiments of the present application, in multi-window display, the mixing process of each audio data is first stopped; then, according to the determined target relationship, a channel for audio data transmission is established between the target window and the preset audio devices (i.e., the first preset audio device and the second preset audio device) to implement the transmission of audio data of different windows to different audio devices, thereby improving the audio-visual experience of users in multi-window display.

[0178] It should be understood that before the first operation, the smart TV usually outputs the mixed audio data through a single audio device. That is to say, before the first operation, the smart TV mixes the first audio data and the second audio data to obtain mixed data and outputs the mixed data to a single audio device to play the sound.

[0179] In order to provide users with a coherent and comfortable audio-visual experience, after entering multi-window display, the mixed data is output to the corresponding audio device as the first preset audio device corresponding to the first window.

[0180] Generally, the first window is the main window that is mainly displayed in multi-window display.

[0181] It should be understood that in the scenario of two windows being displayed side by side, such as Figure 5 shown in (a) of Figure 5 shown in (b) of

[0182] It should be understood that the first preset audio device is the audio device corresponding to the output of the mixing data before the smart TV responds to the first operation, and in the embodiments of the present application, the smart TV supports multiple audio devices at the same time, that is, the target audio device. Therefore, the first preset audio device is the first target audio device, or the first preset audio device is the second target audio device.

[0183] For example, before the first operation, the smart TV usually outputs the mixing data through the built-in speaker. After entering the multi-window display, the built-in speaker is used as the first preset audio device corresponding to the first window.

[0184] It should be noted that the smart TV can send query instructions to the target audio device adapted to it through different connection methods such as Bluetooth, 3.5mm audio interface, HDMI interface, etc. With these interaction instructions, the smart TV can accurately determine the connection status of the target audio device and obtain relevant information of the target audio device, including the name, volume, channel mode, etc. of the audio device, which is convenient for the management of the target audio device. In the embodiments of the present application, the query or monitoring methods corresponding to different connection methods are not limited.

[0185] In some embodiments, in order to improve the accuracy of the determined first preset audio device, when entering the multi-window display, a query or monitoring of information such as the connection status of the target audio device will be triggered. If the result of the query or monitoring indicates that the audio device corresponding to the mixing data is still in the connected state, the audio device corresponding to the mixing data will be used as the first preset audio device.

[0186] For example, before entering the multi-window display, the audio device is a Bluetooth speaker, and after entering the multi-window display, the Bluetooth speaker is in the connected state, then the first preset audio device corresponding to the first window is the Bluetooth speaker.

[0187] For another example, before entering the multi-window display, the audio device is the built-in speaker, and after entering the multi-window display, the first preset audio device corresponding to the first window is the built-in speaker. It should be understood that in the normal operation state of the smart TV, the built-in speaker can continuously and stably output normal audio data. Therefore, in this scenario, after entering the multi-window display, the query or monitoring of information such as the connection status of the target audio device may not be triggered.

[0188] In some embodiments, the connection status, device information, etc. of the target audio device can be stored in the audio device list. When querying or monitoring that the target audio device adapted to the smart TV has changed (such as a change in the connection status, the audio device is in standby, etc.), the audio device list is updated to facilitate the query and management of the target audio device.

[0189] It should be understood, referring to Figure 8 in (a), Figure 8As shown in (b) in [the context], during the process of controlling the audio data transmitted to each audio device, CH1-CH4 are used to receive the first audio data; CH5 is used to receive the second audio data. Therefore, in order to determine the target relationship between each window in the multi-window and the corresponding audio device, it is also necessary to determine the number of channels corresponding to each window. Figure 9 Fig. [the figure number] shows a schematic flowchart of another audio control method provided by an embodiment of the present application. For the above step 530, to determine the corresponding relationship between each target window and the target audio device to obtain the target relationship, as Figure 9 shown, the following steps are further included:

[0190] S531. Determine that the audio device corresponding to the first window is the first preset audio device, and the number of channels of the audio data corresponding to the first window is the first number of channels.

[0191] Among them, the first preset audio device is determined by the audio device output from the mixing data before entering the multi-window display.

[0192] It should be understood that the smart TV supports audio data with multiple channels and two channels. In order to enable the audio data of the first window and the second window to be output through different audio devices respectively, it is necessary to make the number of channels of the audio data of the first window and the number of channels of the audio data of the second window different.

[0193] That is to say, when the first number of channels is multiple channels, adjust the number of channels of the audio data corresponding to the second window to two channels to obtain the second number of channels. When the first number of channels is two channels, adjust the number of channels of the audio data corresponding to the second window to multiple channels to obtain the second number of channels. Among them, the second number of channels is the number of channels of the audio data corresponding to the second window.

[0194] The number of channels of the audio data corresponding to the first window can be multiple channels or two channels. In order to improve the efficiency of audio data processing, the determination of the first number of channels can be divided into the following three cases: (1)-(3):

[0195] (1) Before entering the multi-window display, the smart TV is playing the first media asset, and after entering the multi-window interface, continue to play the first media asset in the first window to keep the continuous playback of the first media asset.

[0196] In this case, determine the first number of channels according to the number of channels of the audio stream corresponding to the first media asset.

[0197] It should be understood that before entering the multi-window display, the audio device of the smart TV playing the first media asset is used as the first preset audio device. In this scenario, the first window maintains the number of channels of the audio device playing the first media asset and the audio data in the first media asset, reducing the data processing process; at the same time, it makes the first media asset displayed full-screen before entering the multi-window continuous with the first media asset displayed in the first window, and the played sound effects remain unchanged, providing users with a coherent and comfortable audio-visual experience.

[0198] (2) Before and after entering the multi-window display, when no media asset is being played, determine the first number of channels according to the preset number of channels.

[0199] Before entering the multi-window display, the smart TV is not playing any media asset. After entering the multi-window display, it still does not play any media asset, but instead displays the interface shown in (b) of Figure 7 , the first interface of the split-screen application, also known as the boot interface, where the left and right windows in the boot page are displayed in equal proportions.

[0200] Therefore, when there is no media asset being played in the first window and no media asset to be played, the first number of channels can be determined through the preset number of channels.

[0201] It should be understood that the smart TV supports playing audio data with multiple different numbers of channels simultaneously, and the played sound. The preset number of channels supported by the smart TV includes multi-channel and stereo, and the first number of channels can be flexibly determined to be stereo or multi-channel through random allocation, or based on historical data, user preferences, etc.

[0202] It should be understood that after determining the first number of channels, if the window receives a media asset to be played, the audio stream of the media asset needs to be adjusted to the corresponding first number of channels.

[0203] For example, if the determined first number of channels is stereo and the number of channels in the audio stream of the media asset to be played in the first window is multi-channel, the audio stream of the media asset to be played needs to be adjusted to stereo.

[0204] (3) Before entering the multi-window display, when no media asset is being played, and after entering the multi-window display, when a second media asset is to be played in the first window, determine the first number of channels according to the number of channels of the audio stream corresponding to the second media asset.

[0205] When the number of channels of the audio stream corresponding to the second media asset is clear and unique, determine this number of channels as the first number of channels.

[0206] In some embodiments, there may be multiple coexisting numbers of channels for the audio stream corresponding to the second media asset. The first number of channels can be determined according to the screen mirroring method corresponding to the second media asset. For example, if the second media asset for screen mirroring is pushed in the form of a URL, the number of channels of the audio data corresponding to the first window can be set to multi-channel; if the second media asset for screen mirroring is through mirror screen mirroring, the number of channels of the audio data corresponding to the first window can be set to stereo.

[0207] It should be understood that after screen mirroring, in the initially transmitted audio data (such as MP4 format or AAC format), the first few frames of this audio data will contain source format data in a pre-agreed format. Decoding these few frames of data can obtain information such as the encoding format, sampling rate, and number of channels of the pushed source; therefore, for the second media asset to be played, the number of channels of the audio data of the second media asset can be obtained through its corresponding audio data, and the number of channels of the audio data corresponding to the first window can be determined from the number of channels of the audio data of the second media asset.

[0208] S532. Determine the first correspondence between the first window and the first preset audio device according to the first number of channels.

[0209] When the first number of channels of the audio data corresponding to the first window is multi-channel, the audio data corresponding to the first window belongs to the first audio data. When the first audio data is output through the first preset audio device in the first window, determine the first correspondence between the first window and the first preset audio.

[0210] When the first number of channels of the audio data corresponding to the first window is stereo, the audio data corresponding to the first window belongs to the second audio data. When the second audio data is output through the first preset audio device in the first window, determine the first correspondence between the first window and the first preset audio.

[0211] S533. Determine the second correspondence between the second window and the second preset audio device.

[0212] When the first number of channels of the audio data corresponding to the first window is multi-channel, the number of channels of the audio data corresponding to the second window is stereo, and the audio data corresponding to the second window belongs to the second audio data; when the second audio data is output through the second preset audio device in the second window, determine the second correspondence between the first window and the first preset audio.

[0213] When the first number of channels of the audio data corresponding to the first window is stereo, the number of channels of the audio data corresponding to the second window is multi-channel, and the audio data corresponding to the second window belongs to the first audio data; when the first audio data is output through the second preset audio device in the second window, determine the second correspondence between the first window and the first preset audio.

[0214] It should be understood that the target relationship includes a first corresponding relationship and a second corresponding relationship.

[0215] Figure 10 FIG. shows a schematic diagram of another application scenario of multi-window display provided by an embodiment of the present application, which is based on (b) in Figure 7 and further shows a schematic diagram after receiving the media assets cast on the screen. The process presented from (b) in Figure 7 to (a) in Figure 10 corresponds to the situation described in the above case (3). After entering multi-window display, when the media asset A (i.e., the second media asset) to be played is in the left window 41, the number of channels of the audio data corresponding to the left window 41 is determined to be the first number of channels according to the number of channels of the audio stream corresponding to the media asset A; furthermore, the number of channels of the audio data corresponding to the second window is determined to be the second number of channels. Then, according to the fact that the number of channels of the audio data corresponding to the left window 41 is the first number of channels, the first corresponding relationship between the left window and the first preset audio device is determined, and according to the fact that the number of channels of the audio data corresponding to the right window is the second number of channels, the second corresponding relationship between the right window and the second preset audio device is determined.

[0216] Furthermore, through the target relationship, and the first number of channels and the second number of channels involved in the target relationship, a first audio output channel between the left window and the first preset audio device, and a second audio output channel between the right window and the second preset audio device are established. Then, the media asset A is played first in the left window 41 (i.e., the main window), as shown in (a) in Figure 10 ; furthermore, after receiving the screen cast of the new media asset B, the media asset B is played in the right window 42, as shown in (b) in Figure 10 .

[0217] In an embodiment of the present application, multi-window display can be implemented in multiple scenarios. In each scenario, first, the first preset audio device corresponding to the first window, and the first number of channels of the audio data corresponding to the first window are determined; since the audio data of the first window and the second window are output through different audio devices, the second preset audio device corresponding to the second window, and the second number of channels of the audio data corresponding to the second window can be determined; furthermore, the determined target relationship; to achieve the transmission of the audio data of different windows to different audio devices, this process improves the data processing efficiency in the smart TV and provides a coherent and comfortable audio-visual experience for the user.

[0218] In a possible case, Figure 11 FIG. shows a system architecture diagram of another display device provided by an embodiment of the present application, as shown in Figure 11As shown in the figure, the system architecture diagram of the display device includes a service module and an audio processing module. The service module includes a first application for multi-window display, such as a split-screen application. The audio processing module is used to control the target chip to process audio data, such as mixing processing and so on.

[0219] Among them, the service module is in the application layer of the system architecture, and the audio processing module is in the kernel layer of the system architecture.

[0220] Among them, AudioTrack is the main API (Application Programming Interface) for audio playback in the Android operating system. It can send audio data into the audio processing process of the system and finally play it through the audio device. For example, in a video playback application, AudioTrack can be used to play the decoded audio data (such as decoding MP3 into PCM audio data).

[0221] AudioTrack supports multiple audio formats, such as Pulse Code Modulation (PCM) format. PCM is a basic encoding method for converting analog audio signals into digital signals. It can handle different sampling rates (such as common 44.1kHz, 48kHz, etc.), channel numbers (stereo, multi-channel, etc.) and sample formats (such as 8bit, 16bit, etc.).

[0222] It should be understood that AudioTrack can also control the volume of the audio stream. It sends the audio data in the application to the audio device for playback, and during this process, it can adjust the volume of the audio data. For example, the volume control can be achieved by changing the sampling value of the audio data. In digital audio, the volume is proportional to the amplitude of the audio sampling value. By amplifying or reducing the sampling value, the volume of the audio can be changed.

[0223] Among them, AudioFlinger is the audio processing service in the Android operating system. It is mainly responsible for managing and processing operations such as routing and playback of audio data.

[0224] The application sends the audio data to AudioFlinger through the audio APIs (such as AudioTrack, MediaPlayer, etc.) of the Android operating system. After AudioFlinger processes these data, it then passes the data to the audio hardware through the audio driver for playback.

[0225] It should be understood that audio data is output through a Bluetooth connection. The corresponding audio data is obtained by controlling AudioPatch, and then this path of audio data is output to a Bluetooth headset or the like.

[0226] For the sake of convenience of description, Figure 11 also reflects the hardware layer that interacts with the above system structure. The hardware layer provides physical hardware support for the operating system. All software functions in the operating system rely on the support of the hardware layer to be effectively implemented. The hardware layer and the kernel layer interact through a driver to ensure smooth information interaction and precise control of physical hardware.

[0227] In the embodiments of the present application, it mainly involves built-in speakers in the hardware layer, audio output devices (such as Bluetooth speakers, Bluetooth headsets, etc.).

[0228] When the display device runs a split-screen application, according to the scenarios before and after entering multi-window display, determine the audio device corresponding to the main window and the number of channels for outputting audio data, and then determine the target relationship. Control the audio processing module to establish channels according to the target relationship. For example, realize the transmission of audio data corresponding to the first window to the built-in speaker, and the transmission of audio data corresponding to the second window to the Bluetooth headset, so as to realize the output of audio data corresponding to different windows through different audio devices.

[0229] Next, through Figure 12 The embodiments shown will be described in detail.

[0230] Figure 12 shows a schematic flowchart of another audio control method provided by the embodiments of the present application. As Figure 12 shown, the method includes:

[0231] S610. In response to a first operation, the service module generates a trigger message.

[0232] Among them, the trigger message is used to instruct the audio processing module to stop the mixing process of the first audio data and the second audio data.

[0233] In some embodiments, the trigger message is used to instruct the audio processing module to switch the control of the target chip from the mixing mode to the target mode; among them, in the mixing mode, the first audio data and the second audio data are mixed, and the mixed data after mixing is transmitted to a single audio device, which can be the first preset audio device or the second preset audio device; in the target mode, the mixing process of the first audio data and the second audio data is stopped, but the first audio data is transmitted to the first preset audio device, and the second audio data is transmitted to the second preset audio device.

[0234] Specifically, the trigger message is used to instruct the audio processing module to control the target chip to switch fromFigure 2 The pattern shown in (a) in Figure 8 is switched to (a) in Figure 8 or the pattern shown in (b) in

[0235] It should also be understood that in response to the first operation, the interface of the over-window display will also be entered.

[0236] S620. According to the preset message passing policy, the service module will trigger the message transmission to the audio processing module.

[0237] Among them, the preset message passing policy includes broadcasting, cross-process communication or encapsulated interface.

[0238] Broadcasting is a global message notification mechanism that allows applications to send and receive messages within the system. In some embodiments, in response to the first operation, the trigger message generated by the service module can be transmitted to the audio processing module in the form of broadcasting.

[0239] It should be understood that cross-process communication can be implemented through Android Interface Definition Language (AIDL), and AIDL is based on Binder. Through AIDL, communication between different applications can be achieved, such as the interaction between system services and applications, and between components of different applications.

[0240] In some embodiments, the interaction between the service module and the audio processing module can be implemented through AIDL, so that the service module transmits the trigger message to the audio processing module.

[0241] The encapsulated interface refers to the encapsulated native private interface. Among them, the native private interface is the interface used inside the Android operating system, and these native private interfaces can be packaged in some ways to make them safer and more controllable to use in the application. In some embodiments, the service module can transmit the trigger message to the audio processing module through the encapsulated interface. Through the encapsulated interface, for the application scenario of the embodiments of the present application, it can be better adapted and controlled through customization to meet its required functions and interactions.

[0242] In the application scenario, the preset message passing policy can be selected according to compatibility requirements, development and maintenance costs, etc., so that the service module transmits the trigger message to the audio processing module.

[0243] In some embodiments, the trigger message may include a first audio configuration parameter and a second audio configuration parameter. Among them, the first audio configuration parameter is used to indicate the parameter for transmitting the audio data corresponding to the first window to the audio processing module, and the second audio configuration parameter is used to indicate the parameter for transmitting the audio data corresponding to the second window to the audio processing module.

[0244] Among them, the first audio configuration parameter includes the identifier of the first preset audio device and the number of the first channels; the second audio configuration parameter includes the identifier of the second preset audio device and the number of the second channels.

[0245] For example, the first audio configuration parameter and the second audio configuration parameter are transmitted to the audio processing module in the kernel layer of the system architecture through the AudioManager.setParameters interface; for example, the first audio configuration parameter and the second audio configuration parameter are transmitted to the audio processing module by implementing AudioManager.setParameters(speaker+mch) and AudioManager.setParameters(bt_e+2ch), where speaker represents that the identifier of the first preset audio device is the identifier of the built-in speaker, mch represents that the number of the first channels is multi-channel, bt_e represents that the identifier of the second preset audio device is the identifier of the Bluetooth headset, and 2ch represents that the number of the first channels is dual-channel.

[0246] It should be understood that AudioManager is a class used to manage audio-related functions in the Android operating system, which can be used for the setting of audio devices, querying the status of the current audio device, and controlling the volume of the audio device, etc.

[0247] In some embodiments, in an application scenario where multi-screen projection is used to achieve multi-window display, the number of channels in the audio stream corresponding to the first screen projection can be used to determine the number of the first channels of the audio data output by the first window; furthermore, according to the identifier of the first preset audio device corresponding to the first window and the number of the first channels, the first audio configuration parameter corresponding to the first window is determined, and according to the identifier of the second preset audio device corresponding to the second window and the number of the second channels, the second audio configuration parameter corresponding to the second window is determined.

[0248] In some embodiments, the trigger message may not include the first audio configuration parameter and the second audio configuration parameter, and these parameters will be transmitted separately after the trigger message is transmitted.

[0249] In some embodiments, in the audio processing module, the first audio configuration parameter or the second audio configuration parameter will be further parsed and passed. Among them, the first parameter is used to determine the preset audio device for outputting the audio data, and the second parameter is used to determine the number of channels for playing the audio data corresponding to the preset audio device.

[0250] It should be understood that when there is the first parameter, it can be determined that in the application scenario of multi-window display, the audio data corresponding to different numbers of channels is supported to be output through different audio devices.

[0251] For example, the first parameter is output_device, and the second parameter is acap_source; when output_device = bt and acap_source = 2ch, it means that the audio data of two channels is output through the Bluetooth audio device; when output_device = bt and acap_source = mch, it means that the audio data of multiple channels is output through the Bluetooth audio device.

[0252] S630. The audio processing module stops mixing the first audio data and the second audio data.

[0253] S640. The audio processing module determines the corresponding relationship between each target window and the target audio device according to the number of channels of the audio data corresponding to the target window in the multiple windows, and obtains the target relationship.

[0254] Among them, the audio device corresponding to the target window and the number of channels of the audio data are sent to the audio processing module by the service module. For example, the first audio configuration parameter corresponding to the first window and the first audio configuration parameter corresponding to the second window are sent to the audio processing module through AudioManager.setParameters(speaker + mch) and AudioManager.setParameters(bt_e + 2ch).

[0255] After receiving the first audio configuration parameter and the second audio configuration parameter, the audio processing module determines the first corresponding relationship according to the first audio configuration parameter; determines the second corresponding relationship according to the second audio configuration parameter, and further obtains the target corresponding parameter according to the first corresponding parameter and the second corresponding parameter.

[0256] For example, when the audio processing module receives speaker + mch, it determines that the audio data corresponding to the first window belongs to the first audio data (the number of channels of the first audio data is multiple channels), and determines the first corresponding relationship between the first window and the built-in speaker when the first audio data is output through the built-in speaker in the first window. When the audio processing module receives bt_e + 2ch, it determines that the audio data corresponding to the second window belongs to the second audio data (the number of channels of the second audio data is two channels), and determines the second corresponding relationship between the second window and the Bluetooth headset when the first audio data is output through the Bluetooth headset in the second window.

[0257] S650. The audio processing module establishes a first audio output channel between the first window and the first preset audio device, and a second audio output channel between the second window and the second preset audio device according to the target relationship.

[0258] It should be understood that the audio processing module includes a first audio input interface and a second audio input interface. The first audio input interface is used to receive multi-channel audio data, and the second audio input interface is used to receive stereo audio data.

[0259] When the number of channels of the first channel is multi-channel, the audio data corresponding to the first window is transmitted to the first audio input interface, and the audio data corresponding to the second window is transmitted to the second audio input interface; furthermore, according to the first correspondence, the audio data of the first window is transmitted through the first audio input interface to the first audio output channel of the first preset audio device; according to the second correspondence, the audio data of the second window is transmitted through the second audio input interface to the second audio output channel of the second preset audio device, as Figure 13 shown in (a) of

[0260] In addition, when the number of channels of the first channel is stereo, the audio data corresponding to the first window is transmitted to the second audio input interface, and the audio data corresponding to the second window is transmitted to the first audio input interface; furthermore, according to the first correspondence, the audio data of the first window is transmitted through the second audio input interface to the first audio output channel of the first preset audio device; according to the second correspondence, the audio data of the second window is transmitted through the first audio input interface to the second audio output channel of the second preset audio device, as Figure 13 shown in (a) of

[0261] S660. Transmit the audio data corresponding to the first window to the first preset audio device through the first audio output channel, and transmit the audio data corresponding to the second window to the second preset audio device through the second audio output channel.

[0262] Through the established first audio output channel and second audio output channel, the audio data corresponding to the first window can be played through the first preset audio device, and the audio data corresponding to the second window can be played through the second preset audio device.

[0263] In the embodiment of the present application, under multi-window display, in response to the first operation, the service module generates a trigger message and transmits the trigger message to the audio processing module, so that the audio processing module can stop the mixing process of each audio data and determine the target relationship; then, according to the determined target relationship, establish a channel for audio data transmission between the target window and the preset audio device (i.e., the first preset audio device and the second preset audio device), so as to realize the transmission of audio data of different windows to different audio devices, and further improve the audio-visual experience of the user under multi-window display.

[0264] In Figure 10 such as (a) of Figure 10In the embodiment corresponding to the application scenario presented in (b) of [], after step 650 and before executing step 660, the following steps (1)-(2) are further included:

[0265] (1) Under multi-window display, in response to a first screen mirroring instruction, play a first target media asset in a first window, and play the audio data of the first target media asset through a first audio output channel; and establish a session list.

[0266] Among them, the first screen mirroring instruction is used to indicate, under the multi-window display interface, when receiving an instruction for the first media asset to be played corresponding to the first path of screen mirroring (i.e., the first target media asset); that is to say, the first target media asset is to be played in the first window (main window).

[0267] It should be understood that the first window is the main window in the multi-window display interface. Therefore, when receiving the screen mirroring instruction for the first time, play the audio data of the first target media asset played in the first window through the first audio output channel, and a session list will be established.

[0268] It should be understood that the session list is used to record the attribute information of each terminal device mirroring the screen to the smart TV. Among them, the session list includes the attribute information of the first window, and the attribute information includes the terminal device information mirroring the screen to the display device and the display position information, where the display position information is used to indicate the layout of the target window in the multi-window display interface.

[0269] In some embodiments, the attribute information further includes sound information.

[0270] For example, the terminal device information is the identifier of the terminal device, and the display position information includes the display position corresponding to the upper left corner of the window, the width and height of the window, etc.

[0271] (2) In response to a second screen mirroring instruction, play a second target media asset in a second window, and play the audio data of the second target media asset through a second audio output channel; and update the session list.

[0272] Among them, the second screen mirroring instruction is used to indicate, under the multi-window display interface, when receiving an instruction for the second media asset to be played corresponding to the second path of screen mirroring (i.e., the second target media asset); that is to say, the second target media asset is to be played in the second window.

[0273] It should be understood that when receiving the screen mirroring instruction for the second time, play the audio data of the second target media asset played in the second window through the second audio output channel, and update the session list. Among them, the updated session list includes the attribute information of the first window and the attribute information of the second window.

[0274] In a multi-window display, in response to a first screen mirroring instruction, a first target media asset is played in a first window, and the audio data of the first target media asset is played through a first audio output channel. In response to a second screen mirroring instruction, a second target media asset is played in a second window, and the audio data of the second target media asset is played through a second audio output channel. During this period, a session list is established and updated. In the application scenario of screen mirroring and displaying multiple windows, the audio data of different windows is transmitted through corresponding channels to an audio device for playback, achieving the goal of outputting multiple windows from multiple audio devices. Furthermore, the audio-visual experience of the user in a multi-window display is improved.

[0275] During the presentation of multi-window content, the audio output device corresponding to each window can be dynamically adjusted. The user can perform modification operations such as switching and replacing the audio device. In this scenario, it can be achieved through methods such as a remote control, body buttons, voice input, and playback control buttons. For the sake of easy understanding, the switching of the audio device of the first window is taken as an example for description.

[0276] In response to an audio switching instruction for the first window, a list of audio devices currently supported by the display device is presented; in response to a selection instruction for the audio device list, the first preset audio device corresponding to the first window is switched to the third preset audio device corresponding to the selection instruction.

[0277] Among them, the third preset audio device can be the same audio device as the second preset audio device or a different audio device from the second preset audio device.

[0278] It should be understood that if the third preset audio device is the same audio device as the second preset audio device, the second preset audio device corresponding to the second window needs to be switched, or the user is reminded and guided to make the switch.

[0279] Figure 14 Shows a schematic diagram of another application scenario provided by an embodiment of the present application. It is in Figure 10 the scenario shown in (b) in, a schematic diagram of further switching the audio device corresponding to each window through methods such as a remote control, body buttons, voice input, and playback control buttons. In Figure 10 the scenario presented in (b) in, taking the control of the playback control button as an example, the audio device corresponding to the first window is switched.

[0280] Such as Figure 14As shown in (a) in [reference], a set of broadcast control buttons corresponding to the window is displayed below the left window 41. For example, the set of broadcast control buttons successively includes volume down (the first broadcast control button), volume up (the second broadcast control button), audio output device (the third broadcast control button), play / pause (the fourth broadcast control button), full screen display (the fifth broadcast control button), hide window (the sixth broadcast control button), and close window (the seventh broadcast control button).

[0281] As Figure 14 shown in (a) in [reference], the third broadcast control button in the set of broadcast control buttons is the built-in speaker button 51, indicating that the audio device corresponding to the window is the built-in speaker; by selecting the built-in speaker button 51, a list 61 of audio devices currently supported by the smart TV is displayed; as Figure 14 shown in (b) in [reference], the user can select and switch the audio device in the list 61 of audio devices, such as switching the audio device corresponding to the left window from the built-in speaker to a Bluetooth speaker; as Figure 14 shown in (c) in [reference], the third broadcast control button in the set of broadcast control buttons below the left window 41 is the Bluetooth speaker button 52, indicating that the audio device corresponding to the window is the Bluetooth speaker.

[0282] It should be understood that if the audio devices of the left window 41 after switching and the right window 42 are the same audio device, a prompt message can be generated to inform the user that the audio devices of the two windows conflict, or the audio device of the right window 42 can be switched to a different audio device from the left window 41.

[0283] It should be understood that the audio device of the second window can also be directly switched. For the implementation process, reference can be made to the detailed description of the operation of the first window above, which will not be elaborated here.

[0284] It should also be understood that in some embodiments, if the connection of the currently used audio device is interrupted, etc., resulting in no sound in the window corresponding to the audio data output through the audio device, within a preset time period, wait for the audio device to reconnect, or in the case of non-reconnection, remind the user whether to automatically switch to other playable audio devices, or display a list of audio devices for the user to select independently.

[0285] The above examples mainly describe how to dynamically switch the control of the audio output device. However, in the scenario of multi-window display, there are also cases where the window content or the number of channels changes. For example, when two video contents are played simultaneously or the screen mirroring content changes, how to control the audio output device of the screen mirroring content on both sides. For example, the media asset A is presented in the left window and the media asset B is presented in the right window, and they can be replaced by the media asset D, etc.

[0286] That is to say, there is a situation where the user replaces or closes the first target media asset or the second target media asset being played. In some embodiments, it further includes updating the attribute information of the first window in the session list when it is monitored that the first target media asset being played in the first window is replaced or closed; and updating the attribute information of the second window in the session list when it is monitored that the second target media asset being played in the second window is replaced or closed.

[0287] It should be understood that in some embodiments, when it is monitored that the first target media asset being played in the first window is replaced or closed, or when it is monitored that the second target media asset being played in the second window is replaced or closed, the attribute information of the first window and the attribute information of the second window in the session list are updated.

[0288] For example, in Figure 10 the scenario presented in (b) below, if the user switches the screen mirroring of the second window to the mirror of the mobile phone, at this time, if it still Figure 10 displays the interface presented in (b) below, the experience brought to the user is poor. At this time, the Figure 10 interface presented in (b) below can be switched to Figure 5 the interface presented in (b) below, which is achieved by updating the attribute information of the first window and the attribute information of the second window in the session list.

[0289] It should be understood that in the process of implementing the above audio control, there may be more modules involved. The embodiments of the present application do not limit the types and uses of the involved modules.

[0290] In the process of the above-mentioned various modules executing audio control, the establishment of each window and the corresponding audio device in the multi-window display of the display device is realized, and the independent output of the audio data corresponding to each window is realized, without interference with each other, improving the user's audio-visual experience.

[0291] During the use of the smart TV, according to the user's needs, mutual switching between multiple windows and other interfaces will be performed. For example, switching from multiple windows to full screen, and switching from full screen to multiple windows; since the requirements for the corresponding audio devices (such as speakers, external audio output devices) are different in the above-mentioned different interface display scenarios, therefore, in the scenario of switching between multiple windows and other interfaces, the audio output device and display, etc. after the interface switch can be dynamically adjusted according to the audio device and display configuration before the interface switch, so that the user can maintain a stable audio-visual and visual experience during the process of switching back and forth between different display interfaces.

[0292] As Figure 14 shown in (a) below, the corresponding broadcast control button group of the window is displayed below the left window 41. For example, full screen display (the fifth broadcast control button), hide window (the sixth broadcast control button), disconnect playback (the seventh broadcast control button).

[0293] Among them, the fifth playback control button is used to display the left window 41 in full screen. At this time, the right window 42 will be closed.

[0294] It should be understood that when the multi-window includes three or four windows, if one of the windows is controlled to be displayed in full screen, the other windows will all be closed, which helps the system to recycle and manage resources, reduce unnecessary resource occupation, and thus affect the overall performance of the system.

[0295] In some embodiments, the left window can be displayed in the form of a full-screen window, a full-screen interface, etc.

[0296] Among them, the sixth playback control button is used to hide the left window 41 and display the right window 42 in full screen.

[0297] It should be understood that when the multi-window includes three or four windows, if one of the windows is controlled to be hidden, the other windows can preset a multi-window layout strategy to rearrange and display the other windows to improve the visual effect of the display.

[0298] The seventh playback control button is used to close the left window 41, then the playback of the left window is disconnected, and the main interface in the first interface shown in (b) in the left window is displayed, and the other windows maintain the original display. Figure 7 Among them, the main interface in the first interface shown in (b) in the left window is displayed, and the other windows maintain the original display.

[0299] In such a process, in order to maintain a stable auditory and visual experience and ensure that the corresponding audio data can be accurately output from the audio device expected by the user, Figure 15 The flowchart of a multi-window control method provided by an embodiment of the present application is shown. It should be understood that, Figure 15 The multi-window control method is a further control based on the above audio control method. Therefore, the multi-window control method is a control under multi-window display to determine the audio data corresponding to the target window and output it from the preset audio device corresponding to the target window. Among them, the preset audio device is one of the target audio devices, and the preset audio device includes a first preset audio device and a second preset audio device.

[0300] Such as Figure 15 As shown, the multi-window control method includes the following steps:

[0301] S710. Under multi-window display, based on the target audio device corresponding to each window, output the audio data of the corresponding window.

[0302] For the implementation process of step 710, reference can be made to the detailed description of the above steps 510 to 540 and steps 610 to 660, which will not be repeated here.

[0303] It should be understood that in the application scenario of multi-window display corresponding to screen mirroring, a session list will also be established and updated, where the session list is used to record the attribute information of each terminal device for screen mirroring to the display device.

[0304] S720. In response to a hide instruction for the first window, adjust the target audio device corresponding to the first window to mute output, and maintain the normal output of the target audio device corresponding to the second window.

[0305] It should be understood that before receiving the hide instruction for the first window, the target audio device corresponding to the first window outputs normally. That is to say, the audio data played corresponding to the first window is transmitted to the target audio device, and the sound is played through the target audio device.

[0306] Each target window in the multi-windows includes the first window and the second window, where the first window or the second window may refer to one or more windows.

[0307] And, for the sake of convenience in description, referring to the above embodiments, the target audio device corresponding to the first window is the first preset audio device, and the target audio device corresponding to the second window is the second preset audio device. For the detailed description of the target window and the preset audio device, reference can be made to the above, and details are not described here again.

[0308] That is to say, in response to the hide instruction for the first window, adjust the first preset audio device corresponding to the first window to mute output, and maintain the normal output of the second preset audio device corresponding to the second window.

[0309] In response to the hide instruction for the first window, control the target audio devices corresponding to each target window (adjust the target audio device corresponding to the first window to mute output, and maintain the normal output of the target audio device corresponding to the second window). In order to synchronously improve the user's visual experience, a hide operation will also be performed on the first window.

[0310] It should be understood that in response to the hide instruction for the first window, adjust the first preset audio device corresponding to the first window to mute output, and perform a hide operation on the first window. That is to say, in response to the hide instruction for the first window, both the display state of the first window and the playback state of the first preset audio device change. In the scenario where the first window is hidden, adjust the first preset audio device from the normal playback state to the mute state, and hide the first window, bringing visual and auditory matching and consistency to the user, thereby improving the user's audio-visual experience.

[0311] It should be understood that the hide operation on the first window can be achieved by the following method: adjust the first display parameter of the first window to obtain the adjusted first target parameter; and based on the display area size and display position in the first target parameter, display the adjusted first window.

[0312] Among them, the display area size in the first target parameter is smaller than the display area size in the first display parameter. The display area size in the first display parameter can be obtained from the attribute information in the session list.

[0313] The first piece of attribute information in the session list includes: Session:SessionClient1, Position:(0, 50), Width:400, Height:1000. It can be seen that the display position in the first display parameter is the coordinate of (0, 50), and the display area size in the first display parameter is 400px × 1000px, where px is the abbreviation of pixel.

[0314] The display area in the adjusted first target parameter is smaller than the display area in the first display parameter. Therefore, the display area in the first target parameter can be 1px × 1px, can be 2px × 5px, or can also be 4px × 10px, etc.

[0315] Generally, when the size of the display area corresponding to the first target parameter is adjusted to be smaller, or even adjusted to be very small, such as 1px × 1px, the occupation of system resources can be reduced.

[0316] Figure 16 Fig. shows an application scenario diagram of window hiding control in multiple windows provided by an embodiment of the present application. The multiple-window display includes two windows, as Figure 16 shown in (a) of, the left window 41 and the right window 42 are displayed; for the playback control button group of the left window 41, among them, the sixth playback control button is the hide button 53, which represents hiding this window and not displaying it. By selecting the hide button 53, the first preset audio device corresponding to the left window 41 is adjusted from the normal playback state to the mute state, and the left window 41 is hidden, as Figure 16 shown in (b) of, the display area size of the first window is adjusted to be smaller, and the adjusted first window 62 is displayed.

[0317] It should be understood that when the display area size of the first window is adjusted to be smaller, such as adjusted to 1px × 1px, although the user is not sensitive to the display of this first window, when the contrast between this window and the second window or the background is high, it will still bring a poor visual experience to the user.

[0318] Therefore, in some embodiments, taking the hiding of the first window as an example above, the hiding of the first window can be realized by the following methods (1)-(3):

[0319] (1) When the display position in the first target parameter is the first position outside the visible area corresponding to the display, the adjusted first window is displayed.

[0320] Among them, the display size area in the first target parameter is less than or equal to the first area, and the first area is the area between the first position and the visible area.

[0321] Figure 17 Fig. shows another application scenario diagram of window hiding control in multiple windows provided by the embodiments of the present application. For Figure 16 the adjustment of the display position of the first window 62 shown in (b) therein, at this time, the adjusted first window 63 is displayed based on the first position P1, where the first position P1 is a position outside the visible area corresponding to the display, and the display size area of the adjusted first window 63 is less than the first area A1 determined between the first position P1 and the visible area.

[0322] It should be understood that if the display size area in the first target parameter is greater than the first area, a part of the adjusted first window will enter the visible area, thus affecting the user's visual experience.

[0323] Figure 18 Fig. shows another application scenario diagram of window hiding control in multiple windows provided by the embodiments of the present application. For Figure 17 an example after adjusting the display size area of the first window 63 shown, as Figure 18 shown, if the display size area in the first target parameter corresponding to the adjusted first window 64 is greater than the first area A1, the user can see a part of the first window 64 in the visible area, thus affecting the user's visual experience.

[0324] (2) When the display position in the first target parameter is the second position corresponding to the interface control, display the adjusted first window, where the display size area in the first target parameter is less than the area corresponding to the interface control.

[0325] In some embodiments, there will be interface controls in the interface of the display. For example, the icon of the application program is displayed in the upper left corner of the interface; the time, network connection status, etc. are displayed in the upper right corner of the interface, etc.

[0326] When there are interface controls in the interface, the adjusted first window can be displayed at the second position corresponding to the interface control, and the display size area in the first target parameter corresponding to the adjusted first window is less than the area corresponding to the interface control.

[0327] It should be understood that the second position is the position corresponding to the interface control, and this second position is below the interface control. Furthermore, the adjusted first window presents a hidden visual effect. During the normal use process, the user's eyes usually focus on the interface control, and the adjusted first window is blocked by the interface control and is not easily noticed by the user, thus achieving the hidden effect.

[0328] (3) When the display position in the first target parameter is the third position in the second layer, the adjusted first window is displayed, where the second layer is the layer below the first layer where the second window is located.

[0329] In some embodiments, the displayed interface includes multiple layers, and different types of content are displayed in different layers, such as a background layer, an information graphic layer, an interactive layer, a displayed image, etc.

[0330] In the case where the first window is hidden, the second window remains in the display state. Therefore, the display layer of the second window is called the first layer. To hide the first window, the display position of the first window is set to the third position in the second layer below the first layer. If the first layer is opaque and there is no opaque area, the third position can be any position in the second layer.

[0331] In some embodiments, if there is a transparent area in the first layer, the third position belongs to the second area in the second layer, where the second area refers to the area in the second layer corresponding to the opaque area of the first layer.

[0332] By one of the above methods, the first window is hidden and displayed, which is convenient for the user to focus more on the content in other windows (such as the second window), and the user can't feel the existence of the first window at all. It can not only optimize the system performance and save memory, but also create an immersive experience and achieve seamless transition. At the same time, it is convenient for background tasks to run and the system to be maintained and updated.

[0333] To improve the user's audio-visual experience after the first window is hidden, the second display parameter corresponding to the second window can also be adjusted to obtain the adjusted second target parameter, and based on this second target parameter, the adjusted second window is displayed.

[0334] Among them, the second display parameter includes the display position and the display area size, and the second target parameter also includes the display position and the display area size.

[0335] In some embodiments, in the scenario where the first window is hidden, if the number of windows involved in the second window is 1, the second display parameter of the second window is adjusted to full-screen display, and the second window is full-screen displayed while maintaining the normal output of the audio device corresponding to the second window (i.e., the second preset audio device).

[0336] Figure 19 Shows another application scenario diagram of window hiding control in multiple windows provided by the embodiments of the present application. The multiple-window display includes two windows, such as Figure 19As shown in (a) therein, the left window 41 and the right window 42 are displayed; for the playback control button group of the left window 41, among which, the sixth playback control button is the hide button 53, which indicates hiding this window and no longer displaying it; by selecting this hide button 53, the left window 41 is hidden, the first preset audio device corresponding to the left window 41 is adjusted from the normal playback state to the mute state; and the right window 42 is adjusted to be displayed as the full-screen window 43, as Figure 19 shown in (b) therein.

[0337] In some other embodiments, in the scenario where the first window is hidden, if the number of windows involved in the second window is greater than 1, it is necessary to re-layout each window in the second window to improve the user's visual experience. The second display parameters of each window in the second window can be adjusted according to the preset multi-window layout strategy; based on the adjusted second display parameters of each window in the second window, each window in the second window is re-arranged, and the normal output of the audio devices corresponding to each window in the second window is maintained.

[0338] Figure 20 Fig. shows an application scenario diagram of window hiding control in multiple windows provided by an embodiment of the present application. There are four windows in the multiple-window display, as Figure 20 shown in (a) therein, the upper-left window 44, the upper-right window 45, the lower-left window 46 and the lower-right window 47 are displayed; for the playback control button group of the upper-left window 44, among which, the sixth playback control button is the hide button 54, which indicates hiding this window and no longer displaying it; by selecting this hide button 54, the upper-left window 44 is hidden, the first preset audio device corresponding to the upper-left window 44 is adjusted from the normal playback state to the mute state; and the displays of the three windows of the upper-right window 45, the lower-left window 46 and the lower-right window 47 are re-arranged, and the normal outputs of the audio devices corresponding to the upper-right window 45, the lower-left window 46 and the lower-right window 47 are maintained respectively, as Figure 20 shown in (b) therein.

[0339] For Figure 20 the display interface shown in (a) therein, among which, the audio data corresponding to the upper-left window 44 is transmitted to the built-in speaker for output, the audio data corresponding to the upper-right window 45 is transmitted to the Bluetooth headset for output, the audio data corresponding to the lower-left window 46 is transmitted to the Bluetooth speaker for output, and the audio data corresponding to the lower-right window 47 is transmitted to the USB speaker for output. In Figure 20 the display interface shown in (b) therein, the built-in speaker is muted, and the normal outputs of the Bluetooth headset, the Bluetooth speaker and the USB speaker are maintained.

[0340] In some embodiments, multiple windows can correspond to one audio device. For example Figure 20In the display interface shown in (a) therein, media asset C is played in the upper left window 44, media asset D is played in the upper right window 45, media asset E is played in the lower left window 46, and the media asset F played in the lower right window 47 belong to the display windows corresponding to different players in the same game. At this time, the audio devices of the lower left window 46 and the lower right window 47 can be set to the same audio device.

[0341] When multiple windows correspond to one audio device, if one of the windows is hidden, the audio device corresponding to that window is muted. When it is detected that other windows also output audio data through this audio device, the normal output of this audio device is restored.

[0342] Figure 21 It shows another application scenario diagram of window hiding control in multiple windows provided by the embodiment of the present application, which is a further explanation of the interface shown in (b) in Figure 20 As shown, its interface includes a first layer 71 and a second layer 72. The first layer 71 is located above the second layer 72. Among them, the rearranged upper right window 45, lower left window 46, and lower right window 47 are displayed in the first layer 71, and the adjusted upper left window 66 is displayed in the second layer 72. Figure 21 As shown, its interface includes a first layer 71 and a second layer 72. The first layer 71 is located above the second layer 72. Among them, the rearranged upper right window 45, lower left window 46, and lower right window 47 are displayed in the first layer 71, and the adjusted upper left window 66 is displayed in the second layer 72.

[0343] It should be understood that the adjusted upper left window 66 is displayed in the second layer 72 at a position corresponding to the lower right window 47 in the first layer 71. Furthermore, the hiding of the upper left window and the rearrangement of the upper right window, lower left window, and lower right window are realized.

[0344] In the application scenario where the first window is hidden in step 720, the multi-window display of the original first window and the second window can be restored. Figure 22 It shows a schematic flowchart of another multi-window control method provided by the embodiment of the present application. As shown in Figure 22 As shown, after step 720, the multi-window control method further includes:

[0345] S730. In response to the restoration instruction of the multi-window display, adjust the target audio device corresponding to the first window from muted output to normal output.

[0346] Restore the function of the target audio device corresponding to the first window to play audio data normally. It should be understood that when the target audio device is in muted output, it will play the audio data corresponding to the first window, but the played volume is set to 0.

[0347] S740. Adjust the first target parameter of the first window and the second target parameter of the second window to obtain an updated session list.

[0348] In addition to restoring the target audio device, the display of the first window and the second window also needs to be restored. Therefore, the first target parameter of the first window and the second target parameter of the second window need to be adjusted to adapt to the multi-window display before the first window is hidden.

[0349] The session list can be updated by the number of windows in the multi-window display corresponding to the restoration instruction and the layout method corresponding to the number of windows in the preset multi-window layout strategy.

[0350] In some embodiments, the first display parameter of the first window and the second display parameter of the second window can be stored. When responding to the restoration instruction of the multi-window display, the stored first display parameter of the first window and the second display parameter of the second window can be directly called to adjust the first target parameter of the first window and the second target parameter of the second window, and then, the session list is updated.

[0351] S750. Based on the attribute information in the updated session list, the first window and the second window are displayed simultaneously.

[0352] In some embodiments, the first display parameter of the first window and the second display parameter of the second window can be stored. When responding to the restoration instruction of the multi-window display, the stored first display parameter of the first window and the second display parameter of the second window can be directly called to display the first window and the second window simultaneously.

[0353] Exemplarily, Figure 23 FIG. shows an application scenario diagram of full-screen window restoration control provided by an embodiment of the present application. As shown in (a) in Figure 23 In the interface where the right window is full-screen displayed in the full-screen window 43, the playback control controls such as "play mode", "multiple play", and "more" can be controlled through the remote control. By selecting "more", the playback control control "split-screen display" can be displayed. After "split-screen display" is selected, a multi-window page as shown in (b) in Figure 23 is displayed.

[0354] It should be understood that during the switching process from (a) to (b) in the above Figure 23 there is also the regulation of the corresponding target audio device. Operations such as restoring the normal output of the corresponding target audio device cannot be intuitively represented by the visual elements of the interface, but the existence of its process brings an auditory experience to the user. Figure 23

[0355] Figure 14 ​​As shown in (a) in [reference], the playback control button group corresponding to the window is displayed below the left window 41, and it also includes disconnecting playback (the seventh playback control button). Therefore, in some embodiments, in order to maintain a stable auditory and visual experience and ensure that the corresponding audio data can be accurately output from the audio device expected by the user, after outputting the audio data of the corresponding window based on the target audio device corresponding to each window in step 710, the multi-window control method further includes the following steps (1)-(3):

[0356] (1) In response to a disconnection instruction for the first window, disconnect the screen mirroring connection corresponding to the first window.

[0357] The media resource played in the first window is screen mirroring played by the terminal device. In response to the selection of the seventh playback control button (disconnect playback), disconnect the screen mirroring connection corresponding to the played media resource.

[0358] For example, Figure 24 shows an application scenario diagram of disconnection control under multiple windows provided by an embodiment of the present application. As shown in (a) in Figure 24 it shows the left window 41 and the right window 42. The media resource A played in the left window 41 is played after being screen mirrored from the smart phone A to the smart TV; for the playback control button group of the left window 41, among them, the seventh playback control button is the disconnection playback button 55, which represents disconnecting the screen mirrored media resource played in this window. By selecting the disconnection playback button 55, the screen mirroring connection corresponding to the first window is disconnected.

[0359] (2) Delete the attribute information corresponding to the first window in the session list.

[0360] The session list is used to record the attribute information of screen mirroring from each terminal device to the display device, which also includes the specific window of screen mirroring and the display parameters corresponding to the window, etc. Therefore, after disconnecting the screen mirroring connection corresponding to the first window, it is necessary to delete the attribute information corresponding to the first window from the session list.

[0361] (3) Maintain the display interfaces of the first window and the second window.

[0362] It should be understood that after the screen mirroring connection corresponding to the first window is disconnected, it does not affect the display of the first window. At this time, the display interface of multiple windows is still displayed. For example, it includes the first window and the second window.

[0363] It should be understood that in the scenario where the screen mirroring connection corresponding to the first window is disconnected, the normal output of the target audio devices corresponding to the first window and the second window is maintained.

[0364] Since the screen mirroring connection corresponding to the first window is disconnected, the media resource played in the first window cannot be obtained. Therefore, the first window is not playing this media resource. Therefore, the following content can be displayed in the first window:

[0365] First, display information such as a guiding picture, the Wi-Fi name accessed by the smart TV, and the name of the smart TV in the first window, and restore the display before screen mirroring, as shown in (b) of Figure 24 . It should be understood that since no media assets are played in the first window, there is no audio data to be played on the target audio device corresponding to the first window.

[0366] Second, display a background picture, a background video, etc. in the first window. It should be understood that if a background picture is played in the first window, there is no audio data to be played on the target audio device corresponding to the first window; if a background video is played in the first window, the audio data corresponding to the background video is transmitted to the target audio device corresponding to the first window.

[0367] Third, display an advertisement in the first window. If the advertisement has audio data, the audio data corresponding to the advertisement is transmitted to the target audio device corresponding to the first window.

[0368] Fourth, display information related to the media assets in the second window in the first window, such as a plot introduction, actor information, etc. At this time, there is no audio data to be played on the target audio device corresponding to the first window.

[0369] In the embodiment of the present application, under multi-window display, in response to a disconnection instruction for the first window, the screen mirroring content of one of the windows is disconnected, meeting the user's needs. At the same time, network and system resources can be released, the device performance can be optimized, the display layout can be made more reasonable, the viewing and interaction experience of other windows can be improved, and at the same time, connection management is facilitated and the stability of device operation is enhanced.

[0370] It should be understood that as shown in (a) of Figure 14 , a set of broadcast control buttons corresponding to this window is displayed below the left window 41, and it also includes full-screen display (the fifth broadcast control button). Therefore, in some embodiments, in order to maintain a stable auditory and visual experience and ensure that the corresponding audio data can be accurately output from the audio device expected by the user, after outputting the audio data of the corresponding window based on the target audio device corresponding to each window in step 710, the multi-window control method further includes the following steps (1)-(2):

[0371] (1) In response to a full-screen instruction for the first window, maintain the normal output of the audio data corresponding to the first window through the target audio device, and adjust the first window to full-screen display.

[0372] For example, Figure 25 shows an application scenario diagram of full-screen control under multi-windows provided by an embodiment of the present application, as shown in Figure 25As shown in (a) of , the left window 41 and the right window 42 are displayed. The media asset A played in the left window 41 is cast from the smart phone A to the smart TV and then played. For the playback control button group of the left window 41, among them, the fifth playback control button is the full-screen display button 56, which represents full-screen display of this window. By selecting the full-screen display button 56, the first window is full-screen displayed, and the normal output of the audio data corresponding to the first window through the target audio device is maintained, such as Figure 25 as shown in (b) of .

[0373] (2) Disconnect the screen mirroring connection corresponding to the second window and delete the attribute information corresponding to the second window in the session list.

[0374] It should be understood that the media assets played in the first window and the second window are both cast and played by the terminal device. In response to the selection of the fifth playback control button (full-screen display) corresponding to the first window, the screen mirroring connection corresponding to the media asset played in the second window will be disconnected. After disconnecting the screen mirroring connection corresponding to the second window, it is necessary to delete the attribute information corresponding to the second window from the session list.

[0375] It should be understood that after the screen mirroring connection corresponding to the second window is disconnected, no audio data is transmitted to the target audio device corresponding to the second window.

[0376] In the embodiment of the present application, in the case of multi-window display, when one window is full-screen displayed, the screen mirroring content of the other window will be disconnected, which can concentrate system resources to render the full-screen window and improve its picture quality and running fluency; at the same time, the display layout is optimized to avoid interference from other screen mirroring content and the output of audio data, allowing users to focus on the full-screen window and obtain a more immersive viewing experience.

[0377] In some embodiments, in the scenario of multi-window display, it is also possible to exit the multi-window display through control devices such as a remote control and a terminal device. At this time, the multi-window control method further includes the following process: in response to the exit instruction of the multi-window, resume the mixing process of the first audio data and the second audio data, and output the mixed audio data to a single audio device; and switch from multi-window display to full-screen display.

[0378] In multi-window display, based on the output of corresponding audio data by different target audio devices for each window, accurate matching of audio devices and windows can be achieved, meeting the user's demand for different audio device outputs for different windows.

[0379] When a hide instruction for the first window is received, adjust the corresponding target audio device to mute output to avoid interference from the hidden window audio to the user's audio-visual experience of the currently focused window content; and since the hidden window is no longer in the user's field of view, the continuous playback of its corresponding audio data will distract the user's attention.

[0380] Maintaining the normal output of the target audio device corresponding to the second window can ensure that the user can focus on the visible content of the second window, guarantee the coherence and effectiveness of audio playback, and enable the user to have a smoother audio-visual experience.

[0381] For the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0382] In the present application, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0383] It should be understood that in various embodiments of the present application, the magnitudes of the sequence numbers of the above processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0384] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0385] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0386] In several embodiments provided by this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is only a logical function division, and there can be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0387] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0388] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0389] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A display device, characterized in that, Including: A display; An audio output interface configured to communicatively connect with a plurality of target audio devices for outputting audio data; A controller configured to: Under multi-window display, output audio data of a corresponding window based on the target audio device corresponding to each window; In response to a hide instruction for a first window, adjust the target audio device corresponding to the first window to mute output and maintain the normal output of the target audio device corresponding to a second window, where the first window and the second window are windows under the multi-window display.

2. The display device according to claim 1, wherein When the controller adjusts the target audio device corresponding to the first window to mute output and maintains the normal output of the target audio device corresponding to the second window, it is further configured to: Perform a hide operation on the first window; Adjust second display parameters corresponding to the second window to obtain adjusted second target parameters, where the second target parameters include a display position and a display area size.

3. The display device according to claim 2, characterized in that, The controller's performing a hide operation on the first window is configured to: Adjust first display parameters of the first window to obtain adjusted first target parameters, where the display area size in the first target parameters is smaller than the display area size in the first display parameters; Based on the display area size and the display position in the first target parameters, display the adjusted first window.

4. The display device according to claim 3, characterized in that, The controller's displaying the adjusted first window based on the display area size and the display position in the first target parameters is configured to: When the display position in the first target parameters is a first position outside the visible area corresponding to the display, display the adjusted first window, where the display size area in the first target parameters is less than or equal to a first area, and the first area is the area between the first position and the visible area.

5. The display device according to any one of claims 2 to 4, characterized in that, The controller's adjusting second display parameters of the second window is further configured to: When the number of windows involved in the second window is greater than 1, adjust the second display parameters of each window in the second window; Arrange each window in the second window based on the adjusted second display parameters of each window in the second window.

6. The display device according to any one of claims 2 to 4, characterized in that, The controller's adjusting second display parameters of the second window is further configured to: When the number of windows involved in the second window is equal to 1, adjust the second display parameters of the second window to full-screen display; Full-screen display the second window.

7. The display device according to claim 6, characterized in that, When the controller full-screen displays the second window, it is further configured to: In response to a resume instruction for multi-window display, adjust the target audio device corresponding to the first window from mute output to normal output; Adjust the first target parameters of the first window and the second target parameters of the second window to obtain an updated session list, where the session list is used to record attribute information of each terminal device casting a screen to the display device; Based on the attribute information in the updated session list, display the first window and the second window simultaneously.

8. The display device according to any one of claims 1 to 4, characterized in that After the controller outputs audio data of a corresponding window based on the target audio device corresponding to each window, it is further configured to: In response to a disconnection instruction for the first window, disconnect the screen mirroring connection corresponding to the first window, and delete the attribute information corresponding to the first window in the session list, where the session list is used to record the attribute information of screen mirroring from each terminal device to the display device; Maintain the display interfaces of the first window and the second window.

9. The display device according to any one of claims 1 to 4, characterized in that, After the controller outputs the audio data of the corresponding window based on the target audio device corresponding to each window, it is further configured to: In response to a full-screen instruction for the first window, maintain the normal output of the target audio device corresponding to the first window, and adjust the first window to full-screen display; Disconnect the screen mirroring connection corresponding to the second window, and delete the attribute information corresponding to the second window in the session list, where the session list is used to record the attribute information of screen mirroring from each terminal device to the display device.

10. A multi-window control method, characterized in that, Applied to the display device according to any one of claims 1 to 9, the multi-window control method includes: Under multi-window display, output the audio data of the corresponding window based on the target audio device corresponding to each window; In response to a hide instruction for the first window, adjust the target audio device corresponding to the first window to mute output, and maintain the normal output of the target audio device corresponding to the second window, where the first window and the second window are windows under the multi-window display.

Images