A display device and a method for adjusting video display sequence

Abstract

The present application provides a display device and a method for adjusting the video display order. The method can extract the identification information of the video call window whose display order is adjusted after the user inputs a control instruction for adjusting the display order, and subscribe to the call data stream of the corresponding window according to the identification information, so as to replace the display content of the video call window with the call data stream according to the display order customized by the user. The method can enable multiple display devices participating in a multi-person video conversation to freely sort the video call windows according to user needs, so that different users can control the video display order according to different conversation needs, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a method for adjusting a video display sequence. Background Art

[0002] Display devices refer to terminal devices that can output specific display images, which can be terminal devices such as smart TVs, mobile terminals, smart advertising screens, projectors, etc. Taking smart TVs as an example, smart TVs are based on Internet application technology, have open operating systems and chips, have open application platforms, can realize two-way human-computer interaction functions, and are TV products that integrate multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users.

[0003] Some display devices may have built-in or external image acquisition devices and / or audio input devices, which can obtain the user's video images and sound audio in real time through the image acquisition device and audio input device, and realize functions such as video calls and video conferencing with other display devices based on network connection. Due to the needs of specific video usage scenarios, the display device can adjust the video display order. For example, during a video conference, important guests need to be displayed in the front position of the participant window, while ordinary participants are set in the back position.

[0004] In order to adjust the video display order, users can implement order adjustment control through the conference background, that is, they can customize the window display order in the conference background. However, this adjustment method will affect all participants after execution, that is, the video display order of all participants will be displayed according to the order set in the background, and other display devices cannot be customized separately. Summary of the invention

[0005] The present application provides a display device and a method for adjusting the video display order to solve the problem that the traditional method for adjusting the video display order does not support individual customization of each display device.

[0006] In one aspect, the present application provides a display device, comprising: a display, a communicator, and a controller. The display is configured to display a user interface, wherein the user interface includes a plurality of video call windows displayed in an initial order; the communicator is configured to connect to a server; and the controller is configured to execute the following program steps:

[0007] Obtaining a control instruction input by a user for adjusting the display order of video call windows;

[0008] In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction;

[0009] Acquiring a call data stream of the target window according to the target window identification information;

[0010] According to the customized order, the call data stream is used to replace the display content of the video call window in the user interface.

[0011] On the other hand, the present application also provides a video display order adjustment method, which is applied to the above-mentioned display device, and the video display order adjustment method comprises the following steps:

[0012] Obtaining a control instruction input by a user for adjusting the display order of video call windows;

[0013] In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction;

[0014] Acquiring a call data stream of the target window according to the target window identification information;

[0015] According to the customized order, the call data stream is used to replace the display content of the video call window in the user interface.

[0016] As can be seen from the above technical solutions, the display device and video display order adjustment method provided by the present application can extract the identification information of the video call window whose display order is adjusted after the user inputs the control instruction for adjusting the display order, and subscribe to the call data stream of the corresponding window according to the identification information, so as to replace the display content of the video call window with the call data stream according to the user-defined display order. The method can enable multiple display devices participating in a multi-person video conversation to freely sort the video call windows according to user needs, so that different users can control the video display order according to different conversation needs, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 It is a usage scenario of the display device in the embodiment of the present application;

[0019] Figure 2 This is a hardware configuration block diagram of the control device in the embodiment of the present application;

[0020] Figure 3A hardware configuration diagram of a display device in an embodiment of the present application;

[0021] Figure 4 A software configuration diagram of a display device in an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of a multi-person call scenario in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a process for displaying a multi-person call interface in an embodiment of the present application;

[0024] Figure 7 This is a schematic diagram of a multi-person call interface in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a process for setting an initial sequence in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of a process for automatically adjusting the display order in an embodiment of the present application;

[0027] Fig.10 This is a flow chart of a method for adjusting the video display order in an embodiment of the present application;

[0028] Fig.11 This is a schematic diagram of the position of the mobile window in the embodiment of the present application;

[0029] Fig.12 A schematic diagram of a process for generating a custom sequence in an embodiment of the present application;

[0030] Fig.13 This is a schematic diagram of clicking on window layout options in an embodiment of the present application;

[0031] Fig.14 This is a schematic diagram of the window layout interface in the embodiment of the present application;

[0032] Fig.15 This is a schematic diagram of the window setting interface in the embodiment of the present application;

[0033] Fig.16 This is a schematic diagram of the user list interface in the embodiment of the present application;

[0034] Fig.17 This is a schematic diagram of the adjusted multi-person call interface in the embodiment of the present application;

[0035] Fig.18 This is a flow chart of replacing a call data flow in an embodiment of the present application;

[0036] Fig.19 This is a timing diagram of a video conferencing scenario in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0038] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0039] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0040] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0041] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0042] Figure 1 FIG. 1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0043] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, etc.

[0044] In some embodiments, a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

[0045] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0046] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0047] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200.

[0048] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown.

[0049] In some embodiments, the display apparatus 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0050] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output.

[0051] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface.

[0052] In some embodiments, the display 260 may be a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0053] In some embodiments, the communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0054] In some embodiments, the user interface may be used to receive a control signal from the control device 100 (eg, an infrared remote controller, etc.).

[0055] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0056] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0057] In some embodiments, the tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0058] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0059] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0060] In some embodiments, the object may be any one of the selectable objects, such as a hyperlink, an icon or other operable controls. Operations related to the selected object include: displaying a page, document, image, etc. connected to the hyperlink, or executing a program corresponding to the icon.

[0061] In some embodiments, the controller includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0062] CPU processor. Used to execute operating system and application instructions stored in the memory, and to execute various applications, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video content. CPU processor may include multiple processors. For example, it includes a main processor and one or more sub-processors.

[0063] In some embodiments, the graphics processor is used to generate various graphic objects, such as icons, operation menus, and user input command display graphics. The graphics processor includes an operator, which performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes; and a renderer, which renders various objects obtained based on the operator, and the rendered objects are used to be displayed on the display.

[0064] In some embodiments, the video processor is used to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal to obtain a signal that can be directly displayed or played on the display device 200.

[0065] In some embodiments, the video processor includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc. Among them, the demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling processing. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to the user input or itself with the video image after scaling processing to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to receive the video output signal after frame rate conversion and change the signal to conform to the display format signal, such as outputting RGB data signal.

[0066] In some embodiments, the audio processor is used to receive an external audio signal, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion, and amplification processing according to a standard codec protocol of the input signal to obtain a sound signal that can be played in a speaker.

[0067] In some embodiments, the user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0068] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0069] In some embodiments, the system of the display device may include a kernel, a command parser (shell), a file system, and an application. The kernel, shell, and file system together constitute the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. After startup, the application is compiled into machine code to form a process.

[0070] See also Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0071] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0072] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0073] like Figure 4As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0074] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0075] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0076] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0077] The display device 200 can also realize social functions, that is, different users can realize video calls or audio calls through different display devices 200. In order to realize the social function, the display device 200 can have a built-in or external image acquisition device and / or an audio input device. Figure 5As shown, a camera may be provided on the display device 200 to collect image information of the user in front of the display device 200. A microphone may also be provided on the display device 200 to collect voice information of the user. When executing the social function, the display device 200 may enable the camera and microphone to collect video data and audio data by running a social application. The video data and audio data are then sent to the server 400 through the communicator 220, so that the video data and audio data are forwarded to other display devices through the server 400.

[0078] Similarly, other display devices can also send video data and audio data to the server 400 in the above manner. Therefore, while sending video data and audio data, the display device 200 can also obtain video data and audio data sent by other display devices from the server 400, and display the obtained video data and play the obtained audio data, thereby realizing video calls and audio calls with other display devices.

[0079] The display device 200 can also realize multi-person video calls based on the above social functions. That is, the social application can build a multi-person call platform through the server 400, and multiple display devices 200 can be connected to the multi-person call platform by running the same social application. When one of the display devices 200 uploads audio and video data to the platform, the remaining platforms can simultaneously subscribe to the data stream corresponding to the audio and video data, so that multiple display devices 200 participating in the call at the same time can obtain audio and video data from each other, realizing multi-person calls.

[0080] In order to realize the above multi-person call function, Figure 6 As shown, in some embodiments, the display device 200 can obtain a conference entry instruction input by the user for starting or joining a multi-person video call. For example, a video conference application such as "×× Conference" or "×link" can be installed on the display device 200. After the user starts the application, he can enter a predetermined conference ID and click the "Join" option after the multi-person call has been started, so that the display device 200 can access the conference platform corresponding to the conference ID, so that the display device 200 can receive video data and / or audio data from other display devices participating in the conference from the server 400 and join the multi-person call.

[0081] After joining a multi-person call, the display device 200 can jump to the user interface corresponding to the multi-person call process. Figure 7 As shown, in the multi-person call user interface, multiple windows for displaying video call content, namely, video call windows, may be included. Video data from multiple different display devices 200 obtained from the server 400 may be displayed in each video call window respectively.

[0082] Multiple video call windows in the user interface can be arranged in a default order for the user to view. For example, the video call windows can be arranged in order according to the identity information of the users participating in the conference, that is, the host of the conference and the local screen of the display device 200 can be displayed preferentially in the user interface. Specifically, in the user interface, multiple rows can be displayed from top to bottom, and each row can display the host screen window and the local screen window in sequence from left to right.

[0083] When there are a large number of display devices 200 participating in a multi-person conference, some users have not set their identity information or the identity information of multiple users is of the same category, so the display device 200 can also arrange the order of the remaining video call windows according to a specific arrangement rule. For example, while the display device 200 gives priority to displaying the host and the local window on the user interface, it can also detect the hardware startup status of each display device 200, thereby sorting multiple call windows according to the hardware status.

[0084] That is, after the display device 200 obtains the joining instruction input by the user, it can obtain the participant list information from the server 400 to set the display order of multiple video call windows according to the participant list information. The participant list information includes user ID and audio and video status. The user ID can be a user name registered based on the application service provider; it can also be the device name after the server 400 detects the display device 200 and passes the authentication, such as mac address, network address, etc. The audio and video status is data used to represent the output status of the audio data and video data of the display device 200. For example, if some display devices 200 do not turn on the camera after joining the meeting, they can only communicate through audio; if some display devices 200 do not turn on the microphone after joining the meeting, they can only communicate through video.

[0085] To facilitate communication, the display device 200 can preferentially display the windows corresponding to users who have both video and audio turned on in the user interface, and then sequentially display the windows of users who have not turned on video but have turned on audio, the windows of users who have turned on video but not audio, and the windows of users who have neither video nor audio turned on.

[0086] For this reason, Figure 8As shown, in some embodiments, the display device 200 can first obtain the identity information set for the user ID in the current video conversation in the step of setting the initial order according to the participant list information, so as to set the first priority based on the identity information. For example, after the user joins the meeting, the display device 200 can extract the user ID of each participant in the participant list information, and match the identity information of each user in the current meeting in the database according to the user ID to determine the user whose identity information is the current meeting host. At the same time, for each display device 200 that joins the meeting, the local user ID can also be recorded to determine that the identity information is the local user.

[0087] For different video call processes, other identity information can be set according to specific needs. For example, multiple identity information levels can be set according to the management hierarchy; multiple identity information can be set according to the geographical location, etc. At the same time, if a user ID is not set with identity information, it can be defaulted to an ordinary participant. Therefore, after obtaining the identity information, the display device 200 can set the display order of the windows corresponding to the user ID according to the identity information. For example, the host window can be displayed first in the user interface, then the local window, and finally the window corresponding to the ordinary participant.

[0088] Obviously, when there are many windows corresponding to the same identity information, the display device 200 can parse the audio switch value and the video switch value from the audio and video status, calculate the judgment value according to the audio switch value and the video switch value, and set the second priority based on the judgment value. The judgment value is the weighted sum of the audio switch value and the video switch value. In order to highlight the user window of a special identity, the first priority can be set higher than the second priority.

[0089] For example, the audio switch value can be assigned a value of 1 when the user turns on the audio, and a value of 0 when the user does not turn on the audio; similarly, the video switch value can be assigned a value of 1 when the user turns on the video, and a value of 0 when the user does not turn on the video. Then, according to different conference types, the weights of the audio switch value and the video switch value are set. For conferences that are mainly for calls, the audio weight can be set to be greater than the video weight, such as 0.6 for audio and 0.4 for video; and for conferences that are mainly for presentations, the audio weight can be set to be less than the video weight, such as 0.4 for audio and 0.6 for video.

[0090] Then, according to the set weight and switch value, the judgment value corresponding to each user ID is calculated. That is, for the user ID that has both video and audio turned on, the judgment value P1=0.6×1+0.4×1=1; for the user ID that has not turned on the video but turned on the audio, the judgment value P2=0.6×1+0.4×0=0.6; for the user that has turned on the video but not turned on the audio, the judgment value P3=0.6×0+0.4×1=0.4; for the user that has neither turned on the video nor turned on the audio, the judgment value P4=0.6×0+0.4×0=0. After calculating the judgment value corresponding to each user ID, the display order of multiple windows can be determined by comparing the size of the judgment value, that is, the host window, the local window, the user window that has both video and audio turned on, the user window that has not turned on the video but turned on the audio, the user window that has turned on the video but not turned on the audio, and the window of the user that has neither turned on the video nor turned on the audio are displayed in sequence in the user interface.

[0091] It can be seen that when each user joins a multi-person call, the video call windows in the user interface can be sorted according to the above display order determination method. For ease of description, the arrangement order determined in the above manner can be referred to as the initial order in the embodiment of the present application. The display device 200 can subscribe to the video data stream corresponding to each video window through the server 400, and draw multiple video call windows in the user interface according to the initial order to display the acquired video data stream.

[0092] like Fig. 9 As shown, in some embodiments, the display device 200 can also adjust the display order according to the actual participation status of the meeting. That is, the display device 200 can monitor the audio and video status of each video call window in the user interface through the server 400. For example, the server 400 can maintain a real-time updated data table, in which the hardware startup status of each display device 200 participating in a multi-person call can be recorded. When any display device 200 turns on or off the audio (or video), the server 400 can detect the corresponding status and make changes in the data table. The modified data table parameters can be sent to each display device 200 participating in the video call along with the data stream, so that the display device 200 can monitor the audio and video status of each video call window in the user interface.

[0093] If the audio and video status of any video call window changes, the second priority can be recalculated according to the changed audio and video status; and the display order of the video call windows can be adjusted according to the recalculated second priority. For example, when a user's display device 200, which does not turn on the video but turns on the audio, turns on the video during the meeting, the corresponding audio and video status changes, that is, the video switch value changes from "0" to "1". Therefore, the judgment value corresponding to the user ID will change from "0.6" to "1", so that the display order of the window corresponding to the user ID can be adjusted according to the recalculated judgment value, and the display position of the window can be advanced.

[0094] It can be seen that based on the video call window display method provided in the above embodiment, the application can display the user interface of the video call window arranged in the initial order on each display device 200. However, since users have different requirements for multi-person calls, different users have different requirements for the window display order. For example, during a video conference, due to specific scene requirements, the user may adjust the display order of the video call window, such as placing important guests in front and ordinary participants in the back.

[0095] To this end, a method for adjusting the video display order is provided in some embodiments of the present application, and the method can be applied to the display device 200 provided in the above embodiments. To meet the needs of the method implementation process, the display device 200 may include a display 260, a communicator 220, and a controller 250. Among them, the display 260 can be configured to display a user interface, and the user interface includes multiple video call windows displayed in an initial order. The communicator 220 is configured to connect to the server 400 to obtain the call data stream and related data during the multi-person call from the server 400. Fig.10 As shown, the controller 250 is configured to execute the above-mentioned video display order adjustment method, which specifically includes the following contents:

[0096] A control instruction input by a user for adjusting the display order of video call windows is obtained. Among them, the control instruction for adjusting the display order of video call windows can be completed by a series of interactive actions. For example, a sequence adjustment option can be provided in the multi-person call user interface. When the user clicks on the option, the user interface can be triggered to enter the setting state. At this time, the user can perform key operations through the control device 100 supporting the display device 200, and perform actions such as "select-adjust position-unselect" on the video call window in the user interface by pressing the direction key and the confirmation key. That is, the control instruction for adjusting the display order of the video call window can be input through a series of key operations.

[0097] For some display devices 200, they can support more forms of interaction methods, so the input of the control instructions can be completed through the corresponding interaction methods. For example, for a display device 200 that supports touch interaction operations, the user can trigger the user interface to enter the setting state by long pressing the video call window with the pre-adjusted display position. Then, through the sliding operation, the video call window at the finger position is moved. After moving the video call window to the target position, the user can release the finger to keep the video call window at the target position. That is, the control instruction for adjusting the display order of the video call window can be input through touch interaction operations.

[0098] The display device 200 can also display a specific window setting interface for the user to input the control instruction for adjusting the display order of the video call windows. For example, the user interface may include a window setting option. When the user selects the window setting option, the display device 200 can jump to the display window setting interface. The window setting interface may include options for representing each video call window. The user can select from multiple options in order to set the display order. That is, the control instruction for adjusting the display order of the video call windows can be input by selecting a specific option on the user interface.

[0099] After the user inputs a control instruction for adjusting the display order of the video call windows, the display device 200 can respond to the control instruction and extract the target window identification information and the custom order from the control instruction. The target window is the video call window whose display order is to be adjusted as specified in the control instruction, and the custom order is the display order set by the user through a specific interactive action when inputting the control instruction.

[0100] For example, Fig.11 As shown, after the user triggers the display device 200 to enter the setting state of the user interface, the user selects the video call window with window number 003 in the current user interface, that is, the selected video call window is the target window. Then, by pressing the direction key on the control device 100, the target window is adjusted to the position of window number 001. Therefore, the display device 20 can extract the user ID of the adjusted window, that is, the target window identification information. At the same time, the custom order, that is, the order of each window after adjustment, is obtained.

[0101] After obtaining the target window identification information and the custom order, the display device 200 can also obtain the call data stream of the target window according to the target window identification information. The call data stream of the target window can be obtained through the data table recorded in the multi-person call platform, that is, in some embodiments, the display device 200 can traverse the user ID of the target window in the step of obtaining the call data stream of the target window according to the target window identification information. Then, based on the traversal result, a data subscription request is sent to the server 400, that is, the data subscription request includes the user ID. After receiving the data subscription request, the server 400 can query the corresponding data stream channel information according to the user ID contained in the request, and feedback the queried data stream channel information to the display device 200.

[0102] The display device 200 then receives the data stream channel information fed back by the server 400 in response to the data subscription request, so as to connect to the data stream channel corresponding to the user ID and obtain the call data stream. For example, when the video call window with window number 003 is selected, the user ID corresponding to the 003 window is: xx0003, then the display device 200 can send a data subscription request to the server 400, so that the server 400 can query the data stream channel information corresponding to xx0003 according to the data subscription request, such as channel003. The queried data stream channel information is then fed back to the display device 200, so that the display device 200 can obtain the video data and / or audio data of the xx0003 user by subscribing to the data stream channel.

[0103] After acquiring the call data stream corresponding to the target window, the display device 200 can use the call data stream to replace the display content of the video call window in the user interface according to the custom order parsed in the control instruction. For example, in the process of adjusting the target window with window number 003 to the position of window number 001, the display device 200 can use the call data stream corresponding to user ID: xx0003 to replace the display content of the video call window with number 001 after acquiring the call data stream corresponding to user ID: xx0003, so that the video call window with number 001 can display the video call screen corresponding to the target window with number 003.

[0104] It should be noted that, in the process of adjusting the window order, after the display position of a display window is changed, the position change process may also cause the position of other windows to change. For example, when the target window numbered 003 is adjusted to the position of window 001, the video call screen corresponding to the original window position 001 needs to be adjusted to the position of window 002 in sequence. Similarly, the video call screen corresponding to the window position 002 needs to be adjusted to the position of window 003. Therefore, in the adjustment process, the target window is not limited to the video call window directly implemented by the user interaction action, but also includes other video call windows that are changed due to the adjustment of the position of some windows.

[0105] In some embodiments, Fig.12 As shown, when a user inputs a control instruction for adjusting the display order through the window setting interface, the display device 200 can first detect a first interactive action input by the user in the step of obtaining the control instruction input by the user for adjusting the display order of the video call window. The first interactive action is an interactive action for triggering the window setting interface. For example, when the multi-person call user interface includes a window setting option, the user can click the window setting option through a button or touch operation on the control device 100 to input the first interactive action.

[0106] After the user inputs the first interactive action, the display device 200 can jump to the display window setting interface in response to the first interactive action to adjust the window display order in the user interface, so the window setting interface includes a video call window option. For example, after the user clicks the window setting option, the display device 200 can jump to the window setting interface. In the window setting interface, options with the same number of windows as in the multi-person call interface can be included for the user to select. In order to facilitate the user to perform subsequent operations, the window setting interface displayed by the display device 200 can include the user ID corresponding to each window and identification information associated with the user ID, such as avatars, video call screenshots, etc.

[0107] The user can select from the multiple window options in turn according to the expected display order to customize the display order of the windows. Therefore, after displaying the window setting interface, the display device 200 can detect the interactive action performed by the user on the window setting interface, that is, detect the second interactive action input by the user based on the video call window option. The second interactive action is an interactive action for defining the display order. For example, the second interactive action is a selection operation on multiple window options.

[0108] By detecting the second interactive action, the display device 200 can record in real time the video call window options selected in sequence by the second interactive action, thereby generating a custom order. For example, when the user selects "Host", "Guest 2", and "Guest 6" in sequence in the window setting interface, a custom order of "001 Window - Host", "002 Window - Guest 2", and "003 Window - Guest 6" can be generated, and the corresponding windows of other window users can be displayed in the initial order.

[0109] It should be noted that, in the process of defining the display order through the window setting interface, the display device 200 can implement the order definition function through a multi-level interface. In addition, in order to facilitate the selection of window options, when displaying the window setting interface, a specific option can be used to trigger the display of the window option list. For example, Fig.13 As shown in the figure, after the user selects the "Window Layout" option in the multi-person call interface, he can jump to the "Window Layout" interface. Fig.14 As shown, in the "window layout" interface, an option of "customizing window display order" may be included. When the user selects this option, the display device 200 may be triggered to jump to the "window settings" interface, such as Fig.15 As shown, the "Window Settings" interface can provide a setting bar for "Display settings based on number", and use multiple selection boxes to indicate the definable content for each window number.

[0110] When the user selects the "+" button on any selection box, the display device 200 may be triggered to display a window option list interface (or dialog box) to select the display content defined by the window. Fig.16 As shown, the user can click the "+" button on the selection box numbered "1" to trigger the display of the "Select Participants" list interface, and select the "Host" option in it to define the video call content corresponding to the "Host" to be displayed in the window position No. 1, such as Fig.17 shown.

[0111] Through the above-mentioned window setting interface, the user can more intuitively define the window display order, and achieve the purpose of order adjustment through simple selection operations, so that the display device 200 can display in sequence according to user needs while reducing operation difficulty and improving user experience.

[0112] like Fig.18As shown, in some embodiments, in order to enable the display device 200 to display in a user-defined order, in the step of using the call data stream to replace the display content of the video call window in the user interface, the display device 200 can first extract the user ID of each of the video call windows in the original order, that is, determine the original user ID of each window; and at the same time, extract the current user ID of each target window in a customized order, that is, determine the current user ID of each window.

[0113] The extracted user ID is then compared. If the original user ID is different from the current user ID, that is, the content displayed in the current window needs to be adjusted, the call data stream of the target window can be used to replace the display content in the video call window; if the original user ID is the same as the current user ID, that is, at the current window position, the video call content displayed in the initial order and the custom order has the same user ID, and there is no need to replace the display content of the window, so the display content of the video call window can be retained.

[0114] In order to facilitate users to participate in multi-person conferences, a window can be used as the main video window in the multi-person call interface. The main video window has a larger window area than other windows, so the corresponding video call content can be displayed more clearly. After customizing the display order, if the user ID corresponding to the video data stream displayed in the current main video window changes, there is a video stream loading process because it involves the process of subscribing to the video stream of the current user ID from the server 400. This process will consume some time. Therefore, in the process of switching the video stream corresponding to the current user ID, the default avatar screen of the user can be displayed before the video stream is pulled.

[0115] Similarly, if the original user ID is different from the current user ID, that is, the display content of the current window needs to be replaced, the display device 200 can detect whether the video function of the target window is turned on. If the video function of the target window is turned on, the call data stream of the target window is obtained to replace the display content in the video call window with the call data stream; if the video function of the target window is not turned on, the identification screen corresponding to the current user ID is obtained to display the identification screen in the video call window.

[0116] For example, when the main video window is switched to user ID: xx0004, if the user with ID xx0004 does not turn on the video function, the call data stream uploaded to the server 400 only contains the audio data stream, not the video data stream. Therefore, when the call data stream corresponding to the user is switched to the main video window for display, the video screen cannot be formed in the main video window. At this time, the display device 200 can extract the avatar of the user xx0004 and display it in the main video window.

[0117] When there are a large number of users participating in a multi-person call, since the user generally only selects the windows that he or she is concerned about during the process of customizing the display order, only some of the target windows will be included in the user-defined display order. In order to ensure that the display device 200 can still display the call windows of all users after the user customizes the order, in some embodiments, the display device 200 can obtain the number of target windows and the total number of windows in the user interface in the step of replacing the display content of the video call window in the user interface with the call data stream, and compare the obtained number of target windows with the total number of windows in the user interface. If the number of target windows is equal to the total number of windows, that is, the user has set the order for all windows, the window order in the user interface can be directly displayed according to the customized order.

[0118] If the number of target windows is less than the total number of windows, that is, the user has not defined the display order of all windows, the display device 200 can replace the first window in the user interface with the call data stream of the target window. Among them, the first window is the target window number video conversation window in the user interface. For example, the display order customized by the user in the window setting interface is "001 window - host", "002 window - guest 2", "003 window - guest 6", then the first window is the window numbered "001", "002", and "003" respectively. At this time, the display device 200 can first replace the windows "001", "002", and "003" with the video call screens of "host", "guest 2", and "guest 6" respectively.

[0119] After replacing the call data stream of the first window, the display device 200 can also display the call data stream of the second window in the user interface according to the initial order. The second window is a video conversation window other than the first window in the user interface. For example, the second window is other video call windows other than the windows numbered "001", "002", and "003" in the user interface. For other windows, the display device 200 can sort and display them in the initial order after eliminating the user windows of "Host", "Guest 2", and "Guest 6".

[0120] Based on the above video display order adjustment method, some embodiments of the present application further provide a display device 200. The display device 200 includes a display 260, a communicator 220 and a controller 250. The controller 250 is configured to execute the following program steps:

[0121] Obtaining a control instruction input by a user for adjusting the display order of video call windows;

[0122] In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction;

[0123] Acquiring a call data stream of the target window according to the target window identification information;

[0124] According to the customized order, the call data stream is used to replace the display content of the video call window in the user interface.

[0125] The following will describe in detail the video display order adjustment method of the display device 200 in conjunction with a specific video conference example. Fig.19 As shown, after the user initiates and joins a meeting in the main video window displayed by the display device 200, the display device 200 will request the signaling service of the audio and video platform system to obtain the participant list data in the meeting. The participant list data will call back the participant data through the signaling software development kit (Software Development Kit, SDK). The display device 200 then stores the participant list data, including user ID, image, audio status, video status, etc. After obtaining the participant data, the display device 200 can also sort the participant list according to the default sorting of the video conferencing application. Then trigger the refresh of the main video window in the form of a message, so that each participant on the main video window creates a window view, which will be bound to the participant and display the corresponding user video screen.

[0126] When the user needs to adjust the order of the video conference window, he can switch to the corresponding window number through the custom window UI and select the window corresponding to the corresponding participant. The display device 200 can then extract the corresponding participant data content from the stored data according to the switching operation input by the user. After returning the participant data, the display device 200 will display the participant data in a UI for the user to select and adjust the order. After the user selects the window to adjust the order, selecting the participant window will return the selected participant to the custom window UI, so that the user can click the OK button in the custom window UI after the selection is completed to determine the custom participant list order of the corresponding window layout, and send the custom participant list order to the participant database.

[0127] After receiving the list order, the participant database can record the received list order. The main video window can monitor the status of the records in the participant database in real time, and compare the currently displayed video window order with the latest customized list order in real time. If the list order is the same, the screen remains unchanged. For example, the user ID of each participant's video window view position can be compared with the user ID of the corresponding layout participant data module for the main video window. If they are the same, keep the content corresponding to the user ID in the main video window unchanged; otherwise, replace the content in the main video window with the sorted user ID screen.

[0128] The display order can also be automatically adjusted when the status of other participants changes. That is, when other participants turn on or off local audio or video, they can synchronize the audio and video status to the audio and video platform system, and the audio and video platform system broadcasts the audio and video status through the signaling service. After receiving the broadcast audio and video status, the signaling SDK can call back the user's audio and video status to the participant database to update the status data recorded in the participant database, triggering the participant list to be re-sorted to sort according to the custom order of the corresponding window layout and the overall audio and video status. The overall sorted order is then used to trigger the refresh of the main video window in the form of a message to complete the adjustment of the video display order.

[0129] As can be seen from the above technical solution, the display device 200 provided by the present application can extract the identification information of the video call window whose display order is adjusted after the user inputs the control instruction for adjusting the display order, and subscribe to the call data stream of the corresponding window according to the identification information, so as to replace the display content of the video call window with the call data stream according to the display order customized by the user. The display device 200 can enable multiple display devices participating in a multi-person video conversation to freely sort the video call windows according to the needs of the user, so that different users can control the video display order according to different conversation needs, thereby improving the user experience.

[0130] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface, wherein the user interface includes a plurality of video call windows displayed in an initial sequence; a communicator, configured to connect to the server; The controller is configured as: Obtaining a control instruction input by a user for adjusting the display order of video call windows; In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction; According to the target window identification information, traverse the user ID of the target window; Sending a data subscription request to a server, wherein the data subscription request includes the user ID; receiving data stream channel information fed back by the server in response to the data subscription request, so as to connect to the data stream channel corresponding to the user ID and obtain the call data stream; According to the customized order, the call data stream is used to replace the display content of the video call window in the user interface.

2. The display device according to claim 1, characterized in that The controller is further configured to: In the step of obtaining a control instruction input by a user for adjusting the display order of the video call windows, detecting a first interactive action input by the user, where the first interactive action is an interactive action for triggering a window setting interface; In response to the first interactive action, displaying a window setting interface, wherein the window setting interface includes a video call window option; detecting a second interaction action input by a user based on the video call window option, where the second interaction action is an interaction action for defining a display order; The video call window options selected in sequence by the second interactive action are recorded to generate the custom order.

3. A display device, characterized in that: include: a display configured to display a user interface, wherein the user interface includes a plurality of video call windows displayed in an initial sequence; a communicator, configured to connect to the server; The controller is configured as: Obtaining a control instruction input by a user for adjusting the display order of video call windows; In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction; Acquiring a call data stream of the target window according to the target window identification information; Extracting the original user ID of each of the video call windows in the initial order; Extract the current user ID of each target window according to the custom order; If the original user ID is different from the current user ID, replacing the display content in the video call window with the call data stream of the target window; If the original user ID is the same as the current user ID, the display content of the video call window is retained.

4. The display device according to claim 3, characterized in that If the original user ID is different from the current user ID, the controller is further configured to: Detect whether the target window has a video function enabled; If the target window has a video function turned on, obtaining a call data stream of the target window to replace display content in the video call window with the call data stream; If the target window does not have a video function enabled, an identification screen corresponding to the current user ID is obtained to display the identification screen in the video call window.

5. A display device, characterized in that: include: a display configured to display a user interface, wherein the user interface includes a plurality of video call windows displayed in an initial sequence; a communicator, configured to connect to the server; The controller is configured as: Obtaining a conference entry instruction input by a user for starting or joining a multi-person video conversation; In response to the conference joining instruction, obtaining conference participant list information from the server, the conference participant list information including user ID and audio and video status; Setting the initial order according to the participant list information; Drawing a plurality of video call windows in the user interface according to the initial order; Obtaining a control instruction input by a user for adjusting the display order of video call windows; In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction; Acquiring a call data stream of the target window according to the target window identification information; According to the customized order, the call data stream is used to replace the display content of the video call window in the user interface.

6. The display device according to claim 5, characterized in that The controller is further configured to: In the step of setting the initial order according to the participant list information, obtaining identity information set for the user ID in the current video conversation; setting a first priority based on the identity information; Parsing an audio switch value and a video switch value from the audio and video status; Calculating a judgment value according to the audio switch value and the video switch value, wherein the judgment value is a weighted sum of the audio switch value and the video switch value; A second priority is set based on the judgment value, the first priority being higher than the second priority.

7. The display device according to claim 6, characterized in that The controller is further configured to: Monitoring the audio and video status of each video call window in the user interface; If the audio or video status of any of the video call windows changes, recalculate the second priority level according to the changed audio or video status; The display order of the video call windows is adjusted according to the recalculated second priority.

8. A display device, characterized in that: include: a display configured to display a user interface, wherein the user interface includes a plurality of video call windows displayed in an initial sequence; a communicator, configured to connect to the server; The controller is configured as: Obtaining a control instruction input by a user for adjusting the display order of video call windows; In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction; Acquiring a call data stream of the target window according to the target window identification information; Obtaining the number of target windows and the total number of windows of the user interface according to the custom order; If the number of the target windows is less than the total number of windows, replacing the first window in the user interface with the call data stream of the target window; The first window is the number of video conversation windows in the target window in the user interface; The call data stream of a second window is displayed in the user interface according to the initial sequence, where the second window is a video conversation window in the user interface except the first window.

9. A method for adjusting video display order, characterized in that: Applied to a display device, the video display order adjustment method includes: Obtaining a control instruction input by a user for adjusting the display order of video call windows; In response to the control instruction, extracting target window identification information and a custom order from the control instruction, the target window being the video call window whose display order is adjusted as specified in the control instruction; According to the target window identification information, traverse the user ID of the target window; Sending a data subscription request to a server, wherein the data subscription request includes the user ID; receiving data stream channel information fed back by the server in response to the data subscription request, so as to connect to the data stream channel corresponding to the user ID and obtain the call data stream; According to the customized sequence, the call data stream is used to replace the display content of the video call window in the user interface.

Images