Display device and video conference interface display method

Abstract

The application discloses a display device and a video conference interface display method. The method comprises the following steps: a central processor responds to a user's joining video conference instruction input through a browser interface, and controls a display to display a video conference interface on a graphics layer. If the video processor detects that the camera is a coding camera, the video processor receives local video data collected by the camera, decodes the local video data, and sends the decoded local video data to the graphics processor. The graphics processor controls the display to display the decoded local video data on a video layer. The display area of the decoded local video data corresponds to the position of the local video display frame. The application directly decodes the local collected video data through the video processor of the display device, and displays the decoded local video data on the video layer, so that the CPU and memory occupancy rate is reduced, and the performance of the display device is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent display device technology, and in particular to a display device and a video conferencing interface display method. Background Technology

[0002] For most smart devices, such as smartphones and tablets, browser-based video conferencing solutions utilize native browser modules for software decoding and display of video. Due to the unique nature of display device chip platforms, their central processing units (CPUs) are weaker and have significantly less memory compared to tablets and smartphones in the same price range. If multiple video streams in a video conference rely on native browser modules for software decoding and display, the CPU and memory usage will increase dramatically, leading to a sharp drop in performance. Summary of the Invention

[0003] This application provides a display device and a video conferencing interface display method, which directly decodes locally acquired video data through the video processor of the display device and displays it on the video layer through the graphics processor, thereby reducing CPU and memory usage and improving the performance of the display device.

[0004] The first aspect of this application provides a display device, including:

[0005] The monitor is configured to display the browser interface;

[0006] The camera is configured to capture local video data;

[0007] The controller includes a central processing unit, a video processor, and a graphics processor;

[0008] The central processing unit is configured as follows:

[0009] In response to a user's command to join a video conference input through a browser interface, the display is controlled to show a video conference interface in the graphics layer. The video conference interface includes a local video display frame, which is transparent.

[0010] The video processor is configured to: if it detects that the camera is an coded camera, receive local video data captured by the camera; decode the local video data; and send the decoded local video data to the graphics processor.

[0011] The graphics processor is configured to control the display to display decoded local video data on the video layer, wherein the display area of ​​the decoded local video data corresponds to the position of the local video display frame, and the video layer is located below the graphics layer.

[0012] In conjunction with the first aspect, in a first implementable manner, the video conferencing interface further includes at least one video display frame for a remote participant, and the central processing unit is configured to:

[0013] Receive video data from remote participants obtained from the server;

[0014] Decode the video data of the remote participants;

[0015] The display is controlled to show the decoded video data of the remote participant in the video display frame of the remote participant.

[0016] In conjunction with the first aspect, in a second possible implementation, the graphics processor is configured to: if the camera is detected to be a non-encoded camera, receive local video data captured by the camera; and control the display to display the local video data on the video layer.

[0017] In conjunction with the first aspect, in a third implementable manner, the central processing unit is configured as follows:

[0018] Receive local video data captured by the camera;

[0019] If the camera is detected to be an coded camera, the local video data is uploaded to the server;

[0020] If the camera is detected to be a non-encoded camera, the local video data is encoded, and the encoded local video data is uploaded to the server.

[0021] In a fourth possible implementation, combining the three possible implementations, the central processing unit is configured as follows:

[0022] If the camera is detected to be a non-encoded camera, the display is controlled to show the local video data in the local video display box of the graphics layer.

[0023] In conjunction with the first aspect, in the fifth feasible method, the video conferencing interface further includes at least one remote participant video display frame, which is transparent;

[0024] The central processing unit is configured to: receive video data of remote participants obtained from the server, and send the video data of remote participants to the video processor;

[0025] The video processor is configured to: decode the video data of the remote participant and send the decoded video data of the remote participant to the graphics processor;

[0026] The graphics processor is configured to control the display to display decoded remote participant video data on the video layer, wherein the display area of ​​the decoded remote participant video data corresponds to the position of the remote participant video display frame.

[0027] In conjunction with the first aspect, in the sixth possible implementation, in response to a user input instruction to change the position of the local video display frame, the central processing unit is configured to: change the position of the local video display frame on the video conferencing interface; and the graphics processor is configured to: change the display area of ​​the decoded local video data so that the display area of ​​the changed local video data corresponds to the position of the changed local video display frame.

[0028] In conjunction with the first aspect, in the seventh implementable manner, the central processing unit is configured as follows:

[0029] In response to a user's input command to turn off the camera or stop displaying local video data, the local video display frame is set to a preset color, or a preset image is displayed in the local video display frame, the width and height of the preset image being the same as the width and height of the local video display frame.

[0030] A second aspect of this application provides a method for displaying a video conferencing interface, including:

[0031] The central processing unit is configured to: in response to a user's command to join a video conference input through a browser interface, control the display to show a video conference interface in the graphics layer, the video conference interface including a local video display frame, the local video display frame being transparent;

[0032] The video processor is configured to: if it detects that the camera is an coded camera, receive local video data captured by the camera; decode the local video data; and send the decoded local video data to the graphics processor.

[0033] The graphics processor is configured to control the display to display decoded local video data on the video layer, wherein the display area of ​​the decoded local video data corresponds to the position of the local video display frame, and the video layer is located below the graphics layer.

[0034] In conjunction with the second aspect, in the first possible implementation, the video conferencing interface further includes at least one video display frame for a remote participant, and the central processing unit is configured to:

[0035] Receive video data from remote participants obtained from the server;

[0036] Decode the video data of the remote participants;

[0037] The display is controlled to show the decoded video data of the remote participant in the video display frame of the remote participant.

[0038] This application provides a method for displaying a video conferencing interface. The method is applicable to a display device, which includes at least a controller and a display. The controller includes a central processing unit (CPU), a video processor, and a graphics processor (GPU). The CPU is configured to: respond to a user's command to join a video conference via a browser interface, control the display to show the video conferencing interface on the graphics layer. The video conferencing interface includes a local video display frame, which is transparent. The video processor is configured to: if the camera is detected to be an encoded camera, receive local video data captured by the camera; decode the local video data; and send the decoded local video data to the GPU. The GPU is configured to: control the display to show the decoded local video data on the video layer, where the display area of ​​the decoded local video data corresponds to the position of the local video display frame, and the video layer is located below the graphics layer. This application directly decodes the locally acquired video data using the display device's video processor and displays it on the video layer using the GPU, instead of using the browser's native modules for decoding and display. This reduces CPU and memory usage while improving the performance of the display device. Attached Figure Description

[0039] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 An operational scenario between a display device and a control device according to some embodiments is illustrated;

[0041] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0042] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0043] Figure 4 A software configuration diagram of a display device 200 according to some embodiments is shown;

[0044] Figure 5 A flowchart for using video conferencing via a browser;

[0045] Figure 6 A flowchart illustrating the interaction between a display device and a server, provided as a feasible embodiment;

[0046] Figure 7 A schematic diagram of a host initiating a video conference, provided as a feasible embodiment;

[0047] Figure 8 A schematic diagram of a video conferencing interface provided for a feasible embodiment;

[0048] Figure 9 A schematic diagram of a preparation interface for joining a video conference provided as a feasible embodiment;

[0049] Figure 10 A schematic diagram of a video conferencing interface provided for a feasible embodiment;

[0050] Figure 11 A schematic diagram of a video conferencing interface provided for a feasible embodiment;

[0051] Figure 12 This is a schematic diagram illustrating the positional relationship between the graphics layer and the video layer.

[0052] Figure 13 A schematic diagram of the graphics layer and video layer interface provided in a feasible embodiment;

[0053] Figure 14 A flowchart illustrating a video conferencing interface display as provided in a feasible embodiment;

[0054] Figure 15 A schematic diagram of a video conferencing interface provided for a feasible embodiment;

[0055] Figure 16 A schematic diagram of a video conferencing interface provided for a feasible embodiment;

[0056] Figure 17 A schematic diagram of a video conferencing interface provided for a feasible embodiment;

[0057] Figure 18 This is a schematic diagram of a video conferencing interface provided for a feasible embodiment. Detailed Implementation

[0058] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0059] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0060] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0061] The terms “include” and “have”, and any variations thereof, are intended to cover but not exclude inclusion. For example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0062] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0063] Figure 1 This is a schematic diagram illustrating the operational scenario between the display device and the control unit according to the embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0064] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0065] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0066] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0067] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0068] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0069] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0070] like Figure 3 The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0071] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0072] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0073] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0074] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0075] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0076] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0077] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0078] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0079] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0080] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0081] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0082] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0083] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0084] See Figure 4 In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Library Layer"), and the kernel layer.

[0085] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0086] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0087] like Figure 4 As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0088] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0089] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0090] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0091] For most smart devices, such as smartphones and tablets, browser-based video conferencing solutions utilize native browser modules for software decoding and display of video. Due to the unique nature of display device chip platforms, their central processing units (CPUs) are generally weaker and have significantly less memory compared to similarly priced tablets and smartphones. If multiple video streams in a video conference are all decoded and displayed using native browser modules, such as… Figure 5 As shown, the CPU and memory usage will increase sharply, and performance will decrease sharply.

[0092] Figure 5 This is a flowchart for using video conferencing through a browser. (Example:) Figure 5 As shown, the process is divided into two parts. The first part involves processing the video captured by the local camera. The camera captures local video data, and after the browser's video processing module receives this data, if the data is encoded, it is decoded by the browser's video software decoding module and displayed through the browser's image rendering module. If the local video data is unencoded, it is directly displayed through the browser's image rendering module, and simultaneously, the local video data is transmitted to the server. The second part involves processing the video data of remote participants. The browser's video processing module receives the video data of remote participants from the server, decodes it using the browser's video software decoding module, and displays it through the browser's image rendering module.

[0093] To address the above technical problems, this application provides a display device. The structure and functions of each part of the display device can be found in the above embodiments. Furthermore, based on the display device shown in the above embodiments, this embodiment further improves some functions of the display device. For example... Figure 6 As shown, Figure 6 This is a flowchart illustrating the interaction between a display device and a server, provided as a feasible embodiment.

[0094] Video conferencing refers to a meeting in which people in two or more locations conduct face-to-face conversations through communication devices and networks. Using a video conferencing system, participants can hear audio from other meeting rooms, see the images, actions, and expressions of other participants in those rooms, and send electronic presentations, giving them an immersive experience. Users can access the video conferencing interface through a video conferencing application or a web browser. This application's embodiments primarily address participating in video conferencing via a web browser, where the display device needs to have a built-in camera or be able to connect an external camera.

[0095] In some embodiments, users can act as hosts to actively book and initiate video conferences. After successfully booking a conference, users can view the video conference ID and password on the booking confirmation screen and inform other participants to join the video conference. Figure 7 This is a schematic diagram of the interface for a host to initiate a video conference, provided as a feasible embodiment. Figure 7 The interface shown includes the video conference time (71), the video conference number (72), the video conference password (73), the video conference link control (74), and the join meeting control (75). Figure 7 The interface also includes a selector 76 that indicates which control is selected. After selecting the video conference link control 74, the user can directly send the video conference link to the participants. Users can also directly enter the video conference by selecting the join control 75 and entering the join command.

[0096] In some embodiments, users can join a video conference as participants. Users can select a video conference link sent by the host and be directly redirected to the browser's join video conference interface. Figure 8 This is a schematic diagram of a video conference joining interface provided for a feasible embodiment. Figure 8 The interface shown includes video conference information 81 and a join-meeting control 82. The video conference information includes the initiator, time, meeting ID, and password. Figure 8 The interface also includes a selector 83 that indicates which control is selected. Users can directly join a video conference by selecting the "Join Conference" control 82 and entering the command to join. Users can also enter the video conference URL in their browser to display a preparation interface for joining the video conference. Figure 9 This is a schematic diagram of a preparation interface for joining a video conference, provided as a feasible embodiment. Figure 9 The interface shown includes a meeting ID input control 91, a password input control 92, a confirmation control 93, and a selector 94. After entering the meeting ID and password, the user can select the confirmation control to enter the command to join the video conference and directly enter the video conference; alternatively, the user can also jump to... Figure 8 The interface shown allows users to directly enter a video conference by selecting the "Join Meeting" control and entering the command to join the video conference.

[0097] The controller includes a central processing unit, a video processor, and a graphics processor.

[0098] The central processing unit executes step S601: In response to the user's command to join the video conference input through the browser interface, it controls the display to show the video conference interface in the graphics layer.

[0099] In some embodiments, the video conferencing interface includes a local video display box, and as more participants join the current video conference, a remote participant video display box corresponding to each participant is added. Figure 10 This is a schematic diagram of a video conferencing interface provided for a feasible embodiment. Figure 10 The interface shown includes a local video display box 101, a first remote participant video display box 102, a second remote participant video display box 103, and a third remote participant video display box 104. The local video display box and the remote participant video display box are the same size. Figure 11 This is a schematic diagram of a video conferencing interface provided for a feasible embodiment. Figure 11 The interface shown includes a first remote participant video display box 111, a local video display box 112, a second remote participant video display box 113, and a third remote participant video display box 114. The first remote participant is the host, and the size of the first remote participant video display box is significantly larger than the sizes of the other video display boxes.

[0100] It should be noted that both the browser interface and the video conferencing interface are displayed in the graphics layer. In this embodiment, the local video data captured by the camera is not decoded and displayed by the central processing unit. Therefore, the local video display frame is set to a transparent state.

[0101] The central processing unit includes a browser graphics rendering module, which is used to draw the video conferencing interface according to user instructions, wherein the local video display box is set to a transparent state.

[0102] In some embodiments, users can choose to set video capture parameters in the video conferencing interface, including resolution and frame rate. Setting these parameters allows the camera to capture data that is better suited to the current display device and network conditions.

[0103] The video processor executes step S602: acquire information from the camera;

[0104] In some embodiments, the camera information specifically refers to the camera's attributes, from which information such as camera model, shooting parameters, and whether it has encoding function can be obtained.

[0105] Cameras are categorized into encoded cameras and non-encoded cameras based on their encoding capabilities. Encoded cameras primarily use H.264 (a digital video compression format) and H.265 (another digital video compression format) encoding. These cameras have built-in encoding capabilities, directly encoding the captured video data into H.264 or H.265 format before transmitting it back to the display device. Decoding is required before playback. The size of the encoded video data is significantly smaller than the unencoded data, facilitating data transmission. Non-encoded cameras primarily use YUV (a color encoding method) cameras. These cameras do not have encoding capabilities; they capture YUV video data, which can be played back directly to the display device without decoding.

[0106] The video processor executes step S603: Based on the information of the camera, it determines whether the camera is an encoded camera; if the camera is a non-encoded camera, the video processor does not execute any steps displayed on the video conferencing interface.

[0107] If the camera is an coded camera, the video processor executes step S604: receiving local video data captured by the camera;

[0108] The video processor executes step S605: Decodes the local video data;

[0109] The video processor can decode H.264 or H.265 format video data into YUV format video data.

[0110] Video conferencing typically involves decoding multiple video streams. Video decoding itself requires significant computational power, and when the chip's computing power is insufficient, using software decoding for multiple video streams severely impacts efficiency. Due to the specific nature of display devices, they incorporate built-in video processors with dedicated hardware decoding modules. Using these hardware modules for video decoding is highly efficient and doesn't consume excessive CPU resources or memory.

[0111] Hardware decoding uses a dedicated decoding module on the chip. Its advantages include low CPU and memory consumption, low resource usage, and high speed.

[0112] The video processor executes step S606: sending the decoded local video data to the graphics processor;

[0113] The graphics processor executes step S607: controlling the display to show the decoded local video data on the video layer.

[0114] It should be noted that the display area of ​​the decoded local video data corresponds to the position of the local video display frame; specifically, the display area of ​​the decoded local video data is the same size as the local video display frame and is displayed in the same position. The graphics processor can directly play YUV format video data.

[0115] In some embodiments, a Cartesian coordinate system can be established with the intersection of two adjacent sides of the display screen as the zero point and the two adjacent sides as the X and Y axes. The local video display frame is a rectangle by default, with four vertices. The graphics processor can obtain the coordinates of the four vertices of the local video display frame from the browser's graphics rendering module, and then create a video playback window in the video layer at the same coordinates as the four vertices, playing the local video data in the video playback window.

[0116] In some embodiments, a Cartesian coordinate system can also be established using the intersection of two adjacent sides of the display screen as the zero point, and the two adjacent sides as the X and Y axes. The local video display frame is rectangular by default, with a standard aspect ratio and a center point. The graphics processor can obtain the coordinates of the center point of the local video display frame and the width or height of the local video display frame from the browser's graphics rendering module, and then create a video playback window with the same width and height as the local video display frame at the same coordinates as the center point in the video layer, and play the local video data in the video playback window.

[0117] Due to the specific functions of display devices, their chips incorporate a graphics processor (GPU). This GPU provides a dedicated hardware video display module, separate from the graphics display. The video display module and the browser's graphics rendering module are displayed on different layers, such as... Figure 12 As shown, the video layer is located below the graphics layer. Therefore, without modifying the logic of the graphics layer, when displaying video using a hardware display module, it will be covered by the graphics layer and thus invisible. However, the local video display frame in the graphics layer has been set to transparent, allowing the video layer below the graphics layer to be displayed smoothly, as shown below. Figure 13 As shown. Similarly, using a hardware display module can also reduce the workload of the graphics layer, save resources, and improve efficiency.

[0118] The graphics processor executes step S608: acquire information from the camera;

[0119] The graphics processor executes step S609: Based on the information from the camera, it determines whether the camera is an coded camera; if the camera is an coded camera, the graphics processor temporarily does not execute any steps related to the video conferencing interface.

[0120] If the camera is a non-encoded camera, the graphics processor executes step S610: receiving local video data captured by the camera;

[0121] The graphics processor executes step S611: controlling the display to show the local video data on the video layer.

[0122] The central processing unit executes step S612: acquire local video data captured by the camera;

[0123] The central processing unit includes a browser video processing module. Local video data captured by the camera is written into the browser's shared memory, and the browser video processing module retrieves local video data from the shared memory.

[0124] The central processing unit executes step S613: Upload the local video data to the server.

[0125] In some embodiments, after the browser's video processing module obtains local video data, it can directly determine the video format of the local video data. If encoded data is detected, the local video data is directly sent back to the WebRTC (Web Real-Time Communication) module and then transmitted to the server. If unencoded data is detected, the local video data is encoded, that is, the YUV format local video data is encoded into H.264 or H.265 format local video data. Then, the encoded local video data is sent back to the WebRTC module and then transmitted to the server. WebRTC is an application programming interface (API) that enables web browsers to conduct real-time voice or video conversations.

[0126] In some embodiments, after the browser's video processing module obtains local video data, it can acquire camera information and determine whether the camera is an encoded camera. If an encoded camera is detected, the local video data is directly sent back to WebRTC and then transmitted to the server. If a non-encoded camera is detected, the local video data is encoded, that is, the YUV format local video data is encoded into H.264 or H.265 format local video data. Then, the encoded local video data is sent back to WebRTC and then transmitted to the server.

[0127] The central processing unit executes step S614: receiving video data of remote participants obtained from the server.

[0128] It should be noted that, in order to reduce the size of video data transmission and download, the video data transmitted to and downloaded from the server is in an encoded format.

[0129] In some embodiments, the WebRTC module receives remote participant video data obtained from the server via the network and sends it to the browser video processing module. The browser video processing module then sends the remote participant video data to the browser video software decoding module in the central processing unit.

[0130] The central processing unit executes step S615: Decodes the video data of the remote participant;

[0131] In some embodiments, the browser video software decoding module decodes the remote participant video data, that is, decodes the remote participant video data in H264 or H265 format into remote participant video data in YUV format.

[0132] Software decoding is performed using software. Its advantages include broad format support and flexible format addition.

[0133] The central processing unit executes step S616: controlling the display to display the decoded video data of the remote participant in the video display frame of the remote participant.

[0134] In some embodiments, the browser's graphics rendering module draws the video conferencing interface and displays the decoded video data of the remote participants in the video display box of the remote participants.

[0135] Figure 14 A flowchart illustrating a video conferencing interface display as provided in a feasible embodiment. (e.g.) Figure 14 As shown, the process is divided into two parts. The first part involves processing video captured by the local camera. After the local video data captured by the camera is written to the browser's shared memory, the browser's video processing module obtains the local video data and transmits it to the server. If the local video data is encoded, the hardware video decoding module decodes it and displays it through the video display module. If the local video data is unencoded, it is directly displayed through the video display module. The second part involves processing video transmitted from a remote location. The browser's video processing module receives video data from remote participants transmitted from the server, decodes it using the browser's software video decoding module, and displays it through the browser's image rendering module. This embodiment of the application, by combining the hardware video decoding and independent video display module of the display device chip platform with the browser's software video decoding and rendering process, greatly saves CPU and memory usage, fully utilizes the performance of the multimedia module of the display device chip, and accelerates decoding and display efficiency.

[0136] In some embodiments, if the camera is a non-encoded camera, the local video data may not be sent to the graphics processor. Instead, the local video data captured by the camera is written to the browser's shared memory, and the browser's video processing module obtains the non-encoded local video data and sends it to the browser's graphics rendering module for display.

[0137] In some embodiments, after the camera captures local video data and writes it to the browser's shared memory, the browser's video processing module obtains the local video data. It determines whether the local video data is encoded video data or whether the camera is an encoded camera. If the local video data is encoded video data or the camera is an encoded camera, the local video data is sent to the video hardware decoding module, which decodes the local video data into YUV local video data and sends the YUV local video data to the video display module for display. If the local video data is unencoded video data or the camera is an unencoded camera, the local video data is sent to the video display module for display.

[0138] In some embodiments, after the WebRTC module receives video data from the remote participant, the video processor obtains the remote participant video data from the WebRTC module.

[0139] The video processor decodes the video data of the remote participant and sends the decoded video data of the remote participant to the graphics processor;

[0140] After receiving the video data from the remote participant, the WebRTC module sends it to the video hardware decoding module in the video processor. The video hardware decoding module decodes the remote participant video data, that is, it decodes the H264 or H265 encoded remote participant video data into YUV format remote participant video data.

[0141] The graphics processor controls the display to show the decoded video data of the remote participant on the video layer, and the display area of ​​the decoded video data of the remote participant corresponds to the position of the video display frame of the remote participant.

[0142] It should be noted that the display area of ​​the decoded remote participant video data corresponds to the position of the remote participant video display frame. Specifically, the display area of ​​the decoded remote participant video data is the same size as the remote participant video display frame and is displayed in the same position.

[0143] In some embodiments, a Cartesian coordinate system can be established with the intersection of two adjacent sides of the display screen as the zero point and the two adjacent sides as the X and Y axes. The video display frame for the remote participant is rectangular by default and has four vertices. The graphics processor can obtain the coordinates of the four vertices of the video display frame for the remote participant from the browser's graphics rendering module, and then create a video playback window in the video layer at the same coordinates as the four vertices, playing the video data of the remote participant in the video playback window.

[0144] In some embodiments, a Cartesian coordinate system can also be established using the intersection of two adjacent sides of the display screen as the zero point, and the two adjacent sides as the X and Y axes. The video display frame for the remote participant is rectangular by default, with a standard aspect ratio and a center point. The graphics processor can obtain the coordinates of the center point and the width or height of the remote participant video display frame from the browser's graphics rendering module, and then create a video playback window with the same width and height as the remote participant video display frame at the same coordinates as the center point in the video layer, and play the remote participant's video data in the video playback window.

[0145] In some embodiments, after the WebRTC module receives video data from remote participants, the central processing unit obtains the remote participant video data from the WebRTC module and sends the remote participant video data to the video processor.

[0146] The video processor decodes the video data of the remote participant and sends the decoded video data of the remote participant to the graphics processor;

[0147] The graphics processor controls the display to show the decoded video data of the remote participant on the video layer, and the display area of ​​the decoded video data of the remote participant corresponds to the position of the video display frame of the remote participant.

[0148] After receiving video data from remote participants, the WebRTC module sends it to the browser video processing module of the central processing unit. The browser video processing module then sends the remote participant video data to the video hardware decoding module. The video hardware decoding module decodes the remote participant video data, converting the H.264 or H.265 encoded remote participant video data into YUV format. The video hardware decoding module then sends the YUV format remote participant video data to the video display module of the graphics processor. The video display module controls the display to show the decoded remote participant video data at the video layer.

[0149] In some embodiments, after the WebRTC module receives video data from remote participants, the central processing unit obtains the video data from the remote participants from the WebRTC module, and at the same time, obtains the current CPU utilization rate.

[0150] The central processing unit (CPU) determines the amount of video data from remote participants that the video processor will decode based on the CPU's utilization rate.

[0151] The central processing unit (CPU) sends a predetermined number of remote participant video data to the video processor. The video processor decodes the remote participant video data and sends the decoded data to the graphics processor (GPU). The GPU controls the display to show the decoded remote participant video data on the video layer, with the display area corresponding to the position of the remote participant video display frame. The CPU then decodes the remaining number of remote participant video data and displays them within the remote participant video display frame.

[0152] The browser video processing module sends a predetermined number of remote participant video data to the hardware video decoding module. The hardware video decoding module decodes the remote participant video data and sends the decoded data to the video display module. The video display module controls the monitor to display the decoded remote participant video data on the video layer, with the display area corresponding to the position of the remote participant video display frame. The browser video processing module then sends the remaining number of remote participant video data to the browser software video decoding module. The software video decoding module decodes the remaining remote participant video data and displays it in the remote participant video display frame.

[0153] In some embodiments, the number of remote participant video data is n. If the CPU utilization rate is lower than a first preset percentage, the video hardware decoding module decodes 0 remote participant video data, meaning all n remote participant video data are decoded by the browser video software decoding module; if the CPU utilization rate is greater than or equal to the first preset percentage and less than a second preset percentage, the video hardware decoding module decodes 1 remote participant video data, meaning n-1 remote participant video data are decoded by the browser video software decoding module, and so on. The number of times the video hardware decoding module can decode is also related to its decoding capability. For example, if the video hardware decoding module can only decode a maximum of two video data simultaneously, then in addition to decoding the local video data, the video hardware decoding module can only decode one remote participant video data.

[0154] In some embodiments, when a user wants to switch the position of the local video display frame from that of other remote participants during a video conference, after the user inputs a command to change the position of the local video display frame, the central processing unit responds to the user's input command to change the position of the local video display frame and changes the position of the local video display frame on the video conference interface; the graphics processing unit responds to the user's input command to change the position of the local video display frame and changes the display area of ​​the decoded local video data so that the display area of ​​the changed local video data corresponds to the position of the changed local video display frame.

[0155] exist Figure 11 In the middle, after the user selects the local video display box 112, the position of the local video display box 112 and the first remote participant video display box 111 are changed. Figure 15 This is a schematic diagram of a video conferencing interface provided for a feasible embodiment. (As shown...) Figure 15 As shown, the local video display box 151 is transparent in the graphics layer, and the video layer displays the local video data decoded by the video hardware decoding module at the location of the locally acquired video 101. The first remote participant video 152 displays the first remote participant video data decoded by the browser's video software decoding module.

[0156] In some embodiments, as the number of participants increases, the local video display frame and the remote participant video display frame may gradually decrease in size. In response to the instruction to shrink the local video display frame, the central processing unit reduces the area of ​​the local video display frame on the video conferencing interface; in response to the instruction to shrink the local video display frame, the graphics processing unit reduces the display area of ​​the decoded local video data so that the reduced display area of ​​the local video data corresponds to the position of the shrunken local video display frame.

[0157] exist Figure 10 In the process, as the number of participants increases by 6, the area of ​​the local video display frame and the video display frames of other remote participants decreases, such as... Figure 16 As shown. Figure 16 This is a schematic diagram of a video conferencing interface provided for a feasible embodiment.

[0158] In some embodiments, during a video conference, a user may turn off their camera. After the user inputs the command to turn off the camera, the central processing unit responds by setting the local video display frame to a preset color, such as... Figure 17 As shown, or, a preset image is displayed within the local video display frame, the width and height of which are the same as the width and height of the local video display frame, such as... Figure 18As shown. At this time, the camera stops acquiring local video data, and the video processor and graphics processor do not perform any operations related to the video conferencing interface display.

[0159] In some embodiments, during a video conference, a user may want to use an alternative image to capture video data instead of the camera. The user inputs a command to stop displaying local video data, and the central processing unit, in response, sets the local video display frame to a preset color, such as... Figure 17 As shown, or, a preset image is displayed within the local video display frame, the width and height of which are the same as the width and height of the local video display frame, such as... Figure 18 As shown.

[0160] This application provides a method for displaying a video conferencing interface. The method is applicable to a display device, which includes at least a controller and a display. The controller includes a central processing unit (CPU), a video processor, and a graphics processor (GPU). The CPU is configured to: respond to a user's command to join a video conference via a browser interface, control the display to show the video conferencing interface on the graphics layer. The video conferencing interface includes a local video display frame, which is transparent. The video processor is configured to: if the camera is detected to be an encoded camera, receive local video data captured by the camera; decode the local video data; and send the decoded local video data to the GPU. The GPU is configured to: control the display to show the decoded local video data on the video layer, where the display area of ​​the decoded local video data corresponds to the position of the local video display frame, and the video layer is located below the graphics layer. This application directly decodes the locally acquired video data using the display device's video processor and displays it on the video layer using the GPU, instead of using the browser's native modules for decoding and display. This reduces CPU and memory usage while improving the performance of the display device.

[0161] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, it may include some or all of the steps of the various embodiments of the custom control button method and the startup method provided by the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0162] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0164] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the browser interface; The camera is configured to capture local video data; The controller includes a central processing unit, a video processor, and a graphics processor; The central processing unit is configured as follows: In response to a user's command to join a video conference input through a browser interface, the display is controlled to show a video conference interface in the graphics layer. The video conference interface includes a local video display frame, which is transparent. The video processor is configured to: if the camera is detected to be an encoded camera, receive the local video data; decode the local video data; and send the decoded local video data to the graphics processor. The graphics processor is configured to control the display to display decoded local video data on the video layer, wherein the display area of ​​the decoded local video data corresponds to the position of the local video display frame, and the video layer is located below the graphics layer.

2. The display device according to claim 1, characterized in that, The video conferencing interface also includes at least one video display frame for remote participants, and the central processing unit is configured to: Receive video data from remote participants obtained from the server; Decode the video data of the remote participants; The display is controlled to show the decoded video data of the remote participant in the video display frame of the remote participant.

3. The display device according to claim 1, characterized in that, The graphics processor is configured to: if it detects that the camera is a non-encoded camera, receive local video data captured by the camera; and control the display to display the local video data on the video layer.

4. The display device according to claim 1, characterized in that, The central processing unit is configured as follows: Acquire local video data captured by the camera; If the camera is detected to be an coded camera, the local video data is uploaded to the server; If the camera is detected to be a non-encoded camera, the local video data is encoded, and the encoded local video data is uploaded to the server.

5. The display device according to claim 4, characterized in that, The central processing unit is configured as follows: If the camera is detected to be a non-encoded camera, the display is controlled to show the local video data in the local video display box of the graphics layer.

6. The display device according to claim 1, characterized in that, The video conferencing interface also includes at least one video display frame for a remote participant, which is transparent. The central processing unit is configured to: receive video data of remote participants obtained from the server, and send the video data of remote participants to the video processor; The video processor is configured to: decode the video data of the remote participant and send the decoded video data of the remote participant to the graphics processor; The graphics processor is configured to control the display to display decoded remote participant video data on the video layer, wherein the display area of ​​the decoded remote participant video data corresponds to the position of the remote participant video display frame.

7. The display device according to claim 1, characterized in that, In response to a user input instruction to change the position of the local video display frame, the central processing unit is configured to: change the position of the local video display frame in the video conferencing interface; The graphics processor is configured to change the display area of ​​the decoded local video data so that the display area of ​​the changed local video data corresponds to the position of the changed local video display frame.

8. The display device according to claim 1, characterized in that, The central processing unit is configured as follows: In response to a user's input command to turn off the camera or stop displaying local video data, the local video display frame is set to a preset color, or a preset image is displayed in the local video display frame, the width and height of the preset image being the same as the width and height of the local video display frame.

9. A method for displaying a video conferencing interface, characterized in that, include: In response to a user's command to join a video conference via a browser interface, the central processing unit controls the display to show the video conference interface on the graphics layer. The video conference interface includes a local video display frame, which is transparent. If the camera is detected to be an encoded camera, the video processor receives the local video data captured by the camera; decodes the local video data, and sends the decoded local video data to the graphics processor; The graphics processor controls the display to display decoded local video data on the video layer. The display area of ​​the decoded local video data corresponds to the position of the local video display frame. The video layer is located below the graphics layer.

10. The method according to claim 9, characterized in that, The video conferencing interface also includes at least one remote participant video display frame. The central processing unit receives remote participant video data obtained from the server; decodes the remote participant video data; and controls the display to display the decoded remote participant video data in the remote participant video display frame.

Images