Display device and device control method

Abstract

The present disclosure provides a display device and a device control method. In the method, when a consumer electronic control device (CEC) transmits first HDMI content to the display device by means of an input port HDMI IN, the first HDMI content is stored in the virtual device node by constructing HDMI IN as a virtual device node. Then, a shared interface corresponding to the virtual device node is added in a hardware abstraction layer (HAL), so that a first application can obtain the first HDMI content from the virtual device node by calling the shared interface, and then shares the first HDMI content with a first device. In this way, when the HDMI content is displayed on the display device, the display device can also share the HDMI content with the first device, thus preventing a content sharing failure during a remote video conference.

Description

Display device and device control method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese application No. 202311735183.7, filed on December 15, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the technical field of display devices, and in particular to a display device and a device control method. Background Art

[0004] High Definition Multimedia Interface (HDMI) is a specialized digital interface suitable for image transmission that can transmit both audio and image signals. HDMI has been standardized, and Consumer Electronics Control (CEC) is a complete single-bus protocol resulting from HDMI standardization. A transmitting device can use CEC signals to control external devices connected to an HDMI port, where the controlled external device is called a Consumer Electronics Control device (CEC device for short). A CEC device can transmit HDMI content to a transmitting device via an HDMI port, and display the HDMI content on the transmitting device.

[0005] During a remote video conference, if the content displayed on the sending device is HDMI content, since there is no mechanism for sharing HDMI content in the Android system (Android), the sending device cannot share the displayed HDMI content with the receiving device, which leads to the failure of content sharing in the remote video conference.

[0006] Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a display device, comprising: a display configured to display first HDMI content transmitted by a consumer electronics control device CEC via an input port HDMIIN of a high-definition multimedia interface HDMI;

[0008] A user input interface is configured to receive instructions from a user; a communication device is configured to communicate with an external device according to a predetermined protocol; a memory is configured to store computer instructions and data associated with a display device; at least one processor is connected to the display, user input interface, communication device and memory, and is configured to execute computer instructions to cause the display device to perform: in response to a first content sharing request sent by a first application, construct the HDMIIN into a virtual device node so that the first HDMI content is stored in the virtual device node, wherein the first content sharing request is used to request that the first HDMI content be shared with the first device; a sharing interface corresponding to the virtual device node is added to the hardware abstraction layer HAL, so that the first application obtains the first HDMI content from the virtual device node by calling the sharing interface, and shares the first HDMI content with the first device.

[0009] In a second aspect, an embodiment of the present disclosure provides a device control method, including: receiving first HDMI content transmitted by a consumer electronics control device CEC through an input port HDMIIN of a high-definition multimedia interface HDMI; in response to a first content sharing request sent by a first application, constructing the HDMIIN into a virtual device node so that the first HDMI content is stored in the virtual device node, wherein the first content sharing request is used to request that the first HDMI content be shared to a first device; adding a shared interface corresponding to the virtual device node in a hardware abstraction layer HAL, so that the first application obtains the first HDMI content from the virtual device node by calling the shared interface, and shares the first HDMI content to the first device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments;

[0011] FIG2 is a block diagram of a hardware configuration of a control device 100 according to some embodiments;

[0012] FIG3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments;

[0013] FIG4 is a diagram illustrating a software configuration in the display device 200 according to some embodiments;

[0014] FIG5 is a schematic diagram of a remote video conferencing scenario according to some embodiments;

[0015] FIG6 is a schematic diagram of another remote video conferencing scenario according to some embodiments;

[0016] FIG7 is a schematic diagram of an HDMI content sharing system framework according to some embodiments;

[0017] FIG8 is a schematic diagram of a flow chart of a device control method executed by a display device 200 according to some embodiments;

[0018] FIG9 is a schematic diagram of a user interface of a display device 200 according to some embodiments;

[0019] FIG10 is a schematic diagram of an overall design framework of an example of HDMI content sharing according to some embodiments;

[0020] FIG11 is a schematic diagram of a process for simulating an HDMI as a virtual camera according to some embodiments;

[0021] FIG12 is a schematic diagram illustrating a process of transferring HDMI content to an Android application layer according to some embodiments;

[0022] FIG13 is a schematic diagram of a user interface of another display device 200 according to some embodiments;

[0023] FIG14 is a schematic diagram of a process for determining whether a virtual camera is available according to some embodiments;

[0024] FIG15 is a schematic diagram of a logic flow for updating a Camera list according to some embodiments;

[0025] FIG16 is a schematic diagram of another remote video conferencing scenario according to some embodiments. DETAILED DESCRIPTION

[0026] FIG1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments. As shown in FIG1 , a user can operate a display device 200 through a mobile terminal 300 and a control device 100 .

[0027] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0028] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.

[0029] In some embodiments, the display device may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0030] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0031] In some embodiments, the display device 200 can also communicate data with the server 400. The display device 200 can be connected to the server 400 via a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 can provide various content and interactions to the display device 200. The server 400 can be a single cluster or multiple clusters, and can include one or more types of servers.

[0032] Figure 2 is a block diagram of the hardware configuration of a control device 100 according to some embodiments. As shown in Figure 2 , the control device 100 includes a processor 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0033] Figure 3 is a hardware configuration block diagram of the display device 200 according to some embodiments. As shown in Figure 3, the display device 200 may include at least one of a tuner and demodulator 210, a communication device 220, a detector 230, an external device interface 240, at least one processor 250, a display 260, an audio output interface 270, a memory, a power supply, and a user input interface 280.

[0034] In some embodiments, at least one processor 250 may include a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output. The display 260 includes a display screen component for displaying images, a driver component for driving the image display, and a component for receiving image signals output from the processor to display video content, image content, and a menu control interface, as well as a user control UI interface. The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and projection screen.

[0035] The communication device 220 is a component that can be used to communicate with external devices or servers using various communication protocols. For example, the communication device may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. The display device 200 can use the communication device 220 to send and receive control signals and data signals with the external control device 100 or the server 400.

[0036] The user input interface 280 may be configured to receive a control signal from the control device 100 (eg, an infrared remote controller, etc.).

[0037] Detector 230 can be used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0038] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0039] The tuner-demodulator 210 may receive broadcast television signals via a wired or wireless reception method, and demodulate audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0040] In some embodiments, at least one processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the at least one processor 250 is located, such as an external set-top box.

[0041] In some embodiments, the at least one processor 250 can control the operation of the display device and respond to user operations through various software control programs stored in the memory. The at least one processor 250 can control the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the at least one processor 250 can perform operations related to the object selected by the user command.

[0042] In some embodiments, at least one processor 250 may also include a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0043] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0044] A user interface is the medium for interaction and information exchange between an application or operating system and the user. It converts information between its internal form and a user-friendly format. A common user interface is the graphical user interface (GUI), which refers to a graphical user interface related to computer operations. It can be an icon, window, control, or other interface element displayed on an electronic device's display. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0045] As shown in Figure 4, the display device system can be divided into three layers: the application layer, the middleware layer, and the hardware layer. The application layer primarily includes commonly used applications on the TV and the application framework. Common applications are primarily browser-based applications, such as HTML5 apps, as well as native apps.

[0046] The Application Framework is a complete programming model that provides all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces for these functions (toolbars, status bars, menus, and dialog boxes).

[0047] Native apps can support online or offline, message push or local resource access.

[0048] The middleware layer includes various TV protocols, multimedia protocols, and system components. Middleware uses the basic services (functions) provided by system software to connect various parts of the application system or different applications on the network, enabling resource and function sharing.

[0049] The hardware layer primarily includes the Hardware Abstraction Layer (HAL) interface (HAL interface for short), hardware, and drivers. The HAL interface is a unified interface for connecting to TV chipsets, while the specific logic is implemented by each chip. Drivers primarily include: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor drivers (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0050] In order to clearly explain the technical solution of the present disclosure, the terms of the present disclosure are first explained:

[0051] CEC (Consumer Electronics Control) allows end users to control multiple CEC-enabled HD (High Definition) devices with a single remote control, eliminating the need to use multiple remote controls to control electronic devices such as televisions, set-top boxes, and portable HD devices.

[0052] EDID (Extended Display Identification Data) is a data specification designed by the video standards organization VESA for optimizing display formats for PC monitors. EDID contains basic parameters of the display device, such as manufacturer, product name, maximum horizontal and vertical frequency, supported resolution, etc., and is the data behind plug-and-play functionality. EDID can be stored in two ways: internal and external. The internal EDID method means that the EDID data is written into the program. When the smart TV is powered on via DC power, the EDID data is written into the Electrically Erasable Programmable Read-Only Memory (EEPROM) chip. When the smart TV is in use, the EDID data is retrieved from the EEPROM chip. The external EDID method means that the EDID data is directly written into an external EEPROM chip and is read directly by an external device through the display data channel without being controlled by the main program of the smart TV.

[0053] HDMI (High Definition Multimedia Interface) is a digital video / audio interface technology. It is a dedicated digital interface suitable for image transmission. It can transmit audio and image signals simultaneously, with a maximum data transmission speed of 48Gbps (version 2.1). It also does not require digital-to-analog or analog-to-digital conversion before signal transmission. HDMI interfaces are hot-swappable.

[0054] HDCP (High-bandwidth Digital Content Protection) processes every pixel in an image, rendering it irregular and unrecognizable. Only after the sender and receiver are synchronized can the reverse process be performed to restore the data. During the decryption process, the HDCP system confirms the connection every 2 seconds and synchronizes the sender and receiver with an identification code every 128 frames to ensure synchronization.

[0055] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and the hardware circuitry. Its purpose is to abstract the hardware. It hides the details of the platform-specific hardware interfaces and provides the operating system with a virtual hardware platform, making it hardware-independent and portable across multiple platforms. From the perspective of software and hardware testing, both hardware and software testing can be performed independently based on the HAL, enabling parallel testing of both software and hardware.

[0056] Remote video conferencing is essentially a process of content sharing between two devices. Content sharing can be divided into four stages: information collection, information processing, information transmission, and information restoration. Information collection can be performed using the audio and video capture card on the sending device. Information processing and transmission rely on the remote video conferencing server. The remote video conferencing server determines the audio and video signal processing method and forwarding logic based on the current video conferencing mode, and finally transmits the processed audio and video data to each receiving device. After receiving the transmitted audio and video data from the remote video conferencing server, the receiving device decodes the transmitted audio and video data and restores the image and sound to the receiving device for playback.

[0057] The remote video conferencing server is actually the server corresponding to the remote video conferencing application. In the remote video conferencing scenario shown in Figure 5, a user can use the remote video conferencing application on display device 200-1 to initiate a remote video conference with display device 200-2. Display device 200-2 also needs to install and run the same remote video conferencing application. In this remote video conferencing scenario shown in Figure 5, display device 200-1 can share its collected audio and video content with display device 200-2, and display device 200-2 can also share its collected audio and video content with display device 200-1. In other words, both display device 200-1 and display device 200-2 can serve as both the sending device and the receiving device in the remote video conference.

[0058] If the audio and video content that needs to be shared is ordinary content, the display device 200-1 and the display device 200-2 can share the content normally. However, if the sending device uses the HDMI port to receive the HDMI content transmitted by the CEC device and needs to share the HDMI content with the receiving device, since there is no mechanism for sharing HDMI content in the relevant technology of the Android system, the sending device cannot share the displayed HDMI content to the receiving device, which leads to the failure of content sharing in the remote video conference.

[0059] For example, in the application scenario shown in Figure 6, display device 200-1 is connected to a CEC device, and the CEC device transmits HDMI content to display device 200-1. At this time, because there is no mechanism for sharing HDMI content in the relevant technologies of the Android system, display device 200-1 cannot share its currently displayed HDMI content with display device 200-2. This will cause the remote video conference content sharing between display devices 200-1 and 200-2 to fail. The HDMI content displayed on display device 200-1 will not be displayed on display device 200-2, and display device 200-2 may even display a black screen.

[0060] Figure 7 is a diagram of an HDMI content sharing system framework according to some embodiments. The content sharing system framework shown in Figure 7 includes display device 200-1 and display device 200-2. Display device 200-1 is connected to CEC device 300 via an HDMI port, and CEC device 300 transmits first HDMI content to display device 200-1 via HDMIIN (i.e., HDMI input port). After display device 200-1 receives the first HDMI content transmitted by CEC device 300, it can display the first HDMI content on a display.

[0061] FIG8 is a flow chart of a device control method provided by an embodiment of the present disclosure executed by a display device 200. As shown in FIG8 , the method includes the following steps:

[0062] Step S800: receiving first HDMI content;

[0063] The first HDMI content is transmitted by the CEC device via HDMI IN. After receiving the first HDMI content, the display device 200 outputs the first HDMI content to the display for display.

[0064] Step S801: In response to a first content sharing request sent by a first application, construct the HDMIIN into a virtual device node so that the first HDMI content is stored in the virtual device node, wherein the first content sharing request is used to request sharing the first HDMI content to a first device.

[0065] The first application may be a remote video conferencing application. The first application needs to be run on display device 200-1 and display device 200-2 (ie, the first device). Display device 200-1 and display device 200-2 use the first application to conduct a remote video conference, ie, to share content.

[0066] The user interface shown in Figure 9 is that of the first application. The user interface of the first application includes a screen display area and a function operation area. The screen display area displays the first HDMI content transmitted by the CEC device 300 to the display device 200-1. The function operation area displays different button controls for implementing sharing functions: an "HDMI Share" button control, a "Desktop Share" button control, and a "Whiteboard Share" button control. Different button controls implement different sharing functions. For example, the "HDMI Share" button control implements HDMI content sharing, while the other button controls implement general content sharing. If a user wishes to share HDMI content, they can click the "HDMI Share" button control to enter a first content sharing request. This request is used to request that the first HDMI content be shared with the display device 200-2. If the user only wishes to share general content with the display device 200-2, they can click other button controls to enter other content sharing requests. In other words, sharing the HDMI content with the display device 200-2 can only proceed after receiving the first content sharing request entered by the user.

[0067] After responding to the first content sharing request sent by the first application, HDMIIN can be constructed into a virtual device node. In the embodiment of the present disclosure, HDMIIN is first registered as V4L2_device to generate the corresponding vedio virtual device node. V4L2 (Video for linux two) is a set of driver frameworks for video devices in the Linux kernel, which provides a unified interface specification for video device driver development and application layer. Devices registered using the V4L2 device driver framework will generate corresponding device node files in the Linux system / dev / directory. The name of the device node is usually videoX (X standard a digital number: / dev / videox), and each videoX device file represents a video device. The application configures and uses the virtual device by performing I / O (Input / Output) operations on the videoX device file.

[0068] The device node corresponding to a video device is / dev / videoX, where X is a number, typically starting at 0. Calling the open() function to open the device node yields the file descriptor fd. After opening the virtual device, you need to query its properties to determine whether it is a video capture device. Using ioctl(), you'll retrieve a struct v4l2_capability data structure, which describes the virtual device's properties, including driver (driver name), card (device name), bus_info (bus name), version (version information), and capabilities (device capabilities). The capabilities field describes the device's capabilities, so you can determine whether the device is a camera by checking whether the capabilities field contains V4L2_CAP_VIDEO_CAPTURE.

[0069] During implementation, HDMIIN can be first registered as V4L2_device to generate a virtual camera node corresponding to HDMIIN. The data stream of the virtual camera node is the first HDMI content. The specific implementation can be to store the first HDMI content in the virtual device node.

[0070] It should be noted that the premise for constructing HDMIIN into a virtual camera node is that the first HDMI content only includes video content, that is, only the video content needs to be shared between the display device 200-1 and the display device 200-2, and the constructed virtual camera node is only used as a virtual camera. Therefore, only video content can be obtained from the virtual camera node, but not audio content.

[0071] Step S802: adding a shared interface corresponding to the virtual device node in the hardware abstraction layer HAL.

[0072] After constructing HDMIIN into a virtual device node, it is also necessary to add a shared interface corresponding to the virtual device node in the HAL. If HDMIIN is constructed as a virtual camera node, the HAL is the Camera HAL, that is, support for the constructed virtual camera node is added to the Camera HAL, that is, a shared interface for the virtual camera node is added to the Camera HAL. Then the first application can open the virtual camera node in the Camera HAL. After the first application opens the virtual camera node in the Camera HAL, the first application can read the first HDMI content from the virtual camera node, and then the first application can share the first HDMI content read from the virtual camera node with the display device 200-2, thereby enabling the display device 200-1 to share the HDMI content with the display device 200-2.

[0073] Figure 10 shows the overall design framework of the HDMI content sharing example described above, including the HDMI port, the VDIN module (hardware processing module), the Camera HAL, the GE2D (2D graphics acceleration engine) module, the Surfacefliger (the core of the graphical user interface, a system service responsible for blending all app graphics data in Z-order and outputting it to the FrameBuffer), and the app. VDIN and GE2D are hardware modules, Surfacefliger and the app are software modules, and the video output and preview are DMA buffers. After receiving HDMI content from the HDMI port, the VDIN module converts (e.g., decodes) the HDMI content, then inputs the camera HAL's video output and passes through the GE2D module for graphics performance processing. This processed data is then returned to the camera HAL's preview plugin. Finally, after blending by Surfacefliger, it is output to the app for display and content sharing.

[0074] FIG11 shows a process of simulating HDMI as a virtual camera in the above HDMI content sharing example, including the following steps:

[0075] Step S1101: registering a V4L2 device;

[0076] Step S1102: when an HDMI RX signal input is detected, the input HDMI RX signal is converted into stream file data through the hardware processing module VDIIN;

[0077] Step S1103: Obtain the stream file data converted from the HDMI RX signal through the vdin_v4l2_isr interface, and import the stream file data into the V4L2 device.

[0078] During implementation, you can first register a virtual device V4L2 and generate a virtual camera device node (virtual camera node), for example, define the node as vedio70. When an HDMI RX (receive) signal input is detected, the input HDMI RX signal is converted into stream file data through the hardware processing VDIIN module. Then, in the registered V4L2 device, the stream file data converted from the HDMI RX signal is obtained through the vdin_v4l2_isr interface. The data is imported into the registered V4L2 device through vdin_v4l2_if_isr. Test that the data in the V4L2 device node is the same as the stream file data converted from the HDMI RX signal, thereby completing the conversion of HDMIIN into a virtual camera device node. The virtual camera device node obtained through the above process can be called by Android's camera HAL.

[0079] FIG12 shows the flow process of HDMI content to the Android application layer in the above HDMI content sharing example, including the following steps:

[0080] Step S1201: The first application Android APP sends a call instruction to the server CameraService;

[0081] Step S1202: the server CameraService calls the hardware abstraction layer Camera HAL to obtain data;

[0082] Step S1203: the first application Android APP sends a display surface Surface or a buffer to the hardware abstraction layer Camera HAL;

[0083] Step S1204: The high-definition multimedia interface HDMI writes the data sent by the CEC device into the V4L2 virtual device;

[0084] Step S1205: the hardware abstraction layer Camera HAL uses the V4L2 protocol to obtain data from the virtual device V4L2;

[0085] Step S1206: The hardware abstraction layer Camera HAL decodes the data;

[0086] Step S1207: The hardware abstraction layer Camera HAL sends the decoded data to the first application Android APP;

[0087] In some embodiments, the Android app can call the Camera HAL through the CameraService, and the Camera HAL opens the real vedioX node, which corresponds to the V4L2 device, i.e., the virtual camera device.

[0088] In addition, the Android app can pass its display surface or the user data buffer to the Camera HAL. The Camera HAL then opens the V4L2 virtual device and continuously retrieves data from it using the V4L2 protocol. This is because HDMI continuously writes data to the V4L2 virtual device through HDMIIN, allowing the Camera HAL to continuously retrieve data written by the CEC device from the V4L2 virtual device.

[0089] There are two ways for the Camera HAL to read data from the V4L2 virtual device: one is the read method, which directly reads the data written by the V4L2 virtual device through the read() system call; the other is the streaming method. Use the VIDIOC_QUERYCAP instruction to query the device's properties and obtain a struct v4l2_capability type data, where the capabilities field records the capabilities of the device. When the field contains V4L2_CAP_READWRITE, it means that the device supports read I / O to read data; when the field contains V4L2_CAP_STREAMING, it means that the device supports streaming I / O. Using the streaming I / O method, you need to apply for a frame buffer from the device and map the frame buffer to the application process address space.

[0090] It should be noted that since only the virtual device node corresponding to HDMIIN is created, the data written by the CEC device needs to be actually stored in physical memory. The data written by the CEC device can be mapped from physical memory to the virtual memory corresponding to the V4L2 virtual device. Specifically, the physical memory can be mapped to the user control through the vedioX node to obtain the virtual address of the virtual memory corresponding to the V4L2 virtual device, and the virtual address points to the physical memory. Therefore, when the Camera HAL reads the first HDMI content from the virtual device node, it can read the first HDMI content from the physical memory based on the virtual address.

[0091] After reading data from the V4L2 virtual device, the Camera HAL begins decoding the data. Finally, the Camera HAL sends each successfully decoded frame of data to the Android app's surface for display on display device 200-1. After the decoded data is sent to a buffer, the Android app processes the data, including displaying it on the screen or sharing it with other devices.

[0092] In a device control method provided by an embodiment of the present disclosure, when the CEC device 300 transmits the first HDMI content to the display device 200-1 through HDMIIN, if the first HDMI content is shared to the display device 200-2 in response to a request, HDMIIN is constructed into a virtual device node so that the first HDMI content is stored in the virtual device node. Then, a shared interface corresponding to the virtual device node is added to the HAL. The first application can obtain the first HDMI content from the virtual device node by calling the shared interface, and then share the first HDMI content to the display device 200-1. In this way, when the HDMI content is displayed on the display device 200-2, the display device 200-2 can also share the HDMI content to the display device 200-2, thereby avoiding the failure of content sharing during the remote video conference.

[0093] In some embodiments, if the first HDMI content is not HDCP encrypted when it is transmitted from the CEC device 300 to the display device 200-1, the first HDMI content only needs to be directly stored in the virtual device node. If the first HDMI content is HDCP encrypted when it is transmitted from the CEC device 300 to the display device 200-1, the first HDMI content needs to be HDCP decrypted and then the HDCP-decrypted first HDMI content needs to be stored in the virtual device node. In this way, the first application obtains the HDCP-decrypted first HDMI content from the virtual device node by calling the sharing interface, and shares the decrypted first HDMI content to the display device 200-2. Therefore, the embodiment of the present disclosure can not only establish a sharing mechanism for sharing HDMI content with other devices through a video conferencing application, but also, for HDCP-encrypted HDMI content, it can also perform HDCP decryption in the current device and share the decrypted HDMI content with other devices, further avoiding the situation where the HDMI content fails to be displayed on other devices when sharing the HDMI content.

[0094] In some embodiments, the display device 200-1 may also include an image collector, such as a built-in camera or a USB camera. Therefore, it can be determined whether to use a real camera or a virtual camera based on the usage scenario. The user interface shown in Figure 13 is a user interface of the first application, which includes a "real camera" area and a "virtual camera" area, wherein the "real camera" area includes a built-in camera and a USB camera, and the "virtual camera" area includes HDMI sharing 1 and HDMI sharing 2. If the user needs to share HDMI content, he can enter a first content sharing request by clicking the "HDMI sharing 1" button control or the "HDMI sharing 2" button control, and then share the first HDMI content to the display device 200-2 according to the first content sharing request. If the user does not need to share HDMI content, he can enter a second content sharing request by clicking the button control in the "real camera" area, and then share the second image content captured by the real camera to the display device 200-2 according to the second content sharing request.

[0095] In the application scenario shown in FIG13 , preview images of different cameras can be displayed in the screen display area, and the user can determine the camera to be selected by viewing the preview image. For example, CEC device 1 and CEC device 2 correspond to HDMI share 1 and HDMI share 2, respectively. CEC device 1 can input high-definition movie resources to display device 200-1, while CEC device 2 can only input movie resources of normal image quality. If a selection instruction is received by the user by clicking "HDMI share 1", the high-definition movie video image can be previewed in the screen display area; if a selection instruction is received by the user by clicking "HDMI share 2", the normal image quality movie video image can be previewed in the screen display area. Users can share different HDMI content input by different CEC devices to display device 200-2 by clicking different camera button controls according to their needs.

[0096] For real cameras, you can also display preview images in the screen display area for users to judge and select. For example, the built-in camera and USB camera shoot at different angles. By displaying preview images of different angles in the screen display area, you can remind users of the different shooting angles corresponding to different cameras.

[0097] In some embodiments, before executing the response to the first content sharing request sent by the first application, it is necessary to determine whether the first content sharing request carries a predetermined parameter, where the predetermined parameter indicates that the current scenario allows the first HDMI content to be shared with the display device 200-2. Only after determining that the first content sharing request carries the predetermined parameter, the HDMI content sharing process is performed. If it is determined that the first content sharing request does not carry the predetermined parameter, the HDMI content sharing process is not performed.

[0098] For example, in some applications that use cameras, some camera references cannot use virtual cameras. Therefore, it is necessary to add the logic processing shown in Figure 14 to the Camera HAL to enable the camera. As shown in Figure 14, the following steps are included:

[0099] Step S1401, defining and adding different camera categories;

[0100] In some embodiments, different camera categories may be defined and added in the Camera HAL and the Camera Service. The categories may include a built-in camera, a USB camera, and an HDMI virtual camera (Vcam0).

[0101] Step S1402: Update the properties of the virtual camera according to the upper application call.

[0102] If the upper-layer application can use the virtual camera, the Vcam0_id attribute is set to 0. If the upper-layer application cannot use the virtual camera, the Vcam0_id attribute is set to -1. The corresponding virtual camera ID (HDMIcameraid) is also updated, for example, HDMIcameraid is set to camera0 or camera1.

[0103] Step S1403, determining whether the virtual camera is available;

[0104] When the upper layer application opens the camera, it determines whether the virtual camera is available based on the attribute setting of Vcam0_id. If it is determined that the virtual camera is available, step S1404 is performed; if it is determined that the virtual camera is not available, step S1405 is performed.

[0105] Step S1404: If the virtual camera is available, open the virtual camera.

[0106] In some embodiments, the virtual camera corresponding to the virtual camera ID can be determined according to HDMI camera ID, and then HDMI data can be read from the virtual camera.

[0107] Step S1405: If the virtual camera is not available, determine whether there is an available real camera;

[0108] Step S1406: If the real camera is available, open the real camera;

[0109] In some embodiments, the real camera may be a built-in camera or a USB camera.

[0110] Step S1407: If the real camera is not available, a prompt indicating that there is no camera is output.

[0111] In some embodiments, a first list can be configured in the display device 200-1, and the first list can record all currently available virtual device nodes. When a virtual device node needs to be called, the application can search for available virtual device nodes from the first list through the HAL. In the embodiment of the present disclosure, after executing the first content sharing request sent by the first application in response to constructing HDMIIN into a virtual device node, the constructed virtual device node can be added to the first list, so that the HAL can find the virtual device node corresponding to HDMIIN from the first list. If the CEC device is unplugged from the display device 200-1, the virtual device node corresponding to HDMIIN can be deleted from the first list according to the unplug signal. In this way, the HAL can no longer find the virtual device node corresponding to HDMIIN from the first list, and thus cannot open the virtual device node corresponding to HDMIIN.

[0112] The specific logical implementation of the above process is shown in Figure 15:

[0113] Step S1501: The Camera HAL process starts and monitors the creation and deletion of the / dev / videoX node. When HDMIIN is connected, step S1501 is performed. When HDMIIN is unplugged, step S1502 is performed.

[0114] Step S1502: If the creation of the / dev / videoX node is detected, a Camera object corresponding to the node is generated;

[0115] Step S1503: Add the Camera object to the maintained Camera list, update the list and notify the server Camera Service to synchronize the update;

[0116] In some embodiments, the Camera HAL may add the Camera object to the maintained Camera list (i.e., the first list), update the IDs of all Cameras in the list, notify the upper-layer Camera Service to synchronize the updated list and CameraID, and notify the upper-layer Camera Service to synchronize the updated list and CameraID;

[0117] Step S1504: If the deletion of the / dev / videoX node is detected, the Camera object corresponding to the node in the Camera list is removed, the list is updated, and the server Camera Service is notified to synchronize the update;

[0118] In some embodiments, the Camera HAL can delete the Camera object from the list, update the IDs of all Cameras in the list, and notify the upper-layer Camera Service to synchronize the updated list and CameraIDs. Based on the above process, Android can dynamically detect the addition and deletion of Camera objects based on the connection and removal of HDMIIN.

[0119] It should be noted that the Camera list contains multiple Camera objects. This situation can occur when a display device can have multiple HDMI ports, and multiple CEC devices 300 are connected to the display device 200-1 through the HDMI ports, and multiple CEC devices 300 can input HDMI content to the display device 200-1. When multiple CEC devices 300 are connected to the display device 200-1, virtual device nodes can be constructed for different HDMIINs. Then, shared interfaces are added to the HAL layer for different virtual device nodes. Applications can call different shared interfaces to read HDMI content from different virtual device nodes, thereby enabling different HDMI content to be shared with the display device 200-2.

[0120] For example, in the application scenario shown in Figure 16, CEC device 300-1 is connected to display device 200-1 via HDMI port 1, while CEC device 300-2 is connected to display device 200-1 via HDMI port 2. Both CEC devices 300-1 and 300-2 input HDMI content to display device 200-1. To facilitate user switching of transmission lines, a virtual device node / dev / video1 can be constructed for the HDMIIN corresponding to CEC device 300-1, and a virtual device node / dev / video1 can be constructed for the HDMIIN corresponding to CEC device 300-2. Camera objects corresponding to each virtual device node are then generated: the Camera object corresponding to / dev / video1 can be Camera1, and the Camera object corresponding to / dev / video2 can be Camera2. Both Camera objects are added to the Camera list. The HAL can then find both Camera1 and Camera2 in the Camera list.

[0121] Here, although two CEC devices are connected to display device 200-1, the user typically selects which CEC device to receive the HDMI content input based on the different HDMI ports connected to the different CEC devices. The HAL can choose to open different virtual device nodes to read different HDMI content based on the selection instructions input by the user. For example, in response to a user selection instruction to play the HDMI content input by CEC device 300-1, the HAL selects to open the virtual device node / dev / video1 and then continuously reads the HDMI content from the virtual device node / dev / video1. This allows the HDMI content input by CEC device 300-1 to be shared with display device 200-2. In response to a user selection instruction to play the HDMI content input by CEC device 300-2, the HAL selects to open the virtual device node / dev / video2 and then continuously reads the HDMI content from the virtual device node / dev / video2. This allows the HDMI content input by CEC device 300-2 to be shared with display device 200-2.

[0122] In some embodiments, if the first HDMI content to be shared with display device 200-2 only includes video content, then in response to the first content sharing request sent by the first application, it is only necessary to construct HDMIIN into a virtual camera node representing the virtual device node, and then store the video content in the virtual camera node. Similarly, it is only necessary to add a sharing interface corresponding to the virtual camera node in the Camera HAL. In this way, the first application can obtain video content from the virtual camera node by calling the sharing interface in the Camera HAL and share the video content with display device 200-2.

[0123] If the first HDMI content that needs to be shared to the display device 200-2 includes video content and audio content, then in response to the first content sharing request sent by the first application, HDMIIN is constructed into a virtual camera node and a virtual microphone node that represent the virtual device node. Then, the video content in the first HDMI content is stored in the virtual camera node, and the audio content in the first HDMI content is stored in the virtual microphone node. And it is necessary to add a shared interface corresponding to the virtual camera node in the Camera HAL, and it is necessary to add a shared interface corresponding to the virtual microphone node in the Audio HAL. In this way, the first application can obtain video content from the virtual camera node by calling the shared interface in the Camera HAL, and can obtain audio content from the virtual microphone node by calling the shared interface in the Audio HAL. Then the first application can share the video content and audio content with the display device 200-2.

[0124] In some embodiments, display device 200 can simultaneously install camera application A and camera application B. Camera application A is an application that uses only a real camera, such as a photo-taking application, while camera application B is an application that can use both a virtual camera and a real camera, such as a remote video conferencing application. When camera application A is running, it sends a call request to invoke the real camera, allowing it to directly obtain image data from the real camera. When camera application B is running, it simultaneously sends a call request to invoke the real camera and a call request to invoke the virtual camera. Upon receiving the call request to invoke the virtual camera, a corresponding virtual device node must be constructed. Therefore, display device 200-1 now has two device nodes: the device node / dev / video0 for the real camera and the virtual device node / dev / videoX for the virtual camera. The device node / dev / video0 for the real camera corresponds to the call request to invoke the real camera, while the virtual device node / dev / videoX for the virtual camera corresponds to the call request to invoke the virtual camera.

Claims

1. A display device, comprising: The display is configured to display a first HDMI content transmitted by a consumer electronics control device CEC through an input port HDMIIN of a high-definition multimedia interface HDMI; A user input interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, the user input interface, the communication device and the memory, is configured to execute computer instructions to cause the display device to perform: In response to a first content sharing request sent by a first application, construct the HDMIIN into a virtual device node so that the first HDMI content is stored in the virtual device node, wherein the first content sharing request is used to request that the first HDMI content be shared to a first device; A shared interface corresponding to the virtual device node is added in the hardware abstraction layer HAL, so that the first application obtains the first HDMI content from the virtual device node by calling the shared interface, and shares the first HDMI content to the first device.

2. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the first HDMI content transmitted from the CEC device to the display device is encrypted using the High-bandwidth Digital Content Protection (HDCP) technology, then in response to the content sharing request sent by the first application, HDCP decrypt the first HDMI content; The decrypted first HDMI content is stored in the virtual device node, so that the first application obtains the first HDMI content decrypted by HDCP from the virtual device node by calling the sharing interface, and shares the decrypted first HDMI content to the first device.

3. The display device according to claim 1, wherein the first application obtains the first HDMI content in the following manner: Calling the shared interface, opening the virtual device node in the HAL, and then reading the first HDMI content from the virtual device node; The buffer corresponding to the first application is transmitted to the HAL, so that the HAL stores the first HDMI content read from the virtual device node into the buffer corresponding to the first application.

4. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After constructing the virtual device node, adding the virtual device node to the first list; wherein, The first list is used to record currently available virtual device nodes, and the HAL is used to search for available virtual device nodes from the first list.

5. The display device according to claim 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If it is detected that the CEC device is not connected to the display device, the virtual device node is deleted from the first list.

6. The display device according to claim 1, further comprising an image collector, wherein the image collector is configured to collect second image content, and the at least one processor is further configured to execute computer instructions to enable the display device to execute: In response to the second content sharing request sent by the first application, the second image content is sent to the first application, so that the first application shares the second image content to the first device.

7. The display device according to claim 6, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Before responding to a first content sharing request sent by a first application, determining whether the first content sharing request carries a predetermined parameter; wherein, The predetermined parameter indicates that the current scenario allows the first HDMI content to be shared to the first device; If the predetermined parameter is not carried, the second image content is sent to the first application, so that the first application shares the second image content to the first device.

8. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device constructs the HDMIIN into a virtual device node by performing the following: If it is detected that the first HDMI content is video content, constructing the HDMIIN as a virtual camera node representing the virtual device node; The at least one processor is specifically configured to execute computer instructions so that the display device adds a shared interface corresponding to the virtual device node in the hardware abstraction layer HAL by performing the following: A shared interface corresponding to the virtual camera node is added in the HAL to store the video content in the virtual camera node, and the first application obtains the first HDMI content from the virtual device node by calling the shared interface, and shares the first HDMI content to the first device.

9. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device constructs the HDMIIN into a virtual device node by performing the following: If it is detected that the first HDMI content includes video content and audio content, construct the HDMIIN as a virtual camera node and a virtual microphone node of the standard virtual device node, so as to store the video content in the first HDMI content to the virtual camera node, and store the audio content in the first HDMI content to the virtual microphone node; The at least one processor is specifically configured to execute computer instructions so that the display device adds a shared interface corresponding to the virtual device node in the hardware abstraction layer HAL by performing the following: A first shared interface corresponding to the virtual camera node and a second shared interface corresponding to the virtual microphone node are added to the HAL, so that the first application obtains the video content in the first HDMI content from the virtual camera node by calling the first shared interface, and the first application obtains the audio content in the first HDMI content from the virtual microphone node by calling the second shared interface, and the video content and audio content in the first HDMI content are shared to the first device.

10. A device control method, comprising: receiving first HDMI content transmitted by a consumer electronics control device CEC through an input port HDMIIN of a high-definition multimedia interface HDMI; In response to a first content sharing request sent by a first application, construct the HDMIIN into a virtual device node so that the first HDMI content is stored in the virtual device node, wherein the first content sharing request is used to request that the first HDMI content be shared to a first device; Add a shared interface corresponding to the virtual device node in the hardware abstraction layer HAL so that the first application The first HDMI content is obtained from the virtual device node by calling the sharing interface, and the first HDMI content is shared to the first device.

Images