Display device, control method of display device

Abstract

The application provides a display device and a control method of the display device. The display device comprises a display screen configured to display an image, an image acquisition device configured to acquire a video stream, wherein each frame of the video stream comprises a remote controller configured to control the display device, and a processor connected to the display screen, wherein the processor is configured to: identify whether a user operates the remote controller according to the video stream; if it is identified that the user operates the remote controller, determine an identifier of a target key of the remote controller according to the video stream, wherein the target key is a key pressed by the user when operating the remote controller; and perform an operation corresponding to the identifier of the target key on the display device. The application improves the flexibility of the user in controlling the display device and improves the user experience.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to control technology. More particularly, to a display device and a control method of the display device. BACKGROUND

[0002] A user can control a display device (for example, a smart television) by pressing different buttons on a remote controller. The remote controller can mainly communicate with the display device through an infrared protocol, or a Bluetooth protocol, and the like, to send a control signal to the display device. In response, the display device can perform corresponding operations based on the control signal received from the remote controller.

[0003] However, when the remote controller has a low battery level, or cannot connect to the display device, the user will not be able to control the display device through the remote controller. SUMMARY

[0004] Exemplary embodiments of the present application provide a display device and a control method of the display device, which can improve the flexibility of the user using a remote controller to control the display device, and thus improve the user experience.

[0005] In a first aspect, the present application provides a display device, comprising:

[0006] a display screen configured to display an image;

[0007] a processor connected to the display screen, the processor being configured to:

[0008] receive a video stream, each frame of the video stream including a control device configured to control the display device;

[0009] identify whether a user operates the control device according to the video stream;

[0010] if it is identified that the user operates the control device, determine an identity of a target button of the control device according to the video stream, the target button being a button pressed by the user when operating the control device;

[0011] perform an operation corresponding to the identity of the target button on the display device.

[0012] In some embodiments, the processor is configured to:

[0013] perform image recognition on each frame of the video stream to obtain a position of each first button of the control device in each frame of the video stream;

[0014] add a key identifier to each first key of the control device according to a position of each first key of the control device in the i-th image in the video stream; and

[0015] track and locate each first key in each image after the i-th image in the video stream according to a position of each first key of the control device in each image after the i-th image, and the identifier of each first key, to obtain the identifier of each first key and the identifier of at least one second key in each image after the i-th image;

[0016] determine the identifier of a target key of the control device according to the identifier of each first key and the identifier of the at least one second key; the target key is one of the at least one second key.

[0017] In some embodiments, the processor is configured to:

[0018] for any image after the i-th image, obtain a distance between any one first key in the image and each first key in an image after the image according to a first position of each first key of the control device in the image and a second position of each first key of the control device in the image after the image;

[0019] for any one first key in the image after the image, take the identifier of the first key with the smallest distance to the first key in the image as the identifier of the first key in the image after the image.

[0020] In some embodiments, the processor is configured to:

[0021] determine at least one first key adjacent to the second key according to the identifier of the second key;

[0022] determine the position of the second key according to the at least one first key;

[0023] obtain a luminance value of a pixel corresponding to the position of the second key and an initial luminance value; the initial luminance value is a luminance value of a pixel corresponding to a position of a first key with the same identifier as the second key in the video stream;

[0024] determine the identifier of a target key of the control device according to a difference between the luminance value of the pixel corresponding to the position of the second key and the initial luminance value, and the identifier of the first key.

[0025] In some embodiments, the processor is configured to:

[0026] determine whether an absolute value of a difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to a second preset luminance threshold value;

[0027] If the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to the second preset luminance threshold value, the processor is configured to determine the identification of the target key of the control device according to the identification of each first key.

[0028] In some embodiments, the processor is configured to:

[0029] determine whether a function of the display device controlled by the control device is available;

[0030] If the function of the display device controlled by the control device is not available, the processor is configured to receive the video stream.

[0031] In some embodiments, the processor is configured to:

[0032] If the function of the display device controlled by the control device is not available, the processor is configured to control the display screen to display prompt information, wherein the prompt information includes a virtual control device corresponding to the control device, and the prompt information is used to prompt a user to perform virtual remote control operation.

[0033] In some embodiments, the processor is configured to:

[0034] After the display screen displays the prompt information for a preset time length, if it is determined that the user does not operate the control device, the processor is configured to control the display screen to stop displaying the prompt information.

[0035] In some embodiments, the processor is configured to:

[0036] perform an operation corresponding to the identification of the target key on the display device according to the identification of the target key and a mapping relationship between the identification of the key and the operation performed on the display device;

[0037] The processor is configured to:

[0038] update the mapping relationship between the identification of the key and the operation performed on the display device.

[0039] In a second aspect, the present application provides a display device control method, the method comprising:

[0040] receiving a video stream, each frame of image of the video stream including a control device used to control the display device;

[0041] According to the video stream, it is identified whether a user operates the control device;

[0042] If it is identified that the user operates the control device, according to the video stream, an identity of a target key of the control device is determined; the target key is a key pressed by the user when operating the control device;

[0043] The display device is executed an operation corresponding to the identity of the target key.

[0044] In a third aspect, the present application provides a display device control apparatus, the apparatus comprising:

[0045] A receiving module is configured to receive a video stream; each frame image of the video stream comprises a control device used to control the display device;

[0046] An identifying module is configured to identify, according to the video stream, whether a user operates the control device;

[0047] A processing module is configured to, when it is identified that the user operates the control device, determine, according to the video stream, an identity of a target key of the control device; the target key is a key pressed by the user when operating the control device;

[0048] An executing module is configured to execute, on the display device, an operation corresponding to the identity of the target key.

[0049] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, the method of the second aspect is implemented.

[0050] The display device and the display device control method provided by the present application can determine whether a user operates a control device by performing image recognition on images of a video stream comprising the control device. When it is determined that the user operates the control device, the display device can determine, according to each frame image in the video stream, an identity of a key (i.e. a target key) of the control device operated by the user. Then, the display device can execute, according to the identity of the target key, an operation corresponding to the identity of the target key on the display device. Through the above method, the display device does not need to be executed a corresponding operation according to a control signal sent by the control device, so that when the control device cannot control the function of the display device, the user can still use the control device to control the display device, thereby improving the flexibility of the user in controlling the display device and further improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the implementation manners in the related art, the accompanying drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art.

[0052] Figure 1 A schematic diagram of an operation scenario between a display device and a control device is provided for the present application.

[0053] Figure 2 A hardware configuration block diagram of a display device 200 is provided for the present application.

[0054] Figure 3 A flowchart of a display device control method is provided for the present application.

[0055] Figure 4 A schematic diagram of an interface of a display device displaying prompt information including a virtual control device is provided for the present application.

[0056] Figure 5 A schematic diagram of an interface of a display device outputting first prompt information is provided for the present application.

[0057] Figure 6 A schematic diagram of an interface of a display device displaying a virtual control device is provided for the present application.

[0058] Figure 7 A schematic diagram of an interface of a display device guiding a user to input a mapping relationship between an identification of a key and an operation performed on the display device is provided for the present application.

[0059] Figure 8 A flowchart of a method for determining an identification of a target key of a control device is provided for the present application.

[0060] Figure 9 A flowchart of another display device control method is provided for the present application.

[0061] Figure 10 A structural schematic diagram of a display device control apparatus is provided for the present application. DETAILED DESCRIPTION

[0062] In order to make the purposes, implementation manners and advantages of the present application more clear, the following will clearly and completely describe the exemplary implementation manners of the present application by combining the accompanying drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some embodiments of the present application, but not all the embodiments.

[0063] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0064] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not have to be limited to the clearly listed components, but can include other components not clearly listed or inherent to these products or devices.

[0065] The term "remote controller" used in the present application can generally be wirelessly controlled within a short distance range. It is generally connected with the display device using infrared and / or radio frequency (RF) signals and / or Bluetooth, and can also include WiFi, wireless USB, Bluetooth, motion sensor, etc. Functional modules.

[0066] Exemplary, Figure 1 A schematic diagram of an operating scenario between a display device and a control device is provided in the present application. As Figure 1 As shown in the figure, a user can operate the display device 200 through the control device 100.

[0067] In some embodiments, the control device 100 can be a remote controller, and the communication between the remote controller and the display device can be infrared protocol communication or Bluetooth protocol communication, and other short-distance communication methods, etc. The display device 200 is controlled by wireless or other wired means. The user can input user instructions through the keys on the remote controller to control the display device 200. For example, the user can input corresponding control instructions through the volume up / down keys, channel control keys, up / down / left / right movement keys, voice input keys, menu keys, power on / off keys, etc. on the remote controller to realize the function of controlling the display device 200.

[0068] The display device 200 can be a liquid crystal display screen, an OLED display screen, or a projection display device. The specific display device type, size, resolution, etc. are not limited, and those skilled in the art can understand that the display device 200 can be changed in performance and configuration as needed. In addition to providing broadcast receiving television functions, the display device 200 can also provide intelligent network television functions supporting computer functions, including but not limited to network television, smart television, Internet protocol television (IPTV), etc.

[0069] Exemplary, Figure 2 A hardware configuration block diagram of a display device 200 is provided in the present application. As Figure 2As shown, in some embodiments, the display device 200 comprises at least one of a communicator 220, a detector 230, a controller 250, a memory 260, a display screen 275, and a user interface 265.

[0070] In some embodiments, the communicator 220 can comprise a WIFI module 221, a Bluetooth module 222, a wired Ethernet module 223, and the like.

[0071] In some embodiments, the display screen 275 is configured to receive image signals originated from the first processor output, and display video contents and images as well as menu control interfaces. In some embodiments, the display screen 275 is a projection display screen, which can further comprise a projection device and a projection screen.

[0072] In some embodiments, the user interface 265 is configured to receive infrared control signals from a control device 100 (e.g., an infrared remote controller).

[0073] In some embodiments, the detector 230 can comprise an image acquisition device 232, a sound acquisition device 231, and the like. The image acquisition device 232 can be a camera, a video camera, and the like, and is configured to acquire external environment scenes.

[0074] For example, the image acquisition device 232 can be a 3D camera, or other cameras configured to capture planar images. For example, the 3D camera can be based on a Structure Light technology, or a Time of Flight (TOF) technology. The 3D camera based on the Structure Light technology, or the 3D camera based on the TOF technology can acquire depth maps and brightness maps of objects.

[0075] The 3D camera based on the Structure Light technology can reduce the complexity of matching algorithms in binoculars, and can improve the robustness of matching algorithms. When the 3D camera based on the Structure Light technology is used for a long time, the camera needs to be replaced, and after the replacement, the replaced 3D camera needs to be calibrated.

[0076] Taking the acquisition of an object's depth map as an example, a Time-of-Flight (TOF) based 3D camera actively illuminates the object by modulating a light source (such as a laser or LED) and uses an image sensor sensitive to the laser wavelength to capture the reflected light, thereby measuring the target distance between the object in the image and the camera. The aforementioned image sensor can measure the time delay ΔT of the emitted laser signal returning to the camera after being reflected from the target. This delay is proportional to twice the distance (round trip) between the camera and the target object. Therefore, the distance (depth) between the object in the image captured by the TOF-based 3D camera and the camera can be estimated as shown in the following formula (1):

[0077]

[0078] Where c represents the speed of light, and d represents the distance between the object in the image taken by the TOF-based 3D camera and the camera.

[0079] like Figure 2 As shown, the controller 250 includes at least one of the following: random access memory 251 (RAM), read-only memory 252 (ROM), other processors 253 (e.g., graphics processing unit (GPU), central processing unit (CPU) 254), and a communication bus 256 (Bus). The communication bus connects the various components.

[0080] In some embodiments, the CPU processor 254 is configured to execute operating system and application instructions stored in memory, and to execute various applications, data, and content based on various interactive instructions received from external input, so as to ultimately display and play various audio and video content.

[0081] In some exemplary embodiments, the CPU processor 254 may include multiple processors. The multiple processors may include a main processor and one or more sub-processors. The main processor is used to perform some operations of the display device 200 in a pre-power-on mode, and / or to display a screen in normal mode. The one or more sub-processors are used for one operation in a standby mode or other state.

[0082] The memory 260 includes various software modules for driving the display device 200. For example, the various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, and a display control module.

[0083] Taking the control device 100 as an example, if the remote controller has a low battery or cannot be connected to the display device, the remote controller cannot send a control signal to the display device, and the user cannot control the display device through the remote controller. Therefore, the flexibility of the display device control method in the related art is poor.

[0084] In view of the above problems of the display device control method in the related art, the present application provides a method for operating a display device based on a video stream including a remote controller collected by an image collection device installed on the display device. Through the above method, the display device does not need to perform corresponding operations according to the control signal sent by the remote controller, so that the user can use the remote controller to control the display device whether the display device can communicate with the remote controller or not, improving the flexibility of the user to control the display device and further improving the user experience.

[0085] The technical solutions of the present application will be described in detail below in combination with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0086] Figure 3 A flowchart of a display device control method provided by the present application is shown in FIG. 1. As shown in FIG. 1, the method includes the following steps: Figure 3

[0087] S101, receiving a video stream.

[0088] Each frame of image of the video stream includes a control device for controlling a display device.

[0089] It should be understood that the present application does not limit the type of each frame of image of the video stream. For example, each frame of image of the video stream can include a depth map corresponding to each frame of image. Alternatively, in some embodiments, each frame of image of the video stream can also include a brightness map corresponding to each frame of image. For any one frame of image, the depth map of the frame of image refers to the distance between each pixel point of the frame of image and the image collection device. For any one frame of image, the brightness map of the frame of image refers to the brightness value of each pixel point of the frame of image.

[0090] ​It should be understood that the application does not limit the manner in which the display device receives the video stream. In some embodiments, the display device can capture the video stream based on an image capture device of the display device. In some embodiments, the display device can also capture the video stream through an external image capture device. In this implementation, the external image capture device can be connected to the display device through a USB interface or the like, for example, and the display device can receive the video stream captured by the image capture device.

[0091] As a possible implementation, before the step S101, the display device can also determine whether the function of the control device to control the display device is available. When it is determined that the function of the control device to control the display device is not available, the display device can perform the step S101 and subsequent steps. In some embodiments, when it is determined that the function of the control device to control the display device is not available, the display device can control the image capture device to capture the video stream, so as to save the computing resources of the display device. When it is determined that the function of the control device to control the display device is available, in some embodiments, the display device can communicate with the control device according to the display device control method of the related art, and control the display device.

[0092] The method of the display device to determine whether the function of the control device to control the display device is available can be different for different types of control devices. Taking the infrared protocol communication as an example of the communication manner between the control device and the display device, if the display device does not receive the control signal of the control device after a first preset time threshold after the display device is powered on, the display device can determine that the function of the control device to control the display device is not available. If the display device receives the control signal of the control device within the first preset time threshold after the display device is powered on, the display device can determine that the function of the control device to control the display device is available. The first preset time threshold can be stored in the display device by the user in advance, for example.

[0093] Taking the Bluetooth protocol communication as an example of the communication manner between the control device and the display device, in some embodiments, the display device can obtain the power of the control device through the Bluetooth protocol communication, and determine whether the function of the control device to control the display device is available according to the power of the control device. If the power of the control device is less than or equal to a preset power threshold, it means that the control device can not be used continuously, and the display device can determine that the function of the control device to control the display device is not available. If the power of the control device is greater than the preset power threshold, it means that the control device can be used continuously, and the display device can determine that the function of the control device to control the display device is available. The preset power threshold can be stored in the display device by the user in advance, for example.

[0094] In this example, in some embodiments, the display device can also determine whether the function of the control device controlling the display device is available by judging whether the Bluetooth connection between the display device and the control device is successful. If the Bluetooth between the display device and the control device is not connected, the display device can determine that the function of the control device controlling the display device is unavailable. If the Bluetooth between the display device and the control device is connected, the display device can determine that the function of the control device controlling the display device is available.

[0095] It should be understood that the display device may execute the above-mentioned steps of determining whether the control device's function of controlling the display device is available after power-on, and when it is determined that the control device's function of controlling the display device is unavailable, control the image acquisition device to acquire the video stream, and execute step S101 and subsequent steps. Alternatively, the display device may periodically execute the above-mentioned steps of determining whether the control device's function of controlling the display device is available during user use of the display device (e.g., watching video on the display device), and when it is determined that the control device's function of controlling the display device is unavailable, control the image acquisition device to acquire the video stream, and execute step S101 and subsequent steps.

[0096] In this implementation, as one possible approach, when it is determined that the function of the control device in controlling the display device is unavailable, the display device can also control the screen to display a prompt message including the virtual control device corresponding to the control device. This prompt message can be used to prompt the user to perform virtual control operations.

[0097] For example, Figure 4 This application provides a schematic diagram of an interface for displaying prompts from a virtual control device. For example... Figure 4 As shown, the interface displaying prompts for the virtual control device may include a virtual control device display area. The content displayed in this virtual control device display area may, for example, be the aforementioned video stream, or the bounding box area corresponding to the control device in each frame of the video stream. Alternatively, in some embodiments, the content displayed in the virtual control device display area may also be a simple line drawing of the control device.

[0098] like Figure 4 As shown, the interface displaying prompts for virtual control devices can also include a video playback area, so that users can be reminded to perform virtual control operations while watching videos is not affected, thus improving the user experience.

[0099] In this implementation, further, after the control device controls the display screen to display the prompt information for the preset time length, if the display device identifies that the user does not operate the control device, it indicates that the user may not be suitable for the control device to control the display device, and then the display device can further control the display screen to stop displaying the prompt information, so as to further reduce the visual impact on the user watching the display device, and further improve the user experience.

[0100] S102, according to the video stream, identifying whether the user operates the control device.

[0101] As a possible implementation, the display device can identify any frame image in the video stream according to a preset image detection algorithm to obtain the number of keys on the control device in the frame image. Then, according to the number of keys on the control device identified and a preset number of keys, it is determined whether the user operates the control device. The preset image detection algorithm may, for example, be a trained first neural network model. The trained first neural network model can identify the number of keys on the control device based on the input video stream. The preset number of keys may, for example, be pre-stored in the display device by the user.

[0102] If the number of keys on the control device identified is less than the preset number of keys, it indicates that the user may cover the fingers on the keys and operate the keys, and then the display device can determine that the user operates the control device. If the number of keys on the control device identified is equal to the preset number of keys, it indicates that the user does not cover the fingers on any key of the control device, and then the display device can determine that the user does not operate the control device.

[0103] As another possible implementation, the display device can also determine whether the user operates the control device based on the difference between at least two frames of images in the video stream. In some embodiments, the display device may, for example, determine whether the user operates the control device according to the difference in luminance values between two frames of images. For example, for any two adjacent frames of images in the video stream, if there are pixel points between the two frames of images with a difference in luminance values greater than or equal to a first preset luminance threshold, the display device can determine that the user operates the control device. If there are no pixel points between the two frames of images with a difference in luminance values greater than or equal to the first preset luminance threshold, the display device can determine that the user does not operate the control device.

[0104] If the display device identifies that the user operates the control device, step S102 can be performed to determine the specific key operated by the user.

[0105] If the display device identifies that the user does not operate the control device, in some embodiments, the display device can return to repeatedly performing the above steps S101 and S102. In this implementation, in some embodiments, the display device can further stop performing the steps S101 and S102 and subsequent steps when the number of times of continuously repeating the steps S101 and S102 is greater than or equal to a preset number threshold. Alternatively, the display device can further stop performing the steps S101 and S102 and subsequent steps when the time of continuously repeating the steps S101 and S102 is greater than or equal to a second preset time threshold. The preset number threshold or the second preset time threshold can be stored in the display device by the user in advance.

[0106] As a possible implementation, the display device can further output first prompt information for prompting the user to operate the control device at a specified angle before performing the step S101 (or simultaneously with performing the steps S101 and S102, or before performing the step S101 and simultaneously with performing the steps S101 and S102), so that the user can operate the control device according to the first prompt information. Through the above method, the angle at which the user holds the control device can be standardized, and it is ensured that each frame of image in the video stream captured by the image capturing device of the display device includes the control device, thereby improving the accuracy of the display device in determining whether the user operates the control device through the above video stream.

[0107] In some embodiments, the display device can output the first prompt information through a display screen, or a voice output device. Taking the case that the display device outputs the first prompt information through the display screen as an example, an exemplary interface of the display device outputting the first prompt information is provided. Figure 5 As shown in the figure, the interface outputting the first prompt information can include an example of the correct angle at which the user holds the control device, for guiding the user to adjust the angle at which the user holds the control device. Figure 5 As shown in the figure, in some embodiments, the interface outputting the first prompt information can further include a video playing area, so that the user can watch the video without being affected when the user is prompted to operate the control device at the correct angle, thereby further improving the user experience. Figure 5

[0108] S103, determining the identity of a target key of the control device according to the video stream.

[0109] The target key is a key pressed by the user when the user operates the control device.

[0110] ​In some embodiments, during the process of identifying whether a user is operating the control device based on the video stream, the display device may also number each identified button on the control device. In this implementation, the button number can serve as an identifier for the control device button. In some embodiments, the display device can determine the number of an unidentified button based on the identified button numbers and the numbers of all buttons in the control device. Then, the display device can use the number of the unidentified button as an identifier for the target button on the control device.

[0111] Alternatively, the labels of each button in the control device can be calibrated by the user offline. In this method, the labels of buttons in different locations on the control device are different. In this implementation, the display device can determine the location of unidentified buttons based on the identified button locations and the location of each button in the control device. Then, the display device can use the label corresponding to the determined location of the unidentified button as the label of the target button in the control device.

[0112] After determining the identifier of the target button of the control device based on the video stream, the display device can execute step S103 to perform the operation corresponding to the identifier of the target button on the display device.

[0113] Furthermore, taking the example that when it is determined that the function of the control device to control the display device is unavailable, the display device can also control the screen to display prompt information including the virtual control device corresponding to the control device. As a possible implementation, after determining the identifier of the target button, the display device can also control the color of the virtual button corresponding to the target button on the virtual control device displayed on the screen to change, so as to prompt the user that the target button of the control device has been pressed, thereby further improving the user experience.

[0114] In this implementation, for example, Figure 6 This application provides a schematic diagram of the interface for displaying a virtual control device. For example... Figure 6 As shown, assuming the user presses button A when operating the control device (i.e., the target button), the display device can control the color of button A in the virtual control device to change. Alternatively, the display device can also control the location of the target button to flash, indicating to the user that the target button of the control device has been pressed.

[0115] S104. Perform the operation corresponding to the identifier of the target button on the display device.

[0116] As a possible implementation, the display device can perform an operation corresponding to the target key identification on the display device according to the target key identification and a mapping relationship between the key identification and the operation on the display device. The mapping relationship between the key identification and the operation on the display device may, for example, be pre-stored in the display device by the user.

[0117] For example, the mapping relationship between the key identification and the operation on the display device may, for example, be as shown in Table 1:

[0118] Table 1

[0119] Serial number Identification of the key Operation performed on the display device 1 Identification 1 Operation 1 2 Identification 2 Operation 2 3 Identification 3 Operation 3 ... ... ...

[0120] Taking the mapping relationship shown in Table 1 as an example, assuming that the target key identification determined by the display device is identification 1, according to the mapping relationship shown in Table 1, the display device can perform operation 1 corresponding to the identification 1 on the display device.

[0121] In this implementation, after performing the operation corresponding to the target key identification on the display device, the display device can also update the mapping relationship between the key identification and the operation on the display device. In some embodiments, the display device can update the mapping relationship between the key identification and the operation on the display device after identifying that the user uses a new control device to control the display device.

[0122] For example, the display device can determine whether the user uses a new control device to control the display device according to the model of the control device. In some embodiments, the display device can obtain the model of the control device after the control device establishes a communication connection with the display device, or when the control device sends a control signal to the display device.

[0123] If the display device determines that the user uses a new control device to control the display device, for example, Figure 7 An interface provided by the display device to guide the user to input the mapping relationship between the key identification and the operation on the display device is provided. As shown in Figure 7 The display device can output second prompt information for prompting the user to face the front of the new control device towards the image acquisition device of the display device, so that the user can face the front of the new control device towards the image acquisition device of the display device according to the second prompt information. Then, the display device can acquire the image of the new control device and display a key identification input interface to receive the mapping relationship between the key identification and the operation on the display device input by the user.

[0124] In some embodiments, the display device can further send a mapping relationship update request including the model of the new control device to the server corresponding to the display device. Then, the server corresponding to the display device can obtain the mapping relationship between the identification of the key corresponding to the new control device and the operation performed on the display device according to the mapping relationship update request, and send it to the display device. Correspondingly, the display device can receive the mapping relationship between the identification of the key corresponding to the new control device and the operation performed on the display device sent by the server.

[0125] In some embodiments, taking the above example that the control device is provided with the two-dimensional code information corresponding to the control device on the control device, the display device can further obtain the model of the control device by scanning the two-dimensional code corresponding to the control device. Alternatively, the two-dimensional code information can further include the mapping relationship between the identification of the key and the operation performed on the display device, and the display device can update the mapping relationship between the identification of the key and the operation performed on the display device by scanning the two-dimensional code.

[0126] As another possible implementation, taking the example that the control device controls the display device when the function of the display device is available, and the control device sends different code words to the display device according to the operation of the user on different keys to control the display device, the display device can further perform the operation corresponding to the identification of the target key on the display device according to the identification of the target key, and the mapping relationship between the identification of the key and the code word corresponding to the key. In this implementation, the display device can first determine the code word corresponding to the target key according to the identification of the target key and the mapping relationship between the identification of the key and the code word corresponding to the key. Then, according to the code word of the target key and the mapping relationship between the code word of the key and the operation performed on the display device, the operation corresponding to the identification of the target key is determined. Then the display device can perform the operation corresponding to the identification of the target key on the display device.

[0127] In this embodiment, by image recognition on the image of the video stream including the control device, it can be determined whether the user operates the control device. When it is determined that the user operates the control device, the display device can determine the identification of the key (i.e. the target key) of the control device operated by the user according to each frame image in the video stream. Then, the display device can perform the operation corresponding to the identification of the target key on the display device according to the identification of the target key. Through the above method, the display device does not need to perform the corresponding operation according to the control signal sent by the control device, so that when the function of the display device controlled by the control device is unavailable, the user can also use the control device to control the display device, improving the flexibility of the user to control the display device, and further improving the user experience.

[0128] The following will describe in detail how the display device determines the identification of the target key of the control device according to the video stream. Figure 8A flowchart of a method for determining the identification of a target key of a control device is provided in the present application. As shown in Figure 8 As a possible implementation, step S102 can include the following steps:

[0129] S201, performing image recognition on each frame image of the video stream to obtain the position of each first key of the control device in each frame image.

[0130] The first key is a key of the control device that is not operated by the user.

[0131] In some embodiments, the user can perform image recognition on each frame image of the video stream by using the aforementioned preset image detection algorithm to obtain the position of each first key of the control device in each frame image. For example, taking each frame image of the video stream as a depth map, the aforementioned preset image detection algorithm can be a three-dimensional point cloud processing algorithm, such as Voxel Net (VoxNet), Multi View Convolutional Neural Networks (MVCNN), Point Net, etc.

[0132] S202, adding a key identification to each first key of the control device according to the position of each first key of the control device in the i-th frame image of the video stream.

[0133] The i-th frame image does not include at least one second key of the control device. The second key is a key of the control device that is operated by the user. i is an integer greater than or equal to 1.

[0134] In some embodiments, the display device can add the key identification to each first key of the control device in the order from top to bottom and from left to right on the control device. The key identification can be a number or a letter, which is not limited in the present application.

[0135] S203, tracking and positioning each first key in each image after the i-th frame image of the video stream according to the position of each first key of the control device in each image after the i-th frame image and the identification of each first key of the control device to obtain the identification of each first key and the identification of each second key of the control device in each image after the i-th frame image.

[0136] The following exemplary describes how the display device obtains the identity of each first key of the control device in each image after the i-th frame image in some embodiments. As one possible implementation, for any frame image after the i-th frame image, the display device can obtain the distance between any one first key in the frame image and each first key in the frame image after one frame, according to the first position of each first key of the control device in the frame image, and the second position of each first key of the control device in the frame image after one frame. Then, considering that even if there is shaking when the user holds the control device, the position of each key of the control device in the image changes, the distance moved by the same key between two adjacent frame images is small. Therefore, for any one first key in the frame image after one frame, the display device can take the identity of the first key in the frame image with the smallest distance from the first key as the identity of the first key in the frame image after one frame, to improve the accuracy of tracking and positioning the first key and the efficiency of tracking and positioning.

[0137] In some embodiments, for any one first key a in the frame image and any one first key b in the frame image after one frame, the distance between the first key a and the first key b can be the distance between the position a of the first key a in the camera coordinate system of the image acquisition device and the position b of the first key b in the camera coordinate system of the image acquisition device.

[0138] It should be understood that the display device can also track and locate the positions of different first keys after determining the identity of each first key of the control device in the i-th frame image by using other preset position tracking and positioning algorithms, and then obtain the identity of each first key of the control device in each frame image after the i-th frame image. The present application does not limit this. For example, the preset position tracking and positioning algorithm can be a trained second neural network model, which can track and locate the positions of the keys.

[0139] The following illustrates how the display device determines the identity of each second key of the control device in each frame image after the i-th frame image in some embodiments. As one possible implementation, the display device can determine the identity of each second key according to the identity of each first key of the control device in any frame image after the i-th frame image after determining the identity of each first key of the control device in the frame image. For example, after determining the identity of each first key in the frame image, the display device can compare the identity of each first key determined in the frame image with the identity of each first key in the i-th frame image, and determine the identity of each first key that does not appear in the frame image. Then, the display device can determine the identity of each second key in the frame image as the identity of each first key that does not appear in the frame image.

[0140] S204, determining the identity of the target key of the control device according to the identity of each first key and the identity of each second key.

[0141] The target key is one of the at least one second key.

[0142] As one possible implementation, taking the identity of each first key on the control device as an example, the number of each key is sequentially increased from top to bottom and from left to right, and considering that when the user holds the control device, in addition to the key (i.e., the target key) that the user wants to operate, the user's hand can also block a part of the key. The inventors have found through research that, in addition to the target key, the key blocked by the user is usually located below the target key. Therefore, the display device can determine the second key with the smallest number as the target key, and determine the identity of the second key as the identity of the target key of the control device.

[0143] As another possible implementation, considering that when the user's hand blocks the control device, the hand also has a greater impact on the brightness of the blocked position, therefore, the display device can also determine the identity of the target key of the control device according to the identity of each first key, the identity of each second key, and the brightness value of the image, to further improve the accuracy of the display device in determining the identity of the target key.

[0144] In some embodiments, the display device can determine at least one first key adjacent to the second key according to the identity of the second key, and then determine the position of the second key according to the at least one first key.

[0145] For example, after determining the at least one first key adjacent to the second key, the display device can obtain the position of the at least one first key in the frame image. In some embodiments, the display device can determine the region between the positions of the at least one first key as the position of the second key.

[0146] After determining the position of the second key, the display device can acquire the luminance value of the pixel point corresponding to the position of the second key, and an initial luminance value. The initial luminance value is the luminance value of the pixel point corresponding to the position of the first key with the same identifier as the second key in the video stream. In some embodiments, the display device can acquire the luminance value of each pixel point in each frame image of the video stream through the image acquisition device with the luminance recognition function.

[0147] Then, the display device can determine the identifier of the target key of the control device according to the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value, and the identifier of the first key. In some embodiments, the display device can determine whether the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to a second preset luminance threshold. Then, when the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to the second preset luminance threshold, the identifier of the target key of the control device is determined according to the identifier of each first key, so as to improve the accuracy of the display device in determining the identifier of the target key.

[0148] If the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is less than the second preset luminance threshold, in some embodiments, the display device can determine that the user does not operate the control device.

[0149] In the embodiment, the position of each first key of the control device in each frame image of the video stream can be acquired through image recognition. The first key identifier is added to each first key of the control device according to the position of each first key of the control device, which considers the influence of the position distribution of each first key on the control device on the determination of the target key, and improves the accuracy of the determination of the target key. Then, the display device can track and locate the first key according to the identifier of the first key between two frame images of the video stream, so as to obtain the identifier of each first key and the identifier of each second key of the control device in each frame image after the i-th frame image. According to the identifier of the first key and the identifier of the second key, the display device determines the identifier of the target key from the second key. Through the above method, the identifier of each first key and the identifier of each second key of the control device in each frame image after the i-th frame image are obtained, which considers the change of the position of the second key in different frame images, and further improves the accuracy of the determination of the target key.

[0150] Taking the image acquisition device as a 3D camera, the display device as a smart television, and the control device as a remote controller as an example, the smart television can acquire the depth map and the luminance map corresponding to each frame image of the video stream from the 3D camera, Figure 9Another flowchart of a display device control method is provided in the present application. As shown in Figure 9 the method comprises the following steps:

[0151] S301, powering on the smart TV.

[0152] The power for powering on the smart TV can be alternating current.

[0153] S302, detecting whether the function of the remote controller controlling the smart TV is available. If available, the smart TV can perform step S303. If not available, step S304 and subsequent steps are performed.

[0154] In a specific implementation, the smart TV can detect whether the function of the remote controller controlling the smart TV is available in the manner described in the foregoing embodiments, which will not be described here.

[0155] S303, receiving the control signal sent by the remote controller.

[0156] S304, controlling the 3D camera to start capturing a video stream.

[0157] For example, the 3D camera can be an RGBD (Red Green Blue Deep) camera.

[0158] S305, creating a virtual remote wireless interaction thread, and controlling the display screen to display prompt information including a virtual remote controller corresponding to the remote controller.

[0159] S306, determining whether the user operates the remote controller.

[0160] If it is determined that the user operates the remote controller, steps S307-S313 are performed. If it is determined that the user does not operate the remote controller, step S314 is performed.

[0161] S307, performing image recognition on each frame of image of the video stream to obtain the position of each first key of the remote controller in each frame of image.

[0162] S308, adding a key identifier to each first key of the remote controller according to the position of each first key of the remote controller in the i-th frame of image of the video stream.

[0163] S309, obtaining the distance between any one first key in the j-th frame of image and each first key in the j+1-th frame of image according to the first position of each first key of the remote controller in the j-th frame of image and the second position of each first key of the remote controller in the j+1-th frame of image. The j-th frame of image is an image after the i-th frame of image in the video stream; j is an integer greater than or equal to i+1 and less than or equal to N, where N is the total number of images included in the video stream.

[0164] S310, for any one first button in the j+1th frame image, the identification of the first button with the smallest distance from the first button in the jth frame image is taken as the identification of the first button in the j+1th frame image.

[0165] S311, according to the identification of each first button, the identification of each second button, and the brightness value of the image, the identification of the target button of the remote controller is determined.

[0166] S312, the color of the target button corresponding to the virtual remote controller displayed on the display screen is changed to prompt the user that the target button is pressed.

[0167] S313, the operation corresponding to the identification of the target button is performed on the display device.

[0168] S314, the floating window used to display the prompt information including the virtual remote controller corresponding to the remote controller is hidden.

[0169] In some embodiments, the display device can also perform the step S314 when no user operation of the remote controller is detected within a preset time length.

[0170] Figure 10 A structural schematic diagram of a display device control apparatus provided in the present application is shown in FIG. 4. As shown in the figure, the apparatus 400 includes a receiving module 401, an identifying module 402, a processing module 403, and an executing module 404. Among them, Figure 10

[0171] The receiving module 401 is configured to receive a video stream. Each frame image of the video stream includes a remote controller used to control the display device.

[0172] The identifying module 402 is configured to identify whether a user operates the control device according to the video stream.

[0173] The processing module 403 is configured to determine the identification of a target button of the control device according to the video stream when it is identified that the user operates the control device. The target button is the button pressed by the user when operating the control device.

[0174] The executing module 404 is configured to perform an operation corresponding to the identification of the target button on the display device.

[0175] ​In some embodiments, the processing module 403 is specifically configured to: perform image recognition on each frame image of the video stream to obtain a position of each first key of the control device in each frame image; add a key identifier to each first key of the control device in the i-th frame image according to the position of each first key of the control device in the i-th frame image; track and locate each first key in each image after the i-th frame image according to the position of each first key of the control device in each image after the i-th frame image and the identifier of each first key and the identifier of at least one second key in each image after the i-th frame image; and determine the identifier of the target key of the control device according to the identifier of each first key and the identifier of the at least one second key. In the i-th frame image, there is no second key of the control device; and the target key is one of the at least one second key.

[0176] In some embodiments, the processing module 403 is specifically configured to: for any frame image after the i-th frame image, obtain a distance between any one first key in the frame image and each first key in a frame image after the frame image according to the first position of each first key of the control device in the frame image and the second position of each first key of the control device in the frame image after the frame image; and for any one first key in the frame image after the frame image, take the identifier of the first key with the minimum distance to the first key in the frame image as the identifier of the first key in the frame image after the frame image.

[0177] In some embodiments, the processing module 403 is specifically configured to: determine at least one first key adjacent to the second key according to the identifier of the second key; determine the position of the second key according to the at least one first key; obtain a luminance value of a pixel point corresponding to the position of the second key and an initial luminance value; and determine the identifier of the target key of the control device according to the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value and the identifier of the first key. The initial luminance value is a luminance value of a pixel point corresponding to the position of the first key with the same identifier as the second key in the video stream.

[0178] In some embodiments, the processing module 403 is specifically configured to determine whether the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to a second preset luminance threshold value; and when the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to the second preset luminance threshold value, determine the identification of the target key of the control device according to the identifications of the first keys.

[0179] In some embodiments, the receiving module 401 is further configured to determine whether the function of the display device controlled by the control device is available before the video stream collected by the image collection device is acquired; and when the function of the display device controlled by the control device is not available, receive the video stream.

[0180] In some embodiments, the apparatus 400 can further include a display module 405 configured to control the display screen to display prompt information when the function of the display device controlled by the control device is not available. The prompt information includes a virtual control device corresponding to the control device, and the prompt information is used to prompt the user to perform virtual remote control operation.

[0181] In some embodiments, the display module 405 is further configured to, after a preset time length of displaying the prompt information on the display screen, control the display screen to stop displaying the prompt information when it is identified that the user does not operate the control device.

[0182] In some embodiments, the execution module 404 is specifically configured to perform an operation corresponding to the identification of the target key on the display device according to the identification of the target key and a mapping relationship between the identification of the key and the operation performed on the display device. In this implementation, the apparatus 400 can further include an updating module 406 configured to update the mapping relationship between the identification of the key and the operation performed on the display device.

[0183] The display device control apparatus provided in the present application is used to execute the display device control method embodiments, and has similar implementation principles and technical effects, which will not be described herein.

[0184] The present application further provides a computer readable storage medium, which can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, and specifically, the computer readable storage medium stores program instructions, which are used for the method in the above embodiments.

[0185] The application also provides a program product including execution instructions stored in a readable storage medium. At least one processor of a display device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to enable the display device to implement the display device control method provided by various embodiments described above.

[0186] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0187] For the convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A display device, characterized by comprising: The application relates to a display device and a control method thereof. The display device comprises: a display screen for displaying images; a processor connected to the display screen, the processor being configured to: determine whether a function of the display device controlled by a control device is available; if the function of the display device controlled by the control device is not available, receive a video stream, each frame of the video stream comprising a control device for controlling the display device; identify whether a user operates the control device according to the video stream; if it is identified that the user operates the control device, perform image recognition on each frame of the video stream to obtain the positions of each first key of the control device in each frame; add key identifiers to the first keys of the control device in the i-th frame according to the positions of the first keys of the control device in the i-th frame; a second key of the control device does not exist in the i-th frame; track and locate the first keys in each frame after the i-th frame according to the positions of the first keys of the control device in each frame after the i-th frame, the identifiers of the first keys, and the identifiers of the first keys in the i-th frame to obtain the identifiers of the first keys and the identifiers of at least one second key in each frame after the i-th frame; the identifier of the second key is the identifier of a first key that does not appear in each frame after the i-th frame compared with the identifiers of the first keys in the i-th frame; wherein for any first key in the j+1-th frame, the identifier of the first key in the j+1-th frame is the identifier of a first key that is closest to the first key in the j-th frame; wherein the j-th frame is an image after the i-th frame in the video stream; j is an integer greater than or equal to i+1 and less than or equal to N, and N is the total number of images included in the video stream; determine at least one first key adjacent to the second key according to the identifier of the second key; determine the position of the second key according to the at least one first key; obtain the luminance value of a pixel point corresponding to the position of the second key and an initial luminance value; the initial luminance value is the luminance value of a pixel point corresponding to the position of a first key with the same identifier as the second key in the video stream; determine the identifier of a target key of the control device according to the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value and the identifier of the first key; the target key is one of the at least one second key; the target key is a key pressed by the user when operating the control device; 2. The display device of claim 1, wherein, perform an operation corresponding to the identifier of the target key on the display device. The processor is configured to:

3. The display device of claim 1, wherein, for any frame after the i-th frame, obtain the distance between any first key in the j-th frame and each first key in the j+1-th frame according to the first positions of the first keys of the control device in the j-th frame and the second positions of the first keys of the control device in the j+1-th frame. The processor is configured to: determine whether the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to a second preset luminance threshold value; if the absolute value of the difference between the luminance value of the pixel point corresponding to the position of the second key and the initial luminance value is greater than or equal to the second preset luminance threshold value, determine the identification of the target key of the control device according to the identifications of the first keys.

4. The display device of claim 1, wherein, The processor is configured to: if the control device cannot control the function of the display device, control the display screen to display prompt information; the prompt information includes a virtual control device corresponding to the control device, and the prompt information is used to prompt the user to perform virtual remote control operation.

5. The display device of claim 4, wherein, The processor is configured to: after a preset time length during which the display screen displays the prompt information, if it is identified that the user does not operate the control device, control the display screen to stop displaying the prompt information.

6. The display device of claim 1, wherein, The processor is configured to: perform an operation corresponding to the identification of the target key on the display device according to the identification of the target key and a mapping relationship between the identification of the key and the operation performed on the display device; The processor is configured to: update the mapping relationship between the identification of the key and the operation performed on the display device.

7. A display device control method characterized by comprising: The method is applied to the display device as claimed in any one of claims 1-6, and the method comprises: receiving a video stream, each frame image of the video stream including a control device used to control the display device; identifying whether a user operates the control device according to the video stream; if it is identified that the user operates the control device, determining the identification of a target key of the control device according to the video stream; the target key is a key pressed by the user when operating the control device; performing an operation corresponding to the identification of the target key on the display device.

Images