Display device and control recognition method

Abstract

The present application provides a display device and a control identification method. The display device includes a display and a controller. The controller is configured to obtain the control keyword in the control trigger instruction when receiving a control trigger instruction input by the user, and control the display to display a candidate control interface. The candidate control interface includes candidate controls whose control texts include control keywords, so that the user can select a target candidate control. Based on the user's operation on the candidate control interface, the target candidate control selected by the user can be determined and triggered. The present application can identify the target control required by the user from multiple similar controls in the user interface, thereby responding to the user's instructions and improving the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a control identification method. Background Art

[0002] Display devices refer to terminal devices capable of outputting specific display images, such as smart TVs, mobile terminals, smart advertising screens, and projectors. With the rapid development of display devices, their functionality and performance are becoming increasingly rich and powerful. They can achieve two-way human-computer interaction, integrating audio, video, entertainment, data, and other functions to meet the diverse and personalized needs of users. Intelligent voice interaction has become a key feature of display devices.

[0003] For display devices with intelligent voice interaction capabilities, users can input control commands through voice to enable certain functions of the display device. For example, a user can say "I want to see X," and the display device will recognize the control name contained in the user's input command and further trigger the corresponding target control to meet the user's needs.

[0004] However, the control name in the user's input command may not be complete. For example, if the user inputs "Open my" by voice, the display device may recognize the control name as "my", which is not a complete control name. The display device may have multiple controls with "my" in the control name. In this case, the display device may not be able to confirm the target control that the user wants to open and thus be unable to respond to the user's command, resulting in a poor user experience. Summary of the Invention

[0005] The present invention provides a display device and a control identification method to solve the problem in the related art that a display device may not be able to identify a target control that a user wants to open, resulting in a poor user experience.

[0006] In a first aspect, the present application provides a display device, comprising a display and a controller. The controller is configured to perform the following steps:

[0007] In response to a control trigger instruction input by a user, a control keyword in the control trigger instruction is obtained; the display is controlled to display a candidate control interface so that the user can select a candidate control according to the candidate control interface; the candidate control interface includes at least one candidate control, and the control text of the candidate control includes the control keyword; based on the user's operation on the candidate control interface, the target candidate control selected by the user is determined; and the target candidate control is triggered.

[0008] In some implementations, the controller is further configured to: before executing the step of obtaining the control keyword in the control trigger instruction, control the display to display a user interface so that the user can input the control trigger instruction.

[0009] In some implementations, before executing the step of controlling the display to display the candidate control interface, all candidate controls in the user interface are acquired according to the control keywords; and a candidate control interface is generated according to all candidate controls.

[0010] In some implementations, the controller is further configured to: in executing the step of acquiring all candidate controls in the user interface according to the control keyword,

[0011] Acquire control information of each control in the user interface, wherein the control information includes control text and control position; determine a control whose control text contains the control keyword as a candidate control; and acquire all candidate controls in the user interface.

[0012] In some implementations, the controller is further configured to: after executing the step of controlling the display to display the candidate control interface, randomly select a candidate control from all candidate controls as the focus control, and highlight it in the candidate control interface to allow the user to confirm whether to trigger the focus control; if a confirmation instruction input by the user is detected, trigger the focus control; if no confirmation instruction input by the user is detected, execute the step of determining the target candidate control selected by the user.

[0013] In some implementations, the controller is further configured to: in executing the step of determining the target candidate control selected by the user, in response to an instruction sent by the control device when the user switches the focus control in the candidate control interface, obtain the switched focus control and determine it as the target candidate control.

[0014] In some implementations, the controller is further configured to: in executing the step of determining the target candidate control selected by the user, send the candidate control interface to the terminal device so that the terminal device displays the candidate control interface; in response to an instruction sent by the mobile terminal when the user touches a candidate control in the candidate control interface, determine the candidate control touched by the user as the target candidate control.

[0015] In some implementations, the display device further includes an image collector configured to collect user images.

[0016] In some implementations, the controller is further configured to: in executing the step of determining the target candidate control selected by the user,

[0017] Obtain the offset angle of each candidate control in the candidate control interface, where the offset angle is used to represent the position of the candidate control relative to the candidate control interface; obtain the user gesture trajectory of the user in the candidate control interface based on the user image captured by the image collector; determine the target candidate control selected by the user based on the offset angle of each candidate control and the user gesture trajectory.

[0018] In some implementations, the controller is further configured to: in executing the step of obtaining the offset angle of each candidate control in the candidate control interface,

[0019] Set the interface focus of the candidate control interface; determine the marking points of each candidate control, where each candidate control has two marking points; obtain the connecting lines between the two marking points of each candidate control and the interface focus, and determine the angle between the two connecting lines as the offset angle of each candidate control.

[0020] In some implementations, the controller is further configured to: in executing the step of acquiring a user gesture trajectory of the user in the candidate control interface according to the user image acquired by the image acquirer,

[0021] The user image is mapped to the candidate control interface, and the initial user gesture focus is determined as the interface focus of the candidate control interface; the user gesture focus after obtaining a preset period is located at a first position in the candidate control interface; and the trajectory between the interface focus and the first position is determined as the user gesture trajectory.

[0022] In some implementations, the controller is further configured to: in the step of determining the target candidate control selected by the user according to the offset angle of each candidate control and the user gesture trajectory,

[0023] Obtain a candidate control whose offset angle includes the user gesture trajectory and determine it as the first candidate control; obtain an associated candidate control of the first candidate control, and the overlapping part of the offset angle of the first candidate control and the associated candidate control is greater than a preset threshold; determine the trajectory distance of the user gesture trajectory, and screen the first candidate control and the associated candidate control according to the trajectory distance to obtain the target candidate control selected by the user.

[0024] In some implementations, the controller is further configured to: after executing the step of determining the target candidate control selected by the user,

[0025] The target candidate control is used as the focus control and is highlighted in the candidate control interface to allow the user to confirm whether to trigger the focus control; if a confirmation instruction input by the user is detected, the step of triggering the target candidate control is executed; if a confirmation instruction input by the user is not detected, the user gesture focus after the next preset cycle is obtained to be located at the second position in the candidate control interface; the trajectory between the interface focus and the second position is determined as the updated user gesture trajectory; based on the updated user gesture trajectory, the step of determining the target candidate control selected by the user is executed.

[0026] In a second aspect, the present application provides a control identification method, applied to a display device, comprising:

[0027] In response to a control trigger instruction input by a user, a control keyword in the control trigger instruction is obtained; the display is controlled to display a candidate control interface so that the user can select a candidate control according to the candidate control interface; the candidate control interface includes at least one candidate control, and the control text of the candidate control includes the control keyword; based on the user's operation on the candidate control interface, the target candidate control selected by the user is determined; and the target candidate control is triggered.

[0028] It can be seen from the above technical solution that the present application provides a display device and a control identification method. When a control trigger instruction input by the user is received, the control keyword in the control trigger instruction can be obtained, and the display can be controlled to display the candidate control interface. Among them, the candidate control interface has candidate controls whose control text includes control keywords, so that the user can select the target candidate control. Then, based on the user's operation on the candidate control interface, the target candidate control selected by the user can be determined and triggered. The present application can identify the target control required by the user from multiple similar controls in the user interface, thereby responding to the user's instructions and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 shows a usage scenario of a display device according to some embodiments;

[0031] Figure 2 shows a hardware configuration block diagram of the control device 100 according to some embodiments;

[0032] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0033] Figure 4 shows a software configuration diagram in the display device 200 according to some embodiments;

[0034] Figure 5 A schematic diagram of a user interface in some embodiments is shown;

[0035] Figure 6 A schematic diagram showing confirmation information of a control recognition mode displayed in a display in a feasible embodiment is shown;

[0036] Figure 7 A flowchart illustrating the interaction between components of a display device in some embodiments is shown;

[0037] Figure 8 A schematic diagram showing an application list in some embodiments is shown;

[0038] Figure 9 A schematic diagram showing a prompt interface displayed on a display in some embodiments is shown;

[0039] Figure 10a A schematic diagram of a candidate control interface in some embodiments is shown;

[0040] Figure 10b A schematic diagram of a candidate control interface in some embodiments is shown;

[0041] Figure 11 shows the offset angles of all candidate controls in the candidate control interface in some embodiments;

[0042] Figure 12 A schematic diagram showing user gesture trajectories in a candidate control interface in some embodiments is shown;

[0043] Figure 13 A schematic diagram of a candidate control interface in some embodiments is shown;

[0044] Figure 14 A schematic diagram of determining a target candidate control in some embodiments is shown;

[0045] Figure 15 A flow chart of an embodiment of a control identification method is shown. DETAILED DESCRIPTION

[0046] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0047] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0048] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0049] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0050] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0051] Figure 1 Schematic diagram of a usage scenario of a display device according to an embodiment. Figure 1 As shown, the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0052] In some embodiments, the control device 100 may be a remote controller, and communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or at least one other short-range communication method, to control the display device 200 wirelessly or wiredly. The user may control the display device 200 by inputting user commands through at least one of buttons on the remote controller, voice input, and control panel input.

[0053] In some embodiments, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a laptop computer, an AR / VR device, etc.

[0054] In some embodiments, the smart device 300 can also be used to control the display device 200. For example, the display device 200 can be controlled using a camera application running on the smart device.

[0055] In some embodiments, data communication may also be performed between the smart device 300 and the display device.

[0056] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive the user's voice command control through a module for obtaining voice commands configured inside the display device 200, or it can receive the user's voice command control through a voice control device set outside the display device 200.

[0057] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0058] In some embodiments, a software step executed by a step execution entity can be transferred to another step execution entity in data communication with the step execution entity as needed. For example, a software step executed by a server can be transferred to a display device in data communication with the server as needed, and vice versa.

[0059] Figure 2 Schematically shows a block diagram of the configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, acting as an interaction medium between the user and the display device 200.

[0060] In some embodiments, the communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, NFC, or an alternative module.

[0061] In some embodiments, the user input / output interface 140 includes at least one of a microphone, a touch pad, a sensor, a button, or alternative modules.

[0062] Figure 3 FIG. 2 is a block diagram showing a hardware configuration of the display device 200 according to an exemplary embodiment.

[0063] In some embodiments, the display apparatus 200 includes at least one of a tuner 210 , a communicator 220 , a detector 230 , an external device interface 240 , a controller 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a user interface.

[0064] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0065] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0066] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0067] In some embodiments, the tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0068] In some embodiments, the communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. The display device 200 can use the communicator 220 to send and receive control signals and data signals with the control device 100 or the server 400.

[0069] In some embodiments, detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or a sound collector, such as a microphone, for receiving external sounds.

[0070] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0071] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0072] In some embodiments, controller 250 controls the operation of the display device and responds to user operations via various software control programs stored in memory. Controller 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object for display on display 260, controller 250 may perform operations related to the object selected by the user command.

[0073] In some embodiments, the object may be any selectable object, such as a hyperlink, an icon, or other operable control. Operations associated with the selected object include: displaying a page, document, or image connected to the hyperlink, or executing a program corresponding to the icon.

[0074] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0075] The CPU processor is used to execute operating system and camera application instructions stored in memory, as well as various interactive instructions received from external input, to execute various camera applications, data, and content, ultimately displaying and playing various audio and video content. The CPU processor can include multiple processors, such as a main processor and one or more sub-processors.

[0076] In some embodiments, a graphics processor is configured to generate various graphical objects, such as at least one of icons, operation menus, and graphics displayed in response to user input commands. The graphics processor includes an arithmetic unit that performs operations based on various user input interactive commands and displays various objects according to display attributes; and a renderer that renders the various objects generated by the arithmetic unit. The rendered objects are then displayed on a display.

[0077] In some embodiments, the video processor is used to receive an external video signal and perform at least one of the following video processing operations, such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis, according to the standard codec protocol of the input signal, to obtain a signal that can be directly displayed or played on the display device 200.

[0078] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, and a display formatting module. The demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to user input or itself with the scaled video image to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to receive the frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting an RGB data signal.

[0079] In some embodiments, the audio processor is used to receive an external audio signal, perform decompression and decoding according to the standard codec protocol of the input signal, and perform at least one of noise reduction, digital-to-analog conversion, and amplification processing to obtain a sound signal that can be played in a speaker.

[0080] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface may recognize the sound or gesture through a sensor to receive the user input command.

[0081] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between the camera application or operating system and the user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on the display screen of an electronic device, where the control can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0082] In some embodiments, the user interface 280 is an interface for receiving control input (eg, a physical button on the display device body, or other interfaces).

[0083] In some embodiments, the system of the display device may include a kernel, a command parser (shell), a file system, and a camera application. The kernel, shell, and file system together constitute the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates kernel space, abstracts hardware, initializes hardware parameters, runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and the user camera application are loaded. After startup, the camera application is compiled into machine code to form a process.

[0084] like Figure 4 As shown in the figure, the display device system can include a kernel, a command parser (shell), a file system, and application programs. The kernel, shell, and file system together form the basic operating system structure, allowing users to manage files, run programs, and use the system. After power is applied, the kernel boots up, activates kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, a scheduler, signals, and inter-process communication (IPC). After the kernel boots up, the shell and user applications are loaded. After booting up, the application is compiled into machine code, forming a process.

[0085] like Figure 4 As shown, the system of the display device is divided into three layers, namely the application layer, the middleware layer and the hardware layer from top to bottom.

[0086] The application layer mainly includes commonly used applications on the TV and the application framework. Common applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0087] The Application Framework is a complete programming model that provides all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces for these functions (toolbars, status bars, menus, and dialog boxes).

[0088] Native apps can support online or offline, message push or local resource access.

[0089] The middleware layer includes various TV protocols, multimedia protocols, and system components. Middleware uses the basic services (functions) provided by system software to connect various parts of the application system or different applications on the network, enabling resource and function sharing.

[0090] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WiFi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0091] Users can input control commands through voice to enable some functions of the display device. However, the control name in the command input by the user may not be complete. For example, if the user inputs "open mine" by voice, the control name that the display device can recognize is "mine", which is not a complete control name. The display device may have multiple controls with "mine" in the control name. In this case, the display device may not be able to confirm the target control that the user needs to open, and thus cannot respond to the user's command, resulting in a poor user experience.

[0092] The present application provides a display device including a display and a controller. The display is configured to display a user interface. The user interface may include a specific target image, such as various media assets obtained from a network signal source, including videos, images, and other content. The user interface may also be some UI interface of the display device.

[0093] In some embodiments, the display device has a voice control function, allowing the user to control the display device through voice input. Specifically, the display device also includes a sound collector, which can be a microphone, for receiving user-input voice commands, such as control triggering commands. The user can use the microphone to send various commands to the display device through voice input.

[0094] In some embodiments, when the user controls the display device to turn on, the controller can control the display to display the user interface. Figure 5 Schematic diagrams of user interfaces in some embodiments are shown. The user interface includes a first navigation bar 500, a second navigation bar 510, a function bar 520 and a content display area 530. The function bar 520 includes multiple function controls such as "watch history", "my favorites" and "my apps". Among them, the content displayed in the content display area 530 will change with the changes in the selected controls in the first navigation bar 500 and the second navigation bar 510. When on the application panel page, the user can trigger entry into the corresponding application panel by clicking on the "my apps" control to input a display instruction for the application panel page. It should be noted that the user can also input a selection operation for the function control in other ways to trigger entry into the application panel. For example, use voice control function or search function to control entry into the application panel page.

[0095] When a user inputs a command to a display device using voice control, for example, the user inputs "open my", since there may be multiple controls in the user interface whose names all contain "my", the user's expression is not clear. Figure 5 In the user interface shown in , the control names of the three controls "My", "My Favorites" and "My Apps" all contain "My", so the display device cannot determine the target control that the user needs to trigger and needs to further identify the target control.

[0096] In some embodiments, the display device has a control recognition function that can identify the target control that the user needs to trigger based on the instruction input by the user. The display device can be set to a control recognition mode. In the control recognition mode, the display device can automatically recognize all controls in the user interface to obtain the target control that the user needs to trigger.

[0097] In some embodiments, a user can send a control recognition mode instruction to the display device by operating a designated button on the remote control. In actual application, the correspondence between the control recognition mode instruction and the remote control button is pre-bound. For example, a control recognition mode button is set on the remote control. When the user touches the button, the remote control sends the control recognition mode instruction to the controller, and the controller controls the display device to enter the control recognition mode. When the user touches the button again, the controller can control the display device to exit the control recognition mode.

[0098] In some embodiments, the correspondence between the control recognition mode instruction and multiple remote control buttons can also be pre-bound. When the user touches multiple buttons bound to the control recognition mode instruction, the remote control issues the control recognition mode instruction. In a feasible embodiment, the buttons bound to the control recognition mode instruction are the direction keys (left, down, left, down) in sequence, that is, when the user touches the buttons (left, down, left, down) continuously within a preset time, the remote control sends the control recognition mode instruction to the controller. The above binding method can avoid the control recognition mode instruction being issued due to user's misoperation. The embodiment of the present application only exemplifies the binding relationship between several control recognition mode instructions and buttons. In actual application, the binding relationship between the control recognition mode instruction and the button can be set according to the user's habits, and no excessive restrictions are made here.

[0099] In some embodiments, the user can use the sound collector of the display device, such as a microphone, to send a control recognition mode instruction to the display device through voice input to control the display device to enter the control recognition mode. The display device may be provided with an intelligent voice system, which can recognize the user's voice to extract the content of the instruction input by the user. The user can input a preset wake-up word through the microphone to start the intelligent voice system, so that the controller can respond to the instruction input by the user. And input the control recognition mode instruction within a certain time to make the display device enter the control recognition mode. For example, the user can enter "classmate XX" to start the intelligent voice system. Then enter "enter control recognition mode" to send the control recognition mode instruction to the display device.

[0100] In some embodiments, the user can also send a control recognition mode instruction to the display device through a preset gesture. The display device can detect the user's behavior through an image collector, such as a camera. When the user makes a preset gesture, it can be considered that the user has sent a control recognition mode instruction to the display device. For example, it can be set as follows: when it is detected that the user makes a V-shape, it is determined that the user has input a control recognition mode instruction to the display device. The user can also send a control recognition mode instruction to the display device through a preset action. For example, it can be set as follows: when it is detected that the user lifts his left foot and right hand at the same time, it is determined that the user has input a control recognition mode instruction to the display device.

[0101] In some embodiments, when a user controls a display device using a smart device, such as a mobile phone, a control recognition mode instruction may be sent to the display device. In actual use, a control may be set up on the mobile phone, which allows the user to select whether to enter control recognition mode. This control then sends a control recognition mode instruction to the controller, which then controls the display device to enter control recognition mode.

[0102] In some embodiments, when a user uses a mobile phone to control a display device, they can issue a continuous click command to the phone. A continuous click command means that the number of times the user clicks the same area of the mobile phone's touchscreen exceeds a preset threshold within a preset period. For example, if a user clicks a certain area of the mobile phone's touchscreen three times within 1 second, it is considered a continuous click command. After receiving the continuous click command, the mobile phone can send a control recognition mode command to the display device, causing the controller to control the display device to enter control recognition mode.

[0103] In some embodiments, when a user uses a mobile phone to control a display device, it can also be set as follows: when it is detected that the user's touch pressure value on a certain area of the mobile phone touch screen exceeds a preset pressure threshold, the mobile phone can send a control recognition mode instruction to the display device.

[0104] You can also set a control recognition mode option in the UI interface of the display device. When the user clicks this option, you can control the display device to enter or exit the control recognition mode.

[0105] In some embodiments, to prevent users from accidentally triggering the control recognition mode, when the controller receives the control recognition mode instruction, it can control the display to display the control recognition mode confirmation information, so that the user can perform a second confirmation whether to control the display device to enter the control recognition mode. Figure 6 A schematic diagram showing control recognition mode confirmation information displayed in a display in a feasible embodiment is shown.

[0106] When the display device enters the control recognition mode, it can automatically detect all controls on the user interface according to the instructions input by the user, thereby identifying the target control that the user needs to trigger.

[0107] Figure 7 A flowchart of the interactions among components of a display device in some embodiments is shown.

[0108] In some embodiments, when the display device enters the control recognition mode, the user can view the user interface and send a control trigger instruction to the display device to trigger the corresponding target control.

[0109] The control trigger instruction can be an instruction containing the full name of a control, for example, the user can send a voice message "I want to watch a movie". Figure 5 As shown, "Movie" is the name of the control. When the display device receives the instruction, it can trigger the "Movie" control and display the interface corresponding to the "Movie" control to the user. For another example, the user can send "Open my application". When the display device receives the instruction, it can trigger the "My Application" control and display the interface corresponding to the "My Application" control to the user. The interface may include all applications installed in the display device, such as Figure 8 As shown, the interface corresponding to the "My Applications" control may include an application list, and the application list includes all applications installed on the display device.

[0110] A control trigger instruction can also be an instruction that contains a partial name of a control. For example, the user can send a voice message saying "Open mine." Since the control name is not complete, the display device needs to filter and match all controls in the user interface to confirm the control that contains the partial name. If there is only one control in the confirmation result, it can be undoubtedly determined that the control is the target control that the user needs to trigger. If the confirmation result contains multiple controls, the display device needs to further determine the specific target control.

[0111] Therefore, when receiving a control trigger instruction input by the user, the display device needs to first determine the control keyword related to the control contained in the control trigger instruction. The control keyword can be the full name of a control or a partial name of a control.

[0112] In some embodiments, when the display device receives a control trigger instruction input by the user, the controller can send the received voice data to the voice recognition service, which is converted into text information to obtain a control trigger text. The voice recognition service is a web service that can be deployed on an electronic device and can include a voice recognition module and a semantic analysis module. Among them, the voice recognition service is used to recognize audio as text, and the semantic service is used to perform semantic analysis on the text. For example, the voice recognition module can recognize the control trigger instruction input by the user as a control trigger text. The semantic analysis module then performs lexical, syntactic and semantic analysis on the control trigger text to understand the user's intention and obtain control keywords.

[0113] In some embodiments, the display device may further include a third-party voice recognition interface. Upon receiving a control trigger instruction input by a user, the controller may send voice data to the third-party voice recognition interface, which may then use a third-party voice recognition device to recognize the user's control trigger instruction as control trigger text. After obtaining the control trigger text, the controller may analyze the control trigger text to obtain control keywords from the control trigger text.

[0114] In some embodiments, the controller may identify the control trigger text to obtain some keywords in the control trigger text.

[0115] The controller can first perform word segmentation on the control trigger text to obtain a word segmentation result including multiple words. For example, for the control trigger instruction "I want to watch a movie", after word segmentation, the word segmentation result can be obtained as the three words "I want, watch, movie". The specific word segmentation method can be referred to in related technologies and will not be described in detail in this application.

[0116] After the control trigger text is segmented, the controller can parse the segmentation results and extract keywords from the segmentation results. Specifically, nouns can be extracted from the segmentation results. For example, for the segmentation result "I want to watch movies", "watch" can be parsed as the control instruction "click", and the noun "movie" can also be extracted. "Movie" is the control keyword. It should be noted that the extracted control keyword may be a complete control name or a partial control name.

[0117] In some embodiments, after determining the control keyword, the controller can match the control keyword with all controls in the user interface.

[0118] The controller may first perform a screenshot process on the user interface to obtain a screenshot image of the user interface. Specifically, the controller may perform a screenshot of the image currently displayed on the display through a screenshot program to obtain a screenshot image of the user interface.

[0119] After obtaining the screenshot image of the user interface, the controller can perform text recognition processing on the screenshot image to obtain image recognition information. The image recognition information includes the control information of each control, and the control information includes the control text and the control position. Among them, the control text is the full name of the control, and the control position is the specific position of the control text of the control in the user interface. The control position can be expressed in the form of (xi, yi, Wi, Hi). Among them, (xi, yi) represents the coordinates of a vertex or center point of the i-th control text, Wi represents the length of the i-th control text, and Hi represents the height of the i-th control text. In the embodiment of the present application, (xi, yi) is exemplarily introduced as the center point of the i-th control text.

[0120] Specifically, the controller can use the OCR (Optical Character Recognition) method to recognize the screenshot image, thereby obtaining the text information contained in the screenshot image. For example, the screenshot image can be recognized and processed using the OCR text model to obtain the corresponding image recognition information. Among them, the OCR method can determine its shape by detecting dark and light patterns, and then use the character recognition method to translate the shape into computer text, which is a prior art. The specific OCR recognition method can be set according to the hardware configuration of the actual display device. For example, it can be a text recognition method based on artificial intelligence, neural networks, genetic algorithms, etc., which is not limited in the embodiments of the present application.

[0121] In some embodiments, after obtaining control information of all controls in the user interface, the controller may match the control keyword with all controls.

[0122] Specifically, the controller can traverse the control names of all controls in the user interface. By comparing all control names with the extracted control keywords, that is, matching the control text and the noun, all controls containing the noun can be determined. In an embodiment of the present application, a control whose control text contains the control keyword is determined as a candidate control. The controller can determine all candidate controls in the user interface based on the control keyword.

[0123] When it is detected that there is only one control whose control name contains the control keyword, that is, when there is only one candidate control, the candidate control is considered to be the target control that the user needs to trigger, and the controller can directly trigger the control.

[0124] When it is detected that there are multiple controls whose names contain the control keyword, that is, when there are multiple candidate controls, it is necessary to further determine the target control that the user needs to trigger.

[0125] In some embodiments, if it is determined that there are multiple candidate controls in the user interface, the controller may randomly select a candidate control and control the display to display a prompt interface to allow the user to confirm whether to trigger the candidate control. Figure 9 A schematic diagram of a display prompt interface in some embodiments is shown. In which, the user interface includes multiple candidate controls, and a control A can be randomly selected to allow the user to confirm whether to trigger the control A.

[0126] In some embodiments, the controller may generate a candidate control interface based on the control information of the candidate controls, and control the display to display the candidate control interface, where the candidate control interface may include all candidate controls.

[0127] In some embodiments, when generating a candidate control interface, candidate controls may be highlighted based on the user interface.

[0128] After all candidate controls are determined, their names can be highlighted using text boxes. Each candidate control will have a text box next to its name. The text boxes can be further highlighted, for example, by making them bold or darkening the area within them. Figure 10a A schematic diagram of a candidate control interface in some embodiments is shown. It can be seen that the candidate control interface includes three candidate controls: "My Collection", "My Applications" and "Create My Game Kingdom".

[0129] In some embodiments, when generating a candidate control interface, the candidate control interface may also include only candidate controls. Specifically, the controller may determine the control information of each candidate control, including the control text and the control position. For each control text, it is displayed in the form of a text box, that is, each text box represents a control text, and the size of the text box is the size of the corresponding control text. At the same time, according to the control position of each control text, all text boxes are displayed, that is, all text boxes are placed at the corresponding control positions, thereby obtaining a candidate control interface. Figure 10b A schematic diagram of a candidate control interface in some embodiments is shown, wherein the candidate control interface includes three candidate controls: "My Collection", "My Controls" and "Create My Game Kingdom".

[0130] It should be noted that Figure 10a and Figure 10bThe candidate controls contained in are the same, but the two are displayed in two different ways on the candidate control interface.

[0131] After generating the candidate control interface, the controller may control the display to display the candidate control interface, and the user may further select a candidate control according to the candidate control interface.

[0132] It should be noted that, in the embodiment of the present application, the candidate control in the candidate control interface refers to: a text box corresponding to the control text of the candidate control, which will not be described in detail hereinafter.

[0133] In some embodiments, when displaying the candidate control interface on the display, the controller may randomly select a candidate control as the focus control and highlight the focus control in the candidate control interface. The text box of the focus control may be set to a preset color, such as red. The user may further confirm whether to trigger the focus control.

[0134] In some embodiments, after viewing the focus control, the user can directly click the confirmation button of the remote control, and the controller can further trigger the focus control.

[0135] If the current focus control is not the one the user wants to trigger, the user can do nothing. If the controller does not receive a confirmation instruction from the user within a preset period, it is considered that the current focus control is not the target control. At this time, the controller can further select the target control that the user wants to trigger.

[0136] In some embodiments, a user can directly use a control device, such as a remote control, to operate the candidate control interface on the display. The user can use the up, down, left, and right keys on the remote control to switch the focus control in the candidate control interface so that the current focus control is the target control that the user wants to trigger. After the focus control is the target control, the user can click the confirmation key on the remote control to send a confirmation instruction to the display device.

[0137] After receiving the confirmation instruction sent by the user, the controller can confirm the focus control in the candidate control interface at the current moment and determine the focus control as the target candidate control.

[0138] In some embodiments, users can also use terminal devices, such as mobile phones and tablet computers, to select target controls. For example, when using a mobile phone, the user can control the mobile phone to display a remote control interface for the device. The controller can send the candidate control interface to the mobile phone, so that the mobile phone and the display device display the screen synchronously. At this time, the candidate control interface can be displayed on the mobile phone.

[0139] The user can directly operate in the candidate control interface of the mobile phone, such as touching a candidate control. The mobile phone can identify the candidate control touched by the user and send a confirmation instruction to the display device, which is used to instruct the user to select the target candidate control to be confirmed, that is, the candidate control touched by the user.

[0140] After receiving the confirmation instruction sent by the mobile phone, the controller may determine the candidate control indicated by the confirmation instruction as the target candidate control.

[0141] In some embodiments, the user may also use gestures to confirm the target control to be triggered. The display device may include an image collector. The image collector may be a camera for capturing images of the user. The camera may be built into the display device as a detector, or may be connected to the display device as an external device. For a camera connected to the display device, the camera may be connected to the external device interface of the display device to access the display device.

[0142] The display device can detect the user's image in real time through a camera. The user's image can include content such as the user's gestures. Specifically, it can be configured to determine that the user has sent a confirmation instruction to the display device when the user makes a preset gesture. For example, it can be configured to determine that the user has sent a confirmation instruction to the display device when the user makes a fist, and the controller can further trigger the focus control.

[0143] If no confirmation instruction is received from the user, the controller may continue to confirm the target control selected by the user.

[0144] In some embodiments, when the candidate control interface is displayed on the display, the user can use gestures to select the candidate control, and the controller can further determine the target control selected by the user.

[0145] In the candidate control interface, the controller may first determine an offset angle of each candidate control. The offset angle may be used to represent the relative position of each candidate control with respect to the candidate control interface, and may represent a general direction of the candidate control.

[0146] Specifically, when determining the offset angle of each candidate control, the controller may first set the interface focus of the candidate control interface. The interface focus is the corner point of all offset angles and is also the core point of the candidate control interface.

[0147] In some embodiments, the interface focus may be set to the center point of the candidate control interface, that is, the center point of the display.

[0148] In some embodiments, when setting the interface focus, a selection method based on the shortest global distance may be used, where the center point of the candidate control with the shortest total distance from other candidate controls is selected as the interface focus.

[0149] It should be noted that, in the embodiment of the present application, the center point of the candidate control refers to: the center point of the text box of the control text of the candidate control, which will not be repeated hereafter.

[0150] Specifically, the selection formula is as shown in formula (1):

[0151]

[0152] in, Indicates the distance between the nth candidate control and the ith candidate control.

[0153] By calculating the distance between the center points of each two candidate controls, the total distance from the center point of each candidate control to the center points of all other candidate controls can be obtained. Furthermore, the center point of the candidate control with the smallest total distance can be selected as the interface focus of the candidate control interface.

[0154] In some embodiments, the center point of the candidate control with the smallest distance from the center point of the display can be selected as the interface focus. That is, for all candidate control centers, the distance from each center point to the center point of the display is calculated, and the center point with the smallest distance is selected as the interface focus.

[0155] In some embodiments, an optimal global distance selection method can be used. The average coordinates of the center points of all candidate controls are selected as the interface focus of the candidate control interface. Assume that there are n candidate controls in the candidate control interface, the coordinates of the center point of the i-th candidate control are (xi, yi), and the coordinates of the interface focus are (x0, y0). Then the selection formula for x0 is shown in formula (2), and the selection formula for y0 is shown in formula (3).

[0156]

[0157]

[0158] The coordinates of the interface focus are the average coordinates of the center points of all candidate controls.

[0159] It should be noted that, in the embodiments of the present application, the interface focus is introduced as an example of the center point of the display.

[0160] In some embodiments, after determining the interface focus of the candidate control interface, the controller may further determine a marker point of each candidate control, and obtain an offset angle of each candidate control based on the marker point and the interface focus.

[0161] When determining the marker point for each candidate control, the text box of the candidate control is completely within the offset angle. Specifically, you can first obtain the line between the center point of the candidate control and the interface focal point as the control centerline. Then, obtain the lines between the four vertices of the candidate control and the interface focal point, and calculate the angles between the four lines and the control centerline. The two vertices with the largest angles are selected as the two marker points of the candidate control.

[0162] Each candidate control has two markers. The lines connecting the two markers to the interface focus can be obtained, and the angle between the two lines is determined as the offset angle of each candidate control. The offset angle of each candidate control is within a range, such as 10-30 degrees.

[0163] Figure 11 The offset angles of all candidate controls in a candidate control interface in some embodiments are shown. Point O is the focal point. The candidate control interface includes four candidate controls. Control A has an offset angle of NA, Control B has an offset angle of NB, Control C has an offset angle of NC, and Control D has an offset angle of ND.

[0164] In some embodiments, the controller can also obtain the user's gesture trajectory in the candidate control interface based on the user image collected by the image collector. The image collector can obtain the user image in real time, and the controller can recognize the user's gesture action based on the user image.

[0165] Specifically, the controller may map the user image to the candidate control interface, thereby obtaining the gesture trajectory of the user's gesture action relative to the candidate control interface.

[0166] The controller can be configured to capture the initial user image after a first period (e.g., 5 seconds) starting from the time the candidate control interface is displayed on the display, indicating that the user has begun using gestures to select candidate controls. For the initial user image, the controller can map the initial user gesture focus to the interface focus of the candidate control interface, i.e., determine the initial user gesture focus as the interface focus of the candidate control interface. The user gesture focus can be a preset gesture position, such as the user's palm position.

[0167] After a preset period T1, which may be 5 seconds, the controller again maps the current user image to the candidate control interface, and obtains the position of the user gesture focus at the current moment in the candidate control interface, which is set as the first position.

[0168] Furthermore, the trajectory between the interface focus and the first position can be determined as the user gesture trajectory. The user gesture trajectory can indicate the approximate direction in which the user selects the candidate control using the gesture. Figure 12A schematic diagram of a user gesture trajectory in a candidate control interface in some embodiments is shown. Point O is the focus interface point, and point S1 is the first position. Therefore, the line segment between point O and point S1 is the user gesture trajectory.

[0169] In some embodiments, positioning technologies such as UWB (Ultra Wide Band) technology or AOA (Angle-of-Arrival) technology can also be used to determine the user's gesture trajectory. In the embodiments of this application, AOA technology is used as an example.

[0170] When utilizing AOA technology, both AOA (angle of arrival) and AOD (angle of departure) positioning are required. AOA positioning refers to the fact that a Bluetooth receiver has multiple antennas, and the distance between the transmitting antenna and each receiving antenna varies. Therefore, the transmitted signal has a time difference at each receiving antenna, allowing the phase difference to be calculated. AOD positioning refers to the fact that a Bluetooth positioning beacon sends a signal through an antenna array, while the receiving device receives the signal through a single antenna and calculates the direction of the signal by decoding the received signal.

[0171] This triangulated distance measurement enables highly accurate real-time indoor positioning. This technology allows us to determine the angle and distance between two devices, and then deduce the trajectory of a device, enabling both direction and distance of movement.

[0172] Specifically, a user may hold a control device, such as a remote controller, and point to the candidate control interface. The remote controller may be provided with a positioning button. When the user clicks the positioning button, the remote controller may send a signal to the display device so that the display device can determine the current position of the remote controller relative to the display device.

[0173] When the user clicks the positioning button for the first time, the remote controller sends an initial signal to the display device, and the display device can determine the initial position of the remote controller based on the initial signal.

[0174] When the user moves the handheld control device to the user-specified position, the user can click the positioning button again, and the remote control sends a signal to the display device again. The display device can determine the user-specified position based on the signal.

[0175] Using the display as the base plane, perpendicular lines from the initial position and the user-specified position to the base plane are obtained, and the intersection of the two perpendicular lines and the base plane is further obtained. The initial position corresponds to the first positioning position, and the user-specified position corresponds to the second positioning position. The first positioning position and the second positioning position can be considered as the movement trajectory of the user gesture.

[0176] At this time, the movement trajectory needs to be mapped to the candidate control interface. Specifically, the first positioning position can be mapped to the interface focus of the candidate control interface, that is, the first positioning position is determined as the interface focus of the candidate control interface. In this way, the user gesture trajectory of the user in the candidate control interface is obtained.

[0177] In some embodiments, after determining the offset angles and user gesture trajectories of all candidate controls, the controller may determine the target candidate control selected by the user based on the offset angle and user gesture trajectory of each candidate control.

[0178] Specifically, the controller may first determine all candidate controls in the direction specified by the user gesture.

[0179] The first candidate control may be determined by obtaining a candidate control having a user gesture trajectory in its offset angle from all candidate controls in the candidate control interface. The user gesture trajectory may be considered as a direction specified by the user gesture, such as 100°.

[0180] In some embodiments, a gesture error threshold β can be set to account for the directional error of the user's gesture, for example, 10°. Assuming the direction of the user's gesture trajectory is 100°, the user's gesture direction can be considered to be 90°-110°. Candidate controls whose offset angles overlap with the user's gesture direction can all be considered first candidate controls.

[0181] In some embodiments, the controller may further obtain an associated candidate control of the first candidate control, wherein the overlapped portion of the offset angles of the first candidate control and the associated candidate control is greater than a preset threshold α, for example, the preset threshold may be 50°. Figure 13 A schematic diagram of a candidate control interface in some embodiments is shown, where the gesture error threshold is β. Therefore, the user gesture direction can be considered to include controls C and D. At the same time, no candidate control exists whose offset angle overlaps with controls C or D by more than a preset threshold α. Therefore, the first candidate controls in this candidate control interface are controls C and D, and there are no associated candidate controls.

[0182] In some embodiments, after determining the first candidate control and the associated candidate controls, the controller may further screen these candidate controls to obtain a target candidate control.

[0183] The controller may first obtain the trajectory distance of the user's gesture trajectory, and further determine the target candidate control selected by the user based on the trajectory distance.

[0184] Specifically, the center points of these candidate controls can be mapped to the line corresponding to the user's gesture trajectory to obtain the distance points of each candidate control on the user's gesture trajectory. The distance between the first position of the user's gesture trajectory and these distance points is then determined, and the candidate control corresponding to the distance point with the smallest distance is determined as the target candidate control selected by the user. Figure 14 A schematic diagram illustrating how to determine candidate target controls in some embodiments is shown. Here, the center point of control C is mapped onto the user's gesture trajectory, resulting in a distance point C1, and the center point of control D is mapped onto the user's gesture trajectory, resulting in a distance point D1. The distance from D1 to the first position is significantly smaller than the distance from C1 to the first position, so control D is the candidate target control selected by the user.

[0185] In some embodiments, after the target candidate control selected by the user is determined, the target candidate control can be used as a new focus control and highlighted in the candidate control interface to allow the user to confirm whether to trigger the focus control.

[0186] If the user sends a confirmation instruction to the display device, the controller triggers the focus control, that is, triggers the target candidate control selected by the user.

[0187] If no confirmation instruction input by the user is detected, it is considered that the focus control is not the target candidate control selected by the user, and the target candidate control needs to be confirmed again.

[0188] At this point, the user continues to use gestures to select controls.

[0189] Specifically, the user image can be mapped to the candidate control interface again after a preset period T1, wherein the initial user gesture focus is still used as the interface focus of the candidate control interface, and at the same time, the current position of the user gesture focus in the candidate control interface can be obtained and set as the second position.

[0190] At this time, the trajectory between the interface focus and the second position can be determined as a new user gesture trajectory, and the target candidate control selected by the user can be determined again based on the user gesture trajectory. The steps for determining the target candidate control selected by the user can refer to the above content and will not be repeated here.

[0191] The controller can continuously determine the focus control until the user confirms that a focus control is a target candidate control, and the controller further triggers the target candidate control, thereby meeting the user's needs.

[0192] This application can identify the control keyword contained in the control trigger instruction based on the control trigger instruction, thereby obtaining all candidate controls containing the keyword. By using an image containing the user's gesture action, the user's gesture trajectory can be obtained, and the direction and distance indicated by the user's gesture can be further confirmed, thereby accurately identifying the user's target control among multiple similar controls. It can respond to user instructions, thereby improving the user experience.

[0193] The present application also provides a control identification method, which is applied to a display device, such as Figure 15 As shown, the method includes:

[0194] Step 1501: In response to a control trigger instruction input by a user, obtain a control keyword in the control trigger instruction.

[0195] Step 1502: Control the display to display a candidate control interface, so that the user can select a candidate control according to the candidate control interface; the candidate control interface includes at least one candidate control, and the control text of the candidate control includes the control keyword.

[0196] Step 1503: Determine the target candidate control selected by the user according to the user's operation on the candidate control interface.

[0197] Step 1504: trigger the target candidate control.

[0198] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be repeated here.

[0199] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention or certain portions of the embodiments.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0201] For ease of explanation, the above description has been presented in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments have been selected and described to better explain the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: monitor; The controller is configured as: In response to a control trigger instruction input by a user, obtaining a control keyword in the control trigger instruction; Controlling the display to display a candidate control interface so that the user can select a candidate control according to the candidate control interface; the candidate control interface includes at least one candidate control, and the control text of the candidate control includes the control keyword; Obtaining an offset angle and a user gesture trajectory of a candidate control; the offset angle is used to represent the position of the candidate control relative to the candidate control interface; Obtaining a candidate control whose offset angle includes the user gesture trajectory and determining it as a first candidate control; obtaining an associated candidate control of the first candidate control, wherein an overlap between the offset angles of the first candidate control and the associated candidate control is greater than a preset threshold; Determining a trajectory distance of the user gesture trajectory, and screening the first candidate control and the associated candidate controls according to the trajectory distance to determine a target candidate control selected by the user; Trigger the target candidate control.

2. The display device according to claim 1, wherein The controller is further configured to: Before executing the step of obtaining the control keyword in the control trigger instruction, Controlling the display to display a user interface so that the user can input the control trigger instruction; Before executing the step of controlling the display to display the candidate control interface, Acquire all candidate controls in the user interface according to the control keyword; Generate a candidate control interface based on all candidate controls.

3. The display device according to claim 2, wherein The controller is further configured to: In the step of acquiring all candidate controls in the user interface according to the control keyword, Obtaining control information of each control in the user interface, wherein the control information includes control text and control position; Determine a control whose control text contains the control keyword as a candidate control; Get all candidate controls in the user interface.

4. The display device according to claim 2, wherein: The controller is further configured to: After executing the step of controlling the display to display the candidate control interface, Randomly select a candidate control from all candidate controls as a focus control, and highlight it in the candidate control interface to allow the user to confirm whether to trigger the focus control; If a confirmation instruction input by the user is detected, triggering the focus control; If no confirmation instruction input by the user is detected, the target candidate control selected by the user is determined.

5. The display device according to claim 4, wherein: The controller is further configured to: In the step of determining the target candidate control selected by the user, In response to an instruction sent by the control device when the user switches the focus control in the candidate control interface, the switched focus control is acquired and determined as the target candidate control.

6. The display device according to claim 1, wherein The controller is further configured to: After executing the step of controlling the display to display the candidate control interface, Sending the candidate control interface to a terminal device so that the terminal device displays the candidate control interface; In response to an instruction sent by the mobile terminal when a user touches a candidate control in the candidate control interface, the candidate control touched by the user is determined as the target candidate control.

7. The display device according to claim 1, wherein Also includes: An image collector configured to collect user images; The controller is further configured to: in executing the step of obtaining the offset angle and user gesture trajectory of the candidate control, Obtaining an offset angle of each candidate control in the candidate control interface, where the offset angle is used to represent a position of the candidate control relative to the candidate control interface; The user gesture trajectory of the user in the candidate control interface is obtained according to the user image collected by the image collector.

8. The display device according to claim 7, wherein: The controller is further configured to: In the step of obtaining the offset angle of each candidate control in the candidate control interface, Setting the interface focus of the candidate control interface; Determine the marking points of each candidate control, where each candidate control has two marking points; The lines between the two marking points of each candidate control and the interface focus are obtained respectively, and the angle between the two lines is determined as the offset angle of each candidate control.

9. The display device according to claim 8, wherein The controller is further configured to: In the step of acquiring the user gesture trajectory of the user in the candidate control interface according to the user image collected by the image collector, Mapping the user image to the candidate control interface, and determining an initial user gesture focus as an interface focus of the candidate control interface; The user gesture focus after obtaining a preset period is located at the first position in the candidate control interface; A trajectory between the interface focus and the first position is determined as a user gesture trajectory.

10. The display device according to claim 9, wherein The controller is further configured to: After performing the steps to determine the target candidate control selected by the user, The target candidate control is used as a focus control and is highlighted in the candidate control interface so that the user can confirm whether to trigger the focus control; If a confirmation instruction input by the user is detected, executing the step of triggering the target candidate control; If no confirmation instruction input by the user is detected, obtaining the user gesture focus after the next preset period is located at a second position in the candidate control interface; Determining a trajectory between the interface focus and the second position as an updated user gesture trajectory; The step of determining the target candidate control selected by the user is performed according to the updated user gesture trajectory.

11. A control identification method, applied to a display device, characterized in that: The method comprises: In response to a control trigger instruction input by a user, obtaining a control keyword in the control trigger instruction; Controlling the display to display a candidate control interface so that the user can select a candidate control according to the candidate control interface; the candidate control interface includes at least one candidate control, and the control text of the candidate control includes the control keyword; Obtaining an offset angle and a user gesture trajectory of a candidate control; the offset angle is used to represent the position of the candidate control relative to the candidate control interface; Obtaining a candidate control whose offset angle includes the user gesture trajectory and determining it as a first candidate control; obtaining an associated candidate control of the first candidate control, where the overlap of the offset angles of the first candidate control and the associated candidate control is greater than a preset threshold; determining a trajectory distance of the user gesture trajectory, and screening the first candidate control and the associated candidate controls based on the trajectory distance to determine a target candidate control selected by the user; Trigger the target candidate control.

Images