Display device and control triggering methods

Abstract

The present application provides a display device and a control triggering method, which can determine the control keywords in the control triggering instruction input by the user. The position information of the control keyword in the user interface is further determined, and finally the control at the position information is triggered, thereby triggering the target control that the user wants to use. The present application can determine the position information of the control keyword in the user interface, rather than the position information of the text obtained by text recognition, so the position of the target control can be accurately determined, thereby triggering the target control. Therefore, the present application has high accuracy when triggering controls, and provides a strong user experience.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a control triggering method. Background Art

[0002] Display devices refer to terminal devices that can output specific display images, such as smart TVs, mobile terminals, smart advertising screens, projectors, etc. With the rapid development of display devices, the functions of display devices will become more and more abundant, and the performance will become more and more powerful. They can realize two-way human-computer interaction functions, integrate multiple functions such as audio and video, entertainment, and data, and are used to meet the diverse and personalized needs of users. Intelligent voice interaction has become one of the main functions of display devices.

[0003] For display devices with intelligent voice interaction functions, users can input control commands by voice to realize some functions of the display device. For example, the user can control the display device to trigger the corresponding target control by saying "I want to see something". The display device can take a screenshot of the current screen of the display and perform text recognition to obtain multiple texts. By matching the user's instructions with these texts, the location information of the text corresponding to the instructions is obtained. The display device triggers the control at the location information to realize the user's instructions.

[0004] However, each text obtained through text recognition may contain multiple control names. In this case, the position information obtained is the position information of the entire text, not the position information of the target control. The control at the position information may be another control, not the target control. Therefore, the display device may trigger the wrong control and fail to implement the user's control instructions, resulting in a poor user experience. Summary of the invention

[0005] The present invention provides a display device and a control triggering method to solve the problem in the related art that a display device may trigger an erroneous control, resulting in a poor user experience.

[0006] In a first aspect, the present application provides a display device. The display device comprises a display, a sound collector, and a controller. The display is configured to display a user interface; the sound collector is configured to receive a voice command input by a user; and the controller is configured to perform the following steps:

[0007] In response to a control trigger instruction input by a user, a control keyword in the control trigger instruction is determined; position information of the control keyword in the user interface is determined; and a control at the position information is triggered.

[0008] In some implementations, the controller is further configured to: in executing the step of determining the control keyword in the control trigger instruction,

[0009] The control trigger instruction is converted into a control trigger text; and the control trigger text is segmented to obtain a control keyword.

[0010] In some implementations, the controller is further configured to: in the step of determining the position information of the control keyword in the user interface,

[0011] The user interface is screenshoted to obtain a screenshot image; the screenshot image is subjected to text recognition processing to obtain image recognition information; and the position information of the control keyword in the user interface is obtained based on the image recognition information.

[0012] In some implementations, the controller is further configured to: in executing the step of acquiring the position information of the control keyword in the user interface based on the image recognition information,

[0013] Determine multiple text recognition information included in the image recognition information; determine control keyword recognition information corresponding to the control keyword in the multiple text recognition information, and obtain position information of the control keyword in the user interface according to the control keyword recognition information.

[0014] In some implementations, the control keyword identification information includes control keyword identification text and identification text position information.

[0015] In some implementations, the controller is further configured to: in executing the step of acquiring the position information of the control keyword in the user interface according to the control keyword identification information,

[0016] The control keyword identification information is detected; when it is detected that the control keyword identification information also includes character information, the position information of each character of the control keyword is determined according to the character information; when it is detected that the control keyword identification information does not include character information, the character information is determined according to the control keyword identification information; the character information includes the position information of each character of the control keyword identification text; the position information of the control keyword in the user interface is determined according to the position information of each character of the control keyword.

[0017] In some implementations, the controller is further configured to: in the step of determining character information according to the control keyword identification information,

[0018] Determine the number of characters and the number of null characters between the first character and each character in the control keyword recognition text, and obtain the character length and the null character length; calculate the first character length between the first character and each character according to the character length and the number of characters; calculate the first null character length between the first character and each character according to the null character length and the number of null characters; calculate the length from the first character to each character according to the first character length and the first null character length; determine the position information of each character according to the length from the first character to each character and the recognition text position information.

[0019] In some implementations, the controller is further configured to: in executing the step of obtaining the length of the null character,

[0020] Determine the length of the control keyword recognition text according to the recognition text position information; determine the total number of characters and the total number of empty characters in the control keyword recognition text, and calculate the total length of characters in the control keyword recognition text according to the total number of characters and the character length; calculate the difference between the length of the control keyword recognition text and the total length of characters; calculate the ratio of the difference and the total number of empty characters to obtain the empty character length.

[0021] In some implementations, the controller is further configured to: in executing the step of calculating the length of each character from the first character according to the first character length and the first empty character length,

[0022] Obtain the length of character intervals; calculate the sum of the number of characters and the number of empty characters to obtain a first number; calculate the total length of character intervals from the first character to each character based on the character interval length and the first number; calculate the sum of the first character length, the first empty character length, and the total length of character intervals to obtain the length from the first character to each character.

[0023] In a second aspect, the present application provides a control triggering method, applied to a display device, comprising:

[0024] In response to a control trigger instruction input by a user, a control keyword in the control trigger instruction is determined; position information of the control keyword in the user interface is determined; and a control at the position information is triggered.

[0025] It can be seen from the above technical solutions that the present application provides a display device and a control triggering method, which can determine the control keywords in the control triggering instruction input by the user. The position information of the control keyword in the user interface is further determined, and finally the control at the position information is triggered, thereby triggering the target control that the user wants to use. The present application can determine the position information of the control keyword in the user interface, rather than the position information of the text obtained by text recognition, so the position of the target control can be accurately determined, thereby triggering the target control. Therefore, the present application has high accuracy when triggering controls, and provides a strong user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 shows a usage scenario of a display device according to some embodiments;

[0028] Figure 2 shows a hardware configuration block diagram of a control device 100 according to some embodiments;

[0029] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0030] Figure 4 shows a software configuration diagram in a display device 200 according to some embodiments;

[0031] Figure 5 A schematic diagram of a user interface in some embodiments is shown;

[0032] Figure 6 A schematic diagram showing voice interaction mode confirmation information displayed on a display in some embodiments is shown;

[0033] Figure 7 A schematic diagram of a user interface in some embodiments is shown;

[0034] Figure 8 A schematic diagram showing an application list in some embodiments;

[0035] Fig. 9 A flowchart of the interaction between components of a display device in some embodiments is shown;

[0036] Fig.10 A schematic diagram showing text recognition information in some embodiments;

[0037] Fig.11A schematic diagram showing prompt information displayed on a display in some embodiments is shown;

[0038] Fig.12 A flow chart of an embodiment of a control triggering method is shown. DETAILED DESCRIPTION

[0039] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0040] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0041] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0042] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0043] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0044] Figure 1 Schematic diagram of a usage scenario of a display device according to an embodiment. Figure 1 As shown, the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0045] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and at least one of other short-range communication methods, to control the display device 200 wirelessly or wired. The user may control the display device 200 by inputting user commands through at least one of buttons on the remote controller, voice input, and control panel input.

[0046] In some embodiments, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a laptop computer, an AR / VR device, etc.

[0047] In some embodiments, the smart device 300 may also be used to control the display device 200. For example, the display device 200 may be controlled using a camera application running on the smart device.

[0048] In some embodiments, the smart device 300 and the display device may also be used to communicate data.

[0049] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0050] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0051] In some embodiments, the software steps executed by a step execution subject can be migrated to another step execution subject in data communication with it as needed for execution. Exemplarily, the software steps executed by a server can be migrated to a display device in data communication with it as needed for execution, and vice versa.

[0052] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200.

[0053] In some embodiments, the communication interface 130 is used for communicating with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module.

[0054] In some embodiments, the user input / output interface 140 includes at least one of a microphone, a touch pad, a sensor, a button, or alternative modules.

[0055] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown.

[0056] In some embodiments, the display apparatus 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0057] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output.

[0058] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface, etc.

[0059] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0060] In some embodiments, the tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0061] In some embodiments, the communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0062] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0063] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0064] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0065] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0066] In some embodiments, the object may be any one of the selectable objects, such as a hyperlink, an icon or other operable controls. Operations related to the selected object include: displaying a page, document, image, etc. connected to the hyperlink, or executing a program corresponding to the icon.

[0067] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM Random Access Memory (RAM), a ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0068] CPU processor. Used to execute operating system and camera application instructions stored in the memory, and execute various camera applications, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video content. CPU processor can include multiple processors. For example, it includes a main processor and one or more sub-processors.

[0069] In some embodiments, the graphics processor is used to generate various graphic objects, such as at least one of icons, operation menus, and user input command display graphics. The graphics processor includes an operator, which performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes; and a renderer, which renders various objects obtained based on the operator, and the rendered objects are used to be displayed on the display.

[0070] In some embodiments, the video processor is used to receive an external video signal and perform at least one of the following video processing operations, such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis, according to a standard codec protocol of the input signal, to obtain a signal that can be directly displayed or played on the display device 200.

[0071] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, and a display formatting module. The demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to the user input or itself with the scaled video image to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to receive the video output signal after frame rate conversion and change the signal to conform to the display format signal, such as outputting an RGB data signal.

[0072] In some embodiments, the audio processor is used to receive an external audio signal, perform decompression and decoding according to a standard codec protocol of the input signal, and perform at least one of noise reduction, digital-to-analog conversion, and amplification processing to obtain a sound signal that can be played in a speaker.

[0073] In some embodiments, the user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0074] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between the camera application or operating system and the user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0075] In some embodiments, the user interface 280 is an interface that can be used to receive control input (such as a physical button on the display device body, or other interfaces).

[0076] In some embodiments, the system of the display device may include a kernel, a command parser (shell), a file system, and a camera application. The kernel, shell, and file system together constitute the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts the hardware, initializes hardware parameters, etc., runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and the user camera application are loaded. After startup, the camera application is compiled into machine code to form a process.

[0077] See also Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the camera application (Applications) layer (referred to as "application layer"), the camera application framework (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0078] In some embodiments, at least one camera application is running in the camera application layer, and these camera applications can be window programs, system settings programs, clock programs, etc. provided by the operating system; or they can be camera applications developed by third-party developers. In specific implementations, the camera application packages in the camera application layer are not limited to the above examples.

[0079] The framework layer provides an application programming interface (API) and programming framework for the camera applications in the camera application layer. The camera application framework layer includes some predefined functions. The camera application framework layer is equivalent to a processing center, which determines the actions to be taken by the camera applications in the application layer. Through the API interface, the camera application can access system resources and obtain system services during execution.

[0080] like Figure 4 As shown, in the embodiment of the present application, the camera application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the camera application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0081] In some embodiments, the activity manager is used to manage the life cycle of each camera application and the common navigation back function, such as controlling the exit, opening, and back of the camera application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0082] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0083] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0084] Each text obtained through text recognition may contain multiple control names. In this case, the position information obtained is the position information of the entire text, not the position information of the target control. The control at the position information may be another control, not the target control. Therefore, the display device may trigger the wrong control and fail to implement the user's control instructions, resulting in a poor user experience.

[0085] The present application provides a display device, including a display, a sound collector and a controller.

[0086] The display is used to display the user interface. The user interface is the screen content currently displayed in the display. The user interface may be a specific target image, such as various media resources obtained from a network signal source, including videos, pictures and other content. The user interface may also be some UI interfaces of the display device. The sound collector may be a microphone for receiving voice commands input by the user, such as control trigger commands.

[0087] In some embodiments, when the user controls the display device to turn on, the controller can control the display to display a user interface. Figure 5 A schematic diagram of a user interface in some embodiments is shown, wherein the user interface includes a media resource interface for playing media resources. The user interface also includes some specific controls, such as "movie". The user can click on the media resource interface to control the display to display the media resource in full screen mode. The user can also click on a control to control the display to display the interface corresponding to the control.

[0088] In some embodiments, the display device has a voice interaction function, and the user can control the display device by inputting voice. The display device can be provided with a voice interaction mode. In the voice interaction mode, the user can perform voice interaction with the display device.

[0089] In some embodiments, the user can send a voice interaction mode instruction to the display device by operating a designated button of the remote control. In the actual application process, the correspondence between the voice interaction mode instruction and the remote control button is pre-bound. For example, a voice interaction mode button is set on the remote control. When the user touches the button, the remote control sends the voice interaction mode instruction to the controller, and the controller controls the display device to enter the voice interaction mode. When the user touches the button again, the controller can control the display device to exit the voice interaction mode.

[0090] In some embodiments, the user can use the sound collector of the display device, such as a microphone, to directly control the display device to enter the voice interaction mode by voice input. The display device may be provided with an intelligent voice system, which can recognize the user's voice to extract the content of the command input by the user. The user can input a preset wake-up word through the microphone to start the intelligent voice system, so that the controller can respond to the command input by the user. For example, the user can enter "classmate XX" to start the intelligent voice system, at which point the display device will enter the voice interaction mode.

[0091] In some embodiments, the user can also send voice interaction mode instructions to the display device through preset gestures. The display device can detect the user's behavior through an image collector, such as a camera. When the user makes a preset gesture, it can be considered that the user has sent a voice interaction mode instruction to the display device. For example, it can be set as follows: when it is detected that the user draws a V, it is determined that the user has input a voice interaction mode instruction to the display device. The user can also send voice interaction mode instructions to the display device through preset actions. For example, it can be set as follows: when it is detected that the user lifts his left foot and right hand at the same time, it is determined that the user has input a voice interaction mode instruction to the display device.

[0092] In some embodiments, when a user uses a smart device to control a display device, such as a mobile phone, a voice interaction mode instruction may also be sent to the display device. In actual application, a control may be set in the mobile phone, through which a user may select whether to enter the voice interaction mode, thereby sending a voice interaction mode instruction to the controller, and the controller may control the display device to enter the voice interaction mode.

[0093] You can also set a voice interaction mode option in the UI interface of the display device. When the user clicks on the option, the display device can be controlled to enter or exit the voice interaction mode.

[0094] In some embodiments, to prevent users from accidentally triggering the voice interaction mode, when the controller receives a voice interaction mode instruction, it can control the display to display voice interaction mode confirmation information, so that the user can make a second confirmation whether to control the display device to enter the voice interaction mode. Figure 6 A schematic diagram showing voice interaction mode confirmation information displayed on a display in some embodiments is shown.

[0095] In some embodiments, when the display device is triggered to enter the voice control mode, the user can also send instructions to the display device in text form through a mobile phone, remote control or other device to prevent the display device from being unable to receive the user's voice commands when there is a problem with the microphone.

[0096] When the display device enters the voice interaction mode, it can interact with the user by voice. The user can input various commands to the display device through the microphone to implement various operations on the display device. The user can send a control trigger command to the display device to trigger the target control that the user needs to use.

[0097] In some embodiments, after the display device enters the voice interaction mode, the user can view the user interface and send a control trigger instruction. Figure 7 A schematic diagram of a user interface in some embodiments is shown.

[0098] The control trigger instruction can be an instruction containing the full name of a control, for example, the user can send a voice message "I want to watch a movie". Figure 7 As shown in the figure, "Movie" is the name of the control. When the display device receives the instruction, it can trigger the "Movie" control and display the interface corresponding to the "Movie" control to the user. For another example, the user can send "Open my app". When the display device receives the instruction, it can trigger the "My App" control and display the interface corresponding to the "My App" control to the user. The interface may include all the applications installed in the display device, such as Figure 8 As shown, the interface corresponding to the "My Applications" control may include an application list, and the application list includes all applications installed on the display device.

[0099] The control trigger instruction may also be an instruction containing a partial name of a control. For example, there is a control "The Strongest Animation Season" in the user interface. The user can send a voice message "I want to watch the animation season". When the display device receives the instruction, it can determine that the control the user wants to trigger is "The Strongest Animation Season", thereby triggering the control.

[0100] In some embodiments, after the user sends a control trigger instruction, the display device may trigger a target control corresponding to the control trigger instruction. Fig. 9 A flowchart of the interactions among components of a display device in some embodiments is shown.

[0101] In response to a control trigger instruction input by a user, the controller may first determine a control keyword related to the control contained in the control trigger instruction. The control keyword may be the full name of a control or a partial name of a control. There may be a one-to-one correspondence between the control keyword and the control name.

[0102] In some embodiments, when the display device receives a control trigger instruction input by the user, the controller can send the received voice data to the voice recognition service, thereby converting it into text information to obtain a control trigger text. The recognition operation of the user's control trigger instruction can refer to the relevant technology, and the embodiments of the present application will not be repeated one by one.

[0103] In some embodiments, the display device may further include a third-party voice recognition interface. Upon receiving a control trigger instruction input by a user, the controller may send voice data to the third-party voice recognition interface, and use a third-party voice recognition device to recognize the user's control trigger instruction as a control trigger text.

[0104] After obtaining the control trigger text, the controller can analyze the control trigger text to obtain the control keyword in the control trigger text.

[0105] In some embodiments, the controller may identify the control trigger text to obtain some keywords in the control trigger text.

[0106] The controller can first perform word segmentation processing on the control trigger text to obtain a word segmentation result including multiple words. The word segmentation processing can use the open source word segmentation tool JIEBA. For example, for the control trigger instruction "I want to watch a movie", after word segmentation processing, the word segmentation result can be obtained as three words "I want, watch, movie". The specific word segmentation method can refer to the relevant technology, and this application will not go into details.

[0107] After the control trigger text is segmented, the controller can parse the segmentation results and extract the keywords in the segmentation results. Specifically, the nouns in the segmentation results can be extracted. For example, for the segmentation result "I want to watch movies", "watch" can be parsed as the control instruction "click", and the noun "movie" can also be extracted. "Movie" is the name of the control in the user interface, that is, the control keyword.

[0108] In some embodiments, considering that the control trigger instruction input by the user may only contain a partial name of a control, for example, the control trigger instruction is "I want to watch animation season". After performing word segmentation processing on the control trigger text, the controller can extract the noun "animation season".

[0109] The controller can traverse all control names in the UI layout information corresponding to the user interface. By comparing all control names with the extracted nouns, that is, matching the control names with the nouns, the repeated words in the noun and all control names can be determined, and the words are determined as control keywords.

[0110] In some embodiments, after determining the control keyword in the control trigger instruction, the controller can determine the position information of the control keyword in the user interface.

[0111] The controller may first perform a screenshot process on the user interface to obtain a screenshot image of the user interface. Specifically, the controller may perform a screenshot of the screen currently displayed on the display through a screenshot program to obtain a screenshot image of the user interface.

[0112] After acquiring the screenshot image of the user interface, the controller can perform text recognition processing on the screenshot image to obtain image recognition information. The image recognition information can be used to determine the position information of the control keyword in the user interface.

[0113] Specifically, the controller can use the OCR (Optical Character Recognition) method to recognize the screenshot image, so as to obtain the text information contained in the screenshot image. For example, the screenshot image is recognized and processed using the OCR text model to obtain the corresponding image recognition information. Among them, the OCR method can determine its shape by detecting dark and light patterns, and then use the character recognition method to translate the shape into computer text, which is a prior art. The specific OCR recognition method can be set according to the hardware configuration of the actual display device. For example, it can be a text recognition method based on artificial intelligence, neural network, genetic algorithm, etc., which is not limited in the embodiments of the present application.

[0114] In some embodiments, the OCR recognition method can convert the screenshot image from a picture form to a text form, and the text information contained in the screenshot image can be obtained. When performing OCR recognition on the screenshot image, a specific text can be obtained, and there are multiple texts. At the same time, the position information of each text in the screenshot image can also be identified, that is, the position information of each text in the user interface. Therefore, the image recognition information includes multiple text recognition information, and each text recognition information can include a specific recognition text and the position information of the recognition text.

[0115] It should be noted that the OCR recognition method cannot guarantee the recognition of specific control names, and can only recognize text information one by one. Therefore, a text may contain multiple control names. For example, the recognized text can be "Selected *UP Master Awards * The Strongest Animation Season * National Creation". Among them, the recognized text includes specific text characters, and the embodiments of this application are uniformly expressed in characters. "*" represents a space, that is, an empty character. At the same time, the position information of the entire text in the user interface can also be identified.

[0116] The text can be regarded as a rectangular area. The position information of the text can be expressed as (x0, y0, W0, H0). Among them, (x0, y0) represents the coordinates of a vertex or center point of the rectangular area, W0 represents the length of the text, and H0 represents the height of the text. In the embodiment of the present application, (x0, y0) is exemplarily set as the lower left corner vertex of the rectangular area. Fig.10 A schematic diagram showing text recognition information in some embodiments.

[0117] In some embodiments, after acquiring the image recognition information, the controller may determine a plurality of text recognition information included in the image recognition information, each piece of text recognition information including a piece of recognition text and position information of the recognition text in the user interface.

[0118] The controller can match the control keyword with all the recognition texts, thereby determining the recognition text where the control keyword is located as the control keyword recognition text. After determining the control keyword recognition text, the text recognition information corresponding to the control keyword can be further determined, that is, the text recognition information corresponding to the recognition text, and use it as the control keyword recognition information. The control keyword recognition information includes the control keyword recognition text and the recognition text position information corresponding to the recognition text.

[0119] In some embodiments, the controller can obtain the position information of the control keyword in the user interface based on the control keyword identification information. Specifically, the control keyword identification information may include character information, and the character information refers to the position information of each character of the control keyword identification text. The controller can determine the position information of each character of the control keyword in the control keyword identification text based on the character information. Then, the position information of the control keyword in the user interface can be determined based on the position information of each character of the control keyword.

[0120] In some embodiments, when the user interface displayed in the display is different, the corresponding control keyword identification information is also different. However, the acquired control keyword identification information may include character information or may not include character information. Therefore, before acquiring the position information of each character of the control keyword, the control keyword identification information may be first detected to determine whether the control keyword identification information includes character information.

[0121] When it is detected that the control keyword identification information includes character information, the position information of each character of the control keyword can be directly determined according to the character information.

[0122] When it is detected that the control keyword identification information does not include the character information, the controller may determine the character information according to the control keyword identification information.

[0123] In some embodiments, the controller may first obtain the sizes of characters and space characters.

[0124] The size of the character can be determined according to the position information of the control keyword recognition text, assuming that the position information of the control keyword recognition text is (x0, y0, W0, H0). The height of each character is the same, which is the height of the control keyword recognition text, that is, the height of the character is H0.

[0125] The length and height of each character can be in a fixed ratio, for example, the length of the character is αH0, where α is the ratio of the length to the height. In the embodiment of the present application, the length and height of each character are set to be the same. Therefore, the size of the character is length = height = H0.

[0126] The height of a character is the same as the height of the text, H0. However, the length and height of a null character may be different, so the length of the null character needs to be determined.

[0127] In some embodiments, the length of the null character can be set to a fixed value. Specifically, the null character length S can be set by the user. NC For example, you can set the length of the null character to be the same as the length of the character, that is, S NC =H0.

[0128] In some embodiments, a fixed ratio θ between the length of the space character and the length of the character may be set, which may be 0.8. NC =θ*H0.

[0129] In some embodiments, the null character length S can be calculated by a preset null character formula: NC .

[0130] Specifically, the length of the control keyword recognition text is W0. The controller can determine the total number of characters N in the control keyword recognition text C and the total number of null characters N NC The total length of characters in the control keyword recognition text can be calculated based on the total number of characters and the character length.

[0131] The total length of null characters can be obtained by calculating the difference between the length W0 of the control keyword recognition text and the total length of characters. The null character length can be determined based on the total length of null characters and the total number of null characters.

[0132] Specifically, the empty character formula is:

[0133]

[0134] In some embodiments, after the character length and the null character length are determined, the position information of each character of the control keyword recognition text may be obtained.

[0135] For each character, the character can also be regarded as a rectangular area, and the position information of each character can be set as (xi, yi, Wi, Hi).

[0136] Specifically, the position information of the control keyword recognition text is (x0, y0, W0, H0). The length and height of the character are the same as the text height, so Wi = Hi = H0, yi = y0. That is, the position information of each character is (xi, y0, H0, H0). At this time, it is necessary to determine the X coordinate of the lower left corner vertex of each character.

[0137] In some embodiments, when determining the X coordinate of the lower left corner vertex of each character, the controller may first determine the number of characters and the number of blank characters between the first character and each character in the control keyword recognition text. The number of characters between the first character and each character is the sequence number of each character, that is, the character is the character number in the control keyword recognition text.

[0138] The total length of characters from the first character to each character can be calculated based on the number of characters and set as the first character length. Similarly, the total length of null characters from the first character to each character can be calculated based on the number of null characters and set as the first null character length.

[0139] Therefore, the length from the first character to each character can be further determined according to the first character length and the first empty character length, and the X coordinate of the lower left corner vertex of each character can be obtained according to the length.

[0140] Specifically, the calculation formula of the X coordinate xi is:

[0141]

[0142] in:

[0143] xi represents the X coordinate of the lower left corner of the ith character of the control keyword recognition text; i represents the character number;

[0144] n represents the number of empty characters between the first character and the i-th character in the control keyword recognition text;

[0145] S NC Indicates the null character length.

[0146] According to the above formula, the X coordinate of the lower left corner vertex of each character can be calculated, and the position information (xi, y0, H0, H0) of each character can be further determined.

[0147] In some embodiments, considering that there may be character intervals between adjacent characters, between adjacent characters and blank characters, and between adjacent blank characters, the controller may obtain the X coordinate of the lower left corner vertex of each character according to the character intervals.

[0148] In the embodiment of the present application, all character intervals are set to be the same, and the length of the character interval is γ.

[0149] When obtaining the X coordinate of the lower left corner vertex of each character, the controller can calculate the sum of the number of characters between the first character and each character and the number of empty characters, and set it as the first number. According to the first number and the character interval length, the total length of the character interval from the first character to each character can be calculated.

[0150] The length between the first character and each character can be further determined according to the total length of the character interval, and the X coordinate of the lower left corner vertex of each character can be obtained according to the length.

[0151] Specifically, the calculation formula of the X coordinate xi is:

[0152]

[0153] The calculation formula of the X coordinate xi can also be expressed as:

[0154]

[0155] According to the above formula, the X coordinate of the lower left corner vertex of each character can be calculated, and the position information (xi, y0, H0, H0) of each character can be further determined.

[0156] In some embodiments, after the position information of each character in the control keyword recognition text is determined, each character of the control keyword can be filtered out from the control keyword recognition text, and the position information of each character of the control keyword can be determined.

[0157] In some embodiments, the controller may determine the position information of the entire control keyword in the user interface based on the position information of each character of the control keyword.

[0158] Specifically, assume that the position information of the first character of the control keyword is (x1, y0, H0, H0), and the position information of the last character is (xz, y0, H0, H0).

[0159] The X coordinate of the lower left corner of the control keyword is the X coordinate x1 of the first character, and the Y coordinate is y0. The length of the control keyword is xz-x1+H0, and the height is H0.

[0160] Therefore, the position information of the control keyword is (x1, y0, xz-x1+H0, H0).

[0161] In some embodiments, the center coordinates of the control keyword can also be calculated, and the center coordinates are [x1+(xz-x1+H0) / 2, y0+H0 / 2], that is, [(xz+x1+H0) / 2, y0+H0 / 2]. The center coordinates can be used as the position information of the control keyword.

[0162] In some embodiments, in addition to characters and space characters, considering that there may be some special characters in the control keyword recognition text, such as numbers and punctuation marks, etc., it is also necessary to determine the size of the special characters.

[0163] The size of each special character may be different. Considering that the control keyword recognition text is a line of text, the height of all special characters can be set to be the same as the height of the control keyword recognition text, that is, the height of all special characters is H0. At this time, it is necessary to determine the length of each special character. Special characters can include numbers, English characters, and punctuation characters.

[0164] In some embodiments, for numeric special characters, it can be considered that the length of each digit is the same, and there is a fixed ratio a between numeric special characters and characters, that is, the length of all numeric special characters is a*H0.

[0165] In some embodiments, for English-type special characters, the length of each English letter may be different, for example, the lengths of the English letters "i" and "m" are obviously different. At this time, a fixed length can be set for each English letter. Taking into account the case of English letters, the 26 uppercase letters and 26 lowercase letters can be set to their own lengths, which can be set by the technician. For example, it is set by the developer of the algorithm related to the control trigger function. The embodiments of this application are not limited. Take the recognition text "Selected * UP Master Awards * The Strongest Animation Season * National Creation" as an example, in which the English letters "U" and "P" have a specific length.

[0166] In some embodiments, for a punctuation-type special character, the controller may detect whether the punctuation is located at the last position of the control keyword recognition text.

[0167] It should be noted that if the last digit of the control keyword recognition text is a punctuation mark, the length of the recognized punctuation mark will be relatively small compared to the character length. Therefore, it is necessary to determine the length of each punctuation mark. For example, for the recognition text "The Strongest Animation Season!", the punctuation mark "!" is located at the last digit of the recognition text, so its length is less than the character length. At this time, the length of the punctuation mark "!" can be determined.

[0168] For common punctuation marks, such as commas, quotation marks, and exclamation marks, a fixed length can be set for each punctuation mark, that is, each punctuation mark has its own specific length.

[0169] In some embodiments, for punctuation-type special characters, and when the punctuation is not the last character of the control keyword recognition text, the controller can detect the sum of the number of characters and the number of special characters in the control keyword recognition text.

[0170] If the sum of the number of characters and the number of special characters is less than the preset control threshold, it is considered that the length of the punctuation-type special character in the recognition text is less than the character length. For example, the control threshold can be set to 6. For the recognition text "Education.Summer vacation", this text includes a total of four characters "Jiao, Yu, Shu, Jia" and one punctuation mark "." The sum of the two types of characters is 5, which is less than the control threshold. Therefore, it can be considered that the control corresponding to this recognition text is a small control, and the length of the punctuation-type special character "." in it is less than the character length. At this time, the length of the punctuation-type special character can be determined as the set fixed length.

[0171] If the sum of the number of characters and the number of special characters is not less than the preset control threshold, it is considered that the length of the punctuation-type special character in the recognition text is equal to the character length. For example, the control threshold is 6. For the recognition text "Education.Summer vacation.Learning", this text includes a total of six characters "Jiao, Yu, Shu, Jia, Xue, Xi" and two punctuation marks "." The sum of the two types of characters is 8, which is greater than the control threshold. Therefore, the length of the punctuation-type special character "." in it is equal to the character length, that is, H0.

[0172] In some embodiments, if the special character is a character in another language, such as Japanese and Russian, etc., it can be considered that the length of each special character is fixed, and there is a fixed ratio b with the character length, that is, the length of this type of special character is b*H0.

[0173] In some embodiments, if a special character is detected and it does not have a set fixed length. For example, for unconventional punctuation marks, which do not have a set fixed length, it is considered that its length is equal to the character length H0.

[0174] In some embodiments, for a Chinese interface, all characters in other languages are special characters.

[0175] For an interface in another language, it can be considered that all characters in languages other than that language are special characters. For example, for an English interface, it is considered that Chinese characters are special characters, and the length of all Chinese characters can be set to a fixed value.

[0176] In some embodiments, when determining the X coordinate of the lower left vertex of each character, the controller can calculate the total character length and the total empty character length from the first character to each character. At the same time, it can also calculate the total special character length from the first character to each character.

[0177] When calculating the total length of special characters, the controller can determine the number of numeric special characters and all punctuation special characters and English special characters between the first character and each character, and further calculate the total length of numeric special characters, the total length of punctuation special characters and the total length of English special characters.

[0178] Based on all the lengths, we can calculate the length from the first character to each character and determine the X coordinate of the lower left corner of each character:

[0179] xi=x0+L

[0180] Where L is the length from the first character to the i-th character.

[0181] After determining the X coordinate of the lower left corner vertex of each character, the position information of each character can be obtained, and the position information of the control keyword in the user interface can be determined.

[0182] In some embodiments, after the position information of the control keyword in the user interface is obtained, the control at the position information can be triggered, and the control at the position information is the target control in the user voice.

[0183] At the same time, the controller can display the interface corresponding to the target control in the real-time display.

[0184] In some embodiments, the control triggering instruction input by the user may not contain the control keyword in the current user interface. For example, the user may have spoken an instruction without the control keyword, or the control keyword may not exist in the user interface currently displayed on the display.

[0185] After word segmentation of the control trigger text, the controller can parse the word segmentation result and extract keywords from the word segmentation result. If the control keyword does not exist in the extracted keywords, the controller cannot trigger any control.

[0186] Alternatively, after the controller determines the control keyword, when matching the control keyword with all the recognized texts in the screenshot image, if no recognized text containing the control keyword is matched, that is, when there is no control keyword recognized text, the controller cannot trigger any control.

[0187] At this time, the display may be controlled to display a preset prompt message, where the prompt message is used to prompt the user that no relevant controls have been searched.

[0188] In some embodiments, the prompt message may use a preset prompt template. For example, the prompt message may be set to "Sorry, no relevant controls were found, please re-confirm." The prompt template may also include relevant media asset names, for example: Sorry, no control "A" was found, please continue to search for other controls. Fig.11 A schematic diagram showing prompt information displayed on a display in some embodiments is shown.

[0189] In some embodiments, the controller may also convert the prompt information into a voice response, and then play the voice response to notify the user.

[0190] The present application also provides a control triggering method, which is applied to a display device, such as Fig.12 As shown, the method includes:

[0191] Step 1201: In response to a control trigger instruction input by a user, determine a control keyword in the control trigger instruction;

[0192] Step 1202: Determine the position information of the control keyword in the user interface;

[0193] Step 1203: trigger the control at the location information.

[0194] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0195] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0197] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; A sound collector is configured to receive a voice command input by a user; The controller is configured as: In response to a control trigger instruction input by a user voice, determining a control keyword in the control trigger instruction; Obtaining the control keyword recognition text and recognition text position information of the control keyword; Determine the number of characters and the number of null characters between the first character and each character in the control keyword recognition text, and obtain the character length and null character length; Calculate the first character length between the first character and each character according to the character length and the number of characters; Calculate the first null character length between the first character and each character according to the null character length and the number of null characters; Calculate the length from the first character to each character according to the first character length and the first null character length; Determine the position information of each character according to the length from the first character to each character and the recognized text position information; Determining the position information of the control keyword in the user interface according to the position information of each character; Trigger the control at the position information.

2. The display device according to claim 1, characterized in that The controller is further configured to: In the step of determining the control keyword in the control trigger instruction, Converting the control trigger instruction into a control trigger text; The control trigger text is segmented to obtain control keywords.

3. The display device according to claim 1, characterized in that The controller is further configured to: After executing the step of determining the control keyword in the control trigger instruction, Taking a screenshot of the user interface to obtain a screenshot image; Perform text recognition processing on the screenshot image to obtain image recognition information.

4. The display device according to claim 3, characterized in that The controller is further configured to: After executing the step of obtaining image recognition information, Determining a plurality of text recognition information included in the image recognition information; Determine the control keyword identification information corresponding to the control keyword in the plurality of text identification information.

5. The display device according to claim 4, characterized in that The control keyword identification information includes control keyword identification text and identification text position information; The controller is further configured to: After executing the step of determining the control keyword identification information corresponding to the control keyword in the plurality of text identification information, Detecting the control keyword identification information; When it is detected that the control keyword identification information also includes character information, determining the position information of each character of the control keyword according to the character information; the character information includes the position information of each character of the control keyword identification text; When it is detected that the control keyword identification information does not include character information, the character information is determined according to the control keyword identification information; and the position information of the control keyword in the user interface is determined according to the position information of each character of the control keyword.

6. The display device according to claim 5, characterized in that The controller is further configured to: In the step of obtaining the null character length, Determine the length of the control keyword recognition text according to the recognition text position information; Determine the total number of characters and the total number of blank characters in the control keyword recognition text, and calculate the total length of characters in the control keyword recognition text according to the total number of characters and the character length; Calculate the difference between the length of the control keyword identification text and the total length of the characters; The ratio of the difference to the total number of null characters is calculated to obtain the null character length.

7. The display device according to claim 5, characterized in that The controller is further configured to: In the step of calculating the length of each character from the first character according to the first character length and the first null character length, Get the length of the character interval; Calculate the sum of the number of characters and the number of blank characters to obtain a first number; Calculate the total length of the character interval from the first character to each character according to the character interval length and the first number; The sum of the first character length, the first empty character length and the total length of the character interval is calculated to obtain the length from the first character to each character.

8. A control triggering method, applied to a display device, characterized in that: The method comprises: In response to a control trigger instruction input by a user voice, determining a control keyword in the control trigger instruction; Obtaining the control keyword recognition text and recognition text position information of the control keyword; Determine the number of characters and the number of null characters between the first character and each character in the control keyword recognition text, and obtain the character length and null character length; Calculate the first character length between the first character and each character according to the character length and the number of characters; calculate the first null character length between the first character and each character according to the null character length and the number of null characters; Calculate the length from the first character to each character according to the first character length and the first null character length; Determine the position information of each character according to the length from the first character to each character and the recognized text position information; Determining the position information of the control keyword in the user interface according to the position information of each character; Trigger the control at the position information.

9. The control triggering method according to claim 8, characterized in that: After determining the control keyword in the control trigger instruction, the method further includes: Taking a screenshot of the user interface to obtain a screenshot image; Perform text recognition processing on the screenshot image to obtain image recognition information; Determining a plurality of text recognition information included in the image recognition information; Determine the control keyword identification information corresponding to the control keyword in a plurality of text identification information, wherein the control keyword identification information includes control keyword identification text and identification text position information.

Images