Method and system for controlling display equipment based on action recognition, display equipment and storage medium

Abstract

The invention discloses a method and system for controlling a display device based on action recognition, the display device and a storage medium, and relates to the technical field of man-machine interaction, and the method comprises the steps: S100, switching a window focus to a target window according to obtained user action information; and step S200, determining and executing a predefined operation according to the user action information and the application scene in the target window so as to control the target window. The operation problem that the remote controller manually switches the window focus in the multi-window state of the display equipment is solved, different window applications are controlled through different body part actions, the window focus is automatically switched, the user operation steps are simplified through predefined operation, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a method, system, display device, and storage medium for controlling display devices based on motion recognition. Background Technology

[0002] With the maturity and mass production of display technology, the cost of large-size panels has decreased, leading to a continuous increase in the screen size of devices such as mobile phones, televisions, and monitors. Large-size display devices can display more content. To fully utilize the advantages of large-screen displays, multi-window information display methods and application scenarios such as split-screen and picture-in-picture are gradually increasing on large-screen display devices. Taking smart TVs as an example, their average screen size has been increasing in recent years, exceeding 70 inches offline by 2025, and is expected to continue to grow in the future. Smart TV products have a wealth of multi-window application scenarios, such as smart fitness: exercising while following a fitness video and correcting posture through a camera window. Another example is smart gaming: watching game guides in a small window while controlling the game in a large window. Large-screen display devices achieve efficient content display through multi-window methods, but this also increases the complexity of interaction. For devices like smart TVs that are mainly operated with a remote control, to control a specific window, the control focus must first be moved to the corresponding window, and then a remote control command must be issued to the window. This step of switching the control focus has become an operational hurdle for most ordinary users. Multi-window scenarios involve complex interactions and high learning costs for users, resulting in many users only briefly trying them out, finding the interaction inconvenient and the experience poor. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, display device, and storage medium for controlling a display device based on motion recognition, in order to solve the problems in the prior art where switching windows on a display device using a remote control requires switching the control focus, resulting in cumbersome and time-consuming operation, inconvenient interaction, and poor user experience.

[0004] The present invention solves the above problems through the following technical solution:

[0005] A method for controlling a display device based on motion recognition, comprising:

[0006] Step S100: Switch the window focus to the target window based on the acquired user action information;

[0007] Step S200: Determine and execute a predefined operation based on the user action information and the application scenario within the target window to control the target window.

[0008] This invention acquires information about the target user's body parts and body movements through a camera, determines the target control window based on the body part information, and determines the control commands based on the body movement information and the application scenario within the target window, thereby achieving the technical effect of controlling the target window of the display device with the body.

[0009] Further, step S100 specifically includes:

[0010] Step S110: Pre-establish a window mapping table, which is used to store the mapping relationship between user action information and windows;

[0011] Step S120: Obtain user action information, determine the target window based on the obtained user action information and the window mapping relationship table, and automatically switch the window focus to the target window.

[0012] Furthermore, the user action information includes body part information and body part action information, and the window mapping table stores at least two mapping relationships between body part information and windows.

[0013] Further, step S200 specifically includes:

[0014] Step S210: Pre-establish an operation mapping relationship table. The operation mapping relationship table is used to store the mapping relationship between user action information and predefined operations in different application scenarios.

[0015] Step S220: Determine the application scenario within the target control window, and determine the predefined operation based on the user action information and the operation mapping relationship table;

[0016] Step S230: Control the target window according to the predefined operation.

[0017] Furthermore, the user action information includes body part information and body part action information. The operation mapping table stores the mapping relationship between a body part action information and a predefined operation in at least one application scenario. The predefined operation is used to predefine a set of operations for the target window application scenario to achieve the user's expected operation goal.

[0018] A system for controlling a display device based on motion recognition, comprising:

[0019] The target window switching module is used to switch the window focus to the target window based on the acquired user action information;

[0020] The execution module is used to determine and execute predefined operations based on the user action information and the application scenario within the target window, thereby controlling the target window.

[0021] Furthermore, the target window switching module includes:

[0022] The acquisition unit is used to acquire user action information;

[0023] The recognition unit is used to identify body part information and body part movement information based on the user action information acquired by the acquisition unit.

[0024] The control unit is used to determine the target window based on body part information and a pre-established window mapping table, and automatically switch the window focus to the target window. The window mapping table is used to store the mapping relationship between user body part information and windows.

[0025] Furthermore, the execution module is used to determine and execute predefined operations based on user body part action information and a pre-established operation mapping table. The operation mapping table is used to store the mapping relationship between user body part action information and predefined operations in different application scenarios.

[0026] This invention acquires user body part information and body part movement information, determines the target window to be controlled based on a pre-established mapping table between body parts and display device windows, and automatically switches the window focus to the target window; based on a pre-established mapping table of predefined operations for typical scenarios, it determines predefined operations by combining body part movement information with the application scenario information of the target window; and executes the predefined operations for the application scenario within the target window. This solves the problem of manually switching window focus with a remote control in a multi-window state of a display device, enabling different body part movements to control different window applications, automatically switching window focus, and simplifying user operation steps through predefined operations.

[0027] A display device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for controlling the display device based on motion recognition.

[0028] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for controlling a display device based on motion recognition.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] (1) By using a camera and recognition algorithm, body part information is mapped to the corresponding window to realize automatic window focus switching and high-frequency operation of body part action to the corresponding window application scenario, further simplifying the interactive operation.

[0031] (2) The present invention solves the problem that when a remote control operates on a target window in a multi-window state of a display device, it is necessary to switch the focus of the window to the target window before operating the target window.

[0032] (3) The present invention solves the problem of cumbersome steps in high-frequency operations (such as switching TV series, switching music, etc.) in display device application scenarios. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the left and right hand gesture control display device split-screen mode in this invention;

[0034] Figure 2 This is a system principle block diagram of the present invention;

[0035] Figure 3 This is a schematic diagram illustrating the mapping relationship between body part information and display device windows in this invention;

[0036] Figure 4 This is a schematic diagram illustrating the predefined operation mapping relationship of the split-screen mode in the scenario of watching movies and TV series + casting mobile phones in this invention;

[0037] Figure 5 This is a schematic diagram of an embodiment of the present invention, which uses left and right hand gestures to control the left and right windows for watching videos and casting. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto.

[0039] Example 1:

[0040] A method for controlling a display device based on motion recognition, comprising:

[0041] Step S100: Switch the window focus to the target window based on the acquired user action information;

[0042] Step S200: Determine and execute a predefined operation based on the user action information and the application scenario within the target window to control the target window.

[0043] This invention acquires information about the target user's body parts and body movements through a camera, determines the target control window based on the body part information, and determines the control commands based on the body movement information and the application scenario within the target window, thereby achieving the technical effect of controlling the target window of the display device with the body.

[0044] Further, step S100 specifically includes:

[0045] Step S110: Pre-establish a window mapping table, which is used to store the mapping relationship between user action information and windows;

[0046] Step S120: Obtain user action information, determine the target window based on the obtained user action information and the window mapping relationship table, and automatically switch the window focus to the target window.

[0047] Furthermore, the user action information includes body part information and body part action information, and the window mapping table stores at least two mapping relationships between body part information and windows.

[0048] Further, step S200 specifically includes:

[0049] Step S210: Pre-establish an operation mapping relationship table. The operation mapping relationship table is used to store the mapping relationship between user action information and predefined operations in different application scenarios.

[0050] Step S220: Determine the application scenario within the target control window, and determine the predefined operation based on the user action information and the operation mapping relationship table;

[0051] Step S230: Control the target window according to the predefined operation.

[0052] Furthermore, the user action information includes body part information and body part action information. The operation mapping table stores the mapping relationship between a body part action information and a predefined operation in at least one application scenario. The predefined operation is used to predefine a set of operations for the target window application scenario to achieve the user's expected operation goal.

[0053] Example 2:

[0054] Combined with appendix Figure 2 As shown, a system for controlling a display device based on motion recognition includes:

[0055] The target window switching module is used to switch the window focus to the target window based on the acquired user action information;

[0056] The execution module is used to determine and execute predefined operations based on the user action information and the application scenario within the target window, thereby controlling the target window.

[0057] Furthermore, the target window switching module includes:

[0058] The acquisition unit is used to acquire user action information;

[0059] The recognition unit is used to identify body part information and body part movement information based on the user action information acquired by the acquisition unit.

[0060] The control unit is used to determine the target window based on body part information and a pre-established window mapping table, and automatically switch the window focus to the target window. The window mapping table is used to store the mapping relationship between user body part information and windows.

[0061] Furthermore, the execution module is used to determine and execute predefined operations based on user body part action information and a pre-established operation mapping table. The operation mapping table is used to store the mapping relationship between user body part action information and predefined operations in different application scenarios.

[0062] Combined with appendix Figure 1 As shown, the display device is in a dual-window split-screen mode. The left window contains application scenario A, and the right window contains application scenario B. Window focus indicates that the current window is in focus state. The window in focus state will respond to operation commands sent by the system. It also includes a camera connected to the display device, which is responsible for acquiring the target user's body image information. The body image includes left hand gesture information and right hand gesture information.

[0063] like Figure 2 As shown, the system modules of the body movement control display device are illustrated: a target window switching module and an execution module. The target window switching module includes an acquisition unit, which is responsible for acquiring image information containing target user movement information from a camera; a recognition unit, which is responsible for recognizing body part information and body part movement information from the image information acquired by the acquisition unit; a control unit, which determines the target window to be controlled according to a pre-established mapping table between body part information and display device windows, and automatically switches the window focus to the target window; and an execution module, which determines and executes predefined operations according to a pre-established mapping table of predefined operations for typical scenarios.

[0064] like Figure 3 As shown, it illustrates the mapping relationship between body part information for manipulating left and right windows and display device windows, including body part information (specifically, the left and right hands) and display device window information (specifically, the left and right windows). The left hand is mapped to the left window, indicating that the left hand is manipulating the left window, and the right hand is mapped to the right window, indicating that the right hand is manipulating the right window.

[0065] like Figure 4As shown, this diagram illustrates the mapping relationship between hand gestures (up and down) and predefined operations in a movie / TV show + screen mirroring scenario. It includes hand gesture information (specifically, hand gestures up and down); typical scenario information (specifically, movie / TV show and screen mirroring scenarios); and predefined operation information (specifically, predefined operations such as "previous episode," "next episode," "swipe up," and "swipe down." Specifically, when the scenario is movie / TV show watching, hand gestures up and down are predefined as "previous episode" and "next episode," while when the scenario is screen mirroring, hand gestures up and down are predefined as "swipe up" and "swipe down."

[0066] like Figure 5 As shown, this illustrates a split-screen mode with the display device in multi-window mode. The left hand controls the previous / next episode of the video-watching scene in the left window, while the right hand controls the swipe up and down of the screen-casting scene in the right window. The left window, running a video-watching application, has predefined mappings for swiping up and down to switch between episodes. Similarly, the right window, running a screen-casting application, has predefined mappings for swiping up and down. Specifically, a mapping table between body part information and display device windows defines the mappings between left-hand gestures and right-hand gestures.

[0067] When the target user makes an upward gesture with their left hand, the acquisition unit obtains an image containing the user's hand information through the camera. The recognition unit identifies the user's body part as the left hand and the body part's movement information as upward. The control unit switches the window focus in the multi-window mode to the left window based on the mapping relationship between the left hand and the left window. The execution module maps the hand movement to the previous episode's instruction in the left window's movie-watching scenario and executes the previous episode's instruction.

[0068] When the target user makes a downward gesture with their right hand, the acquisition unit obtains an image containing the user's hand information through the camera. The recognition unit identifies the user's body part as the right hand and the body part's movement information as downward. The control unit switches the window focus in the multi-window mode to the right window based on the mapping relationship between the right hand and the right window. The execution module maps the downward hand movement to a downward swipe command based on the right window's mobile phone screen projection scenario and executes the downward swipe command.

[0069] This invention acquires user body part information and body part movement information. Based on a pre-established mapping table between body parts and display device windows, it determines the target window to be controlled and automatically switches the window focus to the target window. Based on a pre-established mapping table of predefined operations for typical scenarios, it determines predefined operations by combining body part movement information with the target window's application scenario information. For the application scenario within the target window, the predefined operation is executed. This solves the problem of manually switching window focus with a remote control in a multi-window display device state, enabling different body part movements to control different window applications, automatically switching window focus, and simplifying user operation steps through predefined operations. It also solves the operational obstacle of needing to switch the window focus to the target window before operating it in a multi-window display device state, and addresses the cumbersome steps of high-frequency operations (such as switching TV series or music) in display device application scenarios.

[0070] Example 3:

[0071] A display device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for controlling the display device based on motion recognition as described in Embodiment 1.

[0072] Example 4:

[0073] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for controlling a display device based on motion recognition as described in Embodiment 1.

[0074] The present invention has been described herein with reference to illustrative embodiments. The above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for controlling a display device based on motion recognition, characterized in that, include: Step S100: Switch the window focus to the target window based on the acquired user action information; Step S200: Determine and execute a predefined operation based on the user action information and the application scenario within the target window to control the target window.

2. The method for controlling a display device based on motion recognition according to claim 1, characterized in that, Step S100 specifically includes: Step S110: Pre-establish a window mapping table, which is used to store the mapping relationship between user action information and windows; Step S120: Obtain user action information, determine the target window based on the obtained user action information and the window mapping relationship table, and automatically switch the window focus to the target window.

3. The method for controlling a display device based on motion recognition according to claim 2, characterized in that, The user action information includes body part information and body part action information, and the window mapping table stores at least two mapping relationships between body part information and windows.

4. The method for controlling a display device based on motion recognition according to claim 1, characterized in that, Step S200 specifically includes: Step S210: Pre-establish an operation mapping relationship table. The operation mapping relationship table is used to store the mapping relationship between user action information and predefined operations in different application scenarios. Step S220: Determine the application scenario within the target control window, and determine the predefined operation based on the user action information and the operation mapping relationship table; Step S230: Control the target window according to the predefined operation.

5. The method for controlling a display device based on motion recognition according to claim 4, characterized in that, The user action information includes body part information and body part action information. The operation mapping table stores the mapping relationship between a body part action information and a predefined operation in at least one application scenario.

6. A system for controlling a display device based on motion recognition, characterized in that, include: The target window switching module is used to switch the window focus to the target window based on the acquired user action information; The execution module is used to determine and execute predefined operations based on the user action information and the application scenario within the target window, thereby controlling the target window.

7. The system for controlling a display device based on motion recognition according to claim 6, characterized in that, The target window switching module includes: The acquisition unit is used to acquire user action information; The recognition unit is used to identify body part information and body part movement information based on the user action information acquired by the acquisition unit. The control unit is used to determine the target window based on body part information and a pre-established window mapping table, and automatically switch the window focus to the target window. The window mapping table is used to store the mapping relationship between user body part information and windows.

8. The system for controlling a display device based on motion recognition according to claim 7, characterized in that, The execution module is used to determine and execute predefined operations based on user body part action information and a pre-established operation mapping table. The operation mapping table is used to store the mapping relationship between user body part action information and predefined operations in different application scenarios.

9. A display device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for controlling the display device based on motion recognition as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for controlling a display device based on motion recognition as described in any one of claims 1-5.

Images