Method for controlling display equipment based on gestures

Abstract

The invention mainly discloses a method for controlling display equipment based on gestures. The method comprises the following steps: acquiring skeleton node information of an operator and skeleton node information of a manipulator in an image; according to the skeleton node information of the manipulator, generating a feature vector for describing the hand posture of the manipulator, matching the feature vector for describing the hand posture of the manipulator with a gesture model in a gesture model library, and dividing the image into a gesture category frame image and a gesture-free category frame image according to a matching result, obtaining gesture category information in the frame image with the gesture category, and obtaining gesture category information and hand position information in the frame image without the gesture category according to skeleton node information of the operator and gesture category information of front and back frames of the frame image without the gesture category; and generating an instruction for controlling the display equipment according to the gesture category information in the gesture category frame image and the gesture-free category frame image and the position information of the hand.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a method for controlling a display device based on gestures. Background Art

[0002] With the rapid development of information technology, distributed display devices have been widely used in public places, commercial displays, home entertainment and other fields. However, traditional display device control methods such as physical buttons and remote controls are cumbersome and not intuitive enough. In recent years, although control methods based on visual recognition have emerged, sometimes when the gesture has been completed, the triggering object has not stopped the action, and judgment delays or misjudgments often occur. Its accuracy and robustness still need to be improved. Summary of the invention

[0003] To solve at least one of the aforementioned technical problems, the present disclosure proposes, in a first aspect, a method for controlling a display device based on gestures, comprising: obtaining skeletal node information of a manipulator and skeletal node information of a manipulating hand in an image; generating a feature vector describing the hand posture of the manipulator based on the skeletal node information of the manipulating hand, matching the feature vector describing the hand posture of the manipulator with a gesture model in a gesture model library, dividing the image into a frame image with a gesture category and a frame image without a gesture category according to the matching result, and obtaining gesture category information in the frame image with a gesture category; obtaining gesture category information and hand position information in the frame image without a gesture category based on the skeletal node information of the manipulator and the gesture category information of the previous and next frames of the frame image without a gesture category; generating an instruction to control the display device based on the gesture category information and hand position information of the frame image with a gesture category and the gesture category information and hand position information of the frame image without a gesture category.

[0004] Preferably, it also includes: capturing the real-time skeleton node information of the operator through a depth camera, and the skeleton node information includes the three-dimensional coordinates of the joints and the connection relationship between the joints.

[0005] Preferably, the skeleton node information of the operator in the frame images with gestures in the previous and next frames of the frame images without gestures is interpolated to calculate the skeleton node information of the operator's hand in the frame images without gestures.

[0006] Preferably, the hand skeleton node information of the operator in the frame image without gesture category is matched with the gesture model in the gesture model library to obtain the gesture category of the operator in the frame image without gesture category.

[0007] Preferably, the wrist node position of the operator in the no-gesture category frame image is obtained based on the operator's skeletal node information and the operator's hand's skeletal node information; and the hand position information of the operator in the no-gesture category frame image is obtained based on the operator's wrist node position in the no-gesture category frame image.

[0008] Preferably, the gesture category information in the gesture category frame images of the preceding and following frames of the frame image without gesture category is compared. If the gesture category information of the preceding and following frames belong to the same gesture category, it is determined that the gesture information in the frame image without gesture category is also the same as the gesture category information of the preceding and following frames.

[0009] Preferably, if the gesture category information of the previous and next frames respectively belongs to two consecutive different gesture categories, it is determined that the gesture category information in the frame image without gesture category is the same as the gesture category information of the previous frame or the gesture category information of the next frame.

[0010] Preferably, the gestures of the frame images with gesture category and the frame images without gesture category are classified into first-level response gesture one and second-level response gesture two, and the displacement change data of gesture one and gesture two of the same manipulating hand of the manipulator between frames are obtained; when the displacement change data of gesture one reaches the condition for triggering gesture two, it is determined whether the displacement change data of gesture two reaches the condition for triggering menu display, and if so, an instruction to control the display device is generated.

[0011] In a second aspect, the present disclosure provides a device for controlling a display device based on gestures, comprising: an acquisition unit, configured to acquire skeletal node information of a manipulator and skeletal node information of a manipulating hand in an image; A matching unit is used to generate a feature vector describing the hand posture of the operator according to the skeletal node information of the operator's hand, match the feature vector describing the hand posture of the operator with the gesture model in the gesture model library, and divide the image into a frame image with gesture category and a frame image without gesture category according to the matching result, so as to obtain the gesture category information in the frame image with gesture category; a judging unit is used to obtain the gesture category information and the hand position information in the frame image without gesture category according to the skeletal node information of the operator and the gesture category information of the previous and next frames of the frame image without gesture category; a generating unit is used to generate an instruction to control the display device according to the gesture category information and the hand position information of the frame image with gesture category and the gesture category information and the hand position information of the frame image without gesture category.

[0012] The present disclosure proposes a computer-readable medium in a third aspect, wherein a computer program is stored in the computer-readable medium. The computer program is loaded and executed by a processing module to implement the steps of any of the above-mentioned methods.

[0013] Some technical effects of the present disclosure are: a method for controlling a display device based on gestures, comprising: obtaining the skeleton node information of a manipulator and the skeleton node information of a manipulator's hand in an image; generating a feature vector describing the hand posture of the manipulator according to the skeleton node information of the manipulator's hand, matching the feature vector describing the hand posture of the manipulator with the gesture model in the gesture model library, dividing the image into a frame image with gesture category and a frame image without gesture category according to the matching result, and obtaining the gesture category information in the frame image with gesture category; obtaining the gesture category information and the position information of the hand in the frame image without gesture category according to the skeleton node information of the manipulator and the gesture category information of the previous and next frames of the frame image without gesture category; generating an instruction to control the display device according to the gesture category information and the position information of the hand in the frame image with gesture category and the gesture category information and the position information of the hand in the frame image without gesture category. By combining the gesture posture and the position information of the hand in the frame image without gesture category, the display device responds to the operation corresponding to the gesture posture more accurately and quickly. Response delay is avoided. Finally, the distributed display device can be controlled quickly and accurately. False triggering of the distributed display device or delayed response of the display device due to delay or misjudgment of the control gesture is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To better understand the technical solution of the present disclosure, you may refer to the following drawings for auxiliary explanation of the prior art or embodiments. These drawings will selectively display the products or methods involved in the prior art or some embodiments of the present disclosure. The basic information of these drawings is as follows: Figure 1 The present invention is a flowchart of an embodiment of a method for controlling a display device based on gestures disclosed herein. DETAILED DESCRIPTION

[0015] The technical means or technical effects involved in the present disclosure will be further described below. Obviously, the embodiments (or implementation methods) provided are only some of the implementation methods covered by the present disclosure, but not all of them. Based on the embodiments in the present disclosure and the explicit or implicit descriptions of the figures and texts, all other embodiments that can be obtained by those skilled in the art without making creative efforts will be within the scope of protection requested by the present disclosure.

[0016] In existing methods for controlling large screens with gestures, the performance of computer vision algorithms may be affected by conditions such as lighting and the speed of hand movements. Therefore, when capturing images, some gestures may be blurred in the gesture image or the arm may be waving too fast, which may also cause the gestures to be blurred in the image. This will cause some gestures to control large screens with delays and false touches. For example, when the application action is to control the page turning with a gesture of waving to the right, sometimes after waving to the right a little faster, the display device does not respond to the page turning action or turns the page after a while. There will be some misjudgments or response delays.

[0017] In the current gesture recognition process, if the above-mentioned blurred frames without gesture categories appear, these frames without gesture categories will be discarded. This may lead to information loss in the continuous gesture recognition process, thus affecting the accuracy and timeliness of gesture recognition.

[0018] In this regard, the embodiment of the present disclosure provides a method for controlling a display device based on gestures, which can be applied to various industries and other scenarios where it is necessary to achieve fast and accurate human-computer interaction by controlling the large screen of the display device, such as emergency command and dispatch center, public security command and dispatch center, traffic command and dispatch center, energy command and dispatch center, smart city command and dispatch center and other fields, by controlling the large screen to control the dispatch system, such as switching distributed signal sources, taking over the mouse in the signal source, and thus performing any operation on the content in the signal source. As the central brain of command and dispatch control, the command center plays an extremely important role in social governance and people's livelihood development, and has high requirements for the accuracy and speed of control. In this regard, the method of controlling a display device based on gestures in the embodiment of the present disclosure does not require the use of any complex control equipment or wearable sensors, and can quickly and accurately complete the takeover and control of the control of the large screen of the command center by only using the behavior recognition of living people, and efficiently realizes the fast and accurate interactive operation control of people and large screen content such as signal on-screen, switching, zooming, etc. through simple gesture operations in the air. Of course, the method of the present disclosure can also be applied to the above-mentioned general application scenarios with low requirements, such as turning pages of PPT.

[0019] The exemplary system architecture of the device based on gesture control of display device disclosed in the present invention can be applied. The system architecture may include a camera with a pan / tilt, a server, and a display device. Exemplarily, the display device is a large screen, and the camera with a pan / tilt is connected to the server via a serial cable and a USB cable, and the server is then connected to the display device via a network cable. The image information captured by the camera is transmitted to the server via a USB cable, and the server generates information or commands after processing, analyzing, and making decisions on the received image information. The server can also send the information to the distributed scheduling and image comprehensive management platform through the network port. The distributed scheduling and image comprehensive management platform receives the information and displays the corresponding operation feedback on the large screen.

[0020] Display devices generally require large screen, multi-color, high brightness, and high resolution display effects. Exemplarily, the display device is a large screen display device, which refers to a large screen in a direct-view color TV or a rear projection TV. Usually, the diagonal size of the screen is more than 40 inches. The display surface of the large screen display device can be flat or curved. The large screen display device can also be spliced, without limitation.

[0021] In this embodiment, the camera with a pan-tilt head is located directly above the large screen. The pan-tilt head is a device that carries the camera.

[0022] The target user can use gestures to interact with the server through the camera, and then the server and the large screen can interact with each other, thereby realizing gesture control of the large screen. Among them, the server can be a single server, or a server cluster or cloud computing center composed of several servers. The server can provide various services for the display device. For different applications on the display device, the server can be considered as a background server that provides corresponding network services. It can be considered that the method disclosed in this application is mainly executed by the server side.

[0023] The application scenario under one embodiment of the present disclosure includes a large screen, a camera with a pan / tilt system located above the large screen, and an operable range located in front of the large screen. The operable range is roughly a ring-shaped area. The operator can operate the large screen within the operating range. If it is not within the operable range, for example, if the distance to the large screen is too far, the gesture recognition may be wrong, which may cause an operation error. If the distance to the large screen is too close, the operator may not be able to observe the entire content of the large screen, which is not conducive to operating the large screen. In this embodiment, the width of the large screen is 10 meters, and the operable range in this embodiment is a ring range of 3 to 12 meters from the large screen.

[0024] like Figure 1 , which shows that a method of controlling a display device based on gestures disclosed in the present invention includes: S10: Obtaining the skeleton node information of the operator and the skeleton node information of the operator's hand in the image; S20: generating a feature vector describing the hand posture of the operator according to the skeleton node information of the operator's hand, matching the feature vector describing the hand posture of the operator with the gesture model in the gesture model library, and dividing the image into a frame image with gesture category and a frame image without gesture category according to the matching result, and obtaining gesture category information in the frame image with gesture category; S30: acquiring gesture category information and hand position information in the non-gesture frame image according to the operator's skeleton node information and gesture category information of the frames before and after the non-gesture frame image; S40: Generate an instruction to control the display device according to the gesture category information and the hand position information of the frame image with gesture category and the gesture category information and the hand position information of the frame image without gesture category.

[0025] S10: Obtaining the skeleton node information of the operator and the skeleton node information of the operator's hand in the image; The real-time skeletal node information of the operator is captured by a depth camera, and the skeletal node information includes the three-dimensional coordinates of the joints and the connection relationship between the joints. The image including the above-mentioned operable range can be collected by the depth camera, and the skeletal node information of the operator and the skeletal node information of the manipulator's hand in the image can be obtained by bone recognition of the image collected by the camera. Including the position of each joint point. The captured skeletal node data is preprocessed, including denoising, smoothing, etc., to improve the accuracy and stability of the data. The image within the operable range, that is, the preset range obtained by the above-mentioned camera is preprocessed, such as graying, denoising, edge detection, etc., to improve the efficiency and accuracy of subsequent processing. Use any computer vision algorithm to detect the skeletal nodes in the image, again without restriction. These algorithms are generally able to identify and track multiple key skeletal nodes of the human body, including the head, limbs, etc., and the limbs include wrist joints. As needed, the skeletal node information can be converted from two-dimensional image coordinates to three-dimensional space coordinates. The three-dimensional skeletal node information captured by the depth camera is fused with the two-dimensional skeletal node information detected by the computer vision algorithm. In order to provide more complete and accurate skeletal node information of the operator, especially the skeletal node information of the hand.

[0026] The collected images are preprocessed, and images of a preset range are collected using cameras and other image acquisition devices to obtain images containing the operator and his hands. The images are then processed by denoising and sharpening to improve image quality for subsequent gesture detection. The hand area is then detected, and the hand area is extracted from the image using image processing techniques such as color space conversion, edge detection, and morphological processing. In the hand area, key points of the hand, such as fingertips, wrists, joints, etc., are detected using deep learning algorithms.

[0027] S20: Generate a feature vector describing the hand posture of the operator based on the skeletal node information of the operator's hand, match the feature vector describing the hand posture of the operator with the gesture model in the gesture model library, and divide the image into a frame image with gesture category and a frame image without gesture category according to the matching result, and obtain the gesture category information in the frame image with gesture category.

[0028] The user performs gestures according to the preset gesture specifications. Gestures include specific shapes, speeds, directions, etc. Based on the skeletal node information of the manipulator's hand, a feature vector describing the manipulator's hand posture is generated. Feature vector generation: Based on the position information of the key points, a feature vector describing the hand posture is generated. These feature vectors may include the coordinates, angles, distances, etc. of the key points. Match the feature vector describing the manipulator's hand posture with the gesture model in the gesture model library. Gesture matching and classification: Establish a gesture model library: In the training phase, the gesture model library is obtained by training with deep learning algorithms (such as support vector machines and neural networks) using known gesture images and labels. These models contain feature vectors and corresponding labels for various gestures. Gesture matching: Match the extracted feature vectors with the models in the gesture model library. This can be achieved by calculating the distance or similarity between feature vectors. Gesture classification: Determine the type of gesture in the image based on the matching results. Usually, the model with the smallest distance or the highest similarity is selected as the recognition result.

[0029] According to the matching results, the images are divided into frames with gestures and frames without gestures, and the gesture category information in the frames with gestures is obtained. The distance data is processed by an algorithm to extract the characteristic information of the gesture, such as the shape, speed, direction, etc. of the gesture. Based on the extracted gesture feature information, a classifier (such as a support vector machine, a neural network, etc.) is used to recognize the gesture. The classifier will determine whether the current gesture matches a preset gesture based on the preset gesture template or the gesture samples in the database.

[0030] Based on gesture recognition, the actual measured distance is compared with the preset distance threshold or range to determine whether the gesture is completed. If the actual measured distance meets the preset threshold or range, the gesture is considered to be completed and the corresponding response action is triggered.

[0031] In a gesture recognition system, in order to accurately recognize and respond to user gestures, the system usually needs to process the video stream or image sequence first to segment the hand area and count the fingers or identify the gesture type. However, not all frames contain valid gestures. Some frames may just show the user standing still, making gestures unrelated to the gesture, or the hand does not appear in the picture. These frames are called "non-gesture frames".

[0032] Non-gesture frames are frames in video streams or image sequences that do not contain gestures or have unclear gesture features. These frames usually only contain background information, other non-gesture objects, or images of other parts of the human body, but do not contain the complete form or motion trajectory of the gesture. Valid gesture frames are frames in video streams or image sequences that contain clear, complete gestures that meet the requirements of specific gesture recognition algorithms. These frames have clear gesture forms, motion trajectories, and feature points, and can be accurately recognized by gesture recognition algorithms.

[0033] Extract the position information of key nodes such as the wrist from the video or image sequence. Analyze the motion trajectory and relative position changes of these nodes in the previous and next frames. Compare and match the extracted features with the features in the known gesture pattern library. Pattern matching can be performed using machine learning algorithms (such as support vector machines, random forests, etc.) or deep learning models (such as convolutional neural networks, recurrent neural networks, etc.).

[0034] For situations where you need to capture the dynamic characteristics of gestures, you can use motion sensors (such as gyroscopes, accelerometers) and other devices to assist in the analysis. By calculating the node's motion speed, acceleration and other parameters, you can further determine the type and intention of the gesture. Combined with the gesture information of the previous and next frames, use context information to assist in determining the gesture type of the current non-gesture frame. For example, if the previous frame is the start frame of a gesture and the next frame is the end frame of the gesture, then determine that the current non-gesture frame is a transition frame of the gesture.

[0035] S30: Acquire gesture category information and hand position information in the non-gesture frame image according to the skeleton node information of the operator and the gesture category information of the frames before and after the non-gesture frame image.

[0036] For frames with missing hand position information, the hand position can be estimated by interpolation or prediction between frames. The skeleton node information of the operator in the gesture category frame images before and after the gesture category frame image is interpolated to calculate the hand skeleton node information of the operator in the gesture category frame image. The hand skeleton node information of the operator in the gesture category frame image is matched with the gesture model in the gesture model library to obtain the gesture category of the operator in the gesture category frame image.

[0037] For frames that lack hand position information, the wrist node position of the operator in the frame image without gesture category can be obtained based on the skeleton node information of the operator and the skeleton node information of the operating hand; and the hand position information of the operator in the frame image without gesture category can be obtained based on the wrist node position of the operator in the frame image without gesture category.

[0038] Compare the gesture category information in the gesture category frame images of the previous and next frames of the non-gesture category frame image. If the gesture category information of the previous and next frames belong to the same gesture category, then it is determined that the gesture information in the non-gesture category frame image is also the same as the gesture category information of the previous and next frames. If the gesture category information of the previous and next frames respectively belong to two consecutive different gesture categories, then it is determined that the gesture category information in the non-gesture category frame image is the same as the gesture category information of the previous frame or the gesture category information of the next frame.

[0039] In a video or image sequence, the position information of key nodes such as the wrist is continuously tracked. Algorithms such as optical flow and Kalman filtering can be used to optimize the tracking accuracy of node positions. For non-gesture frames, the position information of key nodes can be calculated by interpolation algorithms. Common interpolation algorithms include linear interpolation and polynomial interpolation. Combined with the position information of the previous and next frames, the position information of the non-gesture frames is corrected and optimized. Smoothing filtering and denoising algorithms can be used to improve the accuracy and stability of position information. Gesture judgment and position information calculation for non-gesture frames are realized. Video or image sequences can be processed in real time, and gesture recognition results and position information can be output.

[0040] S40: Generate an instruction to control the display device according to the gesture category information and the hand position information in the frame image with gesture category and the frame image without gesture category.

[0041] The gestures of the frame images with gesture category and the frame images without gesture category are classified into the first-level response gesture one and the second-level response gesture two, and the displacement change data of gesture one and gesture two of the same manipulating hand of the manipulator between frames are obtained; when the displacement change data of gesture one reaches the condition for triggering gesture two, it is determined whether the displacement change data of gesture two reaches the condition for triggering menu display, and if so, an instruction to control the display device is generated.

[0042] Use a camera (such as a color camera or a depth camera) to capture images of the hand or gesture in real time. Calculate the distance between the hand or gesture and the camera through an image processing algorithm (such as a depth map algorithm). For a depth camera, the depth information can be obtained directly; for a color camera, a specific algorithm (such as stereo vision, structured light, etc.) is required to estimate the depth. Set the corresponding distance threshold based on the expected completion of the gesture. Calculate the distance between the hand or gesture and the camera in real time and compare it with the set threshold. If the distance data is within the set threshold range, it is considered that the gesture is proceeding as expected. If the distance data exceeds the set threshold range, it is considered that the gesture is not completed as expected or there is an error.

[0043] In the second aspect, the present disclosure proposes a device for controlling a display device based on gestures, which is characterized by comprising: an acquisition unit for acquiring the skeleton node information of the operator and the skeleton node information of the operating hand in the image; a matching unit for generating a feature vector describing the operator's hand posture according to the skeleton node information of the operating hand, matching the feature vector describing the operator's hand posture with the gesture model in the gesture model library, and dividing the image into a frame image with gesture category and a frame image without gesture category according to the matching result, and obtaining the gesture category information in the frame image with gesture category. A judgment unit for acquiring the gesture category information and the position information of the hand in the frame image without gesture category according to the skeleton node information of the operator and the gesture category information of the previous and next frames of the frame image without gesture category; a generation unit for generating instructions for controlling the display device according to the gesture category information and the position information of the hand in the frame image with gesture category and the frame image without gesture category.

[0044] The present disclosure proposes a computer-readable medium in a third aspect, wherein a computer program is stored in the computer-readable medium, and the computer program is loaded and executed by a processing module to implement the steps of the acquisition method. It can be understood by those skilled in the art that all or part of the steps in the embodiment can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable medium, and the readable medium can include various media that can store program codes, such as a flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0045] Within the scope of the knowledge and ability level of those skilled in the art, the various embodiments or technical features mentioned herein may be combined with each other as other optional embodiments without conflict. These limited number of optional embodiments, which are not listed one by one and are formed by combining a limited number of technical features, still fall within the technical scope disclosed in the present disclosure and can be understood or inferred by those skilled in the art in combination with the drawings and the above text.

[0046] In addition, the description of most embodiments is based on different focuses. If you need to further understand what is not described in detail, you can refer to the relevant content of the prior art, other relevant descriptions in this article or the purpose of the invention for reasonable reasoning.

[0047] It is emphasized again that the embodiments listed above are typical and preferred embodiments of the present disclosure, which are only used to explain and interpret the technical solutions of the present disclosure in detail to facilitate the understanding of the readers, and are not used to limit the scope or application of the present disclosure. Any technical solutions obtained by modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the present disclosure.

Claims

1. A method for controlling a display device based on gestures, characterized in that: include: Obtain the skeleton node information of the operator and the skeleton node information of the operator's hand in the image; According to the skeletal node information of the manipulator's hand, a feature vector describing the manipulator's hand posture is generated, and the feature vector describing the manipulator's hand posture is matched with the gesture model in the gesture model library. According to the matching result, the image is divided into a frame image with gesture category and a frame image without gesture category, and the gesture category information in the frame image with gesture category is obtained; According to the skeleton node information of the operator and the gesture category information of the previous and next frames of the frame image without gesture category, the gesture category information and the position information of the hand in the frame image without gesture category are obtained; An instruction to control the display device is generated according to the gesture category information and the hand position information of the frame image with gesture category and the gesture category information and the hand position information of the frame image without gesture category.

2. The method according to claim 1, characterized in that Also includes: The operator's real-time bone node information is captured by a depth camera. The bone node information includes the three-dimensional coordinates of the joints and the connection relationship between the joints.

3. The method according to claim 1, characterized in that: The skeleton node information of the operator in the frame images with gestures in the previous and next frames of the frame images without gestures is interpolated to calculate the skeleton node information of the operator's hand in the frame images without gestures.

4. The method according to claim 3, characterized in that The hand bone node information of the operator in the frame image without gesture category is matched with the gesture model in the gesture model library to obtain the gesture category of the operator in the frame image without gesture category.

5. The method according to claim 1, characterized in that According to the skeleton node information of the operator and the skeleton node information of the operator's hand, the wrist node position of the operator in the frame image without gesture is obtained; According to the wrist node position of the operator in the frame image of the no-gesture category, the hand position information of the operator in the frame image of the no-gesture category is obtained.

6. The method according to claim 1, characterized in that Compare the gesture category information in the gesture category frame images of the preceding and following frames of the frame image without gesture category. If the gesture category information of the preceding and following frames belong to the same gesture category, then determine that the gesture information in the frame image without gesture category is also the same as the gesture category information of the preceding and following frames.

7. The method according to claim 6, characterized in that If the gesture category information of the previous and next frames respectively belongs to two consecutive different gesture categories, it is determined that the gesture category information in the frame image without gesture category is the same as the gesture category information of the previous frame or the gesture category information of the next frame.

8. The method according to claim 1, characterized in that The gestures of the frame images with gesture category and the frame images without gesture category are classified into primary response gesture one and secondary response gesture two, and the displacement change data of gesture one and gesture two of the same manipulating hand of the manipulator between frames are obtained; When the displacement change data of gesture one reaches the condition for triggering gesture two, it is determined whether the displacement change data of gesture two reaches the condition for triggering menu display, and if so, an instruction to control the display device is generated.

9. A device for controlling a display device based on gestures, characterized in that: include: An acquisition unit, used to acquire the skeleton node information of the operator and the skeleton node information of the manipulator's hand in the image; A matching unit is used to generate a feature vector describing the hand posture of the operator according to the skeletal node information of the operator's hand, match the feature vector describing the hand posture of the operator with the gesture model in the gesture model library, and divide the image into a frame image with gesture category and a frame image without gesture category according to the matching result, and obtain gesture category information in the frame image with gesture category; A judgment unit, used to obtain gesture category information and hand position information in the non-gesture frame image according to the operator's skeleton node information and gesture category information of the previous and next frames of the non-gesture frame image; The generating unit is used to generate an instruction to control the display device according to the gesture category information and the hand position information of the frame image with gesture category and the gesture category information and the hand position information of the frame image without gesture category.

10. A computer readable medium, characterized in that: The computer readable medium stores a computer program, which is loaded and executed by the processing module to implement the steps of any one of the methods of claims 1 to 8.

Images