Display device and dynamic gesture interaction method

Abstract

Some embodiments of the present application provide a display device and a dynamic gesture interaction method that can obtain a gesture information stream after the user inputs a dynamic gesture, and extract key gesture information from the gesture information stream. Then, a detection model is used to match the key gesture types of each stage in the key gesture information to obtain a node combination with the same key gesture type and a confidence parameter within a set confidence interval as the determined target dynamic gesture. Finally, the control instructions associated with the target dynamic gesture are executed to realize dynamic gesture interaction. The method can detect dynamic gestures based on gesture key points, and then dynamically match key gesture types based on a detection model stored in a tree structure node format, which can enrich the dynamic gesture interaction form and support user-defined dynamic gestures.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a dynamic gesture interaction method. Background Art

[0002] Gesture interaction is a new mode of human-computer interaction. With the development of artificial intelligence (AI) technology, more and more gesture interaction methods can be applied to the interaction process of display devices. The purpose of gesture interaction is to control the display device to execute the corresponding control instructions by detecting specific gestures made by the user. For example, a user can control the display device to rewind or fast forward playback by waving left or right instead of the left and right arrow keys on a remote control device.

[0003] Typically, gesture interaction supported by display devices is based on static gestures, meaning the user's hand shape remains unchanged when performing a specific gesture. For example, when waving left or right, the user needs to keep their five fingers together and move their palm in parallel. During interaction, the display device can first detect static gestures using a gesture type recognition algorithm and then perform the corresponding control action based on the gesture type.

[0004] As can be seen, this static gesture-based interaction method supports a limited number of gestures and is only suitable for simple interaction scenarios. To expand the number of supported gestures, some display devices also support dynamic gesture interaction, which implements specific gesture interactions through continuous movements within a time period. However, due to the limitations of the model used in the dynamic gesture detection process, this dynamic gesture interaction process does not support user-defined gestures and cannot meet user needs. Summary of the Invention

[0005] Some embodiments of the present application provide a display device and a dynamic gesture interaction method to solve the problem that traditional gesture interaction methods are single in form and do not support user customization.

[0006] In one aspect, some embodiments of the present application provide a display device comprising: a display, an image acquisition interface, and a controller. The display is configured to display a user interface; the image acquisition interface is configured to capture gesture interaction images input by a user; and the controller is configured to execute the following program steps:

[0007] Acquire a gesture information stream, where the gesture information stream includes multiple consecutive frames of gesture interaction images;

[0008] Extracting key gesture information from the gesture information stream, the key gesture information including key gesture types of multiple stages and confidence parameters of each stage;

[0009] Matching the key gesture information using a detection model to obtain a target dynamic gesture, the detection model comprising a plurality of nodes stored in a tree structure; each node being provided with a gesture posture template and a designated subordinate node; the target dynamic gesture being a node combination in which the key gesture type is the same as the gesture posture template at each stage and the confidence parameter is within a confidence interval;

[0010] Execute the control instruction associated with the target dynamic gesture.

[0011] On the other hand, some embodiments of the present application further provide a dynamic gesture interaction method, applied to a display device, the dynamic gesture interaction method comprising the following steps:

[0012] Acquire a gesture information stream, where the gesture information stream includes multiple consecutive frames of gesture interaction images;

[0013] Extracting key gesture information from the gesture information stream, the key gesture information including key gesture types of multiple stages and confidence parameters of each stage;

[0014] Matching the key gesture information using a detection model to obtain a target dynamic gesture, the detection model comprising a plurality of nodes stored in a tree structure; each node being provided with a gesture posture template and a designated subordinate node; the target dynamic gesture being a node combination in which the key gesture type is the same as the gesture posture template at each stage and the confidence parameter is within a confidence interval;

[0015] Execute the control instruction associated with the target dynamic gesture.

[0016] It can be seen from the above technical solutions that the display device and dynamic gesture interaction method provided by some embodiments of the present application can obtain a gesture information stream after the user inputs a dynamic gesture, and extract key gesture information from the gesture information stream. Then, the detection model is used to match the key gesture types of each stage in the key gesture information to obtain a node combination with the same key gesture type and a confidence parameter within the set confidence interval as the determined target dynamic gesture, and finally the control instructions associated with the target dynamic gesture are executed to realize dynamic gesture interaction. The method can detect dynamic gestures in stages based on gesture key points, and then dynamically match key gesture types based on the detection model in the form of tree structure node storage, which can enrich the form of dynamic gesture interaction and support user-defined dynamic gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a schematic diagram of the application scenario structure of the display device in the embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the hardware configuration of the display device in the embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the structure of a display device with a built-in image collector in an embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of the structure of a display device connected to an external image collector in an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of the dynamic gesture interaction process in an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of key points in the embodiments of this application;

[0024] Figure 7 This is a schematic diagram of hand orientation in an embodiment of the present application;

[0025] Figure 8 This is a tree structure diagram of a detection model in an embodiment of the present application;

[0026] Figure 9 This is an action path diagram when the pseudo jump is successful in the embodiment of the present application;

[0027] Figure 10 This is an action path diagram when a pseudo jump fails in an embodiment of the present application;

[0028] Figure 11 This is a schematic diagram of the data flow relationship of dynamic gesture interaction in an embodiment of the present application;

[0029] Figure 12 This is a timing diagram of dynamic gesture interaction in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0031] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0032] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0033] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0034] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc.

[0035] Figure 1 Schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the control device 300 or the control apparatus 100 .

[0036] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0037] In some embodiments, a control device 300 (such as a mobile phone, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the control device 300.

[0038] In some embodiments, the display device 200 may not use the above-mentioned control device 300 or control apparatus 100 to receive instructions, but may receive user control through touch or gestures.

[0039] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the control device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0040] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0041] like Figure 2 As shown, the display device 200 may include at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0042] In some embodiments, the controller 250 may include a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output.

[0043] The display 260 may include the following components, namely: a display screen component for presenting images; a driving component for driving image display; a component for receiving image signals output from the controller 250 to display video content, image content, and menu control interface, as well as a component for user control UI interface, etc.

[0044] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0045] Communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. Display device 200 can use communicator 220 to send and receive control signals and data signals with external control device 100 or server 400.

[0046] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0047] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, such as a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures.

[0048] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0049] The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the controller 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device to the main device where the controller 250 is located, such as an external set-top box.

[0050] Controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Controller 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object for display on display 260, controller 250 may perform operations related to the object selected by the user command.

[0051] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0052] In order to support the user's gesture interaction, in some embodiments, the display device 200 can have a built-in or external image collector 231. The image collector 231 built into the display device 200 can be used as a collector 230 of the display device 200. For example, some display devices 200 can have a built-in camera, which can be set at the top or bottom frame of the display device 200, such as Figure 3 The camera of the display device 200 can directly use the data bus inside the display device 200 to connect with other components to achieve data transmission.

[0053] In some embodiments, the camera built into the display device 200 can also support lifting. That is, the camera can be placed on a lifting mechanism. When image capture is required, specific lifting instructions are used to control the lifting mechanism to move, thereby raising the camera for image capture. When image capture is not required, specific lifting instructions can also be used to control the lifting mechanism to move, thereby lowering the camera to hide it.

[0054] The image acquisition device 231 externally connected to the display device 200 can be an independent peripheral device and connected to the display device 200 via a specific image acquisition interface. Figure 4 As shown, the image collector 231 can be an independent camera device, and the display device 200 can be provided with a Universal Serial Bus (USB) interface or a High Definition Multimedia Interface (HDMI), and the image collector 231 is connected to the display device 200 via the USB interface or the HDMI interface. To facilitate the detection of the user's gesture interaction actions, in some embodiments, the image collector 231 externally connected to the display device 200 can be set in a position close to the display device 200, such as the image collector 231 is clamped on the top of the display device 200 by a clamping device, or the image collector 231 is placed on a desktop near the display device 200.

[0055] Obviously, the image collector 231 externally connected to the display device 200 can also support other connection methods based on the specific hardware configuration of the display device 200. In some embodiments, the image collector 231 can also establish a connection with the display device 200 through the communicator 220 of the display device 200 and send the collected image data to the display device 200 according to the data transmission protocol corresponding to the communicator 220. For example, the display device 200 can connect to the image collector 231 via a local area network or the Internet. After the network connection is established, the image collector 231 can send the collected data to the display device 200 via the network transmission protocol.

[0056] In some embodiments, the image collector 231 can also be connected to the display device 200 via a wireless network. For example, for a display device 200 that supports a WiFi wireless network, its communicator 220 is equipped with a WiFi module. Therefore, by connecting the image collector 231 and the display device 200 to the same wireless network, a wireless connection can be established between the display device 200 and the image collector 231. After the image collector 231 captures image data, the image data can be sent to a router device on the wireless network, which then forwards it to the display device 200. Obviously, the image collector 231 can also directly establish a wireless connection with the display device 200 via other wireless connection methods. For example, the image collector 231 is a magnetic camera equipped with a Bluetooth connection module. The camera can be attached to the housing of the display device 200 via magnetic attraction, and a Bluetooth connection can be established with the display device 200 via the Bluetooth connection module to transmit image data. Wireless connection methods include but are not limited to WiFi direct connection, cellular network, analog microwave, Bluetooth, infrared, etc.

[0057] Whether the image collector 231 is built into the display device 200 or is externally connected to the display device 200, the user can start the image collector 231 to collect image data through specific interactive instructions or application control during the use of the display device 200, and perform corresponding processing on the collected image data according to different needs. For example, a "video call" application can be installed in the display device 200. When the user controls the display device 200 to start the application, the application can automatically initiate a wake-up command to the image collector 231 to start the image collector 231 to capture images. As the image capture process progresses, the image collector 231 can send the captured image data to the display device 200 to form an image data stream. Similarly, when the user controls to exit the "video call" application, the application also sends a sleep command to the image collector 231 to control the image collector 231 to stop collecting image data.

[0058] The image data collected by the image collector 231 can be sent to other modules within the display device 200 through the data transmission path between the image collector 231 and the display device 200 for use by the corresponding modules. For example, when running an image recognition-related application, the image collector 231 can send the image data to the control 250 for processing. The controller 250 then identifies the target in the image data by running a specific image processing application to identify the characteristic target in the image. When running a video call application, the image data collected by the image collector 231 can be processed by the controller 250 to form a call data stream, and then the call data stream is sent to the communicator 220 to be transmitted to the other user of the video call through the communicator 220 for display.

[0059] In some embodiments, the display device 200 can perform gesture interaction via the image collector 231. During gesture interaction, a user can make a specific gesture within the capture range of the image collector 231, which will capture a gesture image. The captured gesture image is then sent to the display device 200, allowing the display device 200 to identify the gesture type in the gesture image, determine the control instruction corresponding to the gesture according to the interaction strategy set in the gesture interaction, and implement the control instruction.

[0060] For example, when a user makes a gesture with five fingers together and the palm facing the image collector 231 within the capture range of the image collector 231, the display device 200 can recognize the gesture in the gesture image captured by the image collector 231 and determine the control instruction for the gesture as "pause / start playback." Finally, by executing the control instruction, the current playback interface is paused or started.

[0061] In the above embodiment, one gesture type can correspond to one control instruction, and the display device 200 can set gestures for triggering specific control instructions according to different uses. However, the core of the above gesture recognition is the use of static gesture recognition. Static gesture recognition can only identify the gesture type and gesture position, and it is difficult to identify gesture posture data, and there is no way to achieve the effect of fine control. In addition, there are only a few types of gestures that can be made by the hand, and among the gestures made, there are even fewer types that are easy to detect and recognize.

[0062] Therefore, in some embodiments, the display device 200 may also support dynamic gesture interactions. A dynamic gesture refers to a user performing two or more gesture types within a specific detection period during a single interaction. By combining multiple gesture types, the display device 200's gesture interaction types can be expanded, increasing the richness of gesture interaction.

[0063] For example, if a user adjusts a gesture from spreading their five fingers to closing their five fingers within 2 seconds, i.e., inputs a grabbing action lasting 2 seconds, the display device 200 can continuously capture gesture images within the 2-second detection period and identify the gesture type in the gesture images frame by frame, thereby identifying the grabbing action based on the gesture changes in multiple frames. Finally, the control instruction corresponding to the grabbing action, i.e., "full screen / windowed play," is determined and executed to adjust the size of the playback window.

[0064] In some embodiments, dynamic gesture recognition can be achieved by training a model using deep learning or other training methods. Multiple consecutive frames of gesture image data are then input into the trained dynamic gesture recognition model. The model's internal classification algorithm then calculates the target dynamic gesture corresponding to the current multiple frames of gesture image data. The target dynamic gesture can typically be associated with a specific control instruction, and the display device 200 can implement dynamic gesture interaction by executing this control instruction.

[0065] For example, training data can be generated based on gesture image data. Each frame of gesture interaction image in the training data is assigned a classification label, indicating the type of gesture corresponding to the current frame. Furthermore, multiple consecutive frames of gesture interaction imagery are uniformly assigned dynamic gesture labels, indicating the dynamic gestures corresponding to the multiple frames of gesture interaction imagery. After generating the training data, the training data containing the multiple consecutive frames of gesture interaction imagery can be input into an initial dynamic gesture recognition model to obtain classification probabilities output by the recognition model. The classification probabilities output by the model are then subjected to a loss function operation with the classification labels in the training data to calculate the classification loss. Finally, the model parameters in the recognition model are adjusted using the calculated classification loss through backpropagation. By repeating the aforementioned "classification calculation - loss calculation - backpropagation" model training process, a recognition model capable of outputting accurate classification probabilities can be obtained using a large amount of training data. Using the trained recognition model, the display device 200 can input multiple consecutive frames of gesture interaction imagery detected in real time into the recognition model to obtain classification results, determine the dynamic gestures corresponding to the multiple consecutive frames of gesture interaction imagery, and then match the control instructions corresponding to the dynamic gestures to achieve dynamic gesture interaction.

[0066] In some embodiments, dynamic gesture interaction can also support user-defined operations, that is, a dynamic gesture interaction method is provided, which can be applied to the display device 200. In order to meet the user's gesture interaction, the display device 200 should at least include a display 260 and a controller 250. And at least one image collector 231 is built in or externally connected. Among them, the display 260 is used to display the user interface to assist the user's interactive operations; the image collector 231 is used to collect the gesture interaction images input by the user. Figure 5As shown, the controller 250 is configured to execute the application corresponding to the dynamic gesture interaction method, including the following contents:

[0067] Acquire a gesture information stream. The gesture information stream is video data generated by the image collector 231 through continuous image capture, so the gesture information stream includes multiple frames of continuous gesture interaction images. After starting gesture interaction, the display device 200 can send a start instruction to the image collector 231 to start the image collector 231 to capture images. After starting image capture, the user can make dynamic gestures within the shooting range of the image collector 231, and the image collector 231 can continuously capture multiple frames of gesture interaction images along with the user's dynamic gesture movements. The multiple frames of gesture interaction images captured are sent to the controller 250 in real time to form a gesture information stream.

[0068] Because the gesture information stream includes multiple frames of gesture interaction images, and the gesture interaction images are captured by the image collector 231, the frame rate of the gesture interaction images included in the gesture information stream can be the same as the image capture frame rate of the image collector 231. For example, when the image collector 231 captures images at a frame rate of 30 frames per second (30 FPS), the controller 250 can also obtain the gesture information stream at a frame rate of 30 frames per second.

[0069] However, in some display devices 200 with weaker computing power, an excessively high frame rate will result in an excessive amount of real-time data processing by the controller 250, affecting the response speed of gesture recognition. Therefore, in some embodiments, the display device 200 can also obtain a gesture information stream with a lower frame rate. In order to reduce the frame rate of the gesture information stream, the display device 200 can extract multiple frames of gesture images at equal intervals from the image captured by the image collector 231. For example, the display device 200 can extract one frame of gesture interaction image every frame from the gesture image captured by the image collector 231, thereby obtaining a gesture information stream with a frame rate of 15. The display device 200 can also send a control instruction for frame rate adjustment to the image collector 231, controlling the image collector 231 to capture only 15 frames of gesture image data per second, thereby forming a gesture information stream with a frame rate of 15.

[0070] It should be noted that the input process of dynamic gestures will be affected by the input speed of different user actions, that is, some users' gesture input actions are faster, and some users' gesture input actions are slower. Obviously, for gestures input when the action is slower, the gesture difference between adjacent frames is smaller, and the gesture information stream with a low frame rate can also represent the complete gesture input process. For gestures input when the action is faster, the gesture difference between adjacent frames is larger, and the gesture information stream with a low frame rate may lose some key gestures, affecting the accuracy of gesture recognition. Therefore, in order to improve the accuracy of gesture recognition, the display device 200 should maintain a higher frame rate as much as possible to obtain gesture interaction images. For example, the frame rate of the gesture information stream can be maintained in the range of 15-30FPS.

[0071] Furthermore, in some embodiments, the display device 200 can also dynamically adjust the frame rate of the gesture information stream within a specific interval according to the current operating load, so as to improve the accuracy of gesture recognition by obtaining a high-frame-rate gesture information stream when the computing power is sufficient; and reduce excessive consumption of the computing power of the controller 250 by obtaining a low-frame-rate gesture information stream when the computing power is insufficient.

[0072] After acquiring the gesture information stream, the display device 200 can perform gesture recognition processing on each frame of the gesture interaction image in the gesture information stream to extract key gesture information from the gesture information stream. The gesture recognition processing can be based on an image recognition algorithm to identify the locations of key points such as fingers, joints, and wrists in the gesture interaction image. Specifically, the coordinates of the key points are used to represent the imaging positions of the hand joints in the gesture interaction image.

[0073] For example, the display device 200 can identify the position coordinates of each key point in the current gesture interaction image by matching the feature shape. Then, the coordinates of each key point are combined into an information vector in a set order. Figure 6 As shown, the key points used to characterize gestures can include 21 finger key points, and the position information of each key point can be represented by the coordinates of the corresponding point. For example, for the fingertip key point, the coordinates of the thumb tip are P T1 =(x t1 ,y t1 ), the index fingertip coordinate is P T2 =(x t2 ,y t2 ), the coordinate of the middle finger tip is P T3 =(x t3 ,y t3 )……; Similarly, for the key point in the middle of the finger, the above coordinate representation method is also used, that is, the coordinate of the middle of the thumb is P M1 =(x m1 ,y m1 )……; and the root key point is PB1 =(x b1 ,y b1 ).

[0074] The above-mentioned fingertip, middle and base coordinates can be combined to form a vector for representing the fingertip information, the middle and base information, namely the fingertip information F T for:

[0075] F T =[P T1 ,P T2 ,P T3 ,P T4 ,P T5 ]

[0076] Finger Information F M for:

[0077] F M =[P M1 ,P M2 ,P M3 ,P M4 ,P M5 ]

[0078] Root information F B for:

[0079] F B =[P B1 ,P B2 ,P B3 ,P B4 ,P B5 ]

[0080] Except for the above fingertips F T , middle F M , finger root F B In addition to the coordinate information, the display device 200 can also extract the palm coordinate P in the gesture interaction image. Palm and wrist coordinate P Wrist Then combine these coordinate information to form the gesture key coordinate set H Info That is, the gesture key coordinate set H Info for:

[0081] H Info =[P Palm ,P Wrist ,F T ,F M ,F B ]

[0082] It can be seen that the above-mentioned gesture key coordinate set is a coordinate set composed of multiple key point coordinates. Therefore, based on the mutual relationship of the key point positions in the above-mentioned gesture key coordinate set, the display device 200 can determine the key gesture type from the gesture key coordinate set. In order to determine the key gesture type, in some implementations, when the display device 200 extracts key gesture information from the gesture information stream, it first identifies the key point coordinates in the gesture interaction image, and then extracts the preset key point standard coordinates from the database. Among them, the key point standard coordinates are a template coordinate set determined by the operator of the display device 200 through statistical analysis of crowd gestures, and each gesture can be provided with corresponding key point standard coordinates.

[0083] After extracting the key point coordinates and the standard key point coordinates, the display device 200 can calculate the difference between the key point coordinates and the standard key point coordinates. If the calculated difference is less than or equal to a preset recognition threshold, it is determined that the user gesture in the current gesture interaction image is similar to the gesture type in the standard gesture template. Therefore, the gesture type corresponding to the standard key point coordinates can be determined as the target gesture type.

[0084] For example, if the user makes a five-fingers-together gesture to the image collector 231, the gesture key coordinate set H can be obtained by recognizing a frame of gesture interaction image corresponding to the gesture. Info1 , and then match the standard gestures similar to the five-finger-close gesture from the database to extract the standard coordinates H' of the key points. By calculating the difference between the two coordinate sets, that is, H = H Info1 -H', if the difference is less than or equal to the preset recognition threshold H", that is, H≤H", the target coordinate set is matched, so it can be determined that the target gesture type in the current gesture interaction image is a five-finger close-together gesture.

[0085] In some embodiments, the key gesture information may further include a confidence parameter for characterizing the difference between each gesture type and a standard gesture. In this case, the key gesture information may further include the following parameter items that can represent the key gesture type, that is, the gesture posture information includes but is not limited to: hand facing H F (Hand Face), hand facing H O (HandOrientation), hand orientation angle information H OB , left and right hand H S (Hand Side), gesture extension state H T (Handstretched), etc. Each parameter item can be calculated using the above-mentioned gesture key coordinate set.

[0086] The hand direction information can be used to indicate the direction of the fingertips in the image, such as Figure 7As shown, fingertips pointing up are Up, pointing down are Down, pointing to the left are Left, pointing to the right are Right, and pointing forward (center) are Center. The default is Unknown. Therefore, the hand direction information can be expressed as:

[0087] H O ={Up,Down,Left,Right,Center,Unknown}

[0088] Similarly, while identifying hand orientation information, the hand orientation angle information can also be determined based on the positional relationship between the coordinates of specific key points. This is equivalent to the confidence level of the hand orientation information. For example, although the hand is detected as Left, it still has an angle and may not be facing completely to the left. In this case, some subsequent processing based on the angle information is required to prevent false triggering. The hand orientation angle can be expressed as:

[0089] H Ob =a(0<a<90)

[0090] The display device 200 can preferentially extract the hand orientation information, that is, generate the hand orientation information based on the left and right hand and index finger key point information. The display device 200 can use the index finger root information P B2 , Little finger root information P B5 , wrist information P Wrist , left and right hand information H S Generate hand orientation angle information H OB , hand horizontal and vertical information H XY , hand posture angle information H XB ,H YB , and finally get the hand direction information H O .Right now:

[0091] H O =g(H OB ,H XY ,H XB ,H YB )=f(P B2 ,P B5 ,P Wrist ,H S,α )

[0092] The generation logic is as follows: calculate the index finger root P B2 and the base of the little finger P B5 The deflection angle f(ΔX, ΔY) between the vector and the x-axis direction is in the range of (0°, 90°). The hand orientation information can be obtained based on the deflection angle, and then the deflection angle threshold is set to determine whether the orientation information is valid. For example, the deflection angle threshold β can be set to 5, that is, the orientation information within the range of 45±5 is considered invalid, and the horizontal and vertical information H of the hand is invalid. XY, that is, the generation formula is as follows:

[0093]

[0094] Where ΔX is the horizontal coordinate difference between the base of the index finger and the base of the little finger; ΔY is the vertical coordinate difference between the base of the index finger and the base of the little finger; f(ΔX, ΔY) is the deflection angle; and β is the deflection angle threshold.

[0095] Then calculate the middle point P between the base of the index finger and the base of the little finger M , and calculate the midpoint of the line connecting the four finger roots from the index finger to the little finger, and then calculate P M and wrist coordinate P Wrist The difference ΔY between the index finger and the little finger, and the difference ΔX between the index finger and the little finger, can be used to obtain the hand pitch angle information:

[0096]

[0097] Where H YB is the hand's pitch angle; ΔX is the horizontal coordinate difference between the index finger and the little finger; ΔY is the vertical coordinate difference between the index finger and the little finger.

[0098] If the pitch angle is too large, the hand is considered to be facing the center direction. The specific threshold is α. Since the error in judging the center direction is large, it cannot be used as a criterion for determining the action. Therefore, in some scenarios where precision is not required, it can be directly equated to Unknown. The judgment formula is as follows:

[0099]

[0100] Where H O is the hand orientation information, including Center and other two states, and α is the hand orientation pitch angle threshold.

[0101] Obviously, for some scenes requiring fine movements, more accurate hand posture angle information H is needed. XB ,H YB Therefore, the display device 200 can model the user's hand, preset hand attribute information for different distances, and obtain more accurate hand posture angle information. That is, the user can input the hand size information at different distances in advance, and then according to the current frame distance information, the index finger root information P B2 , Little finger root information P B5 , wrist information P Wrist , left and right hand information H S Can generate hand posture angle information H XB ,H YB .

[0102] According to the middle point PM Information, wrist information P Wrist , hand horizontal and vertical information H XY , left and right hand information H S The corresponding orientation information can be generated. For example, in the case of the right hand being in the vertical position, the Y-axis information of the wrist and the middle point needs to be compared. If the Y value of the middle point is less than the Y value of the wrist, it proves that it is in the vertical position. Therefore:

[0103] H O =l(P M ,P Wrist ,H XY ,H S )

[0104] Hand facing information H F Information indicating the direction of the hand in the picture, which may include a specific value indicating the direction, i.e., Front for forward direction and Back for back direction. F The default value is Unknown.

[0105] H F ={Front,Back,Unknown}

[0106] During the hand orientation recognition process, the hand orientation angle information can also be determined to characterize the degree of hand orientation, which is equivalent to the confidence level of the hand orientation information. For example, although the user's hand is detected as front, it may still have an angle and may not be facing completely forward. In this case, some subsequent processing based on the angle information is required to prevent false triggering of gestures. That is:

[0107] H Fb =a(0<a<90)

[0108] By extracting the gesture orientation information and the index finger root information P B2 , Little finger root information P B5 , left and right hand information H S , gesture direction information H O Generate hand orientation information H F The generation logic is as follows. Taking the right hand facing up as an example, if the x at the base of the index finger is less than the x at the base of the little finger, it is proven to be Front. More details will not be repeated here, and the general formula will be used instead:

[0109] H F =g(P B2 ,P B5 ,H S ,α,H O )

[0110] The left and right hand information can be used to indicate whether the hand image in the picture belongs to the user's left hand or right hand. The left hand is "Left" and the right hand is "Right". Therefore, the left and right hand information can be expressed as:

[0111] H S ={Right,Left,Unknown}

[0112] For the gesture extension state, it can be used to represent the extension state of the finger, that is, the finger state in the extended state can be represented as 1, and the finger state in the contracted state can be represented as 0. Obviously, the extension state of the finger includes not only the two states of extension and contraction, so the extension state can also be represented by setting different values. For example, the value representing the extension state can be set to 0, 1, and 2. Among them, full contraction is 0, half extension is 1, and full extension is 2, which can be flexibly changed according to the specific application scenario. Therefore, the gesture extension state can be represented as:

[0113] H T =[F1,F2,F3,F4,F5]F=0 or 1 or 2)

[0114] Wherein, F1 to F5 represent the extension and retraction states of the five fingers respectively.

[0115] Extract the gesture extension state. In this part, the curling state of each finger is mainly extracted based on information such as hand direction, hand facing, left and right hands, and gesture key points. The final extracted curling state attribute is 0 or 1 (this embodiment takes the state attribute 0 or 1 as an example), where 0 is the curled state and 1 is the extended state. O =Up,H S =Right,H F =Front as an example, that is, the user puts his right hand towards the camera with his hand facing up. Assuming that the index finger tip coordinate is 50 and the index finger middle coordinate is 70, the index finger tip is above the middle, which means the finger is stretched out, which is 1. If the index finger tip is 30 and the middle is 50, it is in a curled state.

[0116] The thumb and the other four fingers are compared differently. While the other four fingers compare their horizontal coordinates, the thumb compares their vertical coordinates. When the hand is facing up or down, the thumb compares its x-coordinate, while the other four fingers compare their y-coordinates. When the hand is facing right or left, the thumb compares its y-coordinate, while the other four fingers compare their x-coordinates. The thumb compares its base and fingertips, while the other four fingers compare their midpoints and fingertips. The comparison points can be adjusted based on the specific scenario, ultimately providing information on the curled state of all five fingers.

[0117] Through the above gesture recognition process, the key gesture information of the current frame can be obtained, including the direction of the hand facing HF , hand toward H O , hand orientation angle information H OB , left and right hand H S , gesture stretch H T Among them, the hand orientation angle information can be used to determine the accuracy of the gesture orientation. In specific scenarios, a threshold can be set to filter out some fuzzy gestures and improve the accuracy of gesture recognition. Taking the right hand, the back of the hand facing the camera, the gesture facing downward (angle of 86 degrees), as an example, the final key gesture information G Info It can be expressed as:

[0118] G Info ={H F =Back,H O =Down,H S =Right,H T ={0,1,0,0,0},H OB =86}

[0119] Because a user's dynamic gesture is a continuous input process, meaning that gesture interaction can be divided into multiple stages, key gesture information includes key gesture types for multiple stages. In some embodiments, the display device 200 can traverse multiple consecutive frames of gesture interaction images to identify the target gesture types corresponding to the key gesture types, thereby dividing the dynamic gesture into multiple stages based on the multiple consecutive frames of gesture interaction images. The gesture interaction images in each stage belong to the same target gesture type.

[0120] For example, the display device 200 may display multiple frames of gesture interaction images photo1 to photo n Analyze the gesture key coordinate set in the image to determine the key gesture types type1 to type n Then compare the key gesture types type1 to type n , so that the multi-frame gesture interaction images with the same key gesture type, such as photo1~photo 30 and photo 31 ~photo n , respectively determined into two stages, thereby determining the key gesture types of these two stages, namely type1=type2=…=type 30 and type 31 =type 32 =…=type n .

[0121] For confidence parameters corresponding to multiple stages, in some embodiments, the confidence parameters include key gesture deflection angles, and the display device 200 can calculate the gesture deflection angles based on the key point coordinates and the key point standard coordinates; then traverse the gesture deflection angles corresponding to multiple consecutive frame gesture interaction images in each stage to obtain the union of deflection angles in each stage; extract the extreme value of the union of deflection angles in each stage as the key gesture deflection angle in the key gesture information of the current stage.

[0122] After extracting the key gesture information, the display device 200 can call the detection model for dynamic gesture matching. The detection model is a matching model, which includes multiple nodes stored in a tree structure, and each node is provided with a gesture posture template. Multiple nodes can be at different levels, and except for the root node and the leaf node, each level of the node is provided with an upper node, and each level of the node is assigned a lower node. For example, in the memory of the display device 200, multiple gesture posture templates can be pre-stored, and each gesture posture template is used to represent a static gesture action. At the same time, the display device 200 also constructs a gesture detection model based on the stored gesture posture templates. In the detection model, each gesture posture template can be assigned corresponding node attributes and lower-level nodes. Therefore, in the display device 200, the gesture posture template can still maintain the original storage quantity, and the detection model can be constructed only by assigning node attributes.

[0123] Obviously, for the detection model, only one gesture posture template is inserted into each node, and each gesture posture template can be assigned multiple node attributes. For example, a "grab-and-release" dynamic gesture includes three stages, namely, five-finger open gesture, five-finger curled gesture, and five-finger open gesture. The corresponding nodes and gesture posture templates in the detection model are: root node - "five-finger open gesture"; first-level node - "five-finger curled gesture"; second-level node - "five-finger open gesture". It can be seen that for each node, only one gesture posture template is inserted, and for each gesture posture template, different levels of node attributes are assigned accordingly, that is, the "five-finger open gesture" target is assigned two node attributes: the root node and the second-level node.

[0124] In the detection model, the root node is used to initialize the matching and may include multiple gesture templates that can be used to match the initial gesture input by the user. For example, the root node may be inserted with a gesture template for characterizing the triggering gesture interaction. The leaf nodes in the detection model generally do not include specific gesture templates, but rather control instructions for representing specific response actions. Therefore, in the embodiments of the present application, unless otherwise specified, the nodes of the detection model do not include leaf nodes.

[0125] After calling the detection model, the display device 200 can use the detection model to match the key gesture information to obtain the target dynamic gesture, where the target dynamic gesture has the same key gesture type as the gesture posture template at each stage and the confidence parameter is a node combination within the confidence interval. Therefore, the target dynamic gesture can be represented by an action path. In order to determine the target dynamic gesture, the display device 200 can match the key gesture type of each stage in the key gesture information with the gesture posture template on each level node in the detection model.

[0126] In the process of using the detection model to match key gestures, the display device 200 can first match the gesture templates of the same type in the corresponding layer based on the key gesture type of each stage. And when a gesture template is matched, the node corresponding to the gesture template is recorded. At the same time, the display device 200 also determines whether the confidence parameter of the node is within a preset reasonable confidence interval. If the key gesture type of the current stage is the same as the gesture template, and the confidence parameter is within the confidence interval, the next stage of matching begins.

[0127] For example, for the dynamic gesture of "grab-and-release", after the user inputs the dynamic gesture, the display device 200 can first match the first-stage "five-finger open gesture" with the gesture posture template in the root node. When the match determines that the "five-finger open gesture" is the same as or similar to the five-finger open gesture template in a root node, it can be determined whether the confidence parameter of the first stage is within the preset confidence interval, that is, whether the gesture orientation angle is within the preset angle interval. If the gesture orientation angle is within the preset angle interval, the second-stage key gesture "five-finger curled-up gesture" is started to perform the above-mentioned matching with the subordinate nodes of the root node.

[0128] After matching the key gestures of each stage with the nodes of the corresponding level, the display device 200 can obtain an action path consisting of multiple matching hit nodes. The action path will eventually point to a leaf node, and the leaf node corresponds to a target dynamic gesture. Therefore, the display device 200 can obtain the target dynamic gesture after the matching is completed and execute the control instructions associated with the target dynamic gesture.

[0129] For example, according to the setting of the gesture interaction strategy of the display device 200, the dynamic gesture of "grab-release" can be used to delete the currently selected file. Therefore, the display device 200 can obtain the deletion instruction after matching the action path of "root node-five fingers open; first-level node-five fingers curled up; second-level node-five fingers open", and delete the currently selected file by executing the deletion instruction.

[0130] It can be seen that in the above embodiment, the display device 200 extracts the gesture posture information at each stage in the gesture information flow and matches the gesture posture information using a detection model in the form of a tree structure node, so that the action path can be determined layer by layer according to the gesture input stage, thereby obtaining the target dynamic gesture. Since the detection model adopts the node form of a tree structure, it is possible to avoid reading the dynamic gesture template each time and repeated monitoring during the process of matching gesture key information. In addition, the tree structure detection model also supports users to insert nodes at any time to realize gesture input. And by adjusting the confidence interval of each node, the hit rate of the node matching process can be customized, so that the detection model can use the gesture habits of different users to realize customized gesture operations.

[0131] In some embodiments, in order to enable the display device 200 to match gesture types for key gesture information, the display device 200 may first extract the first-stage key gesture type from the multi-stage key gesture information when using the detection model to match key gesture information. Then, the first node is matched according to the first-stage key gesture type, wherein the first node is a node whose stored gesture posture template is the same as the first-stage key gesture type. After matching the first node, the display device 200 may extract the second-stage key gesture type from the key gesture information, wherein the second stage is the subsequent action stage of the first stage. Then, the second node is matched according to the second-stage key gesture type. Similarly, the second node is a node whose stored gesture posture template is the same as the second-stage key gesture type, that is, the subordinate node specified by the first node includes the second node. Finally, the first node and the second node are recorded to obtain the action branch.

[0132] For example, the display device 200 may pre-register four key gesture templates, and the key gesture information corresponding to each of them is G info1 -G info4 , corresponding to the combination of five dynamic gestures AM1-AM5. Among them, the first stage key gesture types of AM1-AM4 are the same, and the second stage gesture types of AM3-AM4 are also the same, such as Figure 8 As shown, the corresponding tree structure detection model can be obtained, and the corresponding dynamic gesture is expressed as follows:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138] When performing key gesture information matching, the display device 200 may prioritize the G Info1 and G Info2 If the key gesture information is G Info1 Then according to G Info1 The corresponding root node is designated by the subordinate node to continue monitoring, that is, the key gesture template is matched to G Info2 , G Info3 and G Info4 Similarly, if in the matching process of the second-level node, the key gesture information matched is G Info4 , it will continue to monitor the lower level nodes, that is, G in the third level Info2 and G Info3 The corresponding node. The node matching of subsequent levels is carried out in sequence until the leaf node is detected. For example, in the third level, the matching hit G Info3 If other actions that are not stored in the current level node of the detection model are detected during the matching of a level node, the root node of the tree will be returned to monitor G again. Info1 and G Info2 .

[0139] It should be noted that in the above embodiments, the terms "first stage," "second stage," and "first node" and "second node" are used solely to represent the sequential order of different stages in a dynamic gesture and the hierarchical relationship of different nodes in the detection model, and do not have corresponding numerical meanings. When matching key gesture information using the detection model, gestures in the same stage can be used as both the first and second stages. Similarly, the same node can be used as both the first and second nodes.

[0140] For example, at the beginning of using the detection model to match key gesture information, the key gesture information in the beginning needs to be matched with the root node in the detection model. In this case, the beginning is the first stage, and the next stage after the beginning is the second stage. The root node that is matched is the first node, and the node that is matched in the next level of the root node is the second node. After the matching is completed in the beginning, the display device 200 will continue to use the detection model to match the key gesture information. In this case, the next stage after the beginning is the first stage, and the next stage after the first stage is the second stage. The node that is matched in the node in the next level of the root node is the first node, and the node that is matched in the next level of the first node is the second node. Therefore, in the process of matching using the detection model, the above process can be repeated until the final leaf node is matched.

[0141] The detection model with a tree structure also supports the user's gesture entry process, that is, in some embodiments, the display device 200 can traverse the gesture posture templates stored in the subordinate nodes of the first node when matching the second node according to the second-stage key gesture type; if the gesture posture templates stored in all subordinate nodes are different from the second-stage key gesture type, that is, the dynamic gesture input by the user is a new gesture, then the display device 200 can be triggered to enter the gesture, that is, the display 260 can be controlled to display the entry interface.

[0142] The input interface can prompt the user to input gestures. In order to obtain accurate dynamic gestures, during the gesture input process, the input interface can prompt the user to repeatedly make the dynamic gestures that need to be input through prompt messages. That is, the user inputs the same behavior multiple times in a loop. At the same time, the user can also specify the control instructions associated with the input dynamic gestures through the input interface. Each time the user inputs, the display device 200 extracts the key gesture information according to the above example and matches it with the nodes of the detection model. When the key gesture template is not matched in the nodes of one level, a new node is added to the current level according to the key gesture type of the corresponding stage.

[0143] In order to reduce the impact of the gesture entry process on the user's gesture interaction operations, in some embodiments, the display device 200 can ask the user whether to start the entry through a prompt message or window before displaying the entry interface, and receive the user's input instructions based on the window. If the user enters the entry gesture information, the user's entry gesture information entered based on the entry interface can be received, and in response to the entry gesture information, a new node is set for the detection model, and the new node is a subordinate node of the first node. Finally, the gesture type of the corresponding stage is stored in the new node as a gesture posture template for the new node.

[0144] As can be seen, in the above embodiment, the display device 200 can perform real-time dynamic gesture recording based on a tree-structured detection model. By determining the action to be recorded and recording the user behavior, it detects whether there is a corresponding action branch in the behavior tree structure. If there is no corresponding action branch, the gesture key posture is extracted, and then the corresponding behavior template is obtained. The corresponding node is inserted into the behavior tree to complete the dynamic gesture recording. Obviously, during the dynamic gesture recording process, if the dynamic gesture input by the user has a corresponding action branch in the detection model, the user behavior is detected according to the branch template. If the detection is successful, there is no need to change the node state of the detection model.

[0145] In some embodiments, when the display device 200 uses the detection model to match the key gesture information, it can also judge the corresponding confidence level, wherein the confidence level may include the gesture deflection angle and the number of frames the key gesture is maintained. For the gesture deflection angle, the display device 200 can obtain the confidence interval preset for the corresponding node in the detection model after matching a node; and then compare the key gesture deflection angle at the current stage with the confidence interval of the corresponding node. If the key gesture deflection angle is within the confidence interval, the corresponding current node is recorded and the matching of the subordinate nodes of the current node is started; if the key gesture deflection angle is not within the confidence interval, it is determined that the gesture deviation is large, and therefore further judgment or adaptive adjustment is required.

[0146] Because the confidence parameter not falling within the confidence interval may be due to user input habits, the display device 200 may also adjust the detection model parameters based on user habits. Therefore, in some embodiments, if, during the process of matching key gesture information using the detection model, the key gesture type at a stage is the same as the gesture posture template in the node, but the key gesture deflection angle is not within the confidence interval, the display device 200 may also modify the confidence interval based on the gesture deflection angle.

[0147] It should be noted that during template matching, the display device 200 can match hand orientation, hand facing, and finger extension information. If a match is successful, the confidence threshold is then tested to determine if a match has been successfully achieved. If so, the gesture is considered to have been successfully matched. During gesture entry, the display device 200 only needs to match hand orientation, hand facing, and finger extension information. A successful template match is considered successful. If all gestures in a dynamic gesture are successfully matched, the dynamic gesture is considered to have been successfully matched. Finally, the template confidence is optimized based on the best confidence level.

[0148] The best confidence level can be calculated by taking some keyframes from multiple input gesture interaction images. For example, during gesture detection, a dynamic gesture includes a five-fingered upward movement that occurs 10 times in a specific sequence. A gesture is considered detected if it is detected three times. Of these 10 occurrences, eight consecutive gestures meet the criteria (10-3+1). The one with the lowest average confidence level needs to be selected. This is because at the start and end of a gesture, there may be a large deflection angle at the connection between the gesture and other gestures, resulting in an excessively large deflection angle. Using this deflection angle as the confidence level can result in many false detections.

[0149] The confidence parameter for the key gesture maintenance frame count is the number of consecutive frames in the gesture interaction image that are of the same key gesture type as the first stage. In some embodiments, the display device 200 may also obtain the maintenance frame count before matching the second node according to the second stage key gesture type. If the maintenance frame count of the first stage key gesture type is greater than or equal to the frame count threshold, that is, the user has maintained a gesture for a long time, which is not a case of erroneous input, and thus the second node may be matched according to the second stage key gesture type. If the maintenance frame count of the first stage key gesture type is less than the frame count threshold, the current input may be different from the predetermined dynamic gesture. Therefore, gesture entry may be initiated according to the above embodiment, i.e., the display 260 may be controlled to display the entry interface to update the confidence interval.

[0150] For example, during a gesture interaction action, multiple gesture types may appear. Therefore, it is necessary to extract the more obvious characteristic gestures as the characteristic posture of the action. Among them, the core gesture posture features are the hand direction and the finger extension and retraction state. Therefore, the display device 200 can perform gesture key point recognition and key gesture information extraction on the action frame; then the key gesture information is cyclically matched. If the gesture direction, hand direction, left and right hands, and finger extension and retraction state are the same, it is judged to be a similar gesture. Every time a similar gesture is detected, the deflection angle information and the number of similar gestures are updated. The deflection angle information takes the maximum range, and the number of similar gestures needs to be greater than the threshold. The threshold is determined according to the frame rate and can also be set to a fixed value, such as 3. The action frame is processed and the gesture posture that meets the conditions is selected. When processing multiple action frames, the action communication is taken, and the parameters of each action posture are taken as the union, and finally the corresponding key gesture template is obtained.

[0151] When users record a gesture, their movements are relatively standard. However, when using gesture interaction, they may be more casual and less concerned about whether their posture is standard. Especially when users are in a hurry, their gestures may be very non-standard. This results in inaccurate recognition of dynamic gestures by the display device 200, which reduces the user experience.

[0152] In order to improve the above-mentioned problems and enhance the user experience, in some embodiments, the display device 200 can also adopt a pseudo-jump method when performing dynamic gesture detection. That is, the display device 200 can obtain the confidence parameter of the intermediate stage, and the intermediate stage is a stage between the start stage and the end stage in the multiple stages of key gesture information. Then compare the confidence parameter of the intermediate stage with the confidence interval of the corresponding node. If the confidence parameter of the intermediate stage is not within the confidence interval of the corresponding node, mark the node corresponding to the intermediate stage as a pre-jump node. Then, match the subordinate nodes of the pre-jump node according to the detection model to determine the target dynamic gesture based on the matching results of the subordinate nodes of the pre-jump node.

[0153] When matching the subordinate nodes of the pre-jump node according to the detection model, the display device 200 can obtain the subordinate node matching results of the pre-jump node; if the matching result is hitting any subordinate node, the pre-jump node and the hit subordinate node are recorded as the nodes of the target dynamic gesture; if the matching result is not hitting the subordinate node, the pre-jump node is discarded and the matching is performed again from the upper node.

[0154] For example, Figure 9 As shown, after detecting action G1, the subsequent action G2 will be detected. At this time, if an action G2 appears, but the confidence parameter exceeds the confidence interval, the display device 200 will perform a pseudo jump, that is, the subsequent detection of action G1 and the subsequent action detection of action G2 will be performed at the same time. If action G3 is detected after the pseudo jump, it is considered that the previous pseudo jump is established and the action G3 is directly entered. Figure 10 As shown in FIG, if action G3 is not detected after the pseudo jump, but action G4 appears, and action G1 and action G4 just form another action path, then the pseudo jump is considered invalid and the subsequent action detection of action G4 is continued.

[0155] In order to better implement the pseudo-jump method, the display device 200 can set a pseudo-jump threshold. For example, if a specific confidence parameter value is not within the confidence interval, a pseudo-jump will be performed only when the confidence parameter is less than the pseudo-jump threshold. In addition, there will be a prompt each time a pseudo-jump is performed, and the user can delete the pseudo-jump by pressing a specific button or making a specific gesture. After a certain number of pseudo-jumps, the display device 200 will optimize the Action node involved in the pseudo-jump and increase the specified threshold to adapt to the user's action style.

[0156] Among them, the display device 200 can update the pseudo jump threshold in a variety of ways. For example, each time a pseudo jump is performed, a prompt will pop up and the Act ion node information will be updated by default. If the user believes that this detection is a false detection, it is only necessary to delete this identification. The display device 200 can also update the pseudo jump threshold after multiple pseudo jumps to obtain a better user experience. In addition, for the pseudo jump process, a number threshold can also be set. That is, during the detection process, if there are multiple pseudo jumps, then after exceeding a certain number of times, the previous pseudo jump is considered invalid.

[0157] Based on the above dynamic gesture interaction method, some embodiments of the present application further provide a display device 200. The display device 200 includes: a display 260, an image acquisition interface, and a controller 250. The display 260 is configured to display a user interface; the image acquisition interface is configured to acquire gesture interaction images input by the user; Figure 11 、 Figure 12As shown, the controller 250 is configured to perform the following program steps:

[0158] Acquire a gesture information stream, where the gesture information stream includes multiple consecutive frames of gesture interaction images;

[0159] Extracting key gesture information from the gesture information stream, the key gesture information including key gesture types of multiple stages and confidence parameters of each stage;

[0160] Matching the key gesture information using a detection model to obtain a target dynamic gesture, the detection model comprising a plurality of nodes stored in a tree structure; each node being provided with a gesture posture template and a designated subordinate node; the target dynamic gesture being a node combination in which the key gesture type is the same as the gesture posture template at each stage and the confidence parameter is within a confidence interval;

[0161] Execute the control instruction associated with the target dynamic gesture.

[0162] It can be seen from the above technical solution that the display device 200 provided in the above embodiment can obtain a gesture information stream after the user inputs a dynamic gesture, and extract key gesture information from the gesture information stream. Then, the detection model is used to match the key gesture types of each stage in the key gesture information to obtain a node combination with the same key gesture type and a confidence parameter within the set confidence interval as the determined target dynamic gesture, and finally the control instructions associated with the target dynamic gesture are executed to realize dynamic gesture interaction. The display device 200 detects dynamic gestures based on gesture key points, and then dynamically matches key gesture types based on the detection model stored in the form of a tree structure node, which can enrich the form of dynamic gesture interaction and support user-defined dynamic gestures.

[0163] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A display device, characterized in that: include: monitor; An image acquisition interface configured to acquire gesture interaction images input by a user; The controller is configured as: Acquire a gesture information stream, wherein the gesture information stream includes multiple consecutive frames of gesture interaction images; Extracting key gesture information from the gesture information stream, the key gesture information including key gesture types of multiple stages and confidence parameters of each stage; Obtaining an intermediate stage confidence parameter using a detection model, wherein the intermediate stage is a stage between a start stage and an end stage in multiple stages of key gesture information; the detection model includes a plurality of nodes stored in a tree structure; each node is provided with a gesture posture template and a designated subordinate node; The target dynamic gesture is a node combination in which the key gesture type is the same as the gesture posture template in each stage and the confidence parameter is within the confidence interval; Comparing the confidence parameter of the intermediate stage with the confidence interval of the corresponding node; If the confidence parameter of the intermediate stage is not within the confidence interval of the corresponding node, marking the node corresponding to the intermediate stage as a pre-jump node; performing matching on the subordinate nodes of the pre-jump node according to the detection model, so as to determine a target dynamic gesture according to the matching result of the subordinate nodes of the pre-jump node, so as to obtain a target dynamic gesture; Execute the control instruction associated with the target dynamic gesture.

2. The display device according to claim 1, wherein The controller is configured to: In the step of extracting key gesture information from the gesture information stream, identifying key point coordinates in the gesture interaction image, wherein the key point coordinates are used to represent imaging positions of hand joints in the gesture interaction image; Extract the standard coordinates of preset key points; Calculating the difference between the key point coordinates and the key point standard coordinates; If the difference is less than or equal to a preset recognition threshold, determining that the gesture type corresponding to the standard coordinates of the key point is the target gesture type; The dynamic gesture is divided into multiple stages according to the gesture interaction images of multiple consecutive frames, and the gesture interaction images in each stage belong to the same target gesture type.

3. The display device according to claim 2, wherein The confidence parameter includes a key gesture deflection angle, and the controller is configured to: Calculating the gesture deflection angle according to the key point coordinates and the key point standard coordinates; Traverse the gesture deflection angles corresponding to multiple consecutive frames of gesture interaction images in each stage to obtain the union of the deflection angles in each stage; The extreme values of the deflection angles in each stage are extracted and concentrated as the key gesture deflection angles in the key gesture information of the current stage.

4. The display device according to claim 1, wherein The controller is configured to: In the step of matching the key gesture information using the detection model, extracting the first-stage key gesture type from the multi-stage key gesture information; Matching a first node according to the key gesture type of the first stage, where the first node is a node having the same stored gesture posture template as the key gesture type of the first stage; Extracting a second-stage key gesture type from the key gesture information, where the second stage is a subsequent action stage of the first stage; Matching a second node according to the second-stage key gesture type, where the second node is a node having the same stored gesture posture template as the second-stage key gesture type; The subordinate nodes designated by the first node include a second node; The first node and the second node are recorded to obtain an action branch.

5. The display device according to claim 4, wherein The controller is configured to: In the step of matching the second node according to the key gesture type of the second stage, traversing the gesture posture templates stored in the subordinate nodes specified by the first node; If the gesture posture templates stored in all lower-level nodes are different from the second-stage key gesture type, controlling the display to display an input interface; Receiving input gesture information input by the user based on the input interface; In response to the input gesture information, setting a new node for the detection model, the new node being a subordinate node of the first node; The second-stage gesture type is stored in the new node as a gesture posture template of the new node.

6. The display device according to claim 4, wherein: The controller is configured to: Obtaining a preset confidence interval for each node in the detection model; Compare the confidence intervals of the key gesture deflections and corresponding nodes at each stage; If the key gesture deflection angle is not within the confidence interval, modify the confidence interval according to the gesture deflection angle.

7. The display device according to claim 4, wherein: The confidence parameter also includes the number of frames in which the key gesture is maintained; and the controller is configured to: Before the step of matching the second node according to the key gesture type of the second stage, obtaining a number of maintained frames, where the number of maintained frames is the number of consecutive frames of the same key gesture type as the first stage in the gesture interaction image; If the number of frames maintaining the key gesture type in the first stage is greater than or equal to the frame number threshold, matching the second node according to the key gesture type in the second stage; If the number of frames maintained for the key gesture type in the first stage is less than the frame number threshold, the display is controlled to display an input interface.

8. The display device according to claim 1, wherein The controller is configured to: In the step of performing matching on the subordinate nodes of the pre-jump node according to the detection model, obtaining a matching result of the subordinate nodes of the pre-jump node; If the matching result is hitting any lower-level node, recording the pre-jump node and the hit lower-level node as the node of the target dynamic gesture; If the matching result is that the lower-level node is not hit, the pre-jump node is discarded.

9. A dynamic gesture interaction method, characterized in that: Applied to a display device, the dynamic gesture interaction method includes: Acquire a gesture information stream, wherein the gesture information stream includes multiple consecutive frames of gesture interaction images; Extracting key gesture information from the gesture information stream, the key gesture information including key gesture types of multiple stages and confidence parameters of each stage; A detection model is used to obtain an intermediate stage confidence parameter, wherein the intermediate stage is a stage between a start stage and an end stage in multiple stages of key gesture information; the detection model includes a plurality of nodes stored in a tree structure; each node is provided with a gesture posture template and a designated subordinate node; the target dynamic gesture is a node combination in which the key gesture type in each stage is the same as the gesture posture template and the confidence parameter is within a confidence interval; Comparing the confidence parameter of the intermediate stage with the confidence interval of the corresponding node; If the confidence parameter of the intermediate stage is not within the confidence interval of the corresponding node, marking the node corresponding to the intermediate stage as a pre-jump node; performing matching on the subordinate nodes of the pre-jump node according to the detection model, so as to determine a target dynamic gesture according to the matching result of the subordinate nodes of the pre-jump node, so as to obtain a target dynamic gesture; Execute the control instruction associated with the target dynamic gesture.

Images