Display device and gesture trajectory recognition method

By collecting and processing user behavior images in the display device, identifying gesture information and determining the cursor position, the problem of cursor stuttering and interruption in dynamic gesture interaction is solved, and the continuous and smooth movement of the cursor is achieved, and the user experience is improved.

CN114637439BActive Publication Date: 2024-11-05HISENSE VISUAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210303452.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-24
Publication Date
2024-11-05
Estimated Expiration
2042-03-24

Smart Images

  • Figure CN114637439B_ABST
    Figure CN114637439B_ABST
Patent Text Reader

Abstract

Some embodiments of the present application provide a display device and a gesture trajectory recognition method. The display device includes a display, an image input interface, and a controller. The display device can collect several frames of user behavior images of the user, and perform gesture recognition processing on the user behavior images to obtain user gesture information for each frame of the user behavior image. The cursor position corresponding to each frame of the user behavior image is obtained according to the user gesture information, and the user's gesture movement trajectory can be determined according to the cursor position. Thereby, the cursor in the display is controlled to move along the gesture movement trajectory. For each frame of the user behavior image obtained, the present application can obtain the corresponding cursor position and control the cursor movement, and there will be no situation where the cursor of a certain frame does not move and the cursor freezes or interrupts, thereby improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a gesture trajectory recognition method. Background Art

[0002] Display devices refer to terminal devices that can output specific display images. With the rapid development of display devices, the functions of display devices will become more and more abundant, and the performance will become more and more powerful. They can realize two-way human-computer interaction functions, integrate multiple functions such as audio and video, entertainment, and data, and are used to meet the diverse and personalized needs of users.

[0003] Gesture interaction is a new type of human-computer interaction mode. The purpose of gesture interaction is to control the display device to execute corresponding control instructions by detecting specific gestures made by the user. The gesture interaction methods that the display device can adopt may include static gestures and dynamic gestures. When using static gesture interaction, the display device can detect the gesture type according to the gesture type recognition algorithm, and perform corresponding control actions according to the gesture type. When using dynamic gesture interaction, the user can control the cursor in the display to move. The display device can establish a mapping relationship between the user's gesture and the cursor in the display. At the same time, by continuously detecting the user's image, the user's dynamic gesture can be determined, and then the gesture movement trajectory mapped to the display can be determined, so as to control the cursor to move along the gesture movement trajectory.

[0004] For the interactive process of dynamic gestures, the display device needs to continuously detect the user image. However, the user's gesture may not be detected in some images, resulting in the inability to accurately obtain the gesture movement trajectory corresponding to the user image, and thus the inability to control the cursor to move, resulting in cursor freezes and interruptions, which gives the user a poor experience. Summary of the invention

[0005] The present application provides a display device and a gesture trajectory recognition method to solve the problem in the related art that the gesture movement trajectory corresponding to the user image cannot be accurately obtained, resulting in a poor user experience.

[0006] In a first aspect, the present application provides a display device, including a display, an image input interface, and a controller. The image input interface is configured to connect to an image collector; the controller is configured to perform the following steps:

[0007] Control the image collector to collect several frames of user behavior images of the user;

[0008] Performing gesture recognition processing on the user behavior image to obtain user gesture information of each frame of the user behavior image;

[0009] Acquire a cursor position corresponding to each frame of the user behavior image according to the user gesture information; the cursor position is a display position in the display where the user's target gesture is mapped in the user behavior image;

[0010] The user's gesture movement trajectory is determined according to the cursor position, and the cursor in the display is controlled to move along the gesture movement trajectory.

[0011] In a second aspect, the present application provides a gesture trajectory recognition method, which is applied to a display device, comprising:

[0012] Control the image collector to collect several frames of user behavior images of the user;

[0013] Performing gesture recognition processing on the user behavior image to obtain user gesture information of each frame of the user behavior image;

[0014] Acquire a cursor position corresponding to each frame of the user behavior image according to the user gesture information; the cursor position is a display position in the display where the user's target gesture is mapped in the user behavior image;

[0015] The user's gesture movement trajectory is determined according to the cursor position, and the cursor in the display is controlled to move along the gesture movement trajectory.

[0016] It can be seen from the above technical solutions that the present application provides a display device and a gesture trajectory recognition method. The display device can collect several frames of user behavior images of the user, and perform gesture recognition processing on the user behavior images to obtain user gesture information for each frame of the user behavior image. The cursor position corresponding to each frame of the user behavior image is obtained according to the user gesture information, and the user's gesture movement trajectory can be determined according to the cursor position. Thereby controlling the cursor in the display to move along the gesture movement trajectory. The present application can obtain the corresponding cursor position and control the cursor movement for each frame of the user behavior image obtained, and there will be no situation where the cursor of a certain frame will not move, causing the cursor to freeze or be interrupted, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 shows a usage scenario of a display device according to some embodiments;

[0019] Figure 2shows a hardware configuration block diagram of a control device 100 according to some embodiments;

[0020] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0021] Figure 4 shows a software configuration diagram in a display device 200 according to some embodiments;

[0022] Figure 5 A schematic diagram showing a built-in camera of a display device in some embodiments is shown;

[0023] Figure 6 A schematic diagram showing an external camera of a display device in some embodiments is shown;

[0024] Figure 7 A schematic diagram of a user interface in some embodiments is shown;

[0025] Figure 8 A schematic diagram showing a cursor displayed on a display in some embodiments is shown;

[0026] Fig. 9 A schematic diagram showing cursor control mode confirmation information displayed in a display in some embodiments is shown;

[0027] Fig.10 A flowchart of the interaction between components of a display device in some embodiments is shown;

[0028] Fig.11 A schematic diagram showing user gestures in some embodiments;

[0029] Fig.12 A schematic diagram of a process of determining a cursor position according to user gesture information in some embodiments is shown;

[0030] Fig.13 A schematic diagram showing a camera area displayed on a display in some embodiments is shown;

[0031] Fig.14 A schematic diagram showing the movement of a cursor along a straight line in some embodiments is shown;

[0032] Fig.15 A schematic diagram showing the movement of a cursor along a curve in some embodiments is shown;

[0033] Fig.16 A schematic diagram showing the distance relationship between the cursor and the control in some embodiments;

[0034] Fig.17 The position relationship between the cursor and the control in some embodiments is shown;

[0035] Fig.18A flow chart of an embodiment of a method for identifying a hand gesture trajectory is shown. DETAILED DESCRIPTION

[0036] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0037] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately. It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0038] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0039] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0040] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0041] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to one or more embodiments of the present application, such as Figure 1As shown, the user can operate the display device 200 through the mobile terminal 300 and the control device 100. The control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user instructions through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices can also be used to control the display device 200.

[0042] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and realize connection communication through the network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to realize the synchronous display function. The display device 200 also communicates data with the server 400 through various communication methods. The display device 200 can be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 can provide various content and interactions to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide an intelligent network TV function that provides a computer support function.

[0043] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play the role of an interactive intermediary between the user and the display device 200. The communication interface 130 is used to communicate with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a button or an alternative module.

[0044] Figure 3 FIG. 2 shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment. Figure 3The display device 200 shown includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals. The detector 230 is used to collect signals of the external environment or interact with the outside. The controller 250 and the tuner 210 can be located in different split devices, that is, the tuner 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0045] In some embodiments, the controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. The user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

[0046] In some embodiments, "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on a display screen of an electronic device, wherein a control may include at least one of icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0047] Figure 4 FIG. 1 is a schematic diagram of software configuration in a display device 200 according to one or more embodiments of the present application. Figure 4As shown in FIG. 1 , the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer), the application framework layer (Application Framework layer), the Android runtime and system library layer (System runtime library layer), and the kernel layer. The kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0048] With the rapid development of display devices, people are not limited to controlling display devices with control devices, but want to more conveniently control display devices only with body movements or voice. Users can control display devices using gesture interaction. The gesture interaction methods that display devices can adopt may include static gestures and dynamic gestures. When using static gesture interaction, the display device can detect the gesture type according to the gesture type recognition algorithm, and perform corresponding control actions according to the gesture type. When using dynamic gesture interaction, the display device can detect the user's dynamic gestures, and then determine the gesture movement trajectory mapped to the display, thereby controlling the cursor to move along the gesture movement trajectory.

[0049] When the user uses dynamic gestures to control the movement of the cursor, the display device needs to continuously detect the user image. By identifying each frame of the user image, the user gesture in the image is obtained, and then the coordinates of each frame of the user gesture mapped to the display are determined, so as to control the cursor to move along these coordinates. However, considering many factors such as errors in camera shooting, non-standard user gestures, and errors in gesture recognition, the display device may not be able to recognize the gestures of some user images and thus cannot determine the corresponding coordinates, resulting in the inability to accurately obtain the corresponding gesture movement trajectory. Under normal circumstances, the cursor needs to move according to the position corresponding to each frame of the image to form a continuous motion trajectory. If the position corresponding to the intermediate frame image is missing, the cursor will not move, resulting in a stuck movement. The cursor will continue to move until the position corresponding to the next frame of the image is identified. However, if the position difference is too far, the cursor will suddenly jump, which seriously affects the user's viewing experience.

[0050] The present application provides a display device, including a display and a controller.

[0051] In some embodiments, in order to enable the display device to realize the function of performing gesture interaction with the user, the display device also includes an image input interface for connecting an image collector. The image collector can be a camera for collecting some image data. It should be noted that the camera can be used as an external device, which is externally connected to the display device through the image input interface, or it can be built into the display device as a detector. For a camera externally connected to the display device, the camera can be connected to the external device interface of the display device and connected to the display device. The user can use the camera to complete the photo or filming function on the display device, thereby collecting image data.

[0052] The camera may further include a lens assembly, in which a photosensitive element and a lens are provided. The lens can refract light through a plurality of lenses so that the light of the image of the scene can be irradiated on the photosensitive element. The photosensitive element can be selected based on the detection principle of CCD (Charge-coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) according to the specifications of the camera, convert the light signal into an electrical signal through a photosensitive material, and output the converted electrical signal into image data. The camera can also acquire image data frame by frame according to a set sampling frequency to form video stream data based on the image data.

[0053] In some embodiments, the camera built into the display device can also support lifting. That is, the camera can be set on a lifting mechanism. When image acquisition is required, the lifting mechanism is controlled to move through a specific lifting instruction, thereby driving the camera to rise for image acquisition. When image acquisition is not required, the lifting mechanism can also be controlled to move through a specific lifting instruction, thereby driving the camera to lower to hide the camera. Figure 5 A schematic diagram of a built-in camera of a display device in some embodiments is shown.

[0054] For an image acquisition device connected to a display device, it can be an independent peripheral device and connected to the display device through a specific data interface. Figure 6As shown, the image collector can be an independent camera device, and the display device can be provided with a Universal Serial Bus (USB) interface or a High Definition Multimedia Interface (HDMI) interface, and the image collector is connected to the display device via the USB interface or the HDMI interface. In order to facilitate the detection of the user's gesture interaction actions, in some embodiments, the image collector externally connected to the display device can be set at a position close to the display device, such as the image collector is clamped on the top of the display device by a clamping device, or the image collector is placed on a desktop near the display device.

[0055] Obviously, for the image collector externally connected to the display device, other connection methods can also be supported according to the specific hardware configuration of the display device. In some embodiments, the image collector can also establish a connection relationship with the display device through the communicator of the display device, and send the collected image data to the display device according to the data transmission protocol corresponding to the communicator. For example, the display device can be connected to the image collector through a local area network or the Internet. After the network connection is established, the image collector can send the collected data to the display device through the network transmission protocol.

[0056] In some embodiments, the image collector can also be connected to an external display device via a wireless network. For example, for a display device that supports a WiFi wireless network, a WiFi module is provided in its communicator. Therefore, the display device and the image collector can be connected to the same wireless network to establish a wireless connection. After the image collector collects the image data, the image data can be sent to a router device of the wireless network, and then forwarded to the display device by the router device. Obviously, the image collector can also access the display device via other wireless connection methods. Among them, the wireless connection method includes but is not limited to WiFi direct connection, cellular network, analog microwave, Bluetooth, infrared, etc.

[0057] In some embodiments, after the user controls the display device to turn on, the display device may display a user interface. Figure 7Schematic diagrams of user interfaces in some embodiments are shown. The user interface includes a first navigation bar 700, a second navigation bar 710, a function bar 720, and a content display area 730. The function bar 720 includes multiple function controls such as "watch history", "my favorites" and "my apps". Among them, the content displayed in the content display area 730 will change with the changes in the selected controls in the first navigation bar 700 and the second navigation bar 710. The user can control the display device to display the display panel corresponding to the control by touching a certain control. It should be noted that the user can also input the selection operation of the control in other ways, for example, using the voice control function or the search function to select a certain control.

[0058] Whether it is an image collector built into the display device or an image collector external to the display device, the user can start the image collector to collect image data through specific interactive instructions or application control during the use of the display device, and process the collected image data accordingly according to different needs. For example, a camera application can be installed in the display device, and these camera applications can call the camera to realize their respective related functions. The camera application refers to a camera application that needs to access the camera, which can process the image data collected by the camera to realize related functions, such as video chat. The user can view all the applications installed in the display device by touching the "My Application" control. The application list can be displayed in the display. When the user chooses to open a camera application, the display device can run the corresponding camera application, and the camera application can wake up the image collector, and the image collector can further detect the image data in real time and send it to the display device. The display device can further process these image data, such as controlling the display to display images, etc.

[0059] In some embodiments, the display device can interact with the user through gestures to identify the user's control instructions. The user can use static gestures to interact with the display device to input control instructions. Specifically, during the gesture interaction process, the user can make a specific gesture within the shooting range of the image collector, and the image collector can collect the user's gesture image and send the collected gesture image to the display device. The display device can further recognize the gesture image and detect the type of gesture in the image. The display device can pre-store a gesture interaction strategy to define which control instruction each type of gesture corresponds to. A gesture type can correspond to a control instruction. The display device can set a gesture for triggering a specific control instruction for different purposes according to different purposes. By comparing the type of gesture in the image with the corresponding relationship in the interaction strategy one by one, the control instruction corresponding to the gesture can be determined and implemented.

[0060] For example, when a user makes a gesture with five fingers together and the palm facing the image collector within the shooting range of the image collector, the display device can recognize the gesture in the gesture image captured by the image collector and determine the control instruction for the gesture as "pause / start playing". Finally, by running the control instruction, the current playback interface is paused or started. It should be noted that

[0061] In the above embodiment, gesture recognition adopts static gesture recognition, which can identify the gesture type and then determine the corresponding control instruction. Every time the user presents a static gesture, it means that the user has entered an independent control instruction, such as controlling the volume to increase by one. It should be noted that when the user maintains a static gesture for a long time, the display device may still determine that the user has entered a control instruction. Therefore, for some control instructions that require continuous operation, it is too cumbersome to use static gesture interaction.

[0062] For example, when a user wants to control the focus in the display to select a certain control, the focus may be moved down, right, and down in sequence. At this time, the user needs to constantly change static gestures to control the focus to move, resulting in a poor user experience. Alternatively, if the focus needs to move in one direction multiple times in a row, the user needs to make static gestures continuously. Since even if the user maintains a static gesture for a long time, it will be judged as inputting a control command, so the user needs to put down his hand after making a static gesture, and then make a static gesture again, which affects the user experience.

[0063] In some embodiments, the display device may also support dynamic gesture interaction. The dynamic gesture means that during an interaction, the user can input a control instruction to the display device using a dynamic gesture input method. It can be set as follows: a control instruction can be input to the display device through a series of dynamic gestures, multiple types of different control instructions can be input to the display device in sequence through different types of gestures, or multiple identical control instructions of the same type can be continuously input to the display device through the same type of gestures, thereby expanding the gesture interaction types of the display device and improving the richness of gesture interaction forms.

[0064] For example, if the user adjusts the gesture from five fingers open to five fingers together within 2 seconds, that is, inputs a grabbing action lasting 2 seconds, the display device can continuously obtain gesture images within the 2s detection period, and identify the gesture type in the gesture image frame by frame, thereby identifying the grabbing action according to the gesture changes in multiple frames of images. Finally, the control instruction corresponding to the grabbing action is determined, that is, "full screen / windowed play", and the control instruction is executed to adjust the size of the play window.

[0065] In some embodiments, when a user interface is displayed on a display device, a user can control the focus in the display to select a control and trigger it. Figure 7 As shown, the current focus selects the "My Application" control. Considering that it may be cumbersome for the user to control the movement of the focus using the control device, in order to increase the user experience, the user can also use dynamic gestures to select controls.

[0066] The display device may be provided with a cursor control mode. When the display device is in the cursor control mode, the original focus in the display may be changed to the cursor, such as Figure 8 As shown in the figure, the cursor selects the "My Application" control. The user can use gestures to control the cursor to move, thereby selecting a control to replace the original focus movement.

[0067] In some embodiments, the user can send a cursor control mode instruction to the display device by operating a designated button of the remote controller. In the actual application process, the correspondence between the cursor control mode instruction and the remote controller button is pre-bound. For example, a cursor control mode button is set on the remote controller. When the user touches the button, the remote controller sends the cursor control mode instruction to the controller, and the controller controls the display device to enter the cursor control mode. When the user touches the button again, the controller can control the display device to exit the cursor control mode.

[0068] In some embodiments, the correspondence between the cursor control mode instruction and multiple remote control buttons can be pre-bound, and when the user touches the multiple buttons bound to the cursor control mode instruction, the remote control issues the cursor control mode instruction.

[0069] In some embodiments, the user can use a sound collector of the display device, such as a microphone, to send a cursor control mode instruction to the display device through voice input, so that the display device enters the cursor control mode.

[0070] In some embodiments, the user may also send a cursor control mode instruction to the display device through a preset gesture or action. The display device may detect the user's behavior in real time through an image collector. When the user makes a preset gesture or action, it may be considered that the user has sent a cursor control mode instruction to the display device.

[0071] In some embodiments, when a user uses a smart device to control a display device, such as a mobile phone, a cursor control mode instruction may also be sent to the display device. In actual application, a control may be set in the mobile phone, through which a user may select whether to enter the cursor control mode, thereby sending a cursor control mode instruction to the display device.

[0072] In some embodiments, a cursor control mode option may be set in the UI interface of the display device, and when the user clicks on the option, the display device may be controlled to enter or exit the cursor control mode.

[0073] In some embodiments, to prevent the user from accidentally triggering the cursor control mode, when the controller receives the cursor control mode instruction, it can control the display to display cursor control mode confirmation information, so that the user can make a second confirmation whether to control the display device to enter the cursor control mode. Fig. 9 A schematic diagram showing cursor control mode confirmation information displayed in a display in some embodiments is shown.

[0074] When the display device enters the cursor control mode, the user can use gestures to control the cursor to move, thereby selecting the control to be triggered.

[0075] Fig.10 A flowchart of the interactions among components of a display device in some embodiments is shown.

[0076] In some embodiments, when it is detected that the display device enters the cursor control mode, the controller can wake up the image collector and send a start instruction to the image collector, thereby starting the image collector to capture images. At this time, the user can make dynamic gestures within the shooting range of the image collector, and the image collector can continuously capture multiple frames of user images along with the user's dynamic gestures. In the embodiments of the present application, user behavior images are used to refer to user images captured by the image collector.

[0077] Specifically, the image collector can capture user behavior images at a preset frame rate, for example, 30 frames (30FPS) of user behavior images per second. At the same time, the image collector can also send each frame of the captured user behavior image to the display device in real time. It should be noted that since the image collector sends the captured user behavior image to the display device in real time, the rate at which the display device acquires the user behavior image can be the same as the shooting frame rate of the image collector. For example, when the image collector captures images at a frame rate of 30 frames per second, the controller can also acquire the user behavior image at a frame rate of 30 frames per second.

[0078] In some embodiments, the image collector may collect several frames of user behavior images and send them to the display device in sequence. The display device may recognize each frame of the user behavior image one by one, thereby recognizing the user gestures contained in the user behavior image to determine the control instructions input by the user.

[0079] Specifically, for the collected user behavior images, the controller performs gesture recognition processing on the user behavior images. For example, a preset dynamic gesture recognition model may be used to process each frame of the user behavior image one by one.

[0080] The controller can input the user behavior image into the dynamic gesture recognition model, and the dynamic gesture recognition model can further recognize the user gestures contained in the image. For example, the position information of key points such as fingers, joints, and wrists contained in the user behavior image can be recognized. The key point position refers to the position coordinates of the key point in the user behavior image. After recognition, the user gesture information of each frame of the user behavior image can be output in sequence.

[0081] Fig.11 Schematic diagrams of user gestures in some embodiments are shown. It can be set that the key points used to characterize the user gestures include 21 finger key points. The dynamic gesture recognition model can confirm the user gestures in the user behavior image and identify the position information of the 21 finger key points of the user's hand, that is, the position coordinates located in the user behavior image, and the position information of each key point can be represented by the coordinates of the corresponding point.

[0082] It should be noted that the dynamic gesture recognition model may recognize the user gestures and obtain the position information of each finger key point when recognizing the user behavior image. At this time, the output user gesture information may include the position information of all finger key points. However, due to the influence of different user gestures, some finger key points may be covered by the user, resulting in the absence of these finger key points in the user behavior image. At this time, the dynamic gesture recognition model cannot obtain the position information of these finger key points, and the position information of these finger key points can only be null values. That is, in the user gesture information, the position information of the finger key points recognized by the dynamic gesture recognition model is included, and the position information of the finger key points that are not recognized is null.

[0083] In some embodiments, after the dynamic gesture recognition model obtains the user gesture information of each frame, it can be output to the controller. The controller can further determine the control instruction indicated by the user based on the user gesture information of each frame. Since the user wants to control the cursor to move, the control instruction indicated by the user can be considered as the position instruction indicated by the user to move the cursor. At this time, the controller can obtain the cursor position of each frame based on the user gesture information of each frame.

[0084] In some embodiments, considering that the computing power of the display device may be weak, if the display device is currently implementing some other functions, such as far-field voice, 4K video playback, etc., the display device will be in a high load state. At this time, if the frame rate of the user behavior image input to the dynamic gesture recognition model is high, the real-time data processing volume is too large, and the model may process the user behavior image at a slow rate, thereby making the cursor position acquisition rate slow, resulting in a relatively slow cursor movement in the display.

[0085] Therefore, the controller can first detect the current load rate of the display device. When the load rate is higher than a preset threshold, for example, higher than 60%, the controller can enable the dynamic gesture recognition model to process each frame of the user behavior image at equal intervals at a fixed period. For example, the fixed period can be set to process 15 frames of images per second. This allows the dynamic gesture recognition model to process images stably. When it is detected that the load rate of the display device is not higher than the preset threshold, the dynamic gesture recognition model can be made to process each frame of the user behavior image in real time. At this time, the controller can input the user behavior image sent by the image collector into the dynamic gesture recognition model in real time, and control the model to perform recognition. The dynamic gesture recognition model can also be made to process at equal intervals at a fixed period.

[0086] It should be noted that the rate at which the dynamic gesture recognition model outputs user gesture information and the rate at which it processes user behavior images can be the same. When the dynamic gesture recognition model processes images at regular intervals, it will output user gesture information at regular intervals. When the model processes images in real time, it will also output user gesture information in real time.

[0087] Fig.12 A schematic diagram of a process for determining a cursor position based on user gesture information in some embodiments is shown.

[0088] In some embodiments, in order to enable the cursor displayed on the display to generate a real-time motion trajectory according to the user's dynamic gestures so that the cursor smoothly follows the dynamic gestures, the controller can determine the cursor position of each frame according to the information indicated by the user's gestures.

[0089] Considering that when a user uses gestures to control the cursor, some frames of the gesture images of the user's movements captured continuously over a period of time may be blurred or the gestures may be blocked. At this time, the dynamic gesture recognition model cannot recognize the result and fails to obtain relevant information about the target gesture, such as the user gesture information being null. At this time, it is impossible to obtain the information indicated by the user based on the user gesture information, that is, it is impossible to obtain the cursor position. Therefore, the display device can predict the cursor position corresponding to the frame image to avoid the cursor being stuck, the trajectory being interrupted, or being lost when following the user gesture due to the lack of the cursor position.

[0090] The display device can obtain user gesture information based on the dynamic gesture recognition model, such as Fig.11 The position information of the finger key points shown is used to determine whether the information indicated by the user can be obtained. When the result of the dynamic gesture recognition model is empty, that is, the user gesture information is empty, the cursor position prediction can be performed.

[0091] In the embodiment of the present application, it can be set as follows: when a preset target gesture is detected, it is considered that the user indicates the position information of the cursor movement. The target gesture can be the user showing the preset finger key points. Fig.11 The user gesture diagram shown can be set as key point 9 as the control point for the user to indicate the movement of the cursor, that is, when the position information of the preset finger key point is detected, it is determined that the user indicates the movement of the cursor. The display device can further determine the position information of the cursor movement based on the preset finger key point position information.

[0092] Therefore, when the position information of the preset finger key point is detected in the user gesture information, the position information of the cursor movement can be obtained. In the embodiment of the present application, virtual position information is used to refer to the position information of the preset finger key point, that is, the position information of the target gesture in the user behavior image.

[0093] In some embodiments, the display device can detect whether each frame of user gesture information includes virtual position information. If a frame of user gesture information includes virtual position information, that is, the position information of the preset finger key point is identified, it is considered that the target gesture is detected in the frame of user behavior image, that is, the user specifically indicates how to move the cursor. At this time, the display device can determine the position information where the cursor needs to move based on the virtual position information.

[0094] If a frame of user gesture information does not include virtual position information, that is, the position information of the preset finger key points is null, it is considered that the target gesture is not detected in the user behavior image of this frame. At this time, the user does not specifically indicate how the cursor should move, and the display device needs to predict and supplement the position information where the cursor needs to move.

[0095] In some embodiments, for a certain frame of the target user behavior image, for the two situations of whether the target gesture is detected, the controller can respectively obtain the position information of the cursor to be moved.

[0096] If the target gesture is detected, that is, the frame user gesture information contains virtual position information, then the position information where the cursor needs to move can be obtained according to the virtual position information, that is, the cursor position corresponding to the target user behavior image.

[0097] Specifically, the virtual position information represents the position information of the preset finger key points identified in the target user behavior image, and is used to represent the position information of the user's target gesture. However, the position information is the position of the finger key points in the target user behavior image. Therefore, the display device can map the user's target gesture to the display to obtain the position of the cursor. It should be noted that when mapping the user's target gesture to the display, a reference can be made based on the initial position of the cursor. When the user's target gesture is detected for the first time, the position of the finger key points in the frame image is determined as the initial position of the cursor to form a mapping relationship. In subsequent mappings, the target gestures of subsequent users can be mapped to the display in sequence according to the preset mapping method, thereby obtaining the cursor position corresponding to each frame image.

[0098] In some embodiments, after obtaining the cursor position information, considering that the user's gesture movement is three-dimensional and moves in the air, the movement directions include not only up, down, left, right, but also front and back. During the cursor mapping process, if the gesture moves frequently and the gesture state is unstable, the cursor will shake and other problems. In order to make the cursor movement smoother and the user experience better, the display device can also adjust and optimize the cursor position so that the cursor can be dynamically anti-shake and the movement trajectory is smooth and stable.

[0099] The display device can map the target gesture in the target user behavior image to the display according to the virtual position information to obtain the original cursor position Fc. In the embodiment of the present application, the original cursor position refers to: the coordinates recognized by the dynamic gesture recognition model are directly mapped to the coordinates in the display. By adjusting and optimizing the original cursor position, the target cursor position can be obtained. In the embodiment of the present application, the target cursor position refers to: the coordinate position of the cursor actually displayed in the display after adjustment and optimization.

[0100] Specifically, the display device can adjust the original cursor position according to the following method:

[0101] The display device can obtain a first position value based on the cursor position Fp corresponding to the previous frame of the target user behavior image and a preset adjustment threshold, and can also obtain a second position value based on the original cursor position and the preset adjustment threshold. The target cursor position Fc1 corresponding to the target user behavior image can be obtained based on the first position value and the second position value. It can be expressed by Formula 1:

[0102] F c1 =E 1 *F p +(1-E 1 )*F c (1)

[0103] in:

[0104] F c1 Indicates the adjusted target cursor position;

[0105] E 1 Indicates the preset adjustment threshold;

[0106] F c Indicates the original cursor position before adjustment, F p Indicates the cursor position corresponding to the previous frame of user behavior image.

[0107] Through the preset adjustment threshold, the original cursor position can be adjusted according to the cursor position corresponding to the previous frame image, thereby reducing the jitter deviation that may occur in the target gesture of the frame to optimize the movement of the cursor.

[0108] The adjustment threshold can be preset according to the following method:

[0109]

[0110] in:

[0111] E 1 Indicates the preset adjustment threshold.

[0112] k represents the first adjustment parameter; g represents the second adjustment parameter; the first adjustment parameter and the second adjustment parameter are both numbers between 0 and 1, and can be set by relevant technical personnel.

[0113] S g Indicates the size of the target user behavior image. The size of the user behavior image refers to the size of the user behavior image relative to the display.

[0114] Specifically, the display device can display the captured user behavior image on the display, so that the user can intuitively determine the current gesture situation. Fig.13 A schematic diagram of a display showing a camera area in some embodiments is shown. The camera area displays the image captured by the camera, and the size of the entire camera area can be set by the display device. The user can choose to turn the camera area on or off, but when the camera area is turned off, its size is set to the same as when it is turned on.

[0115] S c Indicates the size of the control at the cursor position corresponding to the previous frame of the target user behavior image. Each time the cursor moves, it can be considered that the cursor selects a certain control. Therefore, the adjustment threshold can be set according to the control selected by the cursor in the previous frame.

[0116] S tv Indicates the size of the display.

[0117] After adjusting the original cursor position, the target cursor position corresponding to the target user behavior image may be determined, that is, the position to which the cursor needs to be moved.

[0118] In some embodiments, in the target user behavior image, if the user's target gesture is not detected, that is, the frame of user gesture information does not contain virtual position information, then the display device can predict the cursor position corresponding to the target user behavior image, so that the cursor can move normally.

[0119] Specifically, in order to better predict the position of the cursor, the display device can first determine the type of cursor movement. It should be noted that the types of cursor movement can be divided into two categories: linear motion and curvilinear motion. When the cursor moves along a straight line, it means that the user's gesture action is also moving along a straight line. It is relatively stable, and generally no frame loss occurs when shooting images. However, when the cursor moves along a curve, it means that the user's gesture action is also moving along the curve. At this time, compared to a straight line, the stability is poor, resulting in a slightly higher frame loss rate. Therefore, a threshold for detecting frame loss can be set in advance to determine whether the cursor moves in a straight line or a curve.

[0120] The display device can detect frame loss in several frames of images before the target user behavior image, which can be a preset number of user behavior images, for example, 20 frames of images, that is, the number of user behavior images in which the user's target gesture is not detected, and whether it exceeds a preset detection threshold. The detection threshold can be set to 0.

[0121] Therefore, it is possible to detect whether the number of images with frame loss in the first 20 frames is greater than 0, that is, to detect whether there are images with frame loss in the first 20 frames. If there is no frame loss, it is considered that the cursor is moving in a straight line, which is set as the first type of movement in the embodiment of the present application; if there is frame loss, it is considered that the cursor is moving in a curve, which is set as the second type of movement in the embodiment of the present application.

[0122] In some embodiments, when it is detected that the cursor is moving in a straight line, the display device may perform a first process on the target user behavior image to predict the target cursor position.

[0123] Fig.14 A schematic diagram of a cursor moving along a straight line in some embodiments is shown. The initial position of the cursor is A1, and the acquired cursor positions are A2, A3 and A4 in sequence. The cursor moves along a straight line, and A5 is the predicted target cursor position of this frame.

[0124] Specifically, the controller may obtain the historical cursor position offset according to the cursor positions corresponding to the first two frames of the target user behavior image, so as to characterize the last movement of the cursor.

[0125] The controller can obtain the cursor movement speed according to the historical cursor position offset and the first time. The first time refers to the time interval between the preset dynamic gesture recognition model processing the first two frames of user behavior images. Generally speaking, the time consumed by the dynamic gesture recognition model to process a frame of image is fixed. Therefore, the first time can also be considered as the time interval between the dynamic gesture recognition model outputting the user gesture information corresponding to the first two frames of user behavior images.

[0126] It should be noted that when the dynamic gesture recognition model processes images at fixed intervals, the first time is a fixed value and does not need to be obtained every time. When the dynamic gesture recognition model processes images in real time, it is necessary to obtain the time difference between the recognition results of the first two frames of images output by the model in real time.

[0127] The controller can obtain the target cursor position offset of the cursor according to the cursor movement speed, the second time and the preset first prediction threshold. The second time is: the time interval between the preset dynamic gesture recognition model processing the target user behavior image and the previous frame of user behavior image, that is, the time interval between the moment when the model outputs the recognition result of the previous frame of image and the moment when the model outputs the recognition result of the current frame of image. The controller can predict the movement of the cursor this time.

[0128] Finally, the controller can sum the coordinate position corresponding to the user behavior image of the previous frame and the offset of the target cursor position, and obtain the target cursor position by performing this offset movement at the position of the cursor of the previous frame.

[0129] The prediction method can be expressed by formula 3 and 4:

[0130] F 0 =v*Δt 0 *E 2 +F 0-1 (3)

[0131] v=(F 0-1 -F 0-2 ) / Δt (4)

[0132] in:

[0133] F 0 represents the target cursor position; v represents the speed of the cursor movement this time, and represents Δt 0 Indicates the second time;

[0134] S frepresents a preset first prediction threshold;

[0135] F 0-1 Indicates the coordinate position corresponding to the previous frame of user behavior image;

[0136] F 0-2 represents the coordinate position corresponding to the second frame of the user behavior image; Δt represents the first time.

[0137] The first prediction threshold can be preset according to the following method:

[0138]

[0139] in:

[0140] E 2 represents the first prediction threshold, which can be 0.6. 1 represents the first prediction parameter; a 2 The first prediction parameter and the second prediction parameter are both numbers between 0 and 1, and can be set by relevant technical personnel.

[0141] D f Indicates the processing rate of user behavior images by the preset dynamic gesture recognition model within the preset time.

[0142] C f Indicates the rate at which the image collector collects user behavior images within a preset time.

[0143] P f Indicates the frame rate of cursor movement within a preset time. The frame rate of cursor movement refers to the frequency of cursor movement times, which can also be considered as the number of times the cursor moves per unit time. The cursor moves from one cursor position to the next cursor position as one movement.

[0144] Specifically, the preset time may be 1 second. Therefore, the rate at which the model processes images, the rate at which the image collector captures images, and the frame rate at which the cursor moves within one second before the target user behavior image can be obtained. Further, a first prediction threshold can be set.

[0145] According to the above formula, the position coordinates of the cursor under linear motion can be predicted.

[0146] In some embodiments, when it is detected that the cursor is moving in a curve, the display device may perform a second process on the target user behavior image to predict the target cursor position.

[0147] Fig.15A schematic diagram of the movement of the cursor along a curve in some embodiments is shown. Among them, the initial position of the cursor is A1, and the cursor positions that have been acquired are A2-A9 in sequence. The image corresponding to cursor position A4 has the first frame loss phenomenon. Since it is the first frame loss, the current movement of the cursor (the movement between A1 and A4) is considered to be a linear movement. Positions A5 and A6 are coordinates obtained by mapping the user's target gesture. The image corresponding to cursor position A7 has the second frame loss phenomenon. Therefore, it is considered that the current movement of the cursor (the movement between A5 and A7) moves along the curve, and the cursor position A7 is obtained according to the prediction. Positions A8 and A9 are coordinates obtained by mapping the user's target gesture. At this time, the target user behavior image has frame loss phenomenon, which is the third frame loss as a whole (preset number of detections). At this time, it is considered that the cursor moves along the curve (the movement between A8 and A10), and the cursor position A10 can be predicted.

[0148] The target user behavior image has a second frame loss, so it is considered that the cursor moves along a curve, and the cursor position predicted by the target user behavior image may be A8.

[0149] It should be noted that when performing curved motion, the method of predicting the cursor position is similar to that of linear motion. In both cases, the last cursor movement, that is, the historical cursor position offset, can be obtained first.

[0150] The cursor movement speed is then acquired according to the historical cursor position offset and the first time, and the target cursor position offset of the cursor is acquired according to the cursor movement speed, the second time and a preset second prediction threshold.

[0151] Finally, the controller can calculate the difference between the coordinate position corresponding to the user behavior image of the previous frame and the offset of the target cursor position, and obtain the target cursor position by performing this offset movement at the position of the cursor of the previous frame.

[0152] The specific prediction method can be expressed by formula 6 and 7:

[0153] F 0 =F 0-1 -v*Δt 0 *E 3 (6)

[0154] v=(F 0-1 -F 0-2 ) / Δt (7)

[0155] in:

[0156] S b Represents the second prediction threshold, which may be 0.3.

[0157] Specifically, the second prediction threshold can be preset according to the following method:

[0158] E 3 =b*E 2 (8)

[0159] Wherein, b represents the third prediction parameter. The third prediction parameter is a number between 0 and 1, which can be set by relevant technical personnel, and can be 0.5.

[0160] According to the above formula, the position coordinates of the cursor under curved motion can be predicted.

[0161] In some embodiments, considering that multiple consecutive frames of user behavior images may have frame loss, a preset threshold for continuous frame loss may be set, which may be 4. Within this threshold, if the user behavior image continues to have frame loss, the display device may continue to predict the cursor position.

[0162] Specifically, before performing gesture recognition on the target user behavior image of this frame, the user behavior images with a preset threshold before the current frame image can be detected first, which can be 4 frames of user behavior images, to see whether the target gesture is not detected in all of these images, that is, whether the first 4 frames of the target user behavior image are all lost.

[0163] If yes, it can be considered that the user no longer indicates the cursor position by gesture. At this time, the user may have put down his hand and has determined the control that the cursor should select. At this time, the cursor can be controlled not to move, and this round of user gesture movement is considered to be over. The next round of gesture recognition can be performed until the camera captures the user gesture again.

[0164] If not, it is considered that the user is still using gestures to indicate the cursor position, but the first few frames are all lost due to some circumstances. At this time, the controller can continue to perform gesture recognition on the target user behavior image and determine the cursor position corresponding to the current frame image.

[0165] In some embodiments, the situation of predicting the cursor position will only occur after the cursor has started to move, that is, the first position of the cursor will not be predicted, but will only be obtained according to the user's instructions. Specifically, when the display device enters the cursor control mode, it can be set as follows: when the user's target gesture is detected for the first time, the cursor is allowed to start moving to avoid frame loss in the first frame image.

[0166] In some embodiments, after determining the target cursor position corresponding to the target user behavior image, the user's gesture movement trajectory can be determined based on the cursor position. Considering that the distance between the cursor positions of each two frames is relatively short, it can be considered that the cursor moves in a straight line between the cursor positions of the two frames. The target cursor position can be made to reach the target cursor position along a straight line from the cursor position of the previous frame. That is, the target cursor position and the cursor position of the previous frame are connected to obtain the gesture movement trajectory.

[0167] The controller can then move the cursor along the gesture movement trajectory.

[0168] In some embodiments, after the cursor moves along the gesture movement trajectory, the user may no longer control the cursor to move. At this time, the cursor may be located within the area of ​​a certain control, or it may be located at the edge of a certain control. When the cursor is located within the area of ​​a certain control, it can be determined that the user has selected the control, and the display device can allow the user to confirm whether to trigger the control. However, if the cursor is located at the edge of the control or outside the area, resulting in a failure to select a certain control, the display device cannot allow the user to confirm triggering the control.

[0169] Therefore, when the cursor does not clearly fall into the area of ​​a certain control, it is necessary to determine the control corresponding to when the cursor stops moving, that is, to determine the control finally selected by the user.

[0170] Specifically, the position information of the preset size can be determined according to the position of the cursor. For example, the preset size can be 500*500. For the cursor position (a, b), a region with a size of 500*500 can be determined with the coordinates as the center.

[0171] The controller can determine all the controls in the area and obtain the distance from all controls to the cursor. The distance from the control to the cursor is set as: the average distance from the midpoint of the four sides of the control to the cursor. Fig.16 As shown, the cursor is at point O. For a control A, the midpoints of its four sides are B1, B2, B3, and B4. The distances from the four midpoints to the cursor are X1, X2, X3, and X4. Therefore, the distance from the control to the cursor is: (X1+X2+X3+X4) / 4

[0172] In some embodiments, considering that when the size of a control is small, the distance from the midpoint of its four sides to the cursor may be short, thus affecting the judgment result. Therefore, the distance from each control to the cursor may also be determined according to the following method.

[0173] Specifically, in the embodiment of the present application, it is set that there are two positional relationships between the cursor and the control: one is that the cursor and the control are located in the same horizontal direction or the same vertical direction, and the other is that the cursor and the control are neither located in the same horizontal direction nor in the same vertical direction.

[0174] Fig.17 A schematic diagram showing the positional relationship between a cursor and a control in some embodiments.

[0175] The cursor position is (a, b). For a control, its size is set to width w and height h. The coordinates of the four vertices are: (x - w, y - h), (x + w, y - h), (x + w, y + h), (x - w, y + h). The two vertical lines corresponding to the two vertical sides of the control are L1 and L2 respectively, and the two horizontal lines corresponding to the two horizontal sides of the control are L3 and L4 respectively. In the embodiments of the present application, it is set that if the cursor is located within the area between the vertical lines, it is considered that the cursor and the control are in the same vertical direction; if the cursor is located within the area between the horizontal lines, it is considered that the cursor and the control are in the same horizontal direction. If the cursor is not located within these two areas, it is considered that the cursor and the control are neither in the same horizontal direction nor in the same vertical direction. As Fig.17 shown in, the cursor O1 and the control A are in the same vertical direction, the cursor O2 and the control A are in the same horizontal direction, and the cursor O3 and the control A are neither in the same horizontal direction nor in the same vertical direction.

[0176] Specifically, for all controls within the area, the relationship between the cursor position and the control position can be judged.

[0177] If x < a < x + w and y < b < y + h. It indicates that the cursor is within the area of this control. At this time, other controls do not need to be considered, and this control can be determined as the control selected by the user.

[0178] If x < a < x + w is satisfied, but y < b < y + h is not satisfied, then the cursor and the control are in the same vertical direction.

[0179] If x < a < x + w is not satisfied, but y < b < y + h is satisfied, then the cursor and the control are in the same horizontal direction.

[0180] If x < a < x + w is not satisfied and y < b < y + h is not satisfied, then the cursor and the control are neither in the same horizontal direction nor in the same vertical direction.

[0181] If the cursor and the control are in the same vertical direction or the same horizontal direction, the distance between the cursor and the control can be calculated according to the following method.

[0182] Respectively obtain the distances from the four sides of the control A to the cursor O: T1, T2, T3, T4. And take the result with the smallest value among the four distances as the distance between the cursor and the control, and the distance is: MIN(T1, T2, T3, T4).

[0183] If the cursor and the control are not in the same horizontal or vertical direction, the distance between the cursor and the control can be calculated as follows.

[0184] Get the distances from the four vertices of control A to the cursor O respectively: P1, P2, P3, P4. And take the smallest value among the four distances as the distance between the cursor and the control, which is: MIN(P1, P2, P3, P4).

[0185] The controller can set the control with the shortest distance as the control selected by the cursor.

[0186] When the user touches the confirmation key, the display device can trigger the control selected by the cursor.

[0187] The present application also provides a method for identifying a gesture trajectory, which is applied to a display device, such as Fig.18 As shown, the method includes:

[0188] Step 1801: Control the image collector to collect several frames of user behavior images of the user.

[0189] Step 1802: Perform gesture recognition processing on the user behavior image to obtain user gesture information of each frame of the user behavior image.

[0190] Step 1803: Acquire a cursor position corresponding to each frame of the user behavior image according to the user gesture information; the cursor position is a display position in the display where the user gesture is mapped to the user behavior image.

[0191] Step 1804: determine the user's gesture movement trajectory according to the cursor position, and control the cursor in the display to move along the gesture movement trajectory.

[0192] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0193] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0195] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, It is characterized in that include: monitor; An image input interface, configured to connect to an image collector; The controller is configured as: Control the image collector to collect several frames of user behavior images of the user; The user behavior image is processed based on a preset dynamic gesture recognition model to obtain user gesture information of each frame of the user behavior image; and whether the user gesture information includes virtual position information, where the virtual position information is position information of a preset target gesture in the user behavior image; If the user gesture information includes virtual position information, it is determined that the target gesture is detected in the user behavior image; if the user gesture information does not include virtual position information, it is determined that the target gesture is not detected in the user behavior image; Determine whether a target gesture is detected in the target user behavior image; If a target gesture is detected, a cursor position corresponding to the target user behavior image is obtained according to the virtual position information; the cursor position is a display position in the display where the user's target gesture is mapped to the user behavior image; If the target gesture is not detected, determining whether the number of user behavior images in which the target gesture is not detected among a preset number of user behavior images detected before the target user behavior image exceeds a preset detection threshold; If not, it is determined that the cursor should perform the first type of movement, and a historical cursor position offset is obtained according to the cursor position corresponding to the first two frames of user behavior images of the target user behavior image; a cursor movement speed is obtained according to the historical cursor position offset and the first time; and a target cursor position offset of the cursor is obtained according to the cursor movement speed, the second time and a preset first prediction threshold; The coordinate position corresponding to the previous frame of the user behavior image and the offset of the target cursor position are summed to obtain the target cursor position; the first time is: the time interval between the preset dynamic gesture recognition model processing the first two frames of the user behavior image; the second time is: the time interval between the preset dynamic gesture recognition model processing the target user behavior image and the previous frame of the user behavior image; If so, it is determined that the cursor should perform the second type of movement, and a historical cursor position offset is obtained according to the cursor position corresponding to the first two frames of the target user behavior image; a cursor movement speed is obtained according to the historical cursor position offset and the first time; and a target cursor position offset of the cursor is obtained according to the cursor movement speed, the second time and a preset second prediction threshold; Calculate the difference between the coordinate position corresponding to the previous frame of the user behavior image and the target cursor position offset to obtain the target cursor position; The user's gesture movement trajectory is determined according to the cursor position, and the cursor in the display is controlled to move along the gesture movement trajectory.

2. The display device according to claim 1, It is characterized in that The controller is further configured to: In the step of acquiring the cursor position corresponding to the target user behavior image according to the virtual position information, According to the virtual position information, mapping the user gesture of the target user behavior image to the display to obtain an original cursor position; Obtaining a first position value according to a cursor position corresponding to a previous frame of the target user behavior image and a preset adjustment threshold; obtaining a second position value according to the original cursor position and the preset adjustment threshold; The target cursor position corresponding to the target user behavior image is acquired according to the first position value and the second position value.

3. The display device according to claim 2, It is characterized in that The method for setting the preset adjustment threshold includes: in: E 1 represents a preset adjustment threshold; k represents a first adjustment parameter; g represents a second adjustment parameter; S g Indicates the size of the target user behavior image; S c Indicates the size of the control at the cursor position corresponding to the previous frame of the target user behavior image; S tv Indicates the size of the display.

4. The display device according to claim 1, It is characterized in that The method for setting the first prediction threshold includes: in: E 2 represents the first prediction threshold; a 1 represents the first prediction parameter; a 2 represents the second prediction parameter; D f Indicates the processing rate of the user behavior image by the preset dynamic gesture recognition model within the preset time; C f Indicates the rate at which the image collector collects user behavior images within a preset time; P f Indicates the frame rate of cursor movement within the preset time.

5. The display device according to claim 1, It is characterized in that The controller is further configured to: Before executing the step of predicting the cursor position corresponding to the target user behavior image, Detecting whether the target gesture is not detected in all user behavior images of a preset threshold before the target user behavior image; If yes, controlling the cursor not to move; If not, the step of predicting the cursor position corresponding to the target user behavior image is performed.

6. A gesture trajectory recognition method, applied to a display device, It is characterized in that The method comprises: Control the image collector to collect several frames of user behavior images of the user; The user behavior image is processed based on a preset dynamic gesture recognition model to obtain user gesture information of each frame of the user behavior image; and whether the user gesture information includes virtual position information, where the virtual position information is position information of a preset target gesture in the user behavior image. If the user gesture information includes virtual position information, it is determined that the target gesture is detected in the user behavior image; if the user gesture information does not include virtual position information, it is determined that the target gesture is not detected in the user behavior image; Determine whether a target gesture is detected in the target user behavior image; if a target gesture is detected, obtain a cursor position corresponding to the target user behavior image according to the virtual position information; the cursor position is a display position in the display where the user's target gesture is mapped in the user behavior image; If the target gesture is not detected, determining whether the number of user behavior images in which the target gesture is not detected among a preset number of user behavior images detected before the target user behavior image exceeds a preset detection threshold; If not, it is determined that the cursor should perform the first type of movement, and a historical cursor position offset is obtained according to the cursor position corresponding to the first two frames of user behavior images of the target user behavior image; a cursor movement speed is obtained according to the historical cursor position offset and the first time; a target cursor position offset of the cursor is obtained according to the cursor movement speed, the second time and a preset first prediction threshold; the coordinate position corresponding to the previous frame of user behavior image and the target cursor position offset are summed to obtain the target cursor position; the first time is: the time interval between the first two frames of user behavior image processed by the preset dynamic gesture recognition model; the second time is: the time interval between the target user behavior image and the previous frame of user behavior image processed by the preset dynamic gesture recognition model; If so, it is determined that the cursor should perform the second type of movement, and a historical cursor position offset is obtained according to the cursor position corresponding to the first two frames of the target user behavior image; a cursor movement speed is obtained according to the historical cursor position offset and the first time; a target cursor position offset of the cursor is obtained according to the cursor movement speed, the second time and a preset second prediction threshold; a difference is calculated between the coordinate position corresponding to the previous frame of the user behavior image and the target cursor position offset to obtain the target cursor position; The user's gesture movement trajectory is determined according to the cursor position, and the cursor in the display is controlled to move along the gesture movement trajectory.