Display screen operation method, electronic device, and readable storage medium

By acquiring and analyzing depth images, generating hand motion trajectories and generating operation instructions, the problem of inconvenient operation of display screens in the prior art is solved, convenient and precise long-distance control is achieved, and user experience is improved.

CN112714900BActive Publication Date: 2024-11-05SHENZHEN HEYTAP TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201880097732.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-10-29
Publication Date
2024-11-05
Estimated Expiration
2038-10-29

AI Technical Summary

Technical Problem

The operating method of display screens of existing electronic devices requires physical touch and is inconvenient to operate, especially when long distances or precise control is required.

Method used

By acquiring at least two depth images, the movement trajectory of the hand is generated, and the operation instructions of the virtual icon are generated according to the movement trajectory, the operation of the display screen is realized.

Benefits of technology

It improves the operational convenience and accuracy of the display, can effectively control components on the display at a long distance, and enhances the user interaction experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112714900B_ABST
    Figure CN112714900B_ABST
Patent Text Reader

Abstract

A display screen operation method, electronic device and computer-readable storage medium. The method includes: acquiring at least two depth images, wherein the depth images contain depth information of a hand; generating a hand motion trajectory of the hand according to the at least two depth images; generating an operation instruction of a virtual icon in the display screen according to the hand motion trajectory, and operating the virtual icon according to the operation instruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a display screen operating method, an electronic device, and a computer-readable storage medium. Background Art

[0002] The display screen of an electronic device can display various interfaces, such as an application interface, a system interface, a lock screen interface, etc. The display screen of an electronic device can be covered with a touch screen, and the user can operate the electronic device by touching the touch screen. When the user needs to operate the electronic device, since the display screen is covered with a touch screen, the user can operate the electronic device by sliding, pressing, clicking, etc. on the display screen with a finger. Summary of the invention

[0003] Embodiments of the present application provide a display screen operating method, an electronic device, and a computer-readable storage medium.

[0004] A display screen operating method, comprising:

[0005] Acquire at least two depth images, wherein the depth images contain depth information of the hand;

[0006] generating a hand motion trajectory of the hand according to the at least two depth images;

[0007] An operation instruction of a virtual icon in a display screen is generated according to the hand movement trajectory, and the virtual icon is operated according to the operation instruction.

[0008] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the following operations:

[0009] Acquire at least two depth images, wherein the depth images contain depth information of the hand;

[0010] generating a hand motion trajectory of the hand according to the at least two depth images;

[0011] An operation instruction of a virtual icon in a display screen is generated according to the hand movement trajectory, and the virtual icon is operated according to the operation instruction.

[0012] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following operations:

[0013] Acquire at least two depth images, wherein the depth images contain depth information of the hand;

[0014] generating a hand motion trajectory of the hand according to the at least two depth images;

[0015] An operation instruction of a virtual icon in a display screen is generated according to the hand movement trajectory, and the virtual icon is operated according to the operation instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A diagram showing an application environment of a display screen operation method in an embodiment;

[0018] Figure 2 is a flow chart of a display screen operation method in one embodiment;

[0019] Figure 3 A flow chart of a method for generating a hand motion trajectory in one embodiment;

[0020] Figure 4 A flow chart of a method for generating a hand motion trajectory in another embodiment;

[0021] Figure 5 is a flow chart of a method for operating a virtual icon in one embodiment;

[0022] Figure 6 is a schematic diagram of generating a first hand motion trajectory in one embodiment;

[0023] Figure 7 is a schematic diagram of generating a second hand motion trajectory in one embodiment;

[0024] Figure 8 is a structural block diagram of a display screen operating device in one embodiment;

[0025] Fig. 9 is a schematic diagram of the internal structure of an electronic device in one embodiment;

[0026] Fig.10 FIG. 4 is a schematic diagram of an image processing circuit in an embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first direction of motion may be referred to as a second direction of motion, and similarly, a second direction of motion may be referred to as a first direction of motion. Both the first direction of motion and the second direction of motion are directions of motion, but they are not the same direction of motion.

[0029] Figure 1 FIG. 1 is a schematic diagram of an application environment of a display screen operation method in an embodiment. Figure 1 As shown, the application environment includes an electronic device 110. The electronic device 110 can capture the surrounding environment through a camera, and the electronic device 110 can obtain at least two depth images, wherein the at least two depth images obtained by the electronic device 110 can be collected using a depth camera. For example, the electronic device 110 can collect at least two depth images through a time-of-flight TOF (Time of Flight) camera. The depth image collected by the electronic device 110 may include depth information of the hand. The electronic device 110 can generate a hand motion trajectory of the hand based on the at least two acquired depth images. The electronic device 110 can generate an operation instruction for a virtual icon in the display screen based on the hand motion trajectory, and the electronic device 110 can operate the virtual icon according to the operation instruction. It can be understood that the above-mentioned electronic device 110 can be a mobile phone, a computer, a tablet computer, a wearable device, etc., which is not limited here.

[0030] In one embodiment, a display screen operation method is provided, which is applied to the above electronic device as an example. Figure 2 As shown, the information processing method includes operations 202 to 206. Among them:

[0031] Operation 202 : Acquire at least two depth images, where the depth images contain depth information of the hand.

[0032] The pixel values ​​in the depth image are used to represent the distance from the camera to each point in the scene. The depth information is the distance between the object in the image and the camera, and the depth information of the hand is the distance between the hand and the camera in the scene.

[0033] Among them, the depth image can be obtained by the electronic device through a camera. Specifically, the electronic device can be provided with a camera for collecting depth images. For example, the camera provided on the electronic device can be a time of flight TOF (Time of Flight) camera. The TOF camera can collect depth images. Specifically, the TOF camera emits continuous light pulses to the target scene, and then uses a sensor to receive the light pulses reflected by the object. By comparing the phase difference between the emitted light pulse and the light pulse reflected by the object, the electronic device can calculate the transmission delay between the light pulses, and then obtain the distance of the object relative to the camera, and finally the electronic device can obtain a depth image. The electronic device can obtain at least two depth images containing the depth information of the hand collected by the camera, and the at least two depth images obtained by the electronic device can be continuous depth images. Among them, the depth information of the hand in the at least two depth images obtained by the electronic device can be different. For example, the electronic device can continuously collect at least two depth images when the hand moves through the camera. Since the hand is moving when the depth image is collected, the depth information of the hand in the at least two depth images collected by the electronic device is different.

[0034] In operation 204 , a hand motion trajectory is generated according to at least two depth images.

[0035] The hand motion trajectory refers to a series of uninterrupted marks formed when the hand moves. The hand motion trajectory can also be the starting point, the end point, and the motion trajectory connecting the starting point and the end point when the hand moves. After the electronic device obtains at least two consecutive depth images, it can generate the hand motion trajectory according to the hand motion information in each depth image. For example, the electronic device can record different hand motion information in at least two depth images respectively, and generate the hand motion trajectory according to the recorded different motion information.

[0036] Operation 206 : generating an operation instruction for a virtual icon on the display screen according to the hand movement trajectory, and operating the virtual icon according to the operation instruction.

[0037] A virtual icon may be an icon displayed on a display screen, and a virtual icon may have multiple states, for example, a virtual icon may be in an off state or in an on state. The state of the virtual icon may be switched automatically by the electronic device, or may be switched by the user through a control in the electronic device. The electronic device may pre-establish a mapping relationship between the hand position in the depth image and the virtual icon position in the display screen, so that the virtual icon position may move with the movement of the hand position. For example, when the user's hand moves from left to right, the virtual icon also moves from left to right in the display screen.

[0038] After acquiring the hand motion trajectory, the electronic device can generate an operation instruction for the virtual icon in the display screen according to the hand motion trajectory. For example, the operation instruction generated by the electronic device can be an operation instruction such as sliding or clicking. The electronic device can operate the virtual icon according to the generated operation instruction. For example, when the operation instruction generated by the electronic device is sliding, the electronic device can control the virtual icon to slide according to the sliding instruction; when the operation instruction generated by the electronic device is clicking, the electronic device can control the virtual icon to click the component in the display screen according to the clicking instruction.

[0039] In this embodiment, the electronic device acquires at least two depth images, and the depth images contain the depth information of the hand. The electronic device can generate the hand motion trajectory of the hand according to the at least two depth images, and generate the operation instructions of the virtual icon in the display screen according to the hand motion trajectory, and the electronic device can operate the virtual icon according to the operation instructions. The electronic device generates the hand motion trajectory according to the acquired multiple depth images, and generates the operation instructions of the virtual icon according to the hand motion trajectory. Since the electronic device generates the hand motion trajectory according to the depth image, and then generates the operation instructions, and operates the virtual icon according to the operation instructions, it can realize the control of the display screen of the electronic device at a distance, and improve the convenience of display screen operation.

[0040] In one embodiment, a display screen operation method provided may also include a process of acquiring a depth image, specifically including: acquiring a historical movement speed of the hand, acquiring an acquisition frame rate based on the historical movement speed, and acquiring at least two depth images based on the acquisition frame rate.

[0041] The historical movement speed of the hand may be the movement speed of the hand obtained by the electronic device before generating the current hand movement trajectory. Specifically, the electronic device may obtain the movement distance of the hand within a preset time length, thereby calculating the movement speed of the hand. The acquisition frame rate refers to the frequency at which the camera of the electronic device acquires frame images. The acquisition frame rate may be a specific value, for example, the acquisition frame rate may be 10 frames per second.

[0042] Different acquisition frame rates can be stored in the electronic device. The electronic device can obtain the corresponding acquisition frame rate according to the historical movement speed of the hand. The faster the historical movement speed of the hand obtained by the electronic device, the more frequent the movement of the hand, and the larger the corresponding acquisition frame rate; the slower the historical movement speed of the hand obtained by the electronic device, the less the movement of the hand, and the smaller the corresponding acquisition frame rate. For example, the historical movement speed of the hand obtained by the electronic device is 10cm / s (centimeter per second), and the corresponding acquisition frame rate is 10 frames / second; the historical movement speed of the hand obtained by the electronic device is 6cm / s (centimeter per second), and the corresponding acquisition frame rate is 12 frames / second. After the electronic device obtains the acquisition frame rate, it can control the camera to obtain at least two depth images according to the acquired acquisition frame rate.

[0043] In this embodiment, the electronic device can obtain the historical movement speed of the hand and obtain the acquisition frame rate according to the historical movement speed, and the electronic device can obtain at least two depth images according to the acquisition frame rate. The electronic device can control the acquisition frequency of the camera to obtain at least two depth images according to the historical movement speed of the hand, and select the corresponding acquisition frequency for different hand movement speeds to obtain at least two depth images, which can improve the accuracy of the acquired depth image and reduce the power consumption of the electronic device.

[0044] like Figure 3 As shown, in one embodiment, a display screen operation method provided may further include a process of generating a hand motion trajectory, and the specific operations include:

[0045] Operation 302: acquiring depth information of a hand in at least two depth images and corresponding position information of the hand in the at least two depth images.

[0046] The depth information of the hand may be the distance between the hand and the camera. The position information of the hand may be the specific position of the hand in the depth image. The specific position of the hand in the depth image may be represented by coordinates.

[0047] After acquiring at least two depth images, the electronic device can respectively acquire the depth information of the hand in the at least two depth images. For example, after the electronic device acquires five depth images, the electronic device can respectively acquire the depth information of the hand in the five depth images. The electronic device can respectively acquire the position information corresponding to the hand in at least two depth images. For example, after the electronic device acquires six depth images, the electronic device can respectively acquire the position information corresponding to the hand in the six depth images.

[0048] Operation 304 : acquiring the movement distance and movement direction of the hand according to the depth information and the position information.

[0049] After obtaining the depth information and position information of the hand, the electronic device can obtain the movement distance and direction of the hand according to the depth information and position information of the hand. Specifically, the electronic device can obtain the movement distance and direction of the hand in three-dimensional space according to the depth information of the hand in each depth image. For example, the electronic device obtains two depth images. In the first depth image, the distance between the hand and the camera of the electronic device is 15 cm. In the second depth image, the distance between the hand and the camera of the electronic device is 20 cm. The electronic device can obtain that the movement distance of the hand in three-dimensional space is 5 cm, and the movement direction is perpendicular to the camera of the electronic device and forward.

[0050] The electronic device can obtain the movement distance and movement direction of the hand in two-dimensional space through the position information of the hand in each depth image. For example, the electronic device obtains three depth images, with mm (Millimeter) as the unit, the coordinates of the position corresponding to the hand in the first depth image are (10, 20), the coordinates of the position corresponding to the hand in the second depth image are (15, 20), and the coordinates of the position corresponding to the hand in the third depth image are (20, 20). The electronic device can obtain that the movement distance of the hand in two-dimensional space is 10mm, and the movement direction is from left to right.

[0051] In operation 306 , a hand motion trajectory is generated according to the motion distance and the motion direction.

[0052] After obtaining the movement distance and movement direction of the hand, the electronic device can generate a hand movement trajectory of the hand according to the movement distance and movement direction. Specifically, in the hand movement trajectory generated by the electronic device, the length of the hand movement trajectory may be the movement distance of the hand, and the direction of the hand movement trajectory may be the movement direction of the hand. For example, the movement distance of the hand in three-dimensional space obtained by the electronic device is 5 cm (centimeter), and the movement direction is perpendicular to the camera of the electronic device and forward. The hand movement trajectory generated by the electronic device is a movement trajectory of 5 cm perpendicular to the camera and forward. For another example, the movement distance of the hand in a two-dimensional control obtained by the electronic device is 10 cm, and the movement direction is from left to right. The hand movement trajectory generated by the electronic device is a movement trajectory of 10 cm from left to right.

[0053] In this embodiment, the electronic device can obtain the depth information of the hand in at least two depth images, and the position information corresponding to the hand in at least two depth images, and the electronic device can obtain the movement distance and movement direction of the hand according to the depth information and the position information, and the electronic device can generate the hand movement trajectory according to the movement distance and the movement direction. The electronic device can obtain the movement distance and movement direction of the hand more accurately according to the depth information and the position information of the hand in at least two depth images, so that the generated hand movement trajectory is more accurate.

[0054] In one embodiment, a display screen operation method provided may also include a process of obtaining the movement distance and direction of the hand, specifically including: drawing a three-dimensional virtual canvas based on at least two depth images, determining the coordinate points of the hand in the three-dimensional virtual canvas based on the depth information and position information of the hand, and obtaining the movement distance and direction of the hand based on the coordinate points.

[0055] The three-dimensional virtual canvas can be implemented by an electronic device through an algorithm. After the electronic device obtains at least two depth images, the three-dimensional virtual canvas can be drawn according to the at least two depth images obtained. Since the electronic device obtains the depth information and position information of the hand, the electronic device can determine the coordinate point of the hand in the three-dimensional virtual canvas according to the depth information and position information of the hand. Specifically, the coordinate point of the hand in the three-dimensional virtual canvas obtained by the electronic device can be a three-dimensional coordinate point. The number of depth images obtained by the electronic device is at least two, and the electronic device can determine a coordinate point of the hand in the three-dimensional virtual canvas according to one depth image. The electronic device can determine at least two coordinate points of the hand in the three-dimensional virtual canvas according to at least two depth images, thereby obtaining the movement distance and movement direction of the hand according to at least two coordinate points.

[0056] In this embodiment, the electronic device can draw a three-dimensional virtual canvas based on the depth image, determine the coordinate points of the hand in the three-dimensional virtual canvas based on the depth information and position information of the hand, and the electronic device can obtain the movement distance and movement direction of the hand based on the coordinate points. By drawing a three-dimensional virtual canvas and obtaining the movement distance and movement direction of the hand based on the coordinate points of the hand in the three-dimensional virtual canvas, the electronic device can improve the accuracy of obtaining the movement distance and movement direction of the hand.

[0057] In another embodiment, Figure 4 As shown, a display screen operation method provided may also include a process of generating a hand motion trajectory, and the specific operations include:

[0058] Operation 402 : acquiring a first movement distance of the hand in a first movement direction according to the depth information of the hand in each depth image.

[0059] The movement direction of the hand acquired by the electronic device may include a first movement direction and a second movement direction. The movement distance of the hand acquired by the electronic device may include a first movement distance and a second movement distance.

[0060] The first movement direction may be based on the display screen of the electronic device, perpendicular to the outward movement direction of the display screen, perpendicular to the inward movement direction of the display screen, inward movement at a preset angle with the display screen, outward movement at a preset angle with the display screen, etc., without limitation herein, wherein the preset angle may be randomly generated by the electronic device. The first movement distance may be the distance the hand moves along the first movement direction.

[0061] After acquiring at least two depth images, the electronic device can acquire the first movement direction of the hand according to the depth information of the hand in each depth image, and the electronic device can acquire the first movement distance of the hand in the first movement direction. For example, the electronic device acquires three depth images, the distance between the hand and the camera in the first depth image is 20cm, the distance between the hand and the camera in the second depth image is 18cm, and the distance between the hand and the camera in the third depth image is 15cm. The electronic device can acquire that the first movement direction of the hand is inward movement perpendicular to the display screen, and the electronic device can acquire that the hand has moved 5cm inward perpendicular to the display screen.

[0062] Operation 404 : acquiring a second movement distance of the hand in a second movement direction according to the position information of the hand in each depth image.

[0063] The second movement direction may be a direction parallel to the movement of the display screen. For example, the second movement direction may be a direction parallel to the display screen moving to the left, moving to the right, moving upward, moving downward, moving to the upper right, moving to the lower left, etc., which is not limited here.

[0064] The position information of the hand in the depth image can be expressed by coordinates, and the position information can also be determined according to the size of the depth image, which is not limited here. After the electronic device obtains at least two depth images, it can obtain the position information of the hand in each depth image, and the electronic device can obtain the second movement direction of the hand and the second movement distance of the hand in the second movement direction according to the position information. For example, in units of mm, the electronic device obtains two depth images, the coordinates of the position corresponding to the hand in the first depth image are (10, 20), and the coordinates of the position corresponding to the hand in the second depth image are (15, 20). The electronic device can obtain that the second movement direction of the hand is parallel to the display screen and moves to the right, and the electronic device can obtain that the hand moves 10 mm to the right parallel to the display screen.

[0065] In operation 406 , a first hand motion trajectory is generated according to the first motion distance and the first motion direction, and a second hand motion trajectory is generated according to the second motion distance and the second motion direction.

[0066] The first hand motion trajectory may be a motion trajectory of the hand moving a first motion distance in a first motion direction; and the second motion trajectory may be a motion trajectory of the hand moving a second motion distance in a second motion direction.

[0067] After respectively acquiring the first movement distance and the first movement direction, the electronic device can generate a first hand movement trajectory in which the hand moves a first movement distance along the first movement direction; after respectively acquiring the second movement distance and the second movement direction, the electronic device can generate a second hand movement trajectory in which the hand moves a second movement distance along the second movement direction.

[0068] In this embodiment, the electronic device can obtain the first movement distance of the hand in the first movement direction based on the depth information of the hand in each depth image, and the electronic device can obtain the second movement distance of the hand in the second movement direction based on the position information of the hand in each depth image. The electronic device can generate a first hand movement trajectory based on the first movement distance and the first movement direction, and generate a second hand movement trajectory based on the second movement distance and the second movement direction. The electronic device can generate different movement trajectories according to different movement directions and different movement distances, thereby improving the accuracy of generating movement trajectories.

[0069] like Figure 5 As shown, in one embodiment, a display screen operation method provided may also include a process of operating a virtual icon, and the specific operations include:

[0070] Operation 502 : generating a first operation instruction of a virtual icon in a display screen according to a first hand motion trajectory, and controlling the virtual icon to operate a component in the display screen according to the first operation instruction.

[0071] The first operation instruction may be an operation instruction such as clicking or dragging, which is not limited here. The electronic device may generate a first operation instruction for a virtual icon in a display screen according to the first hand motion trajectory. Then, the electronic device may control the virtual icon to operate the component in the display screen according to the generated first operation instruction. For example, the first hand motion trajectory obtained by the electronic device is that the hand moves 5 cm inward perpendicular to the display screen, and the electronic device may obtain the corresponding first operation instruction as a click operation instruction, and the electronic device may control the virtual icon to click the component in the display screen.

[0072] Operation 504 : Generate a second operation instruction for the virtual icon on the display screen according to the second hand motion trajectory, wherein the second operation instruction includes the virtual icon motion trajectory.

[0073] The electronic device can pre-establish a mapping relationship between the hand position and the virtual icon position. Specifically, the electronic device can determine the coordinate point of the hand in the three-dimensional virtual canvas based on the depth information and position information of the hand, that is, determine the hand position of the hand in the three-dimensional virtual canvas, and the electronic device can map the hand position to the position of the virtual icon in the display screen. After the electronic device obtains the second hand motion trajectory formed according to the change of the hand position, it can obtain the motion trajectory of the virtual icon in the display screen corresponding to the second hand motion trajectory, and the electronic device can generate a second operation instruction for the virtual icon. Among them, the second operation instruction includes the virtual icon motion trajectory.

[0074] Operation 506 : Control the virtual icon to move along the motion trajectory of the virtual icon according to the second operation instruction.

[0075] Since the second operation instruction generated by the electronic device includes the virtual icon motion trajectory, the electronic device can control the virtual icon on the display screen to move according to the virtual icon motion trajectory according to the second operation instruction. For example, if the second hand motion trajectory acquired by the electronic device is that the hand moves 10mm to the right parallel to the display screen, then the virtual icon moves 10mm to the right on the display screen, and the second operation instruction generated by the electronic device includes the virtual icon motion trajectory that the virtual icon moves 10mm to the right on the display screen, the electronic device can control the virtual icon to move 10mm to the right on the display screen according to the second operation instruction.

[0076] In this embodiment, the electronic device generates a first operation instruction for a virtual icon in the display screen according to the first hand motion trajectory, controls the virtual icon to operate a component in the display screen according to the first operation instruction, and generates a second operation instruction for the virtual icon in the display screen according to the second hand motion trajectory, the second operation instruction includes the virtual icon motion trajectory, and the electronic device controls the virtual icon to move according to the virtual icon motion trajectory according to the second operation instruction. The electronic device can generate different operation instructions according to different hand motion trajectories, which can improve the convenience of display screen operation.

[0077] In one embodiment, Figure 6 As shown in FIG. , a schematic diagram for generating a first hand motion trajectory is provided. Figure 6As shown, the first movement direction of the hand acquired by the electronic device is inward movement perpendicular to the display screen, and the movement distance of the hand in the first movement direction is the distance between the hand position 610 moving inward perpendicular to the display screen to the hand position 620. When the hand moves inward perpendicular to the display screen from the hand position 610 to the hand position 620, the first hand movement trajectory generated by the electronic device is the movement trajectory from the hand position 610 to the hand position 620. The electronic device can generate a first operation instruction for the virtual icon 630 in the display screen according to the first hand movement trajectory. For example, the first operation instruction generated by the electronic device is a click operation instruction, and the electronic device can control the virtual icon 630 to click on the application 2 in the display screen to operate according to the click operation instruction.

[0078] like Figure 7 As shown, in one embodiment, a schematic diagram of generating a second hand motion trajectory is provided. Figure 7 As shown, the second movement direction of the hand acquired by the electronic device is parallel to the display screen and moves to the right, and the movement distance of the hand in the second movement direction is the distance between the hand position 710 moving to the right parallel to the display screen and the hand position 720. Figure 7 As shown, hand position 710 corresponds to virtual icon position 730 in the display screen, and hand position 720 corresponds to virtual icon position 740 in the display screen. When the hand moves from hand position 710 to hand position 720 parallel to the display screen to the right, the second hand motion trajectory generated by the electronic device is the motion trajectory from hand position 710 to hand position 720. The electronic device can generate a second operation instruction for the virtual icon in the display screen according to the second hand motion trajectory, wherein the second operation instruction includes the virtual icon motion trajectory. Figure 7 As shown, the motion trajectory of the virtual icon in the second operation instruction is the motion trajectory from the virtual icon position 730 to the virtual icon position 740 .

[0079] In one embodiment, a display screen operation method provided may also include a process of issuing a prompt message, specifically including: obtaining the interval duration between the moment when the hand motion trajectory is generated and the current moment, and when the interval duration exceeds a duration threshold and no operation instruction is generated, issuing a prompt message, and the prompt message is used to indicate that the hand motion trajectory is invalid.

[0080] When the electronic device generates a hand motion trajectory based on a depth image, it can obtain the time when the hand motion trajectory is generated. The time when the hand motion trajectory is generated and the current time can be specific time points. For example, the time when the hand motion trajectory is generated can be 11:19:20, and the current time can be 11:19:45. The interval duration refers to the duration between two moments. For example, the time when the hand motion trajectory is generated is 11:19:20, and the current time is 11:19:45. The interval duration between the time when the hand motion trajectory is generated and the current time is 20 seconds.

[0081] The duration threshold may be a pre-set duration, for example, 20 seconds, 27 seconds, etc. The duration threshold may be set by the user or automatically generated by the electronic device. The prompt message may be used to prompt the user that the hand motion trajectory is invalid. The prompt message may be in the form of a pop-up window, a vibration prompt, a ring tone prompt, etc., which are not limited here.

[0082] When the electronic device generates the hand motion trajectory according to the depth image, it can obtain the time when the hand motion trajectory is generated and the current time. The electronic device can obtain the interval between the time when the hand motion trajectory is generated and the current time. When the interval time exceeds the time threshold, the electronic device can detect whether an operation instruction is generated. When no operation instruction is generated, the electronic device can issue a prompt message to the user that the hand motion trajectory is invalid.

[0083] In this embodiment, by obtaining the interval time between the moment when the hand motion trajectory is generated and the current moment, when the interval time exceeds the time threshold and no operation instruction is generated, a prompt message is issued, and the prompt message is used to indicate that the hand motion trajectory is invalid. The electronic device determines whether the hand motion trajectory is valid based on the interval time between the moment when the hand motion trajectory is generated and the current moment and whether an operation instruction is generated, and issues a prompt message, and can prompt the user when the hand motion trajectory is invalid so that the user can change the hand motion in time.

[0084] In one embodiment, a display screen operation method is provided, and specific operations for implementing the method are as follows:

[0085] (1) The electronic device can obtain the historical movement speed of the hand and obtain the acquisition frame rate according to the historical movement speed.

[0086] (2) The electronic device can acquire at least two depth images according to the acquisition frame rate.

[0087] (3) The electronic device can obtain the depth information of the hand in at least two depth images, and the corresponding position information of the hand in at least two depth images.

[0088] (4) The electronic device can draw a three-dimensional virtual canvas based on the at least two depth images, and determine the coordinate point of the hand in the three-dimensional virtual canvas based on the depth information and position information of the hand.

[0089] (5) The electronic device can obtain a first movement distance of the hand in the first movement direction and a second movement distance of the hand in the second movement direction according to the coordinate points.

[0090] (6) The electronic device may generate a first hand motion trajectory according to the first motion distance and the first motion direction, and may generate a second hand motion trajectory according to the second motion distance and the second motion direction.

[0091] (7) The electronic device can obtain a first interval duration between the moment when the first hand motion trajectory is generated and the current moment.

[0092] (8) When the first interval duration is less than the first duration threshold and a first operation instruction of a virtual icon in a display screen is generated according to the first hand motion trajectory, the electronic device may control the virtual icon to operate a component in the display screen according to the first operation instruction; when the first interval duration exceeds the first duration threshold and the first operation instruction is not generated, the electronic device may issue a first prompt message, and the first prompt message is used to prompt the user that the first hand motion trajectory is invalid.

[0093] (9) The electronic device may obtain a second interval duration between the moment when the second hand motion trajectory is generated and the current moment.

[0094] (10) When the second interval duration is less than the second duration threshold and a second operation instruction for a virtual icon on the display screen is generated according to the second hand motion trajectory, the second operation instruction includes the virtual icon motion trajectory, and the electronic device can control the virtual icon to move according to the virtual icon motion trajectory according to the second operation instruction; when the second interval duration exceeds the second duration threshold and the second operation instruction is not generated, the electronic device can issue a second prompt message, and the second prompt message is used to prompt the user that the second hand motion trajectory is invalid.

[0095] It should be understood that, although each operation in the above-mentioned each flow chart is shown successively according to the indication of arrow, these operations are not necessarily performed successively according to the order indicated by arrow.Unless there is clear explanation in this article, the execution of these operations does not have strict order restriction, and these operations can be performed in other order.And, at least a part of the operations in the above-mentioned each flow chart can include a plurality of sub-operations or a plurality of stages, and these sub-operations or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-operations or stages is not necessarily performed successively, but can be performed in turn or alternately with at least a part of other operations or sub-operations of other operations or stages.

[0096] In one embodiment, Figure 8 As shown, a display screen operation device is provided, including: a depth image acquisition module 810, a hand motion trajectory generation module 820 and an operation instruction generation module 830, wherein:

[0097] The depth image acquisition module 810 is used to acquire at least two depth images, where the depth images contain depth information of the hand.

[0098] The hand motion trajectory generating module 820 is used to generate the hand motion trajectory of the hand according to at least two depth images.

[0099] The operation instruction generating module 830 is used to generate an operation instruction for a virtual icon in the display screen according to the hand movement trajectory, and operate the virtual icon according to the operation instruction.

[0100] In this embodiment, the electronic device can obtain at least two depth images, the depth images contain the depth information of the hand, the electronic device can generate the hand motion trajectory of the hand according to the at least two depth images, the electronic device can generate the operation instructions of the virtual icon in the display screen according to the hand motion trajectory, and the electronic device can operate the virtual icon according to the operation instructions. The electronic device generates the hand motion trajectory according to the acquired multiple depth images, and generates the operation instructions of the virtual icon according to the hand motion trajectory. Since the electronic device generates the hand motion trajectory according to the depth image, and then generates the operation instructions, and operates the virtual icon according to the operation instructions, it can realize the control of the display screen of the electronic device at a distance, and improve the convenience of display screen operation.

[0101] In one embodiment, the depth image acquisition module 810 is further used to acquire the historical movement speed of the hand, acquire the acquisition frame rate according to the historical movement speed, and acquire at least two depth images according to the acquisition frame rate.

[0102] In one embodiment, the hand motion trajectory generation module 820 is also used to obtain the depth information of the hand in at least two depth images, and the corresponding position information of the hand in at least two depth images; obtain the movement distance and movement direction of the hand according to the depth information and the position information; and generate the hand motion trajectory according to the movement distance and the movement direction.

[0103] In one embodiment, the hand motion trajectory generation module 820 is also used to draw a three-dimensional virtual canvas based on at least two depth images, determine the coordinate points of the hand in the three-dimensional virtual canvas based on the depth information and position information of the hand, and obtain the movement distance and direction of the hand based on the coordinate points.

[0104] In one embodiment, the movement direction includes a first movement direction and a second movement direction, the movement distance includes a first movement distance and a second movement distance, and the hand movement trajectory includes a first hand movement trajectory and a second hand movement trajectory. The hand movement trajectory generation module 820 is also used to obtain the first movement distance of the hand moving in the first movement direction based on the depth information of the hand in each depth image, and to obtain the second movement distance of the hand moving in the second movement direction based on the position information of the hand in each depth image; the hand movement trajectory acquisition module is also used to generate a first hand movement trajectory based on the first movement distance and the first movement direction, and to generate a second hand movement trajectory based on the second movement distance and the second movement direction.

[0105] In one embodiment, the operation instruction generation module 830 is also used to generate a first operation instruction for a virtual icon in a display screen according to a first hand motion trajectory, control the virtual icon to operate a component in the display screen according to the first operation instruction, and generate a second operation instruction for the virtual icon in the display screen according to a second hand motion trajectory, wherein the second operation instruction includes an icon motion trajectory, and the virtual icon is controlled to move according to the icon motion trajectory according to the second operation instruction.

[0106] In one embodiment, a display screen operation device provided may also include an information prompt module, which is used to obtain the interval duration between the moment when the hand motion trajectory is generated and the current moment; when the interval duration exceeds a duration threshold and no operation instruction is generated, a prompt message is issued, and the prompt message is used to indicate that the hand motion trajectory is invalid.

[0107] The division of the modules in the above display screen operating device is only for illustration. In other embodiments, the display screen operating device may be divided into different modules as needed to complete all or part of the functions of the above display screen operating device.

[0108] For the specific definition of the display screen operation device, please refer to the definition of the display screen operation method above, which will not be repeated here. Each module in the above display screen operation device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] The implementation of each module in the display screen operation device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, the operation of the method described in the embodiment of the present application is implemented.

[0110] In one embodiment, Fig. 9 As shown, a schematic diagram of the internal structure of an electronic device is provided. The electronic device includes a processor, a memory, a display, and a network interface connected via a system bus. Among them, the processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory is used to store data, programs, and / or instruction codes, etc., and at least one computer program is stored on the memory, and the computer program can be executed by the processor to implement the display screen operation method applicable to the electronic device provided in the embodiment of the present application. The memory may include non-volatile storage media such as a disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random-Access-Memory, RAM), etc. For example, in one embodiment, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a database, and a computer program. The computer program can be executed by the processor to implement a display screen operation method provided in each embodiment of the present application. The internal memory provides a cache operating environment for the operating system, database, and computer program in the non-volatile storage medium. The display can be used to display an upload interface, etc., and can also be used to display a light effect model. The network interface may be an Ethernet card or a wireless network card, etc., which is used to communicate with external electronic devices, for example, it can be used to communicate with a server or other electronic devices.

[0111] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] The embodiment of the present application further provides an electronic device. The electronic device includes an image processing circuit, which can be implemented using hardware and / or software components and can include various processing units that define an ISP (Image Signal Processing) pipeline. Fig.10 FIG. 1 is a schematic diagram of an image processing circuit in one embodiment. Fig.10 As shown, for ease of explanation, only various aspects of the image processing technology related to the embodiments of the present application are shown.

[0113] like Fig.10As shown, the image processing circuit includes an ISP processor 1040 and a control logic 1050. The image data captured by the imaging device 1010 is first processed by the ISP processor 1040, and the ISP processor 1040 analyzes the image data to capture image statistics that can be used to determine and / or one or more control parameters of the imaging device 1010. The imaging device 1010 may include a camera having one or more lenses 1012 and an image sensor 1014. The image sensor 1014 may include a color filter array (such as a Bayer filter), and the image sensor 1014 may obtain light intensity and wavelength information captured by each imaging pixel of the image sensor 1014, and provide a set of raw image data that can be processed by the ISP processor 1040. The sensor 1020 (such as a gyroscope) may provide the acquired image processing parameters (such as anti-shake parameters) to the ISP processor 1040 based on the interface type of the sensor 1020. The sensor 1020 interface may utilize a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0114] In addition, the image sensor 1014 may also send raw image data to the sensor 1020 , and the sensor 1020 may provide the raw image data to the ISP processor 1040 based on the interface type of the sensor 1020 , or the sensor 1020 may store the raw image data in the image memory 1030 .

[0115] The ISP processor 1040 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 1040 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed at the same or different bit depth precisions.

[0116] The ISP processor 1040 may also receive image data from the image memory 1030. For example, the sensor 1020 interface sends raw image data to the image memory 1030, and the raw image data in the image memory 1030 is then provided to the ISP processor 1040 for processing. The image memory 1030 may be a portion of a memory device, a storage device, or an independent dedicated memory within an electronic device, and may include a DMA (Direct Memory Access) feature.

[0117] When receiving raw image data from the image sensor 1014 interface or from the sensor 1020 interface or from the image memory 1030, the ISP processor 1040 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to the image memory 1030 for additional processing before being displayed. The ISP processor 1040 receives the processed data from the image memory 1030 and processes the processed data in the original domain and in the RGB and YCbCr color spaces. The image data processed by the ISP processor 1040 may be output to the display 1070 for viewing by the user and / or further processed by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 1040 may also be sent to the image memory 1030, and the display 1070 may read the image data from the image memory 1030. In one embodiment, the image memory 1030 may be configured to implement one or more frame buffers. In addition, the output of the ISP processor 1040 may be sent to the encoder / decoder 1060 to encode / decode the image data. The encoded image data may be stored and decompressed before being displayed on the display 1070 device. The encoder / decoder 1060 may be implemented by a CPU or a GPU or a coprocessor.

[0118] The statistical data determined by the ISP processor 1040 may be sent to the control logic 1050 unit. For example, the statistical data may include image sensor 1014 statistical information such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens 1012 shading correction, etc. The control logic 1050 may include a processor and / or microcontroller that executes one or more routines (e.g., firmware) that may determine control parameters of the imaging device 1010 and control parameters of the ISP processor 1040 based on the received statistical data. For example, the control parameters of the imaging device 1010 may include sensor 1020 control parameters (e.g., gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, lens 1012 control parameters (e.g., focal length for focus or zoom), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for auto white balance and color adjustment (e.g., during RGB processing), as well as lens 1012 shading correction parameters.

[0119] The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors perform the operation of the display screen operation method.

[0120] A computer program product comprising instructions, when the computer program product is run on a computer, enables the computer to execute a display screen operation method.

[0121] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A display screen operation method, characterized in that: include: Acquire at least two depth images, wherein the depth images contain depth information of the hand; Acquire depth information of the hand in the at least two depth images, and corresponding position information of the hand in the at least two depth images respectively; Drawing a three-dimensional virtual canvas according to the at least two depth images, and determining a coordinate point of the hand in the three-dimensional virtual canvas according to the depth information and position information of the hand; Acquire the movement distance and movement direction of the hand according to the coordinate points; the movement distance of the hand includes a first movement distance and a second movement distance, and the movement direction of the hand includes a first movement direction and a second movement direction; Acquire a first movement distance of the hand in a first movement direction according to the depth information of the hand in each depth image; Acquire a second movement distance of the hand in a second movement direction according to the position information of the hand in each depth image; generating a first hand motion trajectory according to the first motion distance and the first motion direction, and generating a second hand motion trajectory according to the second motion distance and the second motion direction; the first hand motion trajectory and the second hand motion trajectory both belong to hand motion trajectories; generating a first operation instruction for a virtual icon in the display screen according to the first hand motion trajectory, and controlling the virtual icon to operate a component in the display screen according to the first operation instruction; generating a second operation instruction of a virtual icon on the display screen according to the second hand motion trajectory, wherein the second operation instruction includes the virtual icon motion trajectory; The virtual icon is controlled to move according to the second operation instruction along the motion trajectory of the virtual icon.

2. The method according to claim 1, characterized in that The acquiring of at least two depth images comprises: Acquire the historical movement speed of the hand, and acquire the acquisition frame rate according to the historical movement speed; At least two depth images are acquired according to the acquisition frame rate.

3. The method according to any one of claims 1 to 2, characterized in that: The method further comprises: Obtaining the interval between the moment when the hand motion trajectory is generated and the current moment; When the interval duration exceeds a duration threshold and no operation instruction is generated, a prompt message is issued, where the prompt message is used to prompt that the hand motion trajectory is invalid.

4. A display screen operating device, characterized in that: include: A depth image acquisition module, used to acquire at least two depth images, wherein the depth images contain depth information of the hand; A hand motion trajectory generation module, used for drawing a three-dimensional virtual canvas according to the at least two depth images, determining the coordinate point of the hand in the three-dimensional virtual canvas according to the depth information and position information of the hand; obtaining the movement distance and movement direction of the hand according to the coordinate point; the movement distance of the hand includes a first movement distance and a second movement distance, and the movement direction of the hand includes a first movement direction and a second movement direction; obtaining the depth information of the hand in the at least two depth images, and the corresponding position information of the hand in the at least two depth images respectively; obtaining the first movement distance of the hand moving in the first movement direction according to the depth information of the hand in each depth image; obtaining the second movement distance of the hand moving in the second movement direction according to the position information of the hand in each depth image; generating a first hand motion trajectory according to the first movement distance and the first movement direction, and generating a second hand motion trajectory according to the second movement distance and the second movement direction; the first hand motion trajectory and the second hand motion trajectory both belong to hand motion trajectories; an operation instruction generating module, configured to generate a first operation instruction for a virtual icon in the display screen according to the first hand motion trajectory, and control the virtual icon to operate a component in the display screen according to the first operation instruction; A second operation instruction for a virtual icon in a display screen is generated according to the second hand motion trajectory, wherein the second operation instruction includes the virtual icon motion trajectory; and the virtual icon is controlled to move according to the virtual icon motion trajectory according to the second operation instruction.

5. The device according to claim 4, characterized in that The depth image acquisition module is further used to acquire the historical movement speed of the hand, acquire a capture frame rate according to the historical movement speed; and acquire at least two depth images according to the capture frame rate.

6. The device according to any one of claims 4 to 5, characterized in that The device also includes an information prompt module; the information prompt module is used to obtain the interval duration between the moment when the hand motion trajectory is generated and the current moment; when the interval duration exceeds a duration threshold and no operation instruction is generated, a prompt message is issued, and the prompt message is used to prompt that the hand motion trajectory is invalid.

7. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following operations: Acquire at least two depth images, wherein the depth images contain depth information of the hand; Acquire depth information of the hand in the at least two depth images, and corresponding position information of the hand in the at least two depth images respectively; Drawing a three-dimensional virtual canvas according to the at least two depth images, and determining a coordinate point of the hand in the three-dimensional virtual canvas according to the depth information and position information of the hand; Acquire the movement distance and movement direction of the hand according to the coordinate points; the movement distance of the hand includes a first movement distance and a second movement distance, and the movement direction of the hand includes a first movement direction and a second movement direction; Acquire a first movement distance of the hand in a first movement direction according to the depth information of the hand in each depth image; Acquire a second movement distance of the hand in a second movement direction according to the position information of the hand in each depth image; generating a first hand motion trajectory according to the first motion distance and the first motion direction, and generating a second hand motion trajectory according to the second motion distance and the second motion direction; the first hand motion trajectory and the second hand motion trajectory both belong to hand motion trajectories; generating a first operation instruction for a virtual icon in the display screen according to the first hand motion trajectory, and controlling the virtual icon to operate a component in the display screen according to the first operation instruction; generating a second operation instruction of a virtual icon on the display screen according to the second hand motion trajectory, wherein the second operation instruction includes the virtual icon motion trajectory; The virtual icon is controlled to move according to the second operation instruction along the motion trajectory of the virtual icon.

8. The electronic device according to claim 7, characterized in that: When the computer program is executed by the processor, the processor further performs the following operations when acquiring at least two depth images: Acquire the historical movement speed of the hand, and acquire the acquisition frame rate according to the historical movement speed; At least two depth images are acquired according to the acquisition frame rate.

9. The electronic device according to any one of claims 7 to 8, characterized in that: When the computer program is executed by the processor, the processor performs the following operations: Obtaining the interval between the moment when the hand motion trajectory is generated and the current moment; When the interval duration exceeds a duration threshold and no operation instruction is generated, a prompt message is issued, where the prompt message is used to prompt that the hand motion trajectory is invalid.

10. A computer-readable storage medium, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following operations: Acquire at least two depth images, wherein the depth images contain depth information of the hand; Acquire depth information of the hand in the at least two depth images, and corresponding position information of the hand in the at least two depth images respectively; Drawing a three-dimensional virtual canvas according to the at least two depth images, and determining a coordinate point of the hand in the three-dimensional virtual canvas according to the depth information and position information of the hand; Acquire the movement distance and movement direction of the hand according to the coordinate points; the movement distance of the hand includes a first movement distance and a second movement distance, and the movement direction of the hand includes a first movement direction and a second movement direction; Acquire a first movement distance of the hand in a first movement direction according to the depth information of the hand in each depth image; Acquire a second movement distance of the hand in a second movement direction according to the position information of the hand in each depth image; generating a first hand motion trajectory according to the first motion distance and the first motion direction, and generating a second hand motion trajectory according to the second motion distance and the second motion direction; the first hand motion trajectory and the second hand motion trajectory both belong to hand motion trajectories; generating a first operation instruction for a virtual icon in the display screen according to the first hand motion trajectory, and controlling the virtual icon to operate a component in the display screen according to the first operation instruction; generating a second operation instruction of a virtual icon on the display screen according to the second hand motion trajectory, wherein the second operation instruction includes the virtual icon motion trajectory; The virtual icon is controlled to move according to the second operation instruction along the motion trajectory of the virtual icon.

11. The computer-readable storage medium according to claim 10, wherein: When the computer program is executed by the processor, the processor further performs the following operations when acquiring at least two depth images: Acquire the historical movement speed of the hand, and acquire the acquisition frame rate according to the historical movement speed; At least two depth images are acquired according to the acquisition frame rate.

12. The computer-readable storage medium according to any one of claims 10 to 11, characterized in that: When the computer program is executed by the processor, the processor performs the following operations: Obtaining the interval between the moment when the hand motion trajectory is generated and the current moment; When the interval duration exceeds a duration threshold and no operation instruction is generated, a prompt message is issued, where the prompt message is used to prompt that the hand motion trajectory is invalid.