Frame rate boost based on static gesture recognition

Abstract

A method (1100) includes capturing, by a sensor (1002) in a computing device (1000), a first set of images (110A) at a first frame rate (1102); identifying a static gesture based on at least one image included in the first set of images (110A) (1104); in response to identifying the static gesture, capturing, by the sensor, a second set of images (110B) at a second frame rate, the second frame rate being faster than the first frame rate (1106); and identifying a dynamic gesture associated with the static gesture based on a plurality of images included in the second set of images (110B), the second set of images being captured after the first set of images (1108).

Description

Technical Field

[0001] This manual relates to gesture recognition. Background Technology

[0002] Computing devices can receive input by recognizing gestures based on images captured by sensors. Image processing can consume computational resources. Summary of the Invention

[0003] A computing device, such as a head-mounted device, can capture images at a first frame rate until a static gesture is recognized. A static gesture may include a hand in a predetermined shape or posture. A static gesture can be recognized based on one image and / or a single image. In response to the recognition of a static gesture, the computing device can trigger an increase in the frame rate to capture images at a second frame rate, faster than the first frame rate. While capturing images at the second frame rate, the computing device can recognize dynamic gestures. Dynamic gestures may include hand movements and can be recognized based on multiple images. Capturing images at a first (lower) frame rate before recognizing a static gesture reduces the power consumed by the computing device.

[0004] A method includes: capturing a first set of images at a first frame rate by a sensor in a computing device; identifying a static gesture based on at least one image included in the first set of images; in response to the identification of the static gesture, capturing a second set of images by the sensor at a second frame rate faster than the first frame rate; and identifying a dynamic gesture associated with the static gesture based on a plurality of images included in the second set of images, the second set of images being captured after the first set of images.

[0005] A non-transitory computer-readable storage medium includes instructions stored thereon. When executed by at least one processor, these instructions are configured to cause a computing system to: capture a first set of images at a first frame rate by a sensor in the computing device; identify a static gesture based on at least one image included in the first set of images; in response to the identification of the static gesture, capture a second set of images by the sensor at a second frame rate faster than the first frame rate; and identify a dynamic gesture associated with the static gesture based on a plurality of images included in the second set of images, the second set of images being captured after the first set of images.

[0006] A computing system includes at least one processor and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions stored thereon. When executed by the at least one processor, the instructions are configured to cause the computing system to: capture a first set of images at a first frame rate by a sensor in the computing device; recognize a static gesture based on at least one image included in the first set of images; in response to recognizing the static gesture, capture a second set of images by the sensor at a second frame rate faster than the first frame rate; and recognize a dynamic gesture associated with the static gesture based on a plurality of images included in the second set of images, the second set of images being captured after the first set of images.

[0007] Details of one or more implementations are set forth in the accompanying drawings and the following description. Other features will be apparent from the description, the drawings, and the claims. Attached Figure Description

[0008] Figure 1A A head-mounted device is shown that recognizes static gestures formed by a user's hands.

[0009] Figure 1B A head-mounted device is shown that recognizes dynamic gestures performed by a user's hands.

[0010] Figure 2A The head-mounted device and the hand forming the first static gesture are shown.

[0011] Figure 2B The field of vision includes the hand forming the first static gesture.

[0012] Figure 2C A head-mounted device is shown that presents a menu in response to the recognition of a first static gesture.

[0013] Figure 3A The head-mounted device and the hand performing dynamic gestures are shown.

[0014] Figure 3B The image shows the field of view including the hand forming a dynamic gesture and the menu presented by the head-mounted device in response to the head-mounted device recognizing a static gesture.

[0015] Figure 3C A head-mounted device is shown that presents a cursor that moves along a menu in response to the recognition of a static gesture.

[0016] Figure 4A The head-mounted device and the hand forming the second static gesture are shown.

[0017] Figure 4B The view includes the hand forming the second static gesture, where the menu is removed in response to the head-mounted device recognizing the second static gesture.

[0018] Figure 4C A head-mounted device is shown that removes the menu in response to the recognition of a second static gesture.

[0019] Figure 5A A head-mounted device that displays directions and a map is shown.

[0020] Figure 5B A magnified version of the map is shown.

[0021] Figure 5C The image shows a user's hand performing dynamic gestures, including a pinching motion.

[0022] Figure 5D A map in reduced size is shown.

[0023] Figure 6A It shows the hand and key points superimposed on the hand.

[0024] Figure 6B A schematic diagram of key points of the hand is shown.

[0025] Figure 7 This is a flowchart illustrating a method for improving frame rate based on recognizing static gestures.

[0026] Figure 8A , Figure 8B and Figure 8C The implementation of the head-mounted device is shown.

[0027] Figure 8D This demonstrates another implementation of the head-mounted device.

[0028] Figure 9 It is a timing diagram showing the actions performed by sensors, a first processor, a second processor, and a display included in a computing device.

[0029] Figure 10 It is a block diagram of a computing device.

[0030] Figure 11 It is a flowchart of a method executed by a computing device.

[0031] The same reference numerals refer to the same elements. Detailed Implementation

[0032] Computing devices can process input based on gestures performed by a user, without physical contact with the computing device or electronically coupled to it, such as aerial hand gestures captured by one or more sensors, such as cameras and / or time-of-flight sensors. Technical problems with processing gesture-based input include the high frame rates required to capture and process gestures consuming significant computational resources, and the sensors maintaining high frame rates consuming substantial power, rapidly depleting the batteries of portable computing devices, such as head-mounted displays. At least one technical solution to the problem of high frame rates consuming significant computational resources is to capture images at a relatively low frame rate before recognizing a static gesture. In response to recognizing a static gesture during the low frame rate period, the computing device causes the sensors to capture images at a higher frame rate. Capturing frames at a low frame rate before recognizing a static gesture has at least the technical benefit of reducing the power consumed by the computing device.

[0033] Figure 1A A head-mounted device 104 is shown that recognizes static gestures formed by the hand 106 of user 102. The head-mounted device 104 is an example of a computing device. The head-mounted device 104 is worn on the head of user 102. The head-mounted device 104 may include one or more sensors, such as a camera and / or a time-of-flight sensor, which capture images of objects within a field of view 108 in front of the user 102's face and / or head, such as the user 102's hand 106, and images of other objects and / or the background in front of the user 102.

[0034] sensor( Figure 1A (Unlabeled) A first image set 110A is captured within the field of view 108. The sensor captures the first image set 110A at a first frame rate. The sensor may be an image sensor. The first frame rate is a relatively low frame rate compared to the second frame rate described below. The first image set 110A includes multiple images 112A, 114A, 116A, 118A, and 120A. While the sensor is capturing the first image set 110A, the hand 106 of the user 102 is forming a static gesture, such as an index finger pointing upwards. The static gesture may include the form of a hand 106 without hand movement and may be indicated by a single image. The index finger is included in the hand 106. The static gesture is shown in at least one of the multiple images 112A, 114A, 116A, 118A, and 120A included in the first image set 110A.

[0035] The head-mounted device 104 can recognize static gestures within and / or based on at least one of a plurality of images 112A, 114A, 116A, 118A, 120A included in a first image set 110A. The head-mounted device can also recognize static gestures within and / or based on a single image of the plurality of images 112A, 114A, 116A, 118A, 120A, and / or based on that single image, and / or based on a plurality of images showing the same static gesture and / or a stationary form of the hand 106. In response to recognizing a static gesture, the head-mounted device 104 can perform actions such as launching an application running on the head-mounted device 104. The application can be associated with the recognized static gesture. The head-mounted device 104 can be accessed via a display included in the head-mounted device 104. Figure 1A (Not marked in the text) to present graphical output associated with the application. In some implementations, the head-mounted device 104 stores a library of static gestures, each static gesture associated with a different action and / or application.

[0036] Based on and / or in response to the recognition of a static gesture, the head-mounted device 104 causes the sensor to capture an image at a second frame rate. The second frame rate is higher than and / or faster than the first frame rate. As described herein, frame rate can refer to the image sensor frame rate or the frame processing rate of the image sensor processor.

[0037] Figure 1B A head-mounted device 104 is shown that recognizes dynamic gestures performed by the hand 106 of user 102. Dynamic gestures can be associated with static gestures that cause the head-mounted device 104 to cause a sensor to capture an image at a second frame rate. The head-mounted device 104 may include and / or store a library having one or more dynamic gestures associated with each static gesture. Dynamic gestures may include hand movements. Hand movements may be indicated by multiple images. The multiple images indicating hand movements included in a dynamic gesture may be captured by multiple frames and / or images. Dynamic gestures associated with a static gesture pointing to the index finger may include, for example, swiping up or down, swiping left or right, or tapping. The head-mounted device 104 can recognize dynamic gestures by comparing hand movements with dynamic gestures associated with the recognized static gestures.

[0038] Hand 106 is performing a dynamic gesture within field of view 108. Sensors of head-mounted device 104 capture images at a second frame rate. The images captured by the sensors at the second frame rate include a second image set 110B. Head-mounted device 104 can compare the image sequence with dynamic gestures associated with the static gestures that caused head-mounted device 104 to capture images at the second frame rate. Comparing the image sequence with dynamic gestures associated with the static gestures that caused head-mounted device 104 to capture images at the second frame rate limits the number of dynamic gestures that head-mounted device 104 can compare the image sequence with, thereby reducing computational complexity.

[0039] exist Figure 1B In the example shown, the dynamic gesture includes a swipe made by the index finger. Multiple images 112B, 114B, 116B, 118B, and 120B, included in the second image set 110B, progressively illustrate the movement of the hand 106 corresponding to the dynamic gesture (such as a downward swipe of the index finger). Based on the multiple images 112B, 114B, 116B, 118B, and 120B progressively illustrating a downward swipe of the index finger included in the second image set 110B, the head-mounted device 104 can recognize the downward swipe dynamic gesture. The head-mounted device 104 can perform actions based on the recognized downward swipe dynamic gesture, such as moving down a menu or document presented by a display included in the head-mounted device 104.

[0040] In some implementations, a first processor included in the head-mounted device 104 recognizes static gestures. The first processor may recognize static gestures based on a single image included in a first image set. In some implementations, the first processor is included in an embedded system that includes sensors, the first processor, a memory device, and an interface with a second processor. The first processor may have relatively lower processing power than the second processor. In some implementations, recognizing dynamic gestures may exceed the processing power of the first processor.

[0041] In some implementations, the first processor recognizes static gestures. In some implementations, the first processor sends an interrupt signal to the second processor in response to recognizing a static gesture. The interrupt signal prompts the second processor to perform actions such as launching an application associated with the static gesture. In some implementations, after receiving a second image set from a sensor, the first processor sends the second image set 110B to the second processor, and the second processor recognizes dynamic gestures based on multiple images included in the second image set.

[0042] In some implementations, the first processor determines and / or generates key points based on images, using a first image set 110A and / or a second image set 110B. In some implementations, the first processor recognizes static gestures based on key points generated by the first processor from an image in the first image set 110A. In some implementations, the first processor sends key points generated from images included in the second image set 110B to a second processor. Sending key points instead of images reduces the amount of data transmitted from the first processor to the second processor, thereby reducing the latency in recognizing the second gesture in the second processor and reducing overall power consumption.

[0043] In some examples, the second processor is included in a computing device (an example of a computing device is a head-mounted device 104). The distance between the sensor and the first processor may be less than the distance between the sensor and the second processor. In some implementations, the second processor is included in a computing device different from the first processor, such that the first processor is included in a first computing device and the second processor is included in a second computing device. As a non-limiting example, the second computing device may include, for example, a smartphone, a tablet computer, a laptop computing device, or a desktop computing device. In some implementations, the functions described herein as being performed by the first or second processor are performed by a single processor.

[0044] Figure 2A A head-mounted device 104 and a hand 106 forming a first static gesture are shown. The head-mounted device 104 includes a sensor 202. The sensor 202 may include one or more cameras and / or time-of-flight sensors. The sensor 202 may capture images, such as images 112A, 114A, 116A, 118A, 120A included in a first image set 110A and / or images 112B, 114B, 116B, 118B, 120B included in a second image set 110B.

[0045] User 102 ( Figure 2A (Not shown) The user 102's hand 106 is held within the camera's field of view 204. The camera's field of view 204 can be a three-dimensional cone or a two-dimensional triangle, within which the sensor 202 can capture an image of an object. Figure 2A In the example shown, user 102 makes a first static gesture, that is, user 102's hand 106 faces the sensor 202 with its palm open.

[0046] Figure 2B The field of view 204 is shown, including the hand 106 forming the first static gesture. Figure 2B This shows the result from user 102 ( Figure 2B (not shown in the image) and / or sensor 202 ( Figure 2BThe camera's field of view is 204 degrees (not shown in the image). Figure 2B In the example shown, the field of view of camera 204 and / or sensor 202 is wider than the field of view of user 210 and / or user 102. The hand 106 forming the first static gesture is visible in both camera 204 and user 210.

[0047] A computing device (such as head-mounted device 104) can recognize static gestures, such as a first static gesture of opening one's palm towards sensor 202. The computing device (such as head-mounted device 104) can perform an action in response to recognizing a static gesture (such as the first static gesture). In some implementations, this action includes launching an application associated with the recognized static gesture and presenting graphical output based on the launched application. In some implementations, the action associated with the recognized static gesture includes presenting a menu of the application to be launched.

[0048] Figure 2C A head-mounted device 104 is shown that presents a menu 208 in response to the recognition of a first static gesture. The menu 208 is presented by a display 206 included in the head-mounted device 104. The display 206 may include lenses mounted on the head-mounted device 104. Figure 2C The light projected onto or near the lenses by a liquid crystal display (LCD) or light-emitting diode (LED) display (not otherwise specified), or by a projector included in the head-mounted device 104. Figure 2C In the example shown, menu 208 includes multiple applications for user 102 to select. Headset 104 can respond to the user's selection of an application included in the menu by activating the selected application.

[0049] In response to the recognition of a first static gesture, the head-mounted device 104 can increase the frame rate of the sensor 202, thereby causing the sensor 202 to capture more images per unit time. Capturing more images per unit time enables the head-mounted device 104 to recognize dynamic gestures based on multiple captured images. In response to the recognition of a static gesture, the head-mounted device 104 can generate key points identifying the positions of various parts of the user 102's hand 106. One of these key points can identify the position of the fingertip of the index finger included in the user 102's hand 106.

[0050] Figure 3A A head-mounted device 104 and a hand 106 performing a dynamic gesture are shown. In this example, the dynamic gesture is a slide of the index finger. This slide can be... Figure 2A and Figure 2B The open palm shown, or with Figure 3A The dynamic gesture associated with extending a finger shown in the image ( Figure 3AThe extended finger shown could be a static gesture recognized by head-mounted device 104 and associated with the swipe. The hand 106 with the extended finger is within the camera's field of view of sensor 202. User 102 may be forming Figure 2A and Figure 2B After opening your palm, bend and tuck in your thumb and the other 106 fingers of your hand, excluding the index finger.

[0051] Figure 3B The field of view includes a hand 106 forming a dynamic gesture and a menu 208 presented by a head-mounted device 104 in response to the head-mounted device recognizing a static gesture. Figure 3B This shows the result from user 102 ( Figure 2B (not shown in the image) and / or sensor 202 ( Figure 2B The camera field of view 204 (not shown in the image) is viewed from an angle. The hand 106 is simultaneously located within the camera field of view 204 and the user's field of view 210. A display included in the head-mounted device 104 presents a menu 208 to the user 102. Figure 3B The cursor moves along menu 208 in response to the recognition of a static gesture.

[0052] Figure 3C A head-mounted device 104 is shown that presents a cursor that moves along menu 208 in response to the recognition of a static gesture. Menu 208 and the cursor are shown as being presented on the lenses of the head-mounted device 104.

[0053] exist Figure 3B and Figure 3C In both cases, the cursor is shown moving from a first position 302 to a second position 304, and from a second position 304 to a third position 306. Directional arrow 308 indicates the cursor's movement from the first position 302 to the second position 304. Directional arrow 310 indicates the cursor's movement from the second position 304 to the third position 306. The cursor's position is associated with a key point at the tip of the index finger of the user 102's hand 106. The cursor's movement can be based on a dynamic gesture recognized by the head-mounted device 104. The cursor moving to the position associated with "App 3" and subsequently moving to the left causes the dynamic gesture to be a selection of the application indicated as "App 3". The head-mounted device 104 can respond to the dynamic gesture being recognized as a selection of the application indicated as "App 3" by activating the application indicated as "App 3".

[0054] Figure 4A A head-mounted device 104 and a hand 106 forming a second static gesture are shown. The hand 106 is in front of the sensor 202 and within the camera's field of view 204. In this example, the second static gesture is a clenched fist. In this example, the second static gesture is associated with closing the application and / or exiting the interaction mode.

[0055] Figure 4B The view shown includes the hand 106 forming the second static gesture, wherein the menu 208 is removed in response to the head-mounted device 104 recognizing the second static gesture. Figure 4B (Not shown in the image). The hand 106 is simultaneously located within the camera's field of view 204 and the user's field of view 210.

[0056] The head-mounted device 104 can recognize a second static gesture. The second static gesture can be associated with closing an application and / or exiting an interactive mode. In some examples, the head-mounted device 104 can respond to recognizing a second static gesture by closing an application launched in response to recognizing a dynamic gesture and / or an application selection. In some examples, the head-mounted device 104 can respond to recognizing a second static gesture by closing a menu 208 that opened and / or was displayed in response to recognizing a first static gesture.

[0057] Figure 4C A head-mounted device 104 is shown that removes menu 208 in response to recognizing a second static gesture. Lens 408A included in the head-mounted device 104 no longer displays menu 208 or responds to a... Figure 3B and Figure 3C Any application launched based on the selection shown and described.

[0058] Figure 5A A head-mounted device 104 is shown presenting orientation 502 and map 504A. The head-mounted device 104 may have initiated a map application. The head-mounted device 104 may have responded to a user's selection of a map application (such as based on...). Figure 3A , Figure 3B and Figure 3C (The map application is selected as shown and described) and then the map application is launched.

[0059] Figure 5B A magnified view of map 504B is shown. Map 504B can be displayed by a display included in the head-mounted device 104, such as... Figure 5A As shown in map 504A. User 102 ( Figure 5B (Not shown in the image) A wider terrain view may be desired.

[0060] Figure 5C The hand 106 of a user 102 performing a dynamic gesture including a pinch gesture 506 is shown. Headset 104 ( Figure 5C(Not shown in the image) A static gesture associated with pinching may have been recognized. A static gesture associated with pinching may include, for example, the thumb and forefinger extending while the remaining fingers are bent and folded towards the palm of hand 106. Headset 104 can respond to the recognition of a static gesture by increasing the frame rate and recognizing dynamic pinching gestures. Headset 104 can respond to dynamic pinching gestures by shrinking and / or zooming out the map 504B.

[0061] Figure 5D A reduced-size map 504D is shown. The head-mounted device 104 presents the reduced-size map 504D by shrinking and / or shrinking the map 504B in response to the recognition of a dynamic pinching gesture. If the head-mounted device 104 recognizes a dynamic gesture associated with expanding or enlarging that moves the thumb and forefinger away from each other (reverse pinching), the head-mounted device 104 can present the map in expanded or enlarged form.

[0062] Figure 6A The image shows a hand 106 and key points 600 to 620 superimposed on the hand 106. Key points 600 to 620 can be considered as hand key points. Figure 10 The keypoint generator 1004 shown (which may be part of a gesture recognizer implemented by a first processor included in a computing device (such as a head-mounted device 104) can generate keypoints 600 to 620. These twenty-one key points include the following locations: wrist 600, thumb carpal metacarpophalangeal (CMC) joint 601, thumb metacarpophalangeal (MCP) joint 602, thumb interphalangeal (IP) joint 603, thumb fingertip 604, index finger MCP 605, index finger proximal interphalangeal (PIP) joint 606, index finger distal interphalangeal (DIP) joint 607, index fingertip 608, middle finger MCP joint 609, middle finger PIP joint 610, middle finger DIP joint 611, middle fingertip 612, ring finger MCP joint 613, ring finger PIP joint 614, ring finger DIP joint 615, ring fingertip 616, little finger MCP joint 617, little finger PIP joint 618, little finger DIP joint 619, and little fingertip 620. Key points 600 to 620 can be considered as the key point set. The keypoint generator 1004 can generate a set of keypoints based on each image.

[0063] Figure 6B A schematic diagram 650 shows key points 0 to 21 of the hand. Key points 0 to 21 can be considered as key points of the hand. These twenty-one key points 0 to 21 of the hand correspond to... Figure 6AThe twenty-one keypoints 600 to 620 shown and described. These twenty-one keypoints 0 to 21 are keypoints that the keypoint generator 1004 will generate. These twenty-one keypoints 0 to 21 indicate the following locations: wrist 0, thumb carpal metacarpophalangeal (CMC) joint 1, thumb metacarpophalangeal (MCP) joint 2, thumb interphalangeal (IP) joint 3, thumb fingertip 4, index finger MCP joint 5, index finger proximal interphalangeal (PIP) joint 6, index finger distal interphalangeal (DIP) joint 7, index fingertip 8, middle finger MCP joint 9, middle finger PIP joint 10, middle finger DIP joint 11, middle fingertip 12, ring finger MCP joint 13, ring finger PIP joint 14, ring finger DIP joint 15, ring fingertip 16, little finger MCP joint 17, little finger PIP joint 18, little finger DIP joint 19, and little fingertip 20. This is just an example. The keypoint generator 1004 and / or gesture recognizer can generate additional numbers and / or locations of hand landmarks and / or keypoints.

[0064] Figure 7 This is a flowchart illustrating a method 700 for improving frame rate based on recognizing static gestures. Method 700 can be executed by a computing device (such as a head-mounted device 104).

[0065] Method 700 may include capturing images at a first frame rate (702). The first frame rate may be a relatively low and / or slow frame rate, such as one frame per second (1 fps). Capturing images at the first frame rate (702) may include capturing a first image set, such as a first image set 110A comprising multiple images 112A, 114A, 116A, 118A, 120A. The computing device may capture the images via a sensor (such as sensor 202). Sensor 202 may include one or more cameras and / or time-of-flight sensors.

[0066] Method 700 may include determining whether a static gesture has been recognized (704). Determining whether a static gesture has been recognized (704) may include comparing the captured image with static gestures stored in a gesture library and / or a library of static gestures. In some examples, determining whether a static gesture has been recognized (704) may include generating a set of hand keypoints based on the image captured at a first frame rate (702), such as those mentioned above. Figure 6A The hand keypoints 600 to 620 shown and described. Determining whether a static gesture has been recognized (704) may include comparing these hand keypoint sets with static gestures stored in a gesture library and / or a library of static gestures. The computing device may determine whether a static gesture has been recognized based on whether the comparison between the image and / or the keypoint set and the static gesture meets a similarity threshold (704).

[0067] If no static gesture is recognized, method 700 may include deleting the image (712) and continuing to capture images at the first frame rate (702). Deleting the image (712) after generating keypoints and / or determining whether a static gesture has been recognized has the technical benefit of saving memory in a computing device. Deleting the image (712) also has the benefit of protecting the privacy of people included in the captured image and / or people associated with objects included in the captured image.

[0068] If a static gesture is detected, method 700 may include capturing an image at a second frame rate (706). The second frame rate may be a relatively fast frame rate, such as ten frames per second (10 fps). The second frame rate may be faster than, greater than, and / or higher than the first frame rate. Capturing an image at the second frame rate (706) may include capturing a second image set, such as a second image set 110B comprising multiple images 112B, 114B, 116B, 118B, and 120B. The computing device may capture the image via a sensor, such as sensor 202. Sensor 202 may include one or more cameras and / or time-of-flight sensors.

[0069] After and / or during the capture of an image at a second frame rate (706), method 700 may include determining whether a dynamic gesture has been recognized (708). A dynamic gesture may be associated with a static gesture recognized at (704). In some implementations, each static gesture may be associated with a set of dynamic gestures. Determining whether a dynamic gesture has been recognized (708) may include comparing a plurality of images and / or one or more image sequences captured at (706) with dynamic gestures stored in a gesture library and / or a library of dynamic gestures. In some implementations, determining whether a dynamic gesture has been recognized (708) may include generating a set of keypoints based on the images captured at (706), and comparing a plurality of keypoint sets and / or consecutive keypoint sets with dynamic gestures stored in a gesture library and / or a library of dynamic gestures. In some implementations, a first processor may generate the keypoint sets and send the keypoint sets to a second processor. The second processor may compare a plurality of keypoint sets and / or consecutive keypoint sets with dynamic gestures stored in a gesture library and / or a library of dynamic gestures. The computing device can determine whether a dynamic gesture has been recognized based on whether a comparison of multiple images and / or multiple sets of key points with the dynamic gesture meets a similarity threshold (708).

[0070] If no dynamic gesture is recognized, method 700 may include deleting the image (712) and restoring the image captured at the first frame rate (702). In response to generating a keypoint set based on the image and / or in response to determining that no dynamic gesture was recognized, the image captured at the second frame rate may be deleted (712).

[0071] If a dynamic gesture is detected, the method may include performing an action based on the dynamic gesture (710). This action may be associated with the dynamic gesture. The action may include launching an application associated with the dynamic gesture, launching an application selected based on the dynamic gesture (such as...). Figure 3B and Figure 3C (as shown and described), or performing actions within the application (such as scrolling in a direction associated with a dynamic gesture or selecting an item based on a dynamic gesture), as non-limiting examples.

[0072] Figure 8A , Figure 8B and Figure 8C The implementation of the head-mounted device 104 is shown. For example... Figure 8A , Figure 8B and Figure 8C As shown, the head-mounted device 104 includes a frame 802. The frame 802 includes a front frame portion defined by rim portions 803A, 803B surrounding corresponding optical portions in the form of lenses 807A, 807B, wherein a nose bridge portion 809 connects to the rim portions 803A, 803B. Arm portions 805A, 805B are pivotally or rotatably coupled to the front frame via hinge portions 812A, 812B at the corresponding rim portions 803A, 803B. In some implementations, lenses 807A, 807B may be corrective / prescription lenses. In some implementations, lenses 807A, 807B may be optical materials comprising glass and / or plastic portions, and do not necessarily contain corrective / prescription parameters. Lenses 807A, 807B are Figure 4C An example of lens 408A is shown. Displays 810A and 810B can be coupled into a portion of frame 802. Figure 8BIn the illustrated implementation, displays 810A and 810B are coupled to arm portions 805A and 805B and / or frame portions 803A and 803B of frame 802. In some implementations, the head-mounted device 104 may further include an audio output device 816 (such as one or more speakers), an illumination device 818, a first processor 811A, a second processor 811B, an outward-facing image sensor 814 (or camera), and gaze-tracking cameras 819A and 819B capable of capturing images of the user 102's eyes to track the user 102's gaze. The first processor 811A may receive and / or process images captured by the outward-facing image sensor 814. The first processor 811A may, for example, generate keypoints based on the images. In some implementations, the first processor 811A may recognize static gestures. In some implementations, the first processor 811A may send interrupt signals and / or keypoints to the second processor 811B. In some implementations, the second processor 811B may recognize dynamic gestures. The outward-facing image sensor 814 can be an example of sensor 202. The distance between the first processor 811A and the outward-facing image sensor 814 can be less than the distance between the second processor 811B and the outward-facing image sensor 814.

[0073] In some implementations, the head-mounted device 104 may include a see-through near-eye display. Displays 810A and 810B may be configured to project light from a display source onto a portion of a teleprompter glass, which acts as a beamsplitter positioned at an angle (e.g., 30 to 45 degrees). The beamsplitter can achieve reflectance and transmittance values ​​that allow light from the display source to be partially reflected while the remaining light is transmitted. This optical design allows a user to see physical objects in the world through lenses 807A and 807B, while simultaneously seeing content generated by displays 810A and 810B (such as digital images, user interface elements, virtual content, etc.). In some implementations, waveguide optics may be used to depict content on displays 810A and 810B via coupled-out light 820A and 820B. The images projected from displays 810A and 810B onto lenses 807A and 807B can be semi-transparent, allowing user 102 to see the images projected from displays 810A and 810B as well as physical objects outside the head-mounted device 104.

[0074] Figure 8DAnother implementation of the head-mounted device 104 is shown. In this implementation, the head-mounted device 104 is in the form of goggles, wherein a display is included in the head-mounted device 104, and a housing supporting the display surrounds the face and / or eyes of the user 102. This implementation of the head-mounted device 104 can support virtual reality (VR) experiences, in which the user 102 sees only the content presented by the display included in the head-mounted device 104.

[0075] Figure 9 This is a timing diagram illustrating the actions performed by sensor 902, first processor 904, second processor 906, and display 908. Sensor 202 and outward-facing image sensor 814 are examples of sensor 902. First processor 811A is an example of first processor 904. Second processor 811B is an example of second processor 906. Another example of second processor 906 is a processor included in a computing device (such as a smartphone, tablet computing device, smartwatch, laptop or notebook computing device, or desktop computing device, as a non-limiting example) that is separate from and / or external to the computing device including sensor 902 and first processor 904. Second processor 906 may have more powerful computing capabilities than first processor 904, such as higher clock speeds, the ability to perform more operations per clock cycle, and / or faster memory access (such as larger and / or faster cache). The distance between first processor 904 and sensor 902 may be less than the distance between second processor 906 and sensor 902. Displays 810A and 810B are examples of second processor 906. Any combination of sensor 902, first processor 904, and / or second processor 906 can be included in the execution Figure 7 Method 700 in computing devices (such as head-mounted devices 104).

[0076] Sensor 902 can capture an image (910). Sensor 902 can capture an image at a first frame rate. Images 112A, 114A, 116A, 118A, and 120A, included in the first image set 110A, are examples of images that sensor 902 can capture at (910). Sensor 902 can send the captured image 912 to the first processor 904.

[0077] The first processor 904 can recognize static gestures (914). The first processor 904 can recognize static gestures (914) based on an image 912 captured by the sensor 902 and received by the first processor 904 from the sensor 902. In some implementations, recognizing static gestures (914) includes generating and / or recognizing a set of key points based on the captured image 912, and recognizing the static gesture based on a set of key points in the set of key points.

[0078] In response to the recognition of a static gesture (914), the first processor 904 may send an interrupt signal 916 to the second processor 906. The interrupt signal 916 may cause the second processor 906 to perform an action associated with the interrupt signal 916, such as launching an application or displaying a menu. In response to the recognition of a static gesture (914), the first processor 904 may send a frame rate command 918 to the sensor 902. The frame rate command 918 may cause the sensor 902 to change the rate at which it captures images, such as changing from a first (lower) frame rate to a second (higher) frame rate.

[0079] In response to receiving interrupt signal 916, processor 906 can perform the action associated with interrupt signal 916. An alternative to the first processor 904 sending interrupt signal 916 to the second processor 906 would be for the second processor 906 to send a polling request to the first processor 904. However, a polling request would be less efficient than the first processor 904 sending an interrupt signal to the second processor 906 because a polling request would involve communication between the second processor 906 and the first processor 904 when there is no need to exchange information (such as images and / or key points).

[0080] Actions associated with interrupt signal 916 may include, for example, launching an application. In an example where launching an application is an action associated with interrupt signal 916, second processor 906 may respond to interrupt signal 916 by launching the application (920). The launched application may be associated with and / or identified by interrupt signal 916. After launching the application (920), second processor 906 may send image command 922 to display 908. Image command 922 may have been generated by the launched application. Display 908 may present output (924) in response to image command 922. Output may include application-based output, such as a menu.

[0081] Sensor 902 can capture images at a second frame rate (925). Sensor 902 can capture images at a second frame rate (925) in response to frame rate command 918. The second frame rate is higher than the first frame rate. Sensor 902 can send the captured images to a first processor 904.

[0082] The first processor 904 can generate and / or identify key points (928) based on image 926. Key points can be the locations of various parts of user 102's hand 106. Examples of key points are shown in... Figure 6A and 6BAs shown in the diagram, the first processor 904 can generate a set of key points based on each image in image 926. The first processor 904 can send the set of key points 930 to the second processor 906. The set of key points 930 can be represented with less data than the captured image 926. Representing the set of key points 930 with less data than the captured image 926 makes it possible to send the set of key points 930 to the second processor 906 with a lower latency than that associated with sending image 926 to the second processor 906.

[0083] The second processor 906 can recognize dynamic gestures based on keypoints 930 (932). The second processor 906 can recognize dynamic gestures based on keypoints 930 by comparing a set of keypoints with dynamic gestures stored in a library (932). The dynamic gestures compared with the set of keypoints can be stored in the library in association with the static gestures recognized at (914). The second processor 906 can also recognize dynamic gestures based on the recognition of a dynamic gesture and two or more consecutive sets of keypoints in the set of keypoints 930 received from the first processor 904 satisfying a similarity threshold (932).

[0084] The recognized dynamic gesture can correspond to an input. The input may include, for example, a finger swipe, tap, or pinch. The second processor 906 can input the dynamic gesture and / or associated input to an application (934) launched at (920). The application can receive the input and manipulate it. Based on the input, the application can generate and / or modify the application's output, such as zooming in or out of a map. Based on the generated and / or modified output, the second processor 906 can generate an image instruction 936 and send it to a display 908. The display 908 can present the output (938) based on the image instruction 936.

[0085] The first processor 904 can delete an image (940). The deleted image may include the captured image 912 and / or image 926. The first processor 904 can delete the image (940) in response to generating key points based on the image, recognizing static and / or dynamic gestures based on the image, and / or performing actions based on the recognized static and / or dynamic gestures.

[0086] Figure 10 This is a block diagram of computing device 1000. Computing device 1000 may be an implementation of head-mounted device 104 or other computing devices for executing method 700.

[0087] Computing device 1000 may include sensor 1002. Sensor 1002 may capture images, such as images of hand 106. Sensor 1002 may capture images at different frame rates (such as a first (relatively low) frame rate and a second (relatively high) frame rate). In response to recognizing a static gesture, sensor 1002 may switch from capturing images at the first frame rate to capturing images at the second frame rate. In response to recognizing and / or completing a static gesture, sensor 1002 may switch from capturing images at the second frame rate to capturing images at the first frame rate. Sensor 202 and outward-facing image sensor 814 are examples of sensor 1002.

[0088] The computing device 1000 may include a keypoint generator 1004. The keypoint generator 1004 may include keypoints (which can be considered hand keypoints) based on images captured by the sensor 1002. The keypoint generator 1004 may generate keypoints 600 to 620 based on static data (such as a single image including any image included in the first image set 110A, images 112A, 114A, 116A, 118A, 120A) and / or based on a continuous stream (such as multiple consecutive images 112B, 114B, 116B, 118B, 120B included in the second image set 110B). The keypoint generator 1004 may generate a set of keypoints based on each image, wherein each keypoint in the set corresponds to a different position of the hand. The keypoint generator 1004 can generate keypoints 600 to 620 based on static data and / or a continuous stream by implementing a machine learning model (such as a neural network) trained to output hand landmarks in image coordinates based on images.

[0089] The computing device 1000 may include a gesture recognizer 1006. The gesture recognizer 1006 may recognize gestures based on captured images. In some implementations, the gesture recognizer 1006 recognizes gestures based on one or more sets of key points generated from the image by the key point generator 1004.

[0090] The gesture recognizer 1006 may include a static gesture recognizer 1008 and a dynamic gesture recognizer 1010. The static gesture recognizer 1008 can compare an image and / or a set of keypoints generated from the image with static gestures stored in a gesture library. The static gesture recognizer 1008 can recognize a static gesture based on a similarity threshold between a single image and / or a single set of keypoints and the static gestures stored in the gesture library. The dynamic gesture compared with the sequence of multiple images and / or the sequence of multiple image sets can be a dynamic gesture associated with the recognized static gesture. The dynamic gesture recognizer 1010 can recognize a dynamic gesture based on a similarity threshold between the sequence of multiple images and / or the sequence of multiple keypoints and the dynamic gestures stored in the gesture library.

[0091] The computing device 1000 may include a frame rate controller 1012. The frame rate controller 1012 controls the frame rate at which the sensor 1002 captures images. The frame rate controller 1012 may control the frame rate based on the recognition of gestures by the gesture recognizer 1006. The frame rate controller 1012 may increase the frame rate from a first (relatively low) frame rate to a second (relatively high) frame rate in response to the gesture recognizer 1006 recognizing a static gesture. The frame rate controller 1012 may return the frame rate from the second frame rate to the first frame rate based on the recognition of a dynamic gesture and / or based on the failure to recognize a dynamic gesture after a predetermined number of images and / or keypoint sets. In some examples, the frame rate controller 1012 returns the frame rate from the second frame rate to the first frame rate after a predetermined timeout period. The predetermined timeout period may begin after initiating the second frame rate or after recognizing a dynamic gesture.

[0092] The computing device 1000 may include a processor communication module 1014. The processor communication module 1014 can control communication between the first processor 904 and the second processor 906. The processor communication module 1014 can, for example, control the transmission of interrupt signal 916 and / or key point set 930 from the first processor 904 to the second processor 906.

[0093] The computing device 1000 may include an application module 1016. The application module 1016 can launch and / or execute applications on the computing device 1000. In some implementations, the application module 1016 can launch and / or execute an application in response to recognizing a static gesture associated with the application. In some implementations, the application module 1016 can launch and / or execute an application based on the selection of the application (such as...). Figure 2C and Figure 3CDynamic gestures (such as the selection of applications within menu 208 shown) are used to launch and / or execute applications. In some implementations, application module 1016 may provide input to applications running on computing device 1000 based on dynamic gestures recognized by gesture recognizer 1006. Application module 1016 may provide, for example, swipe gestures, tap gestures, or pinch gestures as input to applications running on computing device 1000.

[0094] The computing device 1000 may include a display controller 1018. The display controller 1018 can control the output of one or more displays included in the computing device 1000. The display controller 1018 can control the output of one or more displays based on an application running on the computing device 1000. The display controller 1018 can transmit the graphical output of an application running on the computing device 1000 to one or more displays included in the computing device 1000.

[0095] The computing device 1000 may include an image deleter 1020. The image deleter 1020 can delete images captured by the sensor 1002. In some implementations, the image deleter 1020 deletes the image after the gesture recognizer 1006 has performed gesture recognition based on the image and / or a comparison between the image and the gesture. In some implementations, the image deleter 1020 deletes the image after the keypoint generator 1004 generates a set of keypoints based on the image.

[0096] The computing device 1000 may include at least one processor 1022. The at least one processor 1022 may execute instructions, such as instructions stored in at least one memory device 1024, to cause the computing device 1000 to perform any combination of the methods, functions and / or techniques described herein.

[0097] Computing device 1000 may include at least one memory device 1024. At least one memory device 1024 may include a non-transitory computer-readable storage medium. At least one memory device 1024 may store data and instructions thereon that, when executed by at least one processor (such as processor 1022), are configured to cause computing device 1000 to perform any combination of the methods, functions, and / or techniques described herein. Therefore, in any of the implementations described herein (even if not explicitly indicated in conjunction with a particular implementation), software (e.g., processing modules, stored instructions) and / or hardware (e.g., processors, memory devices, etc.) associated with or included in computing device 1000 may be configured to perform any combination of the methods, functions, and / or techniques described herein, either alone or in combination with computing device 1000.

[0098] The computing device 1000 may include at least one input / output node 1026. At least one input / output node 1026 may receive and / or transmit data, such as receiving data from and / or sending data to another computing device, and / or receiving input from a user and providing output to that user. Input and output functions may be combined into a single node or may be divided into separate input and output nodes. The input / output node 1026 may include a microphone, camera, IMU, display, speaker, one or more buttons, and / or one or more wired or wireless interfaces for communicating with other computing devices, such as another computing device and / or a server.

[0099] Figure 11 This is a flowchart of method 1100 performed by computing device 1000. Method 1100 includes capturing images at a first frame rate (1102). Capturing images at the first frame rate (1102) may include capturing a first set of images at the first frame rate by a sensor in the computing device. Method 1100 includes recognizing a static gesture (1104). Recognizing a static gesture (1104) may include recognizing a static gesture based on at least one image included in the first set of images. Method 1100 includes capturing images at a second frame rate (1106). Capturing images at the second frame rate (1106) may include: in response to recognizing a static gesture, capturing a second set of images by a sensor at the second frame rate, the second frame rate being faster than the first frame rate. Method 1100 includes recognizing a dynamic gesture (1108). Recognizing a dynamic gesture (1108) may include recognizing a dynamic gesture associated with the static gesture based on a plurality of images included in the second set of images, the second set of images being captured after the first set of images.

[0100] In some examples, method 1100 further includes generating hand keypoints from multiple images included in a second image set. Dynamic gestures are recognized based on hand keypoints.

[0101] In some examples, method 1100 further includes deleting multiple images included in a second image set after generating hand keypoints.

[0102] In some examples, method 1100 further includes sending hand key points to a gesture recognizer.

[0103] In some examples, method 1100 further includes performing an action in response to recognizing a dynamic gesture.

[0104] In some examples, recognizing static gestures is performed by a first processor included in the computing device, and recognizing dynamic gestures is performed by a second processor included in the computing device.

[0105] In some examples, the recognition of static gestures is performed by a first processor included in the computing device, while the recognition of dynamic gestures is performed by a second processor located outside the computing device.

[0106] In some examples, method 1100 further includes a first processor generating hand key points based on a second image set, and the first processor sending the hand key points to a second processor.

[0107] In some examples, the distance between the sensor and the first processor is smaller than the distance between the sensor and the second processor.

[0108] In some examples, method 1100 further includes a first processor sending an interrupt signal to a second processor in response to the first processor recognizing a static gesture.

[0109] In some examples, method 1100 further includes: deleting a first image set in response to recognizing a static gesture; and deleting a second image set in response to recognizing a dynamic gesture.

[0110] In some examples, method 1100 further includes capturing a third set of images at a first frame rate after a dynamic gesture is detected.

[0111] In some examples, method 1100 further includes capturing a third set of images at a first frame rate after a predetermined timeout period.

[0112] In some examples, method 1100 further includes: recognizing a static gesture based on at least one image in a third image set; and in response to recognizing a static gesture based on at least one image in the third image set, capturing a fourth image set by a sensor at a second frame rate.

[0113] In some examples, method 1100 further includes: determining that no dynamic gesture exists in the fourth image set; and in response to determining that no dynamic gesture exists in the fourth image set, capturing a fifth image set at a first frame rate.

[0114] In some examples, the sensor includes a camera.

[0115] In some examples, the sensors include time-of-flight sensors.

[0116] In some examples, the computing device includes a head-mounted device.

[0117] The various technologies described herein can be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or a combination thereof. Implementations can be implemented as computer program products, i.e., computer programs tangibly embodied in an information carrier (e.g., in a machine-readable storage device) for execution by or control of a data processing device (e.g., a programmable processor, a computer, or multiple computers). Computer programs such as those described above can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs can be deployed to execute on a single computer, at a single site, or distributed across multiple sites and interconnected via a communication network.

[0118] The method steps can be executed by one or more programmable processors that execute a computer program to perform a function by manipulating input data and generating output. The method steps can also be executed by a dedicated logic circuit system such as a FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), and the device can be implemented as said dedicated logic circuit system.

[0119] For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. The components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer may also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0120] To provide user interaction, this can be implemented on a computer having a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user, and a keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input.

[0121] The implementation can be implemented in a computing system that includes backend components such as a data server, or middleware components such as an application server, or frontend components such as a client computer having a graphical user interface or web browser through which a user can interact with the implementation, or any combination of such backend, middleware, or frontend components. The components can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0122] While certain features of the described implementations have been shown as illustrated herein, many modifications, substitutions, alterations, and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and alterations falling within the true spirit of the embodiments of the invention.

Claims

1. A method comprising: A first set of images is captured by a sensor in a computing device at a first frame rate; Static gestures are identified based on at least one image included in the first image set; In response to the recognition of the static gesture, the sensor captures a second set of images at a second frame rate, which is faster than the first frame rate. as well as Dynamic gestures associated with the static gestures are identified based on multiple images included in the second image set, which is captured after the first image set.

2. The method of claim 1, further comprising: Hand key points are generated from the plurality of images included in the second image set. The dynamic gestures are identified based on key points on the hand.

3. The method of claim 2, further comprising deleting the plurality of images included in the second image set after generating the hand key points.

4. The method of claim 2 or claim 3, further comprising sending the hand key points to a gesture recognizer.

5. The method as described in any of the preceding claims, further comprising performing an action in response to recognizing the dynamic gesture.

6. The method as described in any of the preceding claims, wherein: The recognition of the static gesture is performed by a first processor included in the computing device; and The recognition of the dynamic gesture is performed by a second processor included in the computing device.

7. The method as described in any one of claims 1 to 5, wherein: The recognition of the static gesture is performed by a first processor included in the computing device; and The recognition of the dynamic gesture is performed by a second processor, which is external to the computing device.

8. The method of claim 2 and any one of claim 6 or 7, further comprising: The first processor generates the hand key points based on the second image set; as well as The first processor sends the key points of the hand to the second processor.

9. The method according to any one of claims 6 to 8, wherein, The distance between the sensor and the first processor is less than the distance between the sensor and the second processor.

10. The method of any one of claims 6 to 9, further comprising: The first processor sends an interrupt signal to the second processor in response to the first processor recognizing the static gesture.

11. The method as described in any of the preceding claims, further comprising: The first image set is deleted in response to the recognition of the static gesture; as well as The second image set is deleted in response to the recognition of the dynamic gesture.

12. The method as described in any of the preceding claims, further comprising capturing a third set of images at the first frame rate after the dynamic gesture is detected.

13. The method of any one of claims 1 to 11, further comprising capturing a third set of images at the first frame rate after a predetermined timeout period.

14. The method of claim 12 or 13, further comprising: The static gesture is identified based on at least one image within the third image set; as well as In response to the recognition of the static gesture based on at least one image in the third image set, the sensor captures a fourth image set at the second frame rate.

15. The method of claim 14, further comprising: It is determined that the dynamic gesture does not exist in the fourth image set; as well as In response to determining that the dynamic gesture does not exist in the fourth image set, a fifth image set is captured at the first frame rate.

16. The method as described in any of the preceding claims, wherein, The sensor includes a camera.

17. The method as described in any of the preceding claims, wherein, The sensor includes a time-of-flight sensor.

18. The method as described in any of the preceding claims, wherein, The computing device includes a head-mounted device.

19. A non-transitory computer-readable storage medium comprising instructions stored thereon, the instructions being configured, when executed by at least one processor, to cause a computing system to perform the method as claimed in any one of claims 1 to 18.

20. A computing system, comprising: At least one processor; as well as A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions stored thereon, the instructions being configured, when executed by the at least one processor, to cause the computing system to perform the method as described in any one of claims 1 to 18.

Images