Leveraging intent detection on computing devices

Abstract

The present disclosure relates to intent detection using a computing device. A method may utilize a method including: capturing an image; determining an environment in which a user is operating a computing device; detecting a gesture based on an object in the image; using a machine learning model, determining the user's intent based on the gesture and the environment; and performing a task based at least on the determined intent.

Description

Technical Field

[0001] Embodiments are directed to detecting intent of a user of a computing device based on the presence of an object (eg, a hand, a book, an item for sale, etc.) captured by a camera of the computing device. Background Art

[0002] Pointing devices in computing are used to control or activate certain elements in a user interface. On a computer, this can be achieved by using a separate controller, such as a mouse that can be moved on a flat surface, and the movement of the mouse is converted into a pointer / cursor on a computer screen. In addition, a mouse may have buttons for clicking and scrolling, which can enable various types of tasks, such as opening an application, selecting an application, scrolling down, etc. However, with the development of smartphones, tablet computers, etc., touch screens are widely used and fingers, for example, can replace physical controllers. User actions such as tapping, scrolling, swiping, pinching, and long pressing have become common modes of interaction with smartphones, tablet computers, etc. Summary of the invention

[0003] In a general aspect, a device, a system, a non-transitory computer-readable medium (having a computer-executable program code stored thereon that can be executed on a computer system), and / or a method can perform processing by a method including the following: capturing an image; determining an environment in which a user is operating a computing device; detecting a gesture based on an object in the image; using a machine learning model, determining the user's intent based on the gesture and the environment; and performing a task based at least on the determined intent.

[0004] The system may include a memory storing a set of instructions; and a processor configured to execute the set of instructions to cause the system to capture an image; determine an environment in which a user is operating a computing device; detect a gesture based on an object in the image; use a machine learning model to determine the user's intent based on the gesture and the environment; and perform a task based at least on the determined intent.

[0005] Implementations may include one or more of the following features. For example, determining the user's intent may further include translating the user's interaction with the real world, and using the interaction and the gesture to determine the user's intent. The machine learning model may be based on a computer vision model. The user's intent may be determined using a first machine learning model and a second machine learning model. The method may further include continuous tracking of a hand associated with a gesture using a second machine learning model. Images may be captured using a single non-depth sensing camera of a computing device. The task may be based on the use of a computer assistant. The task may include at least one of a visual and an audible output. The machine learning model may be trained using multiple images including at least one gesture, the machine learning model may be trained using multiple ground-truth images of the gesture, a loss function may be used to confirm a match between the gesture and the ground-truth image of the gesture, and detection of the gesture based on an object in the image includes matching the object with the gesture that matches the ground-truth image of the gesture. The machine learning model may be trained using multiple images, each of the multiple images including at least one object, and at least one object may have an associated ground-truth box. The machine learning model may generate multiple bounding boxes, the machine learning model may determine multiple features based on at least a portion of the object within the bounding box, the machine learning model may identify the object based on the multiple features, and may determine the user's intent based on the identified object. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The exemplary embodiments will be more fully understood from the detailed description given below and the accompanying drawings, wherein like elements are denoted by like reference numerals, which are given by way of example only and thus do not limit the exemplary embodiments, and wherein:

[0007] Figure 1 A flow chart of a method for detecting a user's intent and performing a task based on the intent according to at least one example embodiment is shown.

[0008] Figure 2 A trigger for detecting a gesture according to at least one example embodiment is shown.

[0009] Figure 3 Determining intent based on disambiguation according to at least one example embodiment is shown.

[0010] Figure 4 Pointing gestures are shown according to at least one example embodiment.

[0011] Figure 5 A block diagram illustrating signal flow according to at least one example embodiment is shown.

[0012] Figure 6 A flow chart of the method is shown.

[0013] Figure 7 A block diagram of a gesture processing system is shown.

[0014] Fig. 8A Shown are the layers in a convolutional neural network without sparsity constraints.

[0015] Figure 8B Shown are the layers in a convolutional neural network with a sparsity constraint.

[0016] Fig. 9 A block diagram of a model according to an exemplary embodiment is shown.

[0017] Fig.10 A block diagram illustrating signal flow for a machine learning process according to an exemplary embodiment is shown.

[0018] Fig.11A and Fig. 11B A head mounted display device according to at least one example embodiment is shown.

[0019] Fig.12 A wearable computing device is shown according to at least one example embodiment.

[0020] Fig.13A , 13B , 13C, 14A and 14B illustrate reading assistant tasks according to exemplary embodiments.

[0021] Fig.15 Examples of a computer device and a mobile computer device according to at least one example embodiment are shown.

[0022] It should be noted that these drawings are intended to illustrate the general characteristics of the methods, structures and / or materials used in certain exemplary embodiments, and to supplement the written description provided below. However, these drawings are not scaled to scale, and may not accurately reflect the precise structure or performance characteristics of any given embodiment, and should not be interpreted as limiting or restricting the range of values ​​or properties covered by the exemplary embodiments. For example, for clarity, the relative thickness and positioning of molecules, layers, regions and / or structural elements may be reduced or enlarged. The use of similar or identical reference numerals in the various drawings is intended to indicate the presence of similar or identical elements or features. DETAILED DESCRIPTION

[0023] Some computing devices lack screens (e.g., display screens) and may rely on real-world interactions using natural gestures (or gesture patterns) using fingers. Computing devices that do not include a controller that interfaces with typical input devices (e.g., a mouse, keyboard, etc.) may include a user interface configured to detect user intent via a non-typical (e.g., as a user intent input device) computing device component (e.g., a camera). In some embodiments, the user interface may detect the user's intent based on natural gestures used to perceive the computing device, and trigger a task (via the computing device) based on the detected user intent.

[0024] In augmented reality (AR) applications, the objects to be recognized can be limited to objects generated by the AR application. For example, if the AR application uses 200 objects, only 200 detectors are needed to recognize the objects. In contrast, the exemplary embodiment uses a trained ML model to recognize any possible real-world object (e.g., hand gestures, products, business cards, etc.) to determine the user's intent. Therefore, the exemplary detector can be configured to detect and / or recognize any real-world object and / or changes (e.g., gestures) of a real-world object.

[0025] In some embodiments, computer vision (e.g., a computer vision model) and / or a machine learning (ML) model can be used to determine the user's intent (e.g., user intent) from the user's gestures (e.g., as captured by a camera of the device). Examples of such gestures may include pointing, clicking, scrolling, circling, pinching to zoom, tapping, swiping, etc. In some embodiments, when used, for example, on a single-person viewing device (e.g., a wearable smart device, a head-mounted display, etc.), natural user intent through pointing gestures can be supported, for example, capturing an entire paragraph in a text document by circling the entire paragraph.

[0026] The user interface can transform (e.g., transform, convert, etc.) the user's interaction with the physical or digital world into a digital semantic understanding of the user's intent. The user's intent can be used to trigger tasks applicable to the physical or digital world. In some embodiments, the user interface can support a process or mechanism for detecting gestures (e.g., a user holding a business card), determining the user's intent (e.g., the intent to save a business card) based on the user's intent and / or verbal commands (e.g., holding the business card in hand (along with a verbal command of "save this business card"), and triggering a task (e.g., to save a business card) based on the determined intent. In some exemplary embodiments, gestures can be used to query (e.g., instruct, ask, etc.) a digital assistant about the definition of a word, the ingredients of a product, or to purchase the item that the user is holding.

[0027] In some embodiments, the user interface and / or mechanism described above may be integrated into the operating system and / or system architecture of the computing device and may be used by other application (e.g., App) developers as a cursor or gesture input medium without any physical input hardware (e.g., mouse, keyboard, etc.). In addition, the user interface and / or mechanism may be extended to interact and manipulate the VR / AR world (e.g., using interactions that are not included as functionality in AR / VR applications) via semantic understanding of things that a user may achieve using pointing and user gestures, such as highlighting text. In some embodiments, the user interface may detect (or help detect) a hand in a pointing position and the position of a pointer (e.g., the tip of the visible portion of an index finger) from a first-person viewing perspective. In an exemplary embodiment, the user interface may be a deep neural network built on, for example, a convolutional neural network (CNN) architecture.

[0028] As a result of the execution of software code stored in a memory (e.g., a non-transitory computer-readable storage medium) associated with the device and executed by at least one processor associated with the device, the instructions regarding Figure 1 The method described. However, alternative embodiments can be envisioned, such as a system embodied as a dedicated processor. The dedicated processor can be a graphics processing unit (GPU). In other words, the user interface can be implemented in the GPU of a single-person viewing device (e.g., a wearable smart device, a head-mounted display, etc.).

[0029] The GPU may be a component of a graphics card. The graphics card may also include video memory, a random access memory digital-to-analog converter (RAMDAC), and driver software. The video memory may be a frame buffer that stores digital data representing an image, video frame, image object, or frame scene. The RAMDAC may be configured to read the contents of the video memory, convert the contents into an analog RGB signal, and send the analog signal to a display or monitor.

[0030] The driver software may be software code stored in the memory. The software code may be configured to implement the methods described herein. Although the methods described below are described as being performed by a processor and / or a dedicated processor, these methods do not necessarily have to be performed by the same processor. In other words, at least one processor and / or at least one dedicated processor may perform the following description of the method. Figure 1 Describe the method.

[0031] Figure 1 A flow chart of a method for detecting a user's intent and triggering the execution of a task based on the intent is shown according to at least one exemplary embodiment. Figure 1As shown, in step S110, a gesture is detected. For example, a computing device including a user interface may use a camera of the computing device to detect a gesture of the user. The camera may be a non-depth sensing camera (e.g., a two-dimensional (2D) camera), and (compared to other gesture detection technologies that may require multiple camera inputs) the user interface may use only one camera to detect gestures. In an exemplary embodiment, the user interface may be configured to detect the position of a user's hand and a pointer (e.g., the tip of the visible portion of the user's index finger) in a pointing position from a first-person perspective based on a machine learning (ML) model trained using different image sets (e.g., thousands of images).

[0032] In step S120, the user's intent is determined based at least on the detected gesture. For example, the gesture may be a user pointing at an object (e.g., using an index finger). In some embodiments, the user interface may be configured to determine the user's intent (e.g., using an ML model). In some embodiments, for example, an ML model (e.g., a computer vision model) may be developed using a camera input of a computing device. Although a computer vision model may require a depth-sending camera or multiple camera inputs, a computing device may use a single non-depth (e.g., 2D) sensing camera input to determine the user's intent. This may allow the ML model to be implemented on a computing device with a single camera or a single non-depth sensing camera.

[0033] In step S130, a task is triggered based at least on the determined intent. For example, a user interface may trigger a task based on the determined intent. The task may be a function of a computing device. Exemplary tasks may include taking a photo or video, increasing / decreasing the volume, skipping a song, etc. Although the present disclosure describes the use of an index finger as a trigger, other fingers may also be used as triggers. The use of an index finger as a trigger is for illustrative purposes. As described above, the ML model may be trained with different sets of images.

[0034] For example, if the gesture is a pointing finger, and the finger is pointing at an object. The user's intent can be determined to be to obtain some information about the object. The interface can trigger the computing device to identify the object and perform a search based on the identified object. For example, the computing device can search for the price of the object in one or more stores.

[0035] Figure 2 A trigger for detecting a gesture according to at least one exemplary embodiment is shown. Figure 2, a bounding box 205 of a user's hand with a finger 210 (e.g., an index finger) is shown. In an exemplary embodiment, the object being detected can cause the bounding box 205 to be generated. The generation of the bounding box 205 can trigger the identification of the object in the bounding box 205. In some embodiments, the bounding box 205 can be one of a plurality of bounding boxes generated in response to receiving an image (e.g., by an ML model). The user interface can cause the capture of an image, which can trigger the generation of the bounding box 205, and the captured image and the ML model are used to determine that the hand is within the bounding box 205.

[0036] After identifying the object as a hand, the user interface may cause the ML model (and / or trigger another ML model) to identify the hand's gesture and / or motion. For example, the ML model may be configured to search for fingertips. Determining that the hand includes a finger 210 in a pointing gesture may trigger another task of the user interface (e.g., as yet another ML model and / or computer code). The task may include determining what the finger 210 is pointing at.

[0037] Figure 3 Determining intent based on disambiguation according to at least one exemplary embodiment is shown. Figure 3 As shown in image 300 of FIG. 3 , finger 210 is pointing at giraffe 305. Finger 210 pointing at an identifiable object (e.g., giraffe 305) can be used to disambiguate (remove uncertainty) and determine the user's intent using the ML model. In an exemplary embodiment, the ML model can determine that the user is likely pointing at giraffe 305. Determining that the user is likely pointing at giraffe 305 can trigger a user interface (e.g., based on the ML model) to cause the computing device to perform a task (e.g., search for information about giraffes using a computer assistant).

[0038] Figure 4 Pointing gestures are shown according to at least one exemplary embodiment. For example, the ML model may determine that the hand includes a pointing finger, as described above. In this example, the ML model may determine that the user may be pointing to text (e.g., in a book) rather than to an object (e.g., giraffe 305). The user's intent may be determined based on the text being pointed at and the posture and / or movement of the hand. For example, the intent may be determined to be translation of text or a portion of text (e.g., a word, phrase, sentence, etc.), reading aloud, finding a definition, etc. Some examples of pointing gestures (e.g., Figure 4 ) can include: a) pointing to a word by pointing directly below the word without covering it (405), b) pointing to a phrase by sliding your finger from left to right (410), c) pointing to a sentence by sliding your finger from left to right and tapping twice to indicate the end of selection (415), d) pointing to a paragraph by drawing a circle around it (420), etc.

[0039] In some embodiments, gestures may be detected in multiple stages (e.g., two or more stages). In other words, a first ML model and a second ML model are used to determine the user's intent. Then, in later stages, the first ML model may not be used. For example, in a first stage, the user interface may be configured to generate a bounding box (e.g., bounding box 205) (e.g., using an ML model) to identify (or help identify) an object as a user's hand. In a second stage, the user interface may be configured to determine the posture of the hand (e.g., using an ML model). This multi-stage approach to gesture recognition can allow continuous tracking of the user's hand (e.g., posture and movement) without having to run at least one of the stages again (e.g., the first stage of identifying the hand), and can make detecting gestures and determining intent (and subsequent execution of tasks) more efficient (e.g., in terms of speed and resource utilization (e.g., processor, memory, etc.)).

[0040] Figure 5 A block diagram of a signal flow according to at least one exemplary embodiment is shown. Figure 5 As shown, signal flow 500 starts with a detection object 505 block. The detection object 505 block can be configured to monitor and detect objects. For example, after initiating a user interface, communication with a camera of a computing device can be established. When an image is captured and transmitted to the user interface, the user interface can determine that an object (or a new object) has entered the field of view of the camera based on the transmitted image. In the recognition object 510 block, the user interface can use a trained ML model to recognize the object. In an exemplary embodiment, the object can be a hand (e.g., indicating the user's intention to present a gesture). However, the object can be, for example, a product for sale, an item in the real world (e.g., a house, a tree, a street sign, furniture, a book, etc.).

[0041] In the identify environment 515 block, the user interface may use user indications, application indications, scans of the surrounding environment (using a camera), etc. to identify the environment in which the computing device is operating. For example, the user interface may be instantiated by a computer application. The application may be a shopping application, an educational application, a translation application, etc. Thus, the identify environment 515 block may identify the environment as a store (or other shopping location), a school (or classroom), a reading location, etc.

[0042] In addition, the recognition environment 515 block can use a trained ML model to recognize the environment. In order to recognize the environment, the computer vision model can be trained using images of objects that can exist in various environments. The image can include tables, chairs, blackboards, etc. for classroom environments. The image can include tables, chairs, bookshelves, cashier counters, etc. for library environments. The image can include trees, vegetation, grass, animals, etc. for outdoor environments. The image captured by the camera of the computing device can be input to the model. The results including the least number of objects can be classified as possible environments. For example, if the image includes several types of trees, grass, and animals, the environment can be classified as an outdoor environment. In addition, the ML model can use tools available to the computer device to identify a specific outdoor (or other classified environment). For example, the ML model can use location information (e.g., global positioning system) and classified environments to more accurately identify the environment (e.g., identified as a national park, state park, golf course, etc.).

[0043] In the Select ML Model Group 520 block, at least one ML model may be selected based on the object and the environment. In an exemplary embodiment, the user interface may be configured to detect and respond to gestures. Thus, at least one ML model may include an ML model trained using gestures. At least one ML model may include an ML model for recognizing gestures and a model and / or algorithm that is unique to the environment and can map gestures to user intent. At least one ML model may be configured to recognize gestures and map gestures to user intent for the environment in a single operation. For example, at least one ML model may include a detection layer or block configured to recognize gestures and map gestures to user intent.

[0044] In addition, there may be multiple gesture technologies. For example, the gesture may be single-handed, double-handed, hand and voice, etc. Thus, the signal flow 500 may include multiple gesture ML models shown as gesture ML model group 1 525, gesture ML model group 2 530, ... and gesture ML model group n 535. The dotted lines indicate that one gesture ML model may be selected at a time. However, other configurations are within the scope of the present disclosure. Other trained ML models may also be included in exemplary embodiments as shown by, but not limited to, object ML model group 540.

[0045] The user interface may also use a combination of trained ML models. For example, an application developer may develop an application for a grocery store chain. Thus, the application developer may rely on gesture ML models available to all application developers and custom ML models trained based on products available in the grocery store (e.g., as an object ML model group 540). A user may open a developed application that may instantiate a user interface. The user may reach for a product, causing the hand to be detected and recognized (detect objects 505 block and recognize objects 510 block). The developed application may identify the environment as a grocery store (recognize environment 515 block) and select a gesture ML model group and an object ML model group. For example, a two-hand ML model group and a custom ML model group may be selected.

[0046] Signal flow 500 may include Figure 5 At least one repeated flow operation shown as a flow 575 block and a flow 580 block shown with a dashed line to indicate that these blocks are not necessarily structurally together or in one location. The flow 575 block includes a recognition gesture 545 block, a trigger task 550 block, and a monitoring 555 block for gestures. The recognition gesture 545 block can be configured to receive an image from a camera of a computing device. The image can be used as an input to a learned ML model to recognize a gesture. Recognizing a gesture can include assigning a variable for identifying a unique training gesture.

[0047] The trigger task 550 block may include instructions (e.g., computer code) that may cause a computing device to perform a task based on the identified gesture. In an exemplary embodiment, each task may be identified by a unique variable. The unique variable may be the same as the variable that identifies the gesture. Alternatively or additionally, the unique variable may be mapped to the identified gesture, or the identified gesture may be mapped to the unique variable. The task may be any task that may be performed by a computing device. For example, a task may be a search, translation, reading (e.g., text to speech), a computer assistant task, storing data (e.g., images), mapping data (e.g., mapping business cards to contacts), etc.

[0048] Continuing with the grocery store application example above, a shopper's gestures may be recognized and tasks performed. For example, reaching out and grabbing an item may result in a display or audible indication of the price, nutrition, or other information about the product. Additionally, a swipe gesture with another hand may result in the item being placed in a shopping cart. These tasks are examples only, and other tasks are within the scope of the present disclosure.

[0049] The monitoring of gestures 555 block can monitor the images captured and transmitted by the camera. The monitoring of gestures 555 block can use a trained ML model that can test the image and determine whether the image is likely to be a gesture. If the image is likely to be a gesture, the image can be transmitted to the recognition gesture 545 block. If the recognition gesture 545 block recognizes the image as a gesture, processing continues to the trigger task 550 block. Otherwise, processing returns to the monitoring of gestures 555 block. In some embodiments, the signal flow can start from flow 575 and / or flow 580. That is, the gesture can be recognized first (flow 575) (for example, the hand and ML model group can be pre-configured via an application) and then the object (flow 560), or vice versa.

[0050] The flow 580 block includes an identification object 560 block, a trigger task 565 block, and a monitoring of the object 570 block. The identification object 560 block can use a trained ML model to identify the object. The trigger task 550 block can cause a task to be performed based on the identification of the object. Continuing with the grocery store application example above, the object can be identified as a product, and the task can be to find information about the product. In addition, two or more ML model groups can be configured to operate together. For example, the trigger task 550 block can trigger the start of the identification object 560 block.

[0051] The monitoring of objects 570 block can monitor images captured and transmitted by the camera. The monitoring of objects 570 block can use a trained ML model that can test an image and determine whether the image is likely an object (e.g., an object different from a previously identified object). If the image is likely an object, the image can be transmitted to the identifying objects 560 block. If the identifying objects 560 block identifies the image as an object, processing continues to the triggering tasks 565 block. Otherwise, processing returns to the monitoring of objects 570 block.

[0052] As a result of the execution of software code stored in a memory (e.g., a non-transitory computer-readable storage medium) associated with the device and executed by at least one processor associated with the device, the instructions regarding Figure 6 The method described herein. However, alternative embodiments are contemplated, such as a system embodied as a dedicated processor. The dedicated processor may be a graphics processing unit (GPU). In other words, the user interface may be implemented in the GPU of a single-person viewing device (e.g., a wearable smart device, a head-mounted display, etc.).

[0053] The GPU may be a component of a graphics card. The graphics card may also include video memory, a random access memory digital-to-analog converter (RAMDAC), and driver software. The video memory may be a frame buffer that stores digital data representing an image, video frame, image object, or frame scene. The RAMDAC may be configured to read the contents of the video memory, convert the contents into an analog RGB signal, and send the analog signal to a display or monitor.

[0054] The driver software may be software code stored in the memory. The software code may be configured to implement the methods described herein. Although the methods described below are described as being performed by a processor and / or a dedicated processor, these methods do not necessarily have to be performed by the same processor. In other words, at least one processor and / or at least one dedicated processor may perform the following description of the method. Figure 6 Describe the method.

[0055] Figure 6 A flow chart of a method according to at least one exemplary embodiment is shown. Figure 6 As shown, in step S605, the user's hand movement is detected. For example, the hand movement can be detected as the hand entering the field of view of the camera of the computing device. The hand movement can also be the hand within the field of view of the camera and changing position (e.g., changing posture, moving from one side to another side, etc.). This movement can indicate that the user is in the process of showing the intention.

[0056] In step S610, a hand gesture is detected. For example, the gesture may be detected as an image captured by a camera. The gesture may be finger pointing, hand grasping, pinching, finger circling, etc.

[0057] In step S615, the environment is identified. The hand gesture may be interpreted differently based on the environment (e.g., use case, scenario, tool, application, etc.). In order to determine the user's intention (e.g., based on the gesture). The environment in which the computing device is operating should be determined. For example, the environment may be based on the location of the computing device, the application interacting with the user interface, etc. The environment may be a store, a classroom, a reading place, a park, an outdoor space (e.g., a forest, a lake, etc.), etc. The environment may be identified based on user input (e.g., voice commands) or computer application settings. For example, a user may say out loud in class that he / she is reading a book, or open a shopping application. Alternatively or in addition, recognition may be performed using an ML model that uses an image of the real-world environment of the computing device.

[0058] For example, in order to recognize an environment, a computer vision model can be trained using images of objects that may exist in various environments. Images may include tables, chairs, blackboards, etc. for classroom environments. Images may include tables, chairs, bookshelves, cashier counters, etc. for library environments. Images may include trees, vegetation, grass, animals, etc. for outdoor environments. Images captured by a camera of a computing device may be input to the model. Results including the least number of objects may be classified as possible environments. For example, if an image includes several types of trees, grass, and animals, the environment may be classified as an outdoor environment. In addition, the ML model may use tools available to the computer device to identify a specific outdoor (or other classified environment). For example, the ML model may use location information (e.g., a global positioning system) and a classified environment to more accurately identify an environment (e.g., a national park, a state park, a golf course, etc.).

[0059] In step S620, a gesture is recognized based on the hand gesture using the trained ML model. The ML model may be trained using a plurality of hand gestures that may be performed by a user of a computer device. The ML model may be trained based on a plurality of images (e.g., images of hand gestures as gestures) and ground truth images. For example, a gesture may be captured as an image using a camera of a computing device. The image may be input to the trained ML model. The trained ML model may recognize a gesture based on the image. The trained ML model may output a gesture identification (e.g., as a unique ID number).

[0060] In step S625, the user's intent is identified based on the gesture and the environment. The ML model may include an ML model for identifying gestures (step S620) and an ML model and / or algorithm that is unique to the environment and can map the gesture to the user's intent. At least one ML model may be configured to recognize gestures and map the gestures to the user's intent for the environment in a single operation. For example, at least one ML model may include a detection layer or block configured to recognize gestures and map the gestures to the user's intent.

[0061] In an exemplary embodiment, the computer device may operate in a real world space. Unlike a computer device executing an AR application (e.g., that can recognize and respond to a limited number of gestures), the exemplary embodiment may be configured to determine the user's intent based on an unlimited number of gestures (e.g., constrained by trained gestures) and an unlimited number of environments (e.g., real world spaces).

[0062] For example, a gesture may indicate different user intents based on the environment. Accordingly, different environments may have different mappings, lookup tables, algorithms, and / or ML models configured to determine the user's intent. Therefore, a mapping, lookup table, algorithm, and / or ML model may be selected based on the environment. In an exemplary embodiment, determining or identifying the user's intent may include mapping the identified gesture to the user's intent. Determining or identifying the user's intent may include identifying the user's intent based on the identified gesture using a mapping, the mapping being based on the environment. Determining or identifying the user's intent may include looking up the user's intent in a lookup table based on the identified gesture (e.g., using the identified gesture as a key). Determining or identifying the user's intent may include using an ML model that includes a detection layer or block configured to recognize gestures and map gestures to user intent.

[0063] For example, a pointing gesture in a reading environment (e.g., of a book) may indicate a different intent than a pointing gesture in a shopping environment. Therefore, the ML model and / or mapping or lookup table configured to map gestures to user intent may be different for the reading environment and the shopping environment. In other words, each ML model may have a mapping (e.g., a lookup table) that is used to determine the user's intent by mapping gestures to possible intents. Alternatively, the application may be configured to use an ML model that is configured to recognize gestures available to application developers. The application may further include a mapping or lookup table configured to map gestures to user intent.

[0064] In step S630, the task is performed based on the user's intent. For example, the task can be a computer-implemented task. The user's intent can be mapped to a task that is performed in response to recognizing the user's intent. The task can be output (e.g., audibly output) a definition of a word, translate a word, store information (e.g., a business card), search for information (e.g., price, encyclopedia information, etc.), turn on / off an appliance, etc.

[0065] Figure 7 A block diagram of a gesture processing system according to at least one exemplary embodiment is shown. Figure 7 As shown, the gesture processing system 700 includes at least one processor 705, at least one memory 710, a controller 720, a user interface 725, an ML model module 730, and a task module 735. The at least one processor 705, at least one memory 710, the controller 720, the user interface 725, the ML model module 730, and the task module 735 are communicatively coupled via a bus 715.

[0066] At least one processor 705 can be used to execute instructions stored on at least one memory 710, thereby implementing various features and functions described herein, or additional or alternative features and functions. At least one processor 705 can be a general-purpose processor. At least one processor 705 can be a graphics processing unit (GPU). At least one processor 705 and at least one memory 710 can be used for various other purposes. In particular, at least one memory 710 can represent various types of memories and examples of related hardware and software that can be used to implement any module described herein.

[0067] At least one memory 710 may be configured to store data and / or information associated with the gesture processing system 700. For example, at least one memory 710 may be configured to store code associated with implementing a user interface for capturing and / or editing images. For example, at least one memory 710 may be configured to store code associated with recognizing gestures, recognizing and implementing ML modules, recognizing and implementing computing tasks, etc. At least one memory 710 may be a non-transitory computer-readable medium with code that, when executed by the processor 705, enables the processor 705 to implement one or more of the techniques described herein. At least one memory 710 may be a shared resource. For example, the gesture processing system 700 may be an element of a larger system (e.g., a server, a personal computer, a mobile device, a head-mounted display, smart glasses, a hands-free computer device, etc.). Therefore, at least one memory 710 may be configured to store data and / or information associated with other elements within the larger system (e.g., image / video rendering, web browsing, computer assistants, and / or wired / wireless communications).

[0068] The controller 720 can be configured to generate various control signals and transmit the control signals to various blocks in the gesture processing system 700. The controller 720 can be configured to generate control signals to implement the techniques described herein. The controller 720 can be configured to control the task module 735 to execute the software code of the computer-based process according to the exemplary embodiment. For example, the controller 720 can generate control signals corresponding to parameters to implement searching, controlling applications, storing data, executing ML models, training ML models, etc.

[0069] The user interface 725 may be configured to communicate with a camera of the computing device. An image and / or multiple images are received from the camera and the images are processed using a trained ML model. After processing the images, the user interface may be configured to recognize and trigger the execution of a computer-implemented task or process.

[0070] The ML model module 730 may be configured to store, train, and execute at least one ML model. The ML model may be based on a convolutional neural network. The ML model may be trained for multiple users and / or a single user. For example, the ML model may be trained and stored on a network device. During initialization, the ML model may be downloaded from the network device to the local device. The ML model may be further trained before use and / or when the local device uses the ML model.

[0071] The task module 735 may be configured to store and execute at least one computer program (e.g., computer code) configured to cause a computer device to perform a task. The task may cause the computer device to perform a search, control an application, control a computer assistant, interpret and store data, translate text, convert text to speech, etc.

[0072] Fig. 8A Shown are the layers in a convolutional neural network without sparsity constraints. Figure 8B Figure 2 shows the layers in a convolutional neural network with a sparsity constraint. Fig. 8A and 8B , various configurations of neural networks used in at least one exemplary embodiment will be described. Fig. 8A An exemplary layered neural network is shown in FIG. The layered neural network includes three layers 810, 820, 830. Each layer 810, 820, 830 may be formed by a plurality of neurons 805. In this embodiment, no sparse constraint has been applied. Therefore, all neurons 805 in each layer 810, 820, 830 are networked to all neurons 805 in any adjacent layer 810, 820, 830.

[0073] Fig. 8A The exemplary neural network shown in is computationally not complex due to the small number of neurons 805 and layers. However, Fig. 8A The arrangement of the neural network shown in may not be scalable to larger sized networks due to the density of connections (e.g., connections between neurons / layers). In other words, the computational complexity may be too great as the network size scales, and in a non-linear manner. Therefore, if the neural network needs to be scaled up to work on inputs having a large number of dimensions, the computations may be too complex for all neurons 805 in each layer 810, 820, 830 to be connected by the network to all neurons 805 in one or more adjacent layers 810, 820, 830.

[0074] Initial sparse conditions can be used to reduce the computational complexity of a neural network. For example, if the neural network acts as an optimization process, the neural network approach can handle high-dimensional data by limiting the number of connections between neurons and / or layers. Figure 8BAn example of a neural network with sparse constraints is shown in . Figure 8B The neural network shown in is arranged so that each neuron 805 is only connected to a small number of neurons 805 in adjacent layers 840, 850, 860. This can form a neural network that is not fully connected and can be scaled to work on higher dimensional data. For example, a neural network with sparse constraints can be used as an optimization process for a model and / or to generate a model for use in rating / downgrading replies based on users posting replies. The fewer number of connections allows the number of connections between neurons to be scaled in a substantially linear manner compared to a fully networked neural network.

[0075] In some embodiments, a fully connected or incompletely connected but with the same Figure 8B The described configurations may be different than those of the convolutional neural network described above. In addition, in some embodiments, a convolutional neural network that is not fully connected and has less complexity than a fully connected neural network may be used. The convolutional neural network may also utilize pooling or max pooling to reduce the dimensionality (and thus complexity) of the data flowing through the neural network. Other methods of reducing the computational complexity of the convolutional neural network may be used.

[0076] Fig. 9 900 is a block diagram of a model according to an exemplary embodiment. The model 900 may be a convolutional neural network (CNN) including a plurality of convolutional layers 915, 920, 925, 935, 940, 945, 950, 955, 960 and an addition layer 930. The plurality of convolutional layers 915, 920, 925, 935, 940, 945, 950, 955, 960 may be one of at least two types of convolutional layers, respectively. Fig. 9 As shown, convolutional layer 915 and convolutional layer 925 can be of the first convolutional type. Convolutional layers 920, 935, 940, 945, 950, 955 and 960 can be of the second convolutional type. An image (not shown) can be input to the CNN. Normalization layer 905 can convert the input image into an image 910 that can be used as an input to the CNN. Model 900 further includes a detection layer 975 and a suppression layer 980. Model 900 can be based on a computer vision model.

[0077] The normalization layer 905 may be configured to normalize the input image. Normalization may include converting the image to MxM pixels. In an exemplary embodiment, the normalization layer 905 may normalize the input image to 300x300 pixels. In addition, the normalization layer 905 may generate a depth associated with the image 910. In an exemplary embodiment, the image 910 may have multiple channels, depths, or feature maps. For example, an RGB image may have three channels: a red (R) channel, a green (G) channel, and a blue (B) channel. In other words, for each of the MxM (e.g., 300x300) pixels, there are three (3) channels. The feature map may have the same structure as the image. However, instead of pixels, the feature map has values ​​based on at least one feature (e.g., color, frequency domain, edge detector, etc.).

[0078] Convolutional layers or convolutions can be configured to extract features from images. Features can be based on color, frequency domain, edge detectors, etc. Convolution can have filters (sometimes called kernels) and strides. For example, the filter can be a 1x1 filter with a stride of 1 (or 1x1xn for transforming to n output channels, sometimes 1x1 filters are called point-by-point convolutions), and the result is the output of the cell generated based on the combination of features of the cells of each channel at the position of the MxM grid (e.g., addition, subtraction, multiplication, etc.). In other words, feature maps with more than one depth or channel are combined into feature maps with a single depth or channel. The filter can be a 3x3 filter with a stride of 1, and the result is an output of fewer cells for each channel of the MxM grid or feature map. The output can have the same depth or number of channels (e.g., 3x3 xn filters, where n=depth or number of channels, sometimes also referred to as depth-by-depth filters) or a reduced depth or number of channels (e.g., 3x3 xk filters, where k<depth or number of channels). Each channel, depth, or feature map can have an associated filter. Each associated filter can be configured to emphasize different aspects of the channel. In other words, different features can be extracted from each channel based on the filter (sometimes referred to as a depth-wise separable filter). Other filters are within the scope of the present disclosure.

[0079] Another type of convolution may be a combination of two or more convolutions. For example, the convolution may be a depth-wise and point-wise separable convolution. This may include, for example, a convolution of two steps. The first step may be a depth-wise convolution (e.g., a 3x3 convolution). The second step may be a point-wise convolution (e.g., a 1x1 convolution). Depth-wise and point-wise convolutions may be separable convolutions because different filters (e.g., filters to extract different features) may be used for each channel or each depth of a feature map. In an exemplary embodiment, a point-wise convolution may transform a feature map to include c channels based on a filter. For example, an 8x8x3 feature map (or image) may be transformed into an 8x8x256 feature map (or image) based on a filter. In some embodiments, more than one filter may be used to transform a feature map (or image) into an MxMxc feature map (or image).

[0080] Convolution can be linear. Linear convolution describes the output as linear time invariant (LTI) based on the input. Convolution can also include rectified linear units (ReLU). ReLU is an activation function that can correct the LTI output of the convolution and limit the corrected output to a maximum value. ReLU can be used to speed up convergence (e.g., more efficient calculation).

[0081] In an exemplary embodiment, the first type of convolution may be a 1x1 convolution, and the second type of convolution may be a depthwise and pointwise separable convolution. Each of the plurality of convolutional layers 920, 935, 940, 945, 950, 955, 960 may have a plurality of cells, and at least one bounding box per cell. The convolutional layers 915, 920, 925 and the addition layer 930 may be used to transform the image 910 into a feature map that is equivalent in size to the feature map of the Conv_3 layer of the VGG-16 standard. In other words, the convolutional layers 915, 920, 925 and the addition layer 930 may transform the image 910 into a 38x38x512 feature map. The convolutional layers 935, 940, 945, 950, 955, 960 may be configured to incrementally transform the feature map into a 1 x 1x256 feature map. This incremental transformation may result in bounding boxes (regions of a feature map or grid) of varying sizes, which enables detection of objects of many sizes. Each cell may have at least one bounding box associated with it. In an exemplary embodiment, the larger the grid (e.g., the number of cells), the fewer the number of bounding boxes per cell. For example, the largest grid may use three (3) bounding boxes per cell, while a smaller grid may use six (6) bounding boxes per cell.

[0082] The detection layer 975 receives data associated with each bounding box. The data may be associated with features in the bounding box. The data may indicate an object in the bounding box (which may not be an object or part of an object). An object may be identified by its features. The data is sometimes cumulatively referred to as a class or classifier. The class or classifier may be associated with an object. The data (e.g., a bounding box) may also include a confidence score (e.g., a number between zero (0) and one (1)).

[0083] After the CNN processes the image, the detection layer 975 can receive and include multiple classifiers indicating the same object. In other words, an object (or a portion of an object) can be within multiple overlapping bounding boxes. However, the confidence score for each classifier can be different. For example, a classifier that identifies a portion of an object can have a lower confidence score than a classifier that identifies a complete (or substantially complete) object. The detection layer 975 can be further configured to discard bounding boxes that have no associated classifiers. In other words, the detection layer 975 can discard bounding boxes that have no objects in them.

[0084] The suppression layer 980 can be configured to sort the bounding boxes based on the confidence score, and can select the bounding box with the highest score as the classifier that identifies the object. The suppression layer can repeat the sorting and selection process for each bounding box with the same or substantially similar classifier. As a result, the suppression layer can include data (e.g., a classifier) ​​that identifies each object in the input image.

[0085] In augmented reality (AR) applications, the objects to be recognized can be limited to objects generated by the AR application. For example, if the AR application uses 200 objects, the detection layer 975 may only include 200 detectors (e.g., similar to ground truth boxes) to recognize objects. Exemplary embodiments use a trained ML model to recognize any possible real-world object (e.g., hand gestures, products, business cards, etc.) to determine the user's intent. Therefore, the detection layer 975 can be configured to detect and / or recognize any real-world object and / or changes (e.g., gestures) of a real-world object.

[0086] As described above, the convolutional layers 915, 920, 925, and the addition layer 930 can generate 38x38x512 feature maps. Each cell (e.g., each of the 1444 cells) can have at least three (3) bounding boxes. Therefore, at least 4332 bounding boxes can be transmitted from the addition layer 930 to the detection layer 975. The convolutional layers 935 and 940 can be second type convolutions and can be configured to perform 3x3x1024 convolutions and 1x1x1024 convolutions. The result can be a 19x19x1024 feature map. Each cell (e.g., each of the 361 cells) can have at least six (6) bounding boxes. Therefore, at least 2166 bounding boxes can be transmitted from the convolutional layer 940 to the detection layer 975.

[0087] Convolution layer 945 can be a second type of convolution and configured to perform a 3x3x512 convolution. The result can be a feature map of 10x10x512. Each cell (e.g., each of the 100 cells) can have at least six (6) bounding boxes. Therefore, at least 600 bounding boxes can be transmitted from convolution layer 945 to detection layer 975. Convolution layer 950 can be a second type of convolution and can be configured to perform a 3x3x256 convolution. The result can be a feature map of 5x5x256. Each cell (e.g., each of the 25 cells) can have at least six (6) bounding boxes. Therefore, at least 150 bounding boxes can be transmitted from convolution layer 950 to detection layer 975.

[0088] Convolution layer 955 may be a second type of convolution and configured to perform a 3x3x256 convolution. The result may be a 3x3x256 feature map. Each cell (e.g., each of the 9 cells) may have at least six (6) bounding boxes. Thus, at least 54 bounding boxes may be transferred from convolution layer 955 to detection layer 975. Convolution layer 960 may be a second type of convolution and configured to perform a 3x3x128 convolution. The result may be a 1x1x128 feature map. The cell may have at least six (6) bounding boxes. Six (6) bounding boxes may be transferred from convolution layer 960 to detection layer 975. Thus, in an exemplary embodiment, detection layer 975 may process at least 7,298 bounding boxes.

[0089] However, additional bounding boxes can be added to the feature maps of each convolutional layer. For example, a fixed number of bounding boxes (sometimes called anchors) can be added to each feature map based on the number of cells (e.g., MxM). These bounding boxes can contain more than one cell. The larger the number of cells, the more bounding boxes are added. As the number of bounding boxes increases, the likelihood of capturing an object within the bounding box also increases. Therefore, by increasing the number of bounding boxes per cell and / or by increasing the number of fixed boxes per feature map, the likelihood of identifying an object in the image can be increased. In addition, the bounding box can have a location on the feature map. As a result, more than one of the same object (e.g., hand, can, dog, cat, etc.) can be identified as being in the image.

[0090] In an exemplary embodiment, model 900 can be used to detect and / or recognize gestures. Gestures can be detected and / or recognized based on objects in an image used as input to model 900. Model 900 can place multiple bounding boxes in the image, and at least one of the bounding boxes should include an object that is a hand configured in a posture that can be a gesture. Model 900 can be trained to detect and / or recognize multiple gestures. In other words, the model can be trained using multiple images and multiple ground truth images (e.g., as ground truth boxes) (described in detail below). An object can be within a bounding box that is a hand configured in a posture that can be a gesture, and it can be determined that the bounding box includes features that recognize a gesture.

[0091] Once the model (e.g., model 900) architecture has been designed (and / or is in operation), the model should be trained (sometimes referred to as developing the model). The model architecture can be used for several models. For example, model 900 can be used in many object recognition models based on the objects used to train the model. In other words, a reading model can be developed by training model 900 using images related to reading. A shopping model can be developed by training model 900 using images of objects (e.g., products) related to shopping. A gesture model can be developed by training model 900 using images related to gestures. Determining the user's intent can be based on using a single trained model (e.g., an ML model) and / or using multiple trained models. Performing tasks based on user intent can be based on using a single trained model (e.g., an ML model) and / or using multiple trained models.

[0092] Fig.10A block diagram of a signal flow for a machine learning process according to an exemplary embodiment is shown. Training the model includes comparing a prediction of an object in an image with data indicating the content in the image. This data is sometimes referred to as ground truth. In other words, the model is used to generate a bounding box that includes data associated with the object in the image. A data set including the ground truth box is input into a training routine along with the bounding box. The training routine includes comparing the generated bounding box with the ground truth box using a loss function. The training routine can be configured to modify the model used to generate the bounding box based on the result of the comparison (e.g., the output of the loss function).

[0093] User data is input as input 1005. User data may include images, gestures, objects, text, etc. For example, an image may include at least one object. Based on the objects in the image, the image may include metadata containing ground truth boxes, data identifying the objects, data identifying ground truth boxes, data identifying ML models, etc. The user data may include multiple ground truth boxes representing objects that can be used to train the model. Other data may also be used. The ground truth boxes may be extracted from or generated by the data in the extract ground truth 1010 block. The extract ground truth 1010 block may also be configured to identify a pre-trained model or a model to be trained. Alternatively, a pre-trained model or a model to be trained may be identified by user input. The identified model may be used to select an ML model from the library 1035 (block 1015).

[0094] The selected ML model is then developed (block 1020) to more accurately perform the desired output function (e.g., detecting or identifying an object in an image) on the input 1005. The output model 1025 block can then be used on the input 1005 immediately (e.g., to continue training or on real-time data (block 1025)) or in the future (e.g., in a user interface configured to determine user intent). The developed model is then saved (block 1030) in a library 1035 for immediate use when needed and / or in the future.

[0095] If similar data is entered in the future, the model does not have to be redeveloped. Library 1035 can grow as the system is used. For example, as the user enters additional gestures, different types of products, or different business cards, the model for the gesture, product, or business card can be updated, revised, developed, and / or added to library 1035.

[0096] The training and optimization process (e.g., as a function of developing a model in block 1020) can be configured to generate a bounding box (or boxes) including data associated with an object in an image. The bounding box (or boxes) can be generated using a selected ML model (e.g., model 900). The object in the bounding box can be identified (or detected) using the selected ML model. The object can be identified (or detected) based on the data associated with the bounding box. The data associated with the bounding box can be features extracted from a portion of the image using the ML model. The training and optimization process (e.g., as a function of developing a model in block 1020) can be configured to compare the generated bounding box with a ground truth box using a loss function. Training can include modifying the ML model used to generate the bounding box based on the result of the comparison (e.g., the output of the loss function).

[0097] Modifying the ML model can include changing features and / or characteristics of features (e.g., key features or importance of features), hyperparameters including: boxes proposal, aspect ratio, data augmentation options, loss function, depth multiplier, number of layers, image input size (e.g., normalization), anchor boxes, location of anchor boxes, number of boxes per cell, feature map size, convolution parameters, etc.

[0098] The training and optimization process can be configured based on the desired tradeoff between the computation time spent and the desired result quality (e.g., as a function of the model developed in block 1020). Typically, the number of iterations used during the training process will produce an approximate logarithmic gain in reconstruction accuracy, so it is preferred to use an automatic threshold to stop further optimization. When the result quality is preferred, the automatic threshold can be set to a predetermined value of the reconstruction error, such as by calculating the mean square error, but other methods can also be used. The automatic threshold can be set to limit the training and optimization process to a predetermined number of iterations. In another alternative, a combination of these two factors can be used.

[0099] Models can also be optimized over time. For example, a product identification goal can be to minimize misidentifications. Therefore, during training (before use and during use), minimizing misidentifications associated with each product can minimize misidentifications in the operating environment. In an exemplary embodiment, a threshold number of misidentifications can be set for each product. The threshold number can be associated with a time period (e.g., a week or a month). In response to determining that the number of misidentifications associated with a product exceeds the threshold number, additional training of the relevant model can be triggered. Although a threshold number is described, other criteria are also within the scope of the present disclosure. Models can be optimized in response to events. For example, when a new product is added to a product line, an optimization (or development) model can be triggered.

[0100] In augmented reality (AR) applications, the objects to be recognized may be limited to objects generated by the AR application. Therefore, only one model is needed, and training the model may not be necessary. For example, if the AR application uses 200 objects, the detection (e.g., detection layer 975) may only include 200 detectors (e.g., similar to ground truth boxes) to recognize objects. Exemplary embodiments use a trained ML model to recognize any possible real-world object (e.g., hand gestures, products, business cards, etc.) to determine the user's intent. Therefore, detection may include detecting and / or recognizing any real-world object and / or changes (e.g., gestures) of a real-world object.

[0101] Fig.11A and 11B 1 is a perspective view of an exemplary head mounted display (HMD) 1100 worn by a user according to at least one exemplary embodiment. The HMD 1100 may include a housing 1110 coupled to a frame 1120, and an audio output device 1130 also coupled to the frame 1120 including, for example, speakers mounted in headphones.

[0102] exist Fig. 11B 110, the front portion 1111A of the housing 1110 is rotated away from the base portion 1111B of the housing 1110 so that some of the components housed in the housing 1110 are visible. The display 1140 can be mounted on the inward-facing side of the front portion 1111A of the housing 1110. The lens 1150 can be mounted in the housing 1110 and located between the user's eyes and the display 1140 when the front portion 1111A is in a closed position relative to the base portion 1111B of the housing 1110. In some embodiments, the HMD 1100 can include a sensing system 1160 including various sensors and a control system 1170 including a processor 1190 and various control system devices that facilitate the operation of the HMD 1100.

[0103] In some embodiments, the HMD 1100 may include a camera 1180 to capture still and moving images. Images captured by the camera 1180 may be used to help track the physical position of the user, the HMD 1100, and / or a handheld electronic device (not shown) in the real world or in real-world space relative to the augmented environment, and / or may be displayed to the user on the display 1140 in a pass-through mode, thereby allowing the user to temporarily leave the augmented environment and return to the physical environment without having to remove the HMD 1100 or otherwise change the configuration of the HMD 1100 to move the housing 1110 out of the user's line of sight.

[0104] For example, in some embodiments, the sensing system 1160 may include an inertial measurement unit (IMU) 1162 that includes a variety of different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The position and orientation of the HMD 1100 may be detected and tracked based on data provided by the sensors included in the IMU 1162. The detected position and orientation of the HMD 1100 may allow the system to then detect and track the user's head gaze direction and movement.

[0105] In some embodiments, the HMD 1100 may include a gaze tracking device 1165 to detect and track the user's eye gaze. The gaze tracking device 1165 may include, for example, an image sensor 1165a or multiple image sensors 1165a to capture images of the user's eyes, such as a specific portion of the user's eyes such as the pupil, to detect and track the direction and movement of the user's gaze. In some embodiments, the HMD 1100 may be configured to process the detected gaze as user input to be translated into a corresponding interaction in an immersive virtual experience.

[0106] In some embodiments, HMD 1100 includes a portable electronic device, such as a smartphone, removably disposed within a chamber of housing 1110. For example, display 1140 and camera 1180 may be provided by the portable electronic device. Fig.11A 11), the display 1140 is aligned with the lens 1150 so that the user can view at least a portion of the display 1140 (provided by the portable electronic device) through each eye. The camera 1180 can be aligned with the aperture in the housing 1110 so that the portable electronic device of the HMD 1100 can capture images when it is set in the housing 1110.

[0107] Fig.12 A wearable computing device according to at least one exemplary embodiment is shown. Fig.12 As shown, wearable computing device 1200 includes lens frame 1205, lens frame 1210, center frame support 1215, lens element 1220, lens element 1225, extended side arm 1230, extended side arm 1235, image capture device 1240 (e.g., camera), on-board computing system 1245, speaker 1250 and microphone 1255.

[0108] Each of the frame elements 1205, 1210, and 1215 and the extending side arms 1230, 1235 can be formed of a solid structure of plastic and / or metal, or can be formed of a hollow structure of similar materials to allow wiring and component interconnections to pass through the wearable computing device 1200 internally. Other materials are also possible. At least one of the lens elements 1220, 1225 can be formed of any material capable of appropriately displaying a projected image or graphic. Each of the lens elements 1220, 1225 can also be sufficiently transparent to allow a user to see through the lens element. These two features of the combined lens element can facilitate augmented reality or a head-up display, in which a projected image or graphic is superimposed on a real-world view perceived by the user through the lens element.

[0109] The central frame support 1215 and the extended side arms 1230, 1235 are configured to secure the wearable computing device 1200 to the user's face via the user's nose and ears, respectively. The extended side arms 1230, 1235 can each be a protrusion extending away from the lens frames 1205, 1210, respectively, and can be positioned behind the user's ears to secure the wearable computing device 1200 to the user. The extended side arms 1230, 1235 can further secure the wearable computing device 1200 to the user by extending around the back of the user's head. Additionally or alternatively, for example, the wearable computing device 1200 can be connected to or attached to a head-mounted helmet structure. Other configurations of the wearable computing device are also possible.

[0110] The on-board computing system 1245 is shown as being positioned on the extended side arm 1230 of the wearable computing device 1200; however, the on-board computing system 1245 may be provided on other portions of the wearable computing device 1200, or may be positioned remote from the wearable computing device 1200 (e.g., the on-board computing system 1245 may be wired or wirelessly connected to the wearable computing device 1200). The on-board computing system 1245 may include, for example, a processor and memory. The on-board computing system 1245 may be configured to receive and analyze data from the image capture device 1240 (and possibly from other sensory devices), and generate images for output by the lens elements 1220, 1225.

[0111] The image capture device 1240 may be, for example, a camera configured to capture still images and / or capture video. In the illustrated configuration, the image capture device 1240 is positioned on the extended side arm 1230 of the wearable computing device 1200; however, the image capture device 1240 may be provided on other portions of the wearable computing device 1200. The image capture device 1240 may be configured to capture images at various resolutions or at different frame rates. Many image capture devices with a small form factor, such as cameras used in mobile phones or webcams, may be incorporated into the example of the wearable computing device 1200.

[0112] One image capture device 1240 is shown. However, more image capture devices may be used, and each device may be configured to capture the same view or to capture a different view. For example, the image capture device 1240 may be forward facing to capture at least a portion of the real world view perceived by the user. This forward facing image captured by the image capture device 1240 may then be used to generate an augmented reality in which the computer generated image appears to interact with or overlay the real world view perceived by the user.

[0113] Wearable computing device 1200 can be used (e.g., on-board computing system 1245) to interpret certain gestures (e.g., performed by one or more hands of the wearer) as user input. For example, wearable computing device 1200 can capture gestures by analyzing image data from image capture device 1240, and initiate tasks defined as corresponding to certain gestures, as discussed in more detail above and exemplified below. In addition, speaker 1250 and microphone 1255 can be used as input / output components. For example, microphone 1255 can detect speech that can be used to supplement and / or modify the intent associated with a gesture. The speaker can be configured to output an audible signal that a user can hear. The audible signal can be related to a task associated with the task in response to the gesture. For example, the audible signal can be a beep indicating that the task has been completed.

[0114] In some embodiments, Figures 13A-13C As shown in Figures 14A-14B, the user interface can determine the user's intent. In some embodiments, it should be noted that the computing device can determine the user's intent based on a combination of gestures and verbal commands from the digital assistant, such as Figures 13A-13C In some implementations, a computing device can perform a task that can include assisting with pronunciation (e.g., Fig.14A ), provide information about the pointed text (e.g. Fig. 14B )wait.

[0115] In some embodiments, the user interface can perform a task that can include saving business card information in the user's contacts. In some embodiments, the computing device can translate text from one language to another. These are just some exemplary tasks, and the present disclosure is not limited to these tasks.

[0116] In some embodiments, the user interface can be integrated with multi-modal context input. For example, the context can come from the content pointed to by the user, voice tasks, applications, historical queries, etc., and can be associated with other applications.

[0117] The techniques described herein do not require any additional hardware and require minimal training sets to train the user interface and can solve real-world scenarios. Furthermore, the proposed mechanism can run on basic devices (e.g., wearables, smartphones, etc.) and is device-independent. That is, an implementation can include a computing device that can be connected to any developer application that requires gestures.

[0118] In some embodiments, frames from a camera of a computing device may be streamed in real time to a processor of the computing device, and a deep learning convolutional network of a user interface may recognize gestures (e.g., whitelisted gestures such as pointing, OK, stop, etc.) and their landmarks (e.g., joints). In an exemplary embodiment, battery efficiency may be improved via an additional process for real-time tracking of hands and landmarks. In some embodiments, an exemplary use case may include integration with a voice-triggered digital assistant. Such exemplary use cases may include: a user points to some text and says "read this text to me", points to a piece of text and says "translate it into French", etc. In the following, additional examples are described.

[0119] Near Field Gestures - In some embodiments, the user interface may capture user interactions in the near field, for example using a single camera. It should be noted that unlike a desktop or mobile device where a pointer may be placed on top of an entity of interest, with a computing device including a user interface, gestures may be placed slightly below the entity of interest to avoid or minimize an opaque finger covering the entity, e.g., text being read, interpreted, translated, etc., as described above.

[0120] In an exemplary embodiment, the gesture may include pointing to pick / select an entity. Such embodiments may include, for example, asking for help translating a word while reading, saving a confirmation number or seat number from a ticket, pointing to ingredients in a food package label, and obtaining (or retrieving) related information.

[0121] In an exemplary embodiment, a gesture may include swiping to select a collection of entities. This is helpful for use cases such as reading a line of text, saving (memorizing) a quote from a book, etc.

[0122] In an exemplary embodiment, gestures may include selecting by drawing a circle or box. This may help to select larger groups of entities.

[0123] In an exemplary embodiment, the gesture may include a double tap or a stroke as an interaction to cancel a task. For example, a double tap may be synonymous with cancel or stroke in a user environment.

[0124] User Experience - In some embodiments, a pointing gesture may be triggered when the device is actively receiving camera input. In an exemplary embodiment, a gesture-based intent may be registered when the computing device is turned on and registers a gesture in the field of view (FOV). In an exemplary embodiment, an explicit trigger command like "this" may be used to register a gesture-based intent. For example, a translation example may include:

[0125] User: “Ok Digital Assistant, What is the meaning of this?”

[0126] Digital assistant (for the first N times): Point with your finger on the word or set of words.

[0127] User: Move finger over word.

[0128] Digital Assistant: Augmented reality is a technology thatsuperimposes a computer-generated image on a user's view of the real world, thus providing a composite view.

[0129] In an exemplary embodiment, the mechanism may include explicit triggers to determine the task and recognize the gesture. For example, an exemplary user journey may include reading help (in a different language or the same language). For example, such an exemplary embodiment may include:

[0130] User: "Hey Digital Assistant, Read (Translate) this."

[0131] Digital assistant (for the first N times or when there is dense text): "Point to what you want to read"

[0132] User: Move finger over word

[0133] Digital Assistant: "Augmented reality is a technology that superimposes acomputer-generated image on a user's view of the real world, thus providing a composite view."

[0134] In an exemplary embodiment, the digital assistant's default language setting may be used to read the text for translation.

[0135] Advanced Learning (definitions, synonyms, pronunciation, etc.) - In some implementations, advanced learning may be supported. For example, such exemplary implementations may include:

[0136] User: "Hey Digital Assistant. What does this word mean" and points to the word

[0137] Digital Assistant: “Diaspora means…”

[0138] User: "Ok Digital Assistant. How do I pronounce this" and point to the word

[0139] Digital Assistant: “This word is pronounced as / / .Would you like me to repeat it? (The pronunciation of this word is / / . Do you want me to repeat it? )”

[0140] User: "Hey Digital Assistant. Can you tell me a synonym of this word?" and points to the word (e.g., augmentation).

[0141] Digital assistants: “The synonym of augmentation is amplification.”

[0142] Object recognition for disambiguating products - In some embodiments, object recognition for disambiguating products may be supported. For example, such exemplary embodiments may include:

[0143] User: "Hey Digital Assistant. What is this?" Point to object in shopping aisle

[0144] Digital Assistant: “This is teriyaki ginger sauce”

[0145] Superhuman Memory - In some implementations, superhuman memory may be supported. For example, such exemplary implementations may include:

[0146] User: "Ok Digital Assistant. Remember this quote." Move your finger over or circle the phrase.

[0147] Digital Assistant: "Saving note to your keep: The time is always right to do what is right."

[0148] Information Seeker - Example implementations may include:

[0149] User: "Hey Digital Assistant. Tell me more about this." Points to the product's ingredients.

[0150] Digital Assistant: "Omega3 fatty acids are polyunsaturated fatty acids characterized by a double bond three atoms away from the terminal methyl group."

[0151] In some embodiments, gestures can support different users to meet their most needs. For example, such exemplary embodiments may include: recognizing pointers on the index finger and thumb (e.g., the thumb is common when lifting an object), training for left and right hands, training for multiple skin colors, finger and nail lengths, training when there is art and accessories on the fingers (e.g., nail polish, jewelry, etc.).

[0152] In some embodiments, advanced intents in the near field view may be supported. Such exemplary embodiments may include pinching and expanding to zoom an entity, raising a hand to stop the digital assistant's voice, swiping the hand left and right to move forward and backward respectively, and giving a thumbs up to indicate good.

[0153] In some embodiments, the field of view may be expanded to support distances longer than the length of a hand. In an exemplary embodiment, such functionality may be supported with one or more cameras that may help triangulate gestures that a user may be referencing in the far field of view. As described above, the interaction of pointing, swiping, etc. may be similar, but the finger may be overlaid on the object behind it. Additional functionality may be supported, such as grabbing and dragging to move virtual objects in the field of view.

[0154] Fig.15 Examples of computer device 1500 and mobile computer device 1550 that can be used with the techniques described herein are shown. Computing device 1500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 1550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the inventions described and / or claimed in this document.

[0155] The computing device 1500 includes a processor 1502, a memory 1504, a storage device 1506, a high-speed interface 1508 connected to the memory 1504 and a high-speed expansion port 1510, and a low-speed interface 1512 connected to a low-speed bus 1514 and the storage device 1506. Each of the components 1502, 1504, 1506, 1508, 1510, and 1512 is interconnected using various buses and can be mounted on a common motherboard or otherwise as appropriate. The processor 1502 can process instructions executed within the computing device 1500, including instructions stored in the memory 1504 or on the storage device 1506 to display graphical information for a GUI on an external input / output device (such as a display 1516 coupled to the high-speed interface 1508). In other embodiments, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Additionally, multiple computing devices 1500 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0156] Memory 1504 stores information within computing device 1500. In one implementation, memory 1504 is one or more volatile memory units. In another implementation, memory 1504 is one or more non-volatile memory units. Memory 1504 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0157] Storage device 1506 can provide mass storage for computing device 1500. In one embodiment, storage device 1506 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a magnetic tape device, a flash memory or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. A computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 1504, storage device 1506, or memory on processor 1502.

[0158] The high-speed controller 1508 manages bandwidth-intensive operations for the computing device 1500, while the low-speed controller 1512 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 1508 is coupled to the memory 1504, the display 1516 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 1510, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 1512 is coupled to the storage device 1506 and the low-speed expansion port 1514. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), is coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0159] As shown in the figure, computing device 1500 can be implemented in many different forms. For example, it can be implemented as a standard server 1520, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 1524. In addition, it can be implemented in a personal computer such as a laptop computer 1522. Alternatively, components from computing device 1500 can be combined with other components in a mobile device (not shown) such as device 1550. Each of such devices can contain one or more of computing devices 1500, 1550, and the entire system can be composed of multiple computing devices 1500, 1550 communicating with each other.

[0160] Computing device 1550 includes processor 1552, memory 1564, input / output devices such as display 1554, communication interface 1566 and transceiver 1568, among other components. Device 1550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 1550, 1552, 1564, 1554, 1566, and 1568 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

[0161] Processor 1552 may execute instructions within computing device 1550, including instructions stored in memory 1564. The processor may be implemented as a chipset of chips including separate and multiple analog and digital processors. For example, the processor may provide coordination for other components of device 1550, such as control of a user interface, applications executed by device 1550, and wireless communications performed by device 1550.

[0162] The processor 1552 can communicate with the user through a control interface 1558 and a display interface 1556 coupled to the display 1554. The display 1554 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 1556 may include appropriate circuits for driving the display 1554 to present graphics and other information to the user. The control interface 1558 may receive commands from the user and convert them for submission to the processor 1552. In addition, an external interface 1562 in communication with the processor 1552 may be provided to enable near-area communication of the device 1550 with other devices. The external interface 1562 may, for example, be provided for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

[0163] The memory 1564 stores information within the computing device 1550. The memory 1564 may be implemented as one or more of: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. The expansion memory 1574 may also be provided and connected to the device 1550 via an expansion interface 1572, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 1574 may provide additional storage space for the device 1550, or may also store applications or other information for the device 1550. Specifically, the expansion memory 1574 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 1574 may be provided as a security module for the device 1550, and may be programmed with instructions that allow the device 1550 to be used securely. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0164] As discussed below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 1564, expansion memory 1574, or memory on processor 1552, which may be received, for example, via transceiver 1568 or external interface 1562.

[0165] Device 1550 can communicate wirelessly through communication interface 1566, which may include digital signal processing circuitry as necessary. Communication interface 1566 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication may occur, for example, through radio frequency transceiver 1568. In addition, short-range communication may occur, for example, using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, GPS (global positioning system) receiver module 1570 may provide additional navigation and location-related wireless data to device 1550, which may be used as appropriate by applications running on device 1550.

[0166] Device 1550 may also communicate audibly using audio codec 1560, which may receive spoken information from a user and convert it into usable digital information. Audio codec 1560 may likewise generate audible sounds for the user, such as through a speaker, for example, in a headset of device 1550. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 1550.

[0167] As shown in the figure, computing device 1550 can be implemented in many different forms. For example, it can be implemented as a cellular phone 1580. It can also be implemented as part of a smart phone 1582, a personal digital assistant, or other similar mobile device.

[0168] In a general aspect, a device, a system, a non-transitory computer-readable medium (having a computer-executable program code stored thereon that can be executed on a computer system), and / or a method can perform processing by a method including the following: capturing an image; determining an environment in which a user is operating a computing device; detecting gestures based on objects in the image; using a machine learning model, determining the user's intent based on the gesture and the environment; and performing a task based at least on the determined intent.

[0169] The system may include: a memory storing a set of instructions; and a processor configured to execute the set of instructions to cause the system to capture an image; determine an environment in which a user is operating a computing device; detect a gesture based on an object in the image; use a machine learning model to determine the user's intent based on the gesture and the environment; and perform a task based at least on the determined intent.

[0170] Implementations may include one or more of the following features. For example, determining the user's intent may further include translating the user's interaction with the real world, and using the interaction and gesture to determine the user's intent. The machine learning model may be based on a computer vision model. The user's intent may be determined using a first machine learning model and a second machine learning model. The method may further include continuous tracking of a hand associated with a gesture using a second machine learning model. Images may be captured using a single non-depth sensing camera of a computing device. The task may be based on the use of a computer assistant. The task may include at least one of a visual and an audible output. The machine learning model may be trained using multiple images including at least one gesture, the machine learning model may be trained using multiple ground truth images of the gesture, a loss function may be used to confirm a match between the gesture and the ground truth image of the gesture, and detection of the gesture based on an object in the image includes matching the object with the gesture that matches the ground truth image of the gesture. The machine learning model may be trained using multiple images, each of the multiple images including at least one object, and at least one object may have an associated ground truth box. The machine learning model may generate multiple bounding boxes, the machine learning model may determine multiple features based on at least a portion of the object within the bounding box, the machine learning model may identify the object based on the multiple features, and the user's intent may be determined based on the identified object.

[0171] Although the example embodiments may include various modifications and alternative forms, their embodiments are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that the example embodiments are not intended to be limited to the specific forms disclosed, but on the contrary, the example embodiments will cover all modifications, equivalents, and alternative forms falling within the scope of the claims. Throughout the description of the drawings, the same reference numerals refer to the same elements.

[0172] Various embodiments of the systems and techniques described herein may be implemented using digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device. Various embodiments of the systems and techniques described herein may be implemented and / or generally referred to herein as circuits, modules, blocks, or systems that can be combined in software and hardware. For example, a module may include functions / behaviors / computer program instructions that are executed on a processor (e.g., a processor formed on a silicon substrate, a GaAs substrate, etc.) or some other programmable data processing device.

[0173] Some of the above example embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of the operations may be rearranged. These processes may be terminated when their operations are completed, but may also have additional steps not included in the figure. These processes may correspond to methods, functions, programs, subroutines, subprograms, etc.

[0174] The methods discussed above (some of which are illustrated by flow charts) may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. A processor may perform the necessary tasks.

[0175] The specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments are embodied in many alternate forms, however, and should not be construed as limited to only the embodiments set forth herein.

[0176] It should be understood that, although the terms first, second, etc. can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first element can be called the second element, and similarly, the second element can be called the first element without departing from the scope of the exemplary embodiment. As used herein, the terms and / or include any and all combinations of one or more of the related enumerated items.

[0177] It should be understood that when an element is referred to as being connected or coupled to another element, it may be directly connected or coupled to the other element or there may be intervening elements. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements. Other words used to describe the relationship between elements (e.g., between versus directly between, adjacent versus directly adjacent, etc.) should be interpreted in a similar manner.

[0178] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms a / an and the are also intended to include plural forms. It should be further understood that the terms contain, include, include and / or include when used in this article, specifying the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0179] It should also be noted that, in some alternative implementations, the functions / behaviors indicated may not occur in the order indicated in the figures. For example, depending on the functionality / behaviors involved, two figures shown in succession may actually be executed concurrently or may sometimes be executed in the reverse order.

[0180] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It should be further understood that, unless explicitly defined as such herein, terms (e.g., those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense.

[0181] The various parts and corresponding detailed descriptions of the above-mentioned example embodiments are presented in terms of software or algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations are descriptions and representations used by those of ordinary skill in the art to effectively convey the essence of their work to other persons of ordinary skill in the art. An algorithm (when the term is used here, and when it is used generally) is considered to be a self-consistent sequence of steps leading to a desired result. These steps are those steps that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of optical, electrical or magnetic signals that can be stored, transferred, combined, compared and otherwise manipulated. Sometimes, mainly due to common reasons, it has proven to be convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0182] In the above illustrative embodiments, references to behaviors and symbolic representations of operations (e.g., in the form of flowcharts) that can be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types and can be described and / or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate arrays (FPGAs), computers, etc.

[0183] It should be remembered, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, or as is apparent from the discussion, terms such as process or compute or calculate or determine display refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical electronic quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0184] It is also noted that the software-implemented aspects of the example embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be a twisted pair, a coaxial cable, an optical fiber, or some other suitable transmission medium known in the art. The example embodiments are not limited by these aspects of any given implementation.

[0185] Finally, it should also be noted that although the appended claims set out particular combinations of features described herein, the scope of the present disclosure is not limited to the particular combinations thereafter claimed, but instead extends to include any combination of features or embodiments disclosed herein, regardless of whether the particular combination has been specifically enumerated in the appended claims at this time.

Claims

1. A method comprising: Capture an image; Using computer vision models to identify the environment in which a user is operating a computing device; detecting a gesture based on an object in the image; selecting a model based on the environment; using the model, determining the user's intent based on the gesture and the environment; as well as performing a task based at least on the intent, Wherein, determining the user's intention includes: mapping the gesture to the user's intent based on the environment, and The user's intent is determined using the mapping of the gesture corresponding to the environment.

2. The method of claim 1, wherein: Determining the user's intent further includes: Translating said user's interaction with the real world, and The interaction and the gesture are used to determine the user's intent.

3. The method of claim 1, wherein: The model is based on the computer vision model.

4. The method according to any one of claims 1 to 3, wherein: The first model and the second model are used to determine the user's intention, and the method further includes: A hand associated with the gesture is continuously tracked using the second model.

5. The method according to any one of claims 1 to 3, wherein: The image is captured using a single non-depth sensing camera of the computing device.

6. The method according to any one of claims 1 to 3, wherein: The tasks are based on the use of a computer assistant.

7. The method according to any one of claims 1 to 3, wherein: The task includes at least one of a visual output and an audible output.

8. The method according to any one of claims 1 to 3, wherein: The model is trained using a plurality of images including at least one gesture, The model is trained using multiple ground truth images of hand gestures. The loss function is used to confirm the match between the gesture and the ground truth image of the gesture, and Detecting the gesture based on an object in the image includes matching the object to the gesture that matches a ground truth image of the gesture.

9. The method according to any one of claims 1 to 3, wherein: The model is trained using a plurality of images, each of the plurality of images containing at least one object, and The at least one object has an associated ground truth box.

10. The method according to any one of claims 1 to 3, wherein: The model generates multiple bounding boxes, The model determines a plurality of features based on at least a portion of the object within the bounding box, The model identifies the object based on the plurality of features, and The user's intent is determined based on the identified object.

11. A non-transitory computer-readable storage medium containing instructions which, when executed by a processor of a computer system, cause the processor to perform the method of any one of claims 1 to 10.

12. A computer system for compressing an image, the computer system being configured to execute the method according to any one of claims 1 to 10.

13. A system comprising: a memory for storing a set of instructions; as well as a processor configured to execute the set of instructions to cause the system to: Capture an image; Using computer vision models to identify the environment in which a user is operating a computing device; detecting a gesture based on an object in the image; selecting a model based on the environment; using the model, determining the user's intent based on the gesture and the environment; as well as performing a task based at least on the intent, Wherein, determining the user's intention includes: mapping the gesture to the user's intent based on the environment, and The user's intent is determined using the mapping of the gesture corresponding to the environment.

14. The system of claim 13, wherein: Determining the user's intent further includes: Translating said user's interaction with the real world, and The interaction and the gesture are used to determine the user's intent.

15. The system of claim 13, wherein: The model is based on a computer vision model.

16. A system as claimed in any one of claims 13 to 15, wherein: The first model and the second model are used to determine the user's intent, and the set of instructions is executed by the processor to further cause the system to: The hand is continuously tracked using the second model.

17. The system of any one of claims 13 to 15, wherein: The image is captured using a single non-depth sensing camera of the computing device.

18. The system of any one of claims 13 to 15, wherein: The tasks are performed using a computer assistant.

19. The system of any one of claims 13 to 15, wherein: The task includes at least one of a visual output and an audible output.

20. The system of any one of claims 13 to 15, wherein: The model generates multiple bounding boxes, The model determines a plurality of features based on at least a portion of the object within the bounding box, The model identifies the object based on the plurality of features, and The user's intent is determined based on the identified object.

21. The system of any one of claims 13 to 15, wherein: The model is trained using a plurality of images including at least one gesture, The model is trained using multiple ground truth images of hand gestures. The loss function is used to confirm the match between the gesture and the ground truth image of the gesture, and Detecting the gesture based on an object in the image includes matching the object to the gesture that matches a ground truth image of the gesture.

Images