COMPUTER IMPLEMENTED METHOD, SYSTEM AND COMPUTER PROGRAM PRODUCT FOR SENSOR-INDEPENDENT GESTURE DETECTION (SENSOR-INDEPENDENT GESTURE DETECTION)

Abstract

To provide a computer-implemented method, system, and computer program product for sensor agnostic gesture detection.SOLUTION: A method for sensor agnostic gesture detection includes: recording a gesture using sensor output from a first user device to a first user profile associated with a ubiquitous gesture agent; associating the gesture with a triggering threshold indicating the gesture and based on the sensor output, and a processor-executable action that is executable by one of a plurality of user devices associated with the user profile; transforming the triggering threshold into a triggering threshold corresponding to a different type of sensor; identifying the gesture using data from the different type sensor that satisfies the corresponding triggering threshold; and implementing, by the one of the plurality of user devices, the processor-executable action associated with the gesture.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the Internet of Things (IoT), and more particularly, to sensor-independent gesture detection in an IoT environment. [Background technology]

[0002] IoT refers to a network of physical objects embedded with sensors, software, and other data processing capabilities that collectively enable the exchange of data between the physical objects and other devices for various purposes. Users are increasingly surrounded by IoT-enabled devices. As a result, IoT-enabled devices are increasingly being used to facilitate usability at the human-machine interface. Summary of the Invention [Problem to be solved by the invention]

[0003] A computer-implemented method, system, and computer program product for sensor-independent gesture detection is provided. [Means for solving the problem]

[0004] Aspects of the disclosure are directed to a computer-implemented method that includes recording a gesture in a first user profile associated with a ubiquitous gesture agent using a sensor output from a first user device. The method further includes associating the gesture with a trigger threshold indicative of the gesture and based on the sensor output and a processor-executable action executable by one of a plurality of user devices associated with the user profile. The method further includes converting the trigger threshold to a trigger threshold corresponding to a different type of sensor. The method further includes identifying the gesture using data from the different type of sensor that satisfies the corresponding trigger threshold. The method further includes performing the processor-executable action associated with the gesture by one of the plurality of user devices.

[0005] Further aspects of the present disclosure are directed to systems and computer program products configured to perform the above-described methods.This Summary is not intended to describe every aspect, example, or embodiment of the present disclosure or combinations thereof.

[0006] The drawings contained herein are incorporated in and form a part of the specification. They illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure. The drawings are merely illustrative of particular embodiments and are not intended to limit the disclosure. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating an example of a computing environment including a ubiquitous gesture agent, according to some embodiments of the present disclosure. [Diagram 2] 1 is a flowchart illustrating an example method for utilizing a ubiquitous gesture agent according to some embodiments of the present disclosure. [Diagram 3] 1 is a flowchart illustrating an example of a method for defining a gesture using a ubiquitous gesture agent according to some embodiments of the present disclosure. [Figure 4] 1 is a flowchart illustrating an example method for identifying a gesture using the same or a different sensor than the sensor originally associated with the gesture, according to some embodiments of the disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating an example computer in accordance with some embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates a cloud computing environment in accordance with some embodiments of the present disclosure. [Figure 7] FIG. 2 illustrates an abstract model layer according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] While the present disclosure is susceptible to various modifications and alternative forms, details of which have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that there is no intention to limit the disclosure to the particular embodiments described. On the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the disclosure.

[0009] Aspects of the present disclosure are directed to the Internet of Things (IoT), and more specifically, to sensor-independent gesture detection in an IoT environment. Although not limited to such applications, embodiments of the present disclosure may be better understood in light of the foregoing context.

[0010] A gesture is a movement of one or more parts of the body (typically the hands and / or head) that expresses an idea or meaning. The gesture can be used to activate pre-programmed functions in a device equipped with a sensor capable of detecting the gesture or in a device communicatively coupled to a device equipped with a sensor capable of detecting the gesture. For example, the device can include, but is not limited to, a smartwatch, a smartphone, earphones, locks (e.g., in a vehicle, house, etc.), health monitors, etc. The detected and pre-programmed gesture can be configured to execute an action on the device. The action can be, for example, unlocking a smartphone, contacting emergency services, issuing an emergency signal, making a call, sending a message, interacting with an application (e.g., starting or stopping an exercise in a health-related application), accessing a remote server (e.g., writing data and / or retrieving data), activating a physical actuator (e.g., unlocking a door), etc. Overall, associating gestures with pre-programmed actions can help improve the usability of various devices in an IoT environment.

[0011] One challenge associated with pre-programmed gestures relates to the various gestures associated with different devices: a gesture to shake a smartwatch is commonly used to launch an app on a smartphone, but if a user forgets their smartwatch, the gesture is useless. To address this and other challenges associated with pre-programmed gestures, aspects of the present disclosure are directed to methods, systems, and computer program products for sensor-independent gesture detection.

[0012] Aspects of the present disclosure enable sensor-independent gesture detection by using a ubiquitous gesture agent to store various recorded gestures and their associated actions. The ubiquitous gesture agent can then turn gestures defined by a first type of sensor output from a first device into a translated output measurable by a second type of sensor output from a second device. For example, following the aforementioned example of shaking a smartwatch, the ubiquitous gesture agent can convert accelerometer data associated with the smartwatch into accelerometer data associated with a smartphone. This allows a user to achieve a similar result (e.g., launching an application) by shaking the smartphone instead of shaking the smartwatch. As another example, the ubiquitous gesture agent can convert accelerometer data associated with the smartwatch into motion data associated with a camera (e.g., a camera in a smart building where a user can opt-in to take advantage of the IoT features of the smart building). In this way, instead of shaking their smartwatch, a user can instead wave their hand, which, when detected by the smart building's cameras, will send a command to the ubiquitous agent, which will then cause the smartphone to perform an action (e.g., launch an application).

[0013] Accordingly, aspects of the present disclosure provide many advantages and improvements. As one general example, aspects of the present disclosure improve usability by allowing users to utilize gestures even in the absence of the device that typically or initially detected those gestures. More specifically, aspects of the present disclosure can translate trigger thresholds detectable by one type of sensor to corresponding trigger thresholds detectable by another type of sensor. Additionally, aspects of the present disclosure utilize a cloud-hosted ubiquitous gesture agent to enable this functionality. Using a cloud-hosted ubiquitous gesture agent provides near-continuous availability without draining computational resources from user devices. Additionally, the ubiquitous gesture agent helps simplify interactions with smart buildings and other IoT resources, allowing users to leverage IoT resources in their domains. Collectively, these improvements, advantages, and functionality can be summarized as sensor-independent gesture detection.

[0014] Referring to the figures, FIG. 1 illustrates an example of a computer environment according to some embodiments of the present disclosure. The computer environment includes a ubiquitous gesture agent 100 communicatively coupled to a plurality of user devices 132 associated with a user profile 102 and one or more environmental sensors 136 via a network 138. The ubiquitous gesture agent 100 can be configured to provide functionality that allows for sensor-independent gesture detection. In other words, the ubiquitous gesture agent 100 can provide functionality that allows a gesture recorded and thereby detectable by a first sensor 1 134-1 to be detectable by a different type of sensor, such as, for example, a sensor 2 134-2 (e.g., another sensor associated with the user profile 102) or an environmental sensor 136 (e.g., a sensor associated with a different user profile (not shown), but still capable of providing functionality to the user profile 102 via the ubiquitous gesture agent 100).

[0015] The ubiquitous gesture agent 100 may include multiple user profiles, such as user profile 102. The user profile may be associated with an individual user (e.g., a person), a business, a company, a municipality, a commercial building, a residential building, and the like. The user profile 102 may include recorded gestures 104. The recorded gestures 104 may include a sensor ID 106 that identifies at least one sensor 134 associated with at least one user device of a plurality of user devices 132 that provides a sensor output 108 indicative of the recorded gestures 104. The sensor output 108 may represent raw data collected from the at least one sensor 134 indicative of the recorded gestures 104. The value category 110 may identify a type of raw data and / or a unit of measurement associated with the raw data from the sensor output 108. As an example, the value category 110 may identify “acceleration” or “meters per second” as the type of raw data and / or a unit of measurement associated with the raw data, respectively. 2 " or both.

[0016] The user profile 102 may further include a threshold 112. The threshold 112 may be configured to take the sensor ID 106, the sensor output 108, or the value category 110, or a combination thereof, and generate a threshold useful for identifying the recorded gesture 104. Thus, the threshold 112 may include sanitized data 114, normalized data 116, or scaled data 118, or a combination thereof. The sanitized data 114 may be generated by removing erroneous, anomalous, or unwanted data. The sanitized data 114 may also be used to infer other data from the sensor output 108. For example, if the sensor output 108 measures acceleration, the sanitized data 114 may convert the raw acceleration measurements into respective changes in acceleration, which may be a more useful metric for identifying the recorded gesture 104 than the acceleration data itself. The normalized data 116 may be configured to normalize the sensor output 108 between 0 and 1, inclusive. The scaled data 118 is an alternative to the normalized data 116. The scaled data 118 can be configured to shift the sensor output 108 such that the mean (or median) of the sensor output 108 is set to 0. The normalized data 116 and / or the scaled data 118 can simplify and / or standardize the data evaluation.

[0017] Regardless of whether or how sanitized data 114, normalized data 116, or scaled data 118, or a combination thereof, may be used to clean and simplify sensor output 108, threshold value 112 ultimately generates trigger threshold 120. Trigger threshold 120 is a value, sequence of values, or pattern of values ​​from sensor ID 106 that identifies a recorded gesture 104. For example, to detect vibrations (e.g., from sensor 1 134-1) from accelerometer data of a smartwatch (e.g., one of multiple user devices 132), trigger threshold 120 may include a threshold number of acceleration changes within a threshold time, where each acceleration change meets a threshold magnitude.

[0018] Advantageously, aspects of the present disclosure further generate one or more corresponding trigger thresholds 122. The corresponding trigger thresholds 122 are thresholds that can identify gestures 104 recorded using one or more different types of sensors. For example, one of the corresponding trigger thresholds 122 can be associated with a different sensor 2 134-2 associated with a different one of a plurality of user devices 132 (e.g., smartphones). In this example, the corresponding trigger thresholds 122 can convert the trigger thresholds 120 into corresponding trigger thresholds 122 detectable by the different sensor 2 134-2 of different types of motion sensors (e.g., gravity sensors, gyroscopes, linear accelerometers, etc.). As another example, another corresponding trigger threshold 122 can be associated with one or more environmental sensors 136. For example, the environmental sensors 136 can be cameras that generate video data in a smart building. In this example, the corresponding trigger thresholds 122 can convert the trigger thresholds 120 based on acceleration data into motion thresholds that are useful for inferring acceleration from the video data. In this manner, aspects of the present disclosure allow a recorded gesture 104 to be identified by additional types of devices and sensors than the devices and sensors that originally created the recorded gesture 104.

[0019] In some embodiments, the threshold 112 may be automatically generated by utilizing a machine learning algorithm, including but not limited to decision tree learning, association rule learning, artificial neural networks (ANN), recurrent neural networks (RNN), deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity / metric training, sparse dictionary learning, genetic algorithms, rule-based learning, or other machine learning techniques or combinations thereof.

[0020] For example, machine learning algorithms include k-nearest neighbors (KNN), learning vector quantization (LVQ), self-organizing maps (SOM), logistic regression, least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression splines (MARS), ridge regression, lasso regression (LASSO), elastic net, least angle regression (LARS), probabilistic classifier, naive Bayes classifier, binary classification, linear classifier, hierarchical classifier, canonical correlation analysis (CCA), factor analysis, independent component analysis (ICA), linear Discriminant Analysis (LDA), Multidimensional Scaling (MDS), Nonnegative Matrix Factorization (NMF), Partial Least Squares Regression (PLSR), Principal Component Analysis (PCA), Principal Component Regression (PCR), Sammon Mapping, t-Distribution Stochastic Neighbor Embedding (SNE), Bootstrap Aggregation, Ensemble Average, Gradient Boosted Decision Tree (GBRT), Gradient Boosted Machine (GBM), Inductive Bias Algorithm, Q-Learning, State-Action-Reward-State-Action (SARSA), Temporal Difference Learning ( One or more examples of machine learning techniques or combinations thereof may be utilized, including but not limited to: TD learning, a priori algorithm, equivalent class transformation algorithm (ECLAT algorithm), Gaussian process regression, gene expression programming, group data processing method (GMDH), inductive logic programming, instance-based learning, logistic model trees, information fuzzy networks (IFN), hidden Markov model Gaussian naive Bayes, polynomial naive Bayes, average-one dependence estimator (AODE), Bayesian networks (BN), classification and regression trees (CART), chi-squared automatic interaction detector (CHAID), expectation maximization algorithm, feed-forward neural networks, logic learning machines, self-organizing maps, single linkage clustering, fuzzy clustering, hierarchical clustering, Boltzmann machines, convolutional neural networks, recurrent neural networks, hierarchical temporal memory (HTM), or other machine learning techniques.

[0021] In summary, machine learning algorithms may be used to generate the sanitized data 114, the normalized data 116, the scaled data 118, the trigger threshold 120, and / or the corresponding trigger threshold 122. Although not shown, in some embodiments, machine learning algorithms may be used to identify salient features of the sensor output 108 (sometimes referred to as feature detection or feature selection).

[0022] The user profile 102 may further include a processor executable action 124 associated with each recorded gesture 104. The processor executable action 124 may include, but is not limited to, activating an actuator (e.g., locking or unlocking a door), accessing a remote server (e.g., reading, writing, or otherwise accessing, transmitting, or refreshing data, or a combination thereof), and performing an action on a smartphone (e.g., unlocking the smartphone, sending a message on the smartphone, calling using the smartphone, sending an emergency signal using the smartphone, opening or interacting with an application executable by the smartphone, etc.). The processor executable action 124 may be associated with one of a plurality of user devices 132. The processor executable action 124 may include an action ID 126 that identifies the action, a target 128 that identifies a device on which to perform the processor executable action 124, and one or more functions 130 that include logic configured to execute the processor executable action 124 by the target 128.

[0023] The plurality of user devices 132 associated with the user profile 102 may include any number of user devices now known or later developed. For example, the plurality of user devices 132 may include smartphones, smart watches, smart clothing, home, appliances, doors, cameras, vehicles, wearable devices, health monitoring devices, accessories (rings, earrings, headbands, etc.), etc. The sensors 134 (referring to any one or more of sensor 1 134-1, sensor 2 134-2, or sensor N 134-N, or combinations thereof) may refer to any type of hardware or software device or both capable of collecting data. The sensors 134 may include, but are not limited to, temperature sensors, chemical sensors, humidity sensors, pressure sensors, angular or linear motion sensors (e.g., accelerometers, linear accelerometers, step counters, gyroscopes, gravity sensors, etc.), rotational sensors, angular or linear motion sensors, angular or linear velocity sensors, biometric sensors, microphones, cameras, etc.

[0024] The environmental sensor 136 may be a sensor stored by the ubiquitous gesture agent 100 that is associated with another user profile other than the user profile 102. In other words, the environmental sensor 136 may be a sensor that does not belong to the user of the user profile 102 but is in the vicinity of the user associated with the user profile 102. The environmental sensor 136 may be any of the aforementioned sensors 134. In some embodiments, the environmental sensor 136 may be a camera associated with a smart building. Advantageously, the ubiquitous gesture agent 100 enables the user profile 102 to utilize a corresponding sensor in the vicinity of the user of the user profile 102 when the user profile 102 selects to use an available environmental sensor 136.

[0025] 2, a flowchart of a method 200 for utilizing a ubiquitous gesture agent is shown, according to some embodiments of the present disclosure. In some embodiments, the method 200 may be implemented by a processor, a computer, a server, the ubiquitous gesture agent 100 of FIG. 1, or another configuration of hardware or software or both.

[0026] Operation 202 includes registering a gesture using a sensor output from the first user device to a first user profile associated with the ubiquitous gesture agent. As an example, the gesture can be a wave of the hand as detected by accelerometer data (e.g., sensor output) from an accelerometer built into a smartwatch (e.g., the first user device).

[0027] Operation 204 includes associating the gesture with a trigger threshold and an action. Continuing with the above example, the trigger threshold can include a threshold number of acceleration changes, where each acceleration change exceeds a threshold magnitude within a threshold time interval. In some embodiments, the action can be a processor-executable action, such as the processor-executable actions described above with respect to FIG. 1. Operation 204 is described in more detail below with reference to FIG. 3.

[0028] Operation 206 includes translating the trigger thresholds into corresponding trigger thresholds for different types of sensors. Continuing with the above example, a trigger threshold based on acceleration data output by an accelerometer in a smart watch may be translated into a corresponding trigger threshold based on video data generated by a video camera operating within a smart building and available to a user profile via a ubiquitous gesture agent.

[0029] Act 208 includes identifying a gesture using data from different types of sensors. Continuing with the above example, act 208 includes identifying a user waving based on video data from a video camera when the user is not wearing a smartwatch or when the smartwatch is malfunctioning.

[0030] Acts 210 include implementing an action associated with the gesture in response to identifying the gesture using data from the different types of sensors. Advantageously, acts 210 allow a user to successfully utilize the gesture to control a device even if the user forgets the particular device typically used to identify the gesture.

[0031] 3 is a flowchart illustrating an example of a method for defining a gesture using a ubiquitous gesture agent according to some embodiments of the present disclosure. In some embodiments, the method 300 can be implemented by a processor, a computer, a server, the ubiquitous gesture agent 100 of FIG. 1, or another configuration of hardware or software or both. In some embodiments, the method 300 is a sub-method of operation 204 of FIG. 2.

[0032] Operation 302 includes defining a recorded gesture by recording sensor output from one or more sensors associated with one or more user devices during a gesture recording interval. In some embodiments, a user can operate the one or more user devices to start and end a gesture recording interval, where data collected from one or more sensors associated with the one or more user devices during the start and end can constitute a sensor output indicative of the recorded gesture.

[0033] Operation 304 includes associating the recorded gesture with an identifier, a sensor output, a category, and a processor-executable action. In some embodiments, operation 304 may be performed by a user providing user input to a device that provides an interface for defining gestures and associating the gestures with various processor-executable actions.

[0034] Operation 306 includes applying a data transformation to the sensor output to make the sensor output detectable by a different type of sensor. The data transformation may include, among others, sanitizing the data, normalizing the data, or scaling the data, or a combination thereof. Additionally, the data transformation may include, for example, converting acceleration data to position data, or other types of data transformations. In some embodiments, operation 306 utilizes a machine learning algorithm to apply a data transformation to the sensor output suitable for making the sensor output detectable by a different type of sensor.

[0035] Act 308 includes defining trigger thresholds for various sensors and corresponding trigger thresholds based on the data transformation. Advantageously, act 308 enables a gesture recorded using one sensor (or set of sensors) to be detectable by a different type of sensor (or set of sensors), thereby enabling sensor-independent gesture detection.

[0036] 4 is a flow chart illustrating an example method for identifying a gesture using a sensor that is the same or different from the sensor originally associated with the gesture, according to some embodiments of the disclosure. In some embodiments, the method 400 can be implemented by a processor, computer, server, ubiquitous gesture agent 100 of FIG. 1, or another configuration of hardware or software or both. In some embodiments, the method 400 is a sub-method of operation 208 of FIG. 2.

[0037] Operation 402 includes collecting sensor data for a plurality of sensors from a plurality of user devices and / or environmental sensors. The sensor data can be collected continuously, semi-continuously, or at a predefined polling interval.

[0038] Operation 404 includes determining whether a sensor and / or corresponding user device is missing from the collected sensor data. If not (NO at operation 404), method 400 proceeds to operation 408 to determine whether any of the collected sensor data matches a trigger threshold. However, if any sensor and / or corresponding user device is missing from the collected sensor data (YES at operation 404), method 400 proceeds to operation 406 to cross-reference the collected sensor data with the corresponding trigger threshold to determine whether any gestures have been detected that would have been detected by one or more missing sensors and / or corresponding user devices.

[0039] Operation 408 includes determining whether there is a match between any of the collected sensor data and any of the trigger thresholds or corresponding trigger thresholds. If not (NO at operation 408), method 400 returns to operation 402 to continue collecting sensor data. If so (YES at operation 408), method 400 proceeds to operation 410.

[0040] Operation 410 includes identifying a gesture associated with the match. Advantageously, method 400 can identify gestures in situations where sensors and / or user devices traditionally used to detect gestures are unavailable by leveraging corresponding trigger thresholds associated with other sensors and / or environmental sensors of other user devices.

[0041] FIG. 5 is a block diagram of an exemplary computer 500 according to some embodiments of the present disclosure. In various embodiments, the computer 500 can perform any or all of the methods described in FIGS. 2-4 and / or implement the functions described in FIG. 1. In some embodiments, the computer 500 receives instructions related to the above-mentioned methods and functions by downloading processor-executable instructions from a remote data processing system over the network 550. In other embodiments, the computer 500 provides instructions for the above-mentioned methods or functions to a client machine such that the client machine performs the method or part of the method based on the instructions provided by the computer 500. In some embodiments, the computer 500 is incorporated (or is virtually provisioned with functionality similar to the computer 500) into one or more entities of a computing environment (e.g., the ubiquitous gesture agent 100) and / or other aspects of the present disclosure.

[0042] The computer 500 includes memory 525, storage 530, an interconnect 520 (e.g., a bus), one or more CPUs 505 (also referred to herein as processors), an I / O device interface 510, I / O devices 512, and a network interface 515.

[0043] Each CPU 505 retrieves and executes programming instructions stored in memory 525 or storage device 530. Interconnect 520 is used to move data, such as programming instructions, between CPU 505, I / O device interface 510, storage device 530, network interface 515, and memory 525. Interconnect 520 may be implemented with one or more buses. CPU 505 may be a single CPU, multiple CPUs, or a single CPU with multiple processing cores, in various embodiments. In some embodiments, CPU 505 may be a digital signal processor (DSP). In some embodiments, CPU 505 includes one or more 3D integrated circuits (3DICs) (e.g., 3D wafer level packaging (3DWLP), 3D interposer-based integration, 3D stacked ICs (3D-SICs), monolithic 3D ICs, 3D heterogeneous integration, 3D system-in-package (3DSip), or package-on-package (PoP) CPU configurations), or combinations thereof. Memory 525 is generally representative of random access memory (static random access memory (SRAM), dynamic random access memory (DRAM), flash, etc.). Storage 530 is generally representative of non-volatile memory such as a hard disk drive, solid state device (SSD), removable memory card, optical storage device, or flash memory device. In another embodiment, storage 530 can be replaced by a storage area network (SAN) device, cloud, or other device connected to computer 500 via I / O device interface 510 or other device connected to computer 500 via network 550 via network interface 515.

[0044] In some embodiments, memory 525 stores instructions 560. However, in various embodiments, instructions 560 are stored partially in memory 525, partially in storage device 530, entirely in memory 525, entirely in storage device 530, or accessed over network 550 via network interface 515.

[0045] The instructions 560 may be computer readable and computer executable instructions for performing any portion or all of the methods of Figures 2-4 or for implementing the functionality described in any portion of Figure 1, or both. Although the instructions 560 are shown in memory 525, the instructions 560 may include program instructions collectively stored across a number of computer readable storage media and executable by one or more CPUs 505.

[0046] In various embodiments, I / O device 512 includes an interface that can present information and receive input. For example, I / O device 512 can present information to and receive input from a user interacting with computer 500.

[0047] The computer 500 is connected to a network 550 via a network interface 515. The network 550 may include physical, wireless, cellular, or different networks.

[0048] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that practice of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be practiced in conjunction with any other type of computing environment now known or later developed.

[0049] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0050] The characteristics are as follows:

[0051] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0052] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin- and thick-client platforms (e.g., cell phones, laptops, PDAs).

[0053] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Different physical and virtual resources are dynamically allocated and reallocated depending on demand. Consumers generally have no control or knowledge of the exact location of the resources provided to them, so there is a sense of location independence, although consumers may be able to determine location at a higher level of abstraction (e.g. country, state, data center).

[0054] Rapid elasticity: Computing capacity can be provisioned quickly and elastically, in some cases automatically, to scale out immediately and to be released quickly to scale in immediately. To the consumer, the computing capacity available for provisioning often appears unlimited and can be purchased at any time and in any quantity.

[0055] Metered Services: Cloud systems leverage metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of the services utilized.

[0056] The service model is as follows:

[0057] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure that can be accessed from a variety of client devices through a thin-client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the individual application functions, except for limited user-specific application configuration settings.

[0058] Platform as a Service (PaaS): The capability offered to the consumer is to deploy applications that the consumer creates or acquires using programming languages ​​and tools supported by the provider onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the hosting environment.

[0059] Infrastructure as a Service (IaaS): The facility offered to the consumer is the provision of processors, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications, and in some cases has partial control over some network components (e.g., the host firewall).

[0060] The deployment model is as follows:

[0061] Private Cloud: The cloud infrastructure is dedicated to a specific organization. It can be managed by that organization or a third party and can exist on-premise or off-premise.

[0062] Community Cloud: The cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). The cloud infrastructure can be managed by the organizations or a third party and can exist on-premise or off-premise.

[0063] Public cloud: The cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0064] Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community or public), each of which retains its inherent semantics but is bound together by standards or specific technologies that enable data and application portability (e.g. cloud bursting for load balancing between clouds).

[0065] A cloud computing environment is a service-oriented environment with an emphasis on statelessness, low coupling, modularity and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0066] Referring now to FIG. 6, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, can communicate. The nodes 10 can communicate with each other. They may be grouped (not shown), physically or virtually, into one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This allows the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. It should be understood that the types of computing devices 54A-54N shown in FIG. 6 are intended to be exemplary only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device over any type of network or network-addressable connection or both (e.g., using a web browser).

[0067] Referring now to Figure 7, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 6) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 7 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0068] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, reduced instruction set computer (RISC) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0069] The virtualization layer 70 provides an abstraction layer from which the following virtual entities can be provided, for example: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0070] In one example, the management layer 780 may provide the following functions: Resource provisioning 81 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables allocation and management of cloud computing resources such that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 enables advance arrangement and procurement of anticipated future needs for cloud computing resources according to SLAs.

[0071] The workload layer 90 provides examples of functionality available to a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and sensor independent gesture detection 96.

[0072] An embodiment of the invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0073] A computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. Computer-readable storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, or semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices with instructions recorded on punch cards or ridge structures in grooves, and any suitable combinations of the foregoing. Computer-readable storage media as used herein should not be construed as ephemeral signals in themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0074] The computer readable program instructions described herein can be downloaded to each computing / processing device from a computer readable storage medium or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in each computing / processing device.

[0075] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a stand-alone software package, or partially on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer readable program instructions by utilizing state information of the computer readable program instructions to customize the electronic circuitry for carrying out aspects of the invention.

[0076] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. Each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer readable program instructions.

[0077] The computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of such computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. The computer readable program instructions may further be stored on a computer readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer readable storage medium having instructions stored thereon constitutes an article of manufacture including instructions for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0078] The computer readable program instructions may also be loaded into a computer, other programmable device, or other apparatus and cause a sequence of operational steps to be executed on the computer, other programmable device, or other apparatus to generate a computer-implemented process, such that the instructions executing on the computer, other programmable device, or other apparatus perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0079] The flowcharts and block diagrams in the drawings of the present disclosure illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specific logical function. In some other implementations, the functions shown in the blocks may be performed in a different order than the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as one step, may be performed simultaneously or substantially simultaneously, may be performed in a partially or fully overlapping manner, or may be performed in reverse order depending on the functions involved. It should be noted that each block in the block diagram and / or flowchart, and combinations of multiple blocks in the block diagram and / or flowchart can be performed by a dedicated hardware-based system performing a specific function or operation, or by a combination of dedicated hardware and computer instructions.

[0080] It will be appreciated that the process software (e.g., any of the instructions stored in instructions 560 of FIG. 5 or any software configured to execute any portion of the method described with respect to FIGS. 2-4 and / or implement any portion of the functionality described in FIG. 1) can be deployed by manually loading a storage medium such as a CD, DVD, etc. directly onto the client, server, and proxy computers, but can also be automatically or semi-automatically deployed onto computer systems by sending the process software to a central server or group of central servers. The process software is then downloaded to the client computers that will run the process software. Alternatively, the process software is sent directly to the client systems via email. The process software is then detached to a directory or loaded into a directory by executing a series of program instructions that detach the process software to a directory. Another method is to send the process software directly to a directory on the hard drive of the client computer. If a proxy server is present, the process selects the proxy server code, determines which computer to place the code on, sends the proxy server code, and installs the proxy server code on the proxy computer. The process software is transferred to the proxy server and stored on the proxy server.

[0081] Embodiments of the invention may also be provided as part of a service agreement with a client company, a non-profit organization, a government agency, an internal organizational structure, etc. These embodiments may include configuring a computer system to perform some or all of the methods described herein, and deploying software, hardware, and web services. These embodiments may also include analyzing consumer behavior, making recommendations in response to the analysis, building a system that implements a subset of the recommendations, integrating the system into existing processes and infrastructure, metering usage of the system, allocating costs to users of the system, and billing, payment (e.g., generating an invoice), or otherwise receiving payment for use of the system.

[0082] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. As used herein, the terms "includes" and / or "including" identify the presence of stated features, integers, steps, operations, elements, or components or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups or combinations thereof. In the preceding detailed description of exemplary embodiments of the various embodiments, reference has been made to the accompanying drawings, in which like numerals represent like elements, which form a part hereof and which are shown to illustrate specific exemplary embodiments in which the various embodiments may be practiced. These embodiments have been described in sufficient detail to enable one skilled in the art to practice the embodiments, but other embodiments may be used and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the preceding description, numerous specific details have been set forth to provide a thorough understanding of the various embodiments. However, the various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the embodiments.

[0083] Different examples of the term "embodiment" as used herein may be referenced, although they do not necessarily refer to the same embodiment. Any data and data structures shown or described herein are merely examples, and other embodiments may use different amounts of data, types of data, fields, numbers and types of fields, field names, numbers and types of rows, records, entries, or organizations of data. Furthermore, any data may be combined with logic, so that separate data structures are not necessary. Thus, the above detailed description is not to be construed in a limiting sense.

[0084] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to allow those skilled in the art to understand the embodiments described herein.

[0085] While the present disclosure has been described with respect to specific embodiments, alterations and modifications thereof will be apparent to those skilled in the art, and it is therefore intended that the following claims be interpreted to cover all such changes and modifications that are within the true spirit and scope of the present disclosure.

[0086] Any advantages discussed in this disclosure are exemplary advantages, and there may be embodiments of the disclosure that achieve all, some, or none of any of the advantages discussed while still being within the spirit and scope of the disclosure.

[0087] A non-limiting list of examples is provided below to illustrate some aspects of the disclosure. Example 1 is a computer-implemented method. The method includes recording a gesture in a first user profile associated with a ubiquitous gesture agent using a sensor output from a first user device. The method includes associating the gesture with a trigger threshold that indicates the gesture and is based on the sensor output and a processor-executable action that is executable by one of a plurality of user devices associated with the user profile. The method includes translating the trigger threshold to a trigger threshold that corresponds to a different type of sensor. The method includes identifying a gesture using data from a different type of sensor that satisfies a corresponding trigger threshold. The method includes implementing, by one of the plurality of user devices, the processor-executable action associated with the gesture.

[0088] Example 2 includes the method of example 1 with or without any features. In this example, the ubiquitous gesture agent is hosted in a public cloud environment.

[0089] Example 3 includes the method of any of examples 1-2 with or without the optional feature, in which different types of sensors are associated with the first user profile.

[0090] Example 4 includes the method of any of Examples 1-2 with or without the optional feature, in which different types of sensors are associated with different user profiles stored in the ubiquitous gesture agent.

[0091] Example 5 includes the method of any of examples 1-4, with or without any feature. In this example, the different type of sensor is a camera in a smart building. Optionally, the sensor output includes acceleration data from an accelerometer, and the corresponding trigger threshold is based on acceleration inferred from video data from the camera in the smart building.

[0092] Example 6 includes the method of any of examples 1-5, with or without any features. In this example, the processor-executable action is selected from the group consisting of: invoking an actuator, accessing a remote server, and implementing the action on the smartphone.

[0093] Example 7 includes the method of any of Examples 1-6, with or without any optional features. In this example, the method is performed by one or more computers in accordance with software downloaded to the one or more computers from a remote data processing system. Optionally, the method further includes metering usage of the software and generating an invoice based on the metered usage.

[0094] Example 8 is a system. The system includes one or more processors and one or more computer-readable storage media storing program instructions configured to, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of Examples 1 to 7.

[0095] Example 9 is a computer program product. The computer program product includes one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media. The program instructions include instructions configured to cause one or more processors to perform a method according to any one of Examples 1 to 7.

Claims

1. recording a gesture in a first user profile associated with the ubiquitous gesture agent using sensor output from the first user device; indicating the gesture and associating the gesture with a trigger threshold based on the sensor output and a processor executable action executable by one of a plurality of user devices associated with the user profile; converting the trigger thresholds to corresponding trigger thresholds for different types of sensors; identifying the gesture using data from the different types of sensors that satisfy the corresponding trigger thresholds; implementing, by one of the plurality of user devices, the processor-executable action associated with the gesture; and 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the ubiquitous gesture agent is hosted in a public cloud environment.

3. The computer-implemented method of claim 1 , wherein the different types of sensors are associated with the first user profile.

4. The computer-implemented method of claim 1 , wherein the different types of sensors are associated with different user profiles stored in the ubiquitous gesture agent.

5. The computer-implemented method of claim 1 , wherein the different types of sensors are cameras in a smart building.

6. 6. The computer-implemented method of claim 5, wherein the sensor output includes acceleration data from an accelerometer, and the corresponding trigger threshold is based on acceleration inferred from video data from the cameras in the smart building.

7. The computer-implemented method of claim 1 , wherein the processor-executable action is selected from the group consisting of: initiating an actuator, accessing a remote server, and executing an action on a smartphone.

8. 10. The computer-implemented method of claim 1, wherein the method is performed by one or more computers in accordance with software downloaded to the one or more computers from a remote data processing system.

9. The method comprises: Measuring usage of the software; generating a bill based on said metering of usage; The computer-implemented method of claim 8 , further comprising:

10. one or more processors; and one or more computer-readable storage media storing program instructions configured, when executed by the one or more processors, to cause the one or more processors to perform a method, the method comprising: recording a gesture in a first user profile associated with the ubiquitous gesture agent using sensor output from the first user device; indicating the gesture and associating the gesture with a trigger threshold based on the sensor output and a processor executable action executable by one of a plurality of user devices associated with the user profile; converting the trigger thresholds to corresponding trigger thresholds for different types of sensors; identifying the gesture using data from the different types of sensors that satisfy the corresponding trigger thresholds; implementing, by one of the plurality of user devices, the processor-executable action associated with the gesture; and Including, the system.

11. The system of claim 10 , wherein the different types of sensors are associated with the first user profile.

12. The system of claim 10 , wherein the different types of sensors are associated with different user profiles stored in the ubiquitous gesture agent.

13. 11. The system of claim 10, wherein the different types of sensors are cameras in a smart building, the sensor outputs include acceleration data from an accelerometer, and the corresponding trigger thresholds are based on acceleration inferred from video data from the cameras in the smart building.

14. The system of claim 10 , wherein the processor-executable action is selected from the group consisting of: activating an actuator, accessing a remote server, and executing an action on a smartphone.

15. one or more computer readable storage media; and program instructions collectively stored on the one or more computer readable storage media, the program instructions causing the one or more processors to: recording a gesture in a first user profile associated with the ubiquitous gesture agent using sensor output from the first user device; indicating the gesture and associating the gesture with a trigger threshold based on the sensor output and a processor executable action executable by one of a plurality of user devices associated with the user profile; converting the trigger thresholds to corresponding trigger thresholds for different types of sensors; identifying the gesture using data from the different types of sensors that satisfy the corresponding trigger thresholds; implementing, by one of the plurality of user devices, the processor-executable action associated with the gesture; 23. A computer program product comprising instructions configured to cause a computer to carry out a method comprising:

16. The computer program product of claim 15 , wherein the different types of sensors are associated with the first user profile.

17. The computer program product of claim 15 , wherein the different types of sensors are associated with different user profiles stored in the ubiquitous gesture agent.

18. 16. The computer program product of claim 15, wherein the different types of sensors are cameras in a smart building, the sensor outputs include acceleration data from an accelerometer, and the corresponding trigger thresholds are based on acceleration inferred from video data from the cameras in the smart building.

19. 16. The computer program product of claim 15, wherein the processor-executable action is selected from the group consisting of: initiating an actuator, accessing a remote server, and executing an action on a smartphone.

20. The program instructions are downloaded into the one or more computer readable storage media from a remote data processing system, the method comprising: Metering usage of the program instructions; generating a bill based on said metering of usage; 16. The computer program product of claim 15, further comprising:

Images