COMPUTER-IMPLEMENTED METHOD, SYSTEM, AND COMPUTER PROGRAM (VISION-BASED NOTIFICATION ESCALATION)

Abstract

To efficiently determine an effective manner to notify a user in situations where time is of essence.SOLUTION: A method can include obtaining notification data for a user, and obtaining a first set of images of the user from one or more cameras. The method can include identifying a first status of the user, and generating a set of notification options on the basis of the first status of the user. The method can include selecting a first notification option, and initiating a first notification by the first notification option. The method can include determining that the user does not acknowledge the first notification, and obtaining a second set of images of the user from the one or more cameras. The method can include identifying a second status of the user, and selecting a second notification option of the set of notification options. The method can include initiating a second notification by the second notification option.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to notifications, and more particularly, to dynamic notifications. [Background technology]

[0002] Notifications, such as messages and alerts directed to users, may be received through multiple devices and in a variety of manners. For example, an employee in an office building may receive an audible alert through the office building's public address system, a visual message notification through the employee's computer screen, or a tactile alert through the employee's mobile phone, or a combination thereof. In some instances, such employees may select the manner in which they receive notifications. Summary of the Invention [Problem to be solved by the invention]

[0003] In situations where time is of the essence, efficiently determining an effective manner for notifying a user can be essential. [Means for solving the problem]

[0004] According to an embodiment of the present disclosure, a method may include obtaining notification data for a user. The method may include obtaining a first set of images of the user from one or more cameras. The method may include identifying a first status of the user based on the first set of images. The method may include generating a set of notification options based at least in part on the first status of the user. The method may include selecting a first notification option of the set of notification options. The method may include initiating a first notification with the first notification option in response to the selection. The method may include determining that the user does not approve the first notification at a first time. The method may include obtaining a second set of images of the user from the one or more cameras in response to the determination. The method may include identifying a second status of the user based on the second set of images. The method may include selecting at least one second notification option of the set of notification options. The method may include initiating at least one second notification with the at least one second notification option in response to the selection of the at least one second notification option.

[0005] Systems and computer program products corresponding to the above methods are also included herein.

[0006] The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure. [Brief description of the drawings]

[0007] The drawings contained in this application are incorporated in and form a part of the specification. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings are only illustrative of particular embodiments and are not intended to limit the disclosure.

[0008] [Figure 1] 1 illustrates an exemplary computing environment having a notification escalation system according to an embodiment of the present disclosure.

[0009] [Diagram 2] 1 illustrates a flowchart of an example method for performing notification escalation according to an embodiment of the present disclosure.

[0010] [Diagram 3] 1 illustrates an example environment in which a notification escalation system may be used, according to an embodiment of the present disclosure.

[0011] [Figure 4] 1 illustrates exemplary major components of a computer system that may be used in accordance with embodiments of the present disclosure.

[0012] [Diagram 5] 1 illustrates a cloud computing environment according to an embodiment of the present disclosure.

[0013] [Figure 6] 1 illustrates an abstraction model layer according to an embodiment of the present disclosure.

[0014] While the invention is susceptible to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Aspects of the present disclosure relate to notifications, and more particularly, aspects relate to visibility-based notification escalation. Although the present disclosure is not necessarily limited to such applications, various aspects of the present disclosure may be understood through the discussion of various examples using this context.

[0016] Notifications, such as messages and alerts directed to users, may be received through multiple devices and in various manners. For example, an employee in an office building may receive an audible alert through the office building's public address system, a visual message notification through the employee's computer screen, or a tactile alert through the employee's mobile phone, or a combination thereof. In some cases, a user may select the manner in which the notification is received. For example, a user in a quiet environment may select to receive visual notifications only on the user's laptop computer screen so as not to disturb others. In another example, a user in a noisy or active environment may select to receive a tactile alert (e.g., a physical vibration) from the user's mobile phone to facilitate drawing the user's attention away from the environment and to the mobile phone.

[0017] Because a user may frequently change his or her location and environment, the effective manner for the user to receive notifications may change as well. In some cases, either the user or the entity attempting to notify the user may not be aware of the most effective manner for notifying the user at a given time. For example, a user who accidentally leaves their cell phone on silent mode may not be aware that a text message displayed on the user's laptop computer screen may be a more effective notification than a text message to the user's cell phone. In another example, a coworker attempting to contact an employee down the hall may not be aware that an audible message through a public address system may be a more effective notification than calling the employee. In situations where time is of the essence, such as when emergency medical attention is required or someone is at risk of injury, it may be essential to efficiently determine an effective manner for notifying the user.

[0018] To address these and other difficulties, embodiments of the present disclosure include a notification escalation system. In some embodiments, the notification escalation system can use visibility to identify one or more notification options for notifying the user. More specifically, the notification escalation system can use a set of cameras to identify a status of the user. Based on such status, the notification escalation system can generate a set of notification options for notifying the user. Furthermore, the notification escalation system can periodically initiate notifications in response to a set of notification options that describe changes in the viewed user's status. In some embodiments, the notification escalation system can continue such periodic initiation of notifications until it obtains acknowledgement data or until an acknowledgement threshold time is exceeded.

[0019] Therefore, the embodiments of the present disclosure can automatically generate notifications that are adapted to the user's status. Furthermore, by using visual recognition to identify the user's status, the embodiments of the present disclosure can provide more options for notifying the user. Thus, the embodiments of the present disclosure can improve the ability to efficiently determine an effective manner for notifying the user.

[0020] 1 illustrates a computing environment 100 including one or more of each of a notification escalation system 105, a user device 125, a notification device 130, a camera 135, a notification source 145, a server 140, or a network 150, or a combination thereof. In some embodiments, at least one of the notification escalation system 105, the user device 125, the notification device 130, the camera 135, the notification source 145, or the server 140, or a combination thereof, can exchange data with at least one other through at least one network 150. For example, in some embodiments, at least one of the notification escalation system 105 can exchange data with at least one of the user device 125 through at least one network 150. One or more of each of the notification escalation system 105, the user device 125, the notification device 130, the camera 135, the notification source 145, the server 140, or the network 150, or a combination thereof, can include a computer system, such as the computer system 401 described with respect to FIG. 4.

[0021] In some embodiments, the notification escalation system 105 may be included in software installed on a computer system of at least one of the user device 125, the notification device 130, the camera 135, the notification source 145, or the server 140, or a combination thereof. In one example, in some embodiments, the notification escalation system 105 may be included as a plug-in software component of software installed on the user device 125. The notification escalation system 105 may include program instructions for performing one or more of the operations described with respect to Figures 2 and 3 that are implemented by a processor, such as the processor of the server 140.

[0022] In some embodiments, notification escalation system 105 may include one or more modules, such as image analysis manager 110, selection manager 115, or data management software 120, or a combination thereof. In some embodiments, image analysis manager 110, selection manager 115, or data management software 120, or a combination thereof, may be integrated into a single module. In some embodiments, image analysis manager 110 may be configured to perform image analysis on images acquired by notification escalation system 105. In some embodiments, image analysis manager 110 may include a trained model (not shown) configured to identify or classify image features, or both. In some embodiments, one or more of image analysis manager 110, selection manager 115, or data management software 120, or a combination thereof, may include program instructions implemented by a processor, such as the processor of server 140, for performing one or more operations described with respect to FIGS. 2 and 3. For example, in some embodiments, image analysis manager 110 may include program instructions for performing operations 215, 220, 240, or 245, or a combination thereof, of FIG. 2. In some embodiments, selection manager 115 may include program instructions for performing operations 225-235 of Figure 2. In some embodiments, data management software 120 may include program instructions for performing operations 205, 210, 245, or 250 of Figure 2, or a combination thereof.

[0023] In some embodiments, the notification escalation system 105 may include and / or use tools for facial recognition analysis, speech analysis, or natural language processing, or a combination thereof. For example, in some embodiments, the data management software 120 may be configured to use natural language processing techniques to identify notification elements (e.g., time, name, or location) included in text notification data. In some embodiments, the data management software may be further configured to use speech analysis techniques (e.g., speech-to-text techniques) to identify notification elements included in audio notification data, such as a voice message.

[0024] In some embodiments, user device 125 may include a user's electronic device, such as a computer-based watch (e.g., a smart watch), a mobile phone, a pager, a laptop computer, a desktop computer, etc. As such, user device 125 may include a user interface component (not shown), such as a screen, a touch screen, a microphone, or a keyboard, or a combination thereof. In some embodiments, user device 125 may be configured to provide tactile, audible, or visual notifications, or a combination thereof.

[0025] In some embodiments, notification device 130 may include an electronic device of a bystander or entity other than the user. For example, in some embodiments, notification device 130 may include an office public address system or a speaker system at a pedestrian crossing. In some embodiments, notification device 130 may include a bystander's electronic device, such as a computer-based watch (e.g., a smart watch), a mobile phone, a pager, a laptop computer, a desktop computer, and the like. As such, in these embodiments, notification device 130 may include a user interface component (not shown), such as a screen, a touch screen, a microphone, or a keyboard, or a combination thereof. In some embodiments, notification device 130 may be configured to provide a tactile, audible, or visual notification, or a combination thereof.

[0026] In some embodiments, the one or more cameras 135 may include stationary cameras mounted in public or office spaces, or both, to capture images and monitor activity within the environment. For example, the one or more cameras 135 may include surveillance cameras. By using such cameras, embodiments of the present disclosure can analyze the user's environment and identify the user's location relative to other people, objects, or devices, or combinations thereof, in such environment. As such, embodiments of the present disclosure can obtain a large amount of data for generating notification options and timing schemes (described in more detail below). In this manner, embodiments of the present disclosure can efficiently identify effective notification options for the user.

[0027] In some embodiments, notification sources 145 may include systems or devices from which notification escalation system 105 obtains notification data. For example, in some embodiments, notification sources 145 may include systems such as public alert / alarm systems and office messaging systems. In some embodiments, notification sources 145 may be communication devices such as computers and cell phones. In one example, notification sources 145 may include cell phones from which coworkers attempt to send text messages to the user.

[0028] In some embodiments, server 140 may be a web server that may store data and / or software used by notification escalation system 105. In some embodiments, network 150 may be a wide area network (WAN), a local area network (LAN), the Internet, or an intranet. In some embodiments, network 150 may be substantially similar to or the same as cloud computing environment 50 described with respect to FIG.

[0029] 2 illustrates a flowchart of an example method 200 for performing notification escalation according to an embodiment of the present disclosure. The method 200 may be performed by a notification escalation system, such as the notification escalation system 105 of FIG.

[0030] In operation 205, the notification escalation system may obtain initial data from an entity such as a user or programmer of the notification escalation system. In some embodiments, the initial data may include a set of one or more notification options. The notification options may refer to a manner in which to send a notification to the user. For example, in some embodiments, the set of notification options may include a haptic alert (e.g., activating a vibration function of a user device, such as a mobile phone, smart watch, pager, etc.), a visual alert (e.g., displaying a notification on a screen of the user device or flashing one or more light-emitting diodes (LEDs) of the user device, or both), and an audible alert (e.g., activating the emission of one or more sounds from a user device, such as a mobile phone, or headphones / earbuds communicatively connected to the mobile phone, or a combination thereof).

[0031] In some embodiments, the set of notification options may include alerting bystanders. A bystander may refer to a person in the vicinity of the user (e.g., within about 6-8 feet of the user). In some embodiments, a bystander may refer to a person included in an image of the user captured by a camera. For example, a public surveillance camera may capture an image of a user standing on a street corner, and a bystander of a pedestrian approaching the street corner. Thus, in some embodiments, alerting such bystanders may include methods such as calling or sending a message to the bystander's notification device (e.g., a cell phone, a smart watch, a pager, etc.) so that the bystander can prompt the user to acknowledge the notification.

[0032] In some embodiments, the set of notification options may include issuing an audible message through a speaker distinct from and not operably connected to the user device, such as a loudspeaker in a public address system in the environment in which the user is located. For example, in some embodiments, a pedestrian signal at a street crossing may include a speaker. In these embodiments, the notification escalation system may use such a speaker to issue an audible message for the user (e.g., a message that the pedestrian user is approaching one or more moving roadways and is in danger).

[0033] In some embodiments, the initial data obtained in operation 205 may include timing data. In some embodiments, the timing data may include an acceptance time threshold. The acceptance time threshold may refer to a predefined threshold time for a user to accept a notification. For example, in some embodiments, the timing data may include a default 5 minute period for a physician to respond to a page or text message. In some embodiments, the timing data may include a minimum process time. The process time may refer to a time interval between successive notifications initiated by the notification escalation system. For example, in some embodiments, the notification escalation system may be configured to wait at least 1 minute between the initiation of a first notification and the initiation of a second notification. In some embodiments, such timing data may be determined by an entity such as a user or programmer of the notification escalation system. In some embodiments, the notification escalation system may select such timing data from a stored data set. For example, in some embodiments, the stored data set may include a first acceptance time threshold of 2 minutes for emergency notifications and a second acceptance time threshold of 5 minutes for non-emergency notifications. In this example, the notification escalation system can select an acknowledgement time threshold from a stored data set based on the level of urgency associated with the notification (ie, emergency or non-emergency).

[0034] In some embodiments, the initial data acquired in operation 205 may include training data for training the notification escalation system to perform image analysis. For example, in some embodiments, operation 205 may include training a machine learning algorithm of the notification escalation system with data such as images of faces or images of scenarios (e.g., images of two people talking, images of people looking at a screen of a device such as a smart watch, images of people carrying a mobile phone and / or pager, etc.) or both. Such training may include performing supervised training, unsupervised training, or semi-supervised training. Furthermore, such training may enable such machine learning algorithm to generate a trained model configured to identify characteristics of one or more people and / or images indicative of a user status (described further below).

[0035] According to embodiments of the present disclosure, machine learning algorithms may include, but are not limited to, decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity / distance training, sparse dictionary learning, genetic algorithms, rule-based learning, or other machine learning techniques, or combinations thereof.

[0036] For example, machine learning algorithms may include the following exemplary techniques: k-nearest neighbors (KNN), learning vector quantization (LVQ), self-organizing maps (SOM), logistic regression, ordinary least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression splines (MARS), ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS), probabilistic classifiers, naive Bayes classifiers, binary classifiers, linear classifiers, hierarchical classifiers, Canonical Correlation Analysis (CCA), Factor Analysis, Independent Component Analysis (ICA), Linear Discriminant Analysis (LDA), Multidimensional Scaling (MDS), Nonnegative Matrix Factorization (NMF), Partial Least Squares Regression (PLSR), Principal Component Analysis (PCA), Principal Component Regression (PCR), Sammon Mapping, t-SNE, Bootstrap Aggregation, Harmonic Mean, Gradient Boosting Decision Tree (GBRT), Gradient Boosting Machine (GBM), Inductive Bias Algorithm, Q learning, state-action-reward-state-action (SARSA), temporal difference (TD) learning, a priori algorithm, equivalent class transformation (ECLAT) algorithm, Gaussian process regression, gene expression programming, group method of data processing (GMDH), inductive logic programming, learning from examples, logic model trees, information fuzzy networks (IFN), hidden Markov models, Gaussian naive Bayes, multinomial naive Bayes, average-one dependence estimator (AODE), Bayesian networks (BN), classification and regression trees (CART), chi-squared automated layer false positives (CHAID), expectation maximization algorithm, forward propagation neural networks, logic learning machines, self-organizing maps, single-link clustering, fuzzy clustering, hierarchical clustering, Boltzmann machines, convolutional neural networks, recurrent neural networks, hierarchical temporary memory (HTM), or other machine learning techniques, or combinations thereof, may be utilized.

[0037] In operation 210, the notification escalation system may obtain notification data from a notification source, such as notification source 145 of FIG. 1. In some embodiments, the notification data may include information to be transmitted to a user. For example, in some embodiments, the notification data may include a text or voice message generated by an entity, such as a user's administrative assistant, such as a doctor. In some embodiments, the notification data may include a text or voice message generated by an entity, such as an automated warning system configured to warn a user, such as a pedestrian near an intersection or construction zone. In some embodiments, the notification data may include one or more elements, such as a name, a time, a duration, a degree of urgency, or a destination, or a combination thereof. In one example, the notification data may include a text message that reads, "Please show up at room 804B for your appointment at 10:30 AM." In this example, the notification data includes the user's destination (i.e., room 804B) and the time of the user's action.

[0038] At operation 215, in response to obtaining the notification data at operation 210, the notification escalation system may obtain image data. The image data may be obtained from at least one camera, such as camera 135 of FIG. 1. The image data may include a set of one or more images of an environment in which the user is located. In some embodiments, the image data may include one or more images of at least one of the user, bystanders, user devices, or notification devices, or combinations thereof. In some embodiments, the image data may include metadata, such as the time or location at which the image was captured, or a combination thereof.

[0039] In operation 220, the notification escalation system may identify a user status based on the image data acquired in operation 215. In some embodiments, operation 220 may include the notification escalation system using image analysis techniques (e.g., a trained machine learning model) to identify the user status from the image data. The user status may include information regarding the location of the user, the activity / action the user is engaged in, one or more devices the user has, or the user's proximity to one or more bystanders, or a combination thereof. For example, in some embodiments, the notification escalation system may identify that the user is waiting for an elevator on the third floor of a building based on imagery from a camera at that location that shows the user lingering in front of the elevator door for a period of time. In another example, in some embodiments, the notification escalation system may identify that the user is looking up information on a mobile phone based on camera imagery of the user looking at a lighted screen of such mobile phone. In another example, in some embodiments, the notification escalation system may identify that the user is in proximity to a bystander working on a computer based on the bystander and computer being included in the camera imagery along with the user due to such bystander and computer being located within the field of view of the camera. Further in this example, the notification escalation system can identify bystanders as colleagues of the user by performing facial recognition on such images.

[0040] In some embodiments, operation 220 may include the notification escalation system determining a status of the user at successive times. For example, in some embodiments, the notification escalation system may determine that at a first time, the user is walking down a hallway in a first direction. Further in this example, at a later second time, the notification escalation system may determine that the user is running down the hallway in a second opposite direction. As described below, in some embodiments, such a change in status may indicate that the user is acknowledging the notification.

[0041] At operation 225, the notification escalation system may generate a set of one or more notification options based at least in part on the user status obtained at operation 220. In this disclosure, the number of notification options may refer to the number of notification options in the set of notification options. For example, in some embodiments, at operation 220, the notification escalation system may identify that a user is typing on a laptop computer with headphones on near a bystander with a mobile phone. In this example, the notification escalation system may generate a set of notification options including at least three notification options, such as: (1) sending a visual alert to the user's laptop computer, (2) sending an audible alert that can be received by the user's headphones (e.g., sending an audible alert to the user's mobile device that may be communicatively connected to the user's headphones), and (3) alerting the bystander by sending a message to the bystander's mobile phone. In some embodiments, the notification escalation system may be configured to prioritize the generated set of notification options. Continuing with the above example, the notification escalation system can prioritize notification option (1) and / or notification option (2) over notification option (3) so as not to disturb bystanders unless the user approves of notifications via notification option (1) and / or (2).

[0042] In some embodiments, operation 225 may include the notification escalation system selecting an acknowledgement time threshold for one or more notification options based at least in part on the notification data obtained in operation 210. For example, continuing with the above example, the notification data obtained by the notification escalation system may include a message that reads, "Board meeting in 10 minutes. Please call the front desk immediately." Continuing with the example, the notification escalation system may use natural language processing techniques to identify elements such as a 10 minute time period and a request to call immediately. Based on these elements of the notification data that indicate a degree of urgency, the notification escalation system may select a threshold such as a 5 minute acknowledgement time threshold. Thus, such a selection may cause the notification escalation system to use a set of three notification options and specify a maximum time of 5 minutes for obtaining acknowledgement data from a user. Acknowledgement data (described further below) may refer to information that indicates that a user has received a notification, or is performing an activity corresponding to the notification, or both. In some embodiments, the acknowledgement time threshold may be selected by an entity such as a programmer or user of the notification escalation system.

[0043] At operation 230, the notification escalation system may select at least one notification option. In some embodiments, such selection may be based at least in part on a user status, a priority associated with the notification option, a previously selected notification option, or an approval time threshold, or a combination thereof. In some embodiments, operation 230 may include the notification escalation system selecting a notification option from the set of notification options generated at operation 225. For example, in some embodiments, at operation 225, the notification escalation system may generate a set of three notification options, with a first notification option having the highest priority and a third notification option having the lowest priority. Further, in this example, the set of three notification options may have a corresponding approval time threshold of 10 minutes (e.g., the notification escalation system may use the set of three notification options and specify a maximum time of 10 minutes for obtaining approval data from the user). In this example, the notification escalation system may select the first notification option at a first time based on its highest priority. Continuing with the example, the notification escalation system may select the second notification option at a second time in response to determining that the user has not responded to the notification with the first notification option. Continuing with the example, the notification escalation system may select all three notification options in the set of notification options at a third time in response to determining that the remaining time of the 10 minute approval time threshold is less than the minimum remaining time (e.g., 3 minutes remain of the 10 minute approval time threshold, which is less than the 5 minute minimum remaining time threshold). Thus, embodiments of the present disclosure may increase the likelihood that a user will approve a notification.

[0044] In operation 235, the notification escalation system may initiate at least one notification. In some embodiments, the notification escalation system may perform operation 235 in response to selecting at least one notification option in operation 230. Initiating a notification may include the notification escalation system issuing a command that enables at least one user device (e.g., user device 125 of FIG. 1) or at least one notification device (e.g., notification device 130 of FIG. 1), or a combination thereof, to perform an action according to the notification option. For example, in some embodiments, operation 235 may include the notification escalation system issuing a command that enables a user's smartwatch to vibrate according to the notification option. In this example, such a vibration may prompt a user who is not responding to a phone call to view a message on the user's smartwatch. In another example, in some embodiments, operation 235 may include the notification escalation system issuing a command that enables a laptop computer to prominently display a text message on its screen according to the notification option. In this example, such a visual display may prompt a user who is not responding to a text message to receive the text message on the user's laptop computer. In another example, in some embodiments, operation 235 may include the notification escalation system issuing a command that enables a public address system in the user's office building to issue an audible message to the user according to the notification options. In another example, in some embodiments, operation 235 may include the notification escalation system simultaneously issuing the commands described in the examples above according to the set of notification options.

[0045] At operation 240, in response to initiating the notification at operation 235, the notification escalation system may acquire image data. Such image data may be acquired during operation in a manner substantially similar to that described at 215. In some embodiments, operation 240 may include the notification escalation system acquiring one or more sets of images over multiple time periods. For example, operation 215 may include the notification escalation system acquiring a first set of images of an environment in which the user is located between 9:00 a.m. and 9:05 a.m., and acquiring a second set of images of an environment in which the user is located between 9:15 a.m. and 9:20 a.m. In this manner, the notification escalation system may determine whether the user acknowledges the first notification or a subsequent notification based on the user's actions captured in the images.

[0046] In operation 245, the notification escalation system may determine whether the user acknowledges the notification. In some embodiments, the notification escalation system may determine that the user acknowledges the notification if the notification escalation system obtains the acknowledgement data. In these embodiments, the notification escalation system may determine that the user does not acknowledge the notification if the notification escalation system does not obtain the acknowledgement data. The acknowledgement data may refer to information indicating that the user has received the notification, is performing an activity corresponding to the notification, or has completed an activity corresponding to the notification. For example, in some embodiments, the acknowledgement data may include a confirmation message from the user device that confirms that the user has read the notification message. In some embodiments, the user may choose to send such a confirmation message to the notification escalation system through the user device.

[0047] In some embodiments, the acknowledgement data may include a determination by the notification escalation system that the user has performed / completed an activity corresponding to the notification. In these embodiments, such a determination may be based on analysis by the notification escalation system of the image data acquired in operation 240. For example, in some embodiments, such image data may indicate that the user has stopped walking, picked up the user's cell phone, and is looking at the cell phone's illuminated screen. In this example, the image data may indicate to the notification escalation system that the user is reading a text message notification that he or she received. Thus, in this example, the notification escalation system may determine that the user is performing an activity corresponding to the notification. In another example, in some embodiments, the image data may indicate that the user is approached by a bystander and is talking to the bystander. In this example, the image data may indicate to the notification escalation system that the bystander is communicating a notification message to the user and / or prompting the user to receive the notification. Thus, in this example, the notification escalation system may determine that the user is performing an activity corresponding to the notification. In some embodiments, the image data may indicate that the user is at a destination specified in the notification message. In these embodiments, the notification escalation system may determine that the user has completed the activity corresponding to the notification message.

[0048] In some embodiments, the approval data may be selected from the group consisting of: (1) a confirmation message from the user device, and (2) image data that indicates to the notification escalation system that the user has completed an activity corresponding to the notification. Thus, in these embodiments, the notification escalation system may determine that it has not received approval data unless it receives at least one of these two types of data. For example, the notification escalation system may initiate a notification requesting that the user go to room A. In this example, the notification escalation system may consider the confirmation message received by the user as approval data indicating that the user approves the notification. Alternatively, in this example, the notification escalation system may obtain a subsequent image of the user standing in room A and may consider the image as approval data indicating that the user approves the notification. Alternatively, in this example, the notification escalation system may obtain a subsequent image of the user moving forward toward room A or not responding to the notification and may consider such image as not approval data. In these cases, the notification escalation system may determine that the user does not approve the notification.

[0049] In some embodiments, the notification escalation system may consider only image data indicating that the user has completed the activity corresponding to the notification as acknowledgment data. For example, if the notification escalation system initiates a notification requesting that the user go to room A, the user's confirmation message to the notification escalation system is not acknowledgment data. However, capturing a subsequent image of the user standing in room A is acknowledgment data to the notification escalation system. Thus, the notification escalation system may be configured to continue method 200 until it obtains an indication that the user has completed the activity corresponding to the notification.

[0050] If, at operation 245, the notification escalation system determines that the user does not acknowledge the notification, the notification escalation system may proceed to operation 250. Alternatively, if the notification escalation system determines that the user does acknowledge the notification, method 200 may terminate (e.g., the notification escalation system may avoid initiating any subsequent notifications and may terminate method 200). Additionally, if the notification escalation system determines that the acknowledgement time threshold has been exceeded, method 200 may terminate.

[0051] At operation 250, the notification escalation system may generate and use a timing scheme for initiating one or more subsequent notifications. A timing scheme may refer to a process for implementing a process time (e.g., a time interval between successive notifications initiated by the notification escalation system). In some embodiments, operation 250 may include the notification escalation system calculating such a process time. In some embodiments, such a calculation may include the notification escalation system dividing an acceptance time threshold by a number of notification options included in a set of notification options.

[0052] In some embodiments, the timing scheme generated at operation 250 may include one or more of the following: If the image data acquired at operation 240 indicates that the user is not performing an activity corresponding to the notification initiated at operation 235, the notification escalation system may proceed to operation 215 in response to allowing one step time to pass. If the image data acquired at operation 240 indicates that the user is performing an activity corresponding to the notification initiated at operation 235 (e.g., the user is moving forward toward a destination specified in the notification message), the notification escalation system may proceed to operation 215 in response to allowing two step times to pass. By using such a timing scheme, embodiments of the present disclosure may allow a user who appears to be responsive to a notification additional time to acknowledge the notification. In this manner, embodiments of the present disclosure may reduce the number of notifications that are initiated, thereby reducing the possibility of overloading the user with excessive notifications.

[0053] 3 illustrates an example environment 300 in which a notification escalation system may be used, according to an embodiment of the present disclosure. (The distances and object sizes illustrated in FIG. 3 are not to scale.) The environment 300 includes a number of moving vehicles 325 at an intersection 305. The environment 300 further includes a public surveillance camera 315 and a crosswalk speaker 320. The public surveillance camera 315 and the crosswalk speaker 320 may be operatively connected to an independent public safety system (not shown) configured to alert pedestrians in the environment 300 of potential traffic hazards. The crosswalk speaker 320 may be configured to emit an audible message, such as a message regarding when it is safe for pedestrians to cross the street.

[0054] The environment 300 may include a user 330 and bystanders 345 on a sidewalk 310. In this example, the independent public safety system may predict, by using methods outside the scope of this disclosure, that the user is moving forward toward an intersection 305 that is in danger and should issue a warning message saying "Approaching an intersection, watch oncoming traffic." To manage the issuance of such a message, the independent public safety system may use a notification escalation system (not shown) according to an embodiment of the present disclosure.

[0055] The notification escalation system may obtain an alert message and image data from the camera 315. Based on the image data obtained at a first time, the notification escalation system may determine that the user 330 has a mobile phone 335 and is wearing headphones 340. In response, the notification escalation system may initiate a text notification to the user's mobile phone 335, including the alert message. Additionally, the notification escalation system may initiate a haptic alert to the user's mobile phone 335 to prompt the user 330 to view the text notification. Based on the image data obtained at a second time, the notification escalation system may determine that the user 330 did not acknowledge the text notification. Based on the image data obtained at a third time, the notification escalation system may determine that the user 330 has moved to a location 355. Additionally, based on such image data, the notification escalation system may determine that the user 330 is near a bystander 345 having a mobile phone 350 and a crosswalk speaker 320. In response, the notification escalation system may initiate the notifications described above, as well as (1) an audible notification emitted from the crosswalk speaker 320 containing a warning message, and (2) a text notification to the bystander's mobile phone 350 requesting the bystander 345 to alert the user 330. Thus, embodiments of the present disclosure may enable an increased set of notification options for notifying users.

[0056] 4 illustrates representative major components of a computer system 401 that may be used in accordance with embodiments of the present disclosure. The specific components illustrated are presented for purposes of example only and are not necessarily the only such variations. Computer system 401 may include a processor 410, a memory 420, an input / output interface (also referred to herein as I / O or I / O interface) 430, and a main bus 440. Main bus 440 may provide a communication path for other components of computer system 401. In some embodiments, main bus 440 may connect to other components, such as a dedicated digital signal processor (not shown).

[0057] The processor 410 of the computer system 401 may be comprised of one or more CPUs 412. The processor 410 may further be comprised of one or more memory buffers or caches (not shown) that provide temporary storage of instructions and data for the CPU 412. The CPU 412 may execute instructions on input provided from the cache or from the memory 420 and output results to the cache or memory 420. The CPU 412 may be comprised of one or more circuits configured to execute one or more methods consistent with embodiments of the present disclosure. In some embodiments, the computer system 401 may include multiple processors 410, which is typical for relatively large systems. However, in other embodiments, the computer system 401 may be a single processor with a single CPU 412.

[0058] The memory 420 of the computer system 401 may be comprised of a memory controller 422 and one or more memory modules (not shown) for temporarily or permanently storing data. In some embodiments, the memory 420 may include a random access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing data and programs. The memory controller 422 may communicate with the processor 410 to facilitate the storage and retrieval of information in the memory module. The memory controller 422 may communicate with the I / O interface 430 to facilitate the storage and retrieval of inputs or outputs in the memory module. In some embodiments, the memory module may be a dual in-line memory module.

[0059] I / O interface 430 may include an I / O bus 450, a terminal interface 452, a storage interface 454, an I / O device interface 456, and a network interface 458. I / O interface 430 may connect main bus 440 to I / O bus 450. I / O interface 430 may direct instructions and data from processor 410 and memory 420 to various interfaces of I / O bus 450. I / O interface 430 may also direct instructions and data from various interfaces of I / O bus 450 to processor 410 and memory 420. The various interfaces may include terminal interface 452, storage interface 454, I / O device interface 456, and network interface 458. In some embodiments, the various interfaces may include a subset of the aforementioned interfaces (e.g., an embedded computer system in an industrial application may not include terminal interface 452 and storage interface 454).

[0060] Logic modules throughout computer system 401, including but not limited to memory 420, processor 410, and I / O interface 430, may communicate failures and changes in one or more components to a hypervisor or operating system (not shown). The hypervisor or operating system may allocate the various resources available in computer system 401 and track the location of data for processes assigned to the various CPUs 412 and in memory 420. In embodiments that combine or configure elements, aspects of the capabilities of the logic modules may be combined or redistributed. These variations will be apparent to those skilled in the art.

[0061] Although this disclosure includes detailed descriptions of cloud computing, it should be understood in advance that implementations of sandwiching as referred to herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0062] Cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort and interaction with the provider of the service. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0063] The characteristics are as follows:

[0064] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the provider of the service.

[0065] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client and thick-client platforms (eg, cell phones, laptops, and PDAs).

[0066] Resource Sharing: A provider's computing resources are shared using a multi-tenant model to serve multiple consumers, with physical and virtual resources being dynamically allocated and reallocated on demand. There is a sense of location independence in that consumers generally have no control or knowledge over the exact location of the resources provided, although location may be specified at a higher level of abstraction (e.g., country, state, or data center).

[0067] Rapid Elasticity: Processing power can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the processing power available for provisioning often appears to be unlimited, and that any amount can be purchased at any time.

[0068] Metering of Services: Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and directed, providing transparency to both providers and consumers of the services utilized.

[0069] The service model is as follows:

[0070] Software as a Service (SaaS): The ability offered to the consumer is to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, except for the possibility of limited user-specified application configuration settings.

[0071] Platform as a Service (PaaS): The capability offered to a consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the environment that hosts the applications.

[0072] Infrastructure as a Service (IaaS): The ability provided to a consumer is to provision processing, storage, network, and other basic computing resources, and the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases has limited control over selected network components (e.g., host firewalls).

[0073] The deployment models are as follows:

[0074] Private Cloud: Cloud infrastructure is operated solely for an organization. It may be managed by that organization or a third party and may be on-site or off-site.

[0075] Community Cloud: Cloud infrastructure is shared by several organizations to support a particular community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organizations or a third party and may exist on-site or off-site.

[0076] Public Cloud: The cloud infrastructure is available to the general public or large industry entities and is owned by an organization that sells cloud services.

[0077] Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain as their own entity, but are tied together by standardized or proprietary technologies that allow data and application portability (e.g., cloud bursting for load balancing between clouds).

[0078] Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0079] Referring now to FIG. 5, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped (not shown) into one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as described herein above. This allows the cloud computing environment 50 to provide infrastructure, platform, or software as a service, or combinations thereof, without the cloud consumer having to maintain resources on the local computing device. It should be understood that the types of computing devices 54A-N shown in FIG. 5 are intended to be exemplary only, and that the computing nodes 10 and the cloud computing environment 50 may communicate with any type of computerized device through any type of network or addressable connection of a network (e.g., using a web browser), or combinations thereof.

[0080] Referring now to Figure 6, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 5) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 6 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0081] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0082] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71 , virtual storage 72 , virtual networks including virtual private networks 73 , virtual applications and operating systems 74 , and virtual clients 75 .

[0083] In one example, management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic acquisition of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost management as resources are utilized in the cloud computing environment and charging or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity proofing for cloud consumers and tasks, and protection of data and other resources. User portal 83 provides consumer and system administrator access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and execution 85 provides advance preparation and acquisition of cloud computing resources anticipated to be needed in the future according to SLAs.

[0084] The workload tier 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and notification escalation logic 96.

[0085] As described in more detail herein, it is contemplated that some or all of the operations of some of the methodology embodiments described herein may be performed in an alternative order, or not performed at all. Additionally, multiple operations may occur simultaneously or within a larger process.

[0086] The present invention may be a system, method, or computer program product, or combination thereof, at any conceivable level of integration of technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0087] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, punch cards or mechanically encoded devices such as ridge structures in grooves with recorded instructions, and any suitable combination of the foregoing. A computer-readable storage medium, as used herein, is not to be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a wave guide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0088] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0089] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instructions of an instruction set architecture (ISA), machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed completely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to customize the electronic circuitry to perform aspects of the present invention.

[0090] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks of the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0091] These computer readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that the instructions, executed via the computer processor, create means for implementing the functions / operations specified in the flowchart and / or block diagram blocks. These computer readable program instructions may also be stored on a computer readable storage medium that can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium having stored thereon includes an article of manufacture including instructions that implement aspects of the functions / operations specified in the flowchart and / or block diagram blocks.

[0092] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in the blocks of the flowcharts and / or block diagrams.

[0093] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions mentioned in the blocks may occur out of the order mentioned in the drawings. For example, two blocks shown in succession may actually be accomplished as one step performed simultaneously, substantially simultaneously, in a partially or completely temporally overlapping manner, or the blocks may be performed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, as well as combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a special-purpose hardware-based system that performs the specified functions or acts or realizes a combination of special-purpose hardware and computer instructions.

[0094] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method, comprising: obtaining notification data for a user; acquiring a first set of images of the user from one or more cameras; determining a first status of the user based on the first set of images; generating a set of notification options based at least in part on the first status of the user; selecting a first notification option from the set of notification options; initiating a first notification according to the first notification option in response to the selection; determining at a first time that the user does not approve the first notification; acquiring a second set of images of the user from the one or more cameras in response to the determination; determining a second status of the user based on the second set of images; selecting at least a second notification option from the set of notification options; initiating at least one second notification via the at least one second notification option in response to the selection of the at least one second notification option. Computer-implemented method.

2. The computer-implemented method of claim 1 , wherein the one or more cameras are surveillance cameras of an environment in which the user is located.

3. The method further includes the step of calculating a process time, the process time being a time interval between the step of initiating the first notification and the step of initiating the at least one second notification; 3. A computer-implemented method according to claim 1 or 2.

4. 4. The computer-implemented method of claim 3, wherein the notification data includes an approval time threshold, and wherein the step of calculating the process time includes dividing the approval time threshold by a number of notification options included in the set of notification options.

5. determining, at a second time subsequent to the first time, that the user acknowledges the at least one second notification; and ceasing to initiate a third notification in response to the determination at the second time. A computer-implemented method according to any one of claims 1 to 4.

6. the notification data includes a destination of the user; and determining that the user acknowledges the at least one second notification includes determining that the user is progressing toward the destination based on a third set of images of the user acquired from the one or more cameras.

6. The computer-implemented method of claim 5.

7. The computer-implemented method of claim 1 , wherein the set of notification options includes initiating a notification to a notification device of an bystander, the bystander being included in the second set of images.

8. A processor; A system comprising a memory in communication with the processor, The memory includes program instructions configured, when executed by the processor, to cause the processor to perform a method for generating a summary sentence structure, the method comprising: obtaining notification data for a user; acquiring a first set of images of the user from one or more cameras; determining a first status of the user based on the first set of images; generating a set of notification options based at least in part on the first status of the user; selecting a first notification option from the set of notification options; initiating a first notification according to the first notification option in response to the selection; determining at a first time that the user does not approve the first notification; acquiring a second set of images of the user from the one or more cameras in response to the determination; determining a second status of the user based on the second set of images; selecting at least a second notification option from the set of notification options; initiating at least one second notification via the at least one second notification option in response to the selection of the at least one second notification option. system.

9. The system of claim 8 , wherein the one or more cameras are surveillance cameras of an environment in which the user is located.

10. The method further includes the step of calculating a process time, the process time being a time interval between the step of initiating the first notification and the step of initiating the at least one second notification; 10. A system according to claim 8 or 9.

11. 11. The system of claim 10, wherein the notification data includes an approval time threshold, and wherein the step of calculating the process time includes dividing the approval time threshold by a number of notification options included in the set of notification options.

12. determining, at a second time subsequent to the first time, that the user acknowledges the at least one second notification; and ceasing to initiate a third notification in response to the determination at the second time. A system according to any one of claims 8 to 11.

13. the notification data includes a destination of the user; and determining that the user acknowledges the at least one second notification includes determining that the user is progressing toward the destination based on a third set of images of the user acquired from the one or more cameras. The system of claim 12.

14. The system of claim 8 , wherein the set of notification options includes initiating a notification to a notification device of a bystander, the bystander being included in the second set of images.

15. The processor: obtaining notification data for a user; acquiring a first set of images of the user from one or more cameras; determining a first status of the user based on the first set of images; generating a set of notification options based at least in part on the first status of the user; selecting a first notification option from the set of notification options; initiating a first notification according to the first notification option in response to the selection; determining at a first time that the user does not approve the first notification; acquiring a second set of images of the user from the one or more cameras in response to the determining; determining a second status of the user based on the second set of images; selecting at least one second notification option from the set of notification options; initiating at least one second notification via the at least one second notification option in response to the selection of the at least one second notification option; A computer program for executing the above.

16. The computer program product of claim 15 , wherein the one or more cameras are surveillance cameras of an environment in which the user is located.

17. A step of calculating a process time by the processor, 17. The computer program product of claim 15 or 16, further comprising the step of: the process time being a time interval between the step of initiating the first notification and the step of initiating the at least one second notification.

18. 20. The computer program product of claim 17, wherein the notification data includes an acceptance time threshold, and wherein calculating the process time includes dividing the acceptance time threshold by a number of notification options included in the set of notification options.

19. The processor, determining, at a second time subsequent to the first time, that the user acknowledges the at least one second notification; 19. The computer program product of claim 15, further comprising: in response to the determination at the second time, ceasing to initiate a third notification.

20. the notification data includes a destination of the user; determining that the user acknowledges the at least one second notification includes determining that the user is progressing toward the destination based on a third set of images of the user acquired from the one or more cameras.

20. A computer program product as claimed in claim 19.

Images