Notification upgrade based on visual recognition

Abstract

A method may include obtaining notification data for a user and obtaining a first set of images of the user from one or more cameras. The method may include identifying a first state of the user and generating a set of notification options based on the first state. The method may include selecting the first notification option and initiating a first notification via the first notification option. The method may include determining that the user has not acknowledged the first notification and obtaining a second set of images from the one or more cameras. The method may include identifying a second state of the user and selecting a second notification option from the set of notification options. The method may include initiating a second notification via the second notification option.

Description

Technical Field

[0001] The present disclosure relates to notifications, and more particularly, to dynamic notifications. Background Art

[0002] Notifications, such as messages and alerts for users, can be received through multiple devices and in various ways. For example, an employee in an office building can receive an audible alert via the building's public address system, a visual message notification via the employee's computer screen, and / or a tactile alert via the employee's mobile phone. In some instances, such an employee can select the manner in which they receive notifications. Summary of the Invention

[0003] According to an embodiment of the present disclosure, a method may include obtaining notification data of a user. The method may include obtaining a first set of images of the user from one or more cameras. The method may include identifying a first state of the user based on the first set of images. The method may include generating a set of notification options based at least in part on the first state of the user. The method may include selecting a first notification option from the set of notification options. The method may include initiating a first notification via the first notification option in response to the selection. The method may include determining at a first time that the user has not acknowledged the first notification. The method may include obtaining a second set of images of the user from one or more cameras in response to the determination. The method may include identifying a second state of the user based on the second set of images. The method may include selecting at least one second notification option from the set of notification options. The method may include initiating at least one second notification via the at least one second notification option in response to the selection of the at least one second notification option.

[0004] Also included herein are systems and computer program products corresponding to the above methods.

[0005] The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings included in this application are incorporated into and form a part of the specification. They illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the present disclosure. The accompanying drawings only illustrate certain embodiments and do not limit the present disclosure.

[0007] Figure 1 An example computing environment with a notification escalation system according to an embodiment of the present disclosure is depicted.

[0008] Figure 2 Depicted is a flow diagram of an example method for performing notification escalation according to an embodiment of the present disclosure.

[0009] Figure 3Depicted are example environments in which a notification escalation system may be employed according to embodiments of the present disclosure.

[0010] Figure 4 Depicted are representative major components of a computer system that may be used in accordance with embodiments of the present disclosure.

[0011] Figure 5 A cloud computing environment according to an embodiment of the present disclosure is depicted.

[0012] Figure 6 Abstract model layers according to an embodiment of the present disclosure are depicted.

[0013] Although the present invention is susceptible to various modifications and alternative forms, details thereof have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that it is not intended to limit the present invention to the specific embodiments described. On the contrary, the present invention covers all modifications, equivalents and alternatives that fall within the spirit and scope of the present invention. DETAILED DESCRIPTION

[0014] Aspects of the present disclosure relate to notifications; more particularly, to notification escalation based on visual recognition. While the present disclosure is not necessarily limited to this application, aspects of the present disclosure may be understood through discussion of various examples using this context.

[0015] Notifications such as messages and alerts for users can be received across multiple devices and in various ways. For example, an employee in an office building can receive an audible alert via the building's public address system, a visual message notification via the employee's computer screen, and / or a tactile alert via the employee's mobile phone. In some instances, a user can choose how to receive notifications. For example, a user in a quiet setting can choose to receive only visual notifications on the user's laptop screen so as not to disturb others. In another example, a user in a loud or active environment can choose to receive a tactile alert (e.g., a physical vibration) from the user's mobile phone to draw the user's attention away from the environment and toward the mobile phone.

[0016] Because users may frequently change their locations and environments, the effective manner in which users receive notifications may likewise change. In some instances, neither the user nor the entity attempting to notify the user may be able to implement the most effective manner to notify the user at a given time. For example, a user who accidentally leaves his or her mobile phone in silent mode may not realize that a text message displayed on the user's laptop screen may be a more effective notification than a text message to the user's mobile phone. In another example, a coworker attempting to contact an employee in a hallway may not realize that an audible message over the public address system may be a more effective notification than a telephone call to the employee. In situations where time is of the essence, such as when emergency medical care is required or when a person is at risk of injury, it may be necessary to effectively determine an effective manner to notify the user.

[0017] To address these and other challenges, embodiments of the present disclosure include a notification escalation system. In some embodiments, the notification escalation system may employ visual recognition to identify one or more notification options for notifying a user. More specifically, the notification escalation system may employ a set of cameras to identify the user's state. Based on this state, the notification escalation system may generate a set of notification options for notifying the user. Additionally, the notification escalation system may cyclically initiate notifications based on the set of notification options, taking into account changes in the user's visually recognized state. In some embodiments, the notification escalation system may continue this cyclic initiation of notifications until it obtains confirmation data or until a confirmation threshold time is exceeded.

[0018] Thus, embodiments of the present disclosure can automatically generate notifications appropriate to the user's status. Additionally, by employing visual recognition to identify the user's status, embodiments of the present disclosure can provide increased options for notifying the user. Thus, embodiments of the present disclosure can improve the ability to efficiently determine effective ways to notify the user.

[0019] Go to the attached figure, Figure 1 A computing environment 100 is shown that includes one or more of each of a notification escalation system 105, a user device 125, a notification device 130, a camera 135, a notification source 145, a server 140, and / or a network 150. In some embodiments, at least one notification escalation system 105, a user device 125, a notification device 130, a camera 135, a notification source 145, and / or a server 140 can exchange data with at least one other notification escalation system via at least one network 150. For example, in some implementations, at least one notification escalation system 105 can exchange data with at least one user device 125 via at least one network 150. One or more of each of the notification escalation system 105, a user device 125, a notification device 130, a camera 135, a notification source 145, a server 140, and / or a network 150 can include a computer system, such as a computer system, for example, a computer system associated with the notification escalation system. Figure 4 The computer system 401 in question.

[0020] In some embodiments, the notification escalation system 105 may be included in software installed on a computer system of at least one of the user device 125, the notification device 130, the camera 135, the notification source 145, and / or the server 140. In one example, in some implementations, the notification escalation system 105 may be included as a plug-in software component of the software installed on the user device 125. The notification escalation system 105 may include program instructions implemented by a processor, such as a processor of the server 140, to perform operations related to the notification escalation system 105. Figure 2 and 3 The operation or operations in question.

[0021] In some embodiments, the notification escalation system 105 may include one or more modules, such as an image analysis manager 110, a selection manager 115, and / or a data manager 120. In some embodiments, the image analysis manager 110, the selection manager 115, and / or the data manager 120 may be integrated into a single module. In some embodiments, the image analysis manager 110 may be configured to perform image analysis on images obtained by the notification escalation system 105. In some embodiments, the image analysis manager 110 may include a trained model (not shown) configured to recognize and / or classify image features. In some embodiments, one or more of the image analysis manager 110, the selection manager 115, and / or the data manager 120 may include program instructions implemented by a processor (e.g., a processor of the server 140) to perform processing related to the image analysis. Figure 2 and 3 One or more of the operations discussed, for example, in some embodiments, the image analysis manager 110 may include a program for performing Figure 2 In some embodiments, the selection manager 115 may include program instructions for performing operations 215, 220, 240 and / or 245. Figure 2 In some embodiments, the data manager 120 may include program instructions for performing operations 225-235. Figure 2 The program instructions of operations 205, 210, 245 and / or 250.

[0022] In some embodiments, the notification escalation system 105 may include and / or employ tools for facial recognition analysis, audio analysis, and / or natural language processing. For example, in some embodiments, the data manager 120 may be configured to employ natural language processing techniques to identify notification elements (e.g., time, name, or location) included in text notification data. In some embodiments, the data manager may also be configured to employ audio analysis techniques (e.g., speech-to-text techniques) to identify notification elements included in audio notification data (such as voicemail messages).

[0023] In some embodiments, user device 125 may include a user's electronic device, such as a computer-based wristwatch (e.g., a smartwatch), a mobile phone, a pager, a laptop computer, a desktop computer, etc. Thus, user device 125 may include user interface components (not shown), such as a screen, a touch screen, a microphone, and / or a keyboard. In some embodiments, user device 125 may be configured to provide tactile, auditory, and / or visual notifications.

[0024] In some embodiments, notification device 130 may include an electronic device for a bystander or entity other than the user. For example, in some embodiments, notification device 130 may include an office public address system or speaker system for a crosswalk. In some embodiments, notification device 130 may include an electronic device for a bystander, such as a computer-based watch (e.g., a smartwatch), a mobile phone, a pager, a laptop computer, a desktop computer, etc. Thus, in these embodiments, notification device 130 may include a user interface component (not shown), such as a screen, a touch screen, a microphone, and / or a keyboard. In some embodiments, notification device 130 may be configured to provide tactile, auditory, and / or visual notifications.

[0025] In some embodiments, one or more cameras 135 may include fixed cameras installed in public spaces and / or office spaces to capture images and monitor activities in the environment. For example, one or more cameras 135 may include surveillance cameras. By employing such cameras, embodiments of the present disclosure can analyze a user's environment and identify the user's position relative to other people, objects, and / or devices in such environment. Thus, embodiments of the present disclosure can obtain a large amount of data for generating notification options and timing schemes (discussed in more detail below). In this way, embodiments of the present disclosure can effectively identify effective notification options for a user.

[0026] In some embodiments, notification source 145 may include a system or device from which notification escalation system 105 obtains notification data. For example, in some embodiments, notification source 145 may include a system such as a public alarm / warning system, an office messaging system, etc. In some embodiments, notification source 145 may be a communication device such as a computer, a mobile phone, etc. In an example, notification source 145 may include a mobile phone from which a coworker attempts to send a text message to the user.

[0027] In some embodiments, server 140 may be a web server capable of storing data and / or software employed by notification escalation system 105. In some embodiments, network 150 may be a wide area network (WAN), a local area network (LAN), the Internet, or an intranet. In some embodiments, network 150 may be substantially similar to or identical to the network described with respect to Figure 5 The cloud computing environment 50 is discussed.

[0028] Figure 2 1 shows a flow chart of an example method 200 for performing notification upgrades according to an embodiment of the present disclosure. The method 200 may be performed by a user such as Figure 1 It is executed by a notification upgrade system such as the notification upgrade system 105.

[0029] In operation 205, the notification upgrade system may obtain initial data from an entity such as a user or programmer of the notification upgrade system. In some embodiments, the initial data may include one or more sets of notification options. The notification options may indicate how the user sends the notification. For example, in some embodiments, a set of notification options may include a tactile alert (e.g., activating a vibration function of a user device such as a mobile phone, smartwatch, pager, etc.); a visual alert (e.g., displaying a notification on a screen of the user device and / or illuminating one or more light emitting diodes (LEDs) of the user device); and an audible alert (e.g., activating one or more sounds from a user device such as a mobile phone and / or a headset / earphone communicatively connected to the mobile phone).

[0030] In some embodiments, a set of notification options may include alerting a bystander. A bystander may refer to a person who is close to the user (e.g., a person within approximately 2 meters (m) to 6 m of the user). In some embodiments, a bystander may refer to a person who is included in an image of the user captured by a camera. For example, a public surveillance camera may capture an image of a user standing on a street corner and a pedestrian approaching the bystander on the street corner. Thus, in some embodiments, alerting such a bystander may include methods such as calling the bystander's notification device (e.g., a mobile phone, smartwatch, pager, etc.) or sending a message to the bystander's notification device so that the bystander can prompt the user to confirm the notification.

[0031] In some embodiments, a set of notification options may include audible messages issued through a speaker that is distinct from and not operatively connected to the user device, such as a speaker of a public address system in the user's environment. For example, in some embodiments, a pedestrian signal at a street intersection may include a speaker. In these embodiments, the notification escalation system may employ such a speaker to issue an audible message to the user (e.g., a message that the pedestrian user is dangerously close to one or more moving vehicles).

[0032] In some embodiments, the initial data obtained in operation 205 may include timing data. In some embodiments, the timing data may include a confirmation time threshold. The confirmation time threshold may refer to a predetermined threshold time for a user to confirm a notification. For example, in some embodiments, the timing data may include a default five-minute time period for a doctor to respond to a page or text message. In some embodiments, the timing data may include a minimum step time. The step time may refer to the time interval between consecutive notifications initiated by the notification escalation system. For example, in some embodiments, the notification escalation system may be configured to wait at least one minute between initiating a first notification and initiating a second notification. In some embodiments, such timing data may be determined by an entity such as a user or programmer of the notification escalation system. In some embodiments, the notification escalation system may select such timing data from a stored dataset. For example, in some embodiments, the stored dataset may include a first confirmation time threshold of two minutes for urgent notifications and a second confirmation time threshold of five minutes for non-urgent notifications. In this example, the notification escalation system may select the confirmation time threshold from the stored dataset based on the urgency associated with the notification (i.e., urgent or non-urgent).

[0033] In some embodiments, the initial data obtained in operation 205 may include training data for training the notification escalation system to perform image analysis. For example, in some embodiments, operation 205 may include training a machine learning algorithm of the notification escalation system using data such as facial images and / or images of scenes (e.g., images of two people talking, images of a person looking at the screen of a device such as a smartwatch, images of a person carrying a mobile phone and / or pager, etc.). Such training may include performing supervised, unsupervised, or semi-supervised training. Additionally, such training may allow such a machine learning algorithm to generate a trained model that is configured to recognize one or more person and / or image characteristics indicative of a user's state (discussed further below).

[0034] According to an embodiment of the present disclosure, the machine learning algorithm may include but is not limited to decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity / metric training, sparse dictionary learning, genetic algorithms, rule-based learning and / or other machine learning techniques.

[0035] For example, the machine learning algorithm can utilize one or more of the following example techniques: K-nearest neighbors (KNN), learning vector quantization (LVQ), self-organizing map (SOM), logistic regression, ordinary least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression splines (MARS), ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS), probabilistic classifier, naive Bayes classifier, binary classifier, linear classifier, hierarchical classifier, canonical correlation analysis (CCA), factor analysis, independent component analysis (ICA), linear discriminant analysis (LDA), multidimensional scaling (MDS), non-negative factor decomposition (NMF), partial least squares regression (PLSR), principal component analysis (PCA), principal component regression (PCR), Sammon map, t-distributed stochastic nearest neighbor embedding (t-SNE), clustering, bootstrap averaging, gradient boosting decision Gradient boosting machine (GBM), inductive bias algorithm, Q-learning, state-action-reward-state-action (SARSA), temporal difference (LAT) learning algorithm, apriori algorithm, equivalence class transformation (ECLAT) algorithm, Gaussian process regression, gene expression programming, grouping method for data manipulation (GMDH), inductive logic programming, instance-based learning, logistic model tree, informative fuzzy network (IFN), hidden Markov model, Gaussian naive Bayes, multinomial naive Bayes, averaged single correlation estimator (AODE), Bayesian network (BN), classification and regression tree (CART), chi-squared automatic interaction detection (CHAID), expectation-maximization algorithm, feedforward neural network, logistic learning machine, self-organizing map, single bond clustering, fuzzy clustering, hierarchical clustering, Boltzmann machine, convolutional neural network, recurrent neural network, hierarchical temporal memory (HTM), and / or other machine learning techniques.

[0036] In operation 210, the notification escalation system may receive notifications from a notification source, such as Figure 1Notification data is obtained from a notification source 145. In some embodiments, the notification data may include information to be sent to the user. For example, in some embodiments, the notification data may include a text or voice message generated by an entity such as an administrative assistant for a user such as a doctor. In some embodiments, the notification data may include a text or voice message generated by an entity such as an automatic alarm system that is configured to issue an alert to a user such as a pedestrian near a traffic intersection or a construction zone. In some embodiments, the notification data may include one or more elements such as a name, time, time period, urgency, and / or destination. In one example, the notification data may include the text message "Please report to room 804B for your 10:30 AM appointment." In this example, the notification data includes the user's destination (i.e., room 804B) and the time at which the user will perform.

[0037] In operation 215, in response to obtaining the notification data in operation 210, the notification upgrade system may obtain image data. The image data may be obtained from at least one camera, e.g. Figure 1 The image data may include one or more images of the user's environment. In some embodiments, the image data may include one or more images of at least one of the user, a bystander, a user device, and / or a notification device. In some embodiments, the image data may include metadata, such as the time and / or location at which the image was captured.

[0038] At operation 220, the notification escalation system may identify the user's status based on the image data obtained at operation 215. In some embodiments, operation 220 may include the notification escalation system employing image analysis techniques (e.g., a trained machine learning model) to identify the user's status from the image data. The user's status may include information regarding the user's location, activities / actions the user is engaged in, one or more devices owned by the user, and / or the user's proximity to one or more bystanders. For example, in some embodiments, the notification escalation system may identify that a user is waiting for an elevator on the third floor of a building based on an image from a camera at that location, which shows the user lingering for a period of time in front of the elevator door. In another example, in some embodiments, the notification escalation system may identify that a user is viewing information on a mobile phone based on a camera image of the user viewing an illuminated screen of the mobile phone. In another example, in some embodiments, the notification escalation system may identify that a user is near a bystander working on a computer based on the bystander and the computer being within the camera's field of view; thus, the bystander and the computer are included in the camera image along with the user. Furthermore, in this example, the notification escalation system may identify the bystander as a colleague of the user by performing facial recognition on the image.

[0039] In some embodiments, operation 220 may include the notification escalation system identifying the user's status at successive times. For example, in some embodiments, the notification escalation system may identify at a first time that the user is walking down a hallway in a first direction. Furthermore, in this example, at a subsequent second time, the notification escalation system may identify that the user is walking down the hallway in a second, opposite direction. As described below, in some embodiments, this changed status may indicate that the user has acknowledged the notification.

[0040] In operation 225, the notification escalation system may generate one or more groups of notification options based at least in part on the user state obtained in operation 220. In the present disclosure, a count of notification options may refer to the number of notification options in a group of notification options. For example, in some embodiments, in operation 220, the notification escalation system may identify that the user is wearing headphones and typing on a laptop computer near a bystander with a mobile phone. In this example, the notification escalation system may generate a group of notification options that includes a count of at least three notification options, such as: (1) sending a visual alert to the user's laptop computer; (2) sending an audible alert that can be received by the user's headphones (e.g., sending an audible alert to the user's mobile device, which can be communicatively connected to the user's headphones); and (3) alerting the bystander by sending a message to the bystander's mobile phone. In some embodiments, the notification escalation system may be configured to prioritize the generated group of notification options. Continuing with the example above, the notification escalation system may prioritize notification option (1) and / or notification option (2) over notification option (3) so as not to disturb bystanders unless the user does not confirm notifications made via notification options (1) and / or (2).

[0041] In some embodiments, operation 225 may include the notification escalation system selecting a confirmation time threshold for one or more notification options based at least in part on the notification data obtained in operation 210. For example, continuing with the example above, the notification data obtained by the notification escalation system may include the message "Meeting starts in 10 minutes, please call the front desk immediately." Continuing with this example, the notification escalation system may employ natural language processing techniques to identify elements such as the 10-minute time period and the request to call immediately. Based on these elements of the notification data that indicate urgency, the notification escalation system may select a threshold, such as a 5-minute confirmation time threshold. Thus, with this selection, the notification escalation system may specify a maximum time of 5 minutes to employ a set of 3 notification options and obtain confirmation data from the user. Confirmation data (discussed further below) may refer to information indicating that the user has received the notification and / or is performing the activity corresponding to the notification. In some embodiments, the confirmation time threshold may be selected by an entity such as a programmer or user of the notification escalation system.

[0042] In operation 230, the notification escalation system may select at least one notification option. In some embodiments, such selection may be based at least in part on the user's status, a priority associated with the notification option, a previously selected notification option, and / or a confirmation time threshold. In some embodiments, operation 230 may include the notification escalation system selecting a notification option from the set of notification options generated in operation 225. For example, in some embodiments, in operation 225, the notification escalation system may generate a set of three notification options, wherein the first notification option has the highest priority and the third notification option has the lowest priority. Additionally, in this example, the set of three notification options may have a corresponding confirmation time threshold of 10 minutes (e.g., the notification escalation system may specify a maximum of 10 minutes to utilize the three notification options in the set and obtain confirmation data from the user). In this example, the notification escalation system may select the first notification option at the first time based on its highest priority. Continuing with this example, the notification escalation system may select the second notification option at the second time in response to determining that the user has not responded to the notification via the first notification option. Continuing with this example, the notification escalation system may select all three notification options in the set of notification options at a third time in response to determining that the remaining time of the 10-minute confirmation time threshold is less than the minimum remaining time (e.g., there are three minutes left in the 10-minute confirmation time threshold, which is less than the 5-minute minimum remaining time threshold). Thus, embodiments of the present disclosure can increase the likelihood that a user will confirm a notification.

[0043] In operation 235, the notification escalation system may initiate at least one notification. In some embodiments, the notification escalation system may perform operation 235 in response to selecting at least one notification option in operation 230. Initiating a notification may include the notification escalation system issuing a request to allow at least one user device (e.g., Figure 1 ) and / or at least one notification device (e.g., Figure 1In some embodiments, operation 235 may include the notification escalation system issuing a command, based on the notification options, to cause the user's smartwatch to vibrate. In this example, such vibration may prompt a user who is unresponsive to a phone call to check the message on the user's smartwatch. In another example, in some embodiments, operation 235 may include the notification escalation system issuing a command, based on the notification options, to cause a laptop computer to prominently display a text message on its screen. In this example, such visual display may prompt a user who is unresponsive to a text message to receive the text message on the user's laptop computer. In another example, in some embodiments, operation 235 may include the notification escalation system issuing a command, based on the notification options, to cause the user's office building's public address system to play an audible message for the user. In another example, in some embodiments, operation 235 may include the notification escalation system issuing a command, based on the notification options, to cause the user's office building's public address system to play an audible message for the user. In another example, in some embodiments, operation 235 may include the notification escalation system issuing the commands discussed in the above examples simultaneously based on a set of notification options.

[0044] In operation 240, in response to initiating the notification in operation 235, the notification escalation system may obtain image data. Such image data may be obtained in a manner substantially similar to that described in operation 215. In some embodiments, operation 240 may include the notification escalation system obtaining one or more sets of images over multiple time periods. For example, operation 215 may include the notification escalation system obtaining a first set of images of the user's environment between 9:00 AM and 9:05 AM, and obtaining a second set of images of the user's environment between 9:15 AM and 9:20 AM. In this manner, the notification escalation system may determine whether the user confirms the first notification or a subsequent notification based on the user's actions captured in the images.

[0045] In operation 245, the notification upgrade system may determine whether the user confirmed the notification. In some embodiments, if the notification upgrade system obtains confirmation data, the notification upgrade system may determine that the user confirmed the notification. In these embodiments, if the notification upgrade system does not obtain confirmation data, the notification upgrade system may determine that the user did not confirm the notification. Confirmation data may refer to information indicating that the user has received the notification, is performing an activity corresponding to the notification, or has completed the activity corresponding to the notification. For example, in some embodiments, the confirmation data may include a confirmation message from a user device confirming that the user has read the notification message. In some embodiments, the user may choose to send such a confirmation message to the notification upgrade system via the user device.

[0046] In some embodiments, the confirmation data may include a determination by the notification escalation system that the user is performing / has completed the activity corresponding to the notification. In these embodiments, this determination may be based on the notification escalation system's analysis of the image data obtained in operation 240. For example, in some embodiments, such image data may show the user stopping walking, retrieving their mobile phone, and viewing and illuminating the mobile phone screen. In this example, the image data may indicate to the notification escalation system that the user is reading a received text message notification. Therefore, in this example, the notification escalation system may determine that the user is performing the activity corresponding to the notification. In another example, in some embodiments, the image data may show the user being approached by a bystander and conversing with the bystander. In this example, the image data may indicate to the notification escalation system that the bystander is transmitting a notification message to the user and / or prompting the user to retrieve the notification. Therefore, in this example, the notification escalation system may determine that the user is performing the activity corresponding to the notification. In some embodiments, the image data may show the user at the destination specified in the notification message. In these embodiments, the notification escalation system may determine that the user has completed the activity corresponding to the notification message.

[0047] In some embodiments, the confirmation data may be selected from the group consisting of (1) a confirmation message from the user device and (2) image data indicating to the notification escalation system that the user has completed the activity corresponding to the notification. Thus, in these embodiments, the notification escalation system may determine that it has not received confirmation data unless it receives at least one of these two types of data. For example, the notification escalation system may initiate a notification requesting the user to go to room A. In this example, the notification escalation system may consider the confirmation message received from the user to be confirmation data indicating that the user has confirmed the notification. Alternatively, in this example, the notification escalation system may obtain a subsequent image of the user standing in room A, and it may consider that image to be confirmation data indicating that the user has confirmed the notification. Alternatively, in this example, the notification escalation system may obtain a subsequent image of the user moving towards room A or not responding to the notification, and it may consider that such image to be not confirmation data. In these cases, the notification escalation system may determine that the user has not confirmed the notification.

[0048] In some embodiments, the notification escalation system may only consider image data as confirmation data indicating that the user has completed the activity corresponding to the notification. For example, if the notification escalation system initiates a notification requesting that the user go to Room A, the user's confirmation message to the notification escalation system would not constitute confirmation data. However, obtaining a subsequent image of the user standing in Room A would constitute confirmation data for the notification escalation system. Therefore, the notification escalation system can be configured to continue method 200 until it obtains an indication that the user has completed the activity corresponding to the notification.

[0049] In operation 245, if the notification escalation system determines that the user has not confirmed the notification, the notification escalation system may proceed to operation 250. Alternatively, if the notification escalation system determines that the user has confirmed the notification, method 200 may end (e.g., the notification escalation system may refrain from initiating any subsequent notifications and end method 200). Additionally, if the notification escalation system determines that the confirmation time threshold has been exceeded, method 200 may end.

[0050] In operation 250, the notification escalation system may generate and employ a timing scheme to initiate one or more subsequent notifications. The timing scheme may refer to a process for implementing a step time (e.g., a time interval between consecutive notifications initiated by the notification escalation system). In some embodiments, operation 250 may include the notification escalation system calculating such a step time. In some implementations, such calculation may include the notification escalation system dividing a confirmation time threshold by a count of notification options included in a set of notification options.

[0051] In some embodiments, the timing scheme generated in operation 250 may include one or more of the following. In the event that the image data obtained in operation 240 indicates that the user does not perform the activity corresponding to the notification initiated in operation 235, the notification escalation system may proceed to operation 215 in response to allowing one step time to pass. In the event that the image data obtained in operation 240 indicates that the user performs the activity corresponding to the notification initiated in operation 235 (for example, the user is traveling to the destination specified in the notification message), the notification escalation system may proceed to operation 215 in response to allowing two step times to pass. By adopting such a timing scheme, embodiments of the present disclosure may allow users who appear to be responsive to a notification additional time to acknowledge the notification. In this way, embodiments of the present disclosure may reduce the number of notifications initiated, which may reduce the possibility of burdening users due to excessive notifications.

[0052] Figure 3 An example environment 300 is depicted in which a notification escalation system may be employed according to an embodiment of the present disclosure. Figure 3 (The distances and sizes of objects shown in the figures are not to scale.) Environment 300 includes a plurality of moving vehicles 325 at a street intersection 305. Environment 300 also includes a public surveillance camera 315 and a crosswalk speaker 320. Public surveillance camera 315 and crosswalk speaker 320 can be operably connected to a separate public safety system (not shown) that is configured to warn pedestrians in environment 300 of potential traffic hazards. Crosswalk speaker 320 can be configured to emit an audible message, such as a message regarding when it is safe for pedestrians to cross the street.

[0053] Environment 300 includes user 330 and bystander 345 on sidewalk 310. In this example, the independent public safety system may predict, by employing methods outside the scope of the present disclosure, that the user is dangerously approaching intersection 305 and should be issued a warning message, "You are approaching an intersection; watch out for oncoming traffic." To manage the issuance of such messages, the independent public safety system may employ a notification escalation system (not shown) according to an embodiment of the present disclosure.

[0054] The notification upgrade system may obtain a warning message and image data from camera 315. Based on the image data obtained at a first time, the notification upgrade system may recognize that user 330 is carrying mobile phone 335 and wearing headset 340. In response, the notification upgrade system may initiate a text notification including a warning message to the user's mobile phone 335. Additionally, the notification upgrade system may initiate a tactile alert to the user's mobile phone 335 to prompt user 330 to check the text notification. Based on the image data obtained at a second time, the notification upgrade system may determine that user 330 has not acknowledged the text notification. Based on the image data obtained at a third time, the notification upgrade system may recognize that user 330 has moved to location 355. Additionally, based on such image data, the notification upgrade system may recognize that user 330 is approaching bystander 345 carrying mobile phone 350 and crosswalk speaker 320. In response, the notification upgrade system may initiate the notification discussed above as well as (1) an audible notification including a warning message from crosswalk speaker 320 and (2) a text notification to the bystander's mobile phone 350 requesting bystander 345 to warn user 330. Thus, embodiments of the present disclosure may allow for an increased set of notification options for notifying a user.

[0055] Figure 4 Depicted are representative major components of an exemplary computer system 401 that can be used according to embodiments of the present disclosure. The specific components depicted are presented for illustrative purposes only and are not necessarily the only such variations. Computer system 401 may include a processor 410, a memory 420, an input / output interface (also referred to herein as I / O or I / O interface) 430, and a main bus 440. Main bus 440 may provide a communication path for other components of computer system 401. In some embodiments, main bus 440 may be connected to other components such as a dedicated digital signal processor (not shown).

[0056] The processor 410 of computer system 401 may include one or more CPUs 412. Processor 410 may additionally include one or more memory buffers or caches (not shown) that provide temporary storage of instructions and data for CPU 412. CPU 412 can execute instructions from cache or from the input provided by memory 420, and output the result to cache or memory 420. CPU 412 may include one or more circuits configured to perform one or more methods according to an embodiment of the present invention. In some embodiments, computer system 401 may include multiple processors 410, which is a typical feature of relatively large systems. However, in other embodiments, computer system 401 may be a single processor with a single CPU 412.

[0057] The memory 420 of the computer system 401 may include a memory controller 422 and one or more memory modules (not shown) for temporary or permanent storage of data. In some embodiments, the memory 420 may include a random access semiconductor memory, a storage device, or a storage medium (volatile or non-volatile) for storing data and programs. The memory controller 422 may communicate with the processor 410 to facilitate the storage and retrieval of information in the memory modules. The memory controller 422 may communicate with the I / O interface 430 to facilitate the storage and retrieval of inputs or outputs in the memory modules. In some embodiments, the memory modules may be dual in-line memory modules.

[0058] The I / O interface 430 may include an I / O bus 450, a terminal interface 452, a storage interface 454, an I / O device interface 456, and a network interface 458. The I / O interface 430 may connect the host bus 440 to the I / O bus 450. The I / O interface 430 may direct instructions and data from the processor 410 and the memory 420 to the various interfaces of the I / O bus 450. The I / O interface 430 may also direct instructions and data from the various interfaces of the I / O bus 450 to the processor 410 and the memory 420. The various interfaces may include the terminal interface 452, the storage interface 454, the I / O device interface 456, and the network interface 458. In some embodiments, the various interfaces may include a subset of the above interfaces (for example, an embedded computer system in an industrial application may not include the terminal interface 452 and the storage interface 454).

[0059] Logic modules throughout computer system 401, including but not limited to memory 420, processor 410, and I / O interface 430, can communicate failures and changes to one or more components to a hypervisor or operating system (not depicted). The hypervisor or operating system can allocate the various resources available in computer system 401 and track the location of data in memory 420 and the locations of processes assigned to the various CPUs 412. In embodiments that combine or rearrange elements, various aspects of the capabilities of the logic modules can be combined or redistributed. Such variations will be apparent to those skilled in the art.

[0060] It is understood in advance that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to a cloud computing environment. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0061] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0062] Features are as follows:

[0063] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without manual interaction with the service provider.

[0064] Wide Area Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0065] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. This is location-independent in the sense that consumers typically do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0066] Rapid elasticity: In some cases, the ability to scale out quickly and in quickly can be provided quickly and elastically. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0067] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the provider and consumer of the utilized service.

[0068] The service model is as follows:

[0069] Software as a Service (SaaS): The ability provided to consumers is to use the provider's applications running on cloud infrastructure. Applications are accessed from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0070] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications onto cloud infrastructure. These applications are built using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but do have control over the deployed applications and possibly the configuration of the application hosting environment.

[0071] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0072] The deployment model is as follows:

[0073] Private cloud: The cloud infrastructure is operated solely for the organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0074] Community cloud: Cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0075] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by the organization that sells cloud services.

[0076] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a unique entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0077] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0078] Now refer to Figure 5 , depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Figure 5 The types of computing devices 54A-N shown in are intended for illustration only, and computing node 10 and cloud computing environment 50 may communicate with any type of computerized device over any type of network and / or network-addressable connection (eg, using a web browser).

[0079] Now refer to Figure 6 , showing the cloud computing environment 50 ( Figure 5 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 6 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0080] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage device 65; and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0081] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0082] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing and other resources for performing tasks within a cloud computing environment. Metering and pricing 82 provides cost tracking when utilizing resources in a cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. A user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0083] The workload layer 90 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and notification escalation logic 96.

[0084] As discussed in greater detail herein, it is contemplated that some or all of the operations of some embodiments of the methods described herein may be performed in an alternate order or may not be performed at all; furthermore, multiple operations may occur concurrently or as part of a larger process.

[0085] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0086] A computer-readable storage medium can be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0087] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0088] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform various aspects of the present invention, an electronic circuit comprising, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0089] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowcharts and / or block diagrams and the combination of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0090] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device and / or other equipment to operate in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0091] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0092] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the functions noted in the frame may not occur in the order noted in the figure. For example, the two frames shown in succession can actually be implemented as a step, simultaneously, substantially simultaneously, in a manner that overlaps part or all of the time, or these frames can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0093] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to explain the principles of the embodiments, practical applications, or improvements over existing technologies in the market, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method comprising: Get the user's notification data; obtaining a first set of images of the user from one or more cameras; identifying a first state of the user based on the first set of images; generating a set of notification options based at least in part on the first state of the user; selecting a first notification option from the set of notification options; In response to the selection, initiating a first notification via the first notification option; Determining at a first time that the user has not confirmed the first notification; responsive to the determining, obtaining a second set of images of the user from the one or more cameras; identifying a second state of the user based on the second set of images; selecting at least one second notification option from the set of notification options based at least in part on the second state of the user; as well as In response to selecting the at least one second notification option, at least one second notification is initiated via the at least one second notification option.

2. The computer-implemented method of claim 1 , wherein: The one or more cameras are surveillance cameras for the environment where the user is located.

3. The computer-implemented method of claim 1 , further comprising calculating a step time, and in, The step time is a time interval between initiating the first notification and initiating the at least one second notification.

4. The computer-implemented method of claim 3, wherein: The notification data includes a confirmation time threshold; and The calculating step time comprises dividing the confirmation time threshold by a count of notification options included in the set of notification options.

5. The computer-implemented method of claim 1 , further comprising determining, at a second time after the first time, that the user acknowledges the at least one second notification; and Responsive to the determination at the second time, initiating a third notification is avoided.

6. The computer-implemented method of claim 5, wherein: The notification data includes the destination of the user; and Wherein, determining that the user acknowledges the at least one second notification comprises determining that the user is traveling toward the destination based on a third set of images of the user obtained from the one or more cameras.

7. The computer-implemented method of claim 1 , wherein: The set of notification options includes initiating a notification to a notification device of a bystander, the bystander being included in the second set of images.

8. A system for notifying an upgrade, comprising: processor; as well as A memory in communication with the processor, the memory containing program instructions that, when executed by the processor, are configured to cause the processor to perform a method for generating a summary text combination, the method comprising: Get the user's notification data; obtaining a first set of images of the user from one or more cameras; identifying a first state of the user based on the first set of images; generating a set of notification options based at least in part on the first state of the user; selecting a first notification option from the set of notification options; In response to the selection, initiating a first notification via the first notification option; Determining at a first time that the user has not confirmed the first notification; responsive to the determining, obtaining a second set of images of the user from the one or more cameras; identifying a second state of the user based on the second set of images; selecting at least one second notification option from the set of notification options based at least in part on the second state of the user; and In response to selecting the at least one second notification option, at least one second notification is initiated via the at least one second notification option.

9. The system according to claim 8, wherein: The one or more cameras are surveillance cameras for the environment where the user is located.

10. The system of claim 8, further comprising calculating a step time, and in, The step time is a time interval between initiating the first notification and initiating the at least one second notification.

11. The system according to claim 10, wherein: The notification data includes a confirmation time threshold; and The calculating step time comprises dividing the confirmation time threshold by a count of notification options included in the set of notification options.

12. The system of claim 8, further comprising determining, at a second time after the first time, that the user acknowledges the at least one second notification; and Responsive to the determination at the second time, initiating a third notification is avoided.

13. The system according to claim 12, wherein: The notification data includes the user's destination; and Wherein determining that the user acknowledges the at least one second notification comprises determining that the user is traveling toward the destination based on a third set of images of the user obtained from the one or more cameras.

14. The system according to claim 8, wherein The set of notification options includes initiating a notification to a notification device of a bystander, the bystander being included in the second set of images.

15. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, wherein: The computer-readable storage medium itself is not a transitory signal, and the program instructions can be executed by a processor to cause the processor to perform a method for generating a summary text composition, the method comprising: Get the user's notification data; obtaining a first set of images of the user from one or more cameras; identifying a first state of the user based on the first set of images; generating a set of notification options based at least in part on the first state of the user; selecting a first notification option from the set of notification options; In response to the selection, initiating a first notification via the first notification option; Determining at a first time that the user has not confirmed the first notification; responsive to the determining, obtaining a second set of images of the user from the one or more cameras; identifying a second state of the user based on the second set of images; selecting at least one second notification option from the set of notification options based at least in part on the second state of the user; and In response to selecting the at least one second notification option, at least one second notification is initiated via the at least one second notification option.

16. The computer program product of claim 15, wherein: The one or more cameras are surveillance cameras for the environment where the user is located.

17. The computer program product of claim 15, further comprising calculating a step time, and in, The step time is a time interval between initiating the first notification and initiating the at least one second notification.

18. The computer program product of claim 17, wherein: The notification data includes a confirmation time threshold; and The calculating step time comprises dividing the confirmation time threshold by a count of notification options included in the set of notification options.

19. The computer program product of claim 15, further comprising determining, at a second time after the first time, that the user acknowledges the at least one second notification; and Responsive to the determination at the second time, initiating a third notification is avoided.

20. The computer program product of claim 19, wherein: The notification data includes the user's destination; and Wherein, determining that the user acknowledges the at least one second notification comprises determining that the user is traveling toward the destination based on a third set of images of the user obtained from the one or more cameras.

Images