Notification escalation based on optical recognition

Abstract

A computer-implemented method comprising: - Receiving notification data for a user; - Receiving an initial set of images of the user from one or more cameras, wherein the one or more cameras are independent of the user and the user's terminal device; - Identifying, based on the initial set of images, an initial status of the user; - Generating, at least partially based on the initial status of the user, an initial set of notification options; - Selecting an initial notification option from the set of notification options, wherein the initial notification option is associated with the user's terminal device; - Triggering, in response to the selection, an initial notification by the initial notification option; - Determining, at an initial time, that the user does not acknowledge the initial notification;- Receiving, in response to the detection of a second set of images of the user from one or more cameras independent of the user and the terminal device; - Identifying, based on the second set of images, a second status of the user; - Selecting at least one second notification option from the set of notification options, wherein the second notification option, selected for one or more devices independent of the user's terminal device, transmits a notification associated with the first notification; and - Triggering, in response to the selection of the at least one second notification option, at least one second notification by the at least one second notification option.

Description

BACKGROUND

[0001] The present disclosure relates to notifications and, in particular, dynamic notifications.

[0002] Notifications, such as messages and alerts directed at a user, can be received via multiple devices and in a variety of ways. For example, an employee in an office building might receive an audible alert through the building's loudspeaker system, a visual message notification on their computer screen, and / or a visual alert on their mobile phone. In some cases, such an employee may be able to choose which method of receiving notifications to use. SUMMARY

[0003] According to embodiments of the present disclosure, a method may include obtaining notification data for a user. The method may include obtaining an initial set of images of the user from one or more cameras. The method may include identifying an initial status of the user based on the initial set of images. The method may include generating an initial set of notification options, at least partially, based on the initial status of the user. The method may include selecting an initial notification option from the set of notification options. In response to the selection, the method may include triggering an initial notification by the initial notification option. The method may include, at an initial time, determining that the user does not acknowledge the initial notification.The method may, in response to the detection, involve receiving a second set of images of the user from one or more cameras. The method may involve identifying a second user state based on the second set of images. The method may involve selecting at least one second notification option from the set of notification options. The method may, in response to the selection of the at least one second notification option, involve triggering at least one second notification by that at least one second notification option.

[0004] A system and a computer program product that conform to the procedure described above are also included herein.

[0005] The foregoing brief description is not intended to describe every illustrated or every form of realization of the present disclosure. List of characters

[0006] The drawings included in this application are incorporated into the specification and form part thereof. They illustrate embodiments of the present disclosure and, together with the description, serve to explain the basic ideas of the disclosure. The drawings merely illustrate certain embodiments and do not limit the disclosure. Fig. Figure 1 represents an exemplary data processing environment that includes a notification escalation system according to embodiments of the present disclosure. Fig. Figure 2 presents a flowchart of an exemplary procedure for performing notification escalation according to embodiments of the present disclosure. Fig. Figure 3 represents an exemplary environment in which a notification escalation system according to embodiments of the present disclosure can be used. Fig. 4 represents characteristic key components of a computer system that can be used according to embodiments of the present disclosure. Fig. Figure 5 represents a cloud computing environment according to embodiments of the present disclosure. Fig. 6 represents abstraction model layers according to embodiments of the present disclosure.

[0007] Although various modifications can be made to the invention and it can take alternative forms, their specific features are shown by way of example in the drawings and described in detail. It should be clear, however, that the intention is not to limit the invention to the specific embodiments described. On the contrary, the invention is intended to encompass all modifications, equivalents, and alternatives that fall within the fundamental concept and scope of protection of the invention. DETAILED DESCRIPTION

[0008] Aspects of the present disclosure relate to notifications; in particular, certain aspects relate to notification escalation based on optical detection. While the present disclosure is not necessarily limited to such use cases, various aspects of the disclosure may become clear through a discussion of different examples using this context.

[0009] Notifications, such as messages and alerts directed at a user, can be received via multiple devices and in a variety of ways. For example, an employee in an office building might receive an audible alert through the building's loudspeaker system, a visual message notification on their computer screen, and / or a visual alert on their mobile phone. In some cases, a user can choose how to receive notifications. For example, in a quiet environment, a user might choose to receive only visual notifications on their laptop screen to avoid disturbing others. In another example, a user in a noisy or active environment might choose to receive haptic alerts (e.g., a buzzer or a light).physical vibrations) are received from the user's mobile phone in order to draw the user's attention away from the surroundings and onto the mobile phone.

[0010] Because users can frequently change their locations and environments, the most effective way for a user to receive a notification can also change. In some cases, neither the user nor the entity attempting to notify the user may recognize the most effective way to notify the user at a given time. For example, a user who has muted their mobile phone may not realize that a text message displayed on the user's laptop screen might be a more effective notification than a text message sent to the user's mobile phone.In another example, a colleague trying to reach an employee in a hallway might not realize that an audible message over a loudspeaker system could be a more effective notification than a phone call. In time-critical situations, such as when urgent medical attention is needed or someone is at risk of injury, efficiently identifying an effective way to notify a user can be crucial.

[0011] To address these and other problems, embodiments of the present disclosure include a notification escalation system. In some embodiments, the notification escalation system can utilize optical detection to identify one or more notification options for notifying a user. In particular, the notification escalation system can use a set of cameras to identify a user's status. Based on such a status, the notification escalation system can generate a set of notification options for notifying the user. Furthermore, according to the set of notification options, the notification escalation system can cyclically trigger notifications that take into account changes in the optically identified status of the user.In some embodiments, the notification escalation system can continue such cyclical triggering of notifications until it receives awareness data or until an awareness threshold time is exceeded.

[0012] Thus, embodiments of the present disclosure can automatically generate notifications tailored to a user's status. Furthermore, embodiments of the present disclosure can provide more options for notifying the user by using optical detection to identify a user's status. Accordingly, embodiments of the present disclosure can improve the ability to efficiently determine an effective way to notify a user.

[0013] With reference to the characters, illustrates Fig. 1. A data processing environment 100 comprising one or more notification escalation systems 105, user units 125, notification units 130, cameras 135, notification sources 145, servers 140, and / or networks 150. In some embodiments, at least one notification escalation system 105, one user unit 125, one notification unit 130, one camera 135, one notification source 145, and / or one server 140 can exchange data with each other via the at least one network 150. For example, in some embodiments, at least one notification escalation system 105 can exchange data with at least one user unit 125 via the at least one network 150.One or more of the notification escalation system 105, the user unit 125, the notification unit 130, the camera 135, the notification source 145, the server 140 and / or the network 150 can be a computer system such as the one in relation to . Fig. The 4 discussed computer system 401 includes.

[0014] In some embodiments, the notification escalation system 105 can be included in software installed on a computer system comprising at least one user unit 125, notification unit 130, camera 135, notification source 145, and / or server 140. In one example, the notification escalation system 105 can be included in some embodiments as a plug-in software component of software installed on a user unit 125. The notification escalation system 105 can include program instructions that are implemented by a processor, such as a processor of the server 140, to perform one or more operations related to the Fig. 2 and Fig. 3 will be discussed.

[0015] In some embodiments, the notification escalation system 105 may comprise one or more modules, such as an image analysis manager 110, a selection manager 115, and / or a data manager 120. In some embodiments, the image analysis manager 110, the selection manager 115, and / or the data manager 120 may be integrated into a single module. In some embodiments, the image analysis manager 110 may be configured to perform image analysis on images received by the notification escalation system 105. In some embodiments, the image analysis manager 110 may include a trained model (not shown) configured to identify and / or categorize image features.In some embodiments, one or more of the image analysis manager 110, the selection manager 115 and / or the data manager 120 may comprise program instructions that are implemented by a processor, such as a processor of the server 140, to perform one or more operations related to the . Fig. 2 and Fig. 3 will be discussed. For example, in some embodiments, the image analysis manager 110 may contain program instructions to perform steps 215, 220, 240 and / or 245, Fig. 2. To perform. In some embodiments, the selection manager may contain 115 program instructions to perform steps 225 to 235. Fig. 2. In some versions, the data manager 120 may contain program instructions to perform steps 205, 210, 245 and / or 250. Fig. 2, to be carried out.

[0016] In some embodiments, the notification escalation system 105 may include tools for facial recognition analysis, audio analysis, and / or natural language processing. For example, in some embodiments, the data manager 120 may be configured to use natural language processing technology to identify notification elements (e.g., a time, name, or location) contained in text notification data. Furthermore, in some embodiments, the data manager may be configured to use audio analysis technology (e.g., speech-to-text technology) to identify notification elements contained in audio notification data, such as a voicemail message.

[0017] In some embodiments, a user unit 125 may comprise a user's electronic device, such as a computerized wristwatch (e.g., a smartwatch), a mobile phone, a pager, a notebook computer, a desktop computer, and the like. Therefore, the user unit 125 may include user interface components (not shown) such as a screen, a touchscreen, a microphone, and / or a keyboard. In some embodiments, the user unit 125 may be configured to provide haptic, audible, and / or visual notifications.

[0018] In some embodiments, the notification unit 130 may include an electronic device belonging to a bystander or entity other than a user. For example, in some embodiments, the notification unit 130 may include an office loudspeaker system or a pedestrian crossing loudspeaker system. In some embodiments, the notification unit 130 may include an electronic device belonging to a bystander, such as a smartwatch, mobile phone, pager, notebook computer, desktop computer, and the like. Therefore, in these embodiments, the notification unit 130 may include user interface components (not shown) such as a screen, touchscreen, microphone, and / or keyboard.In some embodiments, the notification unit 130 can be configured to provide haptic, acoustic and / or visual notifications.

[0019] In some embodiments, one or more cameras may comprise 135 stationary cameras installed in public spaces and / or office spaces to capture images and monitor activity in an environment. For example, one or more cameras may comprise 135 surveillance cameras. By using such cameras, embodiments of the present disclosure can analyze a user's environment and identify a user's position relative to other people, objects, and / or units in such an environment. Thus, embodiments of the present disclosure can obtain a substantial amount of data for generating notification options and a time-based model (as discussed in more detail below). In this way, embodiments of the present disclosure can efficiently identify effective notification options for a user.

[0020] In some embodiments, the notification source 145 may include a system or unit from which the notification escalation system 105 receives notification data. For example, in some embodiments, the notification source 145 may include a system such as a public alarm / warning system, an office messaging system, and the like. In some embodiments, the notification unit 145 may be a data transmission unit such as a computer, a mobile phone, and the like. For example, the notification source 145 may include a mobile phone from which a coworker attempts to send a text message to a user.

[0021] In some embodiments, the server 140 may be a web server on which data and / or software may be stored that are used by the notification escalation system 105. In some embodiments, the network 150 may be a wide area network (WAN), a local area network (LAN), the internet, or an intranet. In some embodiments, the network 150 may essentially resemble a cloud computing environment 50, which, with respect to Fig. 5 is discussed, or be identical to it.

[0022] Fig. Figure 2 illustrates a flowchart of an exemplary method 200 for performing notification escalation according to embodiments of the present disclosure. The method 200 can be implemented by a notification escalation system such as the notification escalation system 105, Fig. 1. will be carried out.

[0023] In step 205, the notification escalation system can receive initial data from an entity such as a user or a programmer of the notification escalation system. In some embodiments, the initial data can include a set of one or more notification options. A notification option can refer to a method of delivering a notification to a user. For example, in some embodiments, a set of notification options can include a haptic alert (e.g., activating a vibration function on a user device such as a mobile phone, smartwatch, pager, and the like); a visual alert (e.g., displaying a notification on a user device screen and / or illuminating one or more light-emitting diodes (LEDs) on a user device); and an audible alert (e.g.,This includes enabling the output of one or more tones from a user unit such as a mobile phone and / or headphones / earphones connected to a mobile phone for data exchange purposes.

[0024] In some embodiments, a set of notification options may include warning a bystander. A "bystander" may refer to a person in close proximity to a user (e.g., a person within a radius of approximately 2 meters (m) to 6 m around the user). In some embodiments, a "bystander" may refer to a person in an image of a user captured by a camera. For example, a public surveillance camera may capture an image of a user standing on a street corner and a bystander approaching the street corner. Thus, in some embodiments, warning such a bystander may involve a method such as calling or transmitting a message to a notification device (e.g., mobile phone, smartwatch, pager, and the like) belonging to the bystander, enabling the bystander to request the user to acknowledge a notification.

[0025] In some embodiments, a set of notification options may include the output of an audible message through a loudspeaker that is separate from and not functionally connected to a user unit, such as a loudspeaker in a public address system located in the user's environment. For example, in some embodiments, a pedestrian signal at a road intersection may include a loudspeaker. In these embodiments, the notification escalation system may use such a loudspeaker to output an audible message to a user (e.g., a message that a pedestrian user is dangerously approaching one or more moving vehicles).

[0026] In some embodiments, the initial data obtained in step 205 may include time-interval data. In some embodiments, time-interval data may include an acknowledgement time threshold. An acknowledgement time threshold may refer to a predefined threshold time for a user to acknowledge a notification. For example, in some embodiments, time-interval data may include a standard 5-minute period for a physician to respond to a Page or text message. In some embodiments, time-interval data may include a minimum step time. A step time may refer to a time interval between successive notifications triggered by the notification escalation system.For example, in some embodiments, the notification escalation system can be configured to wait at least one minute between triggering a first notification and triggering a second notification. In some embodiments, such time interval data can be determined by an entity such as a user or a programmer of the notification escalation system. In such embodiments, the notification escalation system can select such time interval data from a stored data set. For example, in some embodiments, a stored data set can contain a first awareness time threshold of two minutes for an urgent notification and a second awareness time threshold of five minutes for a non-urgent notification.In this example, the notification escalation system can select a notification time threshold from a stored record based on a urgency level (i.e., urgent or non-urgent) associated with a notification.

[0027] In some embodiments, the initial data obtained in step 205 may include training data to train the notification escalation system to perform image analysis. For example, in some embodiments, step 205 may involve training a machine learning algorithm of the notification escalation system with data such as facial images and / or images of scenarios (i.e., images of two people talking to each other, images of a person looking at the screen of a device such as a smartwatch, images of a person carrying a mobile phone and / or pager, and the like). Such training may include supervised, unsupervised, and semi-supervised training.Furthermore, such training can enable such a machine learning algorithm to generate a trained model configured to identify one or more people and / or image properties that indicate a user status (which will be discussed further below).

[0028] According to embodiments of the present disclosure, machine learning algorithms may include decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representative learning, similarity-based or metric-based learning, learning based on small lexicons, genetic algorithms, rule-based learning and / or other machine learning techniques.

[0029] For example, machine learning algorithms can use one or more of the following exemplary techniques: k-nearest neighbor (KNN), learning vector quantization (LVQ), self-organizing map (SOM), logistic regression, ordinary least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression spline (MARS), ridge regression, least absolute shrinkage and selection operator (LASSO), elastic network, least-angle regression (LARS), probabilistic classifier, naiver-Bayes classifier, binary classifier, linear classifier, hierarchical classifier, canonical correlation analysis (CCA), factor analysis, independent component analysis (ICA), discriminant function (linear) discriminant analysis (LDA),Multidimensional scaling (MDS), non-negative metric factorization (NMF), partial least squares regression (PLSR), principal component analysis (PCA), principal component regression (PCR), Sammon projection, t-distributed stochastic neighbor embedding (t-SNE), bootstrap aggregation, ensemble mean, gradient boosted decision tree (GBRT), gradient boosting machine (GBM), inductive bias, Q-learning, state-action-reward-state-action (SARSA), temporal difference learning (TD-learning), apriori algorithms, equivalence class transformation algorithms (ECLAT algorithms) (ECLAT = equivalence class transformation) Gaussian regression, gene expression programming, Group Method of Data Handling (GMDH), inductive logic programming, instance-based learning, logistic model treesInformation Fuzzy Networks (IFN), hidden Markov models, Gaussian Naive Bayes, multinomial Naive Bayes, Averaged One-Dependency Estimators (AODE), Bayesian network (BN), classification and regression tree (CART), CHi-Squared Automatic Interaction Detection (CHAID), expectancy maximization algorithm, forward neural networks, logic learning machine, self-organizing map, single-linkage clustering, fuzzy clustering, hierarchical clustering, Boltzmann machines, convolutional neural networks, recurrent neural networks, hierarchical temporal memory (HTM) and / or other machine learning techniques.

[0030] In step 210, the notification escalation system can receive notification data from a notification source such as notification source 145, Fig. 1. Received. In some embodiments, notification data may include information to be delivered to a user. For example, in some embodiments, notification data may include a text or voice message generated by an entity such as a helper of a user, such as a doctor. In some embodiments, notification data may include a text or voice message generated by an entity such as an automated warning system configured to warn a user, such as a pedestrian, near a road intersection or construction site. In some embodiments, notification data may include one or more elements, such as a name, time, time period, urgency level, and / or destination.For example, notification data might include a text message such as "Please go to your appointment at 10:30 AM in room 804B." In this example, the notification data includes a destination (i.e., room 804B) for the user and a time for the user to be there.

[0031] In step 215, the notification escalation system can receive image data in response to the notification data received in step 210. The image data can come from at least one camera, such as camera 135. Fig. 1. Image data can comprise sets of one or more images of an environment in which a user is located. In some embodiments, image data can comprise one or more images of at least one user, a bystander, a user unit, and / or a notification unit. In some embodiments, image data can include metadata such as a time and / or location where an image is taken.

[0032] In step 220, the notification escalation system can identify a user status based on the image data obtained in step 215. In some embodiments, step 220 may involve the notification escalation system using image analysis technology (e.g., a trained machine learning model) to identify a user status from the image data. A user status may include information relating to a user's location, activities / actions in which the user is involved, one or more units owned by the user, and / or a user's proximity to one or more bystanders.For example, in some embodiments, the notification escalation system can identify, based on images from a camera at that location showing the user standing in front of the elevator doors for a certain period of time, that a user is waiting for an elevator on the third floor of a building. In another example, in some embodiments, the notification escalation system can identify, based on camera images of a user looking at a lit mobile phone screen, that the user is reviewing information on a mobile phone.In another example, in some embodiments, the notification escalation system can identify that a user is near a bystander working at a computer based on the presence of a bystander and a computer within a camera's field of view; thus, the bystander and the computer are included in camera images along with the user. Furthermore, in this example, the notification escalation system can identify the bystander as a colleague of the user by performing facial recognition on such images.

[0033] In some embodiments, step 220 may include the notification escalation system identifying a user's status at successive time points. For example, in some embodiments, the notification escalation system may, at a first time point, identify that a user is walking down a corridor in a first direction. Furthermore, in this example, at a subsequent second time point, the notification escalation system may identify that the user is running down a corridor in a second, opposite direction. As discussed below, in some embodiments, such a change in status may indicate that a user is acknowledging a notification.

[0034] In step 225, the notification escalation system can generate a set of one or more notification options, at least partially, based on a user status obtained in step 220. In this disclosure, a number of notification options can refer to a number of notification options within a set of notification options. For example, in some embodiments, the notification escalation system can identify in step 220 that a user is near a bystander who has a mobile phone, is wearing headphones, and is typing on a notebook computer.In this example, the notification escalation system can generate a set of notification options, including at least three options, such as: (1) transmitting a visual alert to the user's notebook computer; (2) transmitting an audible alert that can be received through the user's headphones (e.g., transmitting an audible alert to a mobile device belonging to the user, which may be connected to the user's headphones for data transmission); and (3) warning bystanders by transmitting a message to their mobile phones. In some embodiments, the notification escalation system can be configured to prioritize the generated set of notification options.Continuing the example above, the notification escalation system can prioritize notification option (1) and / or notification option (2) over notification option (3) so that bystanders are not disturbed unless the user acknowledges a notification through notification options (1) and / or (2).

[0035] In some embodiments, step 225 may include the notification escalation system selecting an awareness time threshold for one or more notification options, at least partially based on notification data received in step 210. For example, continuing the example above, notification data received by the notification escalation system may include the message, “The board meeting will begin in 10 minutes. Please contact reception immediately.” Continuing the example above, the notification escalation system may use natural language processing technology to identify elements such as the 10-minute time period and the request to contact reception immediately.Based on these elements of the notification data, which indicate a level of urgency, the notification escalation system can select a threshold, such as a 5-minute awareness time threshold. Accordingly, by making such a selection, the notification escalation system can specify a maximum time of 5 minutes to utilize the set of three notification options and obtain awareness data from the user. "Awareness data" (which will be discussed further below) can refer to information indicating that a user has received a notification and / or is performing an activity equivalent to receiving a notification. In some implementations, an awareness time threshold can be selected by an entity such as a programmer or a user of the notification escalation system.

[0036] In step 230, the notification escalation system can select at least one notification option. In some embodiments, such a selection can be based at least partially on a user status, a priority associated with a notification option, a previously selected notification option, and / or an awareness time threshold. In some embodiments, step 230 can include the notification escalation system selecting a notification option from the set of notification options generated in step 225. For example, in some embodiments, the notification escalation system can generate a set of three notification options in step 225, with the first notification option having the highest priority and the third notification option having the lowest priority.Furthermore, the set of three notification options in this example can include a 10-minute awareness time threshold (e.g., the notification escalation system can specify a maximum time of 10 minutes to utilize the set of three notification options and obtain awareness data from the user). In this example, the notification escalation system can, at a first point in time, select the first notification option based on its highest priority. Continuing this example, at a second point in time, the notification escalation system can select the second notification option in response to a finding that the user has not responded to a notification via the first notification option.Continuing this example, at a third point in time, the notification escalation system can select all three notification options in the set of notification options in response to a finding that the remaining time of the 10-minute awareness time threshold is shorter than a minimum remaining time (e.g., three minutes of the 10-minute awareness time threshold remain, which is shorter than a 5-minute minimum remaining time threshold). Accordingly, embodiments of the present disclosure can increase the probability that a user will acknowledge a notification.

[0037] In step 235, the notification escalation system can trigger at least one notification. In some embodiments, the notification escalation system can perform step 235 in response to the selection of at least one notification option in step 230. Triggering a notification can involve the notification system issuing a command to at least one user unit (e.g., user unit 125). Fig. 1) and / or at least one notification unit (e.g., notification unit 130, Fig. 1) enables an action to be performed according to a notification option. For example, in some embodiments, step 235 may include the notification escalation system issuing a command, according to a notification option, that enables a user's smartwatch to vibrate. In this example, such a vibration could prompt a user who has not responded to phone calls to view a message on the user's smartwatch. In another example, in some embodiments, step 235 may include the notification escalation system issuing a command, according to a notification option, that enables a notebook computer to unmistakably display a text message on its screen.In one such example, a visual indicator can prompt a user who has not responded to a text message to receive the text message on the user's notebook computer. In another example, in some embodiments, step 235 can include the notification escalation system issuing a command, according to a notification option, that enables a loudspeaker system in the user's office building to emit an audible message to the user. In yet another example, in some embodiments, step 235 can include the notification escalation system issuing the commands discussed in the preceding examples simultaneously, according to a notification option.

[0038] In step 240, the notification escalation system can receive image data in response to a notification being triggered in step 235. Such image data can be obtained in a manner substantially similar to that described in step 215. In some embodiments, step 240 can include the notification escalation system receiving sets of one or more images over multiple time periods. For example, step 215 can include the notification escalation system receiving a first set of images of an environment in which a user is located between 9:00 and 9:05, and a second set of images of the same environment between 9:15 and 9:20.In this way, the notification escalation system can determine, based on the user's actions captured in the images, whether a user is taking note of an initial or a subsequent notification.

[0039] In step 245, the notification escalation system can determine whether a user acknowledges a notification. In some embodiments, the notification escalation system can determine that a user acknowledges a notification if it receives acknowledgement data. In these embodiments, the notification escalation system can determine that a user does not acknowledge a notification if it does not receive any acknowledgement data. "Acknowledgement data" can refer to information indicating that a user has received a notification, is performing an activity that corresponds to a notification, or has completed an activity that corresponds to a notification.For example, in some embodiments, acknowledgement data may include an acknowledgment message from a user unit confirming that a user has read a notification message. In some embodiments, a user may choose to transmit such an acknowledgment message via the user unit to the notification escalation system.

[0040] In some embodiments, awareness data may include a determination by the notification escalation system that a user is performing / has completed an activity corresponding to a notification. In these embodiments, such a determination may be based on an analysis by the notification escalation system of image data received in step 240. For example, in some embodiments, such image data may show a user stopping, taking out their mobile phone, and looking at the illuminated screen of the mobile phone. In this example, the image data may indicate to the notification escalation system that the user is reading a received text message notification. Thus, in this example, the notification escalation system may determine that the user is performing an activity corresponding to a notification.In another example, the image data in some embodiments may show a bystander approaching a user, and the user speaking to the bystander. In this example, the image data may indicate to the notification escalation system that the bystander is delivering a notification message to the user and / or prompting the user to retrieve a notification. Thus, in this example, the notification escalation system can determine that the user is performing an activity corresponding to a notification. In some embodiments, the image data may show that a user is at a destination specified in a notification message. In these embodiments, the notification escalation system can determine that the user has completed an activity corresponding to the notification message.

[0041] In some embodiments, acknowledgement data can be selected from the group consisting of (1) an acknowledgment message from a user unit and (2) image data that indicates to the notification escalation system that a user has completed an activity equivalent to a notification. Accordingly, in these embodiments, the notification escalation system may determine that it is not receiving any acknowledgement data unless it receives at least one of these two types of data. For example, the notification escalation system may trigger a notification requesting a user to go to room A. In this example, the notification escalation system may consider a received acknowledgment message as acknowledgement data indicating that the user has acknowledged the notification.Alternatively, in this example, the notification escalation system may receive a subsequent image of the user standing in room A, and it may consider this image as acknowledgement data indicating that the user has taken note of the notification. Alternatively, in this example, the notification escalation system may receive subsequent images of the user moving towards room A or not responding to the notification, and it may not consider such images as acknowledgement data. In these cases, the notification escalation system may determine that the user has not taken note of the notification.

[0042] In some embodiments, the notification escalation system may consider only image data as acknowledgement data, indicating that a user has completed an activity equivalent to a notification. For example, in a case where the notification escalation system triggers a notification requesting a user to go to room A, a confirmation message from the user to the notification escalation system would not constitute acknowledgement data. However, receiving a subsequent image of the user standing in room A would constitute acknowledgement data for the notification escalation system. Accordingly, the notification escalation system may be configured to continue process 200 until it receives an indication that a user has completed an activity equivalent to a notification.

[0043] In step 245, the notification escalation system can proceed to step 250 if it determines that a user has not acknowledged a notification. Alternatively, procedure 200 can end (e.g., the notification escalation system can refrain from triggering any subsequent notifications and terminate procedure 200) if it determines that a user has acknowledged a notification. Alternatively, procedure 200 can end if it determines that an acknowledgement time threshold has been exceeded.

[0044] In step 250, the notification escalation system can generate and use a timing model for triggering one or more subsequent notifications. A “timing model” can refer to a process for realizing a step time (e.g., a time interval between successive notifications triggered by the notification escalation system). In some embodiments, step 250 can include the notification escalation system calculating such a step time. In some embodiments, such a calculation can involve the notification escalation system dividing an awareness time threshold by a number of notification options contained in a set of notification options.

[0045] In some embodiments, a time-tap model generated in step 250 may include one or more of the following. If the image data obtained in step 240 indicates that a user is not performing any activity corresponding to a notification triggered in step 235, the notification escalation system may proceed to step 215 in response to the expiration of one step time. If the image data obtained in step 240 indicates that a user is performing an activity corresponding to a notification triggered in step 235 (e.g., the user is moving toward a destination specified in a notification message), the notification escalation system may proceed to step 215 in response to the expiration of two step times.By using such a time-based model, embodiments of the present disclosure can grant a user who appears to be responding to a notification additional time to take note of the notification. In this way, embodiments of the present disclosure can reduce the number of triggered notifications, which can decrease the likelihood of a user being burdened by excessive notifications.

[0046] Fig. Figure 3 represents an exemplary environment 300 in which a notification escalation system according to embodiments of the present disclosure can be used. (The in Fig. (Distances and object sizes shown in Figure 3 are not to scale.) The environment 300 contains a plurality of moving vehicles 325 at a road junction 305. The environment 300 also contains public surveillance cameras 315 and a pedestrian crossing loudspeaker 320. The public surveillance cameras 315 and the pedestrian crossing loudspeaker 320 can be functionally connected to an independent public safety system (not shown) configured to warn pedestrians in the environment 300 of potential traffic hazards. The pedestrian crossing loudspeaker 320 can be configured to emit audible messages, such as messages regarding the time at which pedestrians can safely cross the road.

[0047] The environment 300 comprises a user 330 and a bystander 345 on a sidewalk 310. In this example, the independent public safety system can, by using methods outside the scope of this disclosure, predict that the user is moving dangerously toward the intersection 305 and that the warning message “You are approaching an intersection; watch for approaching traffic.” should be issued. To manage the output of such a message, the independent public safety system can use a notification escalation system (not shown) according to embodiments of this disclosure.

[0048] The notification escalation system can receive the warning message and image data from cameras 315. Based on image data received at an initial time, the notification escalation system can identify that user 330 is carrying a mobile phone 335 and wearing headphones 340. In response, the notification escalation system can trigger a text notification to the user's mobile phone 335 containing a warning message. Additionally, the notification escalation system can trigger a haptic alert on the user's mobile phone 335 to prompt user 330 to view the text notification. Based on image data received at a second time, the notification escalation system can determine that user 330 has not acknowledged the notification.Based on image data received at a third time, the notification escalation system can identify that user 330 has moved to position 355. Furthermore, based on such image data, the notification escalation system can identify that user 330 is near bystander 345, who is carrying a mobile phone 350, and the pedestrian crossing loudspeaker 320. In response, the notification escalation system can trigger the notifications discussed above, as well as (1) an audible notification emitted from the pedestrian crossing loudspeaker 320 containing the warning message, and (2) a text notification to bystander 345's mobile phone, requiring bystander 345 to alert user 330.Accordingly, embodiments of the present disclosure may provide a more extensive set of notification options for notifying a user.

[0049] Fig. Figure 4 presents characteristic key components of an exemplary computer system 401, which can be used according to embodiments of the present disclosure. The specific components shown are presented only for illustrative purposes and are not necessarily the only such variations. The computer system 401 may include a processor 410, a memory 420, an input / output interface (also referred to herein as I / O or I / O interface) 430, and a main bus 440. The main bus 440 may provide data transmission paths for the other components of the computer system 401. In some embodiments, the main bus 440 may be connected to other components, such as a special digital signal processor (not shown).

[0050] The processor 410 of the computer system 401 can consist of one or more CPUs 412. The processor 410 can also consist of one or more memory buffers or caches (not shown) that provide temporary storage of instructions and data for the CPU 412. The CPU 412 can execute instructions on inputs provided from the caches or from memory 420 and output the result to the caches or memory 420. The CPU 412 can consist of one or more circuits configured to perform one or more methods according to embodiments of this disclosure. In some embodiments, the computer system 401 can contain multiple processors 410, which is typical for a relatively large system. In other embodiments, the computer system 401 can consist of only one processor with only one CPU 412.

[0051] The memory 420 of the computer system 401 can consist of a memory control unit 422 and one or more memory modules for temporary or permanent storage of data (not shown). In some embodiments, the memory 420 can comprise a direct-access semiconductor memory, a storage unit, or a storage medium (either volatile or non-volatile) for storing data and programs. The memory control unit 422 can exchange data with the processor 410, thereby facilitating the storage and retrieval of data in the memory modules. The memory control unit 422 can exchange data with the I / O interface 430, thereby facilitating the storage and retrieval of inputs and outputs in the memory modules. In some embodiments, the memory modules can be dual inline memory modules.

[0052] The I / O interface 430 can include an I / O bus 450, an end-device interface 452, a memory interface 454, an I / O unit interface 456, and a network interface 458. The I / O interface 430 can connect the main bus 440 to the I / O bus 450. The I / O interface 430 can route instructions and data from the processor 410 and the memory 420 to the various interfaces of the I / O bus 450. The I / O interface 430 can also route instructions and data from the various interfaces of the I / O bus 450 to the processor 410 and the memory 420. The various interfaces can include the end-device interface 452, the memory interface 454, the I / O unit interface 456, and the network interface 458. In some embodiments, the various interfaces may comprise a subset of the interfaces mentioned above (e.g.An embedded computer system in an industrial application may not include the terminal interface 452 and the storage interface 454).

[0053] Logic modules throughout the computer system 401—including, but not limited to, the memory 420, the processor 410, and the I / O interface 430—can transmit failures and changes in one or more of these components to a hypervisor or an operating system (not shown). The hypervisor or operating system can allocate the various resources available in the computer system 401 and track the memory locations of data in the memory 420 and of processes assigned to different CPUs 412. In embodiments where elements are combined or rearranged, aspects of the logic module capabilities can be combined or redistributed. These variations are obvious to a person skilled in the art.

[0054] It is understood from the outset that the implementation of the teachings presented herein is not limited to a cloud computing environment, although this disclosure contains a detailed description of cloud computing. Rather, embodiments of the present invention can be implemented in conjunction with any other type of data processing environment currently known or developed in the future.

[0055] Cloud computing is a service delivery model that provides convenient and on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, main memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management overhead or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0056] The properties are as follows: On-demand Self Service: A cloud customer can unilaterally and automatically provision data processing functions such as server time and network storage as needed, without requiring any human interaction with the service provider. Broad Network Access: Functions are available over a network, accessed via standard mechanisms that support the use of heterogeneous thin or thick source platforms (e.g., mobile phones, notebook computers, and PDAs). Resource pooling: The provider's data processing resources are pooled to serve multiple customers using a multi-user model with diverse physical and virtual resources that are dynamically allocated and reassigned according to demand. There is a perceived location independence in that the customer generally has no control over or knowledge of the exact location of the provided resources, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity: Features can be deployed quickly and elastically, in some cases automatically, to rapidly scale up functionality, and released quickly to rapidly scale down functionality. This often gives customers the impression that the available features are unlimited and can be purchased in any quantity at any time. Measured Service: Cloud systems automatically control and optimize resource usage by employing a metering function at a specific level of abstraction appropriate for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the service provider and the customer.

[0057] The service models are as follows: Software as a Service (SaaS): The functionality provided to the customer consists of using the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices via a thin-client interface, such as a web browser (e.g., web-based email). The customer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage space, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The function provided to the customer is to deploy customer-created or purchased applications on the cloud infrastructure, using programming languages ​​and tools supported by the provider. The customer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, and storage space, but retains control over the deployed applications and potentially over the configurations of the applications' hosting environment. Infrastructure as a Service (IaaS): The functionality provided to the customer consists of supplying processing, storage, networking, and other basic data processing resources, allowing the customer to deploy and run any software, including operating systems and applications. The customer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0058] The deployment models are as follows: Private Cloud: The cloud infrastructure is operated exclusively for one organization. It can be managed by the organization or a third party and can be located on or off-site. Community Cloud: This cloud infrastructure is used by multiple organizations and supports a specific user community with shared interests (e.g., aspects related to a task, security requirements, policies, and compliance with laws and regulations). It can be managed by the organization or a third party and can be located on or off-site. Public Cloud: The cloud infrastructure is made available to the general public or a large group within an industry and is owned by an organization that sells cloud services. Hybrid cloud: The cloud infrastructure is a mixture of two or more clouds (private cloud, community cloud or public cloud) that remain independent units but are connected via a standardized or proprietary technology that enables the portability of data and applications (e.g. cloud bursting for load balancing between clouds).

[0059] A cloud computing environment is service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprised of a network of interconnected nodes.

[0060] With reference to Fig. Figure 5 shows an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 has one or more cloud computing nodes 10 with which local data processing units used by cloud customers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a notebook computer 54C, and / or an automotive computer system 54N, can exchange data. The nodes 10 can exchange data with each other. They can be grouped physically or virtually in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud as described above, or in a combination thereof (not shown). This enables the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services, for which a cloud customer does not need to manage resources on a local data processing unit.It goes without saying that the types of in . Fig. The data processing units 54A to N shown are for illustrative purposes only, and the data processing nodes 10 and the cloud computing environment 50 can exchange data with any type of computer unit via any type of network and / or any type of connection that can be accessed via a network (e.g. using a web browser).

[0061] With reference to Fig. Figure 6 shows a set of functional abstraction layers that are used in the cloud computing environment 50 ( Fig. 5) be provided. It should be clear from the outset that the in Fig. The components, layers, and functions shown in Figure 6 are intended to be illustrative only, and embodiments of the invention are not limited to them. As shown, the following layers and corresponding functions are provided:

[0062] A hardware and software layer 60 comprises hardware and software components. Examples of hardware components include: Mainframe computers 61; servers 62 based on the RISC architecture (RISC = Reduced Instruction Set Computer); servers 63; blade servers 64; storage units 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0063] A virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71; virtual storage 72; virtual networks 73 including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0064] In one example, an administration layer 80 can provide the functions described below. A resource provisioning layer 81 provides dynamic procurement of data processing resources and other resources used to perform tasks within the cloud computing environment. A charge tracking and pricing layer 82 provides cost tracking while resources are used within the cloud computing environment, as well as billing and invoicing for the consumption of these resources. In one example, these resources could include application software licenses. A security layer provides identity verification for cloud customers and tasks, as well as protection for data and other resources. A user portal 83 provides customers and system administrators with access to the cloud computing environment.Service Level Management (SLM) 84 provides the allocation and management of cloud computing resources to ensure that the required level of service is achieved. Service Level Agreement (SLA) planning and fulfillment 85 provides the advance planning and procurement of cloud computing resources for which future requirements are anticipated based on an SLA.

[0065] An operational load layer 90 provides examples of functionalities for which the cloud computing environment can be used. Examples of operational loads and functions that can be provided from this layer include: mapping and navigation 91; software development and management 92 throughout the lifecycle; delivery 93 of training in virtual classrooms; processing 94 of data analytics; transaction processing 95; and notification escalation logic 96.

[0066] As discussed in more detail herein, it is considered that some or all of the steps of some embodiments of the methods described herein may be carried out in alternative sequences or may not be carried out at all; furthermore, several steps may take place simultaneously or as an internal part of a larger process.

[0067] The present invention may comprise a system, a method, and / or a computer program product with any possible level of integration of technical details. The computer program product may include a computer-readable storage medium (or media) on which computer-readable program instructions are stored to cause a processor to execute aspects of the present invention.

[0068] The computer-readable storage medium can be a physical unit on which instructions for use by a unit for executing instructions can be stored and retained. The computer-readable storage medium can be, for example, but is not limited to, an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof.A non-exhaustive list of more precise examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable read-only memory in the form of a compact disc (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically coded unit such as punched cards or raised structures in a groove with instructions recorded on them, or any suitable combination of the foregoing.A computer-readable storage medium, as used herein, is not to be interpreted as consisting of volatile signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through an optical fiber cable), or electrical signals transmitted via a cable.

[0069] The computer-readable program instructions described herein can be downloaded over a network, such as the internet, a local area network, a wide area network, and / or a wireless network, from a computer-readable storage medium to relevant data processing units or to an external computer or external storage device. The network may include copper transmission cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface at each data processing unit receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the respective data processing unit.

[0070] Computer-readable program instructions for executing work steps of the present invention may be assembly instructions, ISA instructions (ISA = Instruction-Set-Architecture), machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++ or the like, as well as procedural programming languages ​​such as the programming language "C" or similar programming languages.The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be to an external computer (for example, via the internet using an internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), which execute computer-readable program instructions, can be used to personalize the electronic circuit by utilizing state information from the computer-readable program instructions, thus implementing aspects of the present invention.

[0071] Aspects of the present invention are described herein with reference to flowchart representations and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It will be clear that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0072] These computer program instructions can be provided to a processor of a computer or other programmable data processing device to create a machine such that the instructions executed by the processor of the computer or other programmable data processing device provide the means to accomplish the functions / actions specified in a block or blocks of the flowchart(s) and / or block diagram(s). These computer-readable program instructions can also be stored on a computer-readable medium that can instruct a computer, other programmable data processing device, or other entity to operate in a particular manner, such that the computer-readable medium with instructions stored on it constitutes a product containing instructions that accomplish the functions / actions specified in a block or block diagram(s).Implement the function / action specified in the blocks of the flowcharts and / or block diagrams.

[0073] The computer-readable program instructions can also be loaded into a computer, other programmable data processing devices, or other units to cause a series of operations to be performed on the computer, other programmable devices, or other units to create a computer-implemented process, such that the instructions executed on the computer, other programmable devices, or units realize the functions / actions specified in a block or blocks of the flowcharts and / or block diagrams.

[0074] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, processes, and computer program products according to various embodiments of the present invention. In this context, each block in the flowcharts or block diagrams can represent a control component, a segment, or a section of instructions that includes one or more executable instructions for implementing the specified logic function(s). In some alternative embodiments, the functions specified in the block may occur in a different order than shown in the figures.For example, two blocks shown consecutively can actually be executed as a single step, simultaneously, essentially concurrently, in a partially or fully overlapping manner, or the blocks can sometimes be executed in reverse order, depending on the functionality associated with them. Furthermore, it should be noted that each block in the block diagrams and / or flowcharts shown, as well as combinations of blocks in the block diagrams and / or flowcharts shown, can be implemented using dedicated hardware systems to perform the specified functions or actions, or using combinations of dedicated hardware and dedicated computer instructions.

[0075] The descriptions of the various embodiments of this disclosure are intended for illustrative purposes only and are not meant to be exhaustive or limited to the disclosed embodiments. Many modifications and variations are conceivable for a person skilled in the art without these representing a deviation from the scope of protection and the fundamental principles of the described embodiments. The terminology used herein has been chosen to explain the fundamental principles of the embodiments, their practical application, and the technical improvements compared to technologies found on the market, or to enable others with the relevant expertise to understand the embodiments disclosed herein.

Claims

[1] A computer-implemented method comprising: Obtaining notification data for a user; Obtaining a first set of images of the user from one or more cameras; Identifying, based on the first set of images, a first status of the user; generating, based at least in part on the user's first status, a first set of notification options; Selecting a first notification option of the set of notification options; triggering, in response to selecting, a first notification through the first notification option; Determining, at an initial point in time, that the user does not acknowledge the first notification; Obtaining, in response to the detecting, a second set of images of the user from one or more cameras; Identifying, based on the first set of images, a first status of the user; Selecting at least a second notification option from the set of notification options; and Triggering, in response to selecting the at least one second notification option, at least one second notification by the at least one second notification option. [2] The computer-implemented method of claim 1, wherein the one or more cameras are surveillance cameras for an environment in which the user is located. [3] The computer-implemented method of claim 1, further comprising calculating a step time, and wherein the step time is a time interval between triggering the first notification and triggering the at least one second notification. [4] The computer-implemented method of claim 3, wherein the notification data comprises an acknowledgement time threshold; and wherein calculating the step time comprises dividing the acknowledgement time threshold by a number of notification options included in the set of notification options. [5] The computer-implemented method of claim 1, further comprising determining, at a second time after the first time, that the user acknowledges the at least one second notification; and refraining from triggering a third notification in response to the determining at the second time. [6] The computer-implemented method of claim 5, wherein the notification data comprises a destination for the user; and wherein determining that the user acknowledges the at least one second notification comprises determining, based on a third set of images of the user obtained from the one or more cameras, that the user is moving toward the destination. [7] The computer-implemented method of claim 1, wherein the set of notification options comprises triggering a notification for a notification entity of a bystander, the bystander being included in the second set of images. [8] System that has: a processor; and a memory in communication with the processor, the memory containing program instructions configured to, when executed by the processor, cause the processor to perform the method for generating a summary text structure according to any one of claims 1 to 7. [9] A computer program product comprising a computer-readable storage medium having program instructions embodied thereon, the computer-readable storage medium not being a transient signal per se, the program instructions being executable by a processor to cause the processor to perform the method for generating a summary text structure according to any one of claims 1 to 7.

Images