Fixing a failed user interface

Abstract

An aspect for resolving failed user interface functionality is described, wherein one or more processors are configured to: detect a first user input having a first input magnitude at a user-selectable element displayed on a local device user interface and determine that the local device user interface is unresponsive to the first user input; detect a second user input at the user-selectable element having a second input magnitude greater than the first input magnitude, where the first user input and the second user input are by a user; determine that the local device user interface is unresponsive to the second user input; send first data corresponding to a user interface offload prompt to a mobile device associated with the user, where the mobile device is within a predetermined proximity from the local device user interface; and receive second data from the mobile device corresponding to the plurality of user inputs of the mobile device.

Description

[Technical field]

[0001] The present invention relates generally to the field of resolving user interface functionality, and more particularly to offloading failed user interface functionality to a nearby mobile device. [Background technology]

[0002] Fast food restaurants have evolved from traditionally taking customer orders over the counter with human interaction to taking customer orders electronically through Internet-enabled computing devices. When a customer enters a restaurant, he or she simply approaches a self-ordering kiosk, selects an item from the menu, enters payment information, and receives an order confirmation. Ordering food or drinks has simply been shifted from interaction with a restaurant employee to a self-service kiosk, replacing human interaction with technology.

[0003] Many mobile device user-interface displays are switching to touch screen displays for self-service kiosks as part of the emergence of Internet of Things (IoT) capabilities. IoT refers to the concept of Internet connectivity extending beyond traditional computing platforms such as personal computers and mobile devices to any range of traditionally non-Internet-enabled physical devices and everyday objects. These devices and objects, which incorporate electronics, Internet connectivity, and other forms of hardware (such as sensors), can communicate and interact with customers via network connectivity, and these devices and objects can be monitored and controlled remotely. Summary of the Invention

[0004] Aspects of the present invention disclose methods, computer program products, and computer systems for resolving failed user interface functions. One embodiment of the present invention may include a local device having one or more processors configured to detect a first user input having a first input magnitude at one or more user selectable elements displayed in a local device user-interface. Further, the one or more processors may be configured to determine that the local device user interface is unresponsive to the first user input. The one or more processors of the local device may be configured to detect a second user input by a user at one or more user selectable elements, the second user input having a second input magnitude greater than the first input magnitude.

[0005] Further, the one or more processors of the local device may be configured to determine that the local device user interface is unresponsive to a second user input. The one or more processors may be configured to determine the unresponsiveness by detecting user annoyance as multiple repeated screen touches, user frustration heard via a microphone, or canceling a user interaction by touching a cancel user selectable element on the local device user interface. If the one or more processors of the local device determine that the local device user interface is unresponsive, the one or more processors of the local device may transmit first data to a mobile device associated with the user, the first data corresponding to a user-interface offload prompt, and the mobile device may be within a predefined proximity range from the local device user interface. Further, the one or more processors of the local device may be configured to receive second data from the mobile device, the second data corresponding to multiple mobile device user inputs received at the mobile device.

[0006] In another embodiment, the one or more processors may be configured to determine that the local device user interface is not responsive to the first user input because the one or more processors failed to execute a command corresponding to the one or more user selectable elements initiated by the first user input. Additionally, a mobile device may include one or more processors that may be configured to execute embodiments of the invention described herein. For example, embodiments of the invention may include one or more processors of a mobile device configured to generate a mobile device user interface on a display of the mobile device, the mobile device user interface may include a mobile device user selectable element corresponding to the one or more user selectable elements.

[0007] In response to determining that the local device user interface is not responsive to the second user input, a user interface offload prompt may be sent to the mobile device. At least one of the plurality of user inputs of the mobile device may correspond to the first user input at the one or more user selectable elements. The one or more processors of the mobile device may be configured to send the second data to a local device comprising the local device user interface, the local device being connected to the network. The local device may include a wireless communication module configured to facilitate wireless communication between the local device and the mobile device when within a predefined proximity range from the local device.

[0008] In another embodiment, the one or more processors of the local device may be configured to monitor user interaction. The user interaction may be monitored by receiving data from a sensor configured to detect user movement, user input, or user activity. Additionally, the one or more sensors may be configured to track indicators received from monitoring the user interaction. Examples of indicators that may be tracked are sensor data (e.g., pressure, input, voice) or some indication of cancellation of input.

[0009] In an embodiment, the one or more processors of the local device may be configured to process and analyze the indicators or indicator data to determine whether the indicators have any correlation with user annoyance. If the analysis produces results indicative of user annoyance, the one or more processors of the local device may be configured to offload functionality of the local device user interface to a proximate mobile device for display on the mobile device user interface and user interaction. In other words, the one or more processors of the mobile device may be configured to mirror functionality of the local device user interface on the mobile device user interface via wireless data transmission from the local device to the mobile device. However, if the analysis produces results not indicative of user annoyance, the one or more processors of the local device may be configured to continue to process and analyze the indicators or indicator data. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of a distributed data processing environment in accordance with an embodiment of the present invention. [Diagram 2] 1 is a flowchart of a method for resolving a failed user interface function according to an embodiment of the present invention. [Diagram 3]11 is a flowchart of another method for resolving a failed user interface function according to an embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram of a computing device in an IoT computing environment according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Touch screen self-service kiosks suffer from the same imperfections as traditional computers: For example, the touch screen user interface may not register user input or may fail to execute software commands requested by the user. If the self-service kiosk is designed to reduce customer skill, customer frustration and annoyance are inevitable when the self-service kiosk malfunctions.

[0012] IoT devices may be configured to detect user input on a user interface display to trigger or invoke their programmed functions. In customer-facing environments (e.g., fast food restaurants, airport check-in stations, retail checkouts), these functions may be facilitated by self-service kiosks, but there appears to be no robust solution for such implementation upon failure of the user interface display. The self-service kiosks may be in network communication with back-end server systems or knowledge engines to trigger or implement functions on network-connected devices (e.g., store point-of-sale devices, applications on mobile devices, kitchen display systems, payment processing). Customer-initiated transactions may be placed from a network-connected user device and triggered by specific user input, sometimes when the user is within close enough range of the self-service kiosk. However, the user interface of the self-service kiosk may fail to receive user input, resulting in user annoyance and frustration when the self-service system is designed to reduce wait times and inconvenience. In other words, if the self-service kiosk's user interface is unable to execute the commands associated with the user input, the customer's order may not be received.

[0013] Aspects of the invention described herein facilitate and enable a self-service kiosk's user interface to detect user input, determine that the user interface is not responding as intended, and offload the user interface to a nearby user's mobile device to allow the user to continue ordering via the user's mobile device. Certain aspects of embodiments of the invention facilitate resolving a failed touch screen by offloading the user interface to the user's mobile device.

[0014] Example embodiments described herein provide methods, systems, and computer program products for resolving failed user interface functionality. One embodiment of the present invention may include a local device having one or more processors configured to detect a first user input having a first input magnitude at one or more user selectable elements displayed in a local device user interface. Further, the one or more processors of the local device may be configured to determine that the local device user interface is unresponsive to the first user input. The one or more processors of the local device may be configured to detect a second user input by a user at one or more user selectable elements, the second user input having a second input magnitude greater than the first input magnitude.

[0015] Further, the one or more processors of the local device may be configured to determine that the local device user interface is unresponsive to a second user input. The one or more processors may determine the unresponsiveness by detecting user annoyance as multiple repeated screen touches, user frustration heard via a microphone, or canceling a user interaction by touching a cancel user selectable element on the local device user interface. If the local device determines that the local device user interface is unresponsive, the local device may be configured to transmit first data to a mobile device associated with the user, the first data corresponding to a user interface offload prompt. The local device may be configured to determine that the mobile device is within a predefined proximity range from the local device user interface prior to transmitting the data to the mobile device. Further, the one or more processors of the local device may be configured to receive second data from the mobile device, the second data corresponding to multiple mobile device user inputs received at a user interface on the mobile device.

[0016] In another embodiment, the one or more processors of the local device may be configured to determine that the local device user interface is not responsive to the first user input because the one or more processors of the local device failed to execute a command corresponding to the one or more user selectable elements initiated by the first user input. Further, the one or more processors of the mobile device may be configured to generate a mobile device user interface on a display of the mobile device, the mobile device user interface may include a mobile device user selectable element corresponding to the one or more user selectable elements.

[0017] In response to determining that the local device user interface is unresponsive to the second user input, a user interface offload prompt may be sent from the local device to the mobile device. For example, after the user interface offload prompt is received at the mobile device, the mobile device may be configured to generate a notification to the mobile device user indicating the unresponsiveness of the local device. In addition to the notification, the mobile device may be configured to generate a prompt to the user, which, when selected on a display of the mobile device, generates a user interface. The mobile device user interface may be configured to receive a plurality of mobile device user inputs at the mobile device. At least one of the plurality of mobile device user inputs may correspond to a first user input at one or more user selectable elements. In other words, the mobile device user input may be the same as a first user input attempted by the user at the local device user interface, but provided at the mobile device user interface because the local device user interface was unresponsive.

[0018] The one or more processors of the mobile device may be configured to transmit the plurality of user inputs of the mobile device as second data to a local device having a local device user interface. The local device may be connected to a network in a facility in which the local device is installed. The local device may include a wireless communication module configured to facilitate wireless communication between the local device and the mobile device when within a predefined proximity range of the local device.

[0019] In another embodiment, the local device may be configured to monitor user interaction by receiving data from a sensor configured to detect user movement, user input, or user activity. Additionally, one or more sensors may be configured to track indicators received from monitoring the user interaction. Examples of indicators that may be tracked are sensor data (e.g., pressure, input, voice) or some indication of input cancellation.

[0020] In an embodiment, the one or more processors of the local device may be configured to process and analyze the indicators or indicator data to determine whether the indicators have any correlation with user annoyance. If the analysis produces results indicative of user annoyance, the one or more processors may be configured to offload functionality of the local device user interface to a proximate mobile device for display on the mobile device user interface and user interaction. In other words, the one or more processors may be configured to mirror functionality of the local device user interface on the mobile device user interface via wireless data transmission from the local device to the mobile device. However, if the analysis produces results not indicative of user annoyance, the one or more processors may be configured to continue to process and analyze the indicators or indicator data.

[0021] The present invention will now be described in detail with reference to the figures.

[0022] 1 illustrates a block diagram of a distributed data processing environment 100 for resolving failed user interface functionality in accordance with an embodiment of the present invention. FIG. 1 is merely intended to provide an example of an embodiment of the present invention and is not intended to imply any limitation with regard to the environments in which different embodiments may be implemented.

[0023] In the illustrated embodiment, the distributed data processing environment 100 includes local devices 120, a database 124, and mobile devices 130 interconnected via a network 110. The network 110 may be, for example, a local area network (LAN), a wide area network (WAN), such as the Internet, or a combination of the two, and may include wired, wireless, or fiber optic connections. The network 110 may include one or more wired and / or wireless networks capable of receiving or transmitting data signals, voice signals, or video signals, including multimedia signals, including voice information, data information, and video information, or combinations thereof. In general, the network 110 may be any combination of connections and protocols that support communication between the local devices 120, the database 124, and the mobile devices 130. In an embodiment, the network 110 may be a private / secure network that may include a firewall to block unauthorized access to and from the network 110 while allowing outbound communication. Distributed data processing environment 100 may include additional servers, computers, or other devices not shown.

[0024] The network 110 may be in communication with a local device 120 having a user interface 122 that displays one or more user selectable elements 126. In one embodiment, the local device 120 may be configured to receive user input by a user 140 at the local device user interface 122. The user input may include detecting a touch action at a user selectable element 126 displayed at the local device user interface 122. The user input may have varying magnitudes that may be determined by measuring the pressure applied to the screen when a touch screen is pressed or touched. Thus, a first user input may have a first input magnitude and a second user input may have a second input magnitude that is less than, greater than, or equal to the first input magnitude.

[0025] Upon detection of user input at one or more user-selectable elements, one or more processors of the local device may determine that the local device user interface 122 is unresponsive to user input. If the local device 120 determines that the local device user interface 122 is unresponsive after the first input or after a subsequent user input, the one or more processors of the local device 120 may then determine whether the mobile device 130 is within a predefined proximity to the local device 120 such that the mobile device 130 is within range to successfully transmit data to the mobile device 130.

[0026] The one or more processors of the local device 120 may be configured to transmit data corresponding to a user interface offload prompt to the mobile device 130 associated with the user 140 after the one or more processors of the local device 120 determine that the mobile device 130 is within a predefined proximity to the local device 120. The data may be transmitted from the local device 120 to the mobile device 130 using a wireless communication medium.

[0027] In an embodiment, the mobile device 130 may be configured to receive data corresponding to the user interface offload prompt. One or more processors of the mobile device 130 may be configured to generate a mobile device user selectable element 136 to be displayed on the mobile device user interface 132 upon receipt of the data. The mobile device 130 may be configured to generate a mobile device user interface 132 that resembles the local device user interface 122. Optionally, the mobile device user interface 132 may mirror the local device user interface 122. However, so long as the mobile device user interface 132 displays a user interface that is the same as or similarly depicted as the user interface displayed on the local device user interface 122, the mobile device user interface 132 will have sufficient accuracy for the user 140 to continue to interact with the local device 120 even if the local device user interface 122 does not respond to user input.

[0028] The mobile device 130 may be configured to detect user input at a mobile device user selectable element 136 displayed on the mobile device user interface 132. The one or more processors of the mobile device 130 may be configured to process commands associated with the user selectable element 136. Additionally, the mobile device 130 may include memory (not shown) for storing data corresponding to the mobile device 130 user input and associated commands. The one or more processors may be configured to transmit data corresponding to the mobile device 130 user input and associated commands to the local device 120 over a wireless communication channel, as previously described.

[0029] In an embodiment, the local device 120 may be configured to receive data from the mobile device 130 corresponding to a number of user inputs and associated commands. Upon receipt of the data from the mobile device 130, the local device 120 may be configured to process the data and execute the commands as intended. For example, if the data from the mobile device 130 corresponds to a selection of an item displayed in a menu represented by the mobile device's user selectable elements 136, one or more processors of the local device 120 may be configured to execute the command associated with such item selection, process any payment information included in the data, and provide confirmation to the user 140 that the order was successfully received.

[0030] Local device 120 operates as a computing device with a local user interface 122 for interfacing with multiple users. In an embodiment, local device 120 operates as a self-service kiosk for customers of a business (e.g., a fast food restaurant, an airport check-in station, or a retail checkout, or a combination thereof). In some embodiments, local device 120 may be an administrative server, a web server, or any other electronic device or computing system capable of receiving and transmitting data, respectively. In some embodiments, local device 120 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a smartphone, or any programmable electronic device capable of communicating with other IoT devices via network 110. Local device 120 may include components as described in further detail in FIG. 4 .

[0031] The mobile device 130 may be operable to execute and display a user interface 132 for interaction with an owner and / or authorized user 140 of the mobile device 130. In an example embodiment, the mobile device 130 may be configured to transmit data to and / or receive data from the network 110 and / or the local device 120. In some embodiments, the mobile device 130 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a smartphone, a smart device, or any programmable electronic device capable of communicating with other computing devices over the network 110. As described in further detail in FIG. 4, the mobile device 130 may include components.

[0032] The user interface 132 operates on the mobile device 130 as a mobile user interface through which one or more users of the mobile device 130 interact with the mobile device 130. In some embodiments, the user interface 132 may be a graphical user interface (GUI), a web user interface (WUI), or a voice user interface (VUI), or a combination thereof, and may display (i.e., visually display) or present (i.e., audibly present) information to the user 140 from one or more processors on the mobile device 130 over the network 105, or allow the user 140 to input or receive information (i.e., graphics, text, or sound, or a combination thereof) to or from those processors, or a combination thereof. In an embodiment, the user interface 132 allows the user 140 to send and receive data (i.e., to and from one or more processors over the network 110).

[0033] In an embodiment, the user interface 132 displays, presents, or allows the user 140 to configure a workflow for the mobile device 130. In an embodiment, the user interface 132 displays, presents, or allows the user 140 to approve a workflow that has been created based on past usage patterns of the mobile device 130. In an embodiment, the user interface 132 displays, presents, or allows the user 140 to receive a notification from one or more processors of the mobile device, for example, displaying a user selectable element of an item in a menu. In an embodiment, the user interface 132 allows the user 140 to input information requested by the one or more processors of the mobile device, for example, receiving a user input corresponding to a selection of an item from a list.

[0034] The data may be stored in a database 124 in communication with the network 110. The database 124 may act as a repository for mobile device data, local device data, and other data that may be associated with the mobile device data or the local device data (e.g., device type profile, device profile). A database is a structured collection of data. The database 124 may be implemented using any type of storage device capable of storing data and configuration files that may be accessed and utilized by either the local device 120 or the mobile device 130, such as a database server, hard disk drive, or flash memory. In an embodiment, the database 124 may be accessed by the local device 120 to store the local device data, the mobile device data, the device type profile, the device profile. In another embodiment, the database 124 may be accessed by the mobile device 130 to access the mobile device data, the device type profile, the device profile. In the embodiment shown, the database 124 may exist independent of the network. In another embodiment, database 124 may reside anywhere within distributed data processing environment 100 provided database 124 is accessible to network 110 .

[0035] In another example embodiment, a user 140 may be allowed to agree to terms and service agreements when setting up a computing device with the network 110. The user agreement may document the purpose of information and data sharing between the local device 120 and the mobile device 130 and provide access privileges for devices on the network 110 designated for participation in the network 110. The user agreement may include all mentioned temporary devices that allow control, trigger, or action to be performed based on the user's original request. In the case of a network with multiple users and multiple computing devices, the system may extend the usage agreement to the aforementioned groups when a new user 140 joins a defined or dynamic group.

[0036] 2, which illustrates a flowchart of a method 200 for resolving a failed user interface function in accordance with an embodiment of the present invention. The method 200 may include a local device 120 including one or more processors configured to detect (202) a first user input by a user at one or more user selectable elements displayed on a local device user interface 122, the first user input having a first magnitude.

[0037] Further, the method 200 may include one or more processors configured to determine (204) that the local device user interface 122 is unresponsive to the first user input. In an embodiment, the one or more processors may be configured to determine that the local device user interface 122 is unresponsive to the first user input in a variety of ways. In an embodiment, the one or more processors may be further configured to determine that the one or more processors failed to execute a command corresponding to one or more user selectable elements 126 displayed on the local device user interface 122. For example, one pound per square inch (psi) (0.07 kilograms per square centimeter (kgf / cm 2 If a first user input having an input magnitude of 1 psi (1 kgf / cm 2 ) corresponds to a user touching a user selectable icon displaying the word "Start," then the command corresponding to that user input may be an executable function that is to be initiated or executed if selected. 2 ) may be sufficient for the touch screen to register the selection on the user interface 122. However, the local device user interface 122 may not properly register the user input as an input to initiate the corresponding command, and therefore the corresponding command may not be executed. There may be technical issues with the touch screen functionality of the local device user interface 122.

[0038] In addition, 2 psi (0.14 kgf / cm 2If a second user input having an input magnitude of 100% corresponds to user 140 touching the same user-selectable icon displaying the word “START,” the corresponding command may still fail to execute for a variety of reasons, including the examples discussed above. Regardless of the reason for the failure, local device 120 may be configured to determine that after a second user input having an input magnitude greater than the input magnitude of the first user input, local device user interface 122 is not responsive to the second user input.

[0039] The method 200 may include other ways for the one or more processors to determine that the local device user interface 122 is unresponsive to user input. For example, the one or more processors may be configured to determine unresponsiveness indicated by user frustration by analyzing an image of the user, the user's body language, or the user's facial expression. In an embodiment, one or more image sensors (e.g., cameras, infrared sensors) communicatively coupled to the local device 120 may be configured to capture an image of the user's face and perform an analysis on the user's facial features to detect an expression indicative of the user's emotion. For example, if the user's facial expression is a frown, the one or more processors may be configured to determine that the user 140 is sad or upset.

[0040] In another example, if the user's facial expression includes upturned inner eyebrows, slightly closed eyes, or downturned corners of the mouth, the one or more processors may be configured to determine that the user 140 is sad. In yet another example, if the user's facial expression includes downturned inner eyebrows, wide open eyes, lips pressed together, or lips open exposing teeth, the one or more processors may be configured to determine that the user 140 is angry. Other combinations of facial expressions may be determined to indicate anger, sadness, or frustration. Thus, in addition to detecting one or more user inputs at the local device user interface 122, if the one or more processors determine that the user 140 is dissatisfied based on the user's facial expression indicating that the user 140 is sad or angry, the one or more processors may determine that the local device user interface is unresponsive.

[0041] In another embodiment, one or more image sensors (e.g., cameras, infrared sensors) communicatively coupled to local device 120 may be configured to capture one or more images of the user's body language. For example, if the one or more processors analyze the images of the user's body and determine that the user's hands are clenched into fists, the one or more processors may be configured to determine that the user's body language indicates frustration. Other types of body language may be determined based on the images, as known by one of skill in the art.

[0042] Further, the method 200 may include one or more processors of the local device configured to detect (206) a second user input by the user at one or more user selectable elements 126, the second user input having a second magnitude that may be greater than, less than, or equal to the magnitude of the first input.

[0043] Further, the method 200 may include one or more processors configured to determine (208) that the local device user interface 122 is not responsive to the second user input, similar to previous embodiments herein.

[0044] The local device 120 may determine that the local device user interface 122 is unresponsive to user input in a variety of ways. In an embodiment, the local device 120 may detect a threshold number of failed user inputs and determine that this threshold number exceeds the number of attempts that may be required to execute a command. For example, if the local device user interface 122 detects two or more user inputs on the same local device user selectable element 126, the local device 120 may determine that the local device user interface 122 is unresponsive.

[0045] In another embodiment, if the local device user interface 122 detects two or more user inputs on any of the one or more user selectable elements 126 of the local device, and the magnitude of the inputs for each successive user input increases by a predefined threshold amount, the local device 120 may determine that the local device user interface 122 is unresponsive. For example, if the magnitude of the second input is greater than the magnitude of the first input by a predefined threshold of 25% of the magnitude of the first input, and the one or more processors are unable to execute the corresponding command, the one or more processors may determine that the local device user interface 122 is unresponsive to the local device user input. Other predefined threshold amounts may be used to determine unresponsiveness. The local device user interface 122 may include a pressure sensor configured to measure the pressure of each user input detected on the display. If the pressure sensor detects an increase in the amount of pressure or a pressure change that exceeds a predefined threshold, the local device 120 may determine that the local device user interface 122 is unresponsive.

[0046] Alternatively, in another embodiment, the local device 120 may include a voice sensor configured to detect the user's voice, and one or more processors may be configured to process and convert the user's voice into user voice data. The user voice data may be processed by a Natural Language Unit (NLU) engine or an Automatic Speech Recognition (ASR) engine or both to determine the user's intent. In this embodiment, the NLU or ASR or both may be configured to determine that the user voice data is indicative of user frustration or user annoyance corresponding to difficulty interacting with the local device user interface 122. Thus, if the one or more processors determine that the user 140 is frustrated or annoyed, the local device 120 may determine that the local device user interface is unresponsive.

[0047] In yet another embodiment, the local device user interface may include a local device user selectable element 126 that corresponds to a command to cancel an interaction with the local device user interface. The interaction cancel user selectable element may be configured to allow the user 140 to cancel an interaction even if the local device user interface is not responsive to user input.

[0048] The one or more processors may be located anywhere within the distributed data processing environment 100 such that data collected at any device connected to the network 110 may be routed to any of the one or more processors for processing, storage, or rerouting. For example, data collected at the local device 120 may be transmitted to the network 110 and stored in the database 124 for processing by some other processing resource available in the distributed data processing environment 100. Similarly, data collected at the mobile device 130 may be transmitted to the network 110 and stored in the database 124 for processing by some other processing resource available in the distributed data processing environment 100. Thus, the one or more processors are not limited to any particular device connected to the network 110. However, some embodiments may be described as processing a process occurring at a device that is best suited to execute the process due to factors of convenience.

[0049] Additionally, the method 200 may include one or more processors configured to transmit 210 local device data to a mobile device associated with the user. The local device data may be collected or generated at the local device and made available to other networked devices via the network 110. The local device data may be described herein as data generated by the local device or first data. The local device data may include data corresponding to a user interface offload prompt, which may include a notification presented to the user 140 that the local device is unresponsive and provides the user with an option to offload the local device user interface to a mobile device associated with the user. In an embodiment, the local device may be configured to determine whether the mobile device is within a predefined proximity to the local device.

[0050] The local device data may be configured to generate a notification on the mobile device when the mobile device receives and processes the local device data. For example, when the mobile device receives local device data or data from the local device, one or more processors of the mobile device may be configured to generate a notification on the mobile device to alert a user. The notification may be an audible notification, a visible notification, a vibrating notification, or some combination thereof to indicate that the local device user interface is not responsive to user input. For example, after the local device determines that the local device user interface is not responsive to user input, the local device may send the local device data including the notification to the mobile device, and when the mobile device processes the local device data, an audible notification, a vibrating notification, or a visible notification may be generated along with a message indicating the failure of the local device user interface.

[0051] The mobile device user interface may be generated on a display of the mobile device as a local software application executes the method 200 on the mobile device. The local device data may include a uniform resource locator (URL) or web address configured to generate a version of the local device user interface on the mobile device user interface. Alternatively, a quick response (QR) code may be displayed on the local device user interface for capture by a camera of the mobile device, and the QR code may provide access to the local device data on the mobile device.

[0052] The notification may include a visible prompt for the user 140 to select the user selectable element to offload the local device user interface to the mobile device user interface. In response to selecting the user selectable element to offload the local device user interface, the mobile device may be configured to execute a command to generate a mobile device user interface that resembles the local device user interface. Alternatively, the mobile device may be configured to execute a command to generate a mobile device user interface that does not resemble the local device user interface, but that includes the mobile device user selectable elements 136 that match the functionality of the local device user selectable elements 126. For example, the local device user selectable elements 126 may be arranged in a particular arrangement that is appropriate for the display characteristics (e.g., display size, resolution) of the display of the local device user interface, while the mobile device user selectable elements 136 may be arranged in a different arrangement than the local device user interface that is appropriate for the display characteristics (e.g., display size, resolution) of the mobile device user interface 132.

[0053] Further, the method 200 may include the local device receiving (212) mobile device data from the mobile device 130. The mobile device data may correspond to a plurality of mobile device user inputs entered by the user 140 at the mobile device user interface 132. The mobile device user interface 132 may resemble the local device user interface 122 such that the user 140 may interact with the mobile device 130 in the same manner as the user 140 interacts with the local device 120. The mobile device data may be described herein as data generated by the mobile device 120 or second data. The mobile device data may include data collected or generated at the mobile device 120.

[0054] 3, which illustrates a flowchart of a method 300 for resolving a failed user interface function in accordance with another embodiment of the present invention. The method 300 may include one or more processors in the local device 120 configured to detect (302) a first user input having a first input magnitude by a user at a local device user selectable element 126 displayed on the local device user interface 122. The method 300 may further include one or more processors configured to determine (304) that the one or more processors failed to execute a first command corresponding to the first local device user selectable element 126.

[0055] Further, the one or more processors may be configured to detect (306) a second user input having a second input magnitude by a user at a second local device user selectable element 126 displayed on the local device user interface 122, determine (308) that the one or more processors have failed to execute a second command corresponding to the second local device user selectable element 126, and determine (310) that the local device user interface 122 is unresponsive to the first user input and the second user input.

[0056] In response to determining 310 that the local device user interface 122 is not responsive to the second user input, the one or more processors may be further configured to transmit 312 first data (e.g., local device data) to a proximate mobile device associated with the user. The first data may correspond to or include data corresponding to a user interface offload prompt, as previously described herein.

[0057] In another embodiment, the one or more processors may be further configured to generate (314) a mobile device user interface 132 on a display of the mobile device 130. The mobile device user interface 132 may include user selectable elements 136 that, when displayed in the local device user interface 122, correspond to the first local device user selectable elements 126 and the second local device user selectable elements 126 described above. In other words, the mobile device user interface 132 may be configured to resemble or mirror the local device user interface 122. The mobile device user interface 132 may be generated using at least the first data received from the local device 120. For example, the mobile device user interface 132 may be generated within an internet browser application installed on the mobile device. Alternatively, the mobile device user interface 132 may be generated within an application published by an owner or agent of the local device 120, which in turn is compatible with the software running on the local device 120. In other words, a software application specifically designed to run and display a version of the local device user interface 122 on the mobile device user interface 132 may be provided to a user of the mobile device 130 for download by the user 140. Thus, when local device data is transmitted from the local device 120 to the mobile device 130, the mobile device 130 may be configured to run the software application on the mobile device 130 using the local device data.

[0058] In yet another embodiment, the method 300 may include one or more processors of the mobile device receiving 316 data from the mobile device user interface 132 corresponding to mobile device user inputs. The mobile device user inputs may include touch actions detected on a touch screen display of the mobile device 130, where the touch actions correspond to a user 140 touching or selecting one or more mobile device user selectable elements 136 displayed on the touch screen. For example, a first user input may be the user 140 selecting a mobile device user selectable element 136 labeled “START ORDER,” which may be configured to display a list of menu options corresponding to the user selectable element of the menu options. A second user input may be the user 140 selecting one or more of the user selectable elements of the menu options, etc. A third user input may be the user 140 providing payment information to complete the order. Nevertheless, each user input may be captured as multiple user inputs to the mobile device and stored in the memory of the mobile device for transmission to the network 110 via the local device 120 or directly to the network via another connection to the network 110.

[0059] Additionally, the method 300 may include one or more processors of the mobile device configured to transmit 318 second data to the local device 120, the second data including the mobile device user inputs. The second data may be transmitted directly to the network 110 via a wireless communication medium or some other form of network connection. Nevertheless, to resolve the failed local device user interface function, data corresponding to the user inputs at the mobile device 130 is captured on the mobile device user interface 132.

[0060] In an embodiment, the local device 120 may be configured to be registered with the network 110 to facilitate communication between the local device 120 and another network-connected device. For example, the local device 120 may include data corresponding to a software application configured to execute on the local device 120. The software application, when executed on the local device 120, may facilitate registration with a server associated with the business at which the local device 120 is being used. The registration may occur via the network 110 to which the local device 120 and the server are connected. Similar to how the local device 120 may be registered with the network 110, a mobile device 130 may be registered with the network 110 to facilitate communication with other network-connected devices, not shown in the figure.

[0061] In embodiments, the user input received at the local device 120 and the mobile device 130 may be agnostic, i.e., the user input may not be limited to touch actions as described herein. Rather, the user input may be voice command input, other peripheral type input, or computer-generated input following instructions by a user.

[0062] 4 illustrates a block diagram of a computer 400 suitable for resolving failed user interface functionality in accordance with an exemplary embodiment of the present invention. FIG. 4 is intended as an illustration of one implementation only and is not intended to imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications may be made to the illustrated environment.

[0063] The computer 400 (e.g., mobile device 130) includes a communications fabric 402 that provides communication between a cache 416, memory 406, persistent storage 408, a communications unit 410, and input / output (I / O) interfaces 412. The communications fabric 402 may be implemented using any architecture designed to pass data and / or control information between processors (such as microprocessors, communications processors, and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, the communications fabric 402 may be implemented using one or more buses or crossbar switches.

[0064] Memory 406 and persistent storage 408 are computer readable storage media. In this embodiment, memory 406 includes random access memory (RAM). In general, memory 406 may include any suitable volatile or non-volatile computer readable storage media. Cache 416 is a high-speed memory that improves the performance of computer processor 404 by holding recently accessed data and data close to the data accessed from memory 406.

[0065] Software and data 414 may be stored in persistent storage 408 and memory 406 for execution and / or access by one or more of the respective computer processors 404 via cache 416. In an embodiment, persistent storage 408 includes a magnetic hard disk drive. As an alternative or in addition to a magnetic hard disk drive, persistent storage 408 may include a solid-state hard drive, a solid-state storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0066] The media used by persistent storage 408 may be removable. For example, a removable hard drive may be used for persistent storage 408. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 408.

[0067] In these examples, the communications unit 410 provides for communication with other data processing systems or devices. In these examples, the communications unit 410 includes one or more network interface cards. The communications unit 410 may provide for communication using either or both physical and wireless communications links. Software and data 414 may be downloaded to the persistent storage 408 via the communications unit 410.

[0068] The I / O interface 412 allows for the input and output of data with other devices that may be connected to the computer 400. For example, the I / O interface 412 may provide a connection to an external device 418, such as a keyboard, keypad, touch screen, or other suitable input device, or a combination thereof. The external device 418 may also include a portable computer-readable storage medium, such as a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data 414 used to practice embodiments of the invention may be stored on such a portable computer-readable storage medium and loaded into the persistent storage 408 via the I / O interface 412. The I / O interface 412 is also connected to a display 420.

[0069] Display 420 provides a mechanism for displaying data to user 140 and may be, for example, a computer monitor.

[0070] IoT data may include any data used by, stored on, transmitted or received by an IoT device, and may include classified IoT data, including, but not limited to, biometric data, credit card numbers, personal identification numbers, medical records, passwords, access codes, etc.

[0071] The present invention may include various accessible data sources, such as databases 124, that may contain personal data, content, or information that the user 140 does not want processed. Personal data includes user information, such as tracking information or geographic location information, as well as personally identifying or sensitive personal information. Processing refers to any automated or non-automated operation or set of operations, such as collection, recording, organization, structuring, storage, adaptation, alteration, retrieval, consultation, use, disclosure by transmitting, disseminating or otherwise making available, combination, restriction, erasure, or destruction, performed on personal data. The software and data 414 may enable authorized and secure processing of personal data. The software and data 414 may be configured to provide informed consent along with notice of the collection of personal data, allowing the user 140 to permit or deny processing of the personal data. Consent can take multiple forms. Authorizing consent can compel the user 140 to take affirmative action before the personal data is processed. Alternatively, the refusal of consent may force the user 140 to take affirmative action to prevent the processing of the personal data before the personal data is processed. The software and data 414 may provide information about the nature (e.g., type, scope, purpose, duration, etc.) of the personal data and the processing. The software and data 414 provides the user 140 with a copy of the stored personal data. The software and data 414 allows for the correction or completion of incorrect or incomplete personal data. The software and data 414 allows for the immediate deletion of personal data.

[0072] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0073] A computer readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in a groove on which the instructions are recorded, and any suitable combination of the above. As used herein, a computer-readable storage medium should not itself be construed as a transitory signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0074] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing device / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof) that may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing device / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage on a computer readable storage medium within the respective computing device / processing device.

[0075] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the invention, electronic circuitry including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer readable program instructions to customize the electronic circuitry by utilizing state information of the computer readable program instructions.

[0076] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks included in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0077] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions for performing aspects of the functions / operations specified in the blocks of the flowcharts and / or block diagrams.

[0078] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in the flowchart and / or block diagram blocks.

[0079] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in a different order than the order shown in the figures. For example, two blocks shown in succession may in fact be executed as one step, executed simultaneously, executed substantially simultaneously in a partially or fully overlapping manner in time, or executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks included in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that executes the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0080] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. It is obvious to those skilled in the art that many changes and modifications are possible without departing from the scope and spirit of the present invention. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to allow other skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for resolving a failed user interface function by information processing in a computer, the method comprising: detecting, by one or more processors, a first user input having a first input magnitude at one or more user selectable elements displayed at a local device user interface of the local device, the first user input being by a user; determining, by the one or more processors, that the local device user interface is not responsive to the first user input; detecting a second user input at the one or more user selectable elements having a second input magnitude greater than the first input magnitude, the second user input being by the user; determining, by the one or more processors, that the local device user interface is not responsive to the second user input; transmitting, by the one or more processors, first data corresponding to a user interface offload prompt to a mobile device associated with the user, the mobile device being within a predefined proximity range from the local device user interface; receiving, by the one or more processors, second data from the mobile device corresponding to a plurality of mobile device user inputs; A method for performing an action including:

2. Determining that the local device user interface is unresponsive to the first user input includes: The method of claim 1 , further comprising determining, by the one or more processors, that the local device failed to execute a command corresponding to the one or more user selectable elements.

3. 3. The method of claim 1, further comprising generating, by the one or more processors, a mobile device user interface on a display of the mobile device, the mobile device user interface including mobile device user selectable elements that correspond to the one or more user selectable elements.

4. 4. The method of claim 1, wherein in response to determining that the local device user interface is not responsive to the second user input, the user interface offload prompt is sent to the mobile device.

5. The method of any one of claims 1 to 4, wherein at least one of a plurality of user inputs of the mobile device corresponds to the first user input at the one or more user selectable elements.

6. 6. The method of claim 1, wherein determining that the local device user interface is unresponsive to the second user input further comprises receiving indicator data from one or more sensors, the indicator data corresponding to user irritation.

7. 7. The method of claim 1, wherein determining that the local device user interface is unresponsive to the second user input further comprises determining that a magnitude of the second input is greater than a magnitude of the first input by a predefined threshold.

8. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 7 for resolving failed user interface functions.

9. 1. A computer system for processing commands in a network, said computer system comprising: detecting a first user input by a user at one or more user selectable elements displayed on a local device user interface, the first user input having a first input magnitude; determining, by one or more processors, that the local device user interface is unresponsive to the first user input; program instructions for detecting a second user input by the user at the one or more user selectable elements, the second user input having a second input magnitude greater than the first input magnitude; determining, by the one or more processors, that the local device user interface is not responsive to the second user input; transmitting, by the one or more processors, first data corresponding to a user interface offload prompt to a mobile device associated with the user, the mobile device being within a predefined proximity range from the local device user interface; receiving, by the one or more processors, second data from the mobile device corresponding to a plurality of mobile device user inputs; A computer system for performing operations including:

10. Determining that the local device user interface is unresponsive to the first user input includes:

10. The computer system of claim 9, further comprising determining, by the one or more processors, that the one or more processors failed to execute a command corresponding to the one or more user selectable elements.

11. 11. The computer system of claim 9 or 10, further comprising generating, by the one or more processors, a mobile device user interface on a display of the mobile device, the mobile device user interface including mobile device user selectable elements corresponding to the one or more user selectable elements.

12. 12. The computer system of claim 9, wherein in response to determining that the local device user interface is not responsive to the second user input, the user interface offload prompt is sent to the mobile device.

13. 13. The computer system of claim 9, further comprising program instructions for transmitting the second data to a local device comprising the local device user interface, the local device being connected to a network.

14. 14. The computer system of claim 9, wherein determining that the local device user interface is unresponsive to the second user input further comprises receiving indicator data from one or more sensors, the indicator data corresponding to user irritation.

15. A computer-readable recording medium having the computer program according to claim 8 recorded thereon.

Images