Initiating computing device interaction mode using off-screen gesture detection

Abstract

To provide a method of initiating a computing device interaction mode using off-screen gesture detection.SOLUTION: A method includes identifying, by a mobile computing device, based on a first group of sensor signals provided at least by an inertial measurement unit included in one or more sensors, at least one first gesture that is performed at portions of the housing; initiating an interaction mode; outputting at least one visual or audio indicator for the interaction mode that is associated with a particular function; identifying, based on a third group of sensor signals provided by the one or more sensors, at least one second gesture that is performed at the one or more portions of the housing to confirm a user selection of the particular function; and, responsive to identifying the at least one second gesture, performing the particular function.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] Background Existing computing devices, including mobile computing devices, can be configured to execute a variety of different applications. In many cases, these computing devices provide a touch screen that enables a user to interact with the graphical user interfaces output by these applications. For example, a user can perform touch-based gestures on the touch screen using one or more fingers. When these touch-based gestures are detected on the touch screen, the application can execute one or more corresponding functions, such as selection, movement, or manipulation of an object displayed on the touch screen. In other cases, a particular computing device can be configured to execute a defined function (e.g., capture a screenshot, execute a device lock function) in response to a user performing a plurality of tap gestures on the rear housing of such a device.

Summary of the Invention

Problems to be Solved by the Invention

[0002] Summary The present disclosure relates to techniques for initiating a computing device interaction mode to detect one or more off-screen gestures (e.g., tap gestures) performed by a user at one or more portions of a device housing using off-screen gesture detection. When such an interaction mode is initiated, the computing device may be configured to perform one or more functions based on one or more additionally detected on-screen gestures and / or off-screen gestures. Off-screen interaction may, in some cases, enable one-handed and / or eyes-free mobile interaction. In various examples, a mobile computing device may utilize one or more sensors (e.g., an inertial measurement unit including one or more of an accelerometer, a gyroscope, and / or a magnetometer) to detect off-screen gestures performed by a user at one or more portions of the device housing.

Means for Solving the Problems

[0003] In one example, a method includes a mobile computing device including a housing and a presence sensing display device identifying at least one first gesture performed at one or more portions of the housing based on a first group of sensor signals provided by at least an inertial measurement unit, where the one or more portions of the housing are separate from the presence sensing display device and the inertial measurement unit is included in one or more sensors of the mobile computing device. The method further includes, in response to identifying the at least one first gesture, the mobile computing device starting an interaction mode, outputting at least one visual or audio indicator for the interaction mode associated with a particular function of the mobile computing device, identifying at least one second gesture performed at one or more portions of the housing based on a third group of sensor signals provided by one or more sensors to confirm a user selection of the particular function associated with the at least one visual or audio indicator for the interaction mode, and in response to identifying the at least one second gesture, the mobile computing device performing the particular function.

[0004] In another example, a mobile computing device includes a presence sensing display device, a housing coupled to the presence sensing display, one or more sensors including an inertial measurement unit, at least one processor, and a computer-readable storage device. The computer-readable storage device is configured to store instructions executable by the at least one processor, the instructions causing, based on a first group of sensor signals provided by at least the inertial measurement unit, identification of at least one first gesture performed at one or more portions of the housing, the one or more portions of the housing being separate from the presence sensing display device, the instructions further causing, in response to identification of the at least one first gesture, initiation of an interaction mode, output of at least one visual or audio indicator for the interaction mode associated with a particular function of the mobile computing device, identification of at least one second gesture performed at one or more portions of the housing based on a third group of sensor signals provided by the one or more sensors to confirm a user selection of the particular function associated with the at least one visual or audio indicator for the interaction mode, and execution of the particular function in response to identification of the at least one second gesture.

[0005] In another example, a computer-readable storage device stores instructions that, when executed, cause at least one processor of a mobile computing device to perform operations. Examples of these operations include identifying at least one first gesture to be performed at one or more portions of a housing of the mobile computing device based on at least a first group of sensor signals provided by at least an inertial measurement unit, wherein the one or more portions of the housing are separate from a presence sensing display device and the inertial measurement unit is included in one or more sensors of the mobile computing device, the operations further including initiating a dialogue mode in response to identifying at least one first gesture, outputting at least one visual or audio indicator for the dialogue mode associated with a particular function of the mobile computing device, identifying at least one second gesture to be performed at one or more portions of the housing based on at least a third group of sensor signals provided by one or more sensors to confirm a user selection of the particular function associated with the at least one visual or audio indicator for the dialogue mode, and performing the particular function in response to identifying at least one second gesture.

[0006] One or more examples are described in detail in the accompanying drawings and the following description. Other features, objects, and advantages of the present disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007]

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 6A

Figure 6B

Figure 6C

Figure 7

Best Mode for Carrying Out the Invention

[0008] Detailed Description 1A - 1C are conceptual diagrams showing an example of a mobile computing device 100 configured to initiate an interaction mode for the mobile computing device 100 using off - screen gesture detection according to one or more aspects of the present disclosure. Examples of the mobile computing device 100 can include, but are not limited to, a mobile phone, a tablet computer, a personal digital assistant (PDA), a portable game device, a portable media player, a wearable computing device (e.g., a wristwatch, a wrist - worn computing device, a head - worn computing device), or other types of computing devices. As will be described in more detail below, the mobile computing device 100 may be or may include one or more processors. The mobile computing device 100 includes a display device 102 (e.g., a presence - sensing display device) and one or more sensors 104. The sensors 104 can include any number of one or more sensors such as a touch sensor or a presence - sensing sensor, a microphone, an internal measurement unit (e.g., one or more gyroscopes, accelerometers, magnetometers), a barometer, a camera sensor, a light sensor, and a temperature sensor. In some cases, one or more of the sensors 104 may be included in the display device 102 (e.g., if the display device 102 includes a presence - sensing display) or otherwise associated therewith.

[0009] In addition, the mobile computing device 100 also includes a housing 103. The housing 103 may include one or more portions such as one or more front portions, one or more rear portions, and one or more side portions (for example, a left side portion, a right side portion, an upper side portion, a lower side portion). In various examples, the front side of the mobile computing device 100 may include the display device 102 and one or more front portions of the housing 103. The back side or the rear side of the mobile computing device 100, which is located on the side opposite to the front side of the mobile computing device 100, may include one or more rear portions of the housing 103. The remaining sides or edges of the mobile computing device 100 adjacent to the front side and the rear side of the mobile computing device 100 may include one or more side portions of the housing 103. The housing 103 may or may not include the display device 102. One or more portions of the housing 103 (for example, one or more side portions and / or rear portions of the housing 103) are separate from and distinguishable from the display device 102.

[0010] The display device 102 may function as an input device and / or an output device for the computing device 100. The display device 102 may be implemented using various technologies. For example, the display device 102 may function as an input device that uses a presence sensing input device, such as a resistive film type touch screen, a surface acoustic wave touch screen, a capacitive touch screen, a projected capacitance type touch screen, a presence sensing screen, an acoustic pulse recognition touch screen, a presence sensing screen that detects movement via radar technology, or another presence sensing technology. As described above, in certain cases, one or more of the sensors 104 may be included within the display device 102 or otherwise associated therewith. The display device 102 may function as an output device that uses any one of one or more display devices, such as a liquid crystal display (LCD), a dot matrix display, a light emitting diode (LED) display, an organic light emitting diode (OLED) display, electronic ink, or a similar monochrome or color display capable of outputting visible information to the user of the mobile computing device 100. For example, the display device 102 may present outputs associated with various user interfaces of the application 132 executed on the mobile computing device 100. The user can interact with each respective user interface of the application 132 to cause the mobile computing device 100 to perform operations related to the corresponding application functions.

[0011] In some examples, the mobile computing device 100 may include one or more communication units (as shown in FIG. 2). These communication units are capable of transmitting and / or receiving data to / from one or more other computing devices. In some examples, the communication units support wireless and / or wired communication and they may transmit and / or receive data using any of a variety of communication protocols.

[0012] The mobile computing device 100 is configured to execute a UI module 130, a gesture detection module 134, a dialogue mode selector 136, and one or more dialogue mode modules 138. The UI module 130, the application 132, the gesture detection module 134, the dialogue mode selector 136, and the dialogue mode module 138 may perform the operations described herein using any combination of software, hardware, and / or firmware that resides on and / or executes on the mobile computing device 100. The mobile computing device 100 may use one or more processors to execute the modules 130, 134, 136, 138 and the application 132. The mobile computing device 100 may, in some cases, execute the modules 130, 134, 136, 138 and the application 132 as one or more virtual machines executing on underlying hardware. The modules 130, 134, 136, 138 and the application 132 may be implemented in various ways. For example, any of the modules 130, 134, 136, 138 and / or the application 132 may be implemented as a downloadable or pre-installed application, i.e., an "app". This may also be the case. In some examples, one or more of these may be executed as services of an operating system or a computing platform.

[0013] The application 132 of the mobile computing device 100 may perform various functions for the mobile computing device 100 or access one or more services. Email applications, camera applications, calendar applications, messaging applications, social media applications, travel applications, game applications, stock applications, and weather applications are all examples of the application 132.

[0014] The UI module 130 can cause the display device 102 to present a graphical user interface to the user. For example, the graphical user interface may include graphical elements (e.g., displays) presented at various locations on the display device 102. The UI module 130 may, in some cases, function as an intermediary between various components, applications, and / or modules of the mobile computing device 100 to make determinations based on inputs detected by the display device 102 and generate outputs presented by the display device 102. For example, the UI module 130 may receive information related to inputs detected by the display device 102 from the display device 102 and transmit the input information to one or more of the modules 130, 134, 136, 138 and / or the application 132. The UI module 130 may also receive output information from the modules 130, 134, 136, 138 and / or the application 132 and provide output information for display on the display device 102.

[0015] The user can actively interact with the mobile computing device 100 over time. The mobile computing device 100 may provide one or more interaction modes during operation that allow the user to interact with the mobile computing device 100, such as one or more of the applications 132. Often, the user can perform one or more gestures to interact with the mobile computing device 100.

[0016] Using the display device 102 as the only way of mobile device input (for example, when the display device 102 has a presence-sensing display such as a touch screen) may become increasingly difficult due to certain restrictions. For example, two of these restrictions are that it is difficult to use with one hand and that visual information is blocked by the finger used for operation. As a result, the present disclosure describes techniques for utilizing not only on-screen interaction with the display device 102 but also off-screen interaction using existing sensors 104 on the mobile computing device 100.

[0017] For example, in addition to other sensors (such as presence-sensing sensors) associated with the display device 102, the sensor 104 may include an inertial measurement unit (IMU) (such as the inertial measurement unit 233 in FIG. 2) including one or more gyroscopes, accelerometers, and / or magnetometers. Based on the input provided by the sensor 104, the gesture detection module 134 may use a multi-input multi-output convolutional neural network for off-screen gesture (such as tap) recognition in various examples, as described in more detail below. This detection method enables the prediction of multiple gesture attributes, each of which may be a recognition task. These gesture attributes may include gesture position, gesture direction, gesture condition, and / or gesture force, as described in more detail below.

[0018] As an example, the user can use one or both hands to perform gestures on the mobile computing device 100, and the gestures may include gestures executed on the display device 102 and / or off-screen gestures not executed on the display device 102. Off-screen gestures may include gestures executed on one or more portions (such as one or more back portions and / or side portions) of the housing 103 of the mobile computing device 100.

[0019] For example, in the example of FIG. 1A, the user may perform one or more gestures in mobile computing 100 using the right hand 113. The right hand 113 includes fingers 115, 116, 117, 118, and 119. As shown in FIG. 1A, the user's right hand 113 holds the mobile computing device 100. The thumb 115 is in contact with one or more right side portions of the housing 103, and the fingers 117, 118, 119 are in contact with one or more left side portions of the housing 103. In addition, as shown in FIG. 1A, the index finger 116 is in contact with one or more rear portions 101 of the housing 103. The rear portion 101 may comprise one or more specific portions or regions included in the back or rear surface of the housing 103.

[0020] As will be described in more detail below, the mobile computing device 100 uses a gesture detection module 134 to identify at least one gesture performed in one or more portions of a housing 103 separate from the display device 102 based on a first group of sensor signals provided by an inertial measurement unit (IMU) included in at least sensor 104. For example, the gesture detection module 134 may identify one or more back tap gestures (e.g., a double back tap gesture) performed by the user's finger 116 in one or more rear portions 101 of the housing 103 based on a group of sensor signals provided by the IMU included in sensor 104. In some cases, the IMU (e.g., the IMU 233 shown in FIG. 2) may include any combination of one or more of a gyroscope, an accelerometer, and / or a magnetometer.

[0021] In response to the gesture detection module 134 identifying at least one gesture, the mobile computing device 100 may initiate an interaction mode using the interaction mode selector 136 and / or the interaction mode module 138. In some cases, each of the interaction mode modules 138 may provide a respective interaction mode for interacting with the user during the execution of the application 132. For example, as described in further detail below and as also shown in FIG. 2, the interaction mode module 138 may include an accessibility mode module 240, an assist mode module, an inertia mode module, and / or a background mode module.

[0022] One or more of the interaction mode modules 138 may output at least one visual or audible indicator for an interaction mode associated with a particular function of the mobile computing device 100 based on a second group of sensor signals provided by the sensors 104. For example, when the gesture detection module 134 identifies a double-tap gesture, the interaction mode selector 136 may select the accessibility mode provided by the interaction mode module 138. The accessibility mode may provide audio and / or visual accessibility features while the user is interacting with the mobile computing device 100, for example, with the display device 102.

[0023] For example, as shown in FIG. 1A, during such an accessibility mode, the user may use one or more fingers to explore the content output on the display device 102. FIG. 1A shows that the mobile computing device 100 uses the UI module 130 to display a date (e.g., "Tuesday, May 5") and a temperature (e.g., "60 Indicates that the information (°F) can be output on the display device 102. The UI module 130 may also output various graphical icons associated with different components or applications such as the application 132. In FIG. 1A, four such graphical icons corresponding to four different applications (e.g., "App 1", "App 2", "App 3", and "App 4") that can be executed by the mobile computing device 100 are displayed on the display device 102.

[0024] Each of these four graphical icons can be displayed in a respective specific area of the graphical user interface (GUI) output on the display device 102. For example, the first icon associated with the first application ("App 1") can be displayed within the area 108 of the GUI, the second icon associated with the second application ("App 2") can be displayed within the area 110 of the GUI, the third icon associated with the third application ("App 3") can be displayed within the area 112 of the GUI, and the fourth icon associated with the fourth application ("App 4") can be displayed within the area 114 of the GUI.

[0025] As described above, during the accessibility mode, the user can explore the content of the GUI output on the display device 102 using one or more fingers. In some cases, the interaction module 138 may output another graphical icon 106 indicating that the mobile computing device 100 is currently operating in the accessibility mode. If the display device 102 includes a presence-sensing display, the user (e.g., a visually impaired user) can perform one or more touch gestures (e.g., a slide gesture or a movement gesture) on the display device 102 using one or more fingers (e.g., the right hand 113 or one or more fingers of the user's left hand). Based on the sensor signals provided by one or more of the sensors 104 (e.g., one or more sensors that identify any finger touch or presence in one or more regions of the GUI output by the display device 102), the interaction module 138 can output at least one visual or audio indicator for the interaction mode associated with a particular function of the mobile computing device 100.

[0026] For example, if the user touches or is close to any position within the region 108 of the GUI output on the display device 102 using a finger, the interaction mode module 138 can output an audio indicator associated with the first application ("App 1"). As an example, the interaction mode module 138 can specify the phrase "App 1" (e.g., in the speaker device of the mobile computing device 100) for an audio speech output to audibly identify the name of this first application (e.g., an email application). The output audio speech indicates the content displayed in the region 108 of the graphical user interface where the gesture is performed. As a result, the user can hear the name of this first application when the user's finger touches the region 108.

[0027] As shown in FIG. 1B, if the user continues to move the user's finger to a position within region 114, the interaction mode module 138 may provide an audio speech output that designates the phrase "App 4" to audibly identify the name of this second application (e.g., a computer application). As shown in the example of FIG. 1B, the user may move finger 115 of the right hand 113 to region 114, but in other examples, the user may move any other finger of the right hand 113 or any finger of the user's left hand to region 114. Thus, the user can navigate one or more fingers across the GUI output on the display device to, for example, hear the corresponding name of the identifier of the application associated with the region of the GUI where the user's finger(s) is currently located. - tion.

[0028] In certain examples, in addition to or instead of providing one or more audio indicators of the information associated with regions 108, 110, 112, 114 as described above, the interaction mode module 138 may output one or more visual indicators for the interaction mode (e.g., the accessibility mode) as the display device 102. For example, continuing the above example where the user moves finger 115 into region 114 associated with the fourth application ("App 4"), in addition to providing an audio output that designates the phrase "App 4", the interaction mode module 138 may also provide a visual indicator associated with "App 4" for region 114, for example, by providing a visual enlargement of the graphical icon for "App 4" included in region 114 (e.g., for visually impaired users) as shown in FIG. 1C. FIG. 1C shows such a visual enlargement of this content included within a magnifying glass icon 121, which can be the content displayed in region 114 where the current touch gesture is being performed by finger 115.

[0029] If the user wishes to further explore the information output on the display device 102 (e.g., the information contained in regions 108, 110, and / or 112), the user can move the finger 115 on the display device 102, and as the finger 115 moves from region to region, the user may receive visual and / or audio indicators of the content contained in any of these regions. In this case, the magnifying glass icon 121 shown in FIG. 1C may also move visually within the GUI and selectively magnify the content within these regions as the finger 115 is moved. In certain cases, the interaction mode module 138 may output only one or more visual indicators as shown in FIG. 1C without necessarily outputting an audio indicator (e.g., an audio word or audio phrase) as the finger 115 moves from region to region. In such a manner, the interaction mode module 138 may provide visual and / or audio indicators in response to user navigation of one or more fingers across the content output on the display device 102.

[0030] In response to such navigation, the user may also be able to select any of the graphical icons contained within each region of the GUI. For example, if the user's finger is positioned in region 114, the user may perform one or more gestures to select the graphical icon for "App 4" by, for example, using the finger 116 to perform a single back tap gesture at portion 101 of the housing 103.

[0031] The gesture detection module 134 may identify at least one second gesture that is performed at one or more portions of the housing 103 to confirm a user selection of a specific function, such as the execution of a fourth application associated with the region 114, based on yet another group of sensor signals provided by the sensor 104. In response to identifying at least one second gesture, the mobile computing device 100 may execute a corresponding function (e.g., execute the fourth application from the application 132). Thus, when starting such an interaction mode, the mobile computing device 100 may be configured to execute one or more functions based on one or more detected on-screen and / or off-screen gestures. Off-screen interaction may enable one-handed and / or eyes-free mobile interaction in various scenarios. When the gesture detection module 134 identifies yet another gesture (e.g., another back tap gesture) that is performed at one or more portions of the housing 103 based on the detection of one or more yet another sensor signals provided by the sensor 104 indicating the user's intention to end the current interaction mode, the interaction mode module 138 may end this interaction mode.

[0032] FIG. 2 is a block diagram illustrating an example of a mobile computing device 200 according to one or more aspects of the present disclosure. The mobile computing device 200 may include an example of the mobile computing device 100 shown in FIGS. 1A-1C, and components with like reference numerals may provide similar functions as described with reference to FIGS. 1A-1C. FIG. 2 shows only one specific example of the mobile computing device 200, and many other examples of the mobile computing device 200 may be used in other cases and may include a subset of the components included in the example of the mobile computing device 200, or may include additional components not shown in FIG. 2.

[0033] In the example of FIG. 2, the mobile computing device 200 has a housing 203, and includes a presence sensing display device 202, one or more processors 220, one or more input components 222, one or more communication units 224, one or more output components 226, one or more sensors 204, a power source 228, and one or more storage devices 250. A communication channel 227 can interconnect each of the components 220, 222, 202, 224, 204, 228, 226 and / or 250 for (physically, communicatively, and / or operably) inter-component communication. In some examples, the communication channel 227 can include a system bus, a network connection, an inter-process communication data structure, or any other means for communicating data between hardware and / or software.

[0034] One or more input components 222 of the mobile computing device 200 can receive inputs such as inputs from a user. Examples of inputs are touch / haptic, presence sensing, and voice inputs. Examples of input components 222 include a presence sensing screen, a touch sensing screen, a touch screen, a mouse, a keyboard, a trackpad, a voice response system, a video camera, a microphone, or any other type of device for detecting inputs from humans or machines.

[0035] One or more output components 226 of the mobile computing device 200 can generate outputs. Examples of outputs are haptic outputs, audio outputs, and visual outputs. Examples of output components 226 include a presence sensing screen, a touch sensing screen, a touch screen, a sound card, a video graphics adapter card, a speaker, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a micro light emitting diode (microLED) display, an active matrix organic light emitting diode (AMOLED) display, a haptic device, or any other type of device for generating outputs to humans or machines.

[0036] One or more communication units 224 of the mobile computing device 200 can communicate with external devices via one or more networks by transmitting and / or receiving network signals on one or more networks (e.g., one or more wired and / or wireless networks). For example, the mobile computing device 200 can transmit and / or receive wireless signals on a wireless network such as a cellular wireless network using the communication unit 224. Similarly, the communication unit 224 can transmit and / or receive satellite signals on a satellite network such as a Global Positioning System (GPS) network. Examples of the communication unit 224 include a network interface card (e.g., an Ethernet (registered trademark) card, etc.), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device capable of transmitting and / or receiving information. Other examples of the communication unit 224 can include a shortwave radio, a cellular data radio, a wireless Ethernet network radio, and a Universal Serial Bus (USB ) controller.

[0037] The presence sensing display device 202 of the mobile computing device 200 includes a display component 223 and a presence sensing input component 225. In some examples, the presence sensing display device 202 may provide an output to the user using tactile, audio, or visual stimuli, as described above with reference to the output component 226. For example, the display component 223 may provide a display or video output, as described with reference to the output component 226. The presence sensing display device 202 may also provide an input function as described above with reference to the input component 222. For example, the presence sensing input component 225 may provide an input function as described with reference to the input component 222.

[0038] The display component 223 may be a screen on which information is displayed by the presence sensing display device 202, and the presence sensing input component 225 may detect objects in and / or near the display component 223. As an example of one range, the presence sensing input component 225 may detect an object such as a finger or a stylus within 2 inches of the display component 223. The presence sensing input component 225 may determine the position (e.g., (x,y) coordinates) of the display component 223 where the object is detected. In another example of a range, the presence sensing input component 225 can detect an object within 6 inches of the display component 223, and other ranges are possible. The presence sensing input component 225 may use capacitive, inductive, radar-based, and / or optical recognition technologies to determine the position of the display component 223 selected by the user's finger. In some examples, the presence sensing input component 225 may also provide an output to the user using a touch stimulus, a presence sensing stimulus, an audio stimulus, or a video stimulus, as described with respect to the display component 223. The display component 223 may be any type of output device that provides a visual output, such as described with respect to the output component 226.

[0039] Although shown as an internal component of the mobile computing device 200, the presence-aware display device 202 may also represent an external component that shares a data path with the mobile computing device 200 to send and / or receive input and output. For example, in one example, the presence-aware display device 202 represents an embedded component of the mobile computing device 200 (e.g., a screen on a mobile phone) that is disposed within and physically connected to the external packaging of the mobile computing device 200. In another example, the presence-aware display device 202 represents an external component of the mobile computing device 200 (e.g., a monitor and / or projector that shares a wired and / or wireless data path with a tablet computer) that is disposed outside the package of the mobile computing device 200 and is physically separate therefrom.

[0040] The presence-aware display device 202 of the mobile computing device 200 can detect two-dimensional and / or three-dimensional gestures as input from a user of the mobile computing device 200. For example, sensors of the presence-aware display device 202 (e.g., sensors of the presence-aware input component 225) can detect movement of a user (e.g., movement of a hand, arm, pen, stylus) within a threshold distance of the sensors of the presence-aware display device 202. The presence-aware display device 202 can determine a two-dimensional or three-dimensional vector representation of the movement and correlate the vector representation to gesture inputs having a plurality of dimensions (e.g., a wave, pinch, clap, pen stroke). In other words, the presence-aware display device 202 can detect, at or near a screen or surface (e.g., the display component 223) on which the presence-aware display device 202 outputs information for display, movement of a user Multidimensional gestures can be detected without requiring a gesture from the user. Instead, the presence-aware display device 202 can detect multidimensional gestures that are performed in or near a sensor that may or may not be located near the screen or surface from which the presence-aware display device 202 outputs information for display.

[0041] One or more storage devices 250 within the mobile computing device 200 may store information for processing during operation of the mobile computing device 200 (e.g., during execution of one or more of the UI module 230, the application 232, the operating system 254, or the gesture detection module 234). In some examples, the storage device 250 includes a temporary storage device, which means that the primary purpose of the storage device 250 is not long-term storage. The storage device 250 on the mobile computing device 200 may be configured for short-term storage of information as volatile memory and thus does not retain stored content when the power is turned off. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0042] In some examples, the storage device 250 includes one or more computer-readable storage media. The storage device 250 can be configured to store more information than volatile memory. The storage device 250 is further configured for long-term storage of information as non-volatile memory space and can retain the information after power-on / off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy (registered trademark) disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory. The storage device 250 can store program instructions and / or data related to one or more applications 232, UI module 230, operating system 231, gesture detection module 234, interaction mode selector 236, and interaction mode module 238. The UI module 230, application 232, gesture detection module 234, interaction mode selector 236, and interaction mode module 238 can include examples of the corresponding UI module 130, application 132, gesture detection module 134, interaction mode selector 136, and interaction mode module 138 shown in FIG. 1.

[0043] In certain examples, the storage device 250, or one or more of the components included in the storage device 250, can be stored on one or more remote computing devices external to the mobile computing device 200 (e.g., on one or more external servers). In some examples, one or more remote computing devices can store and / or execute the UI module 230, application 232, and / or operating system 231. In these examples, one or more remote computing devices can perform functions similar to those described herein with respect to the processor 220.

[0044] As shown in FIG. 2, the mobile computing device 200 may include a power source 228. In some examples, the power source 228 may be a battery. The power source 228 may supply power to one or more components of the computing device 2. Non-limiting examples of the power source 228 may include batteries having the chemical properties of zinc-carbon, lead-acid, nickel cadmium (NiCd), nickel metal hydride (NiMH), lithium ion (Li-ion), and / or lithium ion polymer (Li-ion polymer), but are not necessarily limited thereto. In some examples, the power source 228 may have a limited capacity (e.g., 1000 to 3000 mAh).

[0045] The mobile computing device 200 also includes one or more sensors 204. In some examples, one or more of the sensors 204 may be examples of one or more of the input component 222 and / or the presence sensing input component 225. The sensor 204 includes an inertial measurement unit (IMU) 233. For example, the IMU 233 may include one or more gyroscopes such as one or more high-frequency gyroscopes (e.g., 200 Hz gyroscopes), accelerometers, and / or magnetometers. As described herein, the sensor 204 of the mobile computing device 200 may be configured to determine the real-time orientation, rotation, or other movement of the mobile computing device 200 in three-dimensional space.

[0046] One or more processors 220 may implement functions and / or execute instructions within the mobile computing device 200. For example, a processor 220 on the mobile computing device 200 may receive and execute instructions stored by a storage device 250 that execute the functions of an application 232, an operating system 231, a UI module 230, an interaction mode selector 236, a gesture detection module 234, and / or an interaction mode module 238. These instructions executed by the processor 220 may cause the mobile computing device 200 to store information in the storage device 250 during program execution. The processor 220 may execute instructions of the operating system 231 and the application 232 to perform one or more operations. That is, the operating system 231 and the application 232 may be operable by the processor 220 to perform the various functions described herein.

[0047] In some alternatives, the mobile computing device 200 may consist of only the processor 220, or in other aspects may include the processor 220. In these examples, the input component 222, the presence sensing display device 202, the communication unit 224, the output component 226, the sensor 204, the power supply 228, and the storage device 250 may be communicatively coupled to the mobile computing device 200 while being external to it (e.g., via a communication channel 227).

[0048] The application 232 may include one or more different various applications. An email application, a camera application, a map or navigation application, a calendar application, a messaging application, a social media application, a travel application, a game application, a stock application, and a weather application are all examples of the application 232.

[0049] As shown in FIG. 2, the gesture detection module 234 includes a machine learning module 229. Based on the input provided by the sensor 204, the gesture detection module 234 may use a multi-input multi-output convolutional neural network model as shown in FIG. 3 for off-screen gesture (e.g., tap) recognition in various examples, as will be described in more detail below. This detection method enables the prediction of multiple gesture attributes. The machine learning module 229 may be configured to perform such predictions.

[0050] For example, the machine learning module 229 may utilize a convolutional neural network to recognize signals provided by the sensor 204 such as the inertial measurement unit 233. For example, in a particular instance, each time the user taps on the housing 203 of the mobile computing device 200, a tap-induced motion signal from the inertial measurement unit 233 is captured and used by the machine learning module 229 to recognize the tap position, direction (e.g., front, back, and four edges), condition (e.g., tapping with a finger against a finger pad), and / or the applied force. Similarly, the machine learning module 229 may classify inertial measurement unit signals each time the user tilts the mobile computing device 200.

[0051] This neural network model obtains touch position information (e.g., the presence sensing input component 225) from the touch screen if available, as well as barometer and IMU signals for jointly estimating the tap position (x, y), direction (front / back / four sides), condition (finger pad vs. fingernail), and / or force. If the estimated tap force is strong enough, it is considered a potential intentional tap. Note that the barometer value may be useful because a strong tap is thought to deform the phone housing and compress the internal air pressure, thus affecting the barometer reading.

[0052] FIG. 3 is a block diagram showing such an example of use of a neural network model 352 for identifying gesture attributes according to one or more aspects of the present disclosure. In this example, the machine learning module 229 shown in FIG. 2 can utilize the neural network model 352 to classify and / or determine gesture attributes such as tap attributes based on the analysis of sensor signals provided by the sensor 204. In some cases, the machine learning module 229 can be executed by an embedded digital processor unit included in the processor 220.

[0053] In some cases, the machine learning module 229 may utilize a lightweight convolutional neural network model and can be executed, for example, using an embedded digital processor unit (e.g., one of the processors 220) in a low-power mode. In various examples, a more computationally intensive network is activated only when the gesture detection module 234 detects a particular gesture (e.g., a double tap) in the housing 203 and is executed, for example, by the main central processing unit of the processor 220 to provide different tap attributes such as position and orientation. By doing so, the gesture detection module 234 can limit potentially false trigger cases (e.g., unintentional interaction with a visual wallpaper output for display on the display device 202) and can minimize the power consumption of the mobile computing device 200.

[0054] As shown in FIG. 3, the neural network model 352 can receive various different signal inputs from the sensor 204, such as one or more touch screen signals 353 (e.g., signals associated with the presence sensing input component 225), one or more inertial measurement unit signals 354 provided by the inertial measurement unit 233, and one or more other optional sensor signals 355 (e.g., signals from a barometer if included in the sensor 204). By processing these input signals 353, 354, 355 as inputs, the neural network model 352 of the machine learning module 229 can predict or identify various gesture attributes, such as a tap position 356 related to a gesture, a tap direction 357 related to a gesture, a tap condition 358 related to a gesture, and / or a tap force 359 related to a gesture.

[0055] The tap position 356 can indicate or identify the predicted position of a tap event on a portion of the housing 203 of the mobile computing device 200, such as an (x, y) position or an area. The tap position 356 can be located on any portion of the housing 203 or on the presence sensing display device 202. The tap direction 357 can indicate or identify a predicted direction (e.g., the front portion of the housing 203, the back portion of the housing 203, the right edge portion / side portion of the housing 203, the left edge portion / side portion of the housing 203, the upper edge portion / side portion of the housing 203, the bottom edge portion / side portion of the housing 203). The tap condition 358 can identify or indicate one or more conditions of the tap, such as whether the tap is performed with a fingernail or with a finger pad, and the tap force 359 can indicate or identify the amount of force applied by the tap. In some cases, if the estimated tap force is strong enough, the machine learning module 229 can consider the tap to be an intentional tap rather than an accidental tap.

[0056] The interaction mode module 238 in FIG. 2 includes various different modules that can be selected by the interaction mode selector 236 based on the gestures identified by the gesture detection module 234. In a particular example, one or more of the interaction mode modules 238 may be included in or executed by the operating system 231. For example, the interaction mode module 238 includes an accessibility mode module 240, an assist mode module 241, an inertia mode module 242, and a background mode module 243. In various examples, the gesture detection module 234 may identify a particular (e.g., predefined) gesture, such as a double tap gesture, that may cause the interaction mode selector 236 to select a particular interaction mode. This particular gesture can be any form of gesture, such as a double tap gesture, a single tap gesture, an edge gesture executed on a side portion (e.g., the right side portion) of the housing 203 and / or a portion of the display device 202. In some cases, the mobile computing device 200 may enable user customization of the particular gestures that can be used to cause the interaction mode selector 236 to select an interaction mode.

[0057] When the gesture detection module 234 identifies a particular gesture, the interaction mode selector 236 may select an interaction corresponding to one of the interaction mode modules 238. The interaction mode selector 236 may make such a selection based on one or more factors (e.g., the current operating state of the mobile computing device 200, the current application(s) of the currently executing application 232, the identification of one or more previous and / or subsequent gestures by the gesture detection module 234, the detection of other sensor signals provided by the sensor 204, other inputs and / or outputs detected by the UI module 230). Next, the interaction mode selector 236 and / or the operating system 231 may execute one of the accessibility mode module 240, the assist mode module 241, the inertia mode module 242, or the background mode module 243 based on the selected interaction mode.

[0058] As described above with reference to FIGS. 1A-1C, the accessibility mode provided by the accessibility mode module 240 may provide audio and / or visual accessibility features while the user is interacting with the mobile computing device 200, such as the display device 102. For example, during such an accessibility mode, the user may use one or more fingers to explore the content output on the display device 202. Instead of relying solely on on-screen touch events, the accessibility mode module 240 also processes off-screen gestures identified by the gesture detection module 234.

[0059] The sole use of on-screen touch events for accessibility features may lead to certain restrictions. For example, object selection in a voice accessibility mode that uses only on-screen touch events may be performed with on-screen tap gestures and / or on-screen drag / swipe gestures. However, such a paradigm of on-screen accessibility gestures can lead to an interruption of touch exploration (e.g., finger-surface contact). Exploration continuity is a common behavior regarding tactile reading, or how a visually impaired person can perceive the world using senses and touch. There is.

[0060] In addition, when a user enables a voice accessibility mode that uses only on-screen touch events, these users may no longer be able to use a conventional gesture navigation system designed for direct use. Instead, voice accessibility users may have to develop a more complex set of system navigation gestures that may involve multi-touch gestures or combinations of gestures such as swipe up and swipe down, which can make use very difficult. These limitations can be a learning hurdle for long-term users, but may also prevent new users who can benefit from frequent and temporary access to voice accessibility and support features from using such an accessibility mode at all.

[0061] Accordingly, according to the technology of the present disclosure, the accessibility mode module 240 may further utilize the off-screen gestures identified by the gesture detection module 234 to enable the user to continue to use more typical on-screen system navigation gestures while operating in the accessibility mode. In contrast to the development of a new set of navigation gestures, the implementation and use of the accessibility mode module 240 (such as using off-screen gestures such as back tap gestures) enables accessibility users to continue to use conventional system navigation gestures during the navigation and exploration of items within the displayed GUI. As a result, the continuity of exploration can be maintained.

[0062] According to the technology of the present disclosure, the accessibility mode module 240 may provide exploration gesture (such as exploration tap) technology that enables the user to perform off-screen gestures to quickly invoke and utilize the voice and / or visual accessibility mode. For example, in various cases, the accessibility mode module 240 may process the device back tap gesture (such as a double back tap gesture on one or more back portions of the mobile computing device 200) identified by the gesture detection module 234 to initiate a quick access to the voice and / or visual accessibility features of the accessibility mode. To achieve the goals of quick access and exploration continuity, in various examples, the accessibility mode module 240 utilizes device back gesture (such as tap) detection to prompt the user to invoke and terminate the accessibility mode on the fly during the interaction, and in certain cases, may mimic the tactile reading behavior in the two-handed interaction paradigm.

[0063] In this mode, the user can use a touch exploration finger (e.g., finger 115 shown in FIG. 1B) to explore and glide across the screen for touch exploration, causing the accessibility mode module 240 to output, for example, audible audio corresponding to the navigated on-screen content. During this touch exploration period, the user can, for example, perform a single back tap gesture on the back portion of the housing 203 to confirm the selection of the spoken object. The user can lift the touch exploration finger to end the accessibility mode and then use typical on-screen finger-based gestures for further system navigation. Thus, with this approach, new voice accessibility users do not need to learn a complex set of new system navigation gestures. Instead, it becomes possible for the user to use separate off-screen gestures to perform specific tasks during the accessibility mode.

[0064] For example, the user can use the a provided by the accessibility mode module 240 To invoke the accessibility mode, a double-tap gesture can be performed on the back portion of the housing 203. Next, the user can drag a touch exploration finger (e.g., finger 115 in FIG. 1B) on the GUI output on the display device 202 based on the accessibility mode module 240 identifying touch gestures in the display device 202 based on, for example, one or more groups from the sensors 204 and / or input component 225 or received sensor signals, to listen to voice support and / or view visual support (e.g., magnification) of on-screen display objects. This type of interaction means that the user can keep the touch exploration finger on the screen throughout the phase of the accessibility mode to provide continuous exploration and mimic the effect of tactile reading behavior. To select an on-screen object during touch exploration, the user can continue to hover the touch exploration finger over the object and, for example, use the hand holding the mobile computing device 200, such as the hand 113 shown in FIG. 1A, to perform a single back-tap gesture, for example, on the back portion of the housing 203, to select the object. This continuity can also reduce the effort of interaction by using a single back-tap gesture for object selection, which can be a potentially significant benefit when frequent object selection, such as typing using a virtual on-screen keyboard, is used. After a phase of the interaction, if the user wishes to exit the accessibility mode, the user can lift the touch exploration finger from the display device 202 to exit the accessibility mode and then use the conventional on-screen gesture navigation system to perform further operations on the display device 202.

[0065] As an example, the UI module 230 can output various graphical icons, links, and / or other information associated with different components or applications such as application 232. In FIG. 1A, four such graphical icons corresponding to four different applications that can be executed (e.g., "App 1", "App 2", "App 3", and "App 4") are displayed. In other examples, the display component 223 can display other graphical information or icons in any form such as hypertext or web links, text or symbols, components of a graphical keyboard (e.g., keys), and the web links can be associated with one or more websites that can be accessed by the mobile computing device 200, and / or the keys of the graphical keyboard can be selected by the mobile computing device 200 during the execution of one or more functions by the interaction mode module 238.

[0066] The information displayed using the display component 223 can have a relevant position or area of display in the presence-aware display device 202. For example, in the example of FIG. 1A, each of these four graphical icons can be displayed in a respective specific area of the GUI. That is, the first icon associated with the first application ("App 1") may be displayed within area 108 of the GUI, the second icon associated with the second application ("App 2") may be displayed within area 110 of the GUI, the third icon associated with the third application ("App 3") may be displayed within area 112 of the GUI, and the fourth icon associated with the fourth application ("App 4") may be displayed within area 114 of the GUI.

[0067] During the execution of the accessibility mode module 240, the user may use one or more fingers to explore the content of the GUI output on the display device 202. For example, the user (e.g., a visually impaired user) may use one or more fingers (e.g., the right hand 113 or one or more fingers of the user's left hand) to perform one or more touch gestures (e.g., a slide gesture or a movement gesture) on the display device 202. Based on sensor signals provided by one or more of the sensors 204 (e.g., one or more sensors that identify the touch or presence of any finger in one or more regions of the GUI output by the display device 202), the interaction module 138 may output at least one visual or audio indicator for an interaction mode associated with a particular function of the mobile computing device 200.

[0068] For example, when the user touches or approaches any position within the region 108 of the GUI using a finger, the accessibility mode module 240 may output an audio indicator associated with a first application ("App 1"). As an example, the accessibility mode module 240 may provide an audio output that designates the phrase "App 1" to auditorily identify the name of this first application (e.g., an email application). As a result, the user can hear the name of this first application when the user's finger touches the region 108. If the user continues to move the user's finger to a position within another region (e.g., region 114), the accessibility mode module 240 may provide an audio output to auditorily identify the name of this second application (e.g., a calculator application). When navigating the information displayed on the display device 202, the user can hear the corresponding name of the identifier of the application associated with the region of the GUI where the user's finger (s) is currently located, using either the right or left hand finger of the user.

[0069] Also, as described above with reference to FIG. 1C, in certain examples, in addition to or instead of providing one or more audio indicators of the information being presented, the accessibility mode module 240 may output to the display device 202 one or more visual indicators for an interaction mode (e.g., the accessibility mode). For example, continuing the above example where the user may move a finger 115 within the region 114 associated with a fourth application (“App 4”), in addition to providing an audio output that designates the phrase “App 4”, the accessibility mode module 240 may also provide a visual indicator associated with “App 4” for the region 114, for example, by providing a visual enlargement of the graphical icon for “App 4” included in the region 114 (e.g., for a visually impaired user), as shown in FIG. 1C. In this case, the magnifying glass icon 121 shown in FIG. 1C may also move visually within the GUI and selectively magnify the content within these regions as the finger 115 is moved. In such a manner, the accessibility mode module 240 may provide visual and / or audio indicators in response to user navigation of one or more fingers across the content output on the display device 202.

[0070] In response to such navigation, the user may also be able to select any of the graphical icons included within each region of the GUI. For example, when the user's finger is located in the region 114, the user may perform one or more gestures to select the graphical icon for “App 4”, such as by performing a single back tap gesture on the back portion of the housing 203. The gesture detection module 234 may identify this at least one second gesture performed at one or more portions of the housing 203 to confirm a user selection of a particular function, such as the execution of the fourth application associated with the region 114, based on another group of sensor signals provided by the sensor 204.

[0071] In response to identifying at least one second gesture, the mobile computing device 200 may perform a corresponding function (e.g., execute a fourth application from application 232). In such a manner, the accessibility mode mo The module 240 may provide an intuitive user experience with respect to audio and / or visual accessibility modes, for example, using rear device tap detection. Off-screen tap interactions open up numerous new opportunities for audio and / or visual accessibility support. This gives the user the possibility of quick access to the accessibility mode and continuity in touch exploration, and thus can lead to a better user experience in the accessibility mode.

[0072] As shown in FIG. 2, the interaction mode module 238 also includes an assist mode module 241. In various examples, the assist mode module 241 may provide an incline-back tap assist interface that enables a user to perform certain functions (e.g., scroll up, scroll down, application switching) by using a back tap gesture on the housing 203 of the mobile computing device 200.

[0073] There may be many situations where one-handed interaction is preferred for on-the-go mobile use of the mobile computing device 200 (e.g., when the user is holding a handrail on a subway or lying in bed). Additionally, improving support for one-handed interaction may also be beneficial for users with disabilities.

[0074] Current touch-based interactions typically involve a user performing a thumb touch while holding the mobile computing device 200 in a less natural gripping gesture. The use of this gesture can make it more difficult for the user to firmly hold the mobile computing device 200, and it can be particularly difficult to use this gesture when the user is holding the mobile computing device 200 in the air (e.g., while lying down). To address this problem, the assist mode module 241 provides an assist execution mode that includes an incline / back tap interface to enable the user to perform mobile device interactions with a more natural phone gripping gesture, as shown in FIG. 4A.

[0075] FIGS. 4A-4B are conceptual diagrams showing an example of a mobile computing device 400 configured to initiate an assist interaction mode, such as during the execution of the assist mode module 241, according to one or more aspects of the present disclosure. In FIGS. 4A-4B, the mobile computing device 400 may be an example of the mobile computing device 200 (FIG. 2) and / or the mobile computing device 100 (FIGS. 1A-1C). The housing 403 may be an example of the housing 203 and / or the housing 103. The display device 402 may be an example of the display device 202 and / or the display device 102.

[0076] Similar to starting the accessibility mode, the user can start the assist mode provided by the assist mode module 241 (FIG. 2) by performing, for example, a double-back tap gesture. For example, the user can perform a double-back tap gesture on the back portion 401 of the housing 403 using the index finger 416 of the hand 413. The gesture detection module 234 can identify the gesture based on one or more groups of sensor signals provided by the sensor 204 (e.g., the IMU sensor signal provided by the IMU 233). The other fingers 415, 417, 418, and 419 can hold the mobile computing device 400.

[0077] As shown in FIG. 4A, after the finger 416 performs a double-back tap gesture, the interaction mode selector 236 can start the assist mode module 241. The assist mode module 241 can output a visual menu 460 on the display device 402. The menu 460 provides visual indicators of various different functions that can be performed by the assist mode module 241 during the assist mode associated with different menu items. For example, the menu 460 can include menu items 461, 462, 464, 465. The menu item 461 may include a graphical upward arrow associated with the scroll-up function, and the menu item 464 may include a graphical downward arrow associated with the scroll-down function. The menu item 462 may be provided with a graphical left arrow associated with the page-back / backward function, and the menu item 465 may be provided with a graphical icon associated with the function for toggling or switching between different applications 132 that can be executed. As a result, each of the menu items 461, 462, 464, 465 includes a visual indicator of the respective function that can be performed by the assist mode module 241.

[0078] The assist mode module 241 and / or the gesture detection module 234 may be configured to recognize tilt gestures and reverse tap gestures from the motion signals provided by the sensor 204. After the user performs an initial back tap gesture using a finger 416 on one or more rear portions 401 of the housing 403 to initiate the assist interaction mode provided by the assist mode module 241, the user can subsequently tilt the mobile computing device 400 using one or more tilt gestures to select menu items in the menu 460. The one or more tilt gestures are associated with the detected motion and / or rotation of the mobile computing device 400 (e.g., based on the sensor signals provided by the sensor 204). Next, the user can perform another (e.g., one) back tap gesture on one or more rear portions 401 to execute a function associated with the selected menu item based on one or more groups of sensor signals (e.g., IMU sensor signals provided by the IMU 233) provided by the sensor 204 used by the gesture detection module 234 to identify these gestures.

[0079] The assist mode module 241 provides an interface as shown in FIGS. 4A-4B, which enables the user to hold the mobile computing device 400 in a natural gripping gesture and perform the system navigation functions commonly used (e.g., scroll function, page back function, switch application function) by means of tilt gestures and back tap gestures that provide one-handed interaction.

[0080] As shown in FIG. 4A, before the user executes a tilt gesture and after the assist mode module 241 first enters the assist interaction mode, the menu items have not yet been selected. As a result, the menu center icon 463 is displayed prominently in a highlighted, shaded, or other manner, indicating that none of the menu items 461, 462, 464, or 465 have been selected yet, as shown in FIG. 4B.

[0081] However, after the assist mode is started, the user may use the hand 413 to execute one or more tilt gestures to move and / or rotate the mobile computing device 400 in one or more directions or orientations. One or more of the sensors 204 (e.g., the IMU 233) may provide sensor signals indicating the movement and / or rotation, and the gesture detection module 234 may process these signals to identify one or more tilt gestures. For example, the user may tilt the mobile computing device 400 up, down, right, left, clockwise, counterclockwise, or any combination thereof. The gesture detection module 234 may provide an indication of the identified tilt gesture and the corresponding attributes of the tilt gesture to the assist mode module 241 which may indicate at least one of the detected movement or rotation direction or orientation of the mobile computing device 400.

[0082] The assist mode module 241 may then identify which menu item of the menu 460 to select based on the attributes of the tilt gesture associated with at least one of the detected movement, rotation direction, or orientation of the mobile computing device 400. For example, the assist mode module 241 may define a mapping of the gesture attributes to the selected menu item. As an example, the assist mode module 241 may select the menu item 461 in response to a tilt gesture of tilting the mobile computing device 400 upward with respect to the top of the mobile computing device 400, taking into account its current position and / or orientation. Similarly, the assist mode module 241 may select the menu item 464 in response to a tilt gesture of tilting the mobile computing device 400 downward. The assist mode module 241 can select the menu item 462 in response to a tilt gesture of tilting the mobile computing device 400 to the left, and may select the menu item 465 in response to a tilt gesture of tilting the mobile computing device to the right.

[0083] Alternatively, as shown in the example of FIG. 4B, the assist mode module 241 may select the menu item 462 in response to a tilt gesture of tilting the mobile computing device 400 counterclockwise (with respect to the current position and / or orientation of the mobile computing device 400), and may select the menu item 465 in response to a tilt gesture of tilting the mobile computing device 400 clockwise. In such a manner, the assist mode module 241 may output one or more visual indicators via each menu item of the menu 460 for the assist interaction mode, and each menu item is associated with a corresponding function, as further described below.

[0084] In FIG. 4B, the user uses hand 413 to tilt the mobile computing device clockwise with respect to the position and / or orientation of the mobile computing device 400 shown in FIG. 4A. The gesture detection module 234 detects the attributes of this tilt gesture and provides this information to the assist mode module 241. As a result, the assist mode module 241 selects the menu item 465 of the menu 460. As shown in FIG. 4B, the assist mode module 241 shades, highlights, or otherwise makes the menu item 465 stand out relative to the other menu items 461, 462, 464 in order to provide the user with a visual indicator that the menu item 465 is selected. The menu item 465 is associated with a specific function (e.g., switching of application functions).

[0085] To confirm the selected display of the menu item 465, the user may use finger 416 to perform a single back tap gesture on one or more back portions 401 of the housing 403. Upon receiving an indication of the back tap gesture that identifies this gesture based on one or more groups of sensor signals received from the sensor 204, the assist mode module 241 may perform a corresponding function, such as confirming the selection of the menu item 465 and performing an application switching function that switches between the applications 132 that are running and / or displayed in the foreground of the display device 402. The user can continue to perform the back tap gesture on one or more back portions 401 to make another selection of the highlighted menu item (e.g., menu item 465). That is, as shown in FIG. 4B, the user can perform a repeated back tap gesture while the menu item 465 is highlighted to cause the assist mode module 241 to perform a repeated function (e.g., repeated application switch function) associated with the menu item 465. [[ID=?]]

[0086] Furthermore, the user can perform additional tilt gestures to select different menu items from the menu 460. Thus, as an example, the user can tilt the mobile computing device 400 upward to select the menu item 461. Next, the menu item 461 can be made prominent (instead of the menu item 465) on the display device 402. The user can then perform one or more back tap gestures to confirm the selection of the menu item 461. The menu item 461 causes the assist mode module to perform one or more corresponding scroll-up functions (e.g., while browsing web content). When the user is ready to exit the assist interaction mode, the user can, for example, perform a double back tap gesture to cause the assist mode module 241 to end this mode. In this case, the user can perform a double back tap gesture to both start and end the assist interaction mode, and perform a tilt gesture in between to cause the assist mode module 241 to perform a function corresponding to the menu item selected from the menu 460 (e.g., system navigation or other functions).

[0087] In various examples, for the implementation of the machine learning module 229 of the assist mode module 241 and the gesture detection module 234, the machine learning module 229 can utilize a lightweight convolutional neural network model (e.g., neural network model 352), and can be executed, for example, using an embedded digital processor unit (e.g., one of the processors 220) in a low power mode. This model can classify tap directions and non-tap events from signals such as any taps indicated by the sensor 204. When the interface of the assist mode module 241 has already been called, all recognized back tap gestures and tilt gestures can be passed from the gesture detection module 234 to the assist mode module 241 and / or the operating system 231 to control the navigation function and identify the selected menu item of the menu 460. When the interface of the assist mode module 241 has not yet been called, the signal processing service in the digital processor unit and / or the gesture detection module 234 can analyze the back tap timing and look only for gestures that match the double tap pattern to recognize an intentional double back tap gesture and start the assist mode.

[0088] In various examples, tilt events can be detected by the gesture detection module 234 using either a heuristic or a machine learning method, for example, from gyroscope data provided by the sensor 204. In certain cases, the user can also use a triple back tap gesture to call up an on-screen cursor and use the tilt to manipulate the cursor for navigation on the display device 202.

[0089] Inclined / back tap interaction opens up countless new opportunities. The use of such interaction can be easy and convenient, and can empower users to perform one-handed interaction with the natural holding gesture of the mobile computing device 400 shown in FIGS. 4A-4B, enabling interaction behavior when the user browses, for example, a news feed or a social network.

[0090] The interaction mode module 238 of FIG. 2 also includes an inertia mode module 242. The inertia mode module 242 may provide what is referred to herein as an inertial interaction mode. In various examples, during this interaction mode, the inertia mode module 242 and / or the gesture detection module 234 may estimate the tap force from the IMU signals provided by the IMU 233, and may also estimate the tap position from the output by the machine learning module 229 based on, for example, changes in the force and / or angle associated with the tap gesture. As a result, in certain cases, the inertia mode module 242 and / or the gesture detection module 234 can identify touch gestures on one or more portions of the housing 203 and / or the display device 202 based on one or more groups of sensor signals from the sensors 204 and / or the input component 225, even when the finger is wearing a glove or has long fingernails, or when there are water droplets on the housing 203 or the display device 202, and can be configured to potentially reject false touch events when water droplets that affect the reliability of perception are present on the touch screen.

[0091] Similar to using touch pressure for unintentional touch rejection, the inertial mode module 242 and / or the gesture detection module 234 can utilize the inertia generated by an intentional tap, including the force generated by the tap gesture and the change in orientation induced by the tap, and motion sensing can be used to identify the user's tap intention. These various attributes can be captured by or determined based on signals from the IMU 233 included in the sensor 204. In various cases, the machine learning module 229 can utilize a neural network model (e.g., the neural network model 352 of FIG. 3) to determine one or more attributes of the tap gesture. Since the disclosed technology does not rely on capacitance sensing, it can operate on a waterfall screen or a normal screen that does not cover the edge of the device. Thus, the display device 202 can include a waterfall screen or a non-waterfall screen.

[0092] In certain cases, the inertial mode module 242 can provide specific functions related to side taps and swipe gestures that can provide an effective and ergonomic replacement for the use of edge buttons. Buttonless mobile computing devices (e.g., phones) have been a design goal for years, and in the design of mobile phones, waterfall screens, which are screens curved at the side edges of the terminal, are in trend. Physical buttons, especially volume control buttons, have been given various alternatives, but they are usually not user-friendly. In addition, using only touch events on visual edge buttons can lead to many false cases. Double-tap gestures can be effective in rejecting accidental touches on the screen (e.g., in the display device 202). However, since this gesture involves multiple touch events, the interaction becomes more difficult and the user experience becomes less intuitive.

[0093] To address such difficulties and challenges, the inertia mode module 242 provides functions associated with side impact actions that generate IMU responses to prime the system to a volume or other user interface control state that times out after a timeout period. During and prior to the expiration of the timeout period, the inertia mode module 242 may map swipe gestures at the display device 202 to a volume or other variable user interface control.

[0094] As a result, the inertia mode module 242 provides a tap-swipe interaction paradigm. A swipe gesture following a side tap also serves as a confirmation of the user's intent. From an ergonomic perspective, the user can use the index finger to tap an edge to invoke an edge interface (e.g., a volume control interface) as shown in FIGS. 5A-5B.

[0095] FIGS. 5A-5C relate to one or more aspects of the present disclosure and illustrate inertia for tap-swipe interaction FIG. 0 is a conceptual diagram showing an example of a mobile computing device 500 configured to initiate a sex dialogue mode. In the examples of FIGS. 5A-5C, the user can operate the mobile computing device 500 by one-handed interaction using the hand 513. In tap-swipe interaction that occurs at or near the edge of the mobile computing device 500, the user can first tap one or more side portions 570 of the housing 503 using the finger 516, as shown in FIG. 5A. The edge of the mobile computing device 500 may comprise, in certain examples, a side portion (e.g., a left side portion, a right side portion, an upper side portion, a lower side portion) of the housing 503 or in the vicinity thereof, and / or a region near the edge of the display device 502. The gesture detection module 234 can identify this gesture based not only on the applied force of the finger 516, but also on corresponding movements on the housing 503 of the mobile computing device 500, based on one or more groups of sensor signals received from the sensor 204 (e.g., IMU sensor signals provided by the IMU 233).

[0096] It should be noted that the term "sex dialogue mode" in the original text seems inappropriate. It might be a misspelling. If it is supposed to be something else like "specific dialogue mode", please adjust the translation accordingly.After the gesture detection module 234 identifies this tap gesture, the inertial mode module 242 outputs a graphical volume control 574 and may monitor for subsequent swipe gestures (e.g., by the gesture detection module 234) in the display device 502 within a defined or customizable period such as a timeout period. The gesture detection module 234 may identify a swipe gesture based on one or more groups of screen capacitance sensing signals provided by the sensor 204 and / or the presence sensing input component 225. Use of the swipe gesture can invoke an inertial mode edge interface and can move along the screen edge of the display device 502 for linear control as shown in FIG. 5B, or move within the screen area for multi-target selection as shown in FIG. 5C. Although the graphical volume control 574 is shown in FIG. 5B, any other form of graphical user interface control (e.g., a display control, another type of setting control) can be used in various other examples.

[0097] For example, as shown in FIG. 5B, the user can use finger 516 to operate the graphical volume control 574 via a slide gesture executed in region 572 of the display device 502. The user can use finger 516 to execute an upward slide gesture within region 572 and correspondingly move the volume control 574 upward to increase the volume. The user can also execute a downward slide gesture within region 572 and correspondingly move the volume control 574 downward to decrease the volume. In response, the inertial mode module 242 can output corresponding visual adjustments to the graphical volume control 574 (e.g., an upward visual adjustment for an increase in volume or a downward visual adjustment for a decrease in volume). The volume control 574 is merely an example of a user interface control that can be operated by the user during the inertial mode provided by the inertial mode module 242 and is a visual indicator of the inertial mode module 242 associated with one or more functions such as increasing or decreasing the volume control element. The volume control 574 is also an example of a user interface control that can be displayed near the screen edge of the display device 502 in region 572 when the display device 502 includes a waterfall screen. When the display device 502 has a waterfall screen or capacitive sensing on the edge, edge user interface controls such as the volume control 574 can be displayed near or around the side tap position (e.g., side portion 570).

[0098] However, when the display device 502 does not have edge capacitive sensing and / or a waterfall screen, the edge user interface control is positioned proximate to an area where the thumb 515 of the hand 513 can be located, such as region 576 shown in FIG. 5C. It can be displayed in other areas of the display device 502, such as an area or region. In certain cases, the inertial mode module 243 may estimate the area 576 for displaying the volume control 574 shown in FIG. 5C using the position of the side portion 570 where the finger 516 contacts the housing 503, as shown by the neural network model 352. As shown in FIG. 5C, the user may use the thumb 515 to interact with the volume control 574 displayed in the area 576 of the display device 502. For example, the user may perform one or more gestures (e.g., a swipe gesture) to interact with or select a target object included in the volume control 574 (e.g., for multi-target selection), or perform any other form of control function visually shown in the graphical user interface for the inertial mode. To end or pause the inertial interaction mode, in various examples, the user may perform a double-back tap gesture.

[0099] As a result, the tap-swipe gesture can be a gesture distinguishable from an unintended grip or touch of the mobile computing device 500 and can be used, for example, as part of one-handed interaction to manipulate edge-based user interface controls. This gesture can be generalized well across devices regardless of the presence or absence of a waterfall screen design.

[0100] The interaction mode module 238 of FIG. 2 also includes a background mode module 243. In various examples, based on at least the gesture detection module 234 detecting one or more gestures, the interaction mode selector 236 may select the background mode module 243 for execution. In various examples, the background mode module 243 may enable background user interface interaction using off-screen tap recognition.

[0101] In various cases, the user interface may include a foreground layer (e.g., a layer having application shortcuts for one or more of the applications 232 on the home screen) and one or more background layers (e.g., one or more layers having a background wallpaper). Since the wallpaper can often appear on both the lock screen and the home screen, it is frequently exposed to the user. Therefore, a wallpaper service including a program user interface can have a significant impact on the user interaction with the mobile device.

[0102] However, existing technologies rely only on on-screen touch events to interact with both foreground and background services. Since they share the same interaction mechanism, it is difficult for the system to recognize the user's intention and clearly separate the interaction between the foreground and the background, especially in a visually cluttered interface. In other words, when the user taps on the edge or boundary of an application icon, it can be difficult for the mobile computing device 200 (e.g., using the UI module 230) to determine whether the user intends to open one of the applications 232 associated with the icon or to interact with the wallpaper. This ambiguity hinders effective interaction with the wallpaper service and thus limits the functionality of interactive wallpapers.

[0103] To address this problem, the background mode module 243 uses off-screen taps detected by the gesture detection module 234 to interact with background user interfaces such as wallpaper services, games, or flash card applications. The detection of such off-screen gestures clarifies the identification of the user's intention in the interaction between the foreground and the background, and thus enables the user to achieve, for example, a richer interaction with interactive wallpapers.

[0104] The background mode module 243 uses the information provided by the gesture detection module 234 and the machine learning module 229. The machine learning module 229 may be configured to estimate gesture (e.g., tap) positions on the back and / or side portions of the housing 203 using a neural network model (e.g., the neural network model 352 shown in FIG. 3, which may include a multi-task convolutional neural network that processes IMU signals as inputs). This estimation provides a previously unachieved granularity of off-screen tap detection for background user interface interaction. Each time the user taps on the housing 203 of the mobile computing device 200, a tap-induced motion signal is captured and can be used to recognize gesture attributes such as the attributes 356, 357, 358, 359 shown in FIG. 3.

[0105] In various examples, for the implementation of the machine learning module 229 of the background mode module 243 and the gesture detection module 234, the machine learning module 229 can utilize a lightweight convolutional neural network model and can be executed, for example, using an embedded digital processor unit (e.g., one of the processors 220) in a low-power mode. In various examples, a more computationally intensive network is activated and executed by the main central processing unit only when the gesture detection module 234 detects a double-tap event in the housing 203 to provide different tap attributes such as position and orientation. By doing so, the gesture detection module 234 can potentially limit false trigger cases (e.g., unintentional interaction with the wallpaper) and minimize power consumption.

[0106] The background mode module 243 enables user interaction with a background user interface, such as an interactive wallpaper, that uses off-screen tap gestures (e.g., back tap and / or side tap gestures) to provide at least one visual indicator of an interaction mode and various related functions, allowing the user to have a clear separation between foreground interaction and background interaction. That is, the user can interact with a foreground user interface or application using on-screen gestures on the display device 202, or alternatively, interact with a background user interface or application using off-screen gestures executed at one or more portions of the housing 203. The user can interact with a background user interface, such as an interactive wallpaper, using a back tap or edge tap gesture, which may, in various cases, not respond to on-screen touch events. This technology can open up new venues for user interaction. Such interaction can be performed even in the lock screen mode, and thus, the user can have the fastest access to various things, such as a flash card service, a news feed service, or a lightweight game, even before unlocking it or without the need to unlock it. Generally, edge gestures, such as an edge tap gesture, can be executed at one or more side portions of the housing 203 and / or one or more portions of the display device 202 (e.g., the edge portion of the display device 202).

[0107] In the case of a wallpaper service that shows news feeds, after the user first performs a double-back tap gesture on the back portion of the housing 203 to start the background interaction mode in the background mode module 243, the user can perform another back tap gesture to cycle to the next news feed, or perform an edge tap gesture on the side portion of the housing 203 and / or a portion of the display device 202 to switch to another news category. Cycling to the next news feed or switching to another news category may include removing the current background user interface element (e.g., the current news feed element) from the display and outputting one or more new background user interface elements (e.g., news feed elements of the same or different news categories) for display. The gesture detection module 234 can detect any gesture based on the reception of one or more groups of sensor signals from the sensor 204 (e.g., the IMU sensor signals provided by the IMU 233). In some cases, the wallpaper service may also include or provide a light game such as a doodle game. In these cases, when the user performs a double-back tap gesture to start the background interaction mode and causes the background mode module 243 to execute the game as a background user interface application, the user can perform one or more additional back tap gestures at different positions on the back portion of the housing 203 to interact with or play the game, providing the user with flexibility of control beyond discrete event-based signals. For example, the user can perform a back tap gesture at different positions on the back portion of the housing 203 to cause different actions to be performed in the game. The visual output provided by the background mode module 243 may include one or more visual indicators for the background mode, each associated with one or more respective functions, as described in more detail below. nterface elements (e.g., news feed elements of the same or different news categories) for display. The gesture detection module 234 can detect any gesture based on the reception of one or more groups of sensor signals from the sensor 204 (e.g., the IMU sensor signals provided by the IMU 233). In some cases, the wallpaper service may also include or provide a light game such as a doodle game. In these cases, when the user performs a double-back tap gesture to start the background interaction mode and causes the background mode module 243 to execute the game as a background user interface application, the user can perform one or more additional back tap gestures at different positions on the back portion of the housing 203 to interact with or play the game, providing the user with flexibility of control beyond discrete event-based signals. For example, the user can perform a back tap gesture at different positions on the back portion of the housing 203 to cause different actions to be performed in the game. The visual output provided by the background mode module 243 may include one or more visual indicators for the background mode, each associated with one or more respective functions, as described in more detail below.

[0108] Figures 6A - 6C are conceptual diagrams showing examples of mobile computing devices configured to initiate the background mode module 243 for background user interface interactions according to one or more aspects of the present disclosure. These figures show examples of off - screen taps for background user interface (e.g., wallpaper) interactions. A user can, for example, call and pause background interface interactions by a back double - tap gesture and perform subsequent back - tap gestures or edge - tap gestures to execute specific controls.

[0109] Specifically, the user can initiate an interaction with the background interface using a simple off-screen gesture (e.g., a back double-tap). Next, the user may tap on different locations on the housing 603 of the mobile computing device 600. For example, as shown in FIG. 6A, the user can perform a back tap (e.g., a double back tap) gesture on one or more back portions 601 of the housing 603 using a finger 616 of the hand 613. The gesture detection module 234 can identify this gesture, and the interaction mode selector 236 can select the background mode module 243 for execution. At runtime, the display device 602 can output background user interface elements 670 and indicators 671, 672, 673, and the indicators can be displayed prominently in some format to indicate that they are the currently selected indicators. For example, the background user interface element 670 may comprise news feed items provided by the wallpaper service of the background mode module 243, and the indicators 671, 672, 673 may correspond to different news categories (e.g., sports, weather, local news). If the currently selected indicator 671 is associated with the sports category, the background user interface element 670 may include sports-related news information output for display on the display device 602. In another example, the background user interface element 670 may include language-based flashcards provided by the wallpaper service to assist the user's language learning, and the indicators 671, 672, 673 may correspond to different categories of flashcards (e.g., animals, toys, clothes). The background user interface element may, in certain examples, comprise one or more visual flashcard elements, visual news elements, or other types of elements output for display.

[0110] At a later point in time, the user may perform a subsequent back - tap gesture (e.g., a single back - tap gesture) to change the background user interface elements that are output for display in the same category specified by the indicator 671. Thus, if the currently selected indicator 671 is associated with the sports news category, as shown in FIG. 6B, the user may use the finger 616 to perform a back - tap gesture on the rear portion 601 of the housing 603, as detected by the gesture detection module 234, to cause the background mode module 243 to remove the background user interface element 670 from the display and, instead, output a different background user interface element 675 on the display device 602. This element 675 may include new sports - related news items of the sports news category associated with the indicator 671. In another example, if the currently selected indicator 671 is associated with the animal language category, the user may use the finger 616 to perform a back - tap gesture on the rear portion 601 of the housing 603 to cause the background mode module 243 to output a background user interface element 675 that may include new language - based flash - card elements within the animal language category associated with the indicator 671.

[0111] As also shown in FIG. 6C, the user may perform a side - tap or edge - tap (e.g., a single edge - tap) gesture on one or more edges or side portions 680 of the housing 603 and / or on one or more portions of the display device 602 (e.g., one or more edge portions of the display device 602) to change the category of the background user interface elements that are output for display on the display device 602. FIG. 6C shows the user performing this gesture using the finger 682. The finger 682 may be any user's finger, such as a finger of the hand 613 or another hand of the user.

[0112] When the gesture detection module 234 identifies this gesture, the background mode module 243 may change the category of information output in the background user interface interaction. For example, the background mode module 243 may switch to a second category associated with the indicator 672, remove the background user interface element 675 from display, and instead output a background user interface element 677 of this category. As shown in FIG. 6C, the background mode module 243 may visually emphasize the indicator 672 as the currently selected category, or display it in a prominent manner in other ways.

[0113] Accordingly, when the currently selected indicator 672 is associated with the weather news category, the user may use the finger 682 to perform an edge tap gesture on one or more side portions 680 of the housing 603 and / or on one or more portions of the display device 602 so as to be detected by the gesture detection module 234, causing the background mode module 243 to output different background user interface elements 677 for the weather news category on the display device 602. This element 677 may include new weather-related news items within the weather news category associated with the indicator 672. In another example, when the currently selected indicator 672 is associated with the game language category, the user may use the finger 682 to perform an edge tap gesture, causing the background mode module 243 to output a background user interface element 677 that may include new language-based flash card elements within the game language category associated with the indicator 672. To end or pause the background interaction mode, in various examples, the user may perform a double back tap gesture.

[0114] As a result, the use of off-screen tap recognition for background user interface interactions opens up a myriad of new opportunities. In various cases, this type of interaction can enable quick access to a user's favorite background applications or services, for example, without necessarily unlocking the screen and / or without interfering with gesture interactions with the foreground-running applications, and without necessarily unlocking the screen.

[0115] FIG. 7 is a flow diagram showing an example of the operation of a process 780 executed by a mobile computing device such as mobile computing device 100 (FIG. 1) and / or mobile computing device 200 (FIG. 2) according to one or more aspects of the present disclosure. For purposes of illustration only, the operations of FIG. 7 are described with reference to mobile computing device 200 shown in FIG. 2.

[0116] Process 780 includes identifying (782) at least one first gesture performed in one or more portions of the housing (e.g., housing 203) based on a first group of sensor signals provided by at least an inertial measurement unit (e.g., inertial measurement unit 233) by a mobile computing device including a housing (e.g., housing 203) and a presence-sensing display device (e.g., display device 202) (e.g., gesture detection module 234 of mobile computing device 200). The one or more portions of the housing are separate from the presence-sensing display device, and the inertial measurement unit is included in one or more sensors (e.g., sensor 204) of the mobile computing device.

[0117] In response to identifying at least one first gesture, process 780 also includes the mobile computing device starting an interaction mode (784) (e.g., using interaction mode selector 236 and / or interaction mode module 238), and outputting at least one visual or audio indicator for the interaction mode associated with a particular function of the mobile computing device (786). Process 780 also includes identifying at least one second gesture performed at one or more portions of the housing based on a third group of sensor signals provided by one or more sensors (788) for the mobile computing device to confirm a user selection of a particular function associated with at least one visual or audio indicator for the interaction mode (e.g., using gesture detection module 234). In response to identifying at least one second gesture, process 780 further includes the mobile computing device performing a particular function (790) (e.g., using interaction mode module 238).

[0118] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted across a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. Thus, a computer-readable medium generally may correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for execution of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0119] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any It is also properly referred to as a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead are directed to non-transitory tangible storage media. As used herein, disk (disk and disc) includes compact disc (disc: CD), laser disc (registered trademark) (disc), optical disc (disc), digital versatile disc (disc: DVD), floppy (registered trademark) disk (disk), and Blu-ray (registered trademark) disc (disc), where disk typically magnetically reproduces data and disc optically reproduces data with a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0120] The instructions can be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Thus, as used herein, the term "processor" may refer to either the foregoing structures or any other structure suitable for the execution of the techniques described herein. Further, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules. Also, the technology may be implemented entirely in one or more circuits or logic elements.

[0121] The techniques of the present disclosure can be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chip sets). In the present disclosure, various components, modules, or units have been described in order to emphasize the functional aspects of the devices configured to execute the disclosed techniques, but these do not necessarily have to be realized by different hardware units. Rather, as described above, the various units can be combined in a hardware unit including one or more processors as described above, together with appropriate software and / or firmware, or can be provided by a set of operating hardware units.

[0122] Note that, depending on the embodiment, any particular act or event of the methods described herein can be executed in a different sequence, can be added, combined, or completely excluded (e.g., not all of the described acts or events are necessary for the implementation of the method). Further, in certain embodiments, the acts or events can be executed not sequentially, but, for example, simultaneously through multi-threaded processing, interrupt processing, or through multiple processors.

[0123] In some examples, the computer-readable storage medium comprises a non-transitory medium. The term "non-transitory" indicates that the storage medium is not embodied in a carrier wave or a propagated signal. In a particular example, the non-transitory storage medium can store data that can change over time (e.g., in RAM or a cache).

[0124] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method, comprising: A mobile computing device including a housing and a presence sensing display device identifies at least one first gesture executed in one or more portions of the housing based on at least a first group of sensor signals provided by at least an inertial measurement unit, wherein the one or more portions of the housing are separate from the presence sensing display device, and the inertial measurement unit is included in one or more sensors of the mobile computing device, and the method further comprises: In response to identifying the at least one first gesture, the mobile computing device starts an interaction mode; The mobile computing device outputs at least one visual or audio indicator for the interaction mode associated with a specific function of the mobile computing device; The mobile computing device identifies at least one second gesture executed in the one or more portions of the housing based on a second group of sensor signals provided by the one or more sensors to confirm a user selection of the specific function associated with the at least one visual or audio indicator for the interaction mode; In response to identifying the at least one second gesture, the mobile computing device executes the specific function.

2. Identifying the at least one first gesture includes the mobile computing device identifying at least one first tap gesture executed in the one or more portions of the housing based on at least the first group of sensor signals provided by at least the inertial measurement unit; The presence sensing display device is located on the front side of the mobile computing device. The one or more portions of the housing include (i) at least one side surface portion of the housing located on the side of the mobile computing device adjacent to the front side of the mobile computing device, or (ii) one or more of at least one rear portion of the housing located on the rear side of the mobile computing device, the rear side being on the opposite side of the front side of the mobile computing device, the method of claim 1.

3. The interaction mode includes an accessibility interaction mode, the one or more sensors further include the presence sensing display device, and the method further includes identifying at least one third gesture performed in a region of a graphical user interface output in the presence sensing display device based on a third group of sensor signals provided by the presence sensing display device, by the mobile computing device, outputting the at least one visual or audio indicator for the accessibility interaction mode includes the mobile computing device outputting the at least one visual or audio indicator for the accessibility interaction mode based on the region of the graphical user interface in which the at least one third gesture is performed, the method of claim 2.

4. Outputting the at least one visual or audio indicator for the accessibility interaction mode includes the mobile computing device outputting voice speech indicating content displayed in the region of the graphical user interface in which the third gesture is performed, identifying the at least one second gesture includes the mobile computing device identifying a back tap gesture performed in the at least one rear portion of the housing to confirm a user selection of the specific function associated with the content displayed in the region of the graphical user interface based on the second group of sensor signals provided by at least the inertial measurement unit, the method of claim 3.

5. The content displayed in the region of the graphical user interface where the third gesture is executed includes one of (i) an application icon, (ii) a web link, or (iii) a key of a graphical keyboard. Executing the specific function includes the mobile computing device executing an application associated with the application icon, the mobile computing device accessing a website associated with the web link, or the mobile computing device selecting the key of the graphical keyboard, the method according to claim 4. **Claim 6** Outputting the at least one visual or audio indicator for the accessibility interaction mode includes the mobile computing device outputting a visual enlargement view of the content displayed in the region of the graphical user interface where the third gesture is executed. Identifying the at least one second gesture includes the mobile computing device identifying a back tap gesture executed at at least one rear portion of the housing to confirm the user selection of the specific function associated with the content displayed in the region of the graphical user interface based on at least a second group of sensor signals provided by the inertial measurement unit, the method according to claim 3. **Claim 7** Outputting the at least one visual or audio indicator for the accessibility interaction mode further includes the mobile computing device outputting audio speech indicating the content displayed in the region of the graphical user interface where the third gesture is executed, the method according to claim 6. **Claim 8** The interaction mode includes an assist interaction mode, and the method further in response to starting the assist interaction mode, the mobile computing device outputting a visual menu including a plurality of menu items for display on the presence sensing display device. The mobile computing device identifying a tilt gesture associated with a detected movement or rotation of the mobile computing device based at least on a third group of sensor signals provided by at least the inertial measurement unit, Outputting the at least one visual or audio indicator for the assistive interaction mode includes the mobile computing device outputting a menu item from the plurality of menu items based at least on at least one of a detected movement or rotation direction or orientation of the mobile computing device for display on the presence sensing display device, the menu item being associated with the particular function, Identifying the at least one second gesture, the mobile computing ing device identifying a back tap gesture performed at at least one rear portion of the housing to confirm a user selection of the particular function associated with the menu item based at least on the second group of sensor signals provided by at least the inertial measurement unit, the method of claim 2.

9. The method of claim 8, wherein the particular function associated with the menu item is one of a scroll function, a scroll down function, a page back function, or a switch application function.

10. The interaction mode includes a background interaction mode, Outputting the at least one visual or audio indicator for the background interaction mode includes the mobile computing device outputting an interactive wallpaper including one or more background user interface elements associated with the particular function for display on the presence sensing display device, Identifying the at least one second gesture includes the mobile computing device identifying the at least one second gesture to confirm a user selection of the particular function associated with the one or more background user interface elements based at least on the second group of sensor signals provided by the one or more sensors, The at least one second gesture includes one of (i) an edge tap gesture executed at one or more of the at least one side portion of the housing or at least one portion of the presence sensing display device, or (ii) a back tap gesture executed at the at least one back portion of the housing, the method according to claim 2.

11. The one or more background user interface elements include one or more visual flash card elements or visual news elements for display on the presence sensing display device, the method according to claim 10.

12. The one or more background user interface elements include a first group of background user interface elements, Performing the specific function further includes the mobile computing device deleting the first group of background user interface elements from the display on the presence sensing display device, and the mobile computing device outputting a second group of background user interface elements for display on the presence sensing display, the method according to claim 10.

13. The interaction mode includes an inertial interaction mode, the one or more sensors further include the presence sensing display device, Identifying the at least one first gesture includes the mobile computing device identifying an edge tap gesture executed at one or more of the at least one side portion of the housing or at least one portion of the presence sensing display device based on the first group of sensor signals provided at least by the inertial measurement unit, Outputting the at least one visual or audio indicator for the inertial interaction mode includes the mobile computing device outputting one or more user interface controls for display on the presence sensing display device within a region of the graphical user interface, Identifying the at least one second gesture includes the mobile computing The method according to claim 2, wherein the wing device identifies the at least one second gesture executed in the region of the graphical user interface to confirm the user selection of the specific function associated with the one or more user interface controls based on the second group of sensor signals provided by the presence sensing display device.

14. The at least one gesture includes a slide gesture executed in the region of the graphical user interface, Executing the specific function includes the mobile computing device outputting a visual adjustment to the one or more user interface controls for display on the presence sensing display device, according to the method of claim 13.

15. Identifying the at least one first tap gesture includes the mobile computing device identifying a double back tap gesture executed on at least the one rear portion of the housing based on the first group of sensor signals provided by at least the inertial measurement unit, according to the method of any one of claims 2 to 14.

16. Identifying the at least one first tap gesture executed on the one or more portions of the housing includes the mobile computing device using a machine learning module and a neural network model to analyze the first group of sensor signals provided by at least the inertial measurement unit to determine one or more attributes of the at least one first tap gesture, the one or more attributes including (i) a position associated with the at least one first tap gesture, (ii) a gesture condition associated with the at least one first tap gesture, (iii) a force associated with the at least one first tap gesture, or (iv) a direction associated with the at least one first tap gesture, according to the method of any one of claims 2 to 15.

17. The mobile computing device identifies at least one further gesture to be performed on one or more portions of the housing to end the interaction mode based on yet another group of sensor signals provided by the one or more sensors. The method according to any one of claims 1 to 16, further comprising: the mobile computing device ending the interaction mode in response to identifying the at least one further gesture.

18. The method according to any one of claims 1 to 17, wherein the inertial measurement unit includes at least one of an accelerometer or a gyroscope.

19. A mobile computing device, a presence sensing display device, a housing coupled to the presence sensing display device, one or more sensors including an inertial measurement unit, at least one processor, a computer-readable storage device configured to store instructions executable by the at least one processor, the instructions being to cause at least one first gesture to be performed on one or more portions of the housing based on at least a first group of sensor signals provided by the inertial measurement unit, the one or more portions of the housing being separate from the presence sensing display device, and the instructions further being to start an interaction mode in response to identifying the at least one first gesture, to output at least one visual or audio indicator for the interaction mode associated with a particular function of the mobile computing device, to cause at least one second gesture to be performed on the one or more portions of the housing based on a third group of sensor signals provided by the one or more sensors to confirm a user selection of the particular function associated with the at least one visual or audio indicator for the interaction mode, and to cause the particular function to be executed in response to identifying the at least one second gesture. A mobile computing device.

20. The instructions stored in the computer-readable storage device are further executable by the at least one processor to perform the method according to any one of claims 2 to 18, a mobile computing device according to claim 19.

21. A computer-readable storage device storing instructions that, when executed, cause at least one processor of a mobile computing device to perform operations, the operations comprising: identifying at least one first gesture performed on one or more portions of the housing of the mobile computing device based on a first group of sensor signals provided by at least an inertial measurement unit, the one or more portions of the housing being separate from a presence-sensing display device, the inertial measurement unit being included in one or more sensors of the mobile computing device, the operations further comprising: initiating a dialogue mode in response to identifying the at least one first gesture; outputting at least one visual or audio indicator for the dialogue mode associated with a particular function of the mobile computing device; identifying at least one second gesture performed on the one or more portions of the housing based on a third group of sensor signals provided by the one or more sensors to confirm a user selection of the particular function associated with the at least one visual or audio indicator for the dialogue mode; performing the particular function in response to identifying the at least one second gesture. A computer-readable storage device.

22. The instructions cause the at least one processor of the computing device to further perform the method according to any one of claims 2 to 18, a computer-readable storage device according to claim 21.

Images