Modifying display layout of touch screen display using stylus

Abstract

A method and system including sensing a touch event on a touch screen display of an electronic device; in response to determining that the touch event matches a stylus shaft gesture, changing a display layout presented on the touch screen display, the stylus shaft gesture corresponding to an interaction of a stylus shaft with the touch screen display. The stylus shaft can be between a first end and a second end of the stylus.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. non-provisional application No. 16 / 867,204, filed May 5, 2020, entitled “Using a Stylus to Modify Display layout of Touchscreen Displays,” the contents of which are incorporated herein by reference. Technical Field

[0003] This invention generally relates to touch screen displays and the use of styluses on touch screen displays. Background Technology

[0004] Electronic devices typically feature touchscreen displays to enable user interaction. Users can input information using simple touch gestures or multi-touch gestures by touching the touchscreen display with input devices such as pen-like styluses or one or more fingers. Touchscreens with large sizes and advanced hardware specifications make these devices important tools for improving productivity. Users can perform multiple tasks simultaneously by utilizing the large space on the touchscreen display. For example, electronic devices can run multiple applications concurrently and display information from these applications in multiple windows or views on the touchscreen display at the same time.

[0005] A common way for users to interact with touchscreen displays is by making touch gestures using their fingers or the tip of a pen-like stylus. As an example, based on finger touch events, the Microsoft Surface... TM The gestures that the operating system can recognize and their corresponding descriptions include: "Tap: Press and then release"; "Slide or push: Move the object displayed under the finger by sliding or pushing"; "Slide: Press, slide quickly, and then release"; "Touch and rotate: Slide the content around a point"; "Rotate: Quickly twist to rotate the object"; "Stretch with both hands: Pull the fingers of both hands outward"; "Push with both hands: Bring the fingers of both hands together"; "Twist: Twist an object with two or more fingers, such as turning a knob or turning a page"; "Pinch: Pinch two fingers together on one hand"; "Squeeze: Pinch three or more fingers together on one hand"; "Stretch: Pull the fingers of one hand outward"; "Fix and rotate: Fix the object in place with one finger and drag the object around a fixed point with another finger."

[0006] As can be seen from the above list, most touch screen interactions require finger-based gestures in addition to the basic tap and drag gestures that can be performed using a stylus tip, so users who want to use a stylus often have to switch to finger gestures to take advantage of advanced touch screen capabilities. Many users do not take advantage of some of the display management features provided by touch screen displays because many of the existing supported touch gestures involve too many unintuitive steps.

[0007] Thus, there is a need for a more universal way to manage touch screen display viewing areas on electronic devices. SUMMARY

[0008] The present invention generally relates to using a stylus to implement input interactions on a touch screen display that can modify the touch screen display layout, including implementing screen splitting and input interactions to manage split screen views, etc.

[0009] According to an aspect of the present invention, there is provided a method comprising: sensing a touch event on a touch screen display of an electronic device; in response to determining that the touch event matches a stylus shaft gesture, changing a display layout presented on the touch screen display, the stylus shaft gesture corresponding to an interaction of a stylus shaft with the touch screen display, the stylus shaft being located between a first end and a second end of the stylus.

[0010] The method enables a user to efficiently interact with an electronic device by using a stylus shaft gesture. For example, this can display a desired screen display layout with fewer user interactions, thereby reducing wear and tear or damage that can be caused to the electronic device, and possibly reducing battery power consumption. User experience can also be enhanced, and the electronic device can provide more functionality.

[0011] In the above aspect, changing the display layout comprises: changing a number of viewing areas displayed on the touch screen display, or changing relative sizes of viewing areas displayed on the touch screen display.

[0012] In at least some of the above aspects, determining that the touch event matches the stylus shaft gesture comprises: determining that touch coordinate information of the touch event matches a stylus shaft placement gesture, the stylus shaft placement gesture corresponding to the stylus shaft being placed within a first viewing area on a screen of the touch screen display; and changing the display layout presented on the touch screen display comprises: splitting the first viewing area into a second viewing area and a third viewing area, and causing each of the second viewing area and the third viewing area to display a respective user interface.

[0013] In at least some of the various aspects described above, when the stylus placement gesture corresponds to the stylus being placed in the first viewing area in a vertical orientation, the second and third viewing areas are arranged opposite each other as a left viewing area and a right viewing area.

[0014] In at least some of the various aspects described above, when the stylus placement gesture corresponds to the stylus being placed in the first viewing area in a horizontal orientation, the second and third viewing areas are arranged opposite each other as a top viewing area and a bottom viewing area.

[0015] In at least some of the various aspects described above, at least one of the user interfaces in the second and third viewing areas corresponds to an application that was a background application at the time the touch event occurred; the method includes causing the background application to become a foreground application upon determining that the touch event matches the stylus placement gesture.

[0016] In at least some of the various aspects described above, at least one of the user interfaces in the second and third viewing areas corresponds to an application that was an inactive application at the time the touch event occurred; the method includes causing the inactive application to become a foreground application upon determining that the touch event matches the stylus placement gesture.

[0017] In at least some of the various aspects described above, at least one of the user interfaces in the second and third viewing areas displays a set of icons, each of which is activatable to launch a corresponding application.

[0018] In at least some of the various aspects described above, determining that the touch event matches a stylus gesture includes determining that touch coordinate information of the touch event matches a stylus drag gesture, the stylus drag gesture corresponding to moving the stylus from a location coinciding with a boundary between the first viewing area and the second viewing area, along a screen of the touch screen display; and changing the display layout presented on the touch screen display includes adjusting relative sizes of the first viewing area and the second viewing area in accordance with the movement of the stylus along the screen.

[0019] In at least some of the various aspects described above, determining that the touch event matches a stylus barrel gesture includes determining that touch coordinate information of the touch event matches a stylus barrel drag-and-remove gesture when a first viewing area and a second viewing area are displayed on the touch screen display, the stylus barrel drag-and-remove gesture corresponding to moving the stylus barrel along a screen of the touch screen display within the first viewing area and subsequently removing the stylus barrel from the screen within a threshold period of time of the moving; and changing the display layout presented on the touch screen display includes expanding the first viewing area and closing the second viewing area.

[0020] According to another aspect of the present disclosure, there is provided an electronic device configured to perform the method of any of the example aspects described above.

[0021] According to yet another aspect of the present disclosure, there is provided an electronic device comprising: a touch screen display comprising a display and a touch sensing system configured to generate signals corresponding to touches of a screen of the display; a processing device operatively coupled to the touch screen display; a non-transitory memory coupled to the processing device and storing software instructions that, when executed by the processing device, cause the processing device to: sense a touch event on the touch screen display based on the signals generated by the touch sensing system; determine whether the touch event matches a stylus barrel gesture; and in response to determining that the touch event matches the stylus barrel gesture, change a display layout presented on the touch screen display, the stylus barrel gesture corresponding to an interaction of a stylus barrel with the touch screen display, the stylus barrel being located between a first end and a second end of the stylus.

[0022] According to yet another aspect of the present disclosure, there is provided a computer readable medium storing instructions that, when executed by a processor of an electronic device, configure the electronic device to: sense a touch event on a touch screen display of the electronic device; and in response to determining that the touch event matches a stylus barrel gesture, change a display layout presented on the touch screen display, the stylus barrel gesture corresponding to an interaction of a stylus barrel with the touch screen display, the stylus barrel being located between a first end and a second end of the stylus.

[0023] In at least some of the various example aspects described above, the ability to handle stylus shaft gestures can improve operation of electronic devices and user experience with the electronic devices. For example, a user can be able to replace some finger interactions with a touch screen display with stylus interactions, thereby reducing the potential transfer of dirt, grease, oil, and other contaminants (including bacteria and viruses, etc.) from the user's fingers into the touch screen display. In some cases, reducing contaminants on the screen can reduce the need for cleaning of the touch screen display, thereby reducing potential damage to the device, reducing the consumption of cleaning materials, and can also reduce the spread of contaminants. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are now presented, illustrate example embodiments of the application and together with the general description given above and the detailed description given below, serve to explain the principles of the application.

[0025] Figure 1 A tablet computer employing prior art is shown running two applications and displaying user interfaces of the applications in two viewing areas of its display;

[0026] Figure 2 A touch screen display employing prior art is shown having two viewing areas formed by dragging a window to one end of the display;

[0027] Figure 3 Adjusting a viewing area on a touch screen display employing prior art by dragging a boundary of the viewing area with a finger is shown;

[0028] Figure 4A An electronic device employing a touch screen display is shown in which a stylus shaft is placed in a generally vertical orientation on the screen of the touch screen display;

[0029] Figure 4B An electronic device is shown Figure 4A An electronic device is shown in which a stylus shaft is placed in a generally horizontal orientation on the screen of a touch screen display;

[0030] Figure 4C An electronic device is shown Figure 4A And Figure 4B A block diagram of a touch screen display system selection component of an electronic device is shown;

[0031] Figure 5A An electronic device employing a touch screen display is shown in which a primary viewing area of the touch screen displays a home screen user interface generated by an operating system;

[0032] Figure 5B An electronic device is shown Figure 5AThe electronic device shown has a main viewing area of ​​a touch screen display divided into a first viewing area and a second viewing area, with each viewing area displaying a user interface.

[0033] Figure 6A An electronic device is shown, in which the main viewing area of ​​the touchscreen displays the main screen user interface, an application runs in the background of the electronic device, and a stylus is placed on the screen of the touchscreen display;

[0034] Figure 6B The embodiments of the present invention are shown. Figure 6A or Figure 6B The electronic device shown has a main viewing area of ​​a touchscreen display divided into a first viewing area and a second viewing area. The first viewing area displays the user interface of a running application, and the second viewing area displays the user interface of a note-taking application.

[0035] Figure 6C An electronic device is shown, wherein the main viewing area of ​​the touchscreen displays the user interface of a foreground application running on the electronic device, and a stylus is placed on the screen of the touchscreen display;

[0036] Figure 6D It shows Figure 6A or Figure 6C The electronic device shown has a main viewing area of ​​a touchscreen display divided into a first viewing area and a second viewing area. The first viewing area displays the user interface of a running application, and the second viewing area displays a user interface including multiple icons that represent applications in the second viewing area.

[0037] Figure 6E It shows Figure 6D An alternative arrangement of multiple icons is shown, wherein the multiple icons are arranged near the boundary line between the first viewing area and the second viewing area;

[0038] Figure 7A An electronic device with a touchscreen display is shown, wherein a vertical dividing line divides its main viewing area into a right viewing area and a left viewing area, each of which displays user interface content, and a stylus is placed roughly horizontally on the screen of the touchscreen display.

[0039] Figure 7B It shows Figure 7A The illustrated electronic device has a touchscreen display whose main viewing area is divided into a top viewing area and a bottom viewing area, wherein the top viewing area displays... Figure 7A The user interface content is shown in the left viewing area, and the bottom viewing area displays... Figure 7Auser interface content of the illustrated right viewing area;

[0040] Figure 8A An electronic device is illustrated in which a home screen user interface is displayed in a main viewing area of a touch screen display, a first application and a second application are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0041] Figure 8B An electronic device is illustrated in which a horizontal dividing line divides a main viewing area of a touch screen display into a top viewing area and a bottom viewing area, the top viewing area displays a user interface of a first foreground application, the bottom viewing area displays a user interface of a second foreground application, and a stylus is placed on the screen of the touch screen display;

[0042] Figure 8C An electronic device is illustrated with a touch screen display in which a main viewing area displays a user interface of a first foreground application running on the electronic device, a second background application is also running on the electronic device, and a stylus is placed on the screen of the touch screen display;

[0043] Figure 8D An electronic device is illustrated in which a vertical dividing line divides a main viewing area into a right viewing area and a left viewing area, the right viewing area displays a user interface of a first application, and the left viewing area displays a user interface of a second application;

[0044] Figure 9A An electronic device is illustrated in which a home screen user interface is displayed in a main viewing area of a touch screen display, three applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0045] Figure 9B An electronic device is illustrated in which a vertical dividing line divides a main viewing area into a right viewing area and a left viewing area, each of the right viewing area and the left viewing area displays a screenshot of each of the three applications and a plurality of icons for other applications that are available to run on the electronic device;

[0046] Figure 9C An electronic device is illustrated Figure 9A The screenshot of the first application is illustrated as being selected to activate the first application, and a user interface of the first application is displayed in the left viewing area; the screenshot of the second application is illustrated as being activated Figure 9B The screenshot of the second application is illustrated as being activated, and a user interface of the second application is displayed in the right viewing area; Figure 9B

[0047] ​Figure 9D An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0048] Figure 9E An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display; Figure 9D An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0049] Figure 10A An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0050] Figure 10B An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display; Figure 11A An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0051] Figure 11B An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0052] Figure 12 An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display; Figure 13 An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0053] Figure 1 An electronic device is shown in which a main viewing area of a touch screen displays a user interface of a first foreground application running on the electronic device, two background applications are running in the background of the electronic device, and a stylus is placed on the screen of the touch screen display;

[0054] Figure 2A block diagram illustrating an exemplary electronic device capable of performing the methods described herein is shown in accordance with various embodiments of the present application. DETAILED DESCRIPTION

[0055] In the present application, the term "electronic device" refers to an electronic device having computing capabilities. Examples of the electronic device include, but are not limited to, a personal computer, a notebook computer, a tablet computer ("tablet"), a smartphone, a flat computer, an augmented reality device, an automated teller machine (ATM), a point of sale (POS) terminal, and the like.

[0056] In the present application, the term "display" refers to a hardware component of an electronic device that has a function of displaying graphical images, text, and video content on the electronic device. Examples of the display include, but are not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display.

[0057] In the present application, "screen" refers to a user-facing outer layer of a touchscreen display.

[0058] In the present application, the term "touchscreen display" refers to a combination of a display and a touch-sensing system that can function as an input device by receiving touch inputs. Examples of the touchscreen display include, but are not limited to, a capacitive touchscreen, a resistive touchscreen, and an infrared touchscreen and a surface acoustic wave touchscreen.

[0059] In the present application, the term "touchscreen-enabled device" refers to an electronic device equipped with a touchscreen display.

[0060] In the present application, the term "touch event" refers to an event in which a physical object is detected as interacting with the screen of a touchscreen display.

[0061] In the present application, the term "viewing area" or "view" refers to an area of a display that can be rectangular or the like in shape for displaying information and receiving touch inputs.

[0062] In the present application, the term "primary viewing area" or "primary view" refers to a single viewing area that covers all or almost all (e.g., greater than 95%) of the viewable area of a touchscreen display.

[0063] In the present invention, the term "dividing line" refers to a linear display feature, such as a line that visibly separates two adjacent viewing areas that are displayed simultaneously on a touchscreen display. Examples of dividing lines include: a vertical dividing line, such as one or more vertical lines that provide a border separating a right viewing area from a left viewing area; a horizontal dividing line, such as one or more horizontal lines that provide a border separating a top viewing area from a bottom viewing area. A dividing line can be explicitly displayed or not explicitly displayed, such as a line that marks a border between a first viewing area and a second viewing area.

[0064] In the present invention, the term "split mode" refers to a touchscreen display mode that splits a viewing area, such as a main viewing area. A "vertical split mode" refers to a mode that splits a viewing area into a right viewing area and a left viewing area. A "horizontal split mode" refers to a mode that splits a viewing area into a top viewing area and a bottom viewing area.

[0065] In the present invention, the term "display layout" refers to a configuration of viewing areas that are presented on a display. For example, the main viewing area can have a display layout that employs a split mode that is either a vertical split mode or a horizontal split mode, or a combination of both modes.

[0066] In the present invention, a "window" refers to a user interface form that displays at least a portion of a user interface of an application.

[0067] In the present invention, the term "application" refers to a software program that includes a set of instructions that can be executed by a processing device of an electronic device.

[0068] In the present invention, the terms "running application" and "active application" refer to an application that is resident in memory and that has instructions being executed by a processing device of an electronic device.

[0069] In the present invention, the term "inactive application" refers to an application that is resident in memory but is not an active or running application, but is capable of being launched into an "active" or "running" state.

[0070] In the present invention, the terms "starting" and "launching" an application refer to switching an application from an inactive state to a running state. Launching an application can be accomplished by tapping an icon, swiping, issuing a voice command, or any other method known in the art.

[0071] In the present invention, the terms "executing" and "running" refer to a processing device executing at least some of the instructions that make up an application.

[0072] In the present invention, the expression "activating an icon" refers to interacting with an icon to launch the corresponding application that the icon represents.

[0073] In the present disclosure, the term "foreground application" refers to an application that is running in the foreground of an electronic device. An application is running in the "foreground" or "foreground mode" of an electronic device if the processor of the electronic device is executing the application and the application is displaying its user interface on the display of the electronic device. The user interface can be in a window or in another user interface element such as a scroll bar.

[0074] In the present disclosure, the term "background application" refers to an application that is running in the background of an electronic device. An application is running in the "background" or "background mode" of an electronic device if the processor of the electronic device is executing the application but the application is not displaying its user interface on the touchscreen display of the electronic device. One example of a background application is a running alarm notification application that can be set up and started without displaying its user interface until the alarm goes off, at which point its user interface pops up on the touchscreen display. Another example of a background application is a second application that is maximized to occupy the entire display of an electronic device while a first running application is running, thereby forcing the first application to run in the background without displaying its user interface. Yet another example of a background application is an application that has been minimized by a user, appearing only in the taskbar on the touchscreen display of the electronic device.

[0075] In the present disclosure, the term "home screen user interface" refers to the user interface content displayed by the operating system on the touchscreen display of an electronic device that enables a user to launch applications and functions on the electronic device. A home screen user interface typically displays icons for various applications that can be run on the electronic device. However, a home screen user interface can also include other user interface elements such as scroll bars, window widgets, search fields, etc.

[0076] In the present disclosure, the terms "top," "bottom," "right," "left," "horizontal," and "vertical" when used in the context of a display viewing area refer to the orientation with respect to the display when the content currently displayed on the display is displayed in its intended viewing orientation.

[0077] Known electronic devices can split the main display area into multiple viewing areas. In some examples, these viewing areas can correspond to respective windows. For example, Figure 3An example tablet 10 employing prior art is shown having a display 15. The main viewing area of the display 15 is split into a left viewing area 12 and a right viewing area 14. The left viewing area 12 is displaying the contents of a user interface 16 of a first application program, such as a web browser. The right viewing area 14 is displaying the contents of a user interface 18 of a second application program, such as a photo viewer.

[0078] Referring to Figure 4A An example of a different prior art is shown in which a main viewing area of a touch screen display 25 is split into two viewing areas. The main viewing area of the touch screen display 25 is split into a left viewing area 22 and a right viewing area 24. The user has dragged an application window 28 to the right end of the touch screen display 25. As such, the application window 28 has been maximized to occupy the right viewing area 24 of the touch screen display 25. In the left viewing area 22, a plurality of window screenshots 26 are shown, each representing an active application that is running. The user can select from the plurality of window screenshots 26 the application that the user wishes to maximize to occupy the left viewing area 22.

[0079] Figure 4B An example of prior art adjusting the viewing areas of a touch screen display is shown. An electronic device, such as a tablet 30, has a touch screen display 35 that splits its main viewing area into two viewing areas. A left viewing area 32 displays a first application window 26, while a right viewing area 34 displays a second application window. The user is attempting to adjust the size of the two viewing areas. For example, if the user wishes to shrink the right viewing area 34, the user can touch and drag a dividing line 36, which separates the two viewing areas, to the right. As a result, the right viewing area 34 can have a smaller width, while the left viewing area 32 can have a larger width and expand to the right to fill the space that is no longer used by the right viewing area 34.

[0080] In an example embodiment, an electronic device and a stylus are cooperatively used to modify the display layout of information displayed on the touch screen display of the electronic device in accordance with the interaction of the stylus' pen with the touch screen display. In this regard, Figure 4A and Figure 4CThe electronic device 100 is shown accordingly in accordance with example embodiments, which in the example shown is a tablet device having a touch screen display 45 and a touch input tool, namely a stylus 1000. In example embodiments, the stylus 1000 is an inanimate object like a pen, having a rigid body 1012 extending along an elongate axis 1014 from a first axial end 1016 to a second axial end 1018. The rigid body 1012 includes a stylus shaft 1010 extending along the elongate axis 1014 and between the first and second ends 1016, 1018 of the body 1012. In example embodiments, the stylus shaft 1010 can facilitate user gripping of the stylus and be cylindrical or cuboid in shape along its length. The stylus 1000 can have a tapered tip 1020 disposed at one or more of the axial ends 1016, 1018 of the body 1012. The tip 1020 can be used to activate user interface elements on a touch screen display. In some examples, the stylus 1000 can also include a writing pen. For example, the stylus 1000 can have an ink writing tip at an end opposite the tip 1020.

[0081] In example embodiments, the electronic device 100 is configured to enable a non-tip portion of the stylus 1000, namely the stylus shaft 1010, to provide touch input to the touch screen display 45. In this regard, in Figure 4A and 4B the stylus 1000 is placed on a portion of the touch screen display 45 such that the elongate axis 1014 of the stylus shaft 1010 is parallel to the viewing surface of the screen of the touch screen display 45. In some embodiments, the shaft 1010 has a plurality of contact points spaced along the shaft 1010 for contacting the screen. In other embodiments, a continuous portion of the length of the shaft 1010 is used to contact the screen.

[0082] Figure 4B Selected hardware and software components of the touch screen display system 110 of the electronic device 100 are shown, which are used to detect interaction of the stylus 1000 with the touch screen display 45 and process information about the interaction. The hardware components of the touch screen display system 110 include a touch screen display 45, which includes a display 128, and a touch sensing system 112 for detecting touch interaction with the screen of the display 128.

[0083] In different example embodiments, the touch sensing system 112 can be implemented using different technologies known in the art.

[0084] In one example embodiment, the touch screen display 45 is a capacitive touch screen display, such as a surface capacitive touch screen; the touch sensing system 112 is implemented by a screen that stores an electric charge and a monitoring circuit that monitors the charge throughout the screen. When a conductive object that is capable of drawing a small amount of charge from the screen contacts the capacitive screen of the display 128, the monitoring circuit generates a signal that indicates the point of contact of the touch event. In example embodiments that use a capacitive touch screen display, the pen barrel 1010 of the stylus 1000 is specially designed so that the touch sensing system 112 can detect that the pen barrel 1010 is present on the screen of the display 128. In this regard, in some example embodiments, the pen barrel 1010 includes one or more screen contact points that can transfer an electric charge. For example, the pen barrel 1010 can include conductive contact points that are spaced along the pen barrel 1010 for contacting the screen. The conductive contact points can be electrically connected to one or more human user contact surfaces on the stylus 1000, thereby enabling a conductive path from the human user to the conductive contact points. In some embodiments, a continuous portion of the length of the pen barrel 1010 can have a conductive element for contacting the screen. In some examples, the touch screen display 45 can be a projected capacitive touch screen display rather than a surface touch screen display. In this case, touch events such as stylus drop gestures can occur when the stylus 1000 is close enough to the screen so that it can be detected without actual physical contact.

[0085] In another example embodiment, the touch screen display 45 is a resistive touch screen, and the touch sensing system 112 includes a screen that contains a metallic conductive coating and a resistive layer; and the monitoring circuit generates a signal that indicates the point of contact based on changes in resistance.

[0086] In another example embodiment, the touch screen display 45 is a surface acoustic wave (SAW) or surface wave touch screen, and the touch sensing system 112 sends ultrasonic waves and detects when the screen is touched by recording changes in the waves. In such embodiments, a sound wave absorbing material is provided on the pen barrel 1000 of the stylus 1010.

[0087] In another example embodiment, the touch screen display 45 is an infrared touch screen, and the touch sensing system 112 utilizes a matrix of infrared light beams emitted by LEDs with phototransistor receivers. When an object is near the display, the infrared light beams are blocked, indicating the location of the object.

[0088] In each of the above examples, for a touch event, the touch sensing system 112 generates digital signals indicative of the point of contact of an object with the screen of the display 128. The software of the touch screen display system 110 processes these digital signals. In example embodiments, the software can be part of the operating system (OS) 108 of the electronic device 100. For example, the OS 108 can include a touch screen driver 114 for converting signals from the touch sensing system 112 into spatial touch coordinate information indicative of the physical location of the point of contact of an object on the screen of the display 128 (e.g., a set of multiple X and Y coordinates defining the position of the stylus shaft 1010 relative to the touch screen display 45). In example embodiments, the spatial coordinate information generated by the touch screen driver 114 is provided to a user interface (UI) module 116 of the OS 108, which associates temporal information (e.g., start time and duration) of a touch event with the spatial coordinate information, resulting in touch coordinate information including spatial coordinate information and temporal information. The UI module 116 is configured to determine whether the touch coordinate information matches a touch pattern from a set of candidate touch patterns, each of which corresponds to a respective touch input action, commonly referred to as a gesture. In example embodiments, the set of multiple X and Y coordinates define the position of the stylus shaft 1010 relative to a defined coordinate system of the touch screen display 45, which can be used to determine the shape of the stylus shaft 1010. Thus, in some examples, the touch coordinate information can be indicative of the shape of one or more stylus shaft contact regions, and the set of candidate touch patterns can include patterns matching stylus shapes.

[0089] In example embodiments, in addition to detecting and recognizing conventional finger and stylus tip gestures (e.g., the Microsoft Surface Pen™ gestures described above), the touch screen display system 110 is configured to recognize stylus shaft gestures. For example, the touch screen display system 110 can be configured to recognize a stylus shaft gesture as a touch event, and to determine the shape of the stylus shaft 1010 based on the touch coordinate information. In example embodiments, the touch screen display system 110 is configured to recognize a stylus shaft gesture as a touch event when the touch coordinate information indicates that the stylus shaft 1010 is in contact with the screen of the display 128. In example embodiments, the touch screen display system 110 is configured to determine the shape of the stylus shaft 1010 based on the touch coordinate information. For example, the touch screen display system 110 can be configured to determine the shape of the stylus shaft 1010 based on the set of multiple X and Y coordinates defining the position of the stylus shaft 1010 relative to a defined coordinate system of the touch screen display 45. In example embodiments, the touch screen display system 110 is configured to recognize a stylus shaft gesture as a touch event when the touch coordinate information indicates that the stylus shaft 1010 is in contact with the screen of the display 128, and to determine the shape of the stylus shaft 1010 based on the touch coordinate information. TMIn addition to recognizing the stylus tip gestures, the UI module 116 is also configured to recognize stylus shaft gestures based on the touch coordinate information, the stylus shaft gestures corresponding to: (1) the stylus shaft 1010 of the stylus 1000 being placed on the screen of the display 128 ("stylus shaft placement gesture"); (2) the stylus shaft 1010 of the stylus 1000 being moved on the screen of the display 128 (referred to as a "stylus shaft drag gesture" in the case of linear movement; and a "stylus shaft rotation gesture" in the case of rotational movement); and (3) the stylus shaft 1010 of the stylus 1000 being removed from the screen of the display 128 ("stylus shaft removal gesture"). In example embodiments described in greater detail below, the UI module 116 is configured to further classify the stylus shaft gestures based on the position, orientation, and timing of such gestures. Thus, the touch coordinate information derived by the touch screen driver 114 from the signals generated by the touch sensing system 112 includes information about the position, orientation, and shape of the object causing the touch event, as well as timing information about the touch event. The UI module 116 can use this information to classify the touch event as a particular stylus shaft gesture from a set of possible gestures, each of which has a corresponding predefined touch pattern.

[0090] In example embodiments, the UI module 116 is configured to change the display layout of information presented on the display 128 by providing instructions to the display driver 118 of the OS 108 based on at least one of the type and position of the detected stylus shaft gesture. In example embodiments, components of the OS 108, such as the UI module 116, interact with the UI components of other software programs, such as other applications 120, to coordinate the content displayed in the viewing area on the display 128.

[0091] Now, referring again to Figure 4A and Figure 4B , examples of stylus shaft gestures that can be detected by the touch screen display system 110 are described. Figure 4A and Figure 4B correspond to a vertical stylus shaft placement gesture and a horizontal stylus shaft placement gesture, respectively. In Figure 4A and Figure 4AIn the illustrated example, the electronic device 100 is shown in what is commonly referred to as a "portrait orientation mode." In the portrait orientation mode, the shorter dimension of the rectangular touch screen display 45 defines the width of the primary viewing area (e.g., the distance from the left edge to the right edge), and the longer dimension of the rectangular touch screen display 45 defines the height of the primary viewing area (e.g., the distance from the top edge to the bottom edge). In exemplary embodiments, the electronic device 100 can also operate in a "landscape orientation mode." In the landscape orientation mode, the shorter dimension of the rectangular touch screen display 45 defines the vertical height of the primary viewing area, and the longer dimension of the rectangular touch screen display 45 defines the horizontal width of the primary viewing area. Figure 4B The vertical dashed line 41 is a virtual vertical line that divides the primary viewing area of the touch screen display 55 into a right viewing area and a left viewing area, and represents a touch pattern corresponding to a vertical stylus shaft placement gesture. In Figure 5A to 11B In the illustrated example, the stylus 1000 is placed on the screen of the touch screen display 45 such that the stylus shaft 1010 is substantially parallel to, and generally coincident with, the touch pattern represented by the virtual vertical line 41. The touch sensing system 112 generates signals corresponding to the sensed location of the shaft 1010 on the touch screen display 45. The touch screen driver 114 converts these signals into spatial touch coordinate information. In the illustrated example, the UI module 116 compares the touch coordinate information to a plurality of predefined touch patterns, and determines that the touch coordinate information corresponds to a touch pattern (represented by the virtual line 41) of a vertical stylus shaft placement gesture located at the vertical center of the touch screen display 45. In exemplary embodiments, the UI module 116 allows for some deviation between the orientation of the stylus 1000 and the touch pattern represented by the virtual vertical line 41. For example, the touch screen UI module 116 can consider an angle between the elongate axis of the shaft of the stylus 1000 and the virtual vertical line 41 of no more than + / - 20 degrees to be an ignorable angle. For example, placement of the stylus 1000 with the shaft 1010 parallel to the virtual vertical line 41 or deviating from the orientation by no more than a threshold directional deviation amount (e.g., 20 degrees) is considered to match the touch pattern represented by the virtual vertical line 41, which corresponds to a vertical stylus shaft placement gesture. Similarly, Figure 5AThe illustrated horizontal dashed line 42 is a virtual horizontal line representing a touch pattern on the main viewing area of the touchscreen display 45 corresponding to a horizontal stylus shaft placement gesture. The stylus 1000 is considered to be placed substantially or generally in a horizontal direction at the center of the main viewing area of the touchscreen display 45 if the stylus 1000 is placed on the virtual line 42 or if the angle of deviation between the elongate axis of the shaft 1010 of the stylus 1000 and the touch pattern represented by the horizontal virtual line 42 does not exceed the defined directional deviation threshold (e.g., 20 degrees).

[0092] In addition to or instead of having a defined angular value tolerance for directional deviation, the UI module 116 can be configured to apply a distance deviation threshold in determining the proximity of a stylus shaft gesture to a displayed marker (e.g., a dividing line described below). For example, if the closest portion of the stylus shaft 1010 is within a distance deviation threshold (e.g., within at most 20% of the total screen width horizontally, and within at most 20% of the total screen width vertically) of any portion of a displayed marker, the UI module 116 can consider the stylus shaft 1010 to be placed on or coincident with the marker. In some examples, the distance deviation threshold can be determined as a function of the average of the length of the stylus shaft relative to the length of the marker. In some examples, both a defined angular directional deviation threshold and a distance deviation threshold can be applied in determining whether a stylus shaft is placed on or coincident with a displayed marker having associated position and directional characteristics (e.g., whether the touch coordinates of a stylus shaft placement gesture are within the directional deviation threshold and within the distance threshold of a dividing line).

[0093] The spatial deviation thresholds described above are examples. Other thresholds can be used, and in some examples can be user-defined. Deviation thresholds can also be applied in classifying movement gestures, for example, in some embodiments a stylus shaft drag gesture need not be perfectly linear, and can include a threshold amount of rotation of the stylus 1010 on the screen during movement. Similarly, a stylus shaft rotation gesture need not be a perfect rotation gesture, and can include a threshold amount of linear dragging of the stylus 1010 on the screen during movement. In some examples, a screen movement gesture that simultaneously exceeds both a screen rotation and a screen linear movement threshold can be classified as a combined on-screen "stylus shaft drag and rotation gesture."

[0094] In exemplary embodiments, the touch pattern classification performed by the UI module 116 can include multiple steps. For example, the UI module 116 can be configured to first classify the touch coordinate information as matching a general touch pattern for placement of the stylus shaft 1010 on the touch screen display 45. For example, for touch coordinate information matching a touch pattern, if the touch pattern corresponds to placement of an elongated rigid body in any position or orientation on the touch screen display 45, it can be classified as a stylus shaft placement gesture. The stylus shaft placement gesture can then be further classified as a vertical or horizontal stylus shaft placement gesture using the orientation determined from the touch coordinate information (e.g., horizontal or vertical), and defining the position of the stylus placement gesture relative to the indicia. As described above, in addition to stylus shaft placement gestures, on-screen stylus shaft movement gestures and stylus shaft removal gestures can also be detected and classified.

[0095] Referring now to Figure 5B , display layout modifications made by the touch screen display system 110 in response to detected stylus shaft gestures are described in accordance with exemplary embodiments.

[0096] Figure 5B An exemplary landscape tablet electronic device 100 is depicted in which the touch screen display 45 displays a home screen user interface 50 generated by the UI module 116 on its main viewing area 102. The home screen user interface 50 displays a plurality of icons 52, each representing a corresponding application 120. The stylus 1000 is placed on the screen with its shaft 1010 in a generally vertical orientation. The touch screen display system 110 operates as follows. The touch sensing system 112 senses the presence and position of the stylus shaft 1010 and provides corresponding signals to the touch screen driver 114, which in turn generates corresponding spatial touch coordinate information, which is processed by the UI module 116. The UI module 116 determines that the touch coordinate information corresponds to a vertical placement of the stylus shaft gesture within the central region of the home screen. In response, the UI module 116 causes the display layout of the presented information to be changed, as shown in Figure 5A to Figure 5B .

[0097] In Figure 4BIn some examples, the icons displayed in the right viewing area 56 can be different from the icons displayed in the left viewing area 54, particularly when there are too many applications to fit all of their icons in a single viewing area. In some embodiments, each of the right viewing area 56 and the left viewing area 54 can be an independent scrollable window, which a user can scroll and select the icons in each area using conventional touch gestures.

[0098] In some examples, the icons displayed in the right viewing area 56 can be different from the icons displayed in the left viewing area 54, particularly when there are too many applications to fit all of their icons in a single viewing area. In some embodiments, each of the right viewing area 56 and the left viewing area 54 can be an independent scrollable window, which a user can scroll and select the icons in each area using conventional touch gestures.

[0099] From Figure 5A the transition illustrates how the vertical split mode is activated using the stylus 1000. In some examples, the stylus 1000 is placed on the screen with its shaft 1010 in a generally horizontal orientation, as previously Figure 5B shown. In this case, the horizontal split mode is activated, and the main viewing area of the touch screen display 45 is split horizontally along the line where the stylus is generally placed. The touch screen display 45 then displays a top viewing area and a bottom viewing area, which operate in the same manner as the left viewing area 54 and the right viewing area 56, respectively, as described above.

[0100] In some examples, the icons displayed in the right viewing area 56 can be different from the icons displayed in the left viewing area 54, particularly when there are too many applications to fit all of their icons in a single viewing area. In some embodiments, each of the right viewing area 56 and the left viewing area 54 can be an independent scrollable window, which a user can scroll and select the icons in each area using conventional touch gestures. Figure 6A Figure 6B ​In the above examples, no background application is running, and the user must manually select an icon 52 of an application in one of the viewing areas (54, 56) to launch the application and display content associated with the application in the corresponding viewing area (54, 56).

[0101] In some examples, the touch screen display system 110 detects a vertical stylus placement gesture or a horizontal stylus placement gesture on the home screen user interface 50, which can cause content from a default application to be automatically displayed in one of the displayed corresponding viewing areas (54, 56) without further user interaction. In some examples, the default application can be automatically launched if it is not currently running in the background.

[0102] For example, the default application can be a commonly used application, such as a notes application, a clock application, a calendar application, or any other application. In some examples, the user can configure the UI module 116 to set the default application to a user-specified application.

[0103] In some examples, if one or more background applications are running on the electronic device 100 when the main viewing area 102 of the touch screen display 45 is split, one or more of the viewing areas can automatically display content from a corresponding one of the background applications. Thus, the touch screen display system 110 detecting a stylus placement gesture on the home screen user interface 50 can cause the UI module 116 to switch a background application to a foreground application, and content from the application to be automatically displayed in one of the displayed corresponding viewing areas (54, 56). Figure 6A and Figure 6B Such examples are illustrated. In Figure 6B the electronic device 100 displays a home screen user interface 50 on the main viewing area 102 of the touch screen display 45. A first application is running in the background of the electronic device 100. A stylus 1000 is placed approximately vertically near the center of the screen of the main viewing area 102 of the touch screen display 45. In response to detecting a vertical stylus placement gesture, the UI module 116 causes the main viewing area 102 to split into a right viewing area 56 and a left viewing area 54, separated by a dividing line 51, as shown in Figure 6C Referring to Figure 6B the left viewing area 54 displays a user interface 610 of the first application, which was running in the background when the stylus 1000 was placed on the screen. The right viewing area 56 displays a user interface 620 of a default application. In the illustrated example, the default application is a notes application.

[0104] In another instance, such as Figure 6A As shown, the electronic device 100 displays a user interface 610 of a first application on its main viewing area 102, the first application running in the foreground of the electronic device 100. When the stylus 1000 is placed on the screen in a generally vertical orientation, the UI module 116 divides the main viewing area 102 into a right viewing area 56 and a left viewing area 54, as described above. Figure 6D The left viewing area 54 displays the user interface 610 of the first application, and the right viewing area 56 displays the user interface 620 of the default application.

[0105] In some examples, when the stylus is combined as described above Figure 6D When placed on the screen, the main viewing area 102 is not divided into sub-areas. Instead, the UI module 116 replaces the main screen user interface 50 with content from an application that fills the entire main viewing area 102. For example, depending on the device configuration, the main screen user interface 50 may be replaced with user interface content from a default application or a background application.

[0106] In another example, Figure 6E and 6E A hybrid method for determining newly created viewing areas is illustrated, showing a touchscreen display after the main viewing area 102 is divided into a left viewing area 54 and a right viewing area 56 in response to a detected vertical placement gesture of the stylus. Instead of running a default application in the right viewing area 56, a main screen user interface 50 is displayed in the right viewing area 56, the main screen user interface including multiple icons 52 representing multiple applications. The user can activate any of the icons 52 to launch the corresponding application, causing the user interface of the corresponding application to be displayed in the right viewing area 56. Figure 7A A first display layout of the icons 52 in the right viewing area 56 is shown, wherein the icons 52 are evenly arranged throughout the right viewing area 56. Figure 7B A second display layout of the icon 52 in the right viewing area 56 is shown, wherein the icon 52 is aligned with the dividing line 51.

[0107] In some embodiments, the segmentation mode of the touchscreen display can be switched from a vertical segmentation mode to a horizontal segmentation mode using the stylus 1000. For example, a touchscreen display already segmented into a right viewing area 56 and a left viewing area 54 is in a vertical segmentation mode. The touchscreen display can be switched to a horizontal segmentation mode, in which it is segmented into a top viewing area and a bottom viewing area, and vice versa. The UI module 116 may have a default configuration that determines the content of the new viewing area relative to the content of the old viewing area. For example, when the viewing area is switched to a horizontal segmentation mode, the UI module 116 can be used to display the content of the left viewing area in the vertical segmentation mode in the top viewing area. Similarly, the content of the right viewing area in the vertical segmentation mode can be displayed in the bottom viewing area in the horizontal segmentation mode.

[0108] In this regard, Figure 7B An electronic device 100 is shown, with a touchscreen display 45 in a vertically split mode. The main viewing area 102 is divided into a left viewing area 54 displaying a first user interface 710 of a first application and a right viewing area 56 displaying a second user interface 720. A stylus 1000 is placed generally horizontally on the screen of the touchscreen display 45. In response to detecting a horizontal stylus placement gesture, the user interface module 116 switches the touchscreen display 45 from a vertically split mode to a horizontally split mode, as shown. Figure 7A As shown. Reference Figure 8A The main viewing area includes a top viewing area 55 and a bottom viewing area 57, separated by a horizontal dividing line 59 approximately at the location where the stylus 1000 contacts the screen of the touchscreen display 45. The user interface of the first application is resized and shown as user interface 710 in the top viewing area 55. Similarly, the bottom viewing area 57 is shown as user interface 720 in the right viewing area before the stylus 1000 is placed on the screen of the touchscreen display 45.

[0109] In another example, the stylus 1010 can be placed substantially vertically on the screen of the touchscreen display 45 in a horizontally split mode, wherein the main viewing area 102 is divided into a top viewing area 55 and a bottom viewing area 57. In this case, the situation occurs in conjunction with the above. Figure 8D and 7BThe description is the opposite. Specifically, the touchscreen display 45 switches from a horizontal split mode to a vertical split mode, wherein the main viewing area is divided into a left viewing area 54 and a right viewing area 56. The left viewing area 54 displays the content displayed in the top viewing area 55 before the stylus is placed vertically on the screen, and the content size is adjusted to fit the left viewing area 54. Similarly, the right viewing area 56 displays the content displayed in the bottom viewing area 57 before the stylus 1000 is placed horizontally on the screen, and the content size is adjusted to fit the right viewing area 56.

[0110] In some examples where two applications can run in the background of the electronic device 100, before the main viewing area of ​​the touchscreen display 45 is split, the UI module 116 is used to bring each of the background applications to the foreground and display its respective user interface in one of the two viewing areas 54 and 56 (in vertical split mode) or viewing areas 55 and 57 (in horizontal split mode). In various examples and configurations, the UI module 116 may determine the content of each newly created viewing area after the main viewing area is split, based on predefined parameters (at least some of which may be configurable by the user).

[0111] In one example, such as Figure 8B As shown, the first and second applications in application 120 run in background mode on the electronic device 100. The main viewing area 102 of the touchscreen display 45 of the electronic device 100 displays a main screen user interface 50 including multiple icons 52. A stylus 1000 is positioned generally vertically on the screen of the touchscreen display. Figure 8D In the exemplary configuration shown, in response to detecting the vertical pen placement gesture, the UI module 116 divides the main viewing area 102 into a right viewing area 56 displaying the user interface 820 of the first application and a left viewing area 54 displaying the user interface 810 of the second application. Accordingly, both background applications become foreground applications and are displayed in the respective left and right viewing areas 54 and 56, respectively.

[0112] In another exemplary configuration, such as Figure 8CAs shown, two foreground applications 120 are running on the electronic device 100. The first application has a user interface 810 displayed in a top viewing area 55, and the second application has a user interface 820 displayed in a bottom viewing area 57. A dividing line 59 separates the top viewing area 55 and the bottom viewing area 57. The stylus 1000 is placed generally vertically on the screen of the touch screen display of the electronic device 100. In response to detecting a vertically oriented stylus barrel placement gesture, the UI module 116 of the electronic device 100 causes the main viewing area 102 of the touch screen display 45 to switch from a horizontal split mode to a vertical split mode, which includes a right viewing area 56 and a left viewing area 54, as shown in Figure 8D The first application user interface 810 that was previously displayed in the top viewing area 55 is displayed in the left viewing area 54. The second application user interface 820 that was previously displayed in the bottom viewing area 57 is displayed in the right viewing area 54.

[0113] In yet another example configuration, as shown in Figure 8A to Figure 8C A first application is running in the foreground of the electronic device 100. The first application has a user interface 810 displayed in the main viewing area 102 of the touch screen display 45 of the electronic device 100. In addition, a second application is running in the background of the electronic device 100. The stylus 1000 is placed generally vertically on the screen of the touch screen display of the electronic device 100. In response to detecting a vertically oriented stylus barrel placement gesture, the UI module 116 causes the main viewing area 102 of the touch screen display 45 to split into a right viewing area 54 and a left viewing area 56, as shown in Figure 9A The user interface 810 of the first application is displayed in the left viewing area 56. The user interface 820 of the second application is displayed in the right viewing area 54. In this case, in response to the main viewing area 102 splitting into a right viewing area 54 and a left viewing area 56, the second application switches from a background mode to a foreground mode.

[0114] In another example similar to the example shown in Figure 9B The stylus 1000 can be placed generally horizontally on the screen of the touch screen display of the electronic device 100. In response to the generally horizontal placement of the stylus 1000, the first application is displayed in a top viewing area and the second application is displayed in a bottom viewing area.

[0115] In some examples, three or more applications can be running on the electronic device. In such cases, the UI module 116 can provide a variety of options for determining the content of a newly created viewing area in response to detecting a stylus pen placement gesture in a vertical or horizontal orientation, in different configurations.

[0116] For example, referring to Figure 9C where a home viewing area 102 of a touch screen display 45 of an electronic device 100 is shown. The home viewing area 102 displays a home screen user interface 50 containing a plurality of icons 52 as described above. A first background application, a second background application, and a third background application 120 are running on the electronic device 100. A stylus pen 1000 is placed on the screen of the touch screen display of the electronic device 100 in a generally vertical orientation. In response to the placement of the stylus pen 1000, the touch screen display system 110 causes the home viewing area 102 of the touch screen display to split into a right viewing area 56 and a left viewing area 54 along the line of contact of the stylus pen 1000 with the screen. The content of the left viewing area 54 and the right viewing area 56 is shown in Figure 9B In each of the left viewing area 54 and the right viewing area 56, a screenshot 910 of the user interface of the first background application is displayed. Similarly, in each of the left viewing area and the right viewing area, a screenshot 920 of the user interface of the second background application is displayed. Further, in each of the left viewing area and the right viewing area, a screenshot 930 of the user interface of the third background application is displayed. The three screenshots (910, 920, and 930) are reduced in size to fit within each of the viewing areas. In addition, the three screenshots do not occupy the entire space of the left viewing area and the right viewing area. The space in the left viewing area 54 and the right viewing area 56 not occupied by the screenshots (910, 920, and 930) is used to display a plurality of icons 52 representing other applications that are executable but not currently running.

[0117] In Figure 9C In one embodiment shown in Figure 9B activating the screenshot 910 in the left viewing area causes the user interface 810 of the first application to be displayed in the left viewing area 56 of the touch screen display, as shown inFigure 9C The screenshot 920 in the right viewing area 54 is activated, as shown. Similarly, the screenshot 920 in the right viewing area 54 is activated, as shown. The user interface 820 of the second application is displayed in the right viewing area 56 of the touch screen display, as shown. Figure 9D Figure 9E In one embodiment (not shown), activating an icon 52 in a viewing area causes the corresponding application to start running and display the user interface of the application to be displayed in the viewing area. Background applications with inactive viewing areas remain running in the background.

[0118] In one embodiment, three or more applications are running on the electronic device 100, one of which is a foreground application and the rest are background applications. For example, referring to Figure 9E , a first application is running in the foreground, a second application is running in the background, and a third application is also running in the background of the electronic device 100. The first application has a user interface 810 that is displayed in the main viewing area of the touch screen display of the electronic device 100. As described above, the stylus 1000 is placed on the screen of the touch screen display in a generally vertical orientation. In response to detecting the placement of the stylus, the touch screen display system 110 splits the main viewing area into a left viewing area 54 and a right viewing area 56, as shown. Figure 9C In the left viewing area 54, the size of the user interface 810 of the first (foreground) application is adjusted and the user interface 810 is displayed. In the right viewing area, the screenshots (920 and 930) of the user interfaces of the second (background) application and the third (background) application are displayed. As described above, the screenshots (920 and 930) can be active user interface elements that, when activated, launch the corresponding application and display the user interface of the application in the right viewing area. For example, activating the screenshot 920 in the right viewing area, the user interface of the second (background) application is displayed in the right viewing area, as shown. Figure 9E Figure 10A Figure 10B The right viewing area 54 shown also includes a plurality of icons 52 of applications that are not running on the electronic device 100. In one embodiment (not shown), activating the icon 52 causes the corresponding application to start running and display the user interface of the application in the right viewing area.

[0119] In another aspect of the application, a method of adjusting the size of the viewing area of a touch screen display 45 of an electronic device 100 is provided. Referring to Figure 11A ​​​, the electronic device 100 has a touch screen display 45 that is divided into a right viewing area 56 and a left viewing area 54. In the right viewing area 56, a plurality of icons 52 are displayed. In the left viewing area 54, a user interface 810 of a foreground application is displayed. A stylus 1000 is placed at a dividing line 51 on the screen of the touch screen display that separates the right viewing area 56 and the left viewing area 54. The touch screen display system 110 determines that a vertical stylus placement gesture occurs at a location that coincides with the dividing line 51. The UI module 116 temporarily locks the location of the dividing line 51 to the location of the stylus 1000 while the stylus remains in a vertical orientation on the screen. The UI module 116 detects a stylus bar drag gesture as the stylus 1000 is slid or dragged in either direction (e.g., left or right) generally perpendicular to the axis of the dividing line 51 and recognizes it as a stylus bar drag gesture. In response to detecting the stylus bar drag gesture, the UI module 116 causes the display layout to change so that the dividing line 51 tracks the left and right movement of the stylus bar 1010, provided that the stylus 1000 remains in contact with the screen of the touch screen display. In the illustrated embodiment, the stylus bar 1010 is slid to the right while remaining in contact with the screen. As a result, as shown in Figure 12 Figure 6B, the dividing line 51 between the right viewing area 56 and the left viewing area 54 moves to the right with the stylus 1000. When the stylus 1000 is lifted off the screen, the UI module 116 detects a stylus bar removal gesture and causes the dividing line 51 between the right viewing area 56 and the left viewing area 54 to remain at the last position on the screen where the stylus 1000 was before it was lifted off the screen. Thus, the stylus bar drag gesture causes the right viewing area 56 to become smaller and the left viewing area 54 to become larger. Thus, the stylus 1000 can be used to adjust the size of the viewing areas of the touch screen display.

[0120] Similarly, when the touch screen display 45 is in a horizontal split mode, a horizontal dividing line 59 divides the main viewing area 102 into a top viewing area 55 and a bottom viewing area 57; in conjunction with a subsequent stylus bar drag gesture (e.g., drag up or down the screen) and a stylus bar removal gesture near the dividing line 59, the horizontal stylus placement gesture can be used to cause the touch screen display system 110 to adjust the size of the top viewing area 55 and the bottom viewing area 57.

[0121] In some example embodiments, the touch screen display system 110 is configured to maximize a viewing area and / or close a viewing area in response to a rigid body (e.g., a stylus shaft 1010) being dragged across the screen of the touch screen display 100. Referring to Figure 12 , the display of the device 100 is split into a right viewing area 56 displaying a plurality of icons 52 and a left viewing area 54 displaying a user interface 810 of a foreground application. The stylus 1000 is placed at the split line 510 on the screen of the display. The stylus 1000 is quickly dragged to the right viewing area and lifted off the screen while being dragged in this direction. Accordingly, the stylus 1000 loses contact with the display while still moving relative to the display. In response to the stylus shaft 1010 being lifted while still moving from left to right, the touch screen display system 110 closes the right viewing area 56 and maximizes the content of the left viewing area 54 (e.g., the user interface 810 of the foreground application) to occupy the main viewing area 102 of the display. Thus, in example embodiments, the touch screen display system 110 is configured to detect a "stylus shaft drag and lift gesture" corresponding to the stylus shaft being slid across the touch screen display and then removed. Depending on the location and direction of the detected stylus shaft drag and lift gesture, the UI module 116 closes one viewing area and expands the other viewing area. In some example configurations, the application corresponding to the closed viewing area can also be closed or terminated, and in some examples, such an application can simply be switched to running as a background application.

[0122] Now, with reference to Figure 8C , the operation of the touch screen display system 110 of the electronic device 100 is described for the above example. Figure 8A is a state diagram illustrating various states of the touch screen display system 110 (including the touch screen display 45) as a user interacts with the screen of the touch screen display 45 using the stylus 1000, in accordance with example embodiments. State 1110 represents a scenario in which the touch screen display 45 is in a non-split mode. The main viewing area 102 of the touch screen display 45 displays a user interface of a single application (e.g., as shown in Figure 4A ), or a home screen user interface including a plurality of icons (e.g., as shown in Figure 4B ).

[0123] State 1115 represents a scenario in which the touch screen display system 110 detects a stylus barrel placement gesture in a non-split mode. As a result, as represented by state 1120, the touch screen display system 110 determines a new display layout and content in response to the detected stylus barrel placement gesture. In particular, as known from the examples described above, in example embodiments, the UI module 116 is configured to select a new display layout (e.g., a vertical split mode or a horizontal split mode) and content to display in the viewing area(s) associated with the selected display mode in accordance with one or more of the following: (1) a direction of the detected stylus barrel placement gesture (e.g., a vertical direction or a horizontal direction); (2) a position of the stylus barrel 1010 on the screen within the display (e.g., coinciding with a center vertical axis or a center horizontal axis); (3) current display content at the time the stylus barrel placement gesture is detected (e.g., whether an OS home screen user interface is displayed in the viewing area or whether an application user interface of a foreground application is displayed in the viewing area); (4) current application state information (e.g., a number and names of applications currently running as background applications; a number and names of applications currently running as foreground applications); and (5) predefined parameters (e.g., which inactive application is a default application).

[0124] In the example in which the current state is a non-split mode, detecting a stylus barrel placement gesture that is substantially vertically or horizontally aligned with a middle region of the touch screen display 45 (e.g., as shown in Figs. 11A and 11B, respectively) causes the touch screen display system 110 to determine a new display layout that is a vertical split mode or a horizontal split mode, respectively. This new state and mode is represented by state 1125. State 1125 represents a scenario in which the touch screen display 45 has a first viewing area and a second viewing area (e.g., right viewing area 56 and left viewing area 54, or top viewing area 55 and bottom viewing area 57) and the stylus 1010 remains positioned on the screen. In state 1120, the content of the first viewing area and the second viewing area in state 1125 (e.g., a home screen user interface, a user interface of an application previously running in the background, and / or a user interface of a newly launched default application) is determined in accordance with the examples described above. Figure 10A Figure 10B

[0125] ​​State 1135 represents a scenario in which the touch screen display system 110 detects a stylus shaft removal gesture, in this case, the stylus shaft has been held in the same position on the screen for at least a defined threshold time (e.g., 0.5 seconds). In such a scenario, the first and second viewing areas maintain their current relative sizes, which are determined by the last position of the stylus on the screen (state 1145).

[0126] State 1130 represents a scenario in which the touch screen display system 110 detects that (1) the stylus shaft was previously placed on the touch screen display 45 in a position that substantially coincides with the dividing line 51 or 59 between the two viewing areas, the stylus shaft is dragged along the screen to a new position while maintaining its general orientation (e.g., a right or left drag of the stylus shaft gesture when the stylus is in a generally vertical orientation; or, an up or down drag of the stylus shaft gesture when the stylus is in a generally horizontal orientation); and (2) after being dragged, the stylus shaft 1010 remains on the screen for at least a predefined time (e.g., 0.5 seconds) before being removed from the screen. In such a case, the touch screen display system 110 proportionally adjusts the relative sizes of the viewing areas to each other in accordance with the distance and direction in which the stylus shaft 1010 was dragged (state 1140), as described above with respect to Figure 11A and Figure 11B

[0127] State 1150 represents a scenario in which the touch screen display system 110 detects that the stylus shaft placement gesture substantially coincides with the dividing line (51 or 59) while the touch screen display is in split screen mode. A state that can follow state 1150 is state 1130, in which a stylus shaft drag gesture is detected that results in an adjustment of the sizes of the split mode viewing areas (state 1140). Another state that can follow state 1150 is state 1155, in which the touch screen display system 110 detects a stylus shaft drag and removal gesture, which corresponds to dragging the stylus shaft on the screen and removing the stylus shaft within a threshold time (e.g., 0.5 seconds) of the termination of the drag action. In such a case, the touch screen display system 110 restores the touch screen display 45 to a non-split mode (state 1110), maximizes one of the viewing areas to fill the entire primary viewing area, and closes the other viewing area, as described above with respect to Figure 13 and Figure 13

[0128] ​​The above description discusses manipulation of a viewing area of a display. In some examples, the viewing area can correspond to a user interface window. Further, in some examples, multiple viewing areas or windows can simultaneously display user interfaces from the same application. For example, a word processing application can have a single document window displaying a first document. Placement of a stylus along the application window can cause the application to open a second document window within the area of the original application window. Similarly, a spreadsheet application displaying a single table can create a second window containing a blank spreadsheet in response to placement of a stylus along its window. For another example, placement of a stylus along a border of an application window and then moving the stylus can resize the window in the same manner as resizing a viewing area as described above. For yet another example, the stylus can be placed on a window border, dragged away from the window content and toward a border of the viewing area, and then lifted while the stylus is being dragged, thereby maximizing the application window.

[0129] Although the illustrated example depicts splitting the primary viewing area of the display into two viewing areas by placing the stylus shaft on the display screen, those skilled in the art will appreciate that other splitting of the viewing area is contemplated in the present application. For example, after splitting the primary viewing area into a right viewing area and a left viewing area, the stylus can be placed horizontally on the right viewing area. This can split the right viewing area into an upper right viewing area and a lower right viewing area. The content of each newly created viewing area can be determined using the methods described above.

[0130] In at least some examples, enabling the processing of stylus shaft gestures as described herein can improve one or both of the operation of the electronic device 100 and the user experience of the electronic device 100. For example, facilitating more efficient user interaction with the electronic device 100 through the use of stylus shaft gestures can enable desired screen display layouts to be achieved through fewer user interactions, thereby reducing wear and tear or damage that can be caused to the electronic device 100 and potentially reducing battery power consumption. Further, users can be able to replace some finger interactions with the touch screen display 45 with stylus interactions, thereby reducing the potential transfer of dirt, grease, oil, and other contaminants (including bacteria and viruses, etc.) and other foreign matter from the user’s fingers into the touch screen display 45. In some cases, reducing contaminants on the screen can reduce the cleaning requirements of the touch screen display, thereby reducing damage that can be caused to the device, reducing the consumption of cleaning materials, and can also reduce the spread of contaminants.

[0131] Figure 13is a block diagram of an example processing device 170 that includes components of the touch screen display system 110 and can be used to implement the electronic device 100. The processing device 170 can be used to execute machine-readable instructions to implement the methods and examples described herein. Other processing units suitable for implementing embodiments described herein can be used, which can include different components than those discussed below. Although ​ A single instance of each component is shown, but there can be multiple instances of each component in the processing device 170.

[0132] The processing device 170 can include one or more processing units 172, such as a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a tensor processing unit, a neural network processing unit, or a combination thereof. The processing device 170 can also include one or more input / output (I / O) interfaces 174 that can interface with one or more appropriate input devices 184 and / or output devices 186. The processing device 170 can include one or more network interfaces 176 for wired or wireless communication with a network (e.g., an intranet, the Internet, a P2P network, a WAN, and / or a LAN) or other nodes. The multiple network interfaces 176 can include wired links (e.g., Ethernet lines) and / or wireless links (e.g., one or more antennas) for intranet and / or internetwork communication.

[0133] The processing device 170 can also include one or more storage units 178, which can include mass storage units, such as solid state drives, hard disk drives, disk drives, and / or optical drives. The processing device 170 can include one or more memories 180, which can include volatile memory (e.g., random access memory (RAM)) and non-volatile or non-transitory memory (e.g., flash memory, magnetic memory, and / or read-only memory (ROM)). The non-transitory memory of the memory 180 stores programs 113, including software instructions for execution by the processing unit 172, for implementing examples described herein and the like. In example embodiments, the programs 113 include software instructions for implementing an operating system (OS) 108 (which, as described above, can include a touchscreen driver 114, a UI module 116, and a display driver 118, among other OS components) and other application programs / functions 120. In some examples, the memory 180 can include software instructions for the system 100 that are executed by the processing unit 172 to implement methods described herein. In some other examples, one or more data sets and / or modules can be provided by external memory (e.g., an external drive in wired or wireless communication with the processing device 170) or by a transitory or non-transitory computer-readable medium. Examples of non-transitory computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.

[0134] A bus 182 can be present to provide communication between components of the processing device 170, including the processing unit 172, the I / O interface 174, the network interface 176, the storage unit 178, and / or the memory 180. The bus 182 can be any suitable bus architecture, including a memory bus, a peripheral bus, or a video bus, for example.

[0135] In ​In particular embodiments, the input device 184 includes the touch sensing system 112 of the touch screen display 45, and can also include other input devices (e.g., a keyboard, a mouse, a microphone, an accelerometer, and / or a keypad). The output device 186 includes the display 128 of the touch screen display 24, and can also include other devices, such as a speaker and a haptic generator.

[0136] Although the present application describes methods and processes with steps in a certain order, one or more steps of the methods and processes can be omitted or altered as appropriate. One or more steps can be performed in an order other than the order described.

[0137] Although the present application is described, at least in part, in terms of methods, one of ordinary skill in the art will understand that the application has a range of applications beyond those described, including in various combinations with hardware components, software, or any combination thereof. Accordingly, the technical solutions of the present application can be embodied in the form of a software product. A suitable software product can be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer-readable medium, including a DVD, a CD-ROM, a USB flash disk, a mobile hard disk, or other storage medium, etc. The software product includes instructions tangibly stored thereon, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform the method examples disclosed herein.

[0138] The present application can be embodied in other specific forms without departing from the subject matter of the claims. The described exemplary embodiments are to be considered in all respects as illustrative only and not restrictive. Selected features of the above-described embodiments can be combined in alternative embodiments to create alternative embodiments not explicitly described, and all such combinations are understood to fall within the scope of the present application.

[0139] All values and subranges within the disclosed ranges are also disclosed. In addition, although the systems, devices, and processes disclosed and shown herein can include particular numbers of elements / components, these systems, devices, and components can be modified to include more or less such elements / components. For example, although any of the disclosed elements / components can be singular in number, embodiments disclosed herein can be modified to include multiple such elements / components. The subject matter described herein is intended to encompass and survive any reasonable adaptations of the technology.

Claims

1. A method characterized by, The method comprises: sensing a first touch event on a touch screen display of an electronic device; in response to determining that touch coordinate information of the first touch event matches a stylus shaft placement gesture, the stylus shaft placement gesture corresponding to placement of a stylus shaft within a first viewing area on a screen of the touch screen display, splitting the first viewing area into a second viewing area and a third viewing area, and causing each of the second viewing area and the third viewing area to display a respective user interface, the stylus shaft being positioned between a first end and a second end of the stylus.

2. The method of claim 1, wherein, when the stylus shaft placement gesture corresponds to placement of the stylus shaft within the first viewing area in a vertical orientation, the second viewing area and the third viewing area are arranged opposite each other as a left viewing area and a right viewing area.

3. The method of claim 2, wherein, when the stylus shaft placement gesture corresponds to placement of the stylus shaft within the first viewing area in a horizontal orientation, the second viewing area and the third viewing area are arranged opposite each other as a top viewing area and a bottom viewing area.

4. The method according to any one of claims 1 to 3, characterized in that, at least one of the user interfaces within the second viewing area and the third viewing area corresponds to an application that was a background application when the first touch event occurred; the method comprises causing the background application to become a foreground application upon determining that the first touch event matches the stylus shaft placement gesture.

5. The method according to any one of claims 1 to 3, characterized in that, at least one of the user interfaces within the second viewing area and the third viewing area corresponds to an application that was an inactive application when the first touch event occurred; the method comprises causing the inactive application to become a foreground application upon determining that the first touch event matches the stylus shaft placement gesture.

6. The method according to any one of claims 1 to 3, characterized in that, at least one of the user interfaces within the second viewing area and the third viewing area displays a set of icons, each of the icons being respectively activatable to launch a corresponding application.

7. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: in response to determining that touch coordinate information of a second touch event matches a stylus shaft drag gesture, the stylus shaft drag gesture corresponding to movement of the stylus shaft along a screen of the touch screen display from a position coinciding with a boundary between the second viewing area and the third viewing area; adjusting relative sizes of the second viewing area and the third viewing area in accordance with the movement of the stylus shaft along the screen.

8. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: in response to determining that touch coordinate information of a third touch event matches a stylus shaft drag and removal gesture, the stylus shaft drag and removal gesture corresponding to movement of the stylus shaft along a screen of the touch screen display within the second viewing area and subsequent removal of the stylus shaft from the screen within a threshold period of time of the movement; enlarging the second viewing area and closing the third viewing area.

9. An electronic device, comprising: The electronic device comprises: a touch screen display, the touch screen display comprising a display and a touch sensing system for generating signals corresponding to touching of a screen of the display; a processing device operatively coupled to the touch screen display; a non-transitory memory coupled to the processing device and storing software instructions that, when executed by the processing device, cause the processing device to: sense a first touch event on the touch screen display according to signals generated by the touch sensing system; in response to determining that touch coordinate information of the first touch event matches a stylus shaft placement gesture, the stylus shaft placement gesture corresponding to placement of the stylus shaft within a first viewing area on a screen of the touch screen display, split the first viewing area into a second viewing area and a third viewing area, and cause each of the second viewing area and the third viewing area to display a respective user interface, change the display layout presented on the touch screen display, the stylus shaft being located between a first end and a second end of the stylus.

10. The electronic device of claim 9, wherein, when the stylus shaft placement gesture corresponds to placement of the stylus shaft in a vertical orientation within the first viewing area, the second viewing area and the third viewing area are arranged opposite to each other as a left viewing area and a right viewing area.

11. The electronic device of claim 10, wherein, when the stylus shaft placement gesture corresponds to placement of the stylus shaft in a horizontal orientation within the first viewing area, the second viewing area and the third viewing area are arranged opposite to each other as a top viewing area and a bottom viewing area.

12. The electronic device of any of claims 9-11, wherein, at least one of the user interfaces within the second viewing area and the third viewing area corresponds to an application that is a background application at a time when the first touch event occurs; the processing device is configured to cause the background application to become a foreground application upon determining that the first touch event matches the stylus shaft placement gesture.

13. The electronic device of any of claims 9-11, wherein, at least one of the user interfaces within the second viewing area and the third viewing area corresponds to an application that is an inactive application at a time when the first touch event occurs; the processing device is configured to cause the inactive application to become a foreground application upon determining that the first touch event matches the stylus shaft placement gesture.

14. The electronic device of any of claims 9-11, wherein, at least one of the user interfaces within the second viewing area and the third viewing area displays a set of icons, each of the icons being activatable to launch a respective application.

15. The electronic device of any one of claims 9-11, wherein: the processing device is further configured to: in response to determining that touch coordinate information of a second touch event matches a stylus shaft drag gesture, the stylus shaft drag gesture corresponding to movement of the stylus shaft along a screen of the touch screen display from a position coinciding with a boundary between the second viewing area and the third viewing area; adjust relative sizes of the second viewing area and the third viewing area according to the movement of the stylus shaft along the screen.

16. The electronic device of any one of claims 9-11, wherein: the processing device is further configured to: in response to determining that the touch coordinate information of the third touch event matches a stylus shaft drag and remove gesture, the stylus shaft drag and remove gesture corresponding to moving the stylus shaft along a screen of the touch screen display within the second viewing area and subsequently removing the stylus shaft from the screen within a threshold period of time of the moving; enlarging the second viewing area and closing the third viewing area.

17. A computer readable medium storing instructions, wherein: The instructions, when executed by a processor of an electronic device, configure the electronic device to perform the method of any of claims 1-8.

18. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computer, cause the computer to perform the method of any of claims 1-8.

Images