Managing user interface items in a visual user interface (VUI)

Abstract

A method of managing user interface items in a visual user interface (VUI) is disclosed. In one aspect, the method includes displaying a VUI screen on a display. The VUI screen includes a scrollable region including a plurality of selectable items. The VUI screen provides a pinning bar located at an edge of a display area of the display. A state of the pinning bar and a second pinning bar persists in response to a change in a visual state of the scrollable region. In response to detecting a pinning command, a selected item of the plurality of selectable items is pinned to the pinning bar. In response to detecting a corresponding input, a visual state of the scrollable region is changed. In response to detecting an unpinning command, a pinned item corresponding to a selected pin from the pinning bar is unpinned.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 17 / 063,332, filed on October 5, 2020, entitled “MANAGING USER INTERFACE ITEMS IN A VISUAL USER INTERFACE (VUI),” the contents of which are incorporated herein by reference as if reproduced in their entirety. Technical Field

[0003] The present invention relates generally to visual user interfaces and, more particularly, to managing user interface items. Background Art

[0004] Scrollable lists are a well-known feature of mobile operating systems. Each mobile operating system has its own interaction technique for reordering list items (i.e., changing the position of items in a scrollable list). However, existing interaction techniques do not perform well for long scrollable lists with many items. For long scrollable lists, interaction typically requires the user to press and hold a list item while scrolling the scrollable list by moving their finger to the bottom or top edge of the screen. This makes the interaction complex and cumbersome. Users often miss their target location and lose track of the list items. This interaction is often time-consuming because the speed at which the list scrolls when reordering is typically not the normal scrolling speed. In addition, the interaction is typically limited to reordering one list item at a time. Therefore, there is a need for improved management of user interface items. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram of a computing device suitable for implementing the teachings of the present invention.

[0006] Figures 2A-2D The configuration of a fixed VUI screen provided by an embodiment of the present invention is shown.

[0007] Figure 3 The figure is a flowchart of a method for managing user interface items in a VUI provided by one embodiment of the present invention.

[0008] Figures 4A-4I Shown according to the use Figure 3 A first example of a method of user interaction illustrates a sequence of pinnable VUI screens.

[0009] Figures 5A-5D Shown according to the use Figure 3 A second example of a method of user interaction is shown with a sequence of pinnable VUI screens.

[0010] Figures 6A-6E A sequence of a pinable VUI screen showing a third example of user interaction according to a method of using Figure 3

[0011] Figure 7 is a flowchart of a method of managing user interface items in a VUI provided by another embodiment of the present application.

[0012] Figure 8A and Figure 8B shows a computing device with a pinable VUI screen provided by an embodiment of the present application.

[0013] Figure 9 is a flowchart of a method of managing user interface items in a VUI provided by yet another embodiment of the present application.

[0014] Figures 10A-10F A sequence of a pinable VUI screen showing an example of user interaction according to a method of using Figure 9 SUMMARY

[0015] The present application provides a method, apparatus and non-transitory machine readable medium for managing user interface (UI) items in a VUI. The present application provides a pinning method that can be used to reorder UI items in a list (i.e., “list items”) displayed in a VUI, preserve the position of list items in a list during list operations, or share content across applications through at least one pinning bar. The pinning bar is provided by a portion of a pinable VUI screen at one or more edges of a dynamic content area (e.g., a scrollable area) of the pinable VUI screen, which can include a list. When the dynamic content area is a scrollable area, the pinning bar can extend parallel to a scrollable direction of the scrollable area of the pinable VUI screen. A user can use a designated gesture (e.g., swipe, drag, or long press) to move a list item onto the pinning bar to pin the list item. The pinned UI item remains connected to a fixed position on the pinable VUI screen and is not affected by any changes in the visual state of the VUI screen (e.g., changes in the visual state of the list), such as changes triggered by scrolling, list operations (e.g., sorting), or switching between collaborative applications that interact with the pinning bar.

[0016] ​​Pinning bars can form horizontal bars, vertical bars, or frames at the edges of a VUI screen. In some embodiments, the pinning method can reposition one or more UI items (e.g., list items) in a manner that reduces time requirements, effort, and time needed compared to current reordering techniques and improves accuracy of repositioning UI items compared to current reordering techniques. The pinning method enables a user to set aside UI items that the user wishes to reposition within a dynamic content area (e.g., a list), change the visible state of the VUI screen (e.g., a list view, grid view, or gallery view) by scrolling or performing another operation, and then reposition (e.g., replace, unpin, insert, or add) the UI items back into the dynamic content area. Pinned UI items can be repositioned individually, in groups, or all at once to the same or different VUI screen of the pinned UI items. As described above, pinned UI items are not affected by operations that cause changes in the visual state of the VUI screen and can be accessed at any time from their location on the pinning bar of the pinned UI items. Pins can be repositioned or moved on the pinning bar.

[0017] According to a first embodiment of the first aspect of the present application, there is provided a method of managing user interface items in a visual user interface (VUI) performed by a processor of a computing device. The method includes displaying a first VUI screen of a first application on a display. The first VUI screen includes a dynamic content area that includes selectable content. The first VUI screen provides a first pinning bar located at a first side of a display area of the first VUI screen and a second pinning bar located at a second side of the display area of the first VUI screen opposite the right side. The first and second pinning bars persist across VUI screens of different applications. The method further includes, in response to detecting a pinning command, pinning selected content from the dynamic content area to the first or second pinning bar. Pinning includes displaying a pin in the corresponding one of the first or second pinning bar on the display and storing the selected content associated with the pin in a memory of the computing device. The method further includes, in response to detecting an input to switch an active application from the first application to a second application, displaying a second VUI screen of the second application on the display, wherein the second VUI screen includes an input bar. The state of the first and second pinning bars persists across the first VUI screen of the first application and the second VUI screen of the second application. The method further includes, in response to detecting an unpinning command, adding stored content corresponding to one or more pins located in the first or second pinning bar to the input bar.

[0018] In some or all examples of the first embodiment of the first aspect, the first pinning bar and the second pinning bar are provided in response to a corresponding input.

[0019] In some or all examples of the first embodiment of the first aspect, the first pinning bar is located at a left edge of the dynamic content area and the second pinning bar is located at a right edge of the dynamic content area.

[0020] In some or all examples of the first embodiment of the first aspect, the dynamic content area of the first VUI screen and the input bar of the second VUI screen are scrollable in a vertical direction relative to a screen orientation. In some or all examples of the first embodiment of the first aspect, the first pinning bar and the second pinning bar are defined by a vertically extending area at the left edge of the display area of the display and the right edge of the display area of the display.

[0021] In some or all examples of the first embodiment of the first aspect, the first pinning bar and the second pinning bar are provided without visual representation.

[0022] In some or all examples of the first embodiment of the first aspect, the stored content corresponding to the one or more pins is automatically arranged according to predefined visual layout rules based on a position of the one or more pins in the first pinning bar or the second pinning bar.

[0023] In some or all examples of the first embodiment of the first aspect, the stored content corresponding to the one or more pins is automatically arranged according to predefined visual layout rules based on a size of the one or more pins in the first pinning bar or the second pinning bar.

[0024] In some or all examples of the first embodiment of the first aspect, the state of the first pinning bar and the second pinning bar comprises a position of a pin located on the first pinning bar and the second pinning bar.

[0025] In some or all examples of the first embodiment of the first aspect, the state of the first pinning bar and the second pinning bar comprises a size of a pin located on the first pinning bar and the second pinning bar.

[0026] In some or all examples of the first embodiment of the first aspect, the state of the first pinning bar and the second pinning bar persists in response to a change in visual state of the dynamic content area and persists across applications.

[0027] In some or all examples of the first embodiment of the first aspect, the method further comprises: in response to a corresponding input associated with a selected pin, invoking a context sensitive menu, wherein the context sensitive menu comprises a plurality of menu options; in response to an input selecting an option of the plurality of menu options, performing an action.

[0028] In some or all examples of the first embodiment of the first aspect, the plurality of menu options comprises a delete option for deleting the selected pin, an access option for accessing a source associated with the selected pin, and a search option for searching content associated with the selected pin.

[0029] In some or all examples of the first embodiment of the first aspect, the method further comprises: in response to detecting a corresponding input associated with a respective pin, performing an action associated with the stored content corresponding to the pin.

[0030] According to a second embodiment of the first aspect of the present application, there is provided a method of managing user interface items in a visual user interface (VUI) performed by a processor of a computing device. The method comprises displaying a VUI screen on a display. The VUI screen comprises a scrollable region comprising a plurality of selectable items. The VUI screen provides a pinning bar located at an edge of a display area of the display. The state of the pinning bar and a second pinning bar persists in response to a change in a visual state of the scrollable region. In response to detecting a pinning command, a selected item of the plurality of selectable items is pinned to the pinning bar. In response to detecting a corresponding input, a visual state of the scrollable region is changed. In response to detecting an unpinning command, a pinned item corresponding to a selected pin from the pinning bar is unpinned.

[0031] In some or all examples of the second embodiment of the first aspect, the pinning bar is provided in response to a corresponding input.

[0032] In some examples of the second embodiment of the first aspect, the selectable items are one-dimensionally ordered and arranged.

[0033] In some examples of the second embodiment of the first aspect, the selectable items are two-dimensionally ordered and arranged.

[0034] In some or all examples of the second embodiment of the first aspect, the pinning comprises: displaying a pin in the pinning bar proximate to a location of the selected selectable item, the selected item corresponding to the pin; storing the selected item associated with the pin in a memory of the computing device; deleting the selected item from the scrollable region; and displaying additional items not previously displayed.

[0035] In some or all examples of the second embodiment of the first aspect, the unpinning comprises: adding the stored item corresponding to the selected pin to the scrollable area at a position in the scrollable area corresponding to a position of the selected pin; removing items currently displayed in the scrollable area to accommodate the added item; removing the selected pin from the pinning bar.

[0036] In some or all examples of the second embodiment of the first aspect, the selectable items are ordered and arranged in a sequence, the additional item displayed being a subsequent item in the sequence.

[0037] In some or all examples of the second embodiment of the first aspect, the unpinning comprises: inserting the stored item corresponding to the selected pin to the scrollable area at a position in the sequence corresponding to a position of the selected pin; removing items currently displayed in the sequence to accommodate the inserted item; removing the selected pin from the pinning bar.

[0038] In some or all examples of the second embodiment of the first aspect, the currently displayed item removed is a last item in the sequence.

[0039] In some examples of the second embodiment of the first aspect, the stored item is inserted before items in the sequence proximate to the position of the pin.

[0040] In some examples of the second embodiment of the first aspect, the stored item is inserted after items in the sequence proximate to the position of the pin.

[0041] In some examples of the second embodiment of the first aspect, the pinning command is a specified interaction associated with a selectable item, wherein the interaction of the pinning command comprises detecting a swipe gesture in a direction of the pinning bar.

[0042] In some examples of the second embodiment of the first aspect, the unpinning command is a specified interaction associated with a pin in the pinning bar, wherein the interaction of the pinning command comprises detecting a swipe gesture in a direction of the scrollable area and away from the pinning bar.

[0043] In some or all examples of the second embodiment of the first aspect, the method further comprises: in response to detecting a repositioning input, repositioning a pin in the pinning bar.

[0044] In some or all examples of the second embodiment of the first aspect, the repositioning input is a specified interaction performed in association with a pin in the pinning bar, wherein the interaction of the pinning command includes detecting a drag gesture in a scrollable direction.

[0045] In some or all examples of the second embodiment of the first aspect, changing the visual state of the scrollable region includes scrolling the scrollable region in response to detecting a scroll input.

[0046] In some or all examples of the second embodiment of the first aspect, changing the visual state of the scrollable region includes performing a list operation on the selectable item the scrollable region in response to detecting a corresponding input.

[0047] According to a first embodiment of a second aspect of the present application, there is provided a computing device providing a display, a memory, and a processor system including at least one processor coupled to the display and the memory. The processor system is to cause display of a first VUI screen of a first application. The first VUI screen includes a dynamic content region including selectable content. The first VUI screen provides a first pinning bar located at a first side of a display area of the first VUI screen and a second pinning bar located at a second side of the display area of the first VUI screen opposite the right side. The first pinning bar and the second pinning bar persist across VUI screens of different applications. The processor system is further to, in response to detecting a pinning command, pin selected content from the dynamic content region to the first pinning bar or the second pinning bar. Pinning includes displaying a pin in the corresponding one of the first pinning bar or the second pinning bar on the display and storing the selected content associated with the pin in the memory of the computing device. The processor system is further to cause display of a second VUI screen of a second application in response to detecting an input to switch an active application from the first application to the second application, wherein the second VUI screen includes an input bar. The state of the first pinning bar and the second pinning bar persists across the first VUI screen of the first application and the second VUI screen of the second application. The processor system is further to, in response to detecting an unpinning command, add stored content corresponding to one or more pins located in the first pinning bar or the second pinning bar to the input bar.

[0048] According to a second embodiment of the second aspect of the application, there is provided a computing device providing a display, a memory, and a processor system including at least one processor coupled to the display and the memory. The processor system is configured to cause display of a VUI screen on the display. The VUI screen includes a scrollable region including a plurality of selectable items. The VUI screen provides a pinning bar located at an edge of a display area of the display. The state of the pinning bar and a second pinning bar persists in response to a change in a visual state of the scrollable region. The processor system is further configured to, in response to detecting a pinning command, pin a selected item of the plurality of selectable items to the pinning bar. The processor system is further configured to, in response to detecting a corresponding input, cause a change in the visual state of the scrollable region. The processor system is further configured to, in response to detecting an unpinning command, unpin a pinned item corresponding to a selected pin from the pinning bar.

[0049] According to a first embodiment of the third aspect of the application, there is provided a non-transitory machine-readable medium having tangibly stored thereon executable instructions for execution by a processor system of a computing device. The processing system includes at least one processor, wherein, in response to execution of the processor system, the executable instructions cause the processor system to cause display of a first VUI screen of a first application. The first VUI screen includes a dynamic content region including selectable content. The first VUI screen provides a first pinning bar located at a first side of a display area of the first VUI screen and a second pinning bar located at a second side of the display area of the first VUI screen opposite the right side. The first pinning bar and the second pinning bar persist across VUI screens of different applications. The processor system is further configured to, in response to detecting a pinning command, pin selected content from the dynamic content region to the first pinning bar or the second pinning bar. Pinning includes displaying a pin in the corresponding one of the first pinning bar or the second pinning bar on the display and storing the selected content associated with the pin in a memory of the computing device. The processor system is further configured to cause, in response to detecting an input to switch an active application from the first application to a second application, display of a second VUI screen of the second application, wherein the second VUI screen includes an input bar. The state of the first pinning bar and the second pinning bar persists across the first VUI screen of the first application and the second VUI screen of the second application. The processor system is further configured to, in response to detecting an unpinning command, add stored content corresponding to one or more pins located in the first pinning bar or the second pinning bar to the input bar.

[0050] According to a second embodiment of the third aspect of the application, there is provided a non-transitory machine-readable medium having stored thereon executable instructions for execution by a processor system of a computing device. The processing system includes at least one processor, wherein, in response to execution of the processor system, the executable instructions cause the processor system to cause display of a VUI screen on a display. The VUI screen includes a scrollable area including a plurality of selectable items. The VUI screen provides a pinning bar located at an edge of a display area of the display. A state of the pinning bar and a second pinning bar persists in response to a change in a visual state of the scrollable area. The processor system is further to, in response to detecting a pinning command, pin a selected item of the plurality of selectable items to the pinning bar. The processor system is further to, in response to detecting a corresponding input, cause a change in the visual state of the scrollable area. The processor system is further to, in response to detecting an unpinning command, unpin a pinned item corresponding to a selected pin from the pinning bar.

[0051] Other aspects and features of the present application will become apparent to those ordinarily skilled in the art upon review of the following description of specific implementations in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0052] The present application is made with reference to the accompanying drawings in which are shown embodiments. Many different embodiments can, however, be made and therefore the description should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete. Wherever possible, the same reference numbers used in the drawings and the following description refer to the same elements. Numbers in the figures refer to the same elements throughout. The function of the elements of the illustrated systems and devices can not require physical separation, as communication between the elements can occur through messaging, function calls, shared memory space, etc., without any such physical separation. As such, although functions are shown separately for ease of explanation, these functions need not be implemented in physically or logically separate platforms. Different devices can have different designs such that while some devices implement some functions in fixed function hardware, other devices can implement these functions in programmable processors having code obtained from a machine-readable medium. Finally, elements referred to in the singular can be plural and vice versa, unless explicitly stated otherwise or the context clearly indicates otherwise.

[0053] Reference is first made to Figure 1FIG. 1 illustrates a computing device 102 suitable for implementing the guidance of the present application. The computing device 102 can be a multi-purpose communication device, such as a wired or wireless communication device. Examples of the computing device 102 include, but are not limited to, a smartphone, a tablet, a personal camera or camera peripheral, a personal computer (e.g., a desktop or a laptop), smart glasses or other display mounted on a head-mounted device, a smart speaker or other smart or Internet of Things (IoT) device (e.g., a smart appliance), and the like.

[0054] The computing device 102 includes a processing system that includes a processor 104 (e.g., a microprocessor or central processing unit (CPU)) that controls the overall operation of the computing device 102. The processing system can include one or more other types of processors, such as a graphic processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), an application-specific integrated circuit, or a field programmable gate array (FPGA), coupled to the processor 104 for offloading certain compute tasks. The processor 104 is coupled to a number of components over a communication bus (not shown), which provides a communication path between the components and the processor 104. The processor 104 is coupled to a random access memory (RAM) 108, a read only memory (ROM) 110, and a persistent (non-volatile) memory 112, such as flash memory, and a communication subsystem 130. The communication subsystem 130 includes one or more wireless transceivers for exchanging radio frequency signals with a wireless network. The communication subsystem 130 can also include a wired transceiver for wired communication with a wired network. The wireless transceivers can include a Bluetooth transceiver or other short-range wireless transceiver, a Wi-Fi or other WLAN transceiver for communicating with a WLAN through a WLAN access point (AP), or a wireless wide area network (WW AN) transceiver, such as a cellular transceiver, for communicating with a wireless access network, such as a cellular network. The cellular transceiver can communicate with any of a plurality of fixed transceiver base stations of the cellular network within its geographic coverage area. The wireless transceivers can include multi-band cellular transceivers that support multiple radio frequency bands. Other types of short-range wireless communication include near field communication (NFC), IEEE 802.15.3a (also known as ultrawideband (UWB)), Z-Wave, ZigBee, ANT / ANT+, or infrared (e.g., infrared data association (IrDA) communication). The wireless transceivers can include a satellite receiver for receiving satellite signals from a satellite network that includes a plurality of satellites as part of a global or regional satellite navigation system.

[0055] The computing device 102 also includes a microphone 132 and a speaker 134 coupled to the processor 104. The computing device 102 can also include a camera 136, a touch screen 138, and a satellite receiver 140 for receiving satellite signals from a satellite network, depending on the type of computing device 102. The touch screen 138 is typically a capacitive touch screen. The computing device 102 can also include a plurality of sensors 120 coupled to the processor 104. The sensors 120 can include a biometric sensor 122 (e.g., a facial scanner or a fingerprint scanner), a motion sensor 124 (e.g., an accelerometer), an infrared (IR) sensor 126, and / or a proximity sensor 128. In possible embodiments, the sensors 120 can include other sensors (not shown), such as an orientation sensor, an electronic compass, or an altimeter. Depending on the type of computing device 102, the computing device 102 can also include one or more other input devices 142, such as buttons, switches, dials, a keyboard or keypad, or a navigation tool.

[0056] Operating system software 150, which can be executed by the processing system including the processor 104, is stored in the persistent memory 112, but can be stored in other types of memory devices, such as the ROM 108 or similar storage elements. The operating system software 150 provides a visual user interface (VUI) for user interaction with the computing device 102 in the form of touch input detected via the touch screen 138 and / or other input devices 142. A plurality of application programs 152, which can be executed by the processing system including the processor 104, are also stored in the persistent memory 112. The application programs 152 include a fixed application 170. Alternatively, the fixed application 170 can be part of the operating system software 150, such as part of the VUI, or an application programming interface (API). The fixed application 170 includes instructions for enabling a user to use a fixed management user interface (UI) item in accordance with the guidance of the present application (e.g., the methods 300, 700, and 900).

[0057] Memory 112 stores various data 154, including sensor data acquired by the plurality of sensors 120, including sensor data acquired by the biometric sensor 122, sensor data acquired by the motion sensor 124 (i.e., an accelerometer, gyroscope, or inertial measurement unit (IMU) of the computing device 102), sensor data acquired by the infrared sensor 126, and sensor data acquired by the proximity sensor 128. Memory 112 also stores input data 156 acquired by the touchscreen 138 and / or other input devices 142, user data including user preferences, settings, and possibly authentication and / or identification biometric data about the user, a download cache including data downloaded through the wireless transceiver, and saved files. System software, software modules, particular device applications, or portions thereof, can be temporarily loaded into the RAM 108. Communication signals received by the computing device 102 can also be stored in the RAM 108. Although specific functions are described for various types of memory, this is merely one embodiment, and different allocations of functions to types of memory can be used in other embodiments.

[0058] The computing device 102 can also include a battery (not shown) as a power source, such as one or more rechargeable batteries that can be charged, for example, by a charging circuit coupled to a battery interface (such as a serial data port). The battery provides electrical power to at least some of the components of the computing device 102, and the battery interface (not shown) provides for mechanical and electrical connection for the battery.

[0059] Figures 2A-2D Configurations of the pinable VUI screen (i.e., pinable VUI screen configurations) provided by various embodiments of the present disclosure are shown. Each pinable VUI screen configuration is displayed on the touchscreen display 138 of the computing device 102 and includes a dynamic content area. The dynamic content area can include a scrollable region 202 and one or more pinning bars represented by reference numerals 204, 206a and 206b, 208, and 210a and 210b, respectively, in Figures 2A-2D Content can be pinned to the one or more pinning bars. In some examples, all of the selectable content in the scrollable region 202 is pinable. In other examples, a subset of the selectable content in the scrollable region 202 is pinable. Alternatively, the dynamic content area can be non-scrollable, but can be updated based on data received from the user or the communication subsystem 130.

[0060] The one or more pinning bars can be parallel to Figures 2A-2DThe scrollable region 202 is located within (e.g., centered on) the fixable VUI screen and occupies a majority of the fixable VUI screen. According to embodiments, the scrollable region 202 can be scrollable in a vertical direction relative to the screen orientation of the fixable VUI screen, in a horizontal direction relative to the screen orientation of the fixable VUI screen, or in both the vertical and horizontal directions relative to the screen orientation of the fixable VUI screen. The scrollable region 202 is used to display dynamic content including one or more selectable UI items. According to embodiments, the selectable UI items can be list items in a scrollable list, table grid, array, or gallery, or other dynamic VUI content such as web page content from a web page, etc. The selectable UI items can include pictures, videos, audio, alphanumeric text, multimedia objects, or other suitable logical objects.

[0061] According to embodiments, the pin fixed bar can be located on the left side, right side, top, or bottom of the fixable VUI screen. According to embodiments, the pin fixed bar can form a horizontal bar (or frame), a vertical bar (or frame), or a frame. The pin fixed bar can be located in the scrollable region 202. For example, the pin fixed bar can be a vertical bar located on the left or right side of the scrollable region 202, a horizontal bar located on the top or bottom of the scrollable region 202, or a frame located at the inner edge of the border, frame, or perimeter of the scrollable region 202. Figures 4A-4I An example is shown in which the pin fixed bar is a vertical bar located on the left side of the scrollable region. Alternatively, the pin fixed bar is not located in the scrollable region, the pin fixed bar can be located outside of the scrollable region 202, e.g., at the outer edge of the border, frame, or perimeter of the scrollable region 202.

[0062] The pin fixed bar is defined by a region of the fixable VUI screen, referred to as the pin fixed bar region. The pin fixed bar can be visible through a visual representation as a visual affordance to the user. Examples of visual representations include a bar, frame, or frame defining the border, frame, or perimeter of the pin fixed bar region. Alternatively, the pin fixed bar can be invisible without a visual representation. For example, when the pin fixed bar is located in the scrollable region 202 of the fixable VUI screen, the pin fixed bar can be invisible as shown in the example of Figures 4A-4I .

[0063] The pinning bar, the scrollable region, and the selectable UI items in the scrollable region 202 are touch-sensitive / touch-interactive screen UI items that are each defined by and associated with a corresponding touch area on the touch surface of the touch screen 138 with which a user can interact through touch input. The touch areas can be defined in terms of corresponding display pixels of the display device of the touch screen 138. The touch areas can be slightly larger or smaller than the corresponding display areas in which visual representations of the corresponding screen items are displayed on the display of the touch screen 138.

[0064] According to the present application, the pinning application 170 or the operating system software 150 or application 152 with which the pinning application 170 interacts to provide pinning functionality maintains state information about the pinnable VUI screen, including the state of the dynamic content region (e.g., the scrollable region) and the state of the one or more pinning bars.

[0065] The state information is temporary runtime information stored in RAM 108 or the like. In some examples, the state of the dynamic content region (e.g., the scrollable region) includes the list items in the scrollable list displayed in the scrollable region and the current order of the list items in the scrollable list that are currently displayed in the scrollable region. The order of the list items in the scrollable list displayed in the scrollable region is initially set based on the order of the list items provided by the application 152. The state of the one or more pinning bars includes the pins currently pinned to the one or more pinning bars. The state of the pins can include the order, size, and / or location of the pins, as well as the stored list item (the "pinned item") corresponding to the pin. The state of the stored item corresponding to the pin includes the associated list item data, such as the content, appearance, etc., of the list item that is pinned. The content can include a picture, a video, an audio, a text, a multimedia object, or other suitable logical object, or a combination thereof. The state information will be updated when the scrollable region or the pinning bar is changed. The stored content corresponding to each pinned list item is stored in a corresponding data structure and assigned and associated with a unique identifier that associates the stored content with the corresponding pin displayed on the pinnable VUI screen.

[0066] The location and / or size of the pins in the pinning bar can be interpreted by the application 152 to have a logical or semantic meaning based on predefined visual layout rules stored in the memory 112. The predefined visual layout rules can be defined by the pinning application 170, the application 152 using the pinning application 170 as an API, or the operating system software 150. The predefined visual layout rules are interpreted by a visual interpreter (not shown) forming part of the pinning application 170, the application 152 using the pinning application 170 as an API, or the operating system software 150.

[0067] When there are no pinned list items (e.g., after all pinned list items are unpinned), changes to the order of the list items can be permanently stored by the application 152, e.g., in response to a request to save or store the changes after the list items are reordered. When saved, the order of the list items in the list stored by the application 152 is updated to reflect the state of the scrollable region 202 at the time of saving.

[0068] In Figure 2A , the pin bar 204 is located on the left side of the scrollable region 202a and forms a vertically or vertically extending region. The scrollable region 202a is vertically scrollable but not horizontally scrollable. In Figure 2B , the pin bars 206a and 206b are located on the top and bottom of the scrollable region 202b, respectively, and form a pair of horizontally or horizontally extending bars. The scrollable region 202b is horizontally scrollable but not vertically scrollable. In Figure 2C , the pin bar 208 forms a frame on the edges of the scrollable region 202c. The scrollable region 202c can be horizontally and vertically scrollable. In Figure 2D , the pin bars 210a and 210b are located on the left and right sides of the scrollable region 202d, respectively, and form vertically or vertically extending regions. The scrollable region 202d is vertically scrollable but not horizontally scrollable.

[0069] Figure 3 is a flowchart of a method 300 of managing user interface (UI) items in a VUI provided by an embodiment of the present disclosure. The method 300 can be performed by software, e.g., the pinning application 170, e.g., executed by the processor 104 of the processing system of the computing device 102 as shown in Figure 1 .

[0070] In operation 302, a pinable VUI screen is displayed on the touchscreen 138. The pinable VUI screen includes a scrollable region with one or more selectable UI items and one or more pin bars, e.g., a vertically extending region located on the left or right side of the pinable VUI screen or a horizontally extending bar located on the top or bottom of the pinable VUI screen, as described above.

[0071] In operation 304, it is determined whether a pinning command (also referred to as a pinning event or pinning input) is detected. The pinning command can be a specified touch interaction with the touchscreen 138 that a user performs in association with the selectable UI item. The touch interaction of the pinning command can include one or more touch gestures performed in association with the selectable UI item, sometimes referred to as pinning gestures. In a preferred example, the pinning command includes detecting a swipe gesture in the direction of the pinning bar, the swipe gesture having an initial point or origin in the touch area of the selectable UI item and an end point in the touch area of the pinning bar. In another example, the pinning command includes detecting a long press (also referred to as a hold gesture) in association with the selectable UI item located in the scrollable region within a threshold period of time following a swipe gesture. In yet another example, the pinning command includes a drag gesture in association with the selectable UI item located in the scrollable region from the scrollable region to the pinning bar. Other types of pinning commands can also be possible.

[0072] As known to those skilled in the art, a long press is a touch gesture with a touch point (i.e., a centroid) of touch input that does not move more than a threshold (tolerance) distance and has a duration that exceeds a minimum or threshold duration, which in some examples can be, for example, 500 milliseconds. Other threshold durations can also be possible. A swipe gesture is a directional touch gesture with a single direction. In a swipe gesture, the touch point moves more than the threshold (tolerance) distance of a long press, but the gesture ends (i.e., an end point is detected) while the touch point is still moving and the velocity exceeds a threshold velocity. Rather than using each position or point of contact over the duration of the touch gesture to resolve the direction, the processor 204 receives data from the touchscreen 138 and can evaluate the swipe relative to an origin of the initial contact with the touchscreen 138 and an end point of the end of the contact with the touchscreen 138. Examples of swipes include horizontal swipes, vertical swipes, and diagonal swipes. A horizontal swipe generally includes an origin of the touch gesture initialized toward the left or right side of the touchscreen 138, detecting the object moving horizontally from the origin to an end point toward the right or left side of the touchscreen 138 while maintaining continuous contact with the touchscreen 138, and disengaging contact with the touchscreen 138. Similarly, a vertical swipe generally includes an origin of the touch gesture initialized toward the top or bottom of the touchscreen 138, detecting the object moving vertically from the origin to an end point toward the bottom or top of the touchscreen 138 while maintaining continuous contact with the touchscreen 138, and disengaging contact with the touchscreen 138. Disengaging swipe contact can be gradual in that the contact with the touchscreen 138 is gradually reduced while the swipe is still in progress. A drag gesture differs from a swipe gesture in that the touch point (e.g., the centroid of the touch event) moves more than the threshold (tolerance) distance, but ends when the touch point is no longer moving or moving at a velocity that does not exceed a threshold velocity.

[0073] In response to determining that the pinning command is detected, the process proceeds to operation 306, in which the corresponding UI item is pinned to the pinning bar. Pinning the UI item to the pinning bar includes removing (also referred to as un-displaying) the pinned UI item from the scrollable region; displaying additional UI items that were not previously displayed from the scrollable list; and displaying the pin corresponding to the pinned UI item in the pinning bar next to or near the location of the pinned UI item before the UI item is removed. In examples in which the scrollable region includes a scrollable list, each UI item is a list item, and each list item can include a row. The pin can be located at a particular location in the area of the pinning bar next to the corresponding row, e.g., at its top, bottom, or center. Alternatively, the pin can occupy the entire area next to the pin. Alternatively, the pin can be located at a border between the pinned UI item and the previous item in the scrollable list, or at a border between the pinned UI item and the subsequent item in the scrollable list.

[0074] The additional UI item in the scrollable list is typically the subsequent UI item in the scrollable list. The pinning command updates the state information stored by the pinning application 170 about the state of the scrollable region and the state of the pinning bar.

[0075] The pin is a visual indication of the pinned item. The pin typically has a predefined shape and size. The pin can include identifying information that identifies the pin as being associated with the pinned item. The pin can include a textual label (e.g., a letter, one or more words, or an index or item order number), a thumbnail image, a preview (e.g., the first word of the text of the pinned UI item and / or thumbnail image), or other identifying information associated with the pinned UI item. The shape and nature of the identifying information of the pin can vary. In Figure 4B and Figure 4C In examples, the pin takes the form of a button with an index letter indicating the alphabetical order of the list item in the scrollable list at the time of pinning. Figure 5B and Figure 5C In examples, the pin takes the form of a circle containing the first letter “G” of the name of the pinned item as identifying information.

[0076] After operation 306, or in response to determining that no pinning command is detected (decision block 304), processing proceeds to operation 308, where it is determined whether a scrolling input is detected. Depending on the application, scrolling can be in one or two dimensions. In a preferred example, the scrolling input is a swipe in a scrollable direction. The scrollable direction is the direction of the scrollable region in which additional content is located. It will be appreciated that during scrolling, additional pinnable UI items can be displayed in the VUI. In one-dimensional scrolling applications, for horizontal scrolling relative to the screen orientation of the VUI, the scrollable direction can be up or down, or for vertical scrolling relative to the screen orientation of the VUI, the scrollable direction can be left or right. In two-dimensional scrolling applications, the scrollable direction can be up, down, left, or right relative to the screen orientation of the VUI. In another example, the scrolling input is a drag in the scrollable direction. In yet another example, the scrolling input can be a swipe or a drag in the scrollable direction. The scrolling speed can be fixed or dependent on the velocity of the touch point. The scrolling updates state information stored by the pinning application 170 regarding the state of the scrollable region.

[0077] In response to determining that a scrolling input is detected, processing proceeds to operation 310, where the content of the scrollable region is caused to scroll in the corresponding scrollable direction. The scrolling changes the visual state of the scrollable region (e.g., as a list view, grid view, or gallery view). The scrolling includes updating the content of the scrollable region according to the distance or speed of the scrolling input according to the gesture used for the scrolling input. When a drag gesture is used as the scrolling input, the amount of scrolling relative to the distance of the scrolling input is based on a pixel-to-pixel ratio that is typically 1 : 1, however, different ratios can be used. When a swipe gesture is used as the scrolling input, the scrolling speed is typically dependent on the velocity of the touch point.

[0078] After operation 310, or in response to determining that no scrolling input is detected (decision block 308), when one or more UI items (e.g., list items) have been pinned to the thumb pinning bar, processing proceeds to operation 312. In operation 312, it is determined whether a repositioning input performed by the user in association with a thumb in the thumb pinning bar has been detected. In some examples, the repositioning input includes a drag gesture performed on or in association with a thumb located in the thumb pinning bar in the scrollable direction.

[0079] In response to determining that the repositioning input is detected, processing proceeds to operation 314, in which the pin is repositioned in the pinning bar in accordance with the repositioning input. A visualization can be provided to assist the user in repositioning. The visualization can include displaying directional arrows, guides, grid lines, etc. to provide visual guidance and affordances regarding the direction or position in which the pin can be moved and the gesture required to perform such movement. The visualization can also provide a visualization in the form of directional arrows, guides, grid lines, etc. to provide visual guidance and affordances regarding the unpinning gesture required to unpin the corresponding pinned item. The pin is repositioned (moved) in a direction corresponding to the direction of the repositioning input by a distance corresponding to the distance of the corresponding repositioning input. The distance the pin is repositioned relative to the distance of the repositioning input is typically a 1 : 1 ratio of pixels to pixels, but different ratios can be used. The repositioning updates the state information stored by the pinning application 170 regarding the state of the pinning bar.

[0080] It should be appreciated that operations 312-314 are optional and are only performed when one or more UI items (e.g., list items) have been pinned to the pinning bar. When there are no pinned UI items, operations 312-314 can not be performed.

[0081] After operation 314, or in response to determining that the repositioning input is not detected (decision block 312), when one or more UI items (e.g., list items) have been pinned to the pinning bar, processing proceeds to operation 316. In operation 316, it is determined whether a unpinning command (also referred to as an unpinning event or unpinning input) is detected. The unpinning command can be a specified touch interaction with the touchscreen 138 performed by the user in association with the pin. The touch interaction of the unpinning command can include one or more touch gestures, sometimes referred to as unpinning gestures, performed in association with the pin. In a preferred example, the pinning command includes detecting a swipe gesture along a direction of the scrollable area and away from the pinning bar, the swipe gesture having an initial point or origin in the touch area of the pinned UI item in the pinning bar and a terminal point in the touch area of the scrollable area. In another example, the unpinning command includes detecting a long press in association with the pin within a threshold period of time following the swipe gesture. In yet another example, the unpinning command includes a drag gesture from the pinning bar to the scrollable area in association with the pin located in the pinning bar. There can be other types of unpinning commands as well.

[0082] In response to determining that the unpinning command is detected, the process proceeds to operation 318, in which the corresponding pinned item is unpinned from the pinning bar. Unpinning the pinned item from the pinning bar includes adding (or inserting) the pinned UI item into the scrollable area at a location corresponding to the location of the pin when the pinned UI item is unpinned from the pinning bar, removing the corresponding pin from the pinning bar, and removing one of the currently displayed UI items in the scrollable area to accommodate the added / inserted UI item. In some examples, the pinned UI item is added to the scrollable area before the UI items located next to or near the location of the pin. In other examples, the pinned UI item is added to the scrollable area after the UI items located next to or near the location of the pin. The location where the pinned UI item is added can be application setting. The unpinning command updates the state information stored by the pinning application 170 regarding the state of the scrollable area and the state of the pinning bar.

[0083] It is to be understood that operations 316-318 are optional and are performed only when one or more UI items (e.g., list items) have been pinned to the pinning bar. When there is no pinned UI item, operations 316-318 can not be performed.

[0084] After operation 318, or in response to determining that the unpinning command is not detected (decision block 316), the process proceeds to operation 320, in which it is determined whether to continue the method 300. Alternatively, operation 320 can be omitted or performed in the background by the operating system 150. In response to determining that an input or event designated to end the method 300 has been detected, the process proceeds to operation 322, where the method 300 ends. In response to determining that an input or event designated to end the method 300 has not been detected, the process proceeds to operation 302. Examples of the input or event designated to end the method 300 are closing the pinning application 170, switching the active or foreground application to an application that does not support pinning, or disabling the pinning functionality by changing settings, menu options, etc.

[0085] If the method 300 ends when one or more UI items (e.g., list items) are pinned to the pinning bar, the state information maintained by the pinning application 170 (e.g., changes in the order of the list items in the scrollable list) is discarded. If the method 300 ends when there is no UI item pinned to the pinning bar, the changes in the order of the list items in the scrollable list, if not already stored or discarded, can be automatically saved to memory according to application settings. As described above, when there is no pinned UI item, the application 152 can permanently store the changes in the order of the UI items (e.g., list items) in response to a request to save or store the changes.

[0086] One-dimensional embodiment

[0087] Reference is now made to Figures 4A-4I , describing a sequence of a pinable visual user interface (VUI) screen for a first example of user interaction according to a method of using Figure 3 . In the first example, the scrollable region of the pinable VUI screen comprises a scrollable list 400 displayed on the touchscreen 138. The scrollable list 400 comprises a plurality of list items. Each list item in the scrollable list 400 is presented as a row, where each row in the scrollable list 400 has the same height and width. A pinning bar 402 is located to the left of the scrollable list 400. For purposes of explanation, the pinning bar 402 is shown in Figure 4A with dashed lines and shading, but is not visible during use. The scrollable list 400 comprises a plurality of list items denoted A through Z. In Figure 4A , the list items are arranged in alphabetical order from A through Z, only list items A through F are shown, and list items G-Z are displayable in response to a scroll input. The scrollable list 400 can comprise more or fewer list items in other examples, and can be sorted in other ways than alphabetical order, including but not limited to numerical order. In the example shown, the scrollable list 400 is scrollable only in the vertical direction.

[0088] In response to detecting a pinning command in the form of a right-to-left swipe (also referred to as a left swipe) performed by the user relative to list item E, list item E is pinned to the pinning bar 402, with reference to Figure 4A . As shown in Figure 4B , detecting the pinning command causes a pin 404 corresponding to list item E to be displayed next to or proximate to the location of list item E in the scrollable list 400 while list item E is pinned, the pinned list item (list item E) is removed from the scrollable list 400, and an additional list item (list item G) is displayed at the bottom of the VUI screen, such that list items A-D, F-G are as shown in Figure 4B . In the example shown, the pin 404 is smaller than the height of the row representing a list item. In other examples, the pin 404 can be the same height as the list items in other examples.

[0089] In the example shown, only one pin can be positioned next to or proximate to each row / list item in the scrollable list 400. This effectively limits the number of pins / list items at a given time to the number of rows / list items that can be displayed in the scrollable list 400. In other examples, more than one pin can be positioned next to or proximate to each row / list item in the scrollable list 400.

[0090] Referring to Figure 4B , in response to detecting a pinning command performed by the user in the form of a left swipe relative to list item D, list item D is pinned to pinning bar 402. As shown in Figure 4C , detecting the pinning command causes display of pin 406 corresponding to pinned list item D in pinning bar 402 next to or near the location of pinned list item D in scrollable list 400, removal of pinned list item D from scrollable list 400, and display of additional list items (list items H) at the bottom of the VUI screen such that list items A-C, F-H are displayed as shown in Figure 4C .

[0091] Referring to Figure 4D and Figure 4E , in response to detecting a scrolling input ( Figure 4D ) performed by the user in the scrollable region (i.e., scrollable list 400), the content of the scrollable region is scrolled, causing display of list items P-U ( Figure 4E ). During and after the scrolling, pins 404 and 406 remain pinned to pinning bar 402 at the same locations.

[0092] Referring to Figure 4F and Figure 4G , in response to detecting a repositioning input performed by the user in association with a pin in pinning bar 402, pin 406 corresponding to list item D is moved from a first location ( Figure 4F ) in scrollable list 400 to a second location ( Figure 4F ) in scrollable list 400. As described above, the repositioning input can include a drag gesture in the scrollable direction in association with a pin located in pinning bar 402. In some examples, where only one pin can be positioned next to or near each row / list item in scrollable list 400, there is a pinning location next to or near each row / list item in scrollable list 400 where the pin can be positioned. For example, in a first example, because each row of list items has a fixed height, there can be an index location next to or opposite each row / list item in scrollable list 400 from which the repositioning input repositions the pin from one row to another in scrollable list 400. The index location can be the center of the area of the pinning bar next to or opposite the row in scrollable list 400. In some examples, to move the pin to the index location of a row / list item in scrollable list 400, the centroid of the touch input need only be moved to the touch area corresponding to the respective row in scrollable list 400.

[0093] Referring to Figure 4G and Figure 4H, in response to detecting a de-pinning command in the form of a left-to-right swipe (also referred to as a right swipe) performed by the user with respect to the pin 406, the list item D is de-pinned from the pinning bar 402 and added to or inserted into the scrollable list 400 at a position corresponding to the position of the pin 406 in the scrollable list 400 when the list item D is de-pinned from the pinning bar 402. In the example shown, during de-pinning, the list item D is added to the scrollable list 400 at a position before the list items located beside or near the position of the pin 406.

[0094] In Figure 4G , the display portion of the scrollable list 400 shows the list items P, Q, R, S, T, and U. Since the pin 406 corresponding to the list item D is de-pinned when the list item D is beside or near the list item Q Figure 4G , the list item D is added to the scrollable list 400 at a position before the list item Q. The pin 406 is removed from the pinning bar 402 and the list item U is removed from the list items currently displayed in the scrollable list 400. As a result, after the de-pinning command, the list item D is located between the list items P and Q in the display portion of the scrollable list 400 that displays the list items P, D, Q, R, S, and T Figure 4H .

[0095] Referring to Figure 4H and Figure 4I , in response to detecting a de-pinning command in the form of a right swipe performed by the user with respect to the pin 404, the list item E is de-pinned from the pinning bar 402 and added to or inserted into the scrollable list 400 at a position corresponding to the position of the pin 404 in the scrollable list 400 when the list item E is de-pinned from the pinning bar 402. In the example shown, during de-pinning, the list item E is added to the scrollable list 400 at a position before the list items located beside or near the position of the pin 404.

[0096] In Figure 4H , the display portion of the scrollable list 400 shows the list items P, D, Q, R, S, and T. Since the pin 406 corresponding to the list item D is de-pinned when the list item D is beside or near the list item Q Figure 4H) time, the pin 404 corresponding to list item E is un-pinned, so list item E is added to the scrollable list 400 in a position before list item S. The pin 404 is removed from the pinning bar 402, and list item T is removed from the currently displayed items in the scrollable list 400. As a result, after the un-pinning command, list item E is located between list items R and S in the display portion of the scrollable list 400, which displays list items P, D, Q, R, E, and S Figure 4I .

[0097] Referring now to Figures 5A-5D , a sequence of a pinnable VUI screen is described for a second example of user interaction according to a method of using Figure 3 . In the second example, the pinnable VUI screen includes a scrollable list 500 displayed on the touchscreen 138. The scrollable list 500 includes a plurality of list items. The scrollable list 500 is similar to the scrollable list 400. Each list item in the scrollable list 500 is presented as a row, where each row in the scrollable list 500 has the same height and width. A pinning bar 502 is located to the left of the scrollable list 500. For purposes of explanation, the pinning bar 502 is shown in dashed lines and shading in Figure 5A , but is not visible during use. In the illustrated example, the scrollable list 500 is scrollable only in a vertical direction.

[0098] Referring to Figure 5A and Figure 5B , in response to detecting a pinning command performed by the user in the form of a left swipe relative to list item "Grapes", the list item "Grapes" is pinned to the pinning bar 502. As shown in Figure 5B , detecting the pinning command causes a pin 504 corresponding to the pinned list item "Grapes" in the pinning bar 502 to be displayed next to or near the location of the scrollable list 500, the pinned list item "Grapes" to be removed from the scrollable list 500, and an additional list item (list item "Apple") to be displayed at the bottom of the VUI screen. In the illustrated example, the pin 504 is located at the border between the pinned list item "Grapes" and the previous item in the scrollable list 500, which is list item "Watermelon". The pin 504 takes the form of a circle containing the first letter "G" of the name of the pinned list item as identifying information.

[0099] Referring to Figure 5B and Figure 5C , in response to detecting a scroll input performed by the user in the scrollable list 500, the content of the scrollable region is scrolled, causing the pinnable VUI screen to be displayed that includes list items A-E Figure 5C). During and after scrolling, the pin 504 remains fixed to the pinning bar 502 at the same location.

[0100] Referring to Figure 5C and Figure 5D , in response to detecting a user performing a pinning command in the form of a right swipe relative to the pin 504, the list item "Grapes" is pinned to the pinning bar 502. Pinning includes displaying a pinning VUI screen in which the list item "Grapes" is added or inserted into the scrollable list 500 at a location in the scrollable list 500 corresponding to the location of the pin 504, and the pin 504 is removed. In the illustrated example, during pinning, the list item "Grapes" is added to the scrollable list 400 between list items C and D.

[0101] The one-dimensional pinning method described above has several advantages, including that it reduces the time requirements, effort and time needed to reposition multiple items that exist with current reordering techniques, and improves the accuracy of repositioning items compared to current reordering techniques.

[0102] Two-dimensional embodiment

[0103] Referring now to Figures 6A-6E , a sequence of pinnable visual user interface screens is described for a third example of user interaction according to a method of using Figure 3 . In the third example, the scrollable region of the pinnable VUI screen includes a two-dimensional scrollable array (or grid) 600, such as a table or gallery, displayed on the touchscreen 138. The scrollable array 600 includes a plurality of UI items, sometimes referred to as cells, arranged in rows and columns. The cells are arranged in an order. In the illustrated example, each cell has the same height and width. Pinning bars 602 and 604 are located on the left and right sides of the pinnable VUI screen. For purposes of explanation, the pinning bars 602 and 604 are shown in the pinnable VUI screen illustrated in Figure 6A with dashed lines and shading, but are not visible during use. In the illustrated example, the scrollable array 600 is scrollable only in the vertical direction.

[0104] Referring to Figure 6A and Figure 6B , in response to detecting a user performing a pinning command in the form of a left swipe relative to the cell "S", the cell "S" is pinned to the pinning bar 602. Detecting the pinning command causes a pin 610 in the pinning bar 602 corresponding to the pinned UI item (i.e., the cell "S") to be displayed at a location proximate to the pinned UI item (i.e., the cell "S") in the scrollable array 600 shown in the pinnable VUI screen from Figure 6A Figure 6A ​The pinned item (cell "S") of the scrollable array 600 shown in the fixable VUI screen of Figure 6B The bottom of the scrollable array 660 shown in the fixable VUI screen of Figure 6B In the fixable VUI screen shown, the pin 610 takes the form of a button with an index letter that represents the letter of the UI item (cell "S") that is pinned in the scrollable array 600 at the time of pinning, namely "S".

[0105] Referring to Figure 6B and Figure 6C In response to detecting a pinning command performed by the user in the form of a right swipe relative to the cell "Y", the cell "Y" is pinned to the pinning bar 604. Detecting the pinning command causes the pin 612 corresponding to the pinned UI item (cell "Y") to be displayed in the pinning bar 604 proximate to the location of the pinned UI item, the pinned UI item (cell "Y") is removed from the scrollable array 600, and the additional item (cell "AE") is displayed at the bottom of the fixable VUI screen of Figure 6C In response to detecting a pinning command performed by the user in the form of a right swipe relative to the cell "Y", the cell "Y" is pinned to the pinning bar 604. Detecting the pinning command causes the pin 612 corresponding to the pinned UI item (cell "Y") to be displayed in the pinning bar 604 proximate to the location of the pinned UI item, the pinned UI item (cell "Y") is removed from the scrollable array 600, and the additional item (cell "AE") is displayed at the bottom of the fixable VUI screen of Figure 6C In the fixable VUI screen shown, the pin 612 takes the form of a button with an index letter that represents the letter of the UI item (cell "Y") that is pinned in the scrollable array 600 at the time of pinning, namely "Y".

[0106] Referring to Figure 6C and Figure 6D In response to detecting a pinning command performed by the user in the form of a right swipe relative to the cell "Y", the cell "Y" is pinned to the pinning bar 604. Detecting the pinning command causes the pin 612 corresponding to the pinned UI item (cell "Y") to be displayed in the pinning bar 604 proximate to the location of the pinned UI item, the pinned UI item (cell "Y") is removed from the scrollable array 600, and the additional item (cell "AE") is displayed at the bottom of the fixable VUI screen of Figure 6D In response to detecting a pinning command performed by the user in the form of a right swipe relative to the cell "Y", the cell "Y" is pinned to the pinning bar 604. Detecting the pinning command causes the pin 612 corresponding to the pinned UI item (cell "Y") to be displayed in the pinning bar 604 proximate to the location of the pinned UI item, the pinned UI item (cell "Y") is removed from the scrollable array 600, and the additional item (cell "AE") is displayed at the bottom of the fixable VUI screen of

[0107] Referring to Figure 6D and Figure 6EIn response to detecting a cancel fix command in the form of a drag gesture performed by the user with respect to the pin 612 from the pin fix bar 604 to a desired location (the current location of cell "PY" in the illustrated example), the fixed UI item (cell "Y") is unfixed from the pin fix bar 604. Unfixing includes adding or inserting the fixed UI item (cell "Y") into the scrollable array 600 at a location in the scrollable array 600 that corresponds to the location of the UI item (cell "PY"), and removing the pin 612. Adding or inserting the fixed UI item (cell "Y") causes the existing UI items (cells) in the scrollable array 600, namely cells "PY", "PZ", "QA", "QB", "QC", "QD", "QE", and "QF", to move appropriately based on the added or inserted fixed UI item (cell "Y"), as shown in the fixable VUI screen in Figure 6E

[0108] Third embodiment - preserving UI item position or state

[0109] Figure 7 is a flowchart of a method 700 of managing user interface items in a VUI, provided by one embodiment of the present invention. The method 700 can be performed by software, such as the fix application 170, for example, executed by the processor 104 of the processing system of the computing device 102, as shown in FIG. 1. The method 700 is similar to the method 300. However, rather than changing the position of a few fixed UI items (e.g., list items), the position of all or a group of UI items is changed while preserving those items by fixing the position of one or more UI items. Figure 1

[0110] The operations 302-306 are performed as described above. In operation 312, it is determined whether a repositioning input performed by the user in association with a pin in the pin fix bar has been detected. In response to determining that a repositioning input has been detected, processing proceeds to operation 314 in which the pin is repositioned in the pin fix bar in accordance with the repositioning input. Repositioning in the method 700 changes the order of the pin and the (fixed) UI item relative to each other, but does not change the unfixed UI items.

[0111] It should be appreciated that the operations 312-314 are optional and are performed only when one or more UI items have been fixed to the pin fix bar. When no fixed UI items exist, the operations 312-314 can not be performed.

[0112] After operation 314, or in response to determining that no repositioning input has been detected (decision block 312), processing proceeds to operation 710.

[0113] ​​In operation 710, a list operation is performed on the list displayed in the scrollable area in response to detecting the corresponding input. The input can be generated by invoking the corresponding menu option by calling the corresponding menu, touch gesture, or other suitable means. The list is a list view displayed in the scrollable area. Alternatively, the list can be a row or column in a grid view (e.g., a spreadsheet). The list operation can be a sort operation that sorts unfixed items by a parameter associated with the list items. The sort operation can be by value of a data value of the list items (e.g., numerical or alphabetical order), by date of a date value associated with the list items, or other suitable data parameter. Examples of data values can be cost or time. Examples of date values include add date, edit date, and modification date, among others. The order of the unfixed list items in the list is changed by the list operation.

[0114] Operations 316-322 are then performed as described above.

[0115] In the illustrated example, the fixable VUI screen occupies the entire display area of the touchscreen 138.

[0116] While some of the above-described embodiments have been described in connection with lists and tables, the guidance of the present application is not limited to lists or tables and extends to other fixable VUI screens having ordered UI items, such as a gallery or other similar array.

[0117] Fourth embodiment - fixable clipboard

[0118] A fourth embodiment is a fixable clipboard that supports fixing and unfixing UI items within and between applications 152. A clipboard is a buffer provided by the operating system software 150 that provides short-term storage and transfer within and between applications 152. A clipboard is typically temporary, with its contents residing in RAM 108. The operating system software 150 provides an API by which applications 152 can add and delete items through corresponding cut, copy, and paste operations, among others. The fix application 170 defines the method by which the user commands these operations through the fix and unfix commands described herein.

[0119] The fixable clipboard can be applied to most or all fixable VUI screens having dynamic content. The edge of the fixable VUI screen acts as a pinning bar and the interior of the fixable VUI screen acts as a fixable surface. Figures 2A-2D Any fixable VUI screen configuration of FIGS. 1-3 can be used as a pinning bar, although Figure 2C The fixable VUI screen configurations illustrated in FIGS. 1-3 are preferred. Figure 2D

[0120] ​A fixable clipboard can be used in a reading and / or composition application or scenario. While a user is reading content in a dynamic content area (e.g., a web page in a web browser application), content such as an image or a string of text (e.g., a paragraph) can be fixed (e.g., to an edge of a fixable VUI screen). The user can then switch to another application 152 and unfix the fixed content within the other application 152.

[0121] Reference is now made to Figure 8A and Figure 8B , which show a computing device with a fixable VUI screen provided by embodiments of the present application. In Figure 8A and Figure 8B , a first pin fixed bar 801a is located on the left side of the display area of the touch screen 138 and a second pin fixed bar 801b is located on the right side of the display area of the touch screen 138. The pin fixed bars 801a and 801b are also located on the left and right edges of the VUI screen, respectively. The first pin fixed bar 801a and the second pin fixed bar 801b persist across fixable VUI screens of different applications.

[0122] For purposes of explanation, the pin fixed bars 801a and 801b are shown in dashed lines and shading, but are not visible during use. As best shown in Figure 8A , the fixable VUI screen 800 includes a plurality of sections, including a header 802, a footer 804, a left side 806, and a right side 808. For purposes of illustration, the boundaries of the sections are indicated by dashed lines. These sections are logical divisions of the fixable VUI screen 800 that can be used by the application 152, for example, to interpret pin locations during document composition and the like, as described in more detail below. In use, there can be no visual indication of the sections displayed on the touch screen 128.

[0123] In Figure 8A , a plurality of pins, indicated by reference numerals 812, 814, 816, 818, and 820, are connected to the pin fixed bars. The pins 816 and 820 are positioned parallel to each other on the left and right sides of the fixable VUI screen 800. The pin 816 is located within the pin 818. The pin 812 and the pin 814 overlap. As described below, pin locations (including relative locations with respect to other pins) and / or sizes can be interpreted by the application 152 as having logical or semantic meaning.

[0124] An object can be pinned to one of the pinning bars by a pinning command. The pinning command is a specified touch interaction with the touchscreen 138 that a user performs in association with a selected content. The touch interaction of the pinning command includes one or more touch gestures that are performed in association with the selected content. The selectable content can include a picture, a video, an audio, a text, or a multimedia object, or other suitable logical object. The operating system software 150 or the application 152 includes a logical data object mapping that associates each logical data object to one or more physical data objects stored in the memory, and can include conversion logic. The content can be selected by selecting a screen object or a region of interest (ROI) through a touch input (e.g., a long press). A selection marker can be displayed to provide a visualization of the selected content. The selection marker can be adjusted, i.e., moved or changed in size (e.g., increased or decreased), according to the corresponding touch interaction, e.g., by dragging the selection marker or its end. In one preferred example, the pinning command includes selecting the content, followed by a drag gesture in which the selected content is moved to the pinning bar. The drag gesture has an initial point or origin in the touch area of the content region and a terminal point in the touch area of the pinning bar. A collision of the touch point with the border of the pinning bar can trigger the pinning command. Alternatively, a swipe gesture can be used instead of the drag gesture.

[0125] As described above, in response to detecting the pinning command, a pin corresponding to the pinned object is added to the pinning bar. In some examples, the pins can be located on the pinning bar at positions corresponding to the pinning commands (e.g., at the positions of the end points of the pinning gestures). In other examples, the pins can be arranged in the order in which they appear. The pins can be moved in response to corresponding inputs (e.g., touch gestures), e.g., by a drag gesture that drags the pin along the pinning bar located at the edge of the fixable VUI screen. In some examples, dragging or otherwise moving the pin to the top or bottom of the fixable VUI screen causes the pin to move to the pinning bar on the opposite side of the fixable VUI screen (i.e., from left to right, and vice versa).

[0126] In some examples, the size of the pin can be changed by corresponding inputs (e.g., touch gestures). For example, a pinching gesture to increase or decrease the size of the pin on the touchscreen 138. Figure 8BPins are shown in a first state (or size) indicated by reference numeral 830 and a second state (or size) indicated by reference numeral 840. The first state 830 can represent at least one of an initial size of the pin and a minimum size of the pin. The second state 840 can represent a maximum size of the pin. The size of the pin can be changed by a corresponding touch gesture (e.g., a pinch gesture) performed with two fingers. A pinch open gesture can be used to increase the size of the pin, while a pinch close gesture (also referred to as reverse pinch) can be used to decrease the size of the pin. As noted above, the size of the pin can be interpreted by the application 152 as having a logical or semantic meaning.

[0127] For example, by a corresponding input (e.g., a touch gesture), the pinned object associated with the pin when connected to the pinning bar can be interacted with. In some examples, a long press or detection associated with the pin causes a context sensitive menu to be invoked and displayed on the touch screen 138. The context sensitive menu can be cancelled or closed by touching anywhere on the touch screen 138. The context sensitive menu overlays or pop-up window overlays a portion of the currently pinnable VUI screen. The context sensitive menu includes a plurality of options that depend on the pinned item. The plurality of options can include one or a combination of "delete," "access," "search," "download," "share," "copy," and "play" or "pause." The "play" or "pause" options are only available when the pinned item is a video or music. For another example, the "access" option is only available when the pin was created from an application that is different from the active application. The "search" option is only available when the pinned item is one or more text or images, and selection of the "search" option causes an internet search of the one or more text or images. The "share" option causes the pin or the pinned content to be shared with one or more other applications on the computing device 102.

[0128] Selection of the "delete" option by a corresponding input (e.g., a touch or tap to select the "delete" option from the context sensitive menu) causes the pin and the corresponding pinned object to be deleted. It should be appreciated that the touch or tap is distinguished from a long press with the computing device 102 by the duration of the touch input. When the touch duration is less than a threshold, the input is interpreted as a touch or tap. When the touch duration is equal to or exceeds the touch duration, the input is interpreted as a long press.

[0129] Selecting the "access" option (e.g., touching or tapping to select the "access" option from the context-sensitive menu) in response to the corresponding input causes the computing device 102 to switch the active (or foreground) application to the corresponding pinned object's pinned source application, and optionally to the pinned VUI screen in the source application, and optionally to the location in the VUI screen where the pinned object is located in the source application. If the source application is not running in the background when the "access" option is invoked, the source application is launched or invoked. For example, if the pinned item is created from a website, the "access" option launches or invokes the browser and opens the corresponding website uniform resource locator (URL) if the browser is not already open. If the pinned source location cannot be located, an error message is displayed indicating that the "pinned source is not available or has been deleted." The pinned source can be another application. If the application instance is not available when the "access" option is selected, a screen shot of the content can optionally be displayed.

[0130] Alternatively, at least some of the options or other actions in the context-sensitive menu can be selected and invoked in response to a corresponding touch gesture associated with the respective pin. In response to detecting a corresponding input associated with the respective pin, an action associated with the stored content corresponding to the pin is performed. For example, detecting a double tap on the pin (i.e., tapping twice on the touch area associated with the pin within a threshold duration) or a swipe / drag gesture with an initial contact point on the pin and a direction of travel toward the center of the VUI screen, or an end point near the center of the VUI screen, can cause the "access" option to be selected and the corresponding source of the respective pin to be accessed.

[0131] In some examples, the positions of the pins in the pinning bar can be interpreted by the application program 152 to have logical or semantic meaning based on predefined visual layout rules stored in the memory 112. The predefined visual layout rules can be defined by the pinning application 170, an application program 152 using the pinning application 170 as an API, or the operating system software 150. The meaning can vary based on the application program 152 and / or the context, state, or mode of the application program 152. Table 1 below provides examples of pin position interpretations when the active or foreground application is used for document composition.

[0132] Table 1: Pin Position Interpretations

[0133] Placing Explanation Top pin Become the title of the content Pins placed parallel to each other on the left and right Fixing content into a two-column (dual-column) layout Bottom pin Become the footer of the content Pin order Content order Left pin Left alignment Right pin Right alignment Pin size greater than threshold Center alignment overlay area Two pins overlap Contents are merged in the order of overlap to form a single entity One pin is located in another pin Based on the position, one content is in another content

[0134] The predefined visual layout rules are interpreted by the application 152 for dynamic content generation, e.g., during creation or editing of an electronic document, sometimes referred to as a document in composition. The electronic document can be an electronic message, a word processing document, a presentation document, or other suitable type of electronic document. Examples of electronic messages include email messages, multimedia messaging service (MMS) messages (also known as PXT, picture messages, or multimedia messages), instant messages, social media posts, blog posts, and similar types of electronic messages.

[0135] In some examples, the size of the pin can be interpreted by the application 152 based on the predefined visual layout rules stored in the memory 112 to have a logical or semantic meaning. In some examples, a larger pin can result in a relatively larger size of the fixed object and take up more space when added to the input bar compared to a smaller pin. For example, the input bar can be an input content area, e.g., a document content area (also referred to as the body) of a document in composition, e.g., an electronic message.

[0136] In some examples, the pin is initially generated with an initial default size and can be increased, e.g., by performing a pinch gesture on or associated with the corresponding pin. The size of the object added to the document content area can increase relative to the size of the original fixed object when the size of the corresponding pin increases from the initial default size, or decrease relative to the size of the original fixed object when the size of the corresponding pin decreases from the initial default size. The amount of size adjustment of the object added to the document content area can be directly proportional to the amount of size adjustment of the pin.

[0137] The content generation is dynamic because it does not have a fixed layout or static content. The content composition and layout can change based on the predefined visual layout rules, the fixed content, and the pin size and / or position. The content generation is also dynamic because the electronic document can be created or edited “on the fly.” The electronic document layout and content dynamically change based on changes in the fixed content and the pin size and / or position.

[0138] Figure 9 A flowchart of a method 900 of managing user data items in a VUI provided by one embodiment of the present invention. The method 900 can be performed by software, e.g., the fixed application 170, e.g., at least by the processor 104 of the processing system of the computing device 102 shown in FIG. 1. Figure 1 The method 900 is similar to the methods 300 and 700 in that items can be pinned to a pinning bar, but allows the pinned UI items to be used across different applications 152, thereby providing a pinning clipboard.

[0139] In operation 902, a first pinable VUI screen associated with a first application is displayed on touch screen 138, for example, in response to invocation or launch of the first application, or in response to an active or foreground application changing to the first application. In the illustrated example, the first pinable VUI screen occupies the entire display area of touch screen 138. The first application can be a browsing, reading, or viewing application in which the user can discover content, such as a web browser for web content, a news application for news, a social media application for social media posts, and the like. The first pinable VUI screen includes a dynamic content area with one or more selectable UI items. The first pinable VUI screen also includes one or more pin bar regions. In the described embodiment, the pin bar regions in the form of vertically extending regions are located on the left and right sides of the first pinable VUI screen, as shown. Other types of configurations of one or more pin bar regions can be used in other embodiments. The user can browse the content in the dynamic content area. For example, the dynamic content area can be a scrollable region, such as a web page, which can be browsed by scrolling in response to scrolling input. Figure 8A

[0140] In operation 904, one or more selectable UI items are pinned to one or more of pin bar regions 801a and 801b by a pinning command, each pinning command detected by pinning application 170. The pinning command is a specified interaction performed by the user in association with selected content. The selected content can include a picture, a video, an audio, a text, a multimedia object, or other suitable logical object, or a combination thereof. In one example, the pinning command includes a long press followed by an immediate drag gesture that selects content in the form of a different UI item or a region of interest (ROI), where the selected content is moved to the pin bar. A selection marker is displayed to provide a visualization of the selected content. The selection marker can be adjusted, i.e., moved or changed in size (e.g., increased or decreased), in accordance with a corresponding touch interaction, such as by dragging the selection marker or an end thereof. The drag gesture has an initial point or origin in a touch region of the content area and a terminal point in a touch region of the pin bar. A collision of the touch point with a border of the pin bar can trigger the pinning command. Alternatively, a swipe gesture can be used instead of the drag gesture.

[0141] ​In another example, a long press on selected content is used to invoke a context sensitive menu that includes an option "pin." The context sensitive menu overlays or pops up over a portion of the current pinable VUI screen. The context sensitive menu can include other options, such as "copy," "select," "select all," etc. The "copy" option copies the selected content to a memory, e.g., to a clipboard. In response to selecting the "pin" option from the context sensitive menu, a movable screen indicator, e.g., a caret, cursor, or pointer, is displayed at the location of the selected content. The movable screen indicator is draggable and can be dragged to a pinning bar 801a and 801b at an edge of the first pinable VUI screen to pin it. The movable screen indicator can take any suitable shape, such as a circle, a ball, or a pin. The movable screen indicator can be identical to the visual representation of the pin of the pinning bar 801a and 801b, optionally including preview information based on the selected content.

[0142] There can be other types of pinning commands as well. For example, the pinning command can be a specified touch interaction with the touchscreen 138 that a user performs in association with the selected content. The touch interaction of the pinning command can include one or more touch gestures performed in association with the selected content.

[0143] Pinning the selected content to the pinning bar 801a or 801b in the method 900 includes copying the selected content to a clipboard (i.e., temporarily storing the selected content in a memory) and displaying a pin in the pinning bar 801a or 801b corresponding to the selected content at a location corresponding to the end point of the pinning command. As described above in the context of the methods 300 and 700, the pin is a visual indication of the selected content. The pin typically has a predefined shape and size. The pin can include identifying information that identifies the pin as being associated with the pinned UI item. The pin can include a textual label (e.g., a letter, one or more words, or an index or item order number), a thumbnail image, a preview (e.g., the first word of the text of the selected content and / or a thumbnail image), or other identifying information associated with the selected content. The shape of the pin and the nature of the identifying information can vary.

[0144] In examples in which the location or size of the pin has a logical or semantic meaning, the pinning application 170 can maintain a data structure for the input bar, such as a document content region, e.g., a document in a composition. The data structure defines a visual representation or layout of the document content region and is stored in the memory 112. When the selected content is pinned to one of the pinning bars 801a or 801b, or the "pin" option is selected, the selected content corresponding to each pin is stored in the memory 112 in the data structure for future retrieval.

[0145] When a pin is added to one of the pinning bars 801a or 801b, or when a pin moves within or between the pinning bars 801a or 801b, the position of the pin within the pinning bar 801a or 801b relative to other pins is determined (in pixels), and a visual relationship between the pinned content is generated based on the predefined visual layout rules stored in the memory 112 (e.g., in the pinning application 170). The position of the pin can be determined by the center or centroid of the pin.

[0146] As described above, the predefined visual layout rules are interpreted by a visual interpreter (not shown) forming part of the pinning application 170, an application 152 using the pinning application 170 as an API, or the operating system software 150. The visual interpreter loads the predefined visual layout rules at initialization, and in response to adding a pin, a change to an existing pin (e.g., moving or resizing), or deleting an existing pin, interprets the visual layout of the document content area based on the pin positions, the predefined visual layout rules, and the stored content. Alternatively, the visual interpreter can interpret the visual layout of the document content area in response to unpinning a pinned item or adding / inserting the stored content associated with a pin to the document content area.

[0147] The visual interpreter can also retrieve the stored content corresponding to the current pin from the memory 112 to present and display a mini preview of the visual representation or layout of the document content area at a short time interval based on the current position and / or size of the pin each time a pin is added or moved. The mini preview depends on the currently pinned item. The visual interpreter generates the mini preview based on the position of the currently pinned item and the semantic meaning of the pin position. Since the pinning supports dynamically composing a virtual document, the mini preview enables a preview of the virtual document.

[0148] The mini preview can be a thumbnail image of the document content area presented as an overlay or pop-up window covering a portion of the current VUI screen. The portion of the screen occupied by the mini preview can be 20% or less, 15% or less, or possibly 10% or less of the current VUI screen. The mini preview can be displayed at the location of the added pin or the location of the pin moved or resized. The short time interval during which the mini preview is displayed before being removed can be, for example, 5 seconds or less, or 2-3 seconds or less. Thus, in the example where the position and / or size of the pin has a logical or semantic meaning, the pinning selected content can include displaying a mini preview of the visual representation or layout of the document content area based on the current position and / or size of the pin.

[0149] In operation 906, responsive to the active or foreground application changing from the first application to the second application, the second pinable VUI screen is displayed on the touchscreen 138. In the illustrated example, the second pinable VUI screen occupies the entire display area of the touchscreen 138. The first and second pinable VUI screens are different and separate pinable VUI screens. The active or foreground application is changed responsive to detecting a corresponding input to switch the active or foreground application. The second application has a document composition mode, and the second pinable VUI screen is a document composition screen. The second application can be an electronic messaging application in which electronic messages such as email, text messages, instant messages, posts, etc. can be composed. Alternatively, the second application can be a document creation application for creating a document, such as a word processing application.

[0150] In operation 908, one or more pins are optionally changed responsive to detecting a corresponding input, such as a drag gesture to move a pin or a pinch gesture to change the size of a pin. Thus, in examples in which the location or size of a pin has a logical or semantic meaning, changing the pin location or size can include displaying a mini preview of the visual representation or layout of the document content area based on the current location and / or size of the pin. While the change in the pin is shown after switching to the second application and while the second pinable VUI screen is displayed, the change can occur at any time while the pin remains pinned to one of the pinning bars 801a or 801b.

[0151] In operation 910, one or more pins are optionally deleted responsive to detecting a corresponding input, such as selecting a pin that is pinned to one of the pinning bars 801a or 801b from the context-sensitive menu by selecting the "delete" option from the context-sensitive menu using a long press or other suitable touch gesture. The "delete" option can be selected from the context-sensitive menu by a touch or tap associated with the "delete" option displayed in the context-sensitive menu.

[0152] In operation 912, one or more pins are optionally accessed responsive to detecting a corresponding input, such as selecting a pin that is pinned to one of the pinning bars 801a or 801b from the context-sensitive menu by selecting the "access" option from the context-sensitive menu using a long press or other suitable touch gesture. The "access" option can be selected from the context-sensitive menu by a touch or tap associated with the "access" option displayed in the context-sensitive menu.

[0153] In operation 914, the stored content associated with one or more pins is added to the document content area by unpinning commands, each unpinning command detected by the pin application 170. The unpinning commands are specified interactions performed by the user in association with a selected pin. There can be multiple types of unpinning commands to provide multiple ways of adding the stored content corresponding to the pins to the second pinnable VUI screen of the target application, providing flexibility to the user.

[0154] The stored content corresponding to all the pins can be added to the document content area simultaneously in a manner that the stored content is automatically arranged according to predefined visual layout rules stored in the memory 112. To add the stored content corresponding to all the pins simultaneously, the user invokes a context sensitive menu by a corresponding input, such as a long press. In response to detecting the long press or in association with the document content pane (or area) of the second pinnable VUI screen, the context sensitive menu is invoked and displayed on the touch screen 138. The context sensitive menu can be cancelled or closed by touching any location on the touch screen 138. The document content pane is an area of the second pinnable VUI screen that is internally bounded by the pinning bar 801a or 801b. The context sensitive menu includes multiple options, which include "Pin Pins" and "Pin Selected Pins" when one or more pins are selected. When no pins are selected, the multiple options include only the "Pin Pins" option for pinning the pins. Selecting the "Pin Selected Pins" option by a corresponding input, such as touching or tapping the "Pin Selected Pins" option from the context sensitive menu, causes the stored content corresponding to all the pins to be added to the document content area simultaneously in a manner that the stored content is automatically arranged according to the predefined visual layout rules. The pins are then deleted.

[0155] The stored content corresponding to a subset of the pins can be added to the document content area simultaneously in a manner that the stored content is automatically arranged according to predefined visual layout rules stored in the memory 112. The subset can include one or more pins that the user selects as desired. When the context sensitive menu is invoked and displayed, all the pins can be selected by default, and the user will deselect the pins that are not desired from the selected pins as the subset of pins to be pinned. Alternatively, in other examples, no pins can be selected by default, and the user has to select the desired pins to be pinned. Each selected pin can be indicated by a screen selection marker, such as a glow or highlight around the corresponding pin. The user can select and deselect the pins by a corresponding input, such as a touch or tap associated with the corresponding pin. Selecting the "Pin Pins" option by a corresponding input, such as touching or tapping the "Pin Pins" option from the context sensitive menu, causes the stored content corresponding to all the pins to be added to the document content area simultaneously in a manner that the stored content is automatically arranged according to the predefined visual layout rules. The pins are then deleted.

[0156] The stored content corresponding to the single pin can be added to the document content area at a location specified by the user through a drag gesture, where the end point of the drag gesture specifies the location to add the stored content. The size of the stored content in the document content area can be automatically determined according to predefined visual layout rules even though the location is set by the user through the drag gesture. Alternatively, a swipe gesture can be used instead of a drag gesture. Other types of unpinning commands are also possible.

[0157] Unpinning the stored content from the input field (e.g., the document content area) in the method 900 includes adding the stored content corresponding to the pin from the clipboard (i.e., temporarily stored in memory) to the input field (e.g., the document content area) and removing (or deleting) the pin corresponding to the stored content.

[0158] If the document content area already includes certain content in the document content pane, the visual interpreter generates one or more pins at the corresponding locations of the existing content using a different visualization style compared to the user-generated pin in order to facilitate differentiation. For example, the pin generated by the visual interpreter based on the existing content in the document content pane can be displayed in a different color so that the user can adjust the other pin relative to the existing content.

[0159] Other types of unpinning commands are also possible. For example, a specified touch gesture or other input interaction can be used instead of the context-sensitive menu as described above.

[0160] Reference is now made to Figures 10A-10F , describing a sequence of pinable VUI screens for an example of user interaction according to a method using Figure 9 In Figure 10A , a first pinable VUI screen 1000 of a first application is displayed on the touchscreen 138. The first application is a web browser and the first pinable VUI screen 1000 is a web page having dynamic content including images and text displayed by the web browser. The user selects the image of "Hoover Dam" and swipes left to pin the selected image to a left pinning bar located at the left edge of the touchscreen 138 and the displayable area of the first pinable VUI screen 1000, which is not shown and is not visible to the user. As shown in Figure 10B , a first pin 1025 is generated in the first pinable VUI screen 1000 and is located on the left pinning bar in a position close to the image when pinned. The first pin 1025 includes a preview of the pinned image within a grid shaped like a tab or pin.

[0161] In Figure 10B ​In the example shown in FIG. 10A, the user interacts with the first pinable VUI screen 1000 to select the text in the form of a web page title "Hoover Dam Construction Photos Reveal Human Invention's Wonders" and swipe right to pin the selected text to a right-side pinning bar at the right edge of the displayable area of the touchscreen 138 and the first pinable VUI screen 1000, which is not shown and is not visible to the user. As shown in FIG. 10B, a second pin 1035 is generated and positioned in the right-side pinning bar in a position proximate to the text when pinned. The second pin 1035 includes a preview of the pinned text within a grid of shapes like tabs or pins. Figure 10C

[0162] In the example shown in FIG. 10A, the user interacts with the first pinable VUI screen 1000 to select the text in the form of a web page title "Hoover Dam Construction Photos Reveal Human Invention's Wonders" and swipe right to pin the selected text to a right-side pinning bar at the right edge of the displayable area of the touchscreen 138 and the first pinable VUI screen 1000, which is not shown and is not visible to the user. As shown in FIG. 10B, a second pin 1035 is generated and positioned in the right-side pinning bar in a position proximate to the text when pinned. The second pin 1035 includes a preview of the pinned text within a grid of shapes like tabs or pins. Figure 10D

[0163] In the example shown in FIG. 10A, the user interacts with the first pinable VUI screen 1000 to select the text in the form of a web page title "Hoover Dam Construction Photos Reveal Human Invention's Wonders" and swipe right to pin the selected text to a right-side pinning bar at the right edge of the displayable area of the touchscreen 138 and the first pinable VUI screen 1000, which is not shown and is not visible to the user. As shown in FIG. 10B, a second pin 1035 is generated and positioned in the right-side pinning bar in a position proximate to the text when pinned. The second pin 1035 includes a preview of the pinned text within a grid of shapes like tabs or pins. Figure 10E Figure 10F In the example shown in FIG. 10A, the user interacts with the first pinable VUI screen 1000 to select the text in the form of a web page title "Hoover Dam Construction Photos Reveal Human Invention's Wonders" and swipe right to pin the selected text to a right-side pinning bar at the right edge of the displayable area of the touchscreen 138 and the first pinable VUI screen 1000, which is not shown and is not visible to the user. As shown in FIG. 10B, a second pin 1035 is generated and positioned in the right-side pinning bar in a position proximate to the text when pinned. The second pin 1035 includes a preview of the pinned text within a grid of shapes like tabs or pins.

[0164] ​​​The fourth embodiment provides an interaction method in which content (e.g., images, videos, text, multimedia, or other data, etc.) can be collected from one or more different sources on a browsable visual user interface and organized into discrete parts during a discovery phase of the interaction method, and then added to an input field (e.g., an input content area or a document content area) during a composition phase of the interaction method to create a structured composition in a minimum number of interaction steps. Selected content can be pinned at one or more edges of the VUI screen by a mechanism called pinning. Pinning creates draggable objects called pins that are linked to the stored content. Various operations can be performed on the pins, such as accessing the source, deleting, changing position, web search, etc. The stored content can be added to a document such as an email, a text, an instant message, or a word processing document. The stored content of all pins can be placed at the same time, the stored content of a subset of pins can be placed at the same time, or the stored content can be placed one pin at a time. The position and / or size of the pins can be interpreted to have a logical or semantic meaning that can be used to automatically arrange the stored content based on predefined visual layout rules. If the content pane of the document to which the stored content is being added already has some content, the content pane presents the one or more pins at the corresponding location using a different visual style (e.g., a different color) so that the user can reference the existing content arrangement when placing the other pins.

[0165] Although the methods 300, 700, and 900 have been described individually, features of these methods can be combined in ways understood by those skilled in the art. For example, Figure 7 The execution of the list operation in the method 700 can be performed in the method 300.

[0166] At least some of the above-described embodiments provide a method of managing user interface items in a VUI that enables a user to naturally scroll a list to reposition, resulting in a faster, more efficient, more intuitive, and improved user experience; supports repositioning of multiple items at the same time, resulting in a faster, more efficient, and improved user experience; enables a user to complete a repositioning task when needed, eliminating the time requirement provided by existing solutions in which a user must continuously hold a moved item until the interaction is complete, and resulting in an improved user experience; and supports more accurate repositioning (e.g., unpinning or pinning an item), resulting in a faster, more efficient, and improved user experience, as compared to current available technology in which the next visible item is not known to the user (therefore, the user very frequently misses the target).

[0167] At least some of the above-described embodiments provide a method of managing user interface items in a VUI that supports reusability or easy transfer of content across applications, and reduces the time required to switch between screens and regenerate the content of one application in another application.

[0168] While the described embodiments are in the context of touch gestures detected by a touch screen, the teachings of the present disclosure are not limited to touch interfaces, but can be extended to other modalities and other input devices / methods for displaying content on other types of display devices. For example, the teachings of the present disclosure can be directed to other computing devices that have a computer mouse (or other navigation tool / pointing device) as an input device or use motion gestures as input methods and use other types of display devices to present a VUI, including but not limited to, personal computers with traditional non-touchscreen displays and augmented reality (AR) and / or virtual reality (VR) headsets with displays mounted on headsets, such as a pair of goggles, visors, or smart glasses. The VUI can be fixated on the respective display screen for display. Alternatively, in the context of AR / VR modalities, the fixated VUI can be projected onto a surface or the user’s retina. Examples of how VUI elements can be selectable and fixatable using gestures based on the input modality are described below.

[0169] With respect to computer mouse interactions, left and right clicks of the computer mouse can be used to perform fixation and unfixation. The computer mouse can be used to simulate touch interactions by using a left click (i.e., a button press of the left button of the computer mouse) as the starting point of the interaction, using the scroll wheel of the computer mouse for the scrolling action as the travel distance of the interaction, a right click (i.e., a button press of the right button of the computer mouse) to invoke a context menu, and a left click to select. Dragging a VUI component can be performed by a left or right click followed by moving the position of the computer mouse.

[0170] With respect to AR / VR, interactions can be performed using motion gestures (e.g., air gestures) that are detected using a computer vision system that is used to detect and track the user's body position using known computer vision techniques. The computer vision system can be based on one or a combination of image from a camera and point cloud position data from a detection and ranging (DAR) system. Gesture recognition software can be provided by the AR / VR display mounted on the headgear. The gesture recognition software receives data, such as images captured by a camera of the display mounted on the headgear that captures the user's interaction with the virtual display, and processes the images to recognize gestures performed by the user. For example, a hand splay or pinch splay gesture can be used to point, and a hand / fist close or pinch close gesture can be used to select. A hand / fist close or pinch close gesture can be used to drag a VUI component while moving the hand. A hand / fist close or pinch close gesture can be used to invoke a context menu. Alternatively, touch gestures performed with respect to a touch screen can be replaced, for example, with eye gestures performed with the eyes and detected with eye sensors or motion gestures performed with the hand and detected with motion sensors (e.g., position, proximity, and / or acceleration sensors) embedded in a glove, sleeve, belt, or similar garment worn by the user.

[0171] In one example AR application, a user can fix a real world object captured by the smart glasses to the edge of the visual field of the smart glasses, and then insert the fixed object into a different scene captured by the smart glasses and viewed by the user as an AR object corresponding to the fixed real world object. Fixing a real world object can include generating an AR object to be fixed by performing object edge detection at a location corresponding to a fix gesture performed by the user, and generating a visual representation of the object at the location as an AR object.

[0172] SUMMARY

[0173] The steps and / or operations in the flow charts and diagrams described herein are just for example. There can be many variations of the steps and / or operations without departing from the spirit of the invention. For instance, the steps can be performed in a different order, or where appropriate, added, deleted, modified, and / or combined.

[0174] In view of the present application, it is within the purview of one of ordinary skill in the art to code software for performing the above-described methods. Machine-readable code, executable by one or more processors of one or more respective devices to perform the above-described methods, can be stored in a machine-readable medium such as a memory of a data manager. The terms "software" and "firmware" are interchangeable, and include any computer program stored in memory for execution by a processor, including RAM memory, ROM memory, EPROM memory, EEPROM memory, and non-volatile RAM (NVRAM) memory. The above memory types are exemplary only, and are thus not limiting as to the types of memory usable for storage of a computer program.

[0175] All values and subranges within the disclosed ranges are also disclosed. In addition, although the systems, devices and processes described herein can include, among other things, a specific number of elements, these systems, devices and components can be modified in various ways, such as by adding or subtracting elements, reordering elements, or adding or reordering steps of the disclosed methods.

[0176] Features from one or more of the above-described embodiments can be selected to create alternative embodiments that are comprised of a sub-combination of features that can not have been explicitly described above. In addition, features from one or more of the above-described embodiments can be selected and combined to create alternative embodiments that are comprised of a combination of features that can not have been explicitly described above. Features so selected and combined are obvious in light of the present disclosure and are within the knowledge of one of ordinary skill in the art.

[0177] In addition, numerous specific details are set forth herein to provide a thorough understanding of the example embodiments described herein. However, it should be understood that the example embodiments described herein can be practiced without these specific details. In other instances, well-known methods, procedures and elements have not been described in detail so as not to obscure the example embodiments described herein. The subject matter described herein is intended to cover and encompass all appropriate technical equivalents.

[0178] While the application has been described, at least in part, in terms of methods, one skilled in the art will understand that the application also pertains to various elements for performing at least some aspects and features of the described methods by hardware, software, or a combination thereof. Thus, the technical solutions of the present application can be embodied in a non-volatile or non-transitory machine-readable medium (e.g., optical disc, flash memory, etc.) having stored thereon executable instructions tangibly stored thereon, which enable a processing device to perform examples of the methods disclosed herein.

[0179] The term “processor” can include any programmable system including systems using microprocessors / controllers or nano-processors / controllers, central processing units (CPUs), neural processing units (NPUs), tensor processing units (TPUs), hardware accelerators, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), reduced instruction set circuits (RISCs), logic circuits, and any other circuit or processor capable of executing the functions described herein. The term “database” can refer to a data body, a relational database management system (RDBMS), or both. A database as used herein can include any set of data, including hierarchical databases, relational databases, flat file databases, object- relational databases, object-oriented databases, and any other structured set of records or data stored in a computer system. The above examples are merely illustrative and thus are not intended to limit or restrict the definition and / or meaning of the term “processor” or “database” in any way.

[0180] The application can be embodied in other specific forms without departing from the subject matter of the claims. The described exemplary embodiments are to be considered in all respects only as illustrative and not restrictive. The application is intended to cover and encompass all appropriate technical equivalents. The scope of the application is therefore described by the appended claims rather than by the foregoing description. The scope of the claims should not be limited to the embodiments set forth in the examples, but should be given the broadest interpretation consistent with the entire description.

Claims

1. A method of managing user interface items in a visual user interface (VUI) performed by a processor of a computing device, the method comprising: The method comprises: displaying, on a display, a first VUI screen of a first application, the first VUI screen including a dynamic content area, the dynamic content area including selectable content, the first VUI screen providing a first pinning bar located at a first side of a display area of the first VUI screen and a second pinning bar located at a second side of the display area of the first VUI screen opposite the first side, wherein the first and second pinning bars persist across VUI screens of different applications; in response to detecting a pinning command, pinning selected content from the dynamic content area to the first or second pinning bar, wherein pinning comprises: displaying, on the display, a pin in a corresponding one of the first or second pinning bar, storing the selected content associated with the pin in a memory of the computing device; in response to detecting an input to switch an active application from the first application to a second application, displaying, on the display, a second VUI screen of the second application, wherein the second VUI screen includes an input bar, wherein a state of the first and second pinning bars persists between the first VUI screen of the first application and the second VUI screen of the second application; in response to detecting an unpinning command, adding stored content corresponding to one or more pins located in the first or second pinning bar to the input bar.

2. The method of claim 1, wherein, The first pinning bar is located at a left edge of the dynamic content area and the second pinning bar is located at a right edge of the dynamic content area.

3. The method according to claim 1 or 2, characterized in that, The dynamic content area of the first VUI screen and the input bar of the second VUI screen are scrollable in a vertical direction relative to a screen orientation, wherein the first and second pinning bars are defined by vertically extending areas at the left edge of the display area of the display and the right edge of the display area of the display.

4. The method according to any one of claims 1 to 3, characterized in that, The first and second pinning bars are provided without visual representation.

5. The method according to any one of claims 1 to 4, characterized in that, The stored content corresponding to the one or more pins is automatically arranged according to predefined visual layout rules based on a position of the one or more pins in the first or second pinning bar.

6. The method according to any one of claims 1 to 5, characterized in that, The stored content corresponding to the one or more pins is automatically arranged according to predefined visual layout rules based on a size of the one or more pins in the first or second pinning bar.

7. The method according to any one of claims 1 to 6, characterized in that, The state of the first and second pinning bars includes a position of pins located on the first and second pinning bars.

8. The method according to any one of claims 1 to 7, characterized in that, The state of the first and second pinning bars includes a size of pins located on the first and second pinning bars.

9. The method according to any one of claims 1 to 8, characterized in that, The state of the first pinning bar and the second pinning bar persists in response to changes in a visual state of the dynamic content area and persists across applications.

10. The method according to any one of claims 1 to 9, characterized in that, Also included are: invoking a context sensitive menu in response to a corresponding input associated with a selected pin, wherein the context sensitive menu includes a plurality of menu options; performing an action in response to an input selecting an option of the plurality of menu options.

11. A method of managing user interface items in a visual user interface (VUI) performed by a processor of a computing device, the method comprising: The method includes: displaying a VUI screen on a display, the VUI screen including a scrollable area including a plurality of selectable items, the VUI screen providing a pinning bar located at an edge of a display area of the display, wherein a state of the pinning bar persists in response to changes in a visual state of the scrollable area; in response to detecting a pinning command, pinning a selected item of the plurality of selectable items to the pinning bar, wherein the pinning includes: displaying a pin in the pinning bar proximate a location of the selected selectable item corresponding to the selected item; storing the selected item associated with the pin in a memory of the computing device; in response to detecting a corresponding input, changing the visual state of the scrollable area; in response to detecting an unpinning command, unpinning a pinned item corresponding to a selected pin from the pinning bar.

12. The method of claim 11, wherein, The pinning also includes: removing the selected item from the scrollable area; displaying an additional item.

13. The method according to claim 11 or 12, characterized in that, The unpinning includes: adding the stored item corresponding to the selected pin to the scrollable area at a location in the scrollable area corresponding to a location of the selected pin; removing a currently displayed item in the scrollable area to accommodate the added item; removing the selected pin from the pinning bar.

14. The method of claim 12, wherein, The additional item displayed is a subsequent item in a sequence in which the selectable items are ordered and arranged.

15. The method of claim 13, wherein, The unpinning includes: inserting the stored item corresponding to the selected pin into the scrollable area at a location in the sequence corresponding to a location of the selected pin; removing a currently displayed item in the sequence to accommodate the inserted item; removing the selected pin from the pinning bar.

16. The method of claim 15, wherein, The currently displayed item removed is a last item in the sequence.

17. The method of claim 15, wherein, The stored item is inserted before an item in the sequence proximate the location of the pin.

18. The method according to any one of claims 11 to 17, characterized in that, Also included are: repositioning a pin in the pinning bar in response to detecting a repositioning input.

19. The method according to any one of claims 11 to 18, characterized in that, Changing the visual state of the scrollable area includes: scrolling the scrollable area in response to detecting a scroll input.

20. The method of any one of claims 11 to 19, wherein, Changing the visual state of the scrollable area includes: performing a list operation on the selectable items of the scrollable area in response to detecting a corresponding input.

21. A computing device, comprising: Included are: a processor; a memory storing instructions that, when executed by the processor, cause the computing device to perform the method of any of claims 1-20.

22. A computer readable medium characterized by instructions that, when executed by a processor of a computing device, cause the computing device to perform the method of any one of claims 1-20.

23. A computer program, characterized in that, instructions that, when executed by a processor of a computing device, cause the computing device to perform the method of any one of claims 1-20.

Images