User interface provision device, user interface provision method, and user interface provision program

The XR device user interface system addresses hand-based limitations by using head movement to operate XR devices, ensuring hands-free and natural-looking interactions, enhancing outdoor usability and stability.

JP2024166505AInactive Publication Date: 2024-11-29SHARP KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023082644
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing operation methods for cross reality (XR) devices, such as AR glasses, are limited by the need for hand-based interactions which can be cumbersome outdoors and unnatural for onlookers, and button-based operations cause device shaking due to vibrations.

Method used

A user interface system using head-mounted devices that moves icons on a display based on head movement, allowing operations without hands and minimizing unnatural impressions, utilizing sensors to detect head movement and execute processes when icons are fixed in a setting area.

Benefits of technology

Enables hands-free operation of XR devices outdoors and reduces the unnatural appearance of hand-based interactions, preventing usage restrictions and device shaking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166505000001_ABST
    Figure 2024166505000001_ABST
Patent Text Reader

Abstract

To perform operation related to a head-mounted device, without using a hand, and without giving unnatural impression to the public present in the surrounding.SOLUTION: A user interface provision device comprises: an icon display unit which displays an icon on a display provided with a head-mounted device, and moves the icon according to movement of a head part detected by a sensor provided with the head-mounted device; and a process execution unit which executes processing according to the icon when a state in which the icon is displayed in a set area continues over set time.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a user interface providing device, a user interface providing method, and a user interface providing program. [Background technology]

[0002] Patent Document 1 discloses a communication system. In the communication system, augmented reality (AR) glasses are worn on a user's head. A motion controller is held in the user's hand. A smart device acquires the user's hand movement based on the detection result of an acceleration sensor transmitted from the motion controller (paragraphs 0017-0019). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-185235 A Summary of the Invention [Problem to be solved by the invention]

[0004] There are generally four types of operation methods for a cross reality (XR) device, such as the AR glasses disclosed in Patent Document 1: an operation method using a hand controller, an operation method using a smartphone controller, an operation method using hand tracking, and an operation method using an operating member on the main body.

[0005] In the operation method using a hand controller, a virtual beam emitted from the hand controller held by the user is aimed at an icon to select the icon, and the selection of the icon is confirmed when a button provided on the hand controller is pressed.

[0006] In an operation method using a smartphone controller, a virtual beam emitted from the smartphone held by the user is aimed at an icon to select the icon, and the selection of the icon is confirmed when a button on the smartphone is pressed.

[0007] In the hand tracking operation method, the shape of the user's hand is recognized by a camera provided on the XR device, a hand having a shape corresponding to the recognized hand shape is displayed, and when an icon is touched by the displayed hand, the selection of the icon is confirmed.

[0008] In an operation method using an operation member on the main body, a cursor is moved in response to an operation performed on an operation member such as a button or a touch pad provided on the XR device, and the selection of an icon is confirmed.

[0009] When an operation method using a hand controller or a smartphone controller is adopted, the user's hands are occupied, which limits the uses of the XR device when the XR device is used outdoors.

[0010] When a hand tracking operation method is adopted, the user must hold his / her hand in front of his / her eyes and move the held hand to operate the device, which gives an unnatural impression to the public around the user.

[0011] When a control method using the main unit buttons is adopted, the user must operate the control members of the main unit in order to perform the operation. This requires touching the XR device, which causes the image displayed by the XR device to shake.

[0012] In view of these problems, an aspect of the present disclosure provides a user interface providing device, a user interface providing method, and a user interface providing program that enable operations related to a head-mounted device to be performed without using hands and without giving an unnatural impression to the public around. [Means for solving the problem]

[0013] A user interface providing device of one aspect of the present disclosure includes an icon display unit that displays an icon on a display provided in a head-worn device and moves the icon in accordance with head movement detected by a sensor provided in the head-worn device, and a processing execution unit that executes processing corresponding to the icon when the icon remains displayed within a set area for a set period of time. [Brief description of the drawings]

[0014] [Figure 1] 1 is a front view diagrammatically illustrating a user interface providing system according to a first embodiment and a user wearing augmented reality (AR) glasses included in the user interface providing system. [Diagram 2] 1 is a block diagram of a user interface providing system according to a first embodiment. [Diagram 3] FIG. 2 is a plan view illustrating a schematic display of AR glasses provided in the user interface providing system of the first embodiment. [Figure 4] 1A to 1C are diagrams illustrating schematic changes in the display area of ​​a display of AR glasses provided in the user interface providing system of the first embodiment, and a head wearing the AR glasses. [Diagram 5] 1A to 1C are diagrams illustrating a schematic diagram of a change in a display area of ​​a display of AR glasses provided in the user interface providing system of the first embodiment. [Figure 6]5 is a flowchart showing the flow of processing performed by an icon display unit of a smartphone provided in the user interface providing system of the first embodiment. [Figure 7] 4 is a flowchart showing the flow of processing performed by a processing execution unit of a smartphone provided in the user interface providing system of the first embodiment. [Figure 8] FIG. 2 is a perspective view illustrating a visual effect obtained by the user interface providing system of the first embodiment. [Figure 9] 1 is a block diagram of a configuration for providing a smartphone controller user interface (UI) of a smartphone provided in a user interface providing device of the first embodiment. FIG. [Figure 10] 1 is a plan view illustrating a controller screen displayed on a touch panel display of a smartphone provided in a user interface providing device of the first embodiment. FIG. [Figure 11] 1A to 1C are diagrams illustrating a schematic diagram of a change in a display area of ​​a display of AR glasses provided in the user interface providing system of the first embodiment. [Figure 12] 1 is a plan view illustrating a schematic diagram of an application list display screen displayed on a touch panel display of a smartphone included in the user interface providing system of the first embodiment. FIG. [Figure 13] 1 is a plan view illustrating a gesture operation list display screen displayed on a touch panel display of a smartphone included in the user interface providing system of the first embodiment. FIG. [Figure 14] 2 is a block diagram of hardware of a smartphone included in the user interface providing system of the first embodiment. FIG. [Figure 15] 13A to 13C are diagrams illustrating schematic changes in the display area of ​​the AR glasses provided in the user interface providing system of the second embodiment, and the head when the AR glasses are worn. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0016] 1. First embodiment 1.1 User interface provision system FIG. 1 is a front view that diagrammatically illustrates a user interface providing system according to a first embodiment and a user wearing augmented reality (AR) glasses that are included in the user interface providing system.

[0017] 1 provides a user interface to a user 21 wearing AR glasses 11 included in the user interface providing system 1. The provided user interface includes a user interface for selecting an application and starting the selected application.

[0018] As shown in FIG. 1 , the user interface providing system 1 includes AR glasses 11 and a smartphone 12 .

[0019] The AR glasses 11 are worn on the head 31 of the user 21. The AR glasses 11 display images superimposed on the real world.

[0020] The AR glasses 11 may be replaced with cross reality (XR) glasses other than the AR glasses 11. For example, the AR glasses 11 may be replaced with virtual reality (VR) glasses, mixed reality (MR) glasses, alternative reality (SR) glasses, or the like. The AR glasses 11 may be replaced with glasses-type head mounted devices other than XR glasses. The AR glasses 11 may be replaced with head mounted devices other than glasses-type head mounted device glasses. For example, the AR glasses 11 may be replaced with head mounted devices such as goggles, hats, and headsets.

[0021] The smartphone 12 communicates with the AR glass 11 to receive information from the AR glass 11 and transmit information to the AR glass 11.

[0022] The smartphone 12 may be replaced with a mobile communication terminal other than the smartphone 12. For example, the smartphone 12 may be replaced with a feature phone, a tablet, or the like. The smartphone 12 may be replaced with a terminal other than a mobile communication terminal. For example, the smartphone 12 may be replaced with a personal computer, or the like. The function of the smartphone 12 may be built into the AR glasses 11.

[0023] FIG. 2 is a block diagram of the user interface providing system according to the first embodiment.

[0024] As shown in FIG. 2, the AR glasses 11 include a sensor 41 and a display 42.

[0025] The sensor 41 detects the movement of the head 31. The sensor 41 can detect the movement of three degrees of freedom (3DOF) or six degrees of freedom (6DOF). Therefore, the detected movement of the head 31 includes the movement of moving the viewpoint in the left and right directions. The sensor 41 includes an angular velocity sensor, an acceleration sensor, a geomagnetic sensor, etc.

[0026] The display 42 is disposed in front of the eyes of the head 31, and has optical transparency. Therefore, the display 42 displays an image superimposed on the real world seen through the display 42.

[0027] The smartphone 12 functions as a user interface providing device that provides a user interface to the user 21 .

[0028] 2, the smartphone 12 includes an icon display unit 51 and a processing execution unit 52. The AR glasses 11 may function as a user interface providing device. In this case, the AR glasses 11 include the icon display unit 51 and the processing execution unit 52. The AR glasses 11 and the smartphone 12 may function in cooperation as a user interface providing device. In this case, the AR glasses 11 include the icon display unit 51 and a part of the processing execution unit 52, and the smartphone 12 includes the icon display unit 51 and the remaining part of the processing execution unit 52.

[0029] The icon display unit 51 displays a plurality of icons on the display 42. A plurality of applications are associated with the displayed plurality of icons, respectively. A plurality of functions may be associated with the plurality of icons, respectively. The icon display unit 51 moves the displayed plurality of icons in accordance with the movement of the head 31 detected by the sensor 41.

[0030] When one of the displayed icons continues to be displayed in a set area for a set period of time, the process execution unit 52 executes a process corresponding to that icon.

[0031] As a result, the user 21 can move the icon corresponding to the process to be executed into the setting area by moving the head 31, and then stop moving the head 31 and stare at the icon, thereby causing the user interface providing system 1 to execute the process to be executed. This allows the user 21 to perform operations related to the AR glasses 11 without using his / her hands and without giving an unnatural impression to the public around him / her. This makes it possible to suppress restrictions on the use of the AR glasses 11 even when the AR glasses 11 are used outdoors.

[0032] 1.2 Quick menu screen FIG. 3 is a plan view that illustrates a schematic display of the AR glasses provided in the user interface providing system of the first embodiment.

[0033] 3, the display 42 has a display area 61 that displays an image. The display area 61 is a part of a light-transmitting lens unit 60 provided in the AR glasses 11. Therefore, outside the display area 61, there is a light-transmitting area.

[0034] A setting area 71 is set in the display area 61. The setting area 71 is preferably set in the center of the display area 61. Note that the dashed lines indicating the display area 61 and the setting area 71 drawn in FIG. 3 are not actually displayed on the display 42.

[0035] The user 21 can see the real world beyond the display area 61 through the display area 61. If the AR glasses 11 are equipped with a camera, the camera may capture an image of the real world, and the display 42 may display the captured real world in the display area 61, thereby enabling the user 21 to see the real world.

[0036] The icon display unit 51 displays a quick menu screen 81 in the display area 61 .

[0037] The displayed quick menu screen 81 includes a plurality of icons 91. The plurality of icons 91 is five icons. The plurality of icons 91 may be four or less, or six or more icons. The plurality of icons 91 is preferably three or more and five or less icons.

[0038] The multiple icons 91 are arranged in a line in the left-right direction.

[0039] A plurality of applications are associated with the plurality of icons 91, respectively.

[0040] 1.3 Moving icons FIG. 4 is a diagram illustrating a schematic view of a display area of ​​a display of the AR glasses included in the user interface providing system of the first embodiment, and changes in a head wearing the AR glasses.

[0041] The icon display unit 51 moves the multiple icons 91 left and right in response to the movement of the head 31 which moves the viewpoint left and right. The icon display unit 51 moves the multiple icons 91 in the direction opposite to the direction in which the viewpoint moves due to the movement of the head 31. For this reason, as shown in FIG. 4, when the user 21 turns to the right and the viewpoint moves rightward, the multiple icons 91 move leftward, which is opposite to the rightward direction. The movement distance of the multiple icons 91 increases as the movement distance of the viewpoint increases. In this way, the user 21 can move the viewpoint toward an icon corresponding to a process to be executed, thereby moving that icon into the setting area 71.

[0042] The icon display unit 51 moves the multiple icons 91 in the same direction by the same distance, thereby making it possible to prevent the relative positions of the multiple icons 91 from changing.

[0043] The arrangement intervals of the multiple icons 91 and the movement distances of the multiple icons 91 are preferably determined so that when the field of view (FOV) is 45°, the swing angle of the head 31 required to move an icon displayed next to an icon displayed at the center of the setting area 71 to the center of the setting area 71 is about 10° to 15°. As a result, when the multiple icons 91 are five icons and the icon 101 located in the middle is displayed at the center of the setting area 71, the swing angle of the head 31 required to move the icon 102 located at the far right to the center of the setting area 71 can be set to about 25°. As a result, an arbitrary icon can be selected from the multiple icons 91 by a slight movement of the head 31 that does not cause the user 21 any pain.

[0044] When the horizontal width of a collection of multiple icons 91 is greater than the width of the display area 61 and the entire collection cannot be displayed in the display area 61, the icon display unit 51 performs scrolling.

[0045] 1.4 What happens FIG. 5 is a diagram illustrating a schematic change in the display area of ​​the display of the AR glasses included in the user interface providing system of the first embodiment.

[0046] 5, when one icon 111 included in the plurality of icons 91 continues to be displayed in the setting area 71 for a first set time, the processing execution unit 52 executes a display process of selecting the one icon 111 and displaying an effect 121 indicating that the one icon 111 has been selected in the display area 61. The first set time is, for example, 0.5 seconds. The displayed effect 121 is, for example, an animation of a line 131 running along the periphery of the one icon 111.

[0047] When the entirety of the one icon 111 is displayed within the setting region 71, the process execution unit 52 determines that the one icon 111 is displayed within the setting region 71. When a majority of the one icon 111 is displayed within the setting region 71, the process execution unit 52 may determine that the one icon 111 is displayed within the setting region 71. For example, when 90% or more of the one icon 111 is displayed within the setting region 71, the process execution unit 52 may determine that the one icon 111 is displayed within the setting region 71.

[0048] When the one icon 111 is shifted out of the display area 61 and the state in which the one icon 111 is displayed in the display area 61 does not continue for a first set time, the process execution unit 52 does not select the one icon 111 and does not execute the display process.

[0049] When the state in which the one icon 111 is displayed in the setting area 71 continues for a second setting time longer than the first setting time, the process execution unit 52 ends the execution of the display process and executes a start process for starting an application associated with the one icon 111. As a result, the application screen 141 of the started application is displayed in the display area 61. The second setting time is, for example, one second.

[0050] If the state in which the one icon 111 is displayed in the display area 61 continues for a first set time and then the one icon 111 is shifted out of the display area 61 and the state in which the one icon 111 is displayed in the display area 61 does not continue for a second set time, the process execution unit 52 deselects the one icon 111 and does not execute the startup process.

[0051] When the state in which the one icon 111 or another icon is displayed in the setting area 71 continues for a first set time after the one icon 111 is shifted out of the display area 61, the process execution unit 52 executes a display process for displaying an effect 121 indicating that the one icon 111 or another icon has been selected in the display area 61. Furthermore, when the state in which the one icon 111 or another icon is displayed in the setting area 71 continues for a second set time, the process execution unit 52 executes a start-up process for starting an application associated with the one icon 111 or another icon.

[0052] 1.5 Icons As illustrated in FIG. 3, FIG. 4, and FIG. 5, the plurality of icons 91 includes a previously executed icon 151 associated with the previously executed application, and a carrier icon 152 associated with a carrier application provided by a mobile communication carrier. If the previously executed application is not a carrier application, the carrier icon 152 is an icon different from the previously executed icon 151 and is arranged next to the previously executed icon 151. For example, if the previously executed application is not a carrier application, the carrier icon 152 is arranged to the left of the previously executed icon 151. If the previously executed application is a carrier application, the carrier icon 152 is the same icon as the previously executed icon 151. The plurality of icons 91 may include an icon associated with a home application, an icon associated with a dedicated application such as an MR application, or the like.

[0053] When the icon display unit 51 displays the quick menu screen 81 in the display area 61, it starts displaying a plurality of icons 91 in the display area 61 with the previously executed icon 151 being displayed in the setting area 71.

[0054] Therefore, when the user 21 wants to execute the application that was executed immediately before again, the user 21 does not need to move the head 31 to move the multiple icons 91, but only needs to stare at the icon that was initially displayed in the setting area 71.

[0055] This is because it is believed that there will be many opportunities to execute the application that was executed immediately before again. For example, if the application executed immediately before is a translation application, it is believed that there will be many opportunities to execute the translation application again, and if the application executed immediately before is a city XR application, it is believed that there will be many opportunities to execute the city XR application again.

[0056] 1.6 Flow of processing performed by the icon display unit FIG. 6 is a flowchart showing the flow of processing performed by the icon display unit of the smartphone provided in the user interface providing system of the first embodiment.

[0057] The icon display unit 51 executes steps S101 to S104 shown in FIG.

[0058] In step S101, the icon display unit 51 starts displaying a plurality of icons 91 on the display .

[0059] In the following step S 102 , the icon display unit 51 acquires the detection result from the sensor 41 .

[0060] In the next step S103, the icon display unit 51 determines whether or not there is a movement of the head 31 based on the acquired detection result. If it is determined that there is a movement of the head 31, step S104 is executed. If it is determined that there is no movement of the head 31, step S102 is executed again.

[0061] In step S 104 , the icon display unit 51 moves the plurality of icons 91 in accordance with the movement of the head 31 .

[0062] Through steps S102 to S104, if there is no movement of the head 31, the icon display unit 51 does not move the multiple icons 91, and if there is movement of the head 31, moves the multiple icons 91 in accordance with the movement of the head 31.

[0063] 1.7 Processing flow performed by the processing execution unit FIG. 7 is a flowchart showing the flow of processing performed by the processing execution unit of the smartphone provided in the user interface providing system of the first embodiment.

[0064] The process execution unit 52 executes steps S111 to S118 shown in FIG.

[0065] In step S111, the process execution unit 52 determines whether or not any of the icons included in the plurality of icons 91 is in a state of being displayed within the setting area 71. If it is determined that the icon is in a state of being displayed within the setting area 71, step S112 is executed. If it is determined that the icon is not in a state of being displayed within the setting area 71, step S111 is executed again.

[0066] In step S112, the process execution unit 52 starts measuring the elapsed time.

[0067] In steps S111 and S112, when any icon included in the plurality of icons 91 is not in a state where it is being displayed in the setting area 71, the process execution unit 52 waits until any icon included in the plurality of icons 91 is in a state where it is being displayed in the setting area 71. In addition, in synchronization with any icon included in the plurality of icons 91 being in a state where it is being displayed in the setting area 71, the process execution unit 52 starts measuring the time that has elapsed since the icon in question was in a state where it was displayed in the setting area 71.

[0068] In step S113, the process execution unit 52 determines whether or not the icon is now in a state of being displayed outside the setting area 71. If it is determined that the icon is now in a state of being displayed outside the setting area 71, step S111 is executed again. If it is determined that the icon is not now in a state of being displayed outside the setting area 71, step S114 is executed.

[0069] In step S114, the process execution unit 52 determines whether the measured time has reached the first set time. If it is determined that the measured time has reached the first set time, step S115 is executed. If it is determined that the measured time has not reached the first set time, step S113 is executed again.

[0070] In step S115, the process execution section 52 starts displaying the effect 121.

[0071] In steps S113 to S115, if the icon is displayed outside the setting area 71 before the time that has elapsed since the icon was displayed within the setting area 71 reaches the first set time, the process execution unit 52 waits until any icon included in the plurality of icons 91 is displayed within the setting area 71 again, without starting to display the effect 121. Moreover, the process execution unit 52 starts displaying the effect 121 in synchronization with the time that has elapsed since the icon was displayed within the setting area 71 reaching the first set time.

[0072] In step S116, the process execution unit 52 determines whether or not the icon is now in a state of being displayed outside the setting area 71. If it is determined that the icon is now in a state of being displayed outside the setting area 71, step S111 is executed again. If it is determined that the icon is not now in a state of being displayed outside the setting area 71, step S117 is executed.

[0073] In step S117, the process execution unit 52 determines whether the measured time has reached the second set time. If it is determined that the measured time has reached the second set time, step S118 is executed. If it is determined that the measured time has not reached the second set time, step S116 is executed again.

[0074] In step S118, the process execution unit 52 starts up the application associated with that icon.

[0075] According to steps S116 to S118, when the icon is displayed outside the setting area 71 before the time that has elapsed since the icon was displayed within the setting area 71 reaches the second set time, the process execution unit 52 waits until any icon included in the plurality of icons 91 is displayed again within the setting area 71, without starting the application associated with the icon. Moreover, the process execution unit 52 starts the application associated with the icon in synchronization with the time that has elapsed since the icon was displayed within the setting area 71 reaching the second set time.

[0076] 1.8 Visual Effects FIG. 8 is a perspective view that diagrammatically illustrates a visual effect obtained by the user interface providing system of the first embodiment.

[0077] 8, the user interface providing system 1 gives the user 21 a visual effect in which a plurality of icons 91 appear to be arranged in front of the user 21. The user interface providing system 1 also gives the user 21 a visual effect in which the plurality of icons 91 appear to be arranged on a gentle arc surface.

[0078] The user 21 can cause the user interface providing system 1 to select one of the icons 91 by moving the viewpoint left and right so that the viewpoint is placed on that one icon included in the plurality of icons 91. In this way, the user interface providing system 1 produces a visual effect that gives the impression that the plurality of icons 91 are fixed in space and that the viewpoint can be moved seamlessly by moving the head 31 in 3DOF or 6DOF.

[0079] 1.9 Smartphone Controller User Interface As described above, the user interface providing system 1 provides a user interface (hereinafter referred to as a "center focus UI") that enables the user to move the head 31 to move an icon corresponding to the processing to be executed within the setting area 71, and then stop moving the head 31, thereby causing the user interface providing system 1 to execute the processing to be executed.

[0080] In order to enable operations that are difficult to perform using a center focus UI, the user interface providing system 1 may provide a user interface that uses the smartphone 12 as a controller (hereinafter, referred to as a "smartphone controller UI") The smartphone controller UI is used for operations on existing applications and the like that do not support operations performed using a center focus UI.

[0081] When the user interface providing system 1 provides a center focus UI and a smartphone controller UI, the center focus UI becomes the main operating means for the AR application, and the smartphone controller UI becomes the sub-operating means. The reason why the center focus UI becomes the main operating means is that, when the AR glasses 11 are used outdoors, it is desirable that the AR application be operated in a hands-free manner.

[0082] FIG. 9 is a block diagram of a configuration for providing a smartphone controller UI of a smartphone provided in the user interface providing device of the first embodiment. FIG. 10 is a plan view that typically illustrates a controller screen displayed on a touch panel display of a smartphone provided in the user interface providing device of the first embodiment. FIG. 11 is a diagram that typically illustrates a change in the display area of ​​a display of AR glasses provided in the user interface providing system of the first embodiment. FIG. 12 is a plan view that typically illustrates an application list display screen displayed on a touch panel display of a smartphone provided in the user interface providing system of the first embodiment. FIG. 13 is a plan view that typically illustrates a gesture operation list display screen displayed on a touch panel display of a smartphone provided in the user interface providing system of the first embodiment.

[0083] As shown in FIG. 9, the smartphone 12 includes a controller unit 161 and a touch panel display 162.

[0084] The controller unit 161 causes the touch panel display 162 to display a controller screen 171 shown in FIG.

[0085] As shown in FIG. 10, the controller screen 171 includes a touch pad operation area 181, an end button 182, a menu button 183, a back button 184, an application list button 185, and a gesture operation list button 186.

[0086] When a slide operation is performed on the touchpad operation area 181, the controller unit 161 moves the plurality of icons 91 in a direction corresponding to the slide direction of the performed slide operation by a distance corresponding to the slide amount of the slide operation. Therefore, the slide operation is an alternative operation to the operation of moving the viewpoint by moving the head 31.

[0087] When a double tap operation is performed on the touchpad operation area 181, the controller unit 161 executes a start-up process for starting an application associated with an icon displayed in the setting area 71. Therefore, the double tap operation is an alternative operation to stopping the movement of the head 31 and gazing at the icon.

[0088] When a tap operation is performed on the end button 182 while an application is running, the controller unit 161 ends the running application and puts the display 42 into a non-display state 191 shown in FIG.

[0089] When a tap operation is performed on the MENU button 183 while the quick menu screen 81 is not displayed, the controller unit 161 displays the quick menu screen 81 on the display 42. When a tap operation is performed on the MENU button 183 while the quick menu screen 81 is displayed, the controller unit 161 ends the display of the quick menu screen 81 on the display 42 and sets the display 42 to a non-display state 191 illustrated in Fig. 11. The controller unit 161 may display the quick menu screen 81 on the display 42 when the smartphone 12 is connected to the AR glasses 11, when a short press operation is performed on a power button provided on the smartphone 12, when it is detected that the smartphone 12 has been removed from its case, or the like.

[0090] When a tap operation is performed on the BACK button 184 while an application is running, the controller unit 161 notifies the running application of a BACK event.

[0091] When a tap operation is performed on the application list button 185, the controller unit 161 causes the touch panel display 162 to display an application list display screen 201 showing a list of applications, as illustrated in FIG.

[0092] When a tap operation is performed on the gesture operation list button 186, the controller unit 161 causes a gesture operation list display screen 202 showing a list of gesture operations as illustrated in Fig. 13 to be displayed on the touch panel display 162. The gesture operation list indicates gesture operations supported by the user interface providing system 1.

[0093] The hardware keys provided on the smartphone 12 are, in principle, used for operations independent of the application.

[0094] 1.10 Smartphone Hardware FIG. 14 is a block diagram of hardware of a smartphone included in the user interface providing system of the first embodiment.

[0095] As shown in FIG. 14, the smartphone 12 includes a processor 211, a memory 212, and a storage 213.

[0096] The processor 211 includes a central processing unit (CPU), a graphics processing unit (GPU), and the like.

[0097] The memory 212 includes a random access memory (RAM), a read-only memory (ROM), and the like.

[0098] The storage 213 includes flash storage and the like.

[0099] A user interface providing program 221 is installed in the storage 213 .

[0100] The processor 211 executes the user interface providing program 221 loaded into the memory, and causes the computer including the processor 211, the memory 212, and the storage 213 to operate as the icon display unit 51, the process execution unit 52, and the controller unit 161 described above.

[0101] Therefore, the present disclosure is also directed to a user interface providing program 221 recorded in a non-transitory recording medium such as the memory 212, the storage 213, etc. The user interface providing program 221 causes a computer including the processor 211, the memory 212, and the storage 213 to execute the processes performed by the icon display unit 51, the process execution unit 52, and the controller unit 161.

[0102] 2. Second embodiment The following describes the differences between the second embodiment and the first embodiment. For points that are not described, the second embodiment employs the same configuration as that employed in the first embodiment.

[0103] FIG. 15 is a diagram illustrating a schematic view of a display area of ​​a display of the AR glasses included in the user interface providing system of the second embodiment, and a change in a head wearing the AR glasses.

[0104] In the second embodiment, as shown in FIG. 15, the plurality of icons 91 are six icons arranged in a matrix in the up-down and left-right directions.

[0105] Furthermore, the icon display unit 51 moves the plurality of icons 91 in the up, down, left and right directions in response to the movement of the viewpoint in the up, down, left and right directions. Therefore, as shown in Fig. 15, when the user 21 faces the upper left direction and the viewpoint moves in the upper left direction, the plurality of icons 91 move in the lower right direction, which is opposite to the upper left direction.

[0106] This can reduce the width in the left-right direction of the collection of the multiple icons 91. This can reduce the need for scrolling, and can improve the visibility of the multiple icons 91.

[0107] The present disclosure is not limited to the above-described embodiments, and may be replaced with a configuration that is substantially the same as the configuration shown in the above-described embodiments, a configuration that has the same action and effect, or a configuration that can achieve the same purpose. [Explanation of symbols]

[0108] 1 User interface providing system, 11 AR glasses, 12 Smartphone, 21 User, 31 Head, 41 Sensor, 42 Display, 51 Icon display unit, 52 Processing execution unit, 61 Display area, 71 Setting area, 81 Quick menu screen, 91 Multiple icons, 101 Icon arranged in the middle, 102 Icon arranged on the rightmost side, 111 Single icon, 121 Effect, 131 Line, 141 Application screen, 151 Last executed icon, 152 Carrier icon, 161 Controller unit, 162 Touch panel display, 171 Controller screen, 181 Touch pad operation area, 182 Exit button, 183 MENU button, 184 BACK button, 185 Application list button, 186 Gesture operation list button, 191 Hidden state, 201 Application list display screen, 202 Gesture operation list display screen, 211 Processor, 212 Memory, 213 Storage, 221 User interface providing program.

Claims

1. an icon display unit that displays an icon on a display provided in a head-mounted device and moves the icon in response to a head movement detected by a sensor provided in the head-mounted device; a process execution unit that executes a process corresponding to the icon when the icon is continuously displayed within a set area for a set time; A user interface providing device comprising:

2. The process is a display process for displaying an effect on the display indicating that the icon has been selected. The user interface providing device according to claim 1 .

3. The effect is an animation that runs a line around the perimeter of the icon. The user interface providing device according to claim 2 .

4. the set time is a first set time, The process execution unit executes a start process for starting an application associated with the icon when a state in which the icon is displayed within the setting area continues for a second setting time that is longer than the first setting time. The user interface providing device according to claim 2 .

5. The process is a startup process for starting an application associated with the icon. The user interface providing device according to claim 1 .

6. the display has a display area for displaying an image; The setting area is set in the center of the display area. The user interface providing device according to claim 1 .

7. The icon display unit displays a plurality of icons including the icon on the display, and moves the plurality of icons in the same direction by the same distance in response to the movement. The user interface providing device according to claim 1 .

8. The plurality of icons are arranged in a row in the left-right direction, The movement is a movement of moving the viewpoint left and right. The user interface providing device according to claim 7 .

9. The plurality of icons are arranged in a matrix in the up-down and left-right directions, The movement is in the up, down, left and right directions. The user interface providing device according to claim 7 .

10. the plurality of icons includes a previously executed icon associated with a previously executed application, The icon display unit starts displaying the plurality of icons in a state in which the previously executed icon is displayed within the setting area. The user interface providing device according to claim 7 .

11. The icon display unit moves the icon in a direction opposite to a direction in which the viewpoint moves due to the movement. The user interface providing device according to claim 1 .

12. A user interface providing device according to any one of claims 1 to 11, The head-mounted device; A user interface providing system comprising:

13. Displaying an icon on a display provided on the head-mounted device; moving the icon in response to head movement detected by a sensor provided in the head worn device; executing a process corresponding to the icon when the icon is continuously displayed within a set area for a set time; A method for providing a user interface comprising:

14. Displaying an icon on a display provided on the head-mounted device; moving the icon in response to head movement detected by a sensor provided in the head worn device; executing a process corresponding to the icon when the icon is continuously displayed within a set area for a set time; A user interface providing program that causes a computer to execute the above.

Citation Information

Patent Citations

  • Communication system, communication method, and communication program

    JP2022185235A