Information processing device, system, control method, and computer program

Abstract

To perform suitable operation using a portable terminal in an MR experience.SOLUTION: An information processing device includes image generation means for generating an image to be displayed on a display device attachable to the head of a user, operation mode determination means for determining an operation mode of an application, and application operation means for receiving input operation from a portable terminal held by the user to be operable to perform application operation on the basis of the operation mode and the input operation. The operation mode determination means determines at least either a first operation mode for performing the application mode by allowing the user to operate the portable terminal or a second operation mode for performing the application operation by displaying a virtual UI about the operation in the display device to allow the user to operate the virtual UI.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to application operations using a mobile terminal in XR experiences.

Background Art

[0002] In recent years, XR (X Reality) technology that fuses the virtual world and the real world in real time to create a new experience has been known. In XR, a user uses a head-mounted device (HMD: Head Mount Display) to perform operations while viewing images. When performing operations, it is common to use a controller held by the user. There is a system that uses a mobile terminal such as a smartphone or a tablet with which the user is familiar for the controller.

[0003]

[0004] In VR (Virtual Reality) where the user experiences the virtual world and the real world is not shown, the real mobile terminal cannot be visually recognized. Also, in MR (Mixed Reality) where the user experiences a composite real world in which information of the virtual world is superimposed on the real world, there are cases where the real mobile terminal is superimposed on virtual objects and cannot be visually recognized.

[0005] For example, Patent Document 1 discloses a method in which a mobile terminal of a virtual object is prepared in a virtual space, and information on operations performed on the real mobile terminal is displayed on the mobile terminal of the virtual object so that the user can visually recognize the operations on the mobile terminal.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

[0007] However, in mixed reality (MR), where images from the virtual world are superimposed onto images from the real world, there are cases where the real mobile device is superimposed onto the virtual object and becomes invisible, and cases where it is visible without being superimposed. In environments where the real mobile device is visible, superimposing a virtual mobile device onto the real mobile device may make intuitive operation difficult due to superposition errors. Also, if the virtual mobile device is displayed without superimposing it onto the real mobile device, the virtual and real mobile devices will be displayed separately, which may hinder immersion. In other words, displaying a virtual mobile device is not always preferable.

[0008] Therefore, this disclosure proposes a technology for performing optimal operations using a mobile device in an MR experience. [Means for solving the problem]

[0009] To achieve the above objective, a first aspect of the present invention is: Image generation means for generating images to be displayed on a display device that can be worn on the user's head, An operation mode determination means for determining the operation mode of an application, An application operation means that receives input operations from a portable terminal that can be held and operated by the user, and performs application operations based on the operation mode and the input operations, Equipped with, The display device has an imaging means, The image generation means generates a composite reality image by superimposing a virtual object onto the captured image captured by the imaging means. The operation mode determination means determines whether the user can see the mobile terminal based on whether the mobile terminal can be seen in the mixed reality image without being obstructed by the virtual object, The aforementioned operation mode determination means is The user's gaze direction is obtained, If the user can see the mobile device Furthermore, the line of sight is directed toward the mobile device.In this case, the system is set to a first operation mode in which the user operates the application by operating the mobile terminal, and the user cannot see the mobile terminal. In cases where the line of sight is not directed towards the mobile device. In this case, the system determines a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application by operating the virtual UI. This is an information processing device characterized by the following features.

[0010] A second aspect of the present invention is, A display device that can be worn on the user's head and control means for controlling the display of a GUI on a portable terminal that the user can grasp and operate, An application operation means that accepts input operations using the GUI and performs application operations based on the input operations, Equipped with, The GUI displayed on at least one of the display device and the mobile device differs depending on whether the user can see the mobile device or not. This is an information processing device characterized by the following features.

[0011] A third aspect of the present invention is, A display device that can be worn on the user's head and control means for controlling the display of a GUI on a portable terminal that the user can grasp and operate, An application operation means that accepts input operations using the GUI and performs application operations based on the input operations, Equipped with, The control means controls the display of the GUI, which includes a virtual UI for application operation, on the display device and the display of the GUI for application operation on the mobile terminal to occur simultaneously. The application operation means performs the application operation based on at least one of the operations on the GUI including the virtual UI and the operations on the GUI displayed on the mobile terminal. This is an information processing device characterized by the following features.

[0012] The present invention The The four aspects are as follows A method for controlling a computer, comprising an image generation step of generating an image to be displayed on a display device wearable on a user's head an operation mode determination step of determining an operation mode of an application an application operation step of receiving an input operation from a portable terminal that can be held and operated by the user and performing an application operation based on the operation mode and the input operation and including the display device has imaging means the image generation step generates a composite reality image in which a virtual object is superimposed on an imaging image captured by the imaging means in the operation mode determination step, it is determined whether the user can visually recognize the portable terminal based on whether the portable terminal can be visually recognized without being blocked by the virtual object in the composite reality image in the operation mode determination step The user's gaze direction is obtained, if the user can visually recognize the portable terminal Furthermore, the line of sight is directed toward the mobile device. it is determined as a first operation mode in which the user performs an application operation by operating the portable terminal if the user cannot visually recognize the portable terminal In cases where the line of sight is not directed towards the mobile device. it is determined as a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user performs an application operation by operating the virtual UI The control method is characterized in that

[0013] The fifth aspect of the present invention is a method for controlling a computer, comprising a control step of controlling the display of a GUI on a display device wearable on a user's head and a portable terminal that can be held and operated by the user a determination step of determining whether the user can visually recognize the portable terminal receiving an input operation using the GUI and performing an application operation based on the input operation The application operation steps for performing the operation, Includes, The GUI displayed on at least one of the display device and the mobile device differs depending on whether the user can see the mobile device or not. This is a control method characterized by the following:

[0014] A sixth aspect of the present invention is: A method of controlling a computer, A control step for controlling the display of a GUI in a display device that can be worn on the user's head and a portable terminal that the user can grasp and operate, An application operation step that accepts input operations using the GUI and performs application operations based on the input operations, Includes, In the control step, the display of the GUI, which includes a virtual UI for application operation, on the display device and the display of the GUI for application operation on the mobile terminal are controlled to occur simultaneously. In the application operation step, the application operation is performed based on at least one of the operations on the GUI including the virtual UI and the operations on the GUI displayed on the mobile device. This is a control method characterized by the following: [Effects of the Invention]

[0015] According to the present invention, it is possible to provide suitable operation using a mobile device in an MR experience. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram showing an example of the functional configuration of the system according to the first embodiment. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] This figure shows a specific example of the first operation mode according to the first embodiment. [Figure 4]This figure shows a specific example of the second operation mode according to the first embodiment. [Figure 5] This is a flowchart of the application operation according to the first embodiment. [Figure 6] This is a flowchart of the application operation in the first operation mode according to the first embodiment. [Figure 7] This is a flowchart of the application operation in the second operation mode according to the first embodiment. [Figure 8] This figure shows a specific example of the first operation mode according to the second embodiment. [Figure 9] This is a flowchart of the application operation in the first operation mode according to the second embodiment. [Figure 10] This figure shows a specific example of the second operation mode according to the second embodiment. [Figure 11] This is a flowchart of the application operation in the second operation mode according to the second embodiment. [Figure 12] This diagram shows an example of a mobile device's hardware configuration. [Modes for carrying out the invention]

[0017] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0018] (First Embodiment) In the first embodiment, the operating mode is determined by whether or not the mobile terminal can be seen by the HMD, and an example is shown in which an application operation is performed in which text is added to a virtual object based on the operating mode.

[0019] <Structure> Figure 1 is a block diagram showing an example of the system configuration according to this embodiment. In this embodiment, the information processing device 100 is connected to the HMD 101 and the mobile terminal 102. The connection between the devices may be wired or wireless. Furthermore, the connection between the devices may be achieved by combining wired and wireless connections.

[0020] [Information Processing Device] First, the information processing device 100 will be described. The information processing device 100 receives input operations from the mobile terminal 102 and can perform application operations based on the set operation mode and input operations. Figure 2 is a hardware configuration diagram of the information processing device 100 in this embodiment. As shown in Figure 2, the information processing device 100 has a configuration similar to that of a well-known general-purpose computer and includes a CPU 201, storage device 202, ROM 203, RAM 204, input I / F 205, and output I / F 206.

[0021] The CPU 201 provides centralized control over each device connected via the bus 200. The CPU 201 reads and executes processing steps and computer programs stored in the ROM 203 and RAM 204.

[0022] The storage device 202 stores programs and data for use by the information processing device 100, which includes various programs according to this embodiment.

[0023] ROM203 stores the operating system (OS), device drivers, and boot programs.

[0024] RAM204 has a work area that temporarily stores programs and data loaded from storage device 202 and ROM203, and allows CPU201 to execute each process as appropriate.

[0025] The input interface 205 receives signals from external devices, including the mobile terminal 102, as input signals in a format that can be processed by the information processing device 100.

[0026] Output I / F206 outputs signals to external devices, including the HMD101, as output signals in a format that the external devices can process.

[0027] The CPU 201 reads and executes the program, thereby enabling the information processing device 100 to perform the functions shown in Figure 1.

[0028] The image acquisition unit 103 acquires images of the real world (real images) captured by the imaging unit 114 of the HMD 101. The image acquisition unit 103 outputs the acquired images to the HMD position and orientation measurement unit 104, the mobile terminal position and orientation measurement unit 105, and the mixed reality image generation unit 113.

[0029] The HMD position and orientation measurement unit 104 performs image processing using the real image (captured image) acquired by the image acquisition unit 103, and measures the position and orientation of the HMD 101 by extracting characteristic information such as points and lines in the image. The position and orientation of the HMD 101 can be considered equivalent to the user's line of sight or head position and orientation. The position and orientation information measured by the HMD position and orientation measurement unit 104 is input to the determination unit 106, the virtual UI generation unit 108, and the virtual image generation unit 112.

[0030] This embodiment employs position and orientation measurement using image processing, but the present invention is not limited thereto. The position and orientation of the HMD may be measured using infrared light, or at least one of an ultrasonic sensor, a magnetic sensor, or a depth sensor.

[0031] The mobile terminal position and orientation measurement unit 105 performs image processing using the real-world image acquired by the image acquisition unit 103 to measure the position and orientation of the mobile terminal 102. The position and orientation of the mobile terminal 102 can be measured by displaying a mobile terminal-specific index (marker) on the terminal display unit 118 and estimating the position and orientation of the index displayed on the real-world image. The index is not particularly limited as long as it has a shape that can be uniquely identified.

[0032] This embodiment employs position and orientation measurement of the mobile terminal 102 using indicators, but the present invention is not limited thereto. The position and orientation of the mobile terminal 102 may be measured using infrared light, or at least one of an ultrasonic sensor, a magnetic sensor, a depth sensor, or an angular velocity sensor. Alternatively, the position and orientation of the mobile terminal 102 may be measured using GPS (Global Positioning System).

[0033] The determination unit 106 determines whether the user can see the mobile terminal 102 through the HMD 101. The determination unit 106 makes the above determination based, for example, on the position and orientation of the HMD 101 determined by the HMD position and orientation measurement unit 104 and the position and orientation of the mobile terminal 102 determined by the mobile terminal position and orientation measurement unit 105, and further considering the field of view of the HMD 101 and the display size of the mobile terminal 102. More specifically, the determination unit 106 determines whether the user can see the mobile terminal 102 through the HMD 101 depending on whether the display of the mobile terminal is included within the field of view of the HMD. The determination unit 106 inputs the determination result to the operation mode determination unit 107.

[0034] In this embodiment, the determination unit 106 determines whether the user can view the mobile terminal 102 through the HMD 101 based on the position and orientation of the HMD 101 and the mobile terminal 102, but the present invention is not limited thereto. The determination unit 106 may make the above determination by determining by image recognition whether or not the mobile terminal is included in the real image (captured image) acquired by the image acquisition unit 103. In this case, the determination unit 106 may also determine whether the HMD 101 and the mobile terminal 102 are facing each other in order to determine whether or not the display of the mobile terminal 102 is reflected in the real image of the HMD 101. Preferably, when the real image is displayed on the HMD 101, the determination unit 106 adds to the determination result whether or not the display of the mobile terminal 102 reflected in the real image has a resolution that allows the user to view it with the HMD 101. More preferably, the determination unit 106 adds to the determination result whether or not the HMD 101 has a resolution that allows the user to view not only the display of the mobile terminal 102 but also the information displayed on the display with the HMD 101. Furthermore, the determination unit 106 may also determine whether the mobile terminal 102 is visible in the mixed reality image displayed on the HMD 101. That is, the determination unit 106 may determine whether or not a virtual object is superimposed and drawn over the area of ​​the mobile terminal 102 in the real image in the mixed reality image, which is created by superimposing a virtual space image onto a real image. If a virtual object is not superimposed and drawn over the area of ​​the mobile terminal 102 in the real image, it means that the mobile terminal 102 is visible to the user in the mixed reality image without being obstructed by a virtual object. The determination unit 106 determines that the user can see the mobile terminal 102 via the HMD 101 if the display of the mobile terminal 102 and the information displayed on the display have a resolution that allows the user to see them. Also, the determination unit 106 determines that the user can see the mobile terminal 102 via the HMD 101 if no virtual object is superimposed over the area of ​​the mobile terminal 102 in the real image and the mobile terminal 102 is visible in the mixed reality image without being obstructed by a virtual object. However, the determination unit 106 may determine that the user can see the mobile terminal 102 through the HMD 101 if some of the conditions described above are met.

[0035] The operation mode determination unit 107 determines the application's operation mode based on the determination result of the determination unit 106. Specifically, if the operation mode determination unit 107 determines that the user can see the mobile terminal 102 through the HMD 101, it determines the operation mode to the first mode, in which the user directly operates the mobile terminal 102 to operate the application. If the operation mode determination unit 107 determines that the user cannot see the mobile terminal 102, it determines the operation mode to the second mode, in which a virtual object of the operation UI (virtual UI) is displayed in a virtual space, and the user operates the application by operating the virtual UI. The first and second operation modes can also be described as modes in which the GUI displayed for application operation is different from each other. Here, different GUIs mean that at least a part of either the GUI displayed on the display unit of the mobile terminal 102 or the GUI including the virtual UI displayed on the HMD 101 is different. Even if the operation mode determination unit 107 determines that the user can see the mobile terminal 102 through the HMD 101, it may determine the operation mode to the second mode if the user's gaze is not directed towards the mobile terminal 102. The operation mode determination unit 107 inputs the determined operation mode to the virtual UI generation unit 108 and the operation mode receiving unit 116 of the mobile terminal 102. The operation mode determination unit 107, which inputs the determined operation mode to the virtual UI generation unit 108 and the operation mode receiving unit 116, corresponds to the control means that controls the display of the GUI on the HMD 101 and the mobile terminal 102.

[0036] In this embodiment, an example of the first operation mode for adding text to a virtual object is shown in Figures 3(A) to 3(C), and an example of the second operation mode is shown in Figures 4(A) to 4(C).

[0037] First, the first operation mode will be explained with reference to Figures 3(A) to 3(C). Figure 3(A) shows the MR space during the first operation mode. In the MR space where the virtual object 300 exists, the user 301 experiences the environment using an HMD 302 worn on the head and a mobile terminal 303 held as a controller. Figure 3(B) is the mixed reality image displayed on the HMD 302 in Figure 3(A). The mixed reality image is an image in which the virtual object 300 is superimposed on a real image including the mobile terminal 303. As shown in Figure 3(C), the display unit of the mobile terminal 303 shows an index 304 for the information processing device 100 to measure the position and orientation of the mobile terminal 303, and a keyboard 305, which is a GUI for the user to add text. The user adds text to the virtual object 300 by operating the keyboard 305.

[0038] Next, the second operation mode will be explained with reference to Figures 4(A) to 4(B). The same reference numbers are used for the same configuration as in Figures 3(A) to 3(C), and their explanations will be omitted. Figure 4(A) shows the MR space in the second operation mode. Unlike Figure 3(A), in Figure 4(A), the user 301 is not visually viewing the mobile terminal 303. In the second operation mode, a virtual keyboard 400 as a virtual UI and a pointer 401 for key selection are provided so that the user 301 can add text without using the mobile terminal 303. Figure 4(B) is the mixed reality image displayed on the HMD 302 in Figure 4(A). The mixed reality image is an image in which a virtual object 300, a virtual keyboard 400, and a pointer 401 are superimposed on a real image. The user adds text to the virtual object 300 by moving the pointer 401 by operating the mobile terminal 303 and inputting the key that is focused on by the pointer 401 through a confirmation operation. The virtual UI may be operated by means other than the user operating the mobile device 303. For example, the virtual UI may be operated based on the position and orientation of at least a part of the user's body, such as fingers, head, or gaze, or the position and orientation of the mobile device 102. The position and orientation of the user's body can be measured by performing image recognition processing on real images captured by the HMD 101.

[0039] The virtual UI generation unit 108 determines the operation mode input from the operation mode determination unit 107 to be the second operation When the operation mode is set to the first operation mode, the virtual UI generation unit 108 generates a virtual UI, but when the operation mode is set to the first operation mode, the virtual UI generation unit 108 does not generate a virtual UI. The virtual UI includes graphics objects (virtual objects) composed of GUI elements such as buttons, like the virtual keyboard 400 in Figure 4(B). The virtual UI further includes pointing objects, which are virtual objects for performing selection or confirmation operations, such as the pointer 401.

[0040] This embodiment employs selection and decision operations using the pointer 401, but the present invention is not limited to this as long as it is possible to select and decide on virtual objects. The selection and decision of virtual objects may be performed, for example, by casting rays displayed based on the position and orientation of the mobile device. Graphics objects may be placed in the position and orientation displayed on the HMD 101 according to the position and orientation input from the HMD position and orientation measurement unit 104. For example, as described in Patent Document 2, a technique can be considered in which the viewing frustum of the HMD 101 is calculated and the graphics object is placed in the overlapping region of the viewing frustums of the left and right eyes so as to be perpendicular to the line of sight. Alternatively, graphics objects may be placed in a position and orientation relative to the virtual object to be manipulated.

[0041] The pointing object determines its position and orientation according to the selection operation input from the operation receiving unit 109. Details will be explained using the flowchart described later. The virtual UI generation unit 108 inputs the generated virtual UI to the virtual image generation unit 112.

[0042] The operation receiving unit 109 receives operation signals from the operation unit 119 of the mobile terminal 102. The operation signals will be explained later in the section on the operation unit 119. The operation receiving unit 109 interprets these operation signals and converts them into operation events such as selection operations and confirmation operations. If a confirmation operation is input, the operation receiving unit 109 notifies the application operation unit 110. In the second operation mode, the operation receiving unit 109 also inputs the operation event to the virtual UI generation unit 108.

[0043] The application operation unit 110 receives input operations from the mobile terminal 102 via the operation receiving unit 109 and performs application operations based on the currently set operation mode and the received input operations. In the first operation mode, the application operation unit 110 performs application operations based on input operations using the GUI displayed on the mobile terminal 102. In the second operation mode, the application operation unit 110 performs application operations based on input operations using the GUI, including the virtual UI, displayed on the HMD 101.

[0044] The virtual data holding unit 111 holds data related to virtual objects that constitute the virtual space (shape information and position / orientation information), data related to light sources that illuminate the virtual space, and inputs this data to the virtual image generation unit 112.

[0045] The virtual image generation unit 112 constructs a virtual space from the virtual UI generated by the virtual UI generation unit 108 and virtual objects acquired from the virtual data holding unit 111. The virtual image generation unit 112 further generates an image of the virtual space (virtual image) as seen from the position and orientation measured by the HMD position and orientation measurement unit 104. The virtual image generation unit 112 inputs the generated virtual image to the mixed reality image generation unit 113. Note that the technology for generating an image of the virtual space as seen from a predetermined position is a known technology, so a detailed explanation of it will be omitted.

[0046] The mixed reality image generation unit 113 generates an image to be displayed on the HMD 101. More specifically, the mixed reality image generation unit 113 generates an image in a mixed reality space (mixed reality image) by superimposing a virtual image generated by the virtual image generation unit 112 onto a real image acquired by the image acquisition unit 103. The mixed reality image generation unit 113 outputs the generated mixed reality image to the display unit 115 of the HMD 101.

[0047] The mixed reality image generation unit 113 corresponds to the image generation means that generates images to be displayed on the HMD 101. The image generation means may also be means that instructs other devices or application modules to generate images to be displayed on the HMD 101. For example, if the virtual image generation unit 112 and the mixed reality image generation unit 113 are implemented by devices or modules other than the information processing device 100, the information processing device 100 may instruct the virtual image generation unit 112 and the mixed reality image generation unit 113 to generate mixed reality images. In this case, the information processing device that issues such instructions can be interpreted as corresponding to the image generation means.

[0048] [HMD] Next, I will explain the HMD101.

[0049] The HMD101 is a display device that can be worn on the user's head and has a display unit composed of liquid crystal images or the like. The HMD101 has a display unit for the right eye and a display unit for the left eye, which are mounted so as to be in front of the user's right and left eyes when the HMD101 is worn on the user's head. In addition, the left and right display units (screens) each display images with parallax in stereo. In this embodiment, the HMD101 is described as a video see-through type HMD that displays a composite reality image generated based on a real image captured by an imaging device on the display device. However, the present invention is not limited thereto. The HMD101 may also be an optical see-through type HMD that displays a virtual space image superimposed on a display medium that can be observed by passing through real space.

[0050] The imaging unit 114 captures the real space displayed on the HMD 101 and inputs the captured real-world image to the information processing device 100.

[0051] The display unit 115 displays a composite reality image generated by the information processing device 100.

[0052] [Mobile devices] The mobile terminal 102 will now be described. The mobile terminal 102 is used as a controller that the user can hold and operate. The term "mobile terminal" refers to a general-purpose device, such as a smartphone or tablet. Figure 12 is a hardware configuration diagram of the mobile terminal 102 in this embodiment.

[0053] The CPU 1201 provides comprehensive control over each device connected via the bus 1200. The CPU 1201 reads and executes processing steps and programs stored in the ROM 1203 and RAM 1204.

[0054] The storage device 1202 stores programs and data for use in the mobile terminal 102, including various programs according to this embodiment.

[0055] ROM1203 stores the operating system (OS), device drivers, and boot programs.

[0056] RAM1204 has a work area that temporarily stores programs and data loaded from storage device 1202 and ROM1203, and allows CPU 1201 to execute each process as appropriate.

[0057] The input device 1205 is a device for providing data, information, instructions, etc., and includes a built-in IMU (Internal Measurement Unit), touch panel, and microphone. The input obtained from these is entered as an input signal in a format that can be processed. To be empowered.

[0058] Output device 1206 is a device for receiving and physically presenting the output signal. Examples include displays, speakers, and vibrators.

[0059] The following describes the functions of the mobile terminal 102 according to this embodiment.

[0060] The operation mode receiving unit 116 receives the operation mode from the operation mode determination unit 107 of the information processing device 100 and inputs it to the operation unit 119. In addition, in the case of the first operation mode, the operation mode receiving unit 116 also inputs the operation mode to the operation UI generation unit 117.

[0061] The operation UI generation unit 117 generates a GUI for performing application operations if the operation mode received by the operation mode receiving unit 116 is the first operation mode. The operation UI generation unit 117 inputs the generated GUI to the terminal display unit 118.

[0062] The terminal display unit 118 displays the UI generated by the operation UI generation unit 117. The terminal display unit 118 is located on the front side of the mobile terminal 102 and consists of a display capable of displaying liquid crystal images and the like.

[0063] The operation unit 119 inputs an operation signal corresponding to the input operation of the mobile terminal to the operation UI generation unit 117 or the operation receiving unit 109 of the information processing device 100. Input operations include touch gestures such as tapping and swiping on the touch panel, and back taps detected based on angular acceleration measured by the built-in IMU. In the case of touch gestures, an operation signal is generated based on at least one of the touch position, distance moved, or direction of movement. Furthermore, the position and orientation (position, orientation, and changes thereof) of the mobile terminal 102, acquired by the IMU in response to an input operation of moving the mobile terminal 102, may also be used as an operation signal.

[0064] First, we will explain the behavior of the control unit 119 when the operating mode is set to the first operating mode.

[0065] If the input operation is a decision operation that performs an action on the application, the operation unit 119 inputs an operation signal based on the focused UI to the operation receiving unit 109. If the input operation is a movement or switching operation on the UI, the operation unit 119 may send an operation UI generation unit 117 to update the operation UI. Movement and switching operations may include swiping or dragging operations on the touch panel of the mobile terminal 102, and may also include changes in the orientation of the mobile terminal 102. A decision operation may include tapping on the touch panel of the mobile terminal.

[0066] Next, we will explain the behavior of the control unit 119 when the operating mode is set to the second operating mode.

[0067] The operation unit 119 inputs an operation signal to the operation receiving unit 109 regardless of the type of operation, such as whether the input operation is a movement or switching operation or a decision operation.

[0068] <Processing> Figure 5 is an example of a flowchart showing the process of determining the operation mode and performing application operations. Hereafter, each step will be described by adding the letter S to the beginning of its symbol.

[0069] First, in step S500, the HMD position and orientation measurement unit 104 measures the position and orientation of the HMD 101 based on the captured image acquired from the image acquisition unit 103. Then, the process moves on to step S501.

[0070] In step S501, the mobile terminal position and orientation measurement unit 105 measures the position and orientation of the mobile terminal 102 based on the captured image acquired from the image acquisition unit 103. Then, the process moves on to step S502.

[0071] In step S502, the determination unit 106 determines whether the mobile terminal 102 can be seen by the HMD 101. If it can be seen, the process moves to step S503; otherwise, the process moves to step S504.

[0072] In step S503, application operations are performed using the first operation mode. Details will be explained in the flowchart below. Then, the process moves on to step S505.

[0073] In step S504, application operations are performed using the second operation mode. Details will be explained in the flowchart below. Then, the process moves on to step S505.

[0074] In step S505, if the user enters a command to terminate the application, or if the conditions for terminating the application are met, this process terminates. On the other hand, if no command to terminate the application is entered and the conditions for termination are not met, the process returns to step S500.

[0075] [First Operation Mode] Figure 6 shows a flowchart of the application operation using the first operation mode shown in step S503.

[0076] In step S600, the operation mode determination unit 107 transmits to the operation mode receiving unit 116 that it is in the first operation mode. Then the process moves on to step S601.

[0077] In step S601, the operation mode receiving unit 116 notifies the operation UI generation unit 117 that it is in the first operation mode. Upon receiving the notification, the operation UI generation unit 117 generates a keyboard 305 as shown in Figure 3(C) and displays it on the terminal display unit 118 of the mobile terminal 102. Then, the process moves on to step S602.

[0078] In step S602, the operation unit 119 receives whether or not the user has performed a tap operation on the keyboard 305 displayed on the mobile terminal 102. If a tap operation is received, the process moves to step S603. If no tap operation is received, the process moves to step S505.

[0079] In step S603, the operation unit 119 transmits a key input signal corresponding to the key that was tapped to the operation receiving unit 109. Then, the process moves on to step S604.

[0080] In step S604, the operation receiving unit 109 notifies the application operation unit 110 that the corresponding key has been pressed based on the received key input signal. The application operation unit 110 generates the text for the corresponding key as a virtual object and places it in the virtual space. As a result, the virtual image generation unit 112 generates a virtual image containing the text, and the mixed reality image generation unit 113 superimposes it onto the real image to add the text. Then, the process moves on to step S505.

[0081] In this embodiment, upon receiving notification that it is in the first operation mode, the operation UI is displayed on the display unit of the mobile terminal 102, but the present invention is not limited thereto. For example, the operation UI may always be displayed on the display unit of the mobile terminal 102 regardless of the operation mode, and input may only be displayed when it is in the first operation mode. It's fine to accept them.

[0082] This allows application operation to be performed by inputting commands into the mobile terminal 102.

[0083] [Second Operation Mode] Figure 7 shows a flowchart of the application operation using the second operation mode shown in step S504.

[0084] In step S700, the operation mode determination unit 107 transmits to the virtual UI generation unit 108 and the operation mode receiving unit 116 that it is in the second operation mode. Then the process moves on to step S701.

[0085] In step S701, the operation mode receiving unit 116 notifies the virtual UI generation unit 108 that it is in the second operation mode. Upon receiving the notification, the virtual UI generation unit 108 generates a virtual keyboard 400 and a pointer 401 as shown in Figure 4 and places them in the virtual space. The virtual UI generation unit 108 inputs this placement result to the virtual image generation unit 112. Then, the process moves on to step S702.

[0086] In step S702, the operation mode receiving unit 116 notifies the operation unit 119 that it is in the second operation mode. The operation unit 119 determines whether the user has performed a swipe operation on the mobile terminal's touch panel. If a swipe operation is received, the process moves to S703. If no swipe operation is received, the process moves to S705.

[0087] In step S703, the operation unit 119 notifies the operation receiving unit 109 of the distance and direction the finger has moved on the screen by the swipe operation as a movement operation signal. Then, the process moves on to step S704. In this embodiment, a method for obtaining a movement operation signal by a swipe operation of the mobile terminal 102 is described, but the present invention is not limited thereto. The movement operation signal may also be obtained by a flick operation of the mobile terminal 102, or it may be obtained based on the amount of change in the orientation of the mobile terminal.

[0088] In step S704, the operation receiving unit 109 notifies the virtual UI generation unit 108 of the received movement distance and direction. The virtual UI generation unit 108 changes the position of the pointer 401 based on the acquired movement distance and direction. Then, the process moves to step S505. This makes it possible to change the key on the virtual keyboard 400 that is in focus.

[0089] In step S705, the operation unit 119 determines whether the user has performed a tap operation on the touch panel of the mobile terminal 102. If a tap operation is received, the process moves to step S706. If no tap operation is received, the process moves to step S505.

[0090] In step S706, the operation unit 119 notifies the operation receiving unit 109 of the decision operation as an operation signal. Then the process moves on to step S707.

[0091] In step S707, the operation receiving unit 109 notifies the application operation unit 110 that a decision operation has been performed. The application operation unit 110 determines the focused key based on the position and orientation of the pointer 401 relative to the virtual keyboard 400 generated by the virtual UI generation unit 108, and sets it as the input key. Then, the process moves on to step S708.

[0092] In step S708, the application operation unit 110 generates the text of the key corresponding to the input key as a virtual object and places it in the virtual space. This is done by the virtual image generation unit 11 In step 2, a virtual image containing text is generated, and this is superimposed onto the real image by the composite reality image generation unit 113, thereby adding text. Then the process moves on to step S505.

[0093] This allows users to operate the application by reflecting input operations on the mobile device in a virtual UI.

[0094] <Advantageous effects of this embodiment> According to this embodiment, when the mobile device is visible, application operation is possible based on the UI displayed on the mobile device's screen, and when the mobile device is not visible, application operation is possible based on the UI displayed in the virtual space. This enables optimal application operation in the MR experience.

[0095] (Variation 1) In the first embodiment, an example was shown in which, each time the position and orientation of the HMD 101 and the mobile terminal 102 are measured, it is determined whether or not the mobile terminal 102 can be seen by the HMD 101, and the operation mode is determined based on the determination result to perform application operation, but the present invention is not limited thereto. The operation mode does not need to be changed regardless of the position and orientation of the HMD 101 and the mobile terminal 102 from the time an operation based on the determined operation mode is started until the same operation is completed or canceled. That is, the operation starts when a key is entered in the first or second operation mode, and the operation mode does not need to be changed until the text generation created by multiple key inputs is confirmed or the generation is canceled.

[0096] (Modification 2) In the first embodiment, an example was shown in which the HMD 101 determines whether or not the mobile terminal 102 can be seen, and the operation mode is determined based on the determination result to perform application operation, but the present invention is not limited thereto. The operation mode may also be determined based on user input. User input can be performed by operating the mobile terminal. User input may also be performed by the UI and mouse, or by a button-type device such as a keyboard. Furthermore, user input may be performed by recognizing the user's gestures or by recognizing their voice. When using gestures and voice for input, the mobile terminal 102 needs to be configured to recognize touch gestures and voice, respectively.

[0097] (Variation 3) In the first embodiment, an example was described in which the operation mode determination unit 107 uniquely determines the operation mode, but the present invention is not limited thereto. An operation mode that utilizes both the first operation mode and the second operation mode may be adopted. In this modified example, for example, the virtual UI is displayed regardless of whether the mobile terminal 102 is visible or not, and application operation by the virtual UI is always possible. In this modified example, if the mobile terminal 102 is visible, application operation by operating the mobile terminal 102 is also possible. In other words, in this modified example, if the mobile terminal 102 is visible, both the first operation mode and the second operation mode are enabled, and if the mobile terminal 102 is not visible, the second operation mode is enabled.

[0098] The compatibility of the first operation mode and the second operation mode can be achieved by the information processing device 100 performing the following processing.

[0099] The information processing device 100 instructs the virtual UI generation unit 108 to generate a GUI including a virtual UI and display it on the HMD 101, and also notifies the operation mode receiving unit 116 of the mobile terminal 102 of the operation mode and instructs the mobile terminal 102 to display the GUI. The processing unit 100 controls the display of the GUI, including the virtual UI, on the HMD 101 and the GUI on the mobile terminal 102 to occur simultaneously. Note that "simultaneous display" does not mean that the displays start at the same time, but rather that the GUI is displayed on both the HMD 101 and the mobile terminal 102 for at least a certain period of time.

[0100] First, the information processing device 100 measures the position and orientation of the user's fingers using a measurement unit (not shown) and inputs the measurement result to the operation receiving unit 109. The operation receiving unit 109 determines whether the fingers and the GUI intersect based on the position and orientation of the user's fingers measured by the measurement unit and the position and orientation of the GUI included in the generated virtual UI. If they intersect, the operation receiving unit 109 inputs an operation signal to the application operation unit 110, indicating that a decision operation has been input. On the other hand, if they do not intersect, the operation receiving unit 109 determines that no input operation has been performed by the user and does not input an operation signal to the application operation unit 110. The operation receiving unit 109 also notifies the application operation unit 110 of the input signal sent from the operation unit 119 of the mobile terminal 102.

[0101] This makes it possible to operate the application both by manipulating the virtual UI with the user's fingers and by operating the mobile device. In other words, the information processing device 100 can perform application operations based on at least one of the operations on the GUI, including the virtual UI, and the operations on the GUI of the mobile device 102.

[0102] In this embodiment, an example of operating the virtual UI using the user's fingers as a second operation mode has been described, but the present invention is not limited thereto. In the second operation mode, the user may also operate using their gaze or by voice.

[0103] In this modified example, the operating mode is switched depending on whether the user can see the mobile terminal 102. However, the first and second operating modes may always be enabled regardless of whether the user can see the mobile terminal 102. Alternatively, the user may be able to set whether to enable both the first and second operating modes, and the operating mode determination unit 107 may determine the operating mode based on this setting. That is, if the setting allows for both operating modes to be enabled, the operating mode determination unit 107 determines both the first and second operating modes as operating modes. If the setting does not allow for both modes to be enabled, the operating mode determination unit 107 determines the operating mode according to whether the user can see the mobile terminal 102.

[0104] (Modification 4) The functional configuration of the information processing device 100, HMD 101, and mobile terminal 102 described in the first embodiment is merely an example. Some functions may be performed by devices other than those described above, or they may be performed by multiple devices. For example, the information processing device 100 is not limited to one unit but may be composed of multiple information processing devices. Also, for example, a function described as a function of the information processing device 100, such as the operation mode determination unit 107, may be performed by the HMD 101 or the mobile terminal 102.

[0105] (Second embodiment) In the first embodiment, the case where the same application operation, such as adding text using the keyboard, is performed regardless of the operation mode was described. However, in this embodiment, the case where the application operations that can be performed differ depending on the operation mode is shown.

[0106] When working with 3DCG, as seen in DCC (Digital Content Creation) tools, it is common to have two UIs: First, a 2D UI such as a tree view or property view allows for precise or detailed operation. There is a UI for performing actions. Next, there is a UI that displays a 3D space and allows for intuitive manipulation of that space using a 3D UI such as Drucker's. It is desirable to have a similar UI in XR, which deals with 3DCG.

[0107] The first operation mode, which involves viewing a mobile device through an HMD and operating the application, is suitable for precise and detailed operation using a 2D UI. The second operation mode, which involves displaying a 3D UI in XR space and operating the application, is suitable for spatial or intuitive operation.

[0108] This embodiment describes the case in which an application operation is performed to change the arrangement of virtual objects. Note that the configuration of the information processing device 100, HMD 101, and mobile terminal 102 in this embodiment (Figures 1, 2, and 12) is the same as in the first embodiment, so a description will be omitted.

[0109] Using Figure 8, we will first explain an example of changing the placement of virtual objects using the first operation mode. The same reference numbers are used for the same configuration as in Figure 3, and their explanations will be omitted. Figure 8(A) shows the MR space in the first operation mode. A virtual desk 800 and a virtual cabinet 801 are placed in the MR space. Figure 8(B) is the mixed reality image seen by the HMD 302 in Figure 8(A). The mixed reality image includes a mobile terminal 303. The display unit of this mobile terminal 303 shows a two-dimensional UI for performing operations as shown in Figure 8(C). This two-dimensional UI includes a tree UI 802 that displays a list of virtual objects placed in the MR space in a tree structure, and a property UI 803 that displays detailed settings for the virtual object selected in the tree UI. The user can change the placement of virtual objects by selecting the virtual object to move from the tree UI 802 and inputting its position and orientation numerically in the property UI 803.

[0110] Figure 9 is a flowchart of the process in step S503, which performs an application operation to change the arrangement of virtual objects in the first operation mode. Processes similar to those in Figure 6 are given the same reference numbers, and their explanations are omitted.

[0111] In step S900, the operation mode receiving unit 116 notifies the operation UI generation unit 117 that it is in the first operation mode. Upon receiving the notification, the operation UI generation unit 117 generates a tree UI 802 and a property UI 803 as shown in Figure 8(C), and displays them on the terminal display unit 118 of the mobile terminal 102. Then, the process moves on to step S901.

[0112] In step S901, the operation unit 119 receives whether or not the user has performed a tap operation on the tree UI 802 displayed on the mobile terminal 102. If a tap operation is received, the process moves to step S902. If no tap operation is received, the process moves to step S904.

[0113] In step S902, the operation unit 119 transmits information about the virtual object selected in the tree UI 802 to the operation UI generation unit 117 and the operation receiving unit 109. For example, the operation unit 119 transmits to the operation UI generation unit 117 and the operation receiving unit 109 that desk 800 has been selected. The operation UI generation unit 117 displays the detailed settings of the selected virtual object in the property UI 803. Then, the process moves to step S903.

[0114] In step S903, the operation receiving unit 109 notifies the application operation unit 110 of the information of the selected virtual object. The application operation unit 110 updates the virtual space to indicate that the corresponding virtual object has been selected. The selection of a virtual object may be indicated by displaying a bounding box around the virtual object or by changing the color of the virtual object. Then, the process moves on to step S904.

[0115] In step S904, the operation unit 119 receives whether or not the user has performed a change operation on the property UI 803 displayed on the mobile terminal. In this embodiment, a change operation refers to an operation to change a numerical value indicating the position of a virtual object. If a change operation is received, processing moves to step S905. If no change operation is received, processing moves to step S505.

[0116] In step S905, the operation unit 119 transmits change information of the selected virtual object to the operation receiving unit 109. Then, the process moves on to step S906.

[0117] In step S906, the operation receiving unit 109 notifies the application operation unit 110 of the change information of the selected virtual object. Based on the change information, the application operation unit 110 changes the placement of the corresponding virtual object in the virtual space. As a result, the virtual image generation unit 112 generates a virtual image with the virtual object's placement changed, and this is superimposed on the real image by the mixed reality image generation unit 113, thereby realizing the change in the virtual object's placement.

[0118] Next, an example of changing the placement of virtual objects using the second operation mode will be explained using Figure 10. The same reference numbers are used for the same configuration as in Figure 8, and their explanations will be omitted. Figure 10(A) shows the MR space in the second operation mode. Ray 1000, which is a pointing object, is placed based on the position and orientation of the mobile terminal 303. Figure 10(B) is the mixed reality image seen on the HMD 302 in Figure 10(A). The placement of virtual objects can be changed by selecting and moving the virtual objects pointed to by Ray 1000.

[0119] Figure 11 is a flowchart showing the process of step S504, which is an application operation that changes the arrangement of virtual objects in the second operation mode. Processes similar to those in Figure 7 are given the same reference numbers, and their explanations are omitted.

[0120] In step S1100, the operation mode receiving unit 116 notifies the virtual UI generation unit 108 that it is in the second operation mode. Upon receiving the notification, the virtual UI generation unit 108 generates a ray 1000 in the virtual space as shown in Figure 10(A). The position and orientation of the ray 1000 are updated based on the position and orientation measured by the mobile terminal position and orientation measurement unit 105. Then, the process moves on to step S1101.

[0121] In step S1101, the virtual UI generation unit 108 determines whether the ray 1000 intersects with a virtual object. If they intersect, it updates the virtual space to indicate that the corresponding virtual object has been selected. Then, it moves the process to S1102. If they do not intersect, it moves the process back to S1101.

[0122] In step S1102, the operation unit 119 receives whether or not the user has performed a tap operation. If a tap operation is received, the operation unit 119 sends a selection operation to the operation receiving unit 109. Then, the process moves to step S1103. If no tap operation is received, the process moves to step S1101.

[0123] In step S1103, the operation receiving unit 109 notifies the application operation unit 110 of the selection operation. The application operation unit 110 changes the settings so that the position and orientation of the selected virtual object are updated to follow the position and orientation of the ray. Then, the process moves on to step S1105.

[0124] In step S1104, the operation unit 119 receives whether or not the user has performed a tap operation. If a tap operation is received, the operation unit 119 sends the selection operation to the operation receiving unit 109. Send the data. Then, proceed to step S1105. If no tap operation is received, proceed to step S1106.

[0125] In step S1105, the operation receiving unit 109 notifies the application operation unit 110 of the selection operation. The application operation unit 110 stops updating the virtual object's position and orientation to follow the ray's position and orientation, and positions the virtual object so that its position and orientation at the time the selection operation is received is fixed. Then, the process moves to step S505. As a result, the virtual image generation unit 112 generates a virtual image with the virtual object's arrangement changed, and this is superimposed on the real image by the mixed reality image generation unit 113, thereby realizing the change in the virtual object's arrangement.

[0126] In step S1106, the operation unit 119 receives whether or not the user has performed a flick operation. If a flick operation is received, the operation unit 119 sends a cancel operation to the operation receiving unit 109. Then, the process moves to step S1107. If no flick operation is received, the process moves to step S1103. In this embodiment, the case in which a cancel operation is assigned to a flick operation is described, but the present invention is not limited to this. The input operation assigned to the cancel operation only needs to be distinguishable from the confirm operation. That is, it may be a double-tap operation, or it may be that the amount of change in the posture of the mobile terminal is greater than or equal to a threshold.

[0127] In step S1107, the operation receiving unit 109 notifies the application operation unit 110 of the cancellation operation. The application operation unit 110 stops updating the virtual object's position and orientation to follow the ray's position and orientation, and returns it to its position and orientation before it started following the ray. Then, the process moves on to step S505.

[0128] According to this embodiment, by changing the application operations that can be performed on the mobile device depending on whether the mobile device is visible or not, it becomes possible to perform optimal application operations in the MR experience.

[0129] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0130] (Note) This disclosure includes the following configuration and method. [Configuration 1] Image generation means for generating images to be displayed on a display device that can be worn on the user's head, An operation mode determination means for determining the operation mode of an application, An application operation means that receives input operations from a portable terminal that can be held and operated by the user, and performs application operations based on the operation mode and the input operations, Equipped with, The operation mode determination means determines at least one of the following as the operation mode: a first operation mode in which the user operates the application by operating the mobile terminal, and a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application by operating the virtual UI. An information processing device characterized by the following: [Configuration 2] The operation mode determination means determines the operation mode to the first operation mode if the user can see the mobile terminal, based on the determination result of whether the user can see the mobile terminal, and determines the operation mode to the second operation mode if the user cannot see the mobile terminal. The information processing device according to configuration 1, characterized in that it is a device. [Configuration 3] The operation mode determination means determines whether the user can see the mobile terminal based on the position and orientation of the display device and the position and orientation of the mobile terminal. The information processing apparatus according to configuration 2, characterized in that... [Structure 4] The display device has an imaging means, The operation mode determination means determines whether the user can see the mobile terminal based on whether the mobile terminal is included in the captured image captured by the imaging means. An information processing device according to configuration 2 or 3, characterized by the above. [Composition 5] The operation mode determination means determines whether the user can view the mobile terminal based on whether the captured image has a resolution that allows the user to view the mobile terminal when the captured image is displayed on the display device. The information processing apparatus according to configuration 4, characterized by the features described above. [Composition 6] The image generation means generates a composite reality image by superimposing a virtual object onto the captured image. The operation mode determination means determines whether the user can see the mobile terminal based on whether the mobile terminal can be seen in the mixed reality image without being obstructed by the virtual object. The information processing apparatus according to configuration 4 or 5, characterized by the above. [Composition 7] The operation mode determination means acquires the user's gaze direction, and if the gaze direction is not directed towards the mobile terminal, it determines the operation mode to the second operation mode. An information processing device according to any one of configurations 2 to 6, characterized by the above. [Structure 8] The operation mode determination means does not change the operation mode from the time the application operation in the determined operation mode is started until the application operation is finished. An information processing device according to any one of configurations 2 to 7, characterized by the above. [Composition 9] The operation mode determination means determines the operation mode based on the input from the user. An information processing device according to any one of configurations 2 to 8. [Configuration 10] The second operation mode is an operation mode in which the user operates the virtual UI by operating the mobile terminal. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above. [Composition 11] The system further includes a measurement means for measuring the position and orientation of at least a part of the user's body, The second operation mode is an operation mode in which the virtual UI is operated based on the position and orientation of at least a part of the user's body. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above. [Composition 12] At least a part of the user's body is the user's fingers. The information processing apparatus according to configuration 11, characterized by the features described above. [Composition 13] At least a part of the user's body is the user's line of sight. An information processing device according to configuration 11 or 12, characterized by the above. [Composition 14] The aforementioned virtual UI is a virtual object that includes a GUI. An information processing device according to any one of configurations 1 to 13, characterized by the above. [Composition 15] The virtual UI includes virtual objects for performing selection or decision operations. An information processing apparatus according to any one of configurations 1 to 14, characterized by the features described herein. [Composition 16] In the second operation mode, the virtual UI is operated based on the position and orientation of the mobile device. An information processing apparatus according to any one of configurations 1 to 15, characterized by the above. [Composition 17] The aforementioned mobile device is equipped with a touch panel capable of recognizing touch gestures, In the second operation mode, the virtual UI is operated based on the distance and direction of the touch gesture. An information processing device according to any one of configurations 1 to 16, characterized by the above. [Composition 18] In the first operation mode, a GUI for performing application operations is displayed on the display of the mobile terminal, and the application operations are performed based on operations on the GUI. An information processing device according to any one of configurations 1 to 17, characterized by the features described herein. [Composition 19] The first operation mode and the second operation mode are modes in which the GUI displayed for application operation is different from each other. An information processing apparatus according to any one of configurations 1 to 18, characterized by the above. [Configuration 20] The application operations that can be performed in the first operation mode and the second operation mode are different. An information processing apparatus according to any one of configurations 1 to 19, characterized by the above. [Composition 21] A display device that can be worn on the user's head and control means for controlling the display of a GUI on a portable terminal that the user can grasp and operate, An application operation means that accepts input operations using the GUI and performs application operations based on the input operations, Equipped with, The GUI displayed on at least one of the display device and the mobile device differs depending on whether the user can see the mobile device or not. An information processing device characterized by the following: [Composition 22] A display device that can be worn on the user's head and control means for controlling the display of a GUI on a portable terminal that the user can grasp and operate, An application operation means that accepts input operations using the GUI and performs application operations based on the input operations, Equipped with, The control means controls the display of the GUI, which includes a virtual UI for application operation, on the display device and the display of the GUI for application operation on the mobile terminal to occur simultaneously. The application operation means performs the application operation based on at least one of the operations on the GUI including the virtual UI and the operations on the GUI displayed on the mobile terminal. An information processing device characterized by the following: [Composition 23] The display device and, The aforementioned mobile terminal, An information processing device as described in any one of items 1 to 22, A system that includes this. [Composition 24] A method of controlling a computer, Image generation step of generating an image to be displayed on a display device that can be worn on the user's head, An operation mode determination step to determine the operation mode of the application, An application operation step that receives input operations from a mobile terminal that can be held and operated by the user, and performs application operations based on the operation mode and the input operations, Includes, In the operation mode determination step, at least one of the following is determined as the operation mode: a first operation mode in which the user operates the application by operating the mobile terminal, and a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application by operating the virtual UI. A control method characterized by the following: [Composition 25] A method of controlling a computer, A control step for controlling the display of a GUI in a display device that can be worn on the user's head and a portable terminal that the user can grasp and operate, A determination step of determining whether the user can see the mobile device, An application operation step that accepts input operations using the GUI and performs application operations based on the input operations, Includes, The GUI displayed on at least one of the display device and the mobile device differs depending on whether the user can see the mobile device or not. A control method characterized by the following: [Composition 26] A method of controlling a computer, A control step for controlling the display of a GUI in a display device that can be worn on the user's head and a portable terminal that the user can grasp and operate, An application operation step that accepts input operations using the GUI and performs application operations based on the input operations, Includes, In the control step, the display of the GUI, which includes a virtual UI for application operation, on the display device and the display of the GUI for application operation on the mobile terminal are controlled to occur simultaneously. In the application operation step, the application operation is performed based on at least one of the operations on the GUI including the virtual UI and the operations on the GUI displayed on the mobile device. A control method characterized by the following: [Composition 27] A computer program characterized by causing a computer to perform each step of the control method described in any one of the configurations 24 to 26. [Explanation of Symbols]

[0131] 100: Information processing device 101: HMD 102: Mobile terminal 107: Operation mode determination unit 110: Application operation unit 112: Virtual image generation unit

Claims

1. Image generation means for generating images to be displayed on a display device that can be worn on the user's head, An operation mode determination means for determining the operation mode of an application, An application operation means that receives input operations from a portable terminal that can be held and operated by the user, and performs application operations based on the operation mode and the input operations, Equipped with, The display device has an imaging means, The image generation means generates a composite reality image by superimposing a virtual object onto the captured image captured by the imaging means. The operation mode determination means determines whether the user can see the mobile terminal based on whether the mobile terminal can be seen in the mixed reality image without being obstructed by the virtual object, The operation mode determination means acquires the user's gaze direction and determines a first operation mode in which the user operates the application by operating the mobile terminal if the user can see the mobile terminal and the gaze direction is directed toward the mobile terminal, and determines a second operation mode in which a virtual UI related to operation is displayed on the display device and the user operates the application by operating the virtual UI if the user cannot see the mobile terminal or the gaze direction is not directed toward the mobile terminal. An information processing device characterized by the following:

2. The second operation mode is an operation mode in which the user operates the virtual UI by operating the mobile terminal. The information processing apparatus according to feature 1.

3. The system further includes a measurement means for measuring the position and orientation of at least a part of the user's body, The second operation mode is an operation mode in which the virtual UI is operated based on the position and orientation of at least a part of the user's body. The information processing apparatus according to feature 1.

4. At least a part of the user's body is the user's fingers. The information processing apparatus according to claim 3.

5. At least a part of the user's body is the user's line of sight. The information processing apparatus according to claim 3.

6. The aforementioned virtual UI includes a ray which is a pointing object, The information processing apparatus according to feature 1.

7. In the second operation mode, the virtual UI is operated based on the position and orientation of the mobile terminal. The information processing apparatus according to feature 1.

8. In the first operation mode, a GUI for performing application operations is displayed on the display of the mobile terminal, and the application operations are performed based on operations on the GUI. The information processing apparatus according to feature 1.

9. In the first operation mode, when the user operates the mobile terminal to perform an application operation, the virtual UI is not displayed on the display device. The information processing apparatus according to feature 1.

10. The display device and, The aforementioned mobile terminal, An information processing device according to any one of claims 1 to 9, A system that includes this.

11. A method of controlling a computer, Image generation step of generating an image to be displayed on a display device that can be worn on the user's head, An operation mode determination step to determine the operation mode of the application, An application operation step that receives input operations from a mobile terminal that can be held and operated by the user, and performs application operations based on the operation mode and the input operations, Includes, The display device has an imaging means, The image generation step generates a composite reality image by superimposing a virtual object onto the captured image captured by the imaging means. In the operation mode determination step, it is determined whether the user can see the mobile device based on whether the mobile device can be seen in the mixed reality image without being obstructed by the virtual object, In the operation mode determination step, the user's gaze direction is obtained, and if the user can see the mobile terminal and their gaze direction is directed toward the mobile terminal, a first operation mode is determined in which the user operates the mobile terminal to perform application operations; if the user cannot see the mobile terminal or their gaze direction is not directed toward the mobile terminal, a second operation mode is determined in which a virtual UI related to operation is displayed on the display device and the user operates the virtual UI to perform application operations. A control method characterized by the following:

12. A computer program characterized by causing a computer to perform each step of the control method described in claim 11.

Images