Information processing device, system, control method, and computer program

Abstract

To perform suitable operation using a portable terminal in an MR experience.SOLUTION: An information processing device includes image generation means for generating an image to be displayed on a display device attachable to the head of a user, operation mode determination means for determining an operation mode of an application, and application operation means for receiving input operation from a portable terminal held by the user to be operable to perform application operation on the basis of the operation mode and the input operation. The operation mode determination means determines at least either a first operation mode for performing the application mode by allowing the user to operate the portable terminal or a second operation mode for performing the application operation by displaying a virtual UI about the operation in the display device to allow the user to operate the virtual UI.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to application operation using a mobile terminal in an XR experience. [Background technology]

[0002] In recent years, the virtual world and the real world have been integrated in real time to create new experiences, known as XR (Extensible Reality). XR is a technology that allows users to operate a device while viewing images using a head mounted display (HMD).

[0003] In general, a user uses a controller to operate the device. As the controller, a system is provided that uses a mobile device such as a smartphone or tablet that is familiar to users.

[0004] In VR (Virtual Reality), which allows users to experience a virtual world, the real world is not displayed, so users cannot see their real mobile device. In MR (Mixed Reality), which allows users to experience a mixed reality world where virtual world information is superimposed on the real world, the real mobile device may be superimposed on a virtual object and cannot be seen.

[0005] For example, Patent Document 1 discloses a method in which a mobile terminal representing a virtual object is prepared in a virtual space and information about operations to be performed on a real mobile terminal is displayed on the mobile terminal representing the virtual object, thereby allowing a user to visually recognize operations on the mobile terminal. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-36974 A [Patent Document 2] JP 2019-32713 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in MR, in which an image of the virtual world is superimposed on an image of the real world, the real mobile terminal may be superimposed on a virtual object and not visible, or may be visible without being superimposed. In an environment in which the real mobile terminal can be seen, if a virtual mobile terminal is superimposed on the real mobile terminal and displayed, intuitive operation may be difficult due to superimposition errors. In addition, if the virtual mobile terminal is displayed without being superimposed on the real mobile terminal, the virtual mobile terminal and the real mobile terminal may be displayed separately, which may hinder the sense of immersion. In other words, displaying a virtual mobile terminal may not always be preferable.

[0008] Therefore, this disclosure proposes a technique for performing suitable operations using a mobile terminal in a MR experience. [Means for solving the problem]

[0009] In order to achieve the above object, the first aspect of the present invention is an image generating means for generating an image to be displayed on a display device that can be worn on a user's head; an operation mode determination means for determining an operation mode of an application; an application operation means for receiving an input operation from a portable terminal that can be held and operated by a user, and for performing an application operation based on the operation mode and the input operation; Equipped with the operation mode determination means determines, as the operation mode, at least one of a first operation mode in which the user operates the mobile terminal to perform an application operation, and a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application by operating the virtual UI; The present invention is an information processing device characterized by the above.

[0010] A second aspect of the present invention is A control means for controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; an application operation means for receiving an input operation using the GUI and performing an application operation based on the input operation; Equipped with the GUI displayed on at least one of the display device and the mobile terminal is different depending on whether the user can see the mobile terminal or not; The present invention is an information processing device characterized by the above.

[0011] A third aspect of the present invention is A control means for controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; an application operation means for receiving an input operation using the GUI and performing an application operation based on the input operation; Equipped with the control means controls the display of a GUI including a virtual UI related to application operations on the display device and the display of a GUI related to application operations on the mobile terminal so as to be performed simultaneously; the application operation means performs the application operation based on at least one of an operation on a GUI including the virtual UI and an operation on a GUI displayed on the mobile terminal; The present invention is an information processing device characterized by the above.

[0012] Another fourth aspect of the present invention is a method for producing a composition comprising the steps of: A method for controlling a computer, comprising: an image generating step of generating an image to be displayed on a display device that can be worn on a user's head; an operation mode determination step for determining an operation mode of the application; an application operation step of receiving an input operation from a portable terminal that can be held and operated by a user, and performing an application operation based on the operation mode and the input operation; Including, In the operation mode determination step, at least one of a first operation mode in which the user operates the mobile terminal to perform an application operation and a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application operation by operating the virtual UI is determined as the operation mode. The control method is characterized by the above.

[0013] A fifth aspect of the present invention is A method for controlling a computer, comprising: A control step of controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; a determining step of determining whether the user can see the mobile terminal; An input operation using the GUI is received, and an application is operated based on the input operation. an application operation step for performing an operation; Including, a GUI displayed on at least one of the display device and the mobile terminal differs depending on whether the user can see the mobile terminal or not; The control method is characterized by the above.

[0014] A sixth aspect of the present invention is A method for controlling a computer, comprising: A control step of controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; an application operation step of accepting an input operation using the GUI and performing an application operation based on the input operation; Including, In the control step, a GUI including a virtual UI for application operations is displayed on the display device, and a GUI for application operations on the mobile terminal is displayed simultaneously. In the application operation step, the application operation is performed based on at least one of an operation on a GUI including the virtual UI and an operation on a GUI displayed on the mobile terminal. The control method is characterized by the above. Effect of the Invention

[0015] According to the present invention, it is possible to provide a suitable operation using a mobile terminal in an MR experience. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing an example of a functional configuration of a system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3] FIG. 4 is a diagram showing a specific example of the first operation mode according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a specific example of the second operation mode according to the first embodiment. [Diagram 5] 4 is a flowchart of an application operation according to the first embodiment. [Figure 6] 4 is a flowchart of an application operation in a first operation mode according to the first embodiment. [Figure 7] 5 is a flowchart of an application operation in a second operation mode according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing a specific example of a first operation mode according to the second embodiment. [Figure 9] 10 is a flowchart of an application operation in a first operation mode according to a second embodiment. [Figure 10] FIG. 11 is a diagram showing a specific example of the second operation mode according to the second embodiment. [Figure 11] 10 is a flowchart of an application operation in a second operation mode according to a second embodiment. [Figure 12]FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0018] (First embodiment) In the first embodiment, an example will be shown in which an operation mode is determined depending on whether or not a mobile terminal can be viewed through an HMD, and an application operation is performed in which text is added to a virtual object based on the operation mode.

[0019] <Configuration> 1 is a block diagram showing an example of the configuration of a system according to this embodiment. The system according to this embodiment has a configuration in which an information processing device 100 is connected to an HMD 101 and a mobile terminal 102. The connection between the devices may be either wired or wireless. Furthermore, the connection between the devices may be realized by combining wired and wireless connections.

[0020] [Information processing device] First, the information processing device 100 will be described. The information processing device 100 receives an input operation from a mobile terminal 102, and is capable of operating an application based on a set operation mode and the input operation. Fig. 2 is a hardware configuration diagram of the information processing device 100 in this embodiment. As shown in Fig. 2, the information processing device 100 has a configuration similar to that of a well-known general-purpose computer, and includes a CPU 201, a storage device 202, a ROM 203, a RAM 204, an input I / F 205, and an output I / F 206.

[0021] The CPU 201 comprehensively controls each device connected via the bus 200. The CPU 201 reads out and executes processing steps and computer programs stored in the ROM 203 and the RAM 204.

[0022] The storage device 202 stores programs and data for use by the information processing device 100, including various programs according to this embodiment.

[0023] The ROM 203 stores an operating system (OS), device drivers, and a boot program.

[0024] The RAM 204 temporarily stores the programs and data loaded from the storage device 202 and the ROM 203, and has a work area where the CPU 201 appropriately executes each process.

[0025] The input I / F 205 receives signals from external devices including the mobile terminal 102 as input signals in a format that can be processed by the information processing device 100 .

[0026] The output I / F 206 outputs a signal to an external device including the HMD 101 as an output signal in a format that can be processed by the external device.

[0027] The information processing device 100 realizes the functions shown in FIG. 1 by the CPU 201 reading and executing the programs.

[0028] The image acquisition unit 103 acquires an image of the real world (real image) captured by the imaging unit 114 of the HMD 101. The image acquisition unit 103 outputs the acquired image to the HMD position and orientation measurement unit 104, the mobile terminal position and orientation measurement unit 105, and the mixed reality image generation unit 113.

[0029] The HMD position and orientation measurement unit 104 performs image processing using the real image (captured image) acquired by the image acquisition unit 103, and measures the position and orientation of the HMD 101 by extracting characteristic information such as points and lines in the image. The position and orientation of the HMD 101 can be considered equivalent to the position and orientation of the user's line of sight or head. The position and orientation information measured by the HMD position and orientation measurement unit 104 is input to the determination unit 106, the virtual UI generation unit 108, and the virtual image generation unit 112.

[0030] In this embodiment, position and orientation measurement is performed by image processing, but the present invention is not limited to this. The position and orientation of the HMD may be measured using infrared light, or may be measured using at least one of an ultrasonic sensor, a magnetic sensor, and a depth sensor.

[0031] The mobile terminal position and orientation measurement unit 105 performs image processing using the real image acquired by the image acquisition unit 103, and measures the position and orientation of the mobile terminal 102. The position and orientation of the mobile terminal 102 can be measured by displaying an index (marker) dedicated to the mobile terminal on the terminal display unit 118 and estimating the position and orientation of the index displayed on the real image. In this case, the index is not particularly limited as long as it has a shape that can be uniquely identified.

[0032] In this embodiment, the position and orientation of the mobile terminal 102 is measured using an index, but the present invention is not limited to this. The position and orientation of the mobile terminal 102 may be measured using infrared light, or may be measured using at least one of an ultrasonic sensor, a magnetic sensor, a depth sensor, and an angular velocity sensor. The position and orientation of the mobile terminal 102 may also be measured using a global positioning system (GPS).

[0033] The determination unit 106 determines whether or not the user can view the mobile terminal 102 through the HMD 101. The determination unit 106 makes the above determination based on, for example, the position and orientation of the HMD 101 determined by the HMD position and orientation measurement unit 104 and the position and orientation of the mobile terminal 102 determined by the mobile terminal position and orientation measurement unit 105, and further taking into consideration the angle of view of the HMD 101 and the display size of the mobile terminal 102. More specifically, the determination unit 106 determines whether or not the user can view the mobile terminal 102 through the HMD 101 depending on whether the display of the mobile terminal is included in the angle of view of the HMD. The determination unit 106 inputs the determination result to the operation mode determination unit 107.

[0034] In this embodiment, it is determined whether or not the user can view the mobile terminal 102 through the HMD 101 based on the positions and orientations of the HMD 101 and the mobile terminal 102, but the present invention is not limited thereto. The determination unit 106 may make the above determination by determining whether or not the mobile terminal is included in the real image (captured image) acquired by the image acquisition unit 103 through image recognition. In this case, the determination unit 106 may determine whether or not the HMD 101 and the mobile terminal 102 are facing each other in order to determine whether or not the display of the mobile terminal 102 is displayed in the real image of the HMD 101. The determination unit 106 preferably adds to the determination result whether or not the display of the mobile terminal 102 displayed in the real image has a resolution that allows the user to view the display of the mobile terminal 102 displayed in the real image on the HMD 101 when the real image is displayed on the HMD 101. More preferably, the determination unit 106 adds to the determination result whether or not the display of the mobile terminal 102 has a resolution that allows the user to view the information displayed on the display on the HMD 101 as well as the display of the mobile terminal 102. Furthermore, the determination unit 106 may determine whether the mobile terminal 102 is visible in the mixed reality image displayed on the HMD 101. That is, the determination unit 106 may determine whether or not a virtual object is drawn superimposed on the area of ​​the mobile terminal 102 in the mixed reality image in which a virtual space image is superimposed on the real image. The fact that a virtual object is not drawn superimposed on the area of ​​the mobile terminal 102 in the real image means that the mobile terminal 102 can be viewed by the user in the mixed reality image without being blocked by the virtual object. The determination unit 106 determines that the user can view the mobile terminal 102 through the HMD 101 when the display of the mobile terminal 102 and information displayed on the display have a resolution that allows the user to view them. Furthermore, the determination unit 106 determines that the user can view the mobile terminal 102 through the HMD 101 when a virtual object is not superimposed on the area of ​​the mobile terminal 102 in the real image and the mobile terminal 102 can be viewed in the mixed reality image without being blocked by the virtual object. However, the determination unit 106 may determine that the user can view the mobile terminal 102 through the HMD 101 when some of all of the above-mentioned conditions are satisfied.

[0035] The operation mode determination unit 107 determines the operation mode of the application based on the determination result of the determination unit 106. That is, when it is determined that the user can view the mobile terminal 102 through the HMD 101, the operation mode determination unit 107 determines the operation mode to be a first operation mode in which the user operates the application by directly operating the mobile terminal 102. When it is determined that the user cannot view the mobile terminal 102, the operation mode determination unit 107 determines the operation mode to be a second operation mode in which a virtual object (virtual UI) of an operation UI is displayed in a virtual space and the user operates the virtual UI to operate the application. The first operation mode and the second operation mode can also be expressed as modes in which GUIs displayed for application operation are different from each other. Here, different GUIs means that at least one of the GUI displayed on the display unit of the mobile terminal 102 and the GUI including the virtual UI displayed on the HMD 101 is at least partially different. Even when it is determined that the user can view the mobile terminal 102 through the HMD 101, the operation mode determination unit 107 may determine the operation mode to be the second operation mode if the user's line of sight is not directed toward the mobile terminal 102. The operation mode determination unit 107 inputs the determined operation mode to the virtual UI generation unit 108 and the operation mode receiving unit 116 of the mobile terminal 102. The operation mode determination unit 107, which inputs the determined operation mode to the virtual UI generation unit 108 and the operation mode receiving unit 116, corresponds to a control unit that controls the display of the GUI in the HMD 101 and the mobile terminal 102.

[0036] In this embodiment, an example of the first operation mode when performing an operation of adding text to a virtual object is shown in FIGS. 3(A) to 3(C), and an example of the second operation mode is shown in FIGS. 4(A) to 4(C).

[0037] First, the first operation mode will be described with reference to Fig. 3(A) to Fig. 3(C). Fig. 3(A) is a diagram showing an MR space in the first operation mode. In the MR space in which a virtual object 300 exists, a user 301 experiences using an HMD 302 worn on the head and a mobile terminal 303 held as a controller. Fig. 3(B) is a mixed reality image displayed on the HMD 302 in Fig. 3(A). The mixed reality image is an image in which the virtual object 300 is superimposed on a real image including the mobile terminal 303. As shown in Fig. 3(C), an index 304 for the information processing device 100 to measure the position and orientation of the mobile terminal 303 and a keyboard 305 as a GUI for a user to perform a text addition operation are displayed on the display unit of the mobile terminal 303. The user adds text to the virtual object 300 by operating the keyboard 305.

[0038] Next, the second operation mode will be described with reference to FIG. 4(A) to FIG. 4(B). The same reference numbers are used for the same components as those in FIG. 3(A) to FIG. 3(C), and the description thereof will be omitted. FIG. 4(A) is a diagram showing the MR space in the second operation mode. Unlike FIG. 3(A), FIG. 4(A) shows a state in which the user 301 does not visually recognize the mobile terminal 303. In the second operation mode, a virtual keyboard 400 as a virtual UI and a pointer 401 for key selection are arranged so that the user 301 can add text without using the mobile terminal 303. FIG. 4(B) shows a mixed reality image displayed on the HMD 302 in FIG. 4(A). The mixed reality image is an image in which the virtual object 300, the virtual keyboard 400, and the pointer 401 are superimposed on a real image. The user moves the pointer 401 by operating the mobile terminal 303, and inputs the key focused on by the pointer 401 by a decision operation, thereby adding text to the virtual object 300. The virtual UI may be operated by a user in a manner other than the operation of the mobile terminal 303. For example, the virtual UI may be operated based on the position and orientation of at least a part of the user's body, such as the fingers, head, or line of sight, or the position and orientation of the mobile terminal 102. The position and orientation of the user's body can be measured by performing image recognition processing on a real image captured by the HMD 101.

[0039] The virtual UI generating unit 108 determines whether the operation mode input from the operation mode determining unit 107 is the second operation mode. When the operation mode is the first operation mode, the virtual UI generating unit 108 generates a virtual UI, and when the operation mode is the second operation mode, the virtual UI generating unit 108 does not generate a virtual UI. The virtual UI includes graphics objects (virtual objects) configured with GUIs such as buttons, such as the virtual keyboard 400 in Fig. 4(B). The virtual UI further includes a pointing object, such as a pointer 401, which is a virtual object for performing a selection operation or a decision operation.

[0040] Although the present embodiment employs a selection and decision operation using the pointer 401, the present invention is not limited to this as long as the selection and decision operation of a virtual object can be performed. The selection and decision operation of a virtual object may be performed, for example, by casting a ray displayed based on the position and orientation of a mobile terminal. A graphics object may be placed in a position and orientation displayed on the HMD 101 according to a position and orientation input from the HMD position and orientation measurement unit 104. For example, as in Patent Document 2, a technology is conceivable in which the view frustum of the HMD 101 is calculated and the graphics object is placed in an overlapping area of ​​the view frustums of the left eye and the right eye so as to be perpendicular to the line of sight. Alternatively, the graphics object may be placed in a position and orientation that faces the virtual object to be operated.

[0041] The position and orientation of the pointing object are determined in response to a selection operation input from an operation receiving unit 109. Details will be described later with reference to a flowchart. The virtual UI generating unit 108 inputs the generated virtual UI to a virtual image generating unit 112.

[0042] The operation receiving unit 109 receives an operation signal of the mobile terminal 102 from the operation unit 119 of the mobile terminal 102. The operation signal will be described later in relation to the operation unit 119. The operation receiving unit 109 interprets these operation signals and converts them into operation events such as a selection operation or a decision operation. When a decision operation is input, the operation receiving unit 109 notifies the application operation unit 110. In addition, in the case of the second operation mode, the operation receiving unit 109 inputs an operation event to the virtual UI generation unit 108.

[0043] The application operation unit 110 receives an input operation from the mobile terminal 102 via the operation receiving unit 109, and performs application operations based on the currently set operation mode and the received input operation. In a first operation mode, the application operation unit 110 performs application operations based on input operations using a GUI displayed on the mobile terminal 102. In a second operation mode, the application operation unit 110 performs application operations based on input operations using a GUI including a virtual UI displayed on the HMD 101.

[0044] The virtual data storage unit 111 stores data on the virtual space, such as data on virtual objects that constitute the virtual space (shape information and position and orientation information) and data on a light source that illuminates the virtual space, and inputs the data to the virtual image generation unit 112.

[0045] The virtual image generation unit 112 configures a virtual space from the virtual UI generated by the virtual UI generation unit 108 and the virtual objects acquired from the virtual data storage unit 111. The virtual image generation unit 112 further generates an image (virtual image) of the virtual space seen from the position and orientation measured by the HMD position and orientation measurement unit 104. The virtual image generation unit 112 inputs the generated virtual image to a mixed reality image generation unit 113. Note that the technology for generating an image of the virtual space seen from a predetermined position is a known technology, and therefore a detailed description thereof will be omitted.

[0046] The mixed reality image generation unit 113 generates an image to be displayed on the HMD 101. More specifically, the mixed reality image generation unit 113 generates an image of the mixed reality space (mixed reality image) by superimposing the virtual image generated by the virtual image generation unit 112 on the real image acquired by the image acquisition unit 103. The mixed reality image generation unit 113 outputs the generated mixed reality image to the display unit 115 of the HMD 101.

[0047] The mixed reality image generation unit 113 corresponds to an image generation means that generates an image to be displayed on the HMD 101. Note that the image generation means may be a means that instructs another device or application module to generate an image to be displayed on the HMD 101. For example, when the virtual image generation unit 112 and the mixed reality image generation unit 113 are realized by a device or module other than the information processing device 100, the information processing device 100 may instruct the virtual image generation unit 112 and the mixed reality image generation unit 113 to generate a mixed reality image. In this case, the information processing device that issues the instruction can be interpreted as corresponding to the image generation means.

[0048] [HMD] Next, the HMD 101 will be described.

[0049] The HMD 101 is a display device that has a display unit configured with a liquid crystal image or the like and can be worn on the head of a user. The HMD 101 has a display unit for the right eye and a display unit for the left eye, which are attached so as to be located in front of the right eye and the left eye of the user when the user wears the HMD 101 on the head. In addition, images with parallax are displayed in stereo on the left and right display units (screens). In this embodiment, the HMD 101 is described as a video see-through type HMD that displays a mixed reality image generated based on a real image captured by an imaging device on a display device. However, the present invention is not limited to this. The HMD 101 may be an optical see-through type HMD that displays a virtual space image by superimposing it on a display medium that can be observed through the real space.

[0050] The imaging unit 114 captures an image of the real space displayed on the HMD 101 , and inputs the captured real image to the information processing device 100 .

[0051] The display unit 115 displays the mixed reality image generated by the information processing device 100 .

[0052] [Mobile device] The mobile terminal 102 will be described. The mobile terminal 102 is used as a controller that can be held and operated by a user. The mobile terminal is a general-purpose terminal, and refers to a smartphone or a tablet terminal. FIG. 12 is a hardware configuration diagram of the mobile terminal 102 in this embodiment.

[0053] The CPU 1201 comprehensively controls each device connected via the bus 1200. The CPU 1201 reads out and executes processing steps and programs stored in the ROM 1203 and the RAM 1204.

[0054] The storage device 1202 stores programs and data for use by the mobile terminal 102, including various programs according to this embodiment.

[0055] The ROM 1203 stores an operating system (OS), device drivers, and a boot program.

[0056] The RAM 1204 temporarily stores the programs and data loaded from the storage device 1202 and the ROM 1203, and has a work area where the CPU 1201 appropriately executes each process.

[0057] The input device 1205 is a device for providing data, information, instructions, etc., and includes a built-in IMU (Internal Measurement Unit), touch panel, and microphone. Input obtained from these devices is input as an input signal in a processable format. Be encouraged.

[0058] The output device 1206 is a device for receiving an output signal and physically presenting it, such as a display, a speaker, or a vibrator.

[0059] The following describes the functions of the mobile terminal 102 according to this embodiment.

[0060] The operation mode receiving unit 116 receives the operation mode from the operation mode determination unit 107 of the information processing device 100, and inputs it to the operation unit 119. In addition, in the case of the first operation mode, the operation mode receiving unit 116 also inputs the operation mode to the operation UI generating unit 117.

[0061] The operation UI generating unit 117 generates a GUI for operating an application when the operation mode received by the operation mode receiving unit 116 is the first operation mode. The operation UI generating unit 117 inputs the generated GUI to the terminal display unit 118.

[0062] The terminal display unit 118 displays the UI generated by the operation UI generation unit 117. The terminal display unit 118 is provided on the front side of the mobile terminal 102, and is configured with a display capable of displaying liquid crystal images and the like.

[0063] The operation unit 119 inputs an operation signal corresponding to an input operation of the mobile terminal to the operation UI generation unit 117 or the operation reception unit 109 of the information processing device 100. The input operation includes touch gestures such as tapping and swiping on a touch panel, and back taps detected based on angular acceleration measured by a built-in IMU. In the case of a touch gesture, an operation signal is generated based on at least one of the position of the touch, the distance of movement, and the direction of movement. Furthermore, the position and orientation (position, orientation, and changes therein) of the mobile terminal 102 acquired by the IMU in response to an input operation of moving the mobile terminal 102 may be used as the operation signal.

[0064] First, the behavior of the operation unit 119 when the operation mode is the first operation mode will be described.

[0065] In the case where the input operation is an operation for an application performed by a decision operation, the operation unit 119 inputs an operation signal based on the focused UI to the operation receiving unit 109. In the case where the input operation is a movement or switching operation for the UI, the operation unit 119 may send an operation signal to the operation UI generating unit 117 to update the operation UI. The movement or switching operation may be a swipe operation or a drag operation on the touch panel of the mobile terminal 102, and may further be a change in the attitude of the mobile terminal 102. The decision operation may be a tap operation on the touch panel of the mobile terminal.

[0066] Next, the behavior of the operation unit 119 when the operation mode is the second operation mode will be described.

[0067] The operation unit 119 inputs an operation signal to the operation receiving unit 109 regardless of the type of operation, such as a movement or switching operation or a decision operation.

[0068] <Processing> 5 is an example of a flowchart showing a process for determining an operation mode and operating an application. In the following, each step will be explained with an S added to the beginning of each reference number.

[0069] First, in step S500, the HMD position and orientation measurement unit 104 measures the position and orientation of the HMD 101 based on the captured image acquired by the image acquisition unit 103. Then, the process proceeds to step S501.

[0070] In step S501, the mobile terminal position and orientation measurement unit 105 measures the position and orientation of the mobile terminal 102 based on the captured image acquired by the image acquisition unit 103. Then, the process proceeds to step S502.

[0071] In step S502, the determination unit 106 determines whether or not the mobile terminal 102 is visible through the HMD 101. If it is visible, the process proceeds to step S503, and if it is not visible, the process proceeds to step S504.

[0072] In step S503, the application is operated in the first operation mode (details of which will be explained later in the flowchart), and the process proceeds to step S505.

[0073] In step S504, the application is operated in the second operation mode (details of which will be explained later in the flowchart), and the process proceeds to step S505.

[0074] In step S505, if the user inputs an instruction to end the application or the conditions for ending the application are met, this process ends. On the other hand, if the user has not input an instruction to end the application and the conditions for ending the application are not met, the process returns to step S500.

[0075] [First operation mode] FIG. 6 shows a flow chart of the application operation according to the first operation mode shown in step S503.

[0076] In step S600, operation mode determination section 107 transmits information indicating that the operation mode is the first operation mode to operation mode receiving section 116. Then, the process proceeds to step S601.

[0077] In step S601, the operation mode receiving unit 116 notifies the operation UI generating unit 117 that it is the first operation mode. Upon receiving the notification, the operation UI generating unit 117 generates a keyboard 305 as shown in Fig. 3(C) and displays it on the terminal display unit 118 of the mobile terminal 102. Then, the process proceeds to step S602.

[0078] In step S602, the operation unit 119 receives a signal indicating whether or not the user has performed a tap operation on the keyboard 305 displayed on the mobile terminal 102. If a tap operation has been received, the process proceeds to step S603. If a tap operation has not been received, the process proceeds to step S505.

[0079] In step S603, the operation unit 119 transmits a key input signal corresponding to the key on which the tap operation has been performed to the operation receiving unit 109. Then, the process proceeds to step S604.

[0080] In step S604, the operation receiving unit 109 notifies the application operation unit 110 that a corresponding key has been input based on the received key input signal. The application operation unit 110 generates text for the corresponding key as a virtual object and places it in the virtual space. As a result, the virtual image generation unit 112 generates a virtual image including the text, and the mixed reality image generation unit 113 superimposes the generated virtual image on the real image, thereby adding the text. Then, the process proceeds to step S505.

[0081] In this embodiment, the operation UI is displayed on the display unit of the mobile terminal 102 upon receiving a notification that the operation mode is the first operation mode, but the present invention is not limited to this. For example, the operation UI is always displayed on the display unit of the mobile terminal 102 regardless of the operation mode, and input is only possible when the mobile terminal 102 is in the first operation mode. It may be acceptable to do so.

[0082] This allows application operations to be performed by inputting information into the mobile terminal 102 .

[0083] [Second operation mode] FIG. 7 shows a flow chart of the application operation according to the second operation mode shown in step S504.

[0084] In step S700, the operation mode determination unit 107 transmits information indicating that the operation mode is the second operation mode to the virtual UI generation unit 108 and the operation mode receiving unit 116. Then, the process proceeds to step S701.

[0085] In step S701, the operation mode receiving unit 116 notifies the virtual UI generating unit 108 that the operation mode is the second operation mode. Upon receiving the notification, the virtual UI generating unit 108 generates a virtual keyboard 400 and a pointer 401 as shown in Fig. 4 and places them in the virtual space. The virtual UI generating unit 108 inputs the result of this placement to the virtual image generating unit 112. Then, the process proceeds to step S702.

[0086] In step S702, the operation mode receiving unit 116 notifies the operation unit 119 that the operation mode is the second operation mode. The operation unit 119 determines whether or not the user has performed a swipe operation on the touch panel of the mobile terminal. If a swipe operation has been received, the process proceeds to S703. If a swipe operation has not been received, the process proceeds to S705.

[0087] In step S703, the operation unit 119 notifies the operation receiving unit 109 of the distance and direction that the finger has moved on the screen by the swipe operation as a movement operation signal. Then, the process proceeds to step S704. In this embodiment, a method of obtaining a movement operation signal by a swipe operation of the mobile terminal 102 will be described, but the present invention is not limited to this. The movement operation signal may be obtained by a flick operation of the mobile terminal 102, or may be acquired based on the amount of change in the attitude of the mobile terminal.

[0088] In step S704, the operation receiving unit 109 notifies the virtual UI generating unit 108 of the received movement distance and direction. The virtual UI generating unit 108 changes the position of the pointer 401 based on the acquired movement distance and direction. Then, the process proceeds to step S505. This makes it possible to change the key on the virtual keyboard 400 that is focused on.

[0089] In step S705, the operation unit 119 determines whether or not the user has performed a tap operation on the touch panel of the mobile terminal 102. If a tap operation has been received, the process proceeds to step S706. If a tap operation has not been received, the process proceeds to step S505.

[0090] In step S706, the operation unit 119 notifies the operation receiving unit 109 of the fact that a decision operation has been performed as an operation signal. Then, the process proceeds to step S707.

[0091] In step S707, the operation receiving unit 109 notifies the application operation unit 110 that a confirmation operation has been performed. The application operation unit 110 determines the focused key based on the position and orientation of the pointer 401 with respect to the virtual keyboard 400 generated by the virtual UI generation unit 108, and sets it as the input key. Then, the process proceeds to step S708.

[0092] In step S708, the application operation unit 110 generates a virtual object representing the text of the key corresponding to the input key, and places the virtual object in the virtual space. In step S505, a virtual image including text is generated, and the text can be added by superimposing the virtual image on the real image in the mixed reality image generating unit 113. Then, the process proceeds to step S505.

[0093] This allows input operations on the mobile device to be reflected in the virtual UI, making it possible to operate applications.

[0094] <Advantageous Effects of the Present Embodiment> According to this embodiment, when the mobile device is visible, application operations can be performed based on the UI displayed on the display of the mobile device, and when the mobile device is not visible, application operations can be performed based on the UI displayed in the virtual space. This allows for suitable application operations in a MR experience.

[0095] (Variation 1) In the first embodiment, an example has been shown in which it is determined whether the mobile terminal 102 is visible on the HMD 101 each time the positions and orientations of the HMD 101 and the mobile terminal 102 are measured, an operation mode is determined based on the determination result, and an application operation is performed, but the present invention is not limited to this. The operation mode does not need to be changed regardless of the positions and orientations of the HMD 101 and the mobile terminal 102 from when an operation based on the determined operation mode is started until the same operation is completed or canceled. In other words, the operation start is determined to be when a key is input in the first operation mode or the second operation mode, and the operation mode does not need to be changed until the generation of text created by multiple key inputs is confirmed or the generation is canceled.

[0096] (Variation 2) In the first embodiment, an example is shown in which it is determined whether the mobile terminal 102 is visible through the HMD 101, and an operation mode is determined based on the determination result to perform application operation, but the present invention is not limited to this. The operation mode may be determined based on a user's input. The user's input may be performed by operating the mobile terminal. The user's input may also be performed by a UI and a mouse, or by a button-type device such as a keyboard. The user's input may also be performed by recognizing the user's gestures or by recognizing the user's voice. When using gestures and voice for input, the mobile terminal 102 needs to be configured to be able to recognize touch gestures and voice, respectively.

[0097] (Variation 3) In the first embodiment, an example in which the operation mode is uniquely determined by the operation mode determination unit 107 has been described, but the present invention is not limited to this. An operation mode using both the first operation mode and the second operation mode may be adopted. In this modified example, for example, a virtual UI is displayed regardless of whether the mobile terminal 102 is visible or not, and application operation by the virtual UI is always possible. In this modified example, further, when the mobile terminal 102 is visible, application operation by operating the mobile terminal 102 is possible. In other words, in this modified example, when the mobile terminal 102 is visible, both the first operation mode and the second operation mode are valid, and when the mobile terminal 102 is not visible, the second operation mode is valid.

[0098] The compatibility of the first operation mode and the second operation mode can be realized by the information processing device 100 performing the following process.

[0099] The information processing device 100 instructs the virtual UI generating unit 108 to generate a GUI including a virtual UI and display it on the HMD 101, and also notifies the operation mode receiving unit 116 of the mobile terminal 102 of the operation mode and instructs the mobile terminal 102 to display the GUI. The processing device 100 controls so that the GUI including the virtual UI is displayed simultaneously on the HMD 101 and the mobile terminal 102. Note that "displayed simultaneously" does not necessarily mean that the display starts at the same time, but means that the GUI is displayed on both the HMD 101 and the mobile terminal 102 for at least a certain period of time.

[0100] The information processing device 100 first measures the position and orientation of the user's fingers by a measurement unit (not shown) and inputs the measurement result to the operation receiving unit 109. The operation receiving unit 109 determines whether the fingers and the GUI cross from the position and orientation of the user's fingers measured by the measurement unit and the position and orientation of the GUI included in the generated virtual UI. If the fingers and the GUI cross, the operation receiving unit 109 inputs an operation signal to the application operation unit 110, assuming that a decision operation has been input. On the other hand, if the fingers and the GUI do not cross, the operation receiving unit 109 determines that no input operation has been performed by the user, and does not input an operation signal to the application operation unit 110. The operation receiving unit 109 also notifies the application operation unit 110 of the input signal sent from the operation unit 119 of the mobile terminal 102.

[0101] This makes it possible to operate an application by operating the virtual UI with the user's fingers, and to operate the application by operating the mobile terminal. That is, the information processing device 100 can operate an application based on at least one of an operation on a GUI including the virtual UI and an operation on a GUI of the mobile terminal 102.

[0102] In the present embodiment, an example in which the user operates the virtual UI using his / her fingers as the second operation mode has been described, but the present invention is not limited to this. In the second operation mode, the user may operate the virtual UI using his / her line of sight or voice.

[0103] In this modification, the operation mode is switched depending on whether the user can see the mobile terminal 102, but the first operation mode and the second operation mode may always be enabled regardless of whether the user can see the mobile terminal 102. Alternatively, the user may be allowed to set whether the first operation mode and the second operation mode are compatible, and the operation mode determination unit 107 may determine the operation mode based on this setting. That is, when compatibility of the operation modes is set, the operation mode determination unit 107 determines both the first operation mode and the second operation mode as the operation modes, and when compatibility is not set, the operation mode determination unit 107 determines the operation mode depending on whether the user can see the mobile terminal 102.

[0104] (Variation 4) The functional configurations of the information processing device 100, the HMD 101, and the mobile terminal 102 described in the first embodiment are merely examples. Some functions may be executed by a device different from the devices described above, or may be shared and executed by a plurality of devices. For example, the information processing device 100 is not limited to one device, and may be configured by a plurality of information processing devices. Also, for example, a function described as a function of the information processing device 100, such as the operation mode determination unit 107, may be executed by the HMD 101 or the mobile terminal 102.

[0105] Second embodiment In the first embodiment, the case where the same application operation, that is, a text adding operation using a keyboard, is performed regardless of the operation mode has been described. In the present embodiment, however, a case where the application operation that can be executed differs depending on the operation mode will be described.

[0106] When dealing with 3DCG, as seen in DCC (Digital Content Creation) tools, it is common to have two UIs: First, a 2D UI such as a tree view or property view allows for precise or detailed operations. Next, there is a UI that displays a 3D space and allows intuitive manipulation of the space using a 3D UI such as a dragger. It is desirable to have a similar UI in XR that handles 3DCG.

[0107] The first operation mode, in which applications are operated by viewing a mobile device through an HMD, is suitable for precise and detailed operation using a 2D UI, while the second operation mode, in which applications are operated by displaying a 3D UI in an XR space, is suitable for spatial or intuitive operation.

[0108] In this embodiment, a case where an application operation is performed to change the arrangement of a virtual object will be described. Note that the configurations of the information processing device 100, HMD 101, and mobile terminal 102 in this embodiment (FIGS. 1, 2, and 12) are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0109] First, an example of changing the placement of a virtual object in the first operation mode will be described with reference to FIG. 8. The same reference numbers are used for the same components as in FIG. 3, and their description will be omitted. FIG. 8(A) is a diagram showing an MR space in the first operation mode. A virtual object, a desk 800, and a virtual object, a cabinet 801, are placed in the MR space. FIG. 8(B) is a mixed reality image seen on the HMD 302 in FIG. 8(A). The mixed reality image includes a mobile terminal 303. A two-dimensional UI for performing operations as shown in FIG. 8(C) is displayed on the display unit of the mobile terminal 303. This two-dimensional UI includes a tree UI 802 that displays a list of virtual objects placed in the MR space in a tree format, and a property UI 803 that displays detailed settings of the virtual object selected in the tree UI. The user can change the placement of the virtual object by selecting a virtual object to be moved from the tree UI 802 and inputting a position and orientation in the property UI 803 as a numerical value.

[0110] 9 is a flowchart showing the process of step S503 for performing an application operation for changing the arrangement of a virtual object in the first operation mode. The same processes as those in FIG. 6 are assigned the same reference numerals, and the description thereof will be omitted.

[0111] In step S900, the operation mode receiving unit 116 notifies the operation UI generating unit 117 that it is the first operation mode. Upon receiving the notification, the operation UI generating unit 117 generates a tree UI 802 and a property UI 803 as shown in Fig. 8(C) and displays them on the terminal display unit 118 of the mobile terminal 102. Then, the process proceeds to step S901.

[0112] In step S901, the operation unit 119 receives a signal indicating whether or not the user has performed a tap operation on the tree UI 802 displayed on the mobile terminal 102. If a tap operation has been received, the process proceeds to step S902. If a tap operation has not been received, the process proceeds to step S904.

[0113] In step S902, the operation unit 119 transmits information about the virtual object selected in the tree UI 802 to the operation UI generation unit 117 and the operation receiving unit 109. For example, the operation unit 119 transmits information that a desk 800 has been selected to the operation UI generation unit 117 and the operation receiving unit 109. The operation UI generation unit 117 displays detailed settings of the selected virtual object in the property UI 803. Then, the process proceeds to step S903.

[0114] In step S903, the operation receiving unit 109 notifies the application operation unit 110 of information on the selected virtual object. The application operation unit 110 updates the virtual space so that it is possible to know that the corresponding virtual object has been selected. The fact that the virtual object has been selected may be indicated by displaying a bounding box for the virtual object, or by changing the color of the virtual object. Then, the process proceeds to step S904.

[0115] In step S904, the operation unit 119 receives a signal indicating whether or not the user has performed a change operation on the property UI 803 displayed on the mobile terminal. In this embodiment, a change operation refers to an operation of changing a numerical value indicating the position of a virtual object. If a change operation has been received, the process proceeds to step S905. If a change operation has not been received, the process proceeds to step S505.

[0116] In step S905, the operation unit 119 transmits change information for the selected virtual object to the operation receiving unit 109. Then, the process proceeds to step S906.

[0117] In step S906, the operation receiving unit 109 notifies the application operation unit 110 of change information of the selected virtual object. The application operation unit 110 changes the arrangement of the corresponding virtual object in the virtual space based on the change information. As a result, the virtual image generation unit 112 generates a virtual image in a state in which the arrangement of the virtual object has been changed, and the mixed reality image generation unit 113 superimposes this virtual image on the real image, thereby realizing the change in the arrangement of the virtual object.

[0118] Next, an example of changing the placement of a virtual object in the second operation mode will be described with reference to Fig. 10. The same components as those in Fig. 8 are given the same reference numerals, and their description will be omitted. Fig. 10(A) is a diagram showing the MR space in the second operation mode. A ray 1000, which is a pointing object, is placed based on the position and orientation of the mobile terminal 303. Fig. 10(B) is a mixed reality image seen on the HMD 302 in Fig. 10(A). The placement of the virtual object can be changed by selecting and moving the virtual object pointed to by the ray 1000.

[0119] 11 is a flowchart showing the process of step S504 for performing an application operation to change the arrangement of a virtual object in the second operation mode. The same processes as those in FIG. 7 are assigned the same reference numerals, and the description thereof will be omitted.

[0120] In step S1100, the operation mode receiving unit 116 notifies the virtual UI generating unit 108 that the operation mode is the second operation mode. Upon receiving the notification, the virtual UI generating unit 108 generates a ray 1000 in the virtual space as shown in Fig. 10(A). The position and orientation of the ray 1000 is updated based on the position and orientation measured by the mobile terminal position and orientation measuring unit 105. Then, the process proceeds to step S1101.

[0121] In step S1101, the virtual UI generation unit 108 determines whether the ray 1000 intersects with a virtual object. If the ray intersects with the virtual object, the virtual space is updated so that it is possible to know that the corresponding virtual object has been selected. Then, the process proceeds to S1102. If the ray does not intersect with the virtual object, the process proceeds to S1101.

[0122] In step S1102, the operation unit 119 receives a signal indicating whether or not the user has performed a tap operation. If a tap operation has been received, the operation unit 119 transmits a selection operation to the operation receiving unit 109. Then, the process proceeds to step S1103. If a tap operation has not been received, the process proceeds to step S1101.

[0123] In step S1103, the operation receiving unit 109 notifies the selection operation to the application operation unit 110. The application operation unit 110 changes the settings so that the position and orientation of the selected virtual object are updated to follow the position and orientation of the ray, and then the process proceeds to step S1105.

[0124] In step S1104, the operation unit 119 receives a signal indicating whether or not the user has performed a tap operation. If a tap operation has been received, the operation unit 119 transmits a selection operation to the operation receiving unit 109. If a tap operation is not received, the process proceeds to step S1106.

[0125] In step S1105, the operation receiving unit 109 notifies the application operation unit 110 of the selection operation. The application operation unit 110 stops updating the position and orientation of the virtual object by following the position and orientation of the ray, and positions the virtual object so that the position and orientation at the time of receiving the selection operation are fixed. Then, the process proceeds to step S505. As a result, the virtual image generation unit 112 generates a virtual image in a state in which the position of the virtual object has been changed, and the mixed reality image generation unit 113 superimposes this virtual image on the real image, thereby realizing a change in the position of the virtual object.

[0126] In step S1106, the operation unit 119 receives whether or not the user has performed a flick operation. If a flick operation has been received, the operation unit 119 transmits a cancel operation to the operation receiving unit 109. Then, the process proceeds to step S1107. If a flick operation has not been received, the process proceeds to step S1103. In this embodiment, a case will be described in which a cancel operation is assigned to a flick operation, but the present invention is not limited to this. The input operation assigned to the cancel operation may be differentiated from the confirm operation. In other words, it may be a double tap operation, or the amount of change in the attitude of the mobile terminal may be equal to or greater than a threshold.

[0127] In step S1107, the operation receiving unit 109 notifies the application operation unit 110 of the cancel operation. The application operation unit 110 stops updating the position and orientation of the virtual object by following the position and orientation of the ray, and returns it to the position and orientation before following the ray. Then, the process proceeds to step S505.

[0128] According to this embodiment, application operations that can be performed on a mobile terminal are changed between when the mobile terminal is visible and when the mobile terminal is not visible, thereby enabling suitable application operations in a MR experience.

[0129] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0130] (Additional Note) The present disclosure includes the following configurations and methods. [Configuration 1] an image generating means for generating an image to be displayed on a display device that can be worn on a user's head; an operation mode determination means for determining an operation mode of an application; an application operation means for receiving an input operation from a portable terminal that can be held and operated by a user, and for performing an application operation based on the operation mode and the input operation; Equipped with the operation mode determination means determines, as the operation mode, at least one of a first operation mode in which the user operates the mobile terminal to perform an application operation, and a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application by operating the virtual UI; 23. An information processing apparatus comprising: [Configuration 2] the operation mode determination means determines the operation mode to be the first operation mode when the portable terminal is visible to the user, based on a determination result of whether or not the portable terminal is visible to the user, and determines the operation mode to be the second operation mode when the portable terminal is not visible to the user. 2. The information processing device according to configuration 1. [Configuration 3] the operation mode determination means determines whether the user can view the portable terminal based on the position and orientation of the display device and the position and orientation of the portable terminal; 3. The information processing device according to configuration 2. [Configuration 4] The display device has an imaging means, the operation mode determination means determines whether the user can visually recognize the portable terminal based on whether the portable terminal is included in an image captured by the imaging means; 4. The information processing device according to configuration 2 or 3. [Configuration 5] the operation mode determination means determines whether or not the user can view the portable terminal based on whether or not the captured image has a resolution that allows the user to view the portable terminal when the captured image is displayed on the display device. 5. The information processing device according to configuration 4. [Configuration 6] the image generating means generates a mixed reality image by superimposing a virtual object on the captured image, the operation mode determination means determines whether the user can view the portable terminal based on whether the portable terminal can be viewed without being obstructed by the virtual object in the mixed reality image; 6. The information processing device according to configuration 4 or 5. [Configuration 7] the operation mode determination means acquires a line of sight direction of the user, and when the line of sight direction is not directed toward the mobile terminal, determines the operation mode to be the second operation mode. 7. The information processing device according to any one of configurations 2 to 6. [Configuration 8] the operation mode determination means does not change the operation mode from when the application operation in the determined operation mode is started until when the application operation is ended. 8. The information processing device according to any one of configurations 2 to 7. [Configuration 9] The operation mode determination means determines an operation mode based on an input from the user. 9. The information processing device according to any one of configurations 2 to 8. [Configuration 10] The second operation mode is an operation mode in which the user operates the virtual UI by operating the mobile terminal. 10. The information processing device according to any one of configurations 1 to 9. [Configuration 11] The present invention further includes a measuring means for measuring a position and orientation of at least a part of a body of the user, The second operation mode is an operation mode for operating the virtual UI based on a position and posture of at least a part of the body of the user. 11. The information processing device according to any one of configurations 1 to 10. [Configuration 12] At least a part of the user's body is a finger of the user, 12. The information processing device according to configuration 11. [Configuration 13] At least a portion of the body of the user is the line of sight of the user; 13. The information processing device according to configuration 11 or 12. [Configuration 14] The virtual UI is a virtual object including a GUI. 14. The information processing device according to any one of configurations 1 to 13. [Configuration 15] The virtual UI includes a virtual object for performing a selection operation or a decision operation. 15. The information processing device according to any one of configurations 1 to 14. [Configuration 16] In the second operation mode, the virtual UI is operated based on the position and orientation of the mobile terminal. 16. The information processing device according to any one of configurations 1 to 15. [Configuration 17] the mobile terminal includes a touch panel capable of recognizing touch gestures; In the second operation mode, the virtual UI is operated based on a moving distance and a direction of the touch gesture. 17. The information processing device according to any one of configurations 1 to 16. [Configuration 18] In the first operation mode, a GUI for performing application operations is displayed on a display of the mobile terminal, and the application operations are performed based on operations on the GUI. 18. The information processing device according to any one of configurations 1 to 17. [Configuration 19] The first operation mode and the second operation mode are modes in which GUIs displayed for operating an application are different from each other. 19. The information processing device according to any one of configurations 1 to 18. [Configuration 20] The application operations that can be executed in the first operation mode and the second operation mode are different. 20. The information processing device according to any one of configurations 1 to 19. [Configuration 21] A control means for controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; an application operation means for receiving an input operation using the GUI and performing an application operation based on the input operation; Equipped with the GUI displayed on at least one of the display device and the mobile terminal is different depending on whether the user can see the mobile terminal or not; 23. An information processing apparatus comprising: [Configuration 22] A control means for controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; an application operation means for receiving an input operation using the GUI and performing an application operation based on the input operation; Equipped with the control means controls the display of a GUI including a virtual UI related to application operations on the display device and the display of a GUI related to application operations on the mobile terminal so as to be performed simultaneously; the application operation means performs the application operation based on at least one of an operation on a GUI including the virtual UI and an operation on a GUI displayed on the mobile terminal; 23. An information processing apparatus comprising: [Configuration 23] The display device; The mobile terminal; An information processing device according to any one of configurations 1 to 22; A system including: [Configuration 24] A method for controlling a computer, comprising: an image generating step of generating an image to be displayed on a display device that can be worn on a user's head; an operation mode determination step for determining an operation mode of the application; an application operation step of receiving an input operation from a portable terminal that can be held and operated by a user, and performing an application operation based on the operation mode and the input operation; Including, In the operation mode determination step, at least one of a first operation mode in which the user operates the mobile terminal to perform an application operation and a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application operation by operating the virtual UI is determined as the operation mode. A control method comprising: [Configuration 25] A method for controlling a computer, comprising: A control step of controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; a determining step of determining whether the user can see the mobile terminal; an application operation step of accepting an input operation using the GUI and performing an application operation based on the input operation; Including, a GUI displayed on at least one of the display device and the mobile terminal differs depending on whether the user can see the mobile terminal or not; A control method comprising: [Configuration 26] A method for controlling a computer, comprising: A control step of controlling display of a GUI on a display device that can be worn on a user's head and on a portable terminal that can be held and operated by the user; an application operation step of accepting an input operation using the GUI and performing an application operation based on the input operation; Including, In the control step, a GUI including a virtual UI for application operations is displayed on the display device, and a GUI for application operations on the mobile terminal is displayed simultaneously. In the application operation step, the application operation is performed based on at least one of an operation on a GUI including the virtual UI and an operation on a GUI displayed on the mobile terminal. A control method comprising: [Configuration 27] A computer program causing a computer to execute each step of the control method according to any one of configurations 24 to 26. [Explanation of symbols]

[0131] 100: Information processing device 101: HMD 102: Mobile terminal 107: Operation mode determination unit 110: Application operation unit 112: Virtual image generation unit

Claims

1. an image generating means for generating an image to be displayed on a display device that can be worn on the user's head; an operation mode determination means for determining an operation mode of an application; an application operation means for receiving an input operation from a portable terminal that can be held and operated by a user, and for performing an application operation based on the operation mode and the input operation; Equipped with The display device has an imaging means, the image generation means generates a mixed reality image by superimposing a virtual object on the captured image captured by the imaging means, the operation mode determination means determines whether the user can view the portable terminal based on whether the portable terminal can be viewed without being obstructed by the virtual object in the mixed reality image; the operation mode determination means, when determining that the user can see the mobile terminal, determines a first operation mode in which the user operates the mobile terminal to perform an application operation, and, when determining that the user cannot see the mobile terminal, determines a second operation mode in which a virtual UI related to the operation is displayed on the display device and the user operates the application by operating the virtual UI.

1. An information processing device comprising:

2. the operation mode determination means acquires a line of sight direction of the user, and when the line of sight direction is not directed toward the mobile terminal, determines the operation mode to be the second operation mode; 2. The information processing apparatus according to claim 1, wherein:

3. the second operation mode is an operation mode in which the user operates the virtual UI by operating the mobile terminal; 2. The information processing apparatus according to claim 1, wherein:

4. further comprising a measuring means for measuring the position and orientation of at least a part of the user's body; the second operation mode is an operation mode in which the virtual UI is operated based on the position and posture of at least a part of the body of the user; 2. The information processing apparatus according to claim 1, wherein:

5. At least a part of the user's body is a finger of the user.

5. The information processing apparatus according to claim 4,

6. At least a portion of the body of the user is the user's line of sight; 5. The information processing apparatus according to claim 4,

7. The virtual UI includes a ray that is a pointing object.

2. The information processing apparatus according to claim 1, wherein:

8. In the second operation mode, the virtual UI is operated based on the position and orientation of the mobile terminal.

2. The information processing apparatus according to claim 1, wherein:

9. In the first operation mode, a GUI for operating an application is displayed on a display of the mobile terminal, and the application is operated based on an operation on the GUI.

2. The information processing apparatus according to claim 1, wherein:

10. the display device; the mobile terminal; An information processing device according to any one of claims 1 to 9; A system including:

11. A method for controlling a computer, comprising: an image generation step of generating an image to be displayed on a display device that can be worn on the user's head; an operation mode determination step for determining an operation mode of the application; an application operation step of receiving an input operation from a portable terminal that can be held and operated by a user, and performing an application operation based on the operation mode and the input operation; Including, The display device has an imaging means, the image generating step generates a mixed reality image by superimposing a virtual object on the captured image captured by the imaging means, In the operation mode determination step, it is determined whether the user can view the portable terminal based on whether the portable terminal can be viewed without being obstructed by the virtual object in the mixed reality image; In the operation mode determination step, if it is determined that the user can see the mobile terminal, a first operation mode is determined in which the user operates the mobile terminal to perform application operations, and if it is determined that the user cannot see the mobile terminal, a second operation mode is determined in which a virtual UI related to operation is displayed on the display device and the user operates the virtual UI to perform application operations. A control method comprising:

12. A computer program causing a computer to execute each step of the control method according to claim 11.

Images