Information processing apparatus, method for controlling information processing apparatus, system, and program

Abstract

To improve the operability of a virtual screen by a user of an information processing apparatus.SOLUTION: An information processing apparatus includes: image acquisition means for acquiring a captured image; and display control means for displaying a composite image generated by combining the captured image and a virtual object including a virtual screen on display means. When a mobile terminal is moved, the display control means changes a position at which the virtual screen related to the mobile terminal is displayed such that it is closer to the mobile terminal based on the position of the mobile terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Head-mounted displays (HMDs) and the like are used as devices for experiencing mixed reality (MR). Patent Document 1 discloses that a virtual screen is displayed on an information processing device such as a head-mounted display, and the virtual screen is moved by a user's operation.

[0003] Recently, there is a technology that connects a personal computer (PC) to a head-mounted display and displays the PC screen as a virtual object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-113692 Summary of the Invention [Problem to be solved by the invention]

[0005] When a plurality of virtual screens are displayed on an information processing device, the user's field of vision becomes cluttered due to the virtual screens, making it difficult for the user to operate the virtual screens.

[0006] The present invention has been made in consideration of these points, and has as its object to improve the operability of the virtual screen for users of information processing devices. [Means for solving the problem]

[0007] One aspect of the present invention is an information processing device comprising: an image acquisition means for acquiring a captured image; and a display control means for displaying on a display means a composite image that combines the captured image with a virtual object including a virtual screen, wherein the display control means, when a mobile terminal is moved, changes the position at which the virtual screen relating to the mobile terminal is displayed so as to be closer to the mobile terminal based on the position of the mobile terminal. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the operability of the virtual screen for the user of the information processing device. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are a block diagram and a schematic diagram showing the hardware configuration of a head-mounted display and a device according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a display in the first embodiment. [Figure 3] 3 is an example of a data list according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a display in the first embodiment. [Figure 5] 5 is a flowchart showing processing of the head-mounted display and device in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a display in the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a display in the second embodiment. [Figure 8] 10 is a flowchart showing processing of a head-mounted display and a device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) FIG. 1(a) is a block diagram showing the hardware of an information processing system in the first embodiment. The information processing system is composed of an information processing device for head mounting and a mobile terminal.

[0012] Reference numeral 150 in FIG. 1(a) is a head-mounted display (HMD) which is an information processing device for head mounting, and 151 is a mobile terminal used in connection with the head-mounted display, and is a device such as a PC, a tablet, a smartphone, or the like.

[0013] <Internal Configuration of HMD> Referring to FIG. 1(a), the internal configuration of the HMD 150 will be described.

[0014] The camera 101 is an imaging unit fixed to the housing of the HMD 150. The camera 101 is an image acquisition unit that captures an image in front of the camera 101 to acquire a captured image representing a real space, which is a three-dimensional space. The camera 101 has, for example, two cameras, a camera for the left eye and a camera for the right eye. The two cameras are used to capture images to be used for combining with an image of a virtual space and for generating position and orientation information, and have an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 150, and the imaging unit for the left eye outputs an image of each frame of the moving image (captured image). The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 150, and the imaging unit for the right eye outputs an image of each frame of the moving image (captured image). That is, the camera 101 acquires captured images as stereo images having parallax that approximately matches the positions of the left and right eyes of the wearer of the HMD 150. Furthermore, distance information on the distance from the two cameras to the subject can be acquired by distance measurement using the stereo cameras. Note that in an HMD for an MR system, it is preferable to position the central optical axis of the imaging range of the imaging unit so that it approximately matches the line of sight of the wearer of the HMD.

[0015] The left-eye imaging unit and the right-eye imaging unit each have an optical system and an imaging device. Light entering from the outside enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incoming light as a captured image. Note that the imaging unit 202 may capture and output a video instead of a captured image.

[0016] The camera 101 may be provided with a gaze capturing unit that captures the gaze of the user. The gaze capturing unit is a camera that captures an image for detecting the gaze of the user, and is a gaze capturing unit that is attached to the inside of the HMD to capture an image of the user's eyes when the user wears the HMD 150. The image of the subject (the user's eyes) captured by the camera is output to the control unit 102. The control unit 102 detects the gaze of the user wearing the HMD 150 from the image captured by the gaze capturing unit, and identifies the part of the display 105 that the user is gazing at.

[0017] The control unit 102 is a CPU that controls each component of the HMD 150. When the control unit 102 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the drawing unit 106, the control unit 102 performs display control to display the composite image on the display 105. Note that instead of the HMD control unit 201 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing load.

[0018] The storage unit 103 is a memory used as a work memory for the control unit 102 and the drawing unit 106, and as a place to store images captured by the camera 101. It is also used as a place to store various other data. It is also assumed that the storage unit 103 stores application programs installed in the HMD 150.

[0019] The position and orientation estimation unit 104 estimates the position and orientation of the camera 101 at the time when the camera image input to the storage unit 103 was captured. In this embodiment, the position and orientation is estimated using a method for estimating the camera's own position and orientation, such as SLAM (Simultaneous Localization and Mapping). Note that the position and orientation estimation unit 104 may be able to estimate its own position by LiDAR SLAM (LiDAR; Light Detection And Ranging) using a laser, in addition to Visual SLAM, which estimates its own position from a camera image. Note that it may also be able to estimate its own position by Depth SLAM, which uses a ToF sensor (ToF; Time of Flight).

[0020] For example, by using LiDAR SLAM, the position and orientation of the HMD can be estimated even in dark places, so that a virtual screen can be displayed according to the relative positions of the mobile device and the HMD, even in dark places.

[0021] The display 105 is a display unit that displays the composite image generated by the drawing unit 106. The display 105 is positioned so as to be visible to both eyes of the user and is located near both eyes.

[0022] The drawing unit 106 is a GPU that draws CG data on the captured image stored in the storage unit 103 to generate a composite image. In addition, the drawing unit 106 also draws information such as rays, which are virtual rays indicating the designated position, for display on the display 105.

[0023] The ROM 107 is a non-volatile memory that can be electrically erased and recorded. The ROM 107 stores various programs, such as programs for the overall operation of the HMD 150 and processing programs described later, and can be executed by the control unit 102.

[0024] Communication unit 108 is configured with, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller to transmit and receive data to and from mobile terminal 151. Communication unit 108 outputs a weighted wireless signal from the antenna and demodulates a wireless signal received by the antenna, thereby achieving short-range wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). Note that communication unit 108 may be wired, such as a USB cable (registered trademark), or wireless, such as Wi-Fi (Wireless Fidelity) (registered trademark). Note that communication unit 108 may similarly transmit and receive data to and from mobile terminals 152 and 153, and may also transmit and receive data to and from other devices.

[0025] The inertial measurement unit 109 is a sensor for detecting the position and orientation of the HMD 150. The inertial measurement unit 109 acquires position information and orientation information of the user (the user wearing the HMD 150) that correspond to the position and orientation of the HMD 150. The inertial measurement unit 109 has an inertial measurement unit (IMU) configured with inertial sensors such as an acceleration sensor and an angular acceleration sensor. The inertial measurement unit 109 is used to acquire position information and orientation information of the user, and the control unit 102 acquires the user's position information and orientation information from the inertial measurement unit 109. Note that the inertial measurement unit 109 may be capable of detecting only orientation information, only position information, or both orientation information and position information. In other words, it may be capable of detecting at least one of orientation information and position information. Note that the inertial measurement unit 109 may include a geomagnetic sensor that detects the orientation of the HMD 150. The control unit 102 acquires information on the orientation of the HMD 150 from the geomagnetic sensor.

[0026] The internal bus 110 is connected to the camera 101, the control unit 102, the storage unit 103, the position and orientation estimation unit 104, the display 105, the drawing unit 106, the ROM 107, the communication unit 108, and the inertial measurement unit 109. The units connected to the internal bus 110 are capable of exchanging data with each other via the internal bus 110.

[0027] The HMD 150 may be connected via wire or wireless to a PC that mainly handles image processing, and at least some of the above-mentioned functions may be provided by the PC.

[0028] An output unit (not shown) may be provided to output sound, vibration, light, etc. In particular, when changing the position of the virtual screen (described later), vibration may be transmitted to the user. By transmitting the vibration to the user, the user can feel that the virtual screen has been moved.

[0029] <Internal structure of mobile device> 1(a), the internal configuration of the mobile terminal 151 will be described. Note that the mobile terminals 152 and 153 are assumed to have the same configuration as the mobile terminal 151, and descriptions thereof will be omitted.

[0030] The display 120 is a display unit that displays a screen suited to an application selected on the mobile terminal 151 .

[0031] The control unit 121 is a CPU that controls each component of the mobile terminal 151 .

[0032] The storage unit 122 is a memory that holds various data, and also stores the application programs installed on the mobile terminal 151.

[0033] The position detection unit 123 uses a GPS (Global Positioning System) or the inertial measurement device described above to detect the position of the mobile terminal 151. Note that the position and orientation estimation unit 104 on the HMD 150 side may detect the position of the mobile terminal 151 from a captured image.

[0034] The communication unit 124 has the same configuration as the communication unit 108 and transmits and receives data to and from the HMD 150 .

[0035] The ROM 125 is a non-volatile memory that can be electrically erased and recorded. It stores various programs, such as the overall operation of the mobile terminal 151 and processing programs described later, and can be executed by the control unit 1221.

[0036] The internal bus 126 is connected to the display 120, the control unit 121, the storage unit 122, the position detection unit 123, the communication unit 124, and the ROM 125. The units connected to the internal bus 126 are capable of exchanging data with each other via the internal bus 126.

[0037] 1(b) is a schematic diagram showing the configuration of FIG. 1(a), in which HMD 150 is worn on the user's head, and mobile terminals 151, 152, and 153 are placed on a table and connected to HMD 150. It is assumed that mobile terminals 152 and 153 have the same hardware configuration as mobile terminal 151.

[0038] <Example of a head-mounted display screen> 2(a), 2(b), and 2(c) illustrate a scenario in which a mobile terminal 151, a mobile terminal 152, and a mobile terminal 153 are connected to an HMD 150. The scenario illustrates how an application of the HMD 150 and applications of the mobile terminals 151, 152, and 153 are displayed on the display 105 of the HMD 150.

[0039] FIG. 2(a) shows the display on the display 105 when the HMD 150 is activated and an image of real space is visible. A composite image 205 is an image displayed on the display 105, and represents the visual field of the user wearing the HMD 150. A tree 202 and a table 203 are real objects. A hand 201 is the user's hand, and it is assumed that the hand of the user wearing the HMD 150 is visible. A virtual object 204 is a virtual line (ray) extending from the user's hand 201. A target is designated and operated using the tip of this ray 204.

[0040] 2(b) shows the display on display 105 when mobile terminal 151, mobile terminal 152, and mobile terminal 153 are connected from the state shown in FIG. 2(a) and immediately before the respective applications are started. Virtual object 206 is a list of applications held by HMD 150, and may be displayed when HMD 150 is started. Virtual object 207, virtual object 208, and virtual object 209 are lists of applications held by the connected devices, and an application to be started is selected and determined by ray 204.

[0041] Figure 2(c) shows the display on display 105 when an application is launched from the state shown in Figure 2(b) and the user is operating the application. Once the application is launched, the user can freely operate the application, and in this example, the application is operated using ray 204.

[0042] 3(a) and 3(b) show a data list for managing the positional relationship between the HMD 150, the mobile terminal 151, the mobile terminal 152, and the mobile terminal 153 in this embodiment and for determining the location of the launched application.

[0043] 3(a) shows the position of HMD 150, information on the angle and elevation angle of where it is facing, and location information on the locations of mobile terminal 151, mobile terminal 152, and mobile terminal 153. This information is updated when HMD 150 moves or changes its posture, and similarly updated when mobile terminal 151, mobile terminal 152, and mobile terminal 153 move. The columns for mobile terminal 151, mobile terminal 152, and mobile terminal 153 in FIG. 3(a) are registered when connected to HMD 150, and deleted when disconnected.

[0044] 3(b) shows the applications running on the HMD 150, the mobile terminal 151, the mobile terminal 152, and the mobile terminal 153, and the location of each application within the space of the HMD 150. When the user moves an application, the location of the moved application is updated. The field for each running application in FIG. 3(b) is registered when the application is started, and deleted when the application is closed.

[0045] 3, the origin may be anywhere, and may be fixed to the position when the HMD 150 is started, for example. However, the coordinate axes of the position coordinates of the mobile terminals 151, 152, and 153 must be the same as the coordinate axes of the HMD 150.

[0046] Figures 4(a), 4(b), 4(c), 4(d), and 4(e) show the virtual screen displayed on the display 105 of the HMD 150. Figures 4(a), 4(b), 4(c), 4(d), and 4(e) show how the virtual screen moves as the mobile terminal 151, the mobile terminal 152, and the mobile terminal 153 are moved.

[0047] Figure 4(a) shows the same state as Figure 2(c).

[0048] 4(b) shows how the virtual screen is displayed when the mobile terminal 153 is moved to the left of the mobile terminal 151 from the state of FIG. 4(a). The virtual screen 401 is a group of virtual screens related to applications including apps G and H running on the mobile terminal 153. When the mobile terminal 153 moves, the group of virtual screens 401 moves to the left of app A, which is located on the leftmost side of the applications running on the mobile terminal 151. In other words, the control unit 102 of the HMD 150 controls the virtual screen related to the mobile terminal 153 to move based on the positional relationship between the mobile terminals as the mobile terminal 153 moves.

[0049] In addition, the position of the virtual screen is changed so as to move closer to the moved mobile device as the mobile device moves. The virtual screen may be moved as if it were moving together with the mobile device when viewed from the user, or may be moved closer to the mobile device when the mobile device has stopped moving for a predetermined period of time. For example, if the virtual screen appears to move together with the mobile device when viewed from the user, the position of the virtual screen is determined for each frame based on the position of the mobile device in the captured image so that the virtual screen is closer to the mobile device than the position of the virtual screen in the previous frame.

[0050] 4(c) shows how the virtual screen is displayed when the mobile terminal 151 is moved to the right of the mobile terminal 152 from the state of FIG. 4(b). The virtual screen 402 is a group of virtual screens related to applications running on the mobile terminal 151, including app A, app B, and app C. When the mobile terminal 151 moves, the group of virtual screens 402 moves to the right of app E, which is located at the rightmost position among the applications running on the mobile terminal 152. In other words, the control unit 102 of the HMD 150 controls the virtual screen related to the mobile terminal 151 to move based on the positional relationship between the mobile terminals as the mobile terminal 151 moves.

[0051] FIG. 4(d) shows how the virtual screens are displayed when the mobile terminal 152 is moved from the state shown in FIG. 4(c) to the immediate right of the mobile terminal 153. The virtual screen 403 is a group of virtual screens related to applications running on the mobile terminal 152, including apps D and E. When the mobile terminal 152 moves, the virtual screen group 403 moves to the right of the virtual screen group 401. Note that on the right side of the virtual screen group 401 is a group of virtual screens 404 related to applications running on the HMD 150, including apps 1, 2, and 3. Therefore, the virtual screen group 404 is positioned around a center line 405 of the HMD 150, and the virtual screen group 401, the virtual screen group 402, and the virtual screen group 403 are positioned so as not to overlap with the virtual screen group 404. This operation is performed assuming that the HMD 150 is placed on the center line 405 between the mobile terminals 152 and 151.

[0052] Note that a mode for rearranging the virtual screens may be provided when the virtual screens of multiple mobile devices are displayed. In this case, a group of virtual screens may be displayed for each virtual screen associated with each mobile device, based on the relative positions of the mobile devices in real space determined by the user over a predetermined period of time. For example, the user switches to a mode for rearranging the virtual screens and moves the mobile device from the state shown in FIG. 4(a) to the state shown in FIG. 4(b) and then to the state shown in FIG. 4(c). The user also issues an instruction to end the mode for rearranging the virtual screens. Upon receiving an instruction to end the mode for rearranging the virtual screens, the control unit 102 rearranges the virtual screens so that they are displayed in the state shown in FIG. 4(c). When rearranging the virtual screens in this manner, the virtual screens do not move in accordance with the movement of the multiple mobile devices one by one, but rather the display position of each virtual screen is determined based on the relative positions of the mobile devices at a predetermined time.

[0053] It is also possible to consider separate applications running on the HMD 150. For example, when the mobile terminal 151 is moved to the immediate right of the mobile terminal 153 from the state shown in Fig. 4(c), the virtual screen groups 401 and 402 may not be moved, and the virtual screen group 403 may be placed to the right of the virtual screen group 401 even if it overlaps with the virtual screen group 404.

[0054] Fig. 4(e) shows the state of the display on the display 105 when the mobile terminal 152 is moved behind the mobile terminal 151 from the state of Fig. 4(d). The virtual screen group 402 on which the mobile terminal 152 is running moves behind the virtual screen group 403 on which the mobile terminal 151 is running.

[0055] In the example of Fig. 4, the virtual screens are moved without changing their relative positions within the group of virtual screens, but for example, when the group of virtual screens is moved, the virtual screens within the group of virtual screens may be arranged in a line in the depth direction or in a line in the horizontal direction. When arranged in a line, the virtual screens may overlap each other for each application.

[0056] When applications overlap, the anteroposterior relationship of the applications before the move may be maintained, or the application that the user's gaze was last on may be placed in front. The anteroposterior relationship of the applications may be based on a predetermined application priority, such as table applications being given top priority and text applications being given second priority, or may be determined by application color. Here, the anteroposterior relationship of the applications refers to the relationship between which virtual screen is in the foreground (closer to the user) and which virtual screen is in the background.

[0057] <Virtual screen layout processing> Figures 5(a) and 5(b) are flowcharts showing the details of the processing of this embodiment, where Figure 5(a) is a flowchart for HMD 150 and Figure 5(b) is a flowchart for mobile terminal 151, mobile terminal 152, and mobile terminal 153.

[0058] It is assumed that the HMD 150, the mobile terminal 151, the mobile terminal 152, and the mobile terminal 153 of this embodiment have already been started up.

[0059] The processing of the HMD 150 will be described with reference to FIG.

[0060] In step S501, the control unit 102 controls the camera 101 to acquire a captured image (stereo camera image), stores the captured image in the storage unit 103, and the process proceeds to step S502.

[0061] In step S502, the control unit 102 detects the user's hand from the stereo camera image stored in step S501, monitors the position pointed by the ray 204, determines information about what the user is pointing, and proceeds to step S503. The user's hand can be image-recognized from the stereo image data, and information such as whether the user is moving something or pressing the enter button can be determined by comparing with the past positions of the user's hand, and the target of the instruction can be determined by the position pointed by the ray 204.

[0062] In step S503, the control unit 102 determines whether or not an instruction to connect to a device (mobile terminal) has been received from the user in step S502. If it is determined that an instruction to connect to a device has been received, the process proceeds to step S504; otherwise, the process proceeds to step S505. An instruction to connect to a device can be determined by displaying candidate devices to connect to and connection buttons, etc., in advance, and detecting when the ray 204 pointed by the user presses the connection button, etc. The communication unit 108 may automatically detect devices that can be connected and detect when the user presses the permission button, etc.

[0063] In step S504, the control unit 102 controls the communication unit 108 to connect to the device, and proceeds to step S505. The connected device is registered in the data list of Fig. 4(a). When connected to the device, the control unit 102 may display on the display 105 a list of applications that can be launched, similar to the virtual object 207, virtual object 208, and virtual object 209 of Fig. 2(b).

[0064] In step S505, the control unit 102 determines whether or not an instruction to disconnect the device was received from the user in step S502. If it is determined that an instruction to disconnect the device was received, the process proceeds to step S506; if not, the process proceeds to step S507. An instruction to disconnect the device can be determined by detecting, for example, that the user has pressed the disconnect button on the device.

[0065] In step S506, the control unit 102 controls the communication unit 108 to disconnect the device for which the device disconnection instruction was issued, and the process proceeds to step S507. Note that the disconnected device is deleted from the data list in FIG.

[0066] In step S507, the control unit 102 sets the device count number N to 0, and the process proceeds to step S508.

[0067] In step S508, the control unit 102 compares the number of connected devices with the device count number N, and if N is smaller than the number of connected devices, the process proceeds to step S509, and if not, the process proceeds to step S515.

[0068] In step S509, the control unit 102 controls the communication unit 108 to acquire the location information of the connected device, and the process proceeds to step S510.

[0069] In step S510, the control unit 102 determines whether or not the user has instructed the device to start an application in step S502. If it is determined that an instruction to start an application has been given, the process proceeds to step S511; otherwise, the process proceeds to step S512.

[0070] In step S511, the control unit 102 controls the communication unit 108 to send an application startup instruction to the connected device, and then the process proceeds to step S512. The application for which the startup instruction was given is registered in the startup application field for the device in question in the data list of FIG. 4(b). The application may be located anywhere, for example, in front of the HMD 150.

[0071] In step S512, the control unit 102 determines whether or not the user has instructed the device to end the application in step S502. If it is determined that an instruction to end the application has been given, the process proceeds to step S513; otherwise, the process proceeds to step S514.

[0072] In step S513, the control unit 102 controls the communication unit 108 to send an application termination instruction to the connected device, and then the process proceeds to step S512. Note that the application that has been instructed to be terminated is deleted from the startup application column of the device in question in the data list of FIG. 4(b).

[0073] In step S514, the control unit 102 adds 1 to the device count number N, and the process proceeds to step S508.

[0074] In step S515, the control unit 102 controls the position and orientation estimation unit 104 to acquire the position and orientation information of the HMD 150, stores it in the storage unit 103, and proceeds to step S516. The position and orientation information is information on the position and orientation in FIG. 3(a). Note that the orientation may be an angle or an elevation angle.

[0075] In step S516, the control unit 102 determines whether or not the user has instructed the HMD 150 in step S502 to start an application. If it is determined that an instruction to start an application has been given, the process proceeds to step S517; otherwise, the process proceeds to step S518.

[0076] In step S517, the control unit 102 starts the application, and the process proceeds to step S518. The application can be started by the control unit 102 reading out the corresponding application program from the storage unit 103.

[0077] In step S518, the control unit 102 determines whether or not the user has instructed the HMD 150 in step S502 to end the application. If it is determined that the user has instructed the HMD 150 to end the application, the process proceeds to step S519; otherwise, the process proceeds to step S520.

[0078] In step S519, the control unit 102 ends the application, and the process proceeds to step S520.

[0079] In step S520, control unit 102 determines whether or not an instruction to operate an application was given in step S502. If it is determined that an instruction to operate an application was given, control proceeds to step S521; otherwise, control proceeds to step S522.

[0080] In step S521, the control unit 102 operates the application in accordance with the instruction of the application acquired in step S502, and the process proceeds to step S522. If the application is an operation of an application of the HMD 150, the control unit 102 controls the application, and if the application is an operation of an application of a device, the control unit 102 controls the communication unit 108 and transmits an instruction to operate the application to the device.

[0081] In step S522, the control unit 102 updates the application and proceeds to step S523. If the application was operated in step S521, the image of the application is updated to include the results, and stored in the storage unit 103. Furthermore, from the device side, the control unit 102 controls the communication unit 108, and receives the image of the application from the device side and stores it in the storage unit 103. If the application operation involves moving the application, the position of the application in the data list in FIG. 3(b) is updated.

[0082] In step S523, the control unit 102 determines whether the device has moved based on the device location information acquired in step S509. If it is determined that the device has moved, the process proceeds to step S524; if not, the process proceeds to step S525.

[0083] In step S524, the control unit 102 updates the application positions in response to the device movement, and then proceeds to step S525. The application positions are updated by updating the data list shown in FIG. 4. For example, if the mobile terminal 153 moves to the left of the mobile terminal 151, the virtual screen for the mobile terminal 153 in FIG. 4(b) should be moved to the left of the leftmost virtual screen of the virtual screen for the mobile terminal 151. That is, the applications G and H of the mobile terminal 153 should be moved to the left of the application A, B, and C of the mobile terminal 151 that is located to the left of the application that is located to the left of the application A, B, and C of the mobile terminal 151. Although the application screen sizes are omitted from the data list shown in FIG. 4(b), taking the screen sizes into consideration allows for accurate movement, such as moving applications so that they do not overlap each other. That is, the movement of applications can be controlled simply by monitoring and updating the data list shown in FIG. 4.

[0084] In step S525, the drawing unit 106 reads out the application image stored in the storage unit 103 and draws it on the captured image at a position according to the data list. Furthermore, the drawing unit 106 draws a ray 204 at a position based on the position of the user's hand, and the display 105 displays the composite image of the captured image and the application and the ray 204, and the process proceeds to step S526.

[0085] In step S526, the control unit 102 determines whether or not to terminate the processing of this embodiment. If it is determined that the processing should be terminated, the processing terminates; if not, the processing proceeds to step S501. The determination of whether or not to terminate is made based on whether or not an end instruction has been received from the user in step S502.

[0086] <Processing on mobile devices> Next, a description will be given of the processing of mobile terminal 151, mobile terminal 152, and mobile terminal 153. Mobile terminal 151, mobile terminal 152, and mobile terminal 153 each perform the same processing, and FIG. 5(b) shows the processing of any of mobile terminals 151, 152, and 153.

[0087] In step S531, the control unit 121 controls the communication unit 124 to obtain an instruction from the user, and the process proceeds to step S532. The instruction from the user includes instructions transmitted by the HMD 150 in steps S511, S513, S521, and the like.

[0088] In step S532, the control unit 121 determines whether the instruction from the user acquired in step S531 is an instruction to connect to the HMD 150. If it is determined to be an instruction to connect to the HMD 150, the process proceeds to step S533; otherwise, the process proceeds to step S534. This occurs because the connection instruction is sent by the connection process in step S504 on the HMD 150 side.

[0089] In step S533, the control unit 121 controls the communication unit 124 to connect to the HMD 150, and proceeds to step S534. Note that the control unit 121 controls the communication unit 124 to transmit to the HMD 150 information on a list of applications that can be launched, such as the virtual object 207, the virtual object 208, and the virtual object 209 in FIG. 2B.

[0090] In step S534, the control unit 121 determines whether the instruction from the user acquired in step S531 is a disconnection instruction to the HMD 150. If it is determined to be a disconnection instruction to the HMD 150, the process proceeds to step S533; otherwise, the process proceeds to step S534. This occurs because the disconnection instruction is transmitted by the disconnection process in step S504 on the HMD 150 side.

[0091] In step S535, the control unit 121 controls the communication unit 124 to disconnect from the HMD 150, and the process proceeds to step S536.

[0092] In step S536, the control unit 121 determines whether the instruction from the user acquired in step S531 is an instruction to start an application. If it is determined to be an instruction to start an application, the process proceeds to step S537; if not, the process proceeds to step S538. This is because the application start instruction is sent by the application start process in step S511 on the HMD 150 side.

[0093] In step S537, the control unit 121 starts up the application, and the process proceeds to step S538. The application is started up by the control unit 121 reading out the application program from the storage unit 122.

[0094] In step S538, the control unit 121 determines whether the instruction from the user acquired in step S531 is an instruction to close the application. If it is determined to be an instruction to close the application, the process proceeds to step S539; if not, the process proceeds to step S540. This is because the application close instruction is sent by the application close process in step S513 on the HMD 150 side.

[0095] In step S539, the control unit 121 ends the application, and the process proceeds to step S540.

[0096] In step S540, the control unit 121 determines whether the instruction from the user acquired in step S531 is an instruction to operate an application. If it is determined to be an instruction to operate an application, the process proceeds to step S541; if not, the process proceeds to step S542. This is because an instruction to operate an application is sent by the application operation process in step S521 on the HMD 150 side.

[0097] In step S541, the control unit 121 operates the application, and the process proceeds to step S542. The image of the operated application is stored in the storage unit 122.

[0098] In step S542, the control unit 121 reads out the image of the application held in step S541, and the image is displayed on the display 120, and the process proceeds to step S543.

[0099] In step S543, the control unit 121 determines whether or not the control unit 121 is currently connected to the HMD 150. If it is determined that the control unit 121 is currently connected to the HMD 150, the process proceeds to step S544, and if not, the process proceeds to step S546.

[0100] In step S544, the control unit 121 controls the position detection unit 123 to acquire the position information, controls the communication unit 124 to transmit the information, and then the process proceeds to step S545.

[0101] In step S545, the control unit 121 reads out the image of the latest application currently running from the storage unit 122, controls the communication unit 124 to send it, and then the process proceeds to step S546.

[0102] In step S546, the control unit 121 determines whether or not to end the device processing of this embodiment. If it is determined that the processing should be ended, the processing ends. If not, the processing proceeds to step S531. The determination of whether or not to end the processing is made based on whether or not an end instruction has been received from the user in step S531.

[0103] <How to display the virtual screen> 6 shows the state of the display on the display 105 when the orientation of the HMD 150 is changed to the right from the state shown in FIG. 2(c). In this case, the positions of the applications running on the mobile terminals 151, 152, and 153 do not move, and only the applications running on the HMD 150, that is, app 1, app 2, and app 3, move.

[0104] This results in a situation in step S524 where the position and elevation angle of HMD 150 in Fig. 4(a) remain unchanged, only the angle has changed, and there is no change in the positions of mobile terminals 151, 152, and 153. In this case, it is sufficient to move only App 1, App 2, and App 3 in the activated app column of HMD 150 in Fig. 4(b) by an amount corresponding to the change in angle.

[0105] In this embodiment, when any one of the mobile terminals 151, 152, and 153 is moved out of the field of view of the HMD 150, an application running on the moved device may be moved to the edge of the screen. Outside the field of view of the HMD 150 means, for example, outside the imaging range of the camera 101, which may be outside the imaging range of a camera that acquires video for the user to view, or outside the imaging range of a camera that acquires video for acquiring the position and orientation of the HMD 150. When the mobile terminal is moved out of the field of view of the HMD 150, the virtual screen may be moved behind the other virtual screens (toward the user) or may be hidden. The hidden virtual screen may be displayed when the device is brought back into the field of view. Hiding may mean erasing or minimizing.

[0106] In this embodiment, when the screen of any of mobile terminals 151, 152, and 153 is turned face down, i.e., when the display portion becomes invisible, the applications running on that mobile terminal may be deleted. The determination of turning the screen face down may be made by determining the orientation of the mobile terminal based on a captured image. When the screen is turned face up, i.e., when the display portion becomes visible, the applications that were deleted are redisplayed. In addition, if the mobile terminal is foldable, the applications running on that mobile terminal may be deleted when it is folded.

[0107] (Second embodiment) In the second embodiment, in addition to the first embodiment, a method for determining the placement of applications running on the device depending on whether the user wearing the HMD 150 is holding the device in their hand or wearing it will be described.

[0108] 7 shows the state of the display on the display 105 in the second embodiment when the user holds the mobile terminal 153 in his / her hand from the state shown in FIG. 4(d). The group of virtual screens 401 on which the mobile terminal 153 is running moves to the front, and the group of virtual screens 404 on which the HMD 150 is running, which was originally placed in that location, moves behind the group of virtual screens 401. In other words, the application related to the device being held in the hand is given priority over the application running on the head-mounted display. Note that moving an application to the front is one example of a method of giving priority, and other methods of giving priority may also be used.

[0109] Note that when the mobile terminal 153 is moved away from the state of FIG. 7 or placed on the table 203, the virtual screen group 404 that has been moving to the rear may move to the front.

[0110] When the orientation of the HMD 150 is changed to the right as shown in FIG. 6, not only the applications running on the HMD 150 but also the applications running on the mobile terminal 153 being held in the hand or worn may be moved.

[0111] FIG. 8(a) is a flowchart showing the details of the processing of the HMD 150 of this embodiment, and FIG. 8(b) is a flowchart showing the details of the processing of the mobile terminal 151, the mobile terminal 152, and the mobile terminal 153 of this embodiment.

[0112] The processing of the HMD 150 of this embodiment will be described in detail below with reference to Fig. 8(a). Steps S801 to S823 of the processing flow of the HMD 150 are the same as steps S501 to S523 in Fig. 5, so their description will be omitted. Also, steps S826 to S828 are the same as steps S524 to S526 in Fig. 5, so their description will be omitted, and only steps S824 and S825, which are different, will be described.

[0113] In step S824, the control unit 102 determines whether the moving device (mobile terminal) is a device worn by the user. If the control unit 102 determines that the moving device is worn by the user, the process proceeds to step S825; otherwise, the process proceeds to step S826. Whether the device is worn by the user may be determined by detecting the device from the captured image acquired in step S801, or by information indicating that the device's position information is closer than a predetermined distance to the position information of the HMD 150. Note that it may also be determined that the user is holding the device when the difference between the change in the position or orientation of the HMD 150 and the change in the position or orientation of the device is smaller than a threshold. Note that the user wearing the device may be when the user holds the device, when the user wears the device around their neck, or when the device is stored in their pocket.

[0114] In step S825, the control unit 102 updates the positions of the applications, and then proceeds to step S827. The control unit 102 checks the positional relationship between the applications running on the device determined to be worn in step S825 and the applications running on the HMD 150. If these have the same positional relationship, the control unit 102 updates the data list in FIG. 4(b) so that the applications running on the device determined to be worn are brought to the foreground.

[0115] The processing flow of the device is the same as that shown in FIG. 5(b), and therefore the explanation will be omitted.

[0116] 6 to the right, if the user is holding a device in their hand, the virtual screens related to the device may also move together. This movement determination may also be performed in step S826.

[0117] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0118] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0119] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0120] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0121] [Configuration 1] image acquisition means for acquiring a captured image; a display control means for displaying a composite image obtained by combining the captured image with a virtual object including a virtual screen on a display means, When the mobile terminal is moved, the display control means changes the position at which the virtual screen relating to the mobile terminal is displayed so as to be closer to the mobile terminal based on the position of the mobile terminal. 1. An information processing device comprising:

[0122] [Configuration 2] and when there are a plurality of mobile terminals, when at least one of the plurality of mobile terminals is moved, the display control means changes the positions at which the virtual screens of the plurality of mobile terminals are displayed in accordance with the relative positions of the plurality of mobile terminals. 2. The information processing device according to configuration 1,

[0123] [Configuration 3] When there are a plurality of mobile terminals, when at least one of the plurality of mobile terminals is moved, the display control means changes the position where the virtual screen is displayed so that the virtual screens relating to the plurality of mobile terminals are aligned in accordance with the positional relationship of the plurality of mobile terminals. 3. The information processing device according to configuration 2.

[0124] [Configuration 4] When there are a plurality of virtual screens relating to a predetermined mobile terminal among the plurality of mobile terminals, the display control means displays the plurality of virtual screens relating to the predetermined mobile terminal in an overlapping manner. 4. The information processing device according to configuration 3.

[0125] [Configuration 5] the display control means changes a position where a virtual screen related to the mobile terminal and a position where a virtual screen related to the information processing device are displayed when the mobile terminal is moved; 5. The information processing device according to any one of configurations 1 to 4.

[0126] [Configuration 6] the display control means controls the display so that a virtual screen relating to the mobile terminal on the right side is displayed on the right side, and a virtual screen relating to the mobile terminal on the left side is displayed on the left side; 6. The information processing device according to any one of configurations 2 to 5,

[0127] [Configuration 7] the display control means controls, when the attitude of the display means is changed, the virtual screen related to the information processing device to be displayed so as to move in accordance with the change in the attitude. 7. The information processing device according to any one of configurations 1 to 6.

[0128] [Configuration 8] When the mobile device moves out of an imaging range of the captured image, the display control means controls to hide or minimize a virtual screen related to the mobile device that has moved out of the imaging range. 8. The information processing device according to any one of configurations 1 to 7.

[0129] [Configuration 9] When the user is wearing the mobile terminal, the display control means controls the display so that a virtual screen of the mobile terminal worn by the user is displayed in front. 9. The information processing device according to any one of configurations 1 to 8.

[0130] [Configuration 10] When the user is wearing the mobile terminal, the display control means controls the display so that a virtual screen of the mobile terminal worn by the user moves in accordance with a change in the attitude of the display means. 10. The information processing device according to any one of configurations 1 to 9.

[0131] [Configuration 11] The information processing device according to any one of configurations 1 to 10, characterized in that when the position of the mobile terminal is moved backward, the display control means displays a virtual screen related to the information processing device in front of the user relative to the virtual screen related to the mobile terminal.

[0132] [Configuration 12] The mobile terminal is a device having a display, the display control means controls the mobile terminal to hide or minimize a virtual screen related to the mobile terminal when the mobile terminal is placed face down so that the display is hidden; 12. The information processing device according to any one of configurations 1 to 11.

[0133] [Configuration 13] The mobile terminal is a device having a display, the display control means controls the display to display a virtual screen related to the mobile terminal when the display is changed from the downward facing state to a position where the display is visible. 13. The information processing device according to configuration 12.

[0134] [Configuration 14] when the display control means receives an instruction to fix the position of the first virtual screen related to the first portable terminal, the display control means controls the position of the first virtual screen so that it does not move closer to the first portable terminal even if the first portable terminal is moved. 14. The information processing device according to any one of configurations 1 to 13.

[0135] [Configuration 15] the display control means maintains the front-to-back relationship of the virtual screen when changing the display position of the virtual screen; 15. The information processing device according to any one of configurations 1 to 14.

[0136] [Configuration 16] further comprising a line-of-sight acquisition means for acquiring the line of sight of the user; When changing the display position of the virtual screen, the display control means controls the virtual screen, of the virtual screens displayed on the display means, so that the virtual screen that the user is looking at is displayed in front of the user. 16. The information processing device according to any one of configurations 1 to 15.

[0137] [Configuration 17] When changing the positions at which the plurality of virtual screens related to the mobile terminal are displayed, the display control means controls the display so as to determine a front-to-back relationship of the plurality of virtual screens in accordance with a predetermined priority order. 17. The information processing device according to any one of configurations 1 to 16.

[0138] [Control method] an image acquisition step of acquiring a captured image; a display control step of displaying a composite image on a display means, the composite image being a composite of the captured image and a virtual object including a virtual screen; In the display control step, when the mobile terminal is moved, a position where a virtual screen related to the mobile terminal is displayed is changed so as to be closer to the mobile terminal based on the position of the mobile terminal. 2. A method for controlling an information processing apparatus comprising:

[0139] [program] 18. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17.

[0140] [system] A mobile device, A display device; an image acquisition device that acquires a captured image; a display control device that displays a composite image obtained by combining the captured image with a virtual object including a virtual screen on the display device, When the mobile terminal is moved, the display control device changes the position at which the virtual screen related to the mobile terminal is displayed so as to be closer to the mobile terminal, based on the position of the mobile terminal. An information processing system comprising:

Claims

1. image acquisition means for acquiring a captured image; a display control means for displaying a composite image obtained by combining the captured image with a virtual object including a virtual screen on a display means, When the mobile terminal is moved, the display control means changes the position at which the virtual screen relating to the mobile terminal is displayed so as to be closer to the mobile terminal based on the position of the mobile terminal.

1. An information processing device comprising:

2. and when there are a plurality of mobile terminals, when at least one of the plurality of mobile terminals is moved, the display control means changes the positions at which the virtual screens of the plurality of mobile terminals are displayed in accordance with the relative positions of the plurality of mobile terminals.

2. The information processing apparatus according to claim 1, wherein:

3. When there are a plurality of mobile terminals, when at least one of the plurality of mobile terminals is moved, the display control means changes the position where the virtual screen is displayed so that the virtual screens relating to the plurality of mobile terminals are aligned in accordance with the positional relationship of the plurality of mobile terminals.

3. The information processing apparatus according to claim 2, wherein:

4. When there are a plurality of virtual screens relating to a predetermined mobile terminal among the plurality of mobile terminals, the display control means displays the plurality of virtual screens relating to the predetermined mobile terminal in an overlapping manner.

4. The information processing apparatus according to claim 3,

5. the display control means changes a position where a virtual screen related to the mobile terminal and a position where a virtual screen related to the information processing device are displayed when the mobile terminal is moved; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. the display control means controls the display so that a virtual screen relating to the mobile terminal on the right side is displayed on the right side, and a virtual screen relating to the mobile terminal on the left side is displayed on the left side; 5. The information processing apparatus according to claim 2, wherein the information processing apparatus is a computer.

7. the display control means controls, when the attitude of the display means is changed, the virtual screen related to the information processing device to be displayed so as to move in accordance with the change in the attitude.

2. The information processing apparatus according to claim 1, wherein:

8. When the mobile device moves out of an imaging range of the captured image, the display control means controls to hide or minimize a virtual screen related to the mobile device that has moved out of the imaging range.

2. The information processing apparatus according to claim 1, wherein:

9. When the user is wearing the mobile terminal, the display control means controls the display so that a virtual screen of the mobile terminal worn by the user is displayed in front.

2. The information processing apparatus according to claim 1, wherein:

10. When the user is wearing the mobile terminal, the display control means controls the display so that a virtual screen of the mobile terminal worn by the user moves in accordance with a change in the attitude of the display means.

2. The information processing apparatus according to claim 1, wherein:

11. 2. The information processing device according to claim 1, wherein the display control means displays the virtual screen relating to the information processing device in front of the user relative to the virtual screen relating to the portable terminal when the position of the portable terminal is moved backward.

12. The mobile terminal is a device having a display, the display control means controls the mobile terminal to hide or minimize a virtual screen related to the mobile terminal when the mobile terminal is placed face down so that the display is hidden; 2. The information processing apparatus according to claim 1, wherein:

13. The mobile terminal is a device having a display, the display control means controls the display to display a virtual screen related to the mobile terminal when the display is changed from the downward facing state to a position where the display is visible.

13. The information processing apparatus according to claim 12.

14. when the display control means receives an instruction to fix the position of the first virtual screen related to the first portable terminal, the display control means controls the position of the first virtual screen so that it does not move closer to the first portable terminal even if the first portable terminal is moved.

2. The information processing apparatus according to claim 1, wherein:

15. the display control means maintains the front-to-back relationship of the virtual screen when changing the display position of the virtual screen; 2. The information processing apparatus according to claim 1, wherein:

16. further comprising a line-of-sight acquisition means for acquiring the line of sight of the user; When changing the display position of the virtual screen, the display control means controls the virtual screen, of the virtual screens displayed on the display means, so that the virtual screen that the user is looking at is displayed in front of the user.

2. The information processing apparatus according to claim 1, wherein:

17. When changing the positions at which the plurality of virtual screens related to the mobile terminal are displayed, the display control means controls the display so as to determine a front-to-back relationship of the plurality of virtual screens in accordance with a predetermined priority order.

2. The information processing apparatus according to claim 1, wherein:

18. an image acquisition step of acquiring a captured image; a display control step of displaying a composite image on a display means, the composite image being a composite of the captured image and a virtual object including a virtual screen; In the display control step, when the mobile terminal is moved, a position where a virtual screen related to the mobile terminal is displayed is changed so as to be closer to the mobile terminal based on the position of the mobile terminal.

2. A method for controlling an information processing apparatus comprising:

19. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

20. A mobile device, a display device; an image acquisition device that acquires a captured image; a display control device that displays a composite image obtained by combining the captured image with a virtual object including a virtual screen on the display device, When the mobile terminal is moved, the display control device changes the position at which the virtual screen related to the mobile terminal is displayed so as to be closer to the mobile terminal, based on the position of the mobile terminal. An information processing system comprising:

Images