Information processing apparatus for controlling image display, control method thereof, information processing system, computer program product, and computer-readable storage medium

Abstract

The invention discloses an information processing apparatus for controlling image display, a control method thereof, an information processing system, a computer program product, and a computer readable storage medium. An information processing apparatus communicating with a mobile terminal, the information processing apparatus including: a display control section configured to control a display section to display a virtual space in which a virtual screen related to the mobile terminal is placed in a real space; and an instruction acquisition section configured to acquire an instruction to place a virtual screen related to the mobile terminal in the virtual space, in a case where movement of the mobile terminal is detected, the instruction to place the virtual screen related to the mobile terminal in the virtual space. The display control unit changes a position at which a virtual screen relating to the mobile terminal placed in the virtual space is displayed to a position corresponding to a position of the mobile terminal at which the movement has been detected.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus for controlling image display. Background Art

[0002] As a device for experiencing mixed reality (MR) by a user, a head-mounted display (HMD) is used. Japanese Patent Application Laid-Open No. 2022-113692 discusses a technique for displaying a virtual image on an information processing apparatus such as a head-mounted display and moving the virtual image according to a user operation.

[0003] In recent years, there has been a technique for connecting a personal computer (PC) to a head-mounted display and displaying the screen of the PC as a virtual object.

[0004] In the case of displaying a plurality of virtual images, there is a problem that the user's field of view of the information processing apparatus becomes complicated due to the virtual images and it is difficult for the user to operate the virtual images. Summary of the Invention

[0005] In view of the above problems, an object of the present invention is to improve the operability of a virtual image by a user of an information processing apparatus.

[0006] According to an aspect of the present invention, an information processing apparatus that communicates with a mobile terminal, the information processing apparatus including: a display control unit configured to control a display unit to display a virtual space in which a virtual image related to the mobile terminal is placed in a real space; an instruction acquisition unit configured to acquire an instruction for placing a virtual image related to the mobile terminal in the virtual space, wherein, when a movement of the mobile terminal is detected, the display control unit changes a position of the virtual image related to the mobile terminal placed in the virtual space to a position corresponding to a position of the mobile terminal in which the movement is detected.

[0007] Other features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0008] Figure 1A is a block diagram showing a hardware configuration of a head-mounted display and an apparatus according to a first exemplary embodiment. Figure 1B is a schematic diagram showing a hardware configuration of a head-mounted display and an apparatus according to a first exemplary embodiment.

[0009] Figure 2A is a diagram showing a display example according to a first exemplary embodiment. Figure 2B is a diagram showing a display example according to a first exemplary embodiment. Figure 2C is a diagram showing a display example according to a first exemplary embodiment.

[0010] Figure 3A is an example of a data list according to the first exemplary embodiment. Figure 3B is an example of a data list according to the first exemplary embodiment.

[0011] Figure 4A is a diagram showing a display example according to the first exemplary embodiment. Figure 4B is a diagram showing a display example according to the first exemplary embodiment. Figure 4C is a diagram showing a display example according to the first exemplary embodiment. Figure 4D is a diagram showing a display example according to the first exemplary embodiment. Figure 4E is a diagram showing a display example according to the first exemplary embodiment.

[0012] Figure 5A is a flowchart showing the processing of a head-mounted display according to the first exemplary embodiment. Figure 5B is a flowchart showing the processing of a mobile terminal according to the first exemplary embodiment.

[0013] Figure 6 is a diagram showing a display example according to the first exemplary embodiment.

[0014] Figure 7 is a diagram showing a display example according to the second exemplary embodiment.

[0015] Figure 8A and Figure 8B is a flowchart showing the processing of a head-mounted display according to the second exemplary embodiment. Figure 8C is a flowchart showing the processing of a mobile terminal according to the second exemplary embodiment. Detailed Description of the Invention

[0016] Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. Note that the following exemplary embodiments do not limit the invention according to the appended claims. Although multiple features are described in the exemplary embodiments, not all of the multiple features are essential for the invention, and the multiple features can be arbitrarily combined. In the drawings, the same or similar components are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0017] Figure 1A is a block diagram showing the hardware of an information processing system according to the first exemplary embodiment. The information processing system includes a mobile terminal and an information processing device to be worn on the head.

[0018] Figure 1AThe head-mounted display (HMD) 150 and the mobile terminal 151 are shown. The HMD 150 is an information processing device to be worn on the head, and the mobile terminal 151 is a device such as a personal computer (PC), a tablet, or a smart phone that is used by connecting the device to the head-mounted display 150.

[0019] <Internal Structure of the HMD>

[0020] Reference Figure 1A , the internal structure of the HMD 150 is described.

[0021] The camera 101 is an imaging unit fixed to the housing of the HMD 150. The camera 101 is an image acquisition unit for capturing the space in front of the camera 101 to obtain a captured image representing the real space as a three-dimensional space. For example, the camera 101 includes two cameras, namely a left-eye camera and a right-eye camera. The two cameras capture the captured images to be used for synthesizing the images of the virtual space and generating position / pose information, and the two cameras include a left-eye imaging unit and a right-eye imaging unit. The left-eye imaging unit captures a moving image of the real space corresponding to the left eye of the wearer of the HMD 150 and outputs the images of each frame in the moving image (captured image). The right-eye imaging unit captures a moving image of the real space corresponding to the right eye of the wearer of the HMD 150 and outputs the images of each frame in the moving image (captured image). That is, the camera 101 obtains a captured image that is a stereoscopic image having a parallax that is substantially the same as the positions of the left and right eyes of the wearer of the HMD 150. By using such a stereoscopic camera to measure the distance, information related to the distance from the two cameras to the subject is obtained as distance information. In the HMD 150 for a mixed reality (MR) system, it is desirable that the central optical axis of the imaging range of each imaging unit is placed substantially in line with the line of sight of the wearer of the HMD 150.

[0022] The left-eye imaging unit and the right-eye imaging unit each include an optical system and an imaging device.

[0023] The light entering each imaging unit from the outside passes through the optical system and enters the imaging device, and the imaging device outputs an image corresponding to the light entering the imaging device as a captured image. The imaging unit may capture a video instead of a captured image and output the video.

[0024] In the camera 101, a line-of-sight imaging unit for imaging the user's line of sight can be set. The line-of-sight imaging unit is a camera that acquires an image for detecting the user's line of sight, and is attached inside the HMD 150 to image the user's eyes when the user wears the HMD 150. An image obtained by imaging a subject (the user's eyes) with the camera 101 is output to the control unit 102. The control unit 102 detects the line of sight of the user wearing the HMD 150 from the image captured by the line-of-sight imaging unit, and identifies the portion of the display 105 that the user is gazing at.

[0025] The control unit 102 is a central processing unit (CPU) that controls the components of the HMD 150. In response to the control unit 102 obtaining a composite image from the rendering unit 106 (an image obtained by synthesizing a captured image obtained by imaging the space in front of the user by the imaging unit and computer graphics (CG)), the control unit 102 performs display control for displaying the composite image on the display 105. Instead of the control unit 102 of the HMD 150 controlling the entire device, multiple hardware components can share the processing to control the entire device.

[0026] The storage unit 103 is a memory that serves as a working memory for the control unit 102 or the rendering unit 106 and serves as a place for holding the captured image captured by the camera 101. The storage unit 103 also serves as a place for holding various data. The storage unit 103 also stores programs used by applications installed on the HMD 150.

[0027] The position / pose estimation unit 104 estimates the position and pose of the camera 101 when the camera image of the camera 101 input to the storage unit 103 is captured. In the present exemplary embodiment, for example, the position / pose estimation unit 104 uses techniques for estimating the position and pose of the camera itself (such as simultaneous localization and mapping (SLAM), etc.) to estimate the position and pose. The position / pose estimation unit 104 may be able to estimate the position of the camera 101 itself through visual SLAM for estimating the position of the camera itself from the video of the camera, or through light detection and ranging (Lidar) SLAM using a laser. The position / pose estimation unit 104 may be able to estimate the position of the camera 101 itself through depth SLAM using a time-of-flight (ToF) sensor.

[0028] For example, by using Lidar SLAM, even in the dark, the position / pose estimation unit 104 estimates the position and pose of the HMD 150, and thus a virtual screen can be displayed even in the dark according to the positional relationship between the mobile terminal and the HMD 150.

[0029] The display 105 is a display unit that displays the composite image generated by the rendering unit 106. The display 105 is placed at a position where the user's both eyes can see the display 105 and is near the both eyes.

[0030] The rendering unit 106 is a graphics processing unit (GPU). The rendering unit 106 renders CG data on the captured image stored in the storage unit 103 to generate a composite image. The rendering unit 106 also renders information such as a line (ray) of a virtual light ray that is to be displayed on the display 105 and indicates the indicated position.

[0031] The read-only memory (ROM) 107 is an electrically erasable and recordable non-volatile memory. The ROM 107 stores various programs such as programs for the entire operation of the HMD 150 and processing programs, and the control unit 102 executes the various programs.

[0032] For example, the communication unit 108 includes an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller for transmitting data to and receiving data from the mobile terminal 151. The communication unit 108 outputs a modulated wireless signal from the antenna and demodulates the wireless signal received by the antenna, thereby implementing short-range wireless communication according to the Institute of Electrical and Electronics Engineers (IEEE) 802.15 standard (so-called ). The communication unit 108 can perform wired communication using a Universal Serial Bus (USB) cable (USB cable, registered trademark) or wireless communication using Wi-Fi (registered trademark). Similarly, the communication unit 108 can also transmit data to and receive data from the mobile terminal 152 and the mobile terminal 153, and can further transmit data to and receive data from other devices.

[0033] The inertial measurement unit 109 is a sensor for detecting the position or posture of the HMD 150. Then, the inertial measurement unit 109 acquires position information or posture information related to the user (the user wearing the HMD 150) corresponding to the position or posture of the HMD 150. The inertial measurement unit 109 includes an inertial measurement unit (IMU) composed of inertial sensors such as an acceleration sensor or an angular acceleration sensor. The inertial measurement unit 109 is used to acquire position information or posture information related to the user, and the control unit 102 acquires position information or posture information related to the user from the inertial measurement unit 109. The inertial measurement unit 109 may be capable of detecting only posture information, or may be capable of detecting only position information, or may be capable of detecting both posture information and position information. More specifically, the inertial measurement unit 109 may be capable of detecting at least one of posture information and position information. The inertial measurement unit 109 may include a geomagnetic sensor, which is a sensor for detecting the direction angle of the HMD 150. The control unit 102 acquires information related to the direction angle of the HMD 150 from the geomagnetic sensor.

[0034] The camera 101, the control unit 102, the storage unit 103, the position / posture estimation unit 104, the display 105, the rendering unit 106, the ROM 107, the communication unit 108, and the inertial measurement unit 109 are connected to the internal bus 110. The components connected to the internal bus 110 exchange data with each other via the internal bus 110.

[0035] The HMD 150 may be connected to a PC that mainly performs image processing in a wired or wireless manner, and the PC may have at least some of the above functions.

[0036] The output unit may be set in such a way that an output unit (not shown) outputs sound, vibration, and light. In particular, when the position of the virtual screen changes, vibration may be sent to the user. Vibration is sent to the user so that the user can feel that the user moves the virtual screen.

[0037] <Internal Structure of the Mobile Terminal>

[0038] Reference Figure 1A , the internal structure of the mobile terminal 151 is described. The mobile terminals 152 and 153 have a structure similar to that of the mobile terminal 151, and will not be described again.

[0039] The display 120 is a display unit that displays a screen corresponding to the application selected in the mobile terminal 151.

[0040] The control unit 121 is a CPU that controls the components of the mobile terminal 151.

[0041] The storage unit 122 is a memory that holds various data. The storage unit 122 also stores programs used by applications installed on the mobile terminal 151.

[0042] The position detection unit 123 uses the Global Positioning System (GPS) or the inertial measurement unit (IMU) as described above to detect the position of the mobile terminal 151. The position / pose estimation unit 104 of the HMD 150 can detect the position of the mobile terminal 151 from the captured image.

[0043] The communication unit 124 has a structure similar to that of the communication unit 108, and sends data to the HMD 150 and receives data from the HMD 150.

[0044] The ROM 125 is an electrically erasable and recordable non-volatile memory. The ROM 125 stores various programs such as programs for the overall operation of the mobile terminal 151 and processing programs, and the control unit 121 executes various programs.

[0045] The display 120, the control unit 121, the storage unit 122, the position detection unit 123, the communication unit 124, and the ROM 125 are connected to the internal bus 126. The components connected to the internal bus 126 exchange data with each other via the internal bus 126.

[0046] Figure 1B Shown in Figure 1A is a schematic diagram showing a state in which the HMD 150 worn on the user's head and the mobile terminals 151, 152, and 153 placed on the table are connected together in the structure. The mobile terminals 152 and 153 have a hardware structure similar to that of the mobile terminal 151.

[0047] <Example of the screen of the head-mounted display>

[0048] Figure 2A , Figure 2B and Figure 2C are diagrams showing a scenario where the mobile terminals 151, 152, and 153 are connected to the HMD 150. Figure 2A , Figure 2B and Figure 2C show a state in which the display 105 of the HMD 150 displays the applications of the HMD 150 and the applications of the mobile terminals 151, 152, and 153.

[0049] Figure 2AShows the display state of the display 105 in a state where the HMD 150 is activated and the video of the real space is visible. The composite image 205 is the video displayed on the display 105 and represents the space of the field of view visible to the user wearing the HMD 150. The tree 202 and the table 203 are real objects. The hand 201 is the hand of the user, and the hand of the user wearing the HMD 150 is visible. The virtual object 204 is a virtual line (line) emitted from the user's hand. The user designates and operates an object using the end of the line 204.

[0050] Figure 2B Shows the display state of the display 105 in a state where the mobile terminals 151, 152, and 153 are Figure 2A connected to the HMD 150 from the state in and the state immediately before starting the applications of the mobile terminals 151, 152, and 153. The virtual object 206 is information related to the list of applications included in the HMD 150 and can be displayed when the HMD 150 is activated. The virtual objects 207, 208, and 209 are the lists of applications included in the respective connected devices. The user uses the line 204 to select and determine the application to be started.

[0051] Figure 2C Shows the display state of the display 105 in a state where an application is started from the state in Figure 2B and the user operates the application. After starting the application, the user can freely operate the application. This example shows the operation of the user using the line 204 to operate the application.

[0052] Figure 3A and Figure 3B Shows a data list for managing the positional relationship between the HMD 150 and the mobile terminals 151, 152, and 153 according to this exemplary embodiment and determining the placement location of the started application (APP).

[0053] Figure 3A Shows information related to the position of the HMD 150, information related to the angle and elevation angle indicating where the HMD 150 is pointing, and position information indicating where the mobile terminals 151, 152, and 153 are. This information is updated in response to the movement or change in posture of the HMD 150 and is similarly updated in response to the movement of the mobile terminals 151, 152, and 153. Figure 3A In each column of the mobile terminals 151, 152, and 153 in, is registered in response to the connection of the mobile terminal to the HMD 150 and is deleted in response to the disconnection of the mobile terminal from the HMD 150.

[0054] Figure 3BShows the applications operating in the HMD 150 and the mobile terminals 151, 152, and 153, and the placement positions of the respective applications in the space of the HMD 150. When the user moves an application, the placement position of the moved application is updated. Figure 3B Each column of the operating applications in is registered in response to the start of the application and deleted in response to the end of the application.

[0055] Figure 3A and Figure 3B The origin of the coordinates of the position in can be at any position. For example, the position where the HMD 150 is started can be fixed as the origin. This is applicable as long as the coordinate axes of the position coordinates of the mobile terminals 151, 152, and 153 are consistent with the coordinate axes of the HMD 150.

[0056] Figure 4A , Figure 4B , Figure 4C , Figure 4D and Figure 4E Shows the virtual screen displayed on the display 105 of the HMD 150. Figure 4A , Figure 4B , Figure 4C , Figure 4D and Figure 4E Shows the state of moving the virtual screen by moving the mobile terminals 151, 152, and 153.

[0057] Figure 4A Shows the same state as Figure 2C .

[0058] Figure 4B Shows the display state of the virtual screen when the mobile terminal 153 moves from the state in Figure 4A to the left side of the mobile terminal 151. The virtual screen 401 is a virtual screen group related to the applications including the applications (APP) G and APP H operating in the mobile terminal 153. When the mobile terminal 153 moves, the virtual screen group 401 moves to the left side of APP A, which is in the leftmost position among the applications operating in the mobile terminal 151. More specifically, in response to the movement of the mobile terminal 153, the control unit 102 of the HMD 150 controls so that the virtual screen related to the mobile terminal 153 moves based on the positional relationship between the mobile terminal 151 and the mobile terminal 153.

[0059] In this case, according to the movement of the mobile terminal, the position of the virtual screen is changed in such a way that the virtual screen approaches the moved mobile terminal. The virtual screen can be moved in such a way that when viewed by the user, the virtual screen moves together with the movement of the mobile terminal. After the movement of the mobile terminal is completed and at the timing when the mobile terminal is stationary for a predetermined time, the position of the virtual screen can be made closer to the mobile terminal. For example, in the case of providing a view in such a way that when viewed by the user, the virtual screen moves together with the movement of the mobile terminal, based on the position of the mobile terminal in the captured image for each frame, the position of the virtual screen is determined in such a way that it is closer to the mobile terminal than the position of the virtual screen in the previous frame.

[0060] Figure 4C Shows when the mobile terminal 151 moves from Figure 4B The state shown to the right of the mobile terminal 152. The virtual screen 402 is a virtual screen group related to the applications including APP A, APP B, and APP C operating in the mobile terminal 151. In response to the movement of the mobile terminal 151, the virtual screen group 402 moves to the right of APP E, which is in the rightmost position among the applications operating in the mobile terminal 152. More specifically, in response to the movement of the mobile terminal 151, the control unit 102 of the HMD 150 controls so that the virtual screen related to the mobile terminal 151 moves based on the positional relationship between the mobile terminal 151 and the mobile terminal 152.

[0061] Figure 4D Shows when the mobile terminal 152 moves from Figure 4C The state shown to the immediate right of the mobile terminal 153. The virtual screen 403 is a virtual screen group related to the applications including APP D and APP E operating in the mobile terminal 152. In response to the movement of the mobile terminal 152, the virtual screen group 403 moves to the right of the virtual screen group 401. To the right of the virtual screen group 401, there is a virtual screen group 404 related to the applications including APP 1, APP 2, and APP 3 operating in the HMD 150. Therefore, the virtual screen group 404 is placed centered on the center line 405 of the HMD 150, and the virtual screen groups 401, 402, and 403 are placed so as not to overlap with the virtual screen group 404. This operation is performed under the assumption that the HMD 150 is placed on the center line 405 between the mobile terminal 152 and the mobile terminal 151.

[0062] When displaying virtual screens of respective mobile terminals among a plurality of mobile terminals, a mode for organizing the virtual screens can be provided. In this case, according to the positional relationship between the mobile terminals in the real space determined by the user during a predetermined time period, respective virtual screen groups can be displayed as clusters of virtual screens related to the corresponding mobile terminals. For example, the user changes the current mode to a mode for organizing virtual screens, and moves the positions of the mobile terminals in the order of the state in Figure 4A the state in Figure 4B the state in Figure 4C . The user gives an instruction to complete the mode for organizing virtual screens. When the control unit 102 receives the instruction to complete the mode for organizing virtual screens, the control unit 102 organizes the virtual screens to achieve the display of the virtual screens in the state in Figure 4C . In the case of organizing the virtual screens as described above, the virtual screens do not move according to the movement of each mobile terminal among the plurality of mobile terminals, and the display positions of respective clusters of virtual screens are determined according to the positional relationship between the mobile terminals at a predetermined time.

[0063] The applications operating in the HMD 150 can be considered different from the applications operating in the mobile terminals 151, 152, and 153. For example, when the mobile terminal 151 moves from the state in Figure 4C to the right side immediately adjacent to the mobile terminal 153, the virtual screen groups 401 and 402 may not move, and even when the virtual screen group 403 overlaps with the virtual screen group 404, the virtual screen group 403 can be placed on the right side of the virtual screen group 401.

[0064] Figure 4E Shows the display state of the display 105 when the mobile terminal 151 moves from the state in Figure 4D to the rear of the mobile terminal 152. The virtual screen group 402 operating in the mobile terminal 151 moves to the rear of the virtual screen group 403 operating in the mobile terminal 152.

[0065] In the Figure 4A , Figure 4B , Figure 4C , Figure 4D , Figure 4E examples, the virtual screen groups move without changing the positional relationship between the virtual screens in the virtual screen groups. Alternatively, for example, when the virtual screen groups move, the virtual screens in the virtual screen groups can be placed in a row in the depth direction, or can be placed in a row in the horizontal direction. When the virtual screens are placed in a row, the applications may overlap each other.

[0066] In the case where applications overlap each other, the front / back relationship between the applications before movement can be maintained, or the application that the user's line of sight last pointed to can be placed in the front. The placement of applications can be based on, for example, the priority of pre-determined applications, such that the table application has the highest priority and the text application has the second highest priority. The placement of applications can be determined for the colors of the applications. The front / back relationship between applications indicates the following relationship, which indicates which virtual screen is in the front position (the position near the user) and which virtual screen is in the back position (the position far from the user).

[0067] <Processing of placing virtual screens>

[0068] Figure 5A And 5B is a flowchart showing details of the processing according to this exemplary embodiment. Figure 5A is a flowchart of the HMD 150. Figure 5B is a flowchart of the mobile terminals 151, 152, and 153.

[0069] The startup of the HMD 150 and the mobile terminals 151, 152, and 153 according to this exemplary embodiment is completed.

[0070] Refer to Figure 5A , and describe the processing of the HMD 150.

[0071] In step S501, the control unit 102 controls the camera 101 to acquire a captured image (a stereoscopic camera image) and stores the captured image in the storage unit 103, and the processing proceeds to step S502.

[0072] In step S502, the control unit 102 detects the user's hand from the stereoscopic camera image stored in step S501 and also monitors the position indicated by the line 204, thereby determining information related to what instruction the user gives. Then, the processing proceeds to step S503. The user's hand can be detected by performing image recognition on the stereoscopic image data of the stereoscopic camera image. The control unit 102 can compare the detected position of the user's hand with the past position of the user's hand and determine information related to whether the user is trying to move the device or whether the user presses the confirmation button. The control unit 102 can determine the object of the instruction based on the position indicated by the line 204.

[0073] In step S503, the control unit 102 determines whether the user gives an instruction to connect to a device (mobile terminal) in step S502. When the control unit 102 determines that an instruction to connect to a device is given (Yes in step S503), the process proceeds to step S504. When it is determined to be No (No in step S503), the process proceeds to step S505. The instruction to connect to a device is determined by pre-displaying candidates of devices to be connected and a connection button and detecting that the connection button is pressed by the user via the line 204. The communication unit 108 can automatically detect devices to which the HMD 150 can be connected and can detect that the user presses a permission button.

[0074] In step S504, the control unit 102 controls the communication unit 108 to connect to a device, and the process proceeds to step S505. The control unit 102 registers the device to which the HMD 150 is connected in the data list in Figure 3A . When the HMD 150 is connected to a device, the control unit 102 may display a list of applications that can be launched, such as the virtual objects 207, 208, and 209 in Figure 2B , on the display 105.

[0075] In step S505, the control unit 102 determines whether the user gives an instruction to disconnect from the device in step S502. When the control unit 102 determines that an instruction to disconnect from the device is given (Yes in step S505), the process proceeds to step S506. When it is determined to be No (No in step S505), the process proceeds to step S507. The instruction to disconnect from the device is determined by detecting that the user presses a device disconnection button.

[0076] In step S506, the control unit 102 controls the communication unit 108 to disconnect from the device for which the device disconnection instruction is given, and the process proceeds to step S507. The control unit 102 deletes the device from which the HMD 150 is disconnected from the data list in Figure 3A .

[0077] In step S507, the control unit 102 sets the device count number N to 0, and the process proceeds to step S508.

[0078] In step S508, the control unit 102 compares the number of devices to which the HMD 150 is connected with the device count number N, and when N is less than the number of devices to which the HMD 150 is connected (Yes in step S508), the process proceeds to step S509. When it is determined to be No (No in step S508), the process proceeds to step S515.

[0079] In step S509, the control unit 102 controls the communication unit 108 to obtain position information related to the device to which the HMD 150 is connected, and the process proceeds to step S510.

[0080] In step S510, the control unit 102 determines whether the user has given an instruction to start an application to any of the devices to which the HMD 150 is connected in step S502. When the control unit 102 determines that an instruction to start an application has been given (Yes in step S510), the process proceeds to step S511. When it is determined to be No (No in step S510), the process proceeds to step S512.

[0081] In step S511, the control unit 102 controls the communication unit 108 to send an instruction to start an application to the device to which the HMD 150 is connected, and the process proceeds to step S512. The control unit 102 registers the application for which the start instruction is given in the operation application column of the corresponding device in the Figure 3B data list therein.

[0082] In step S512, the control unit 102 determines whether the user has given an instruction to end an application to any device in step S502. When the control unit 102 determines that an instruction to end an application has been given (Yes in step S512), the process proceeds to step S513. When it is determined to be No (No in step S512), the process proceeds to step S514.

[0083] In step S513, the control unit 102 controls the communication unit 108 to send an instruction to end an application to the device to which the HMD 150 is connected, and the process proceeds to step S514. The control unit 102 deletes the application for which the end instruction is given from the operation application column of the corresponding device in the Figure 3B data list therein.

[0084] In step S514, the control unit 102 increments the device count number N by 1, and the process returns to step S508.

[0085] In step S515, the control unit 102 controls the position / pose estimation unit 104 to obtain position / pose information related to the HMD 150 held in the storage unit 103, and the process proceeds to step S516. The position / pose information is information related to the Figure 3A position and pose therein. The pose can be an angle and an elevation angle.

[0086] In step S516, the control unit 102 determines whether the user gives an instruction to start an application to the HMD 150 in step S502. When the control unit 102 determines that an instruction to start an application has been given (Yes in step S516), the process proceeds to step S517. When it is determined that the answer is no (No in step S516), the process proceeds to step S518.

[0087] In step S517, the control unit 102 starts the application, and the process proceeds to step S518. The control unit 102 starts the application by reading the program of the corresponding application from the storage unit 103.

[0088] In step S518, the control unit 102 determines whether the user gives an instruction to end the application to the HMD 150 in step S502. When the control unit 102 determines that an instruction to end the application has been given (Yes in step S518), the process proceeds to step S519. When it is determined that the answer is no (No in step S518), the process proceeds to step S520.

[0089] In step S519, the control unit 102 ends the application, and the process proceeds to step S520.

[0090] In step S520, the control unit 102 determines whether an instruction to operate the application is given in step S502. When the control unit 102 determines that an instruction to operate the application has been given (Yes in step S520), the process proceeds to step S521. When it is determined that the answer is no (No in step S520), the process proceeds to step S522.

[0091] In step S521, the control unit 102 operates the application according to the instruction to operate the application obtained in step S502, and the process proceeds to step S522. When the operation of the application is an operation of the application of the HMD 150, the control unit 102 controls the application. When the operation of the application is an operation of the application of the device, the control unit 102 controls the communication unit 108 to send an instruction to operate the application to the device.

[0092] In step S522, the control unit 102 updates the application, and the process proceeds to step S523. When the application is operated in step S521, the control unit 102 updates the image of the application including the operation result and stores the image in the storage unit 103. In the device, after the control unit 102 controls the communication unit 108, the device receives the image of the application and stores the image in the storage unit 103. When the operation of the application involves the movement of the application, the control unit 102 updates Figure 3BThe position of the corresponding application in the data list.

[0093] In step S523, based on the position information related to the device obtained in step S509, the control unit 102 determines whether any device has moved. When the control unit 102 determines that any device has moved (Yes in step S523), the process proceeds to step S524. When it is determined that no device has moved (No in step S523), the process proceeds to step S525.

[0094] In step S524, the control unit 102 updates the position of the application according to the movement of the device, and the process proceeds to step S525. The update of the position of the application is achieved by updating Figure 3B the data list in. For example, when the mobile terminal 153 moves to the left of the mobile terminal 151, Figure 4B the virtual screen related to the mobile terminal 153 in moves in the following manner: the virtual screen related to the mobile terminal 153 is to the left of the virtual screen at the leftmost position in the virtual screen related to the mobile terminal 151. More specifically, the APP G and APP H of the mobile terminal 153 move in the following manner: APP G and APP H are to the left of the application at the leftmost position among the APP A, APP B, and APP C of the mobile terminal 151. Although Figure 3B the size of the application screen is omitted from the data list in, when the size is also considered as a factor, the applications are moved with high precision in such a way that, for example, the applications do not overlap each other. That is, the control of the movement of the applications is performed by monitoring and updating Figure 3B the data list in.

[0095] In step S525, the drawing unit 106 reads the image of the application held in the storage unit 103, and draws the read image at the position corresponding to the data list in the captured image. In addition, the drawing unit 106 draws the line 204 at the position based on the position of the user's hand, and the display 105 displays the composite image of the captured image and the application and the line 204. Then, the process proceeds to step S526.

[0096] In step S526, the control unit 102 determines whether to end the process of this exemplary embodiment. When the control unit 102 determines to end the process (Yes in step S526), the process ends. When it is determined that the process is not to end (No in step S526), the process returns to step S501. The determination of whether to end the process is made based on whether an end instruction from the user is obtained in step S502.

[0097] <Processing of the mobile terminal>

[0098] Describe the processing of mobile terminals 151, 152, and 153. The mobile terminals 151, 152, and 153 perform similar processing, and Figure 5B Illustrate the processing of any one of the mobile terminals 151, 152, and 153.

[0099] In step S531, the control unit 121 controls the communication unit 124 to obtain an instruction from the user, and the processing proceeds to step S532. The instruction from the user includes the instruction sent from the HMD 150 in step S511, step S513, or step S521.

[0100] In step S532, the control unit 121 determines whether the instruction from the user obtained in step S531 is an instruction to connect to the HMD 150. When the control unit 121 determines that the instruction from the user is an instruction to connect to the HMD 150 (Yes in step S532), the processing proceeds to step S533. When it is determined to be No (No in step S532), the processing proceeds to step S534. This is based on the sending of the connection instruction in the connection processing of step S504 of the HMD 150.

[0101] In step S533, the control unit 121 controls the communication unit 124 to connect to the HMD 150, and the processing proceeds to step S534. The control unit 121 controls the communication unit 124 to send information related to a list of applications that can be launched, such as the virtual objects 207, 208, and 209 as Figure 2B shown, to the HMD 150.

[0102] In step S534, the control unit 121 determines whether the instruction from the user obtained in step S531 is an instruction to disconnect from the HMD 150. When the control unit 121 determines that the instruction from the user is an instruction to disconnect from the HMD 150 (Yes in step S534), the processing proceeds to step S535. When it is determined to be No (No in step S534), the processing proceeds to step S536. This is based on the sending of the disconnection instruction in the disconnection processing of step S506 of the HMD 150.

[0103] In step S535, the control unit 121 controls the communication unit 124 to disconnect from the HMD 150, and the processing proceeds to step S536.

[0104] In step S536, the control unit 121 determines whether the instruction from the user obtained in step S531 is an instruction to start an application. When the control unit 121 determines that the instruction from the user is an instruction to start an application (Yes in step S536), the process proceeds to step S537. When it is determined to be No (No in step S536), the process proceeds to step S538. This is based on the sending of an application start instruction in the application start process of step S511 of the HMD 150.

[0105] In step S537, the control unit 121 starts the application, and the process proceeds to step S538. The application is started by the control unit 121 reading the program of the application from the storage unit 122.

[0106] In step S538, the control unit 121 determines whether the instruction from the user obtained in step S531 is an instruction to end the application. When the control unit 121 determines that the instruction from the user is an instruction to end the application (Yes in step S538), the process proceeds to step S539. When it is determined to be No (No in step S538), the process proceeds to step S540. This is based on the sending of an application end instruction in the application end process of step S513 of the HMD 150.

[0107] In step S539, the control unit 121 ends the application, and the process proceeds to step S540.

[0108] In step S540, the control unit 121 determines whether the instruction from the user obtained in step S531 is an instruction to operate the application. When the control unit 121 determines that the instruction from the user is an instruction to operate the application (Yes in step S540), the process proceeds to step S541. When it is determined to be No (No in step S540), the process proceeds to step S542. This is based on the sending of an application operation instruction in the application operation process of step S521 of the HMD 150.

[0109] In step S541, the control unit 121 operates the application, and the process proceeds to step S542. The control unit 121 stores the image of the operated application in the storage unit 122.

[0110] In step S542, the control unit 121 reads the image of the application held in step S541, and the display 120 displays the read image. Then, the process proceeds to step S543.

[0111] In step S543, the control unit 121 determines whether the mobile terminal is currently connected to the HMD 150. When the control unit 121 determines that the mobile terminal is currently connected to the HMD 150 (Yes in step S543), the process proceeds to step S544. When it is determined to be No (No in step S543), the process proceeds to step S546.

[0112] In step S544, the control unit 121 controls the position detection unit 123 to obtain position information and controls the communication unit 124 to send the position information. Then, the process proceeds to step S545.

[0113] In step S545, the control unit 121 reads the image of the latest operation application from the storage unit 122 and controls the communication unit 124 to send the image. Then, the process proceeds to step S546.

[0114] In step S546, the control unit 121 determines whether to end the processing of the device according to this exemplary embodiment. When the control unit 121 determines to end the processing (Yes in step S546), the processing ends. When it is determined to be No (No in step S546), the process returns to step S531. The determination of whether to end the processing is based on whether an end instruction from the user is obtained in step S531.

[0115] <How to display the virtual screen>

[0116] Figure 6 Shows the display state of the display 105 when the direction of the HMD 150 changes from Figure 2C the state in the right direction. In this case, the positions of the applications operating in the mobile terminals 151, 152, and 153 do not move, and only the APP 1, APP 2, and APP 3 (as the applications operating in the HMD 150) move.

[0117] This is the case where, in step S524, Figure 4A the position and elevation angle of the HMD 150 in do not change and only the angle of the HMD 150 changes, and the positions of the mobile terminals 151, 152, and 153 do not change. In this case, it is possible to only make Figure 3B the APP 1, APP 2, and APP 3 in the operation application bar of the HMD 150 in move by an amount corresponding to the angle change.

[0118] In this exemplary embodiment, when any one of the mobile terminals 151, 152, and 153 is moved outside the field of view of the HMD 150, the application operating in the moved device can be moved to the end of the screen. For example, outside the field of view of the HMD 150 is outside the imaging range of the camera 101, and can be outside the imaging range of the camera that acquires the video the user wants to view, or can be outside the imaging range of the camera that acquires the video for obtaining the position and orientation of the HMD 150. When the mobile terminal is moved outside the field of view of the HMD 150, the virtual screen can be moved behind other virtual screens (to a position farther from the user), or can be hidden. The hidden virtual screen can be displayed in response to the corresponding device returning to the field of view. Hiding can be elimination, or can be minimization.

[0119] In this exemplary embodiment, when the screen of any one of the mobile terminals 151, 152, and 153 is face down, that is, when the display unit of any one of the mobile terminals 151, 152, and 153 is hidden, the application operating in that mobile terminal can be eliminated. The determination of whether the screen is face down can be made by determining the orientation of the mobile terminal based on the captured image. In response to the screen being face up, that is, in response to the display unit being visible, the eliminated application is displayed again.

[0120] When any one of the mobile terminals 151, 152, and 153 is foldable, in response to the mobile terminal being folded, the application operating in that mobile terminal can be eliminated.

[0121] In the second exemplary embodiment, in addition to the first exemplary embodiment, a method is also described for further determining the placement of the application operating in the device based on whether the user wearing the HMD 150 is holding the device in their hand or wearing the device.

[0122] Figure 7 Shows the display state of the display 105 when the user holds the mobile terminal 153 in their hand from the state in Figure 4D The virtual screen group 401 operating in the mobile terminal 153 is moved to the front, and the virtual screen group 404 operating in the HMD 150 that was previously placed at that location is moved behind the virtual screen group 401. More specifically, compared to the applications operating in the head-mounted display, priority is given to the applications related to the device held in the user's hand. Moving the application to the front is an example of a method for giving priority to the application, and other methods for giving priority to the application can be adopted.

[0123] When moving the mobile terminal 153 from as Figure 7In the case where the state in [reference] moves backward or the mobile terminal 153 is placed on the table 203, the virtual screen group 404 that is already in the rear can move to the front.

[0124] In response to the direction of the HMD 150 changing to the right direction as shown in Figure 6 the applications operating in the mobile terminal 153 that the user holds in their hand or wears can also move together with the applications operating in the HMD 150.

[0125] Figure 8A and Figure 8B are flowcharts showing details of the processing of the HMD 150 according to this exemplary embodiment. Figure 8C are flowcharts showing details of the processing of the mobile terminals 151, 152, and 153 according to this exemplary embodiment.

[0126] Referring to Figure 8A and Figure 8B the details of the processing of the HMD 150 according to this exemplary embodiment will be described below. In the processing flow of the HMD 150, steps S801 to S823 are similar to steps S501 to S523 in Figure 5A and thus will not be described again. Steps S826 to S828 are similar to steps S524 to S526 in Figure 5A and thus will not be described again. Only steps S824 and S825 that are different from Figure 5A will be described.

[0127] In step S824, the control unit 102 determines whether the device (mobile terminal) that is moving is a device worn by the user. In the case where the control unit 102 determines that the device that is moving is a device worn by the user (Yes in step S824), the processing proceeds to step S825. In the case where it is determined to be No (No in step S824), the processing proceeds to step S826. The determination of whether the device that is moving is a device worn by the user can be performed, for example, by detecting the device from the captured image obtained in step S801 and determining that the user wears the device, or can be performed based on information indicating that the distance between the position information related to the device and the position information related to the HMD 150 is less than a predetermined distance. In the case where the difference between the change in the position or orientation of the HMD 150 and the change in the position or orientation of the device is less than a threshold value, it can be determined that the user holds the device. Possible examples of the case where the user wears the device include the case where the user holds the device, the case where the user wears the device around their neck, and the case where the user has the device in their clothes pocket.

[0128] In step S825, the control unit 102 updates the position of the application, and the process proceeds to step S827. The control unit 102 checks the positional relationship between the application operating in the device worn by the user determined in step S825 and the application operating in the HMD 150. When the positional relationship between these applications causes the application to be in the corresponding position, the control unit 102 updates in such a way that the application operating in the determined device worn by the user moves forward. Figure 3B the data list in

[0129] The processing flow of this device is similar to that in Figure 5B and will not be described again.

[0130] When the user holds the device in their hand and the direction of the HMD 150 changes to the right in Figure 6 the virtual picture group related to the device held by the user can also move together. The determination of the movement can also be made in step S826.

[0131] As described above, according to the present invention, the operability of the user of the information processing device with respect to the virtual picture is improved.

[0132] The present invention can also be implemented by performing the following processing. Software (program) for implementing the functions of the above-described exemplary embodiments is provided to a system or device via a network or various storage media, and the computer (or control unit or microprocessing unit (MPU)) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0133] Although the present invention has been described in detail based on the desired exemplary embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope without departing from the gist of the present invention are also included in the present invention. Some of the above-described exemplary embodiments can be appropriately combined.

[0134] Other Embodiments

[0135] Embodiments of the present invention can also be implemented by the following method, that is, a software (computer program product including computer programs / instructions) for executing the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (central processing unit (central processor) microprocessing unit (microprocessor)) of the system or device reads and executes the computer programs / instructions.

[0136] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation so as to cover all such modifications as well as equivalent structures and functions.

Claims

1. An information processing apparatus that communicates with a mobile terminal, the information processing apparatus comprising: a display control unit configured to control a display unit to display a virtual space in which a virtual image related to the mobile terminal is placed in a real space; an instruction acquisition unit configured to acquire an instruction for placing a virtual image related to the mobile terminal in the virtual space, wherein, when movement of the mobile terminal is detected, the display control unit changes a position of the virtual image related to the mobile terminal placed in the virtual space to a position corresponding to the position of the mobile terminal in which the movement is detected.

2. The information processing apparatus according to claim 1, wherein, In a case where there are a plurality of mobile terminals, in response to movement of at least any one of the plurality of mobile terminals, the display control unit changes positions of virtual images respectively related to different mobile terminals among the plurality of mobile terminals according to a positional relationship between the plurality of mobile terminals.

3. The information processing apparatus according to claim 2, wherein In a case where there are a plurality of mobile terminals, in response to movement of at least any one of the plurality of mobile terminals, the display control unit changes positions of virtual images related to the plurality of mobile terminals to display the virtual images arranged in clusters according to a positional relationship between the plurality of mobile terminals, where each cluster is related to a different mobile terminal among the plurality of mobile terminals.

4. The information processing apparatus according to claim 3, wherein, In a case where there are a plurality of virtual images related to a predetermined mobile terminal among the plurality of mobile terminals, the display control unit displays the plurality of virtual images related to the predetermined mobile terminal in an overlapping manner.

5. The information processing apparatus according to any one of claims 1 to 4, wherein, In response to movement of the mobile terminal, the display control unit changes positions of the virtual image related to the mobile terminal and a virtual image related to the information processing apparatus.

6. The information processing apparatus according to any one of claims 2 to 4, wherein, The display control unit performs control to display a virtual image related to a mobile terminal on the right side on the right side and a virtual image related to a mobile terminal on the left side on the left side.

7. The information processing apparatus according to any one of claims 1 to 4, wherein, The display control unit performs control to display a virtual image related to the information processing apparatus in such a manner that the virtual image moves according to a change in the posture in a case where the posture of the display unit changes.

8. The information processing apparatus according to any one of claims 1 to 4, wherein, The display control unit performs control to hide or minimize a virtual image related to a mobile terminal that has moved outside a shooting range of a captured image in a case where the mobile terminal has moved outside the shooting range.

9. The information processing apparatus according to any one of claims 1 to 4, wherein, The display control unit performs control to display a virtual image related to a mobile terminal worn by a user in front in a case where the user wears the mobile terminal.

10. The information processing apparatus according to any one of claims 1 to 4, wherein, The display control unit performs control to display a virtual image related to a mobile terminal worn by a user in such a manner that the virtual image moves according to a change in the posture of the display unit in a case where the user wears the mobile terminal.

11. The information processing apparatus according to any one of claims 1 to 4, wherein, In a case where the position of the mobile terminal moves backward, the display control unit displays a virtual image related to the information processing apparatus at a position closer to the user than a virtual image related to the mobile terminal.

12. The information processing apparatus according to any one of claims 1 to 4, Among them, wherein the mobile terminal is a device including a display, and wherein the display control unit controls so that when the mobile terminal is placed face down such that the display of the mobile terminal is hidden, a virtual screen related to the mobile terminal is hidden or minimized.

13. The information processing apparatus according to claim 12, wherein, The display control unit controls so that when the mobile terminal is placed such that the display of the mobile terminal is visible from a state where the mobile terminal is placed face down, a virtual screen related to the mobile terminal is displayed.

14. The information processing apparatus according to any one of claims 1 to 4, wherein, The display control unit controls so that when an instruction to fix the position of a first virtual screen related to a first mobile terminal is received, even if the first mobile terminal moves, the position of the first virtual screen is not changed to a position corresponding to the position of the first mobile terminal that has detected the movement.

15. The information processing apparatus according to any one of claims 1 to 4, wherein, When the position of the displayed virtual screen changes, the display control unit maintains the front / back relationship of the virtual screen.

16. The information processing apparatus according to any one of claims 1 to 4, further comprising: a line-of-sight acquisition unit configured to acquire the line of sight of a user, and wherein the display control unit controls so that when the position of the displayed virtual screen changes, the virtual screen pointed to by the line of sight of the user among the virtual screens displayed on the display unit is displayed in front of the user.

17. The information processing apparatus according to any one of claims 1 to 4, wherein, The display control unit controls so that when the positions of a plurality of virtual screens related to the mobile terminal change, the plurality of virtual screens are displayed based on the front / back relationship determined between the plurality of virtual screens according to a predetermined priority.

18. The information processing apparatus according to any one of claims 1 to 4, further comprising: an image acquisition unit configured to acquire a captured image, wherein the display control unit displays an image of a mixed reality space obtained by synthesizing a virtual screen related to the mobile terminal and the captured image on the display unit.

19. An information processing apparatus that communicates with a mobile terminal, the information processing apparatus comprising: a display control unit configured to control a display unit to display a virtual space in which a virtual screen related to the mobile terminal is placed in a real space, wherein when an application related to the mobile terminal is started, the display control unit places a virtual screen of the application related to the mobile terminal in the virtual space.

20. The information processing apparatus according to claim 19, wherein, When an application related to the mobile terminal is started, the display control unit places a virtual screen of the application related to the mobile terminal in the front of the display unit.

21. The information processing apparatus according to claim 19, further comprising: an instruction acquisition unit configured to acquire an instruction to change the position of placing a virtual screen related to the mobile terminal; wherein based on the instruction acquired by the instruction acquisition unit, the display control unit changes the position of the virtual screen of the application related to the mobile terminal.

22. The information processing apparatus according to any one of claims 19 to 21, wherein, In the case where the movement of the mobile terminal is detected, the display control unit changes the position of the virtual screen related to the mobile terminal placed in the virtual space to a position corresponding to the position of the mobile terminal where the movement is detected.

23. A control method for an information processing device, the information processing device communicating with a mobile terminal, the control method comprising: a display control step of controlling a display unit to display a virtual space in which a virtual screen related to the mobile terminal is placed in the real space; and an instruction acquisition step of acquiring an instruction for placing a virtual screen related to the mobile terminal in the virtual space, wherein, in the display control step, in the case where the movement of the mobile terminal is detected, the position of the virtual screen related to the mobile terminal placed in the virtual space is changed to a position corresponding to the position of the mobile terminal where the movement is detected.

24. A computer program product comprising a program for causing a computer to execute the steps of the control method for the information processing device according to claim 23.

25. A computer-readable storage medium storing a program for causing a computer to execute the steps of the control method for the information processing device according to claim 23.

26. An information processing system comprising: a mobile terminal; a display unit; a display control device configured to control the display unit to display a virtual space in which a virtual screen related to the mobile terminal is placed in the real space; and an instruction acquisition device configured to acquire an instruction for placing a virtual screen related to the mobile terminal in the virtual space, wherein, in the case where the display control device detects the movement of the mobile terminal, the display control device changes the position of the virtual screen related to the mobile terminal placed in the virtual space to a position corresponding to the position of the mobile terminal where the movement is detected.

Images