Information processor, information processing system, information processing program, and information processing method

Abstract

To provide a more convenient information processor in using an HMD and a portable terminal in combination.SOLUTION: An information processor includes picture acquisition means for acquiring an imaged picture and display control means for performing control so as to display, on the display means, a synthesized picture obtained by synthesizing a virtual screen with the imaged picture. The display control means performs controls so as to display the virtual screen vertically when a portable terminal is vertically seen from a user and display the virtual screen horizontally when the portable terminal is horizontally seen from the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In recent years, there has been a technology that connects a head-mounted display (HMD) to a mobile device such as a smartphone or tablet, and displays the screen of the mobile device on the HMD.

[0003] Patent Document 1 discloses a method for switching whether or not to display the screen of a mobile terminal such as a smartphone or tablet terminal on an HMD depending on the state of the mobile terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6940702 Public Relations Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 does not describe a method for determining the orientation of the screen displayed on the HMD depending on the state of the mobile terminal, and therefore it is not necessarily user-friendly.

[0006] Therefore, an object of the present invention is to provide an information processing device that is easier to use when an HMD and a mobile terminal are used in combination. [Means for solving the problem]

[0007] One aspect of the present invention is an information processing device having an image acquisition means for acquiring a captured image, and a display control means for controlling a display means to display a composite image obtained by combining the captured image with a virtual screen, wherein the display control means controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation from the user's perspective, and controls the display means to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation from the user's perspective. [Effects of the Invention]

[0008] The present invention has an effect of providing an information processing device that is easier to use when an HMD and a mobile terminal are used in combination. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram of a display device and a mobile terminal according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a connection configuration between a display device and a mobile terminal according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing a coordinate system based on a display device at a predetermined time point and the attitude of the display device at the predetermined time point in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating a mobile terminal in a portrait orientation according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating a mobile terminal in a landscape orientation according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing a mobile terminal in a flat state in the first embodiment. [Figure 7] FIG. 3 is a sequence diagram of the display device and the mobile terminal according to the first embodiment. [Figure 8] 4 is a flowchart of the display device according to the first embodiment. [Figure 9] 4 is a flowchart of the mobile terminal according to the first embodiment. [Figure 10] FIG. 10 is a sequence diagram of a display device and a mobile terminal according to a second embodiment. [Figure 11] FIG. 1 is a diagram showing an example of an image viewed by a user in the first embodiment. [Figure 12] FIG. 2 is a diagram showing an example of an image viewed by a user in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) In the present embodiment, an application (hereinafter also referred to as an app) in which an HMD and a mobile terminal are connected and the screen of the mobile terminal is displayed on the HMD will be described. Such an app is, for example, a remote desktop app or a mirroring app downloaded to the HMD.

[0012] Hereinafter, the configuration of the present invention will be described with reference to FIG. 1.

[0013] FIG. 1 is a block diagram of the display device and the mobile terminal in the present embodiment. In the present embodiment, the description will be made assuming an HMD as the display device A100 and a terminal such as a smartphone as the mobile terminal B100.

[0014] <Internal Configuration of HMD> The display device A100 includes a control unit A101, a storage unit A102, a memory A103, an input unit A104, an output unit A105, a sensor A106, and a communication unit A107.

[0015] Although an HMD is described here as an example of an information processing device, the information processing device is not limited to this. For example, part of the processing of the information processing device may be performed by a smartphone, tablet terminal, or PC, and each component of the display device A100 may be included in a smartphone, tablet terminal, or PC. Part of the processing may be performed by a mobile terminal B100 (described later), or part of the processing may be performed by a device other than the mobile terminal B100.

[0016] The control unit A101 is a CPU that controls each component of the display device A100. The control unit A101 is a display control means that, upon acquiring a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG), controls the composite image to be displayed on the display. Note that instead of the control unit A101 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing. Note that the composite image may be generated by the control unit A101, or may be connected to a PC (personal computer) wirelessly or via a wired connection, and the composite image generated by the PC may be acquired. The user can recognize the mixed reality space by viewing such a composite image.

[0017] The memory unit A102 is an electrically erasable and recordable non-volatile memory medium such as an SSD (Solid State Drive) or flash memory, and stores programs executed by the control unit A101, databases, user settings, and the like.

[0018] The memory A103 is, for example, a RAM (Random Access Memory), and is used as a buffer memory for temporarily storing various data, a work area for the control unit A101, and the like.

[0019] The input unit A104 is used to input instructions to the display device A100. The input unit A104 includes, for example, a power button for instructing ON / OFF of the power of the display device A100 and operation buttons for instructing screen transitions. Note that the input unit A104 does not necessarily have to be built into the display device A100. It may also be configured so that input is made via a communication unit A107, which will be described later.

[0020] The output unit A105 displays, for example, a GUI (Graphical User Interface) for interactive operation. It also corresponds to a light-emitting device such as an LED (Light Emitting Diode), an audio output device such as a speaker, and a display unit such as a display. The output unit A105 may be configured to output via a communication unit A107, which will be described later.

[0021] The sensor A106 may be, for example, an imaging sensor that captures surrounding images, or a sensor such as a LiDAR (Light Detection And Ranging) or ToF (Time of Flight) that measures the surrounding conditions. Also, sensors such as an IMU (Inertial Measurement Unit) or a geomagnetic sensor that measure the attitude and position are also included. The imaging sensor is an image acquisition unit that acquires captured images.

[0022] The communication unit A107 is, for example, a USB (Universal Serial Bus) for connecting to other devices, or a connector (RJ45) for connecting to Ethernet, and performs wired communication. The communication unit A107 may also have a communication unit for performing wireless communication with a controller or external device, such as a NIC (Network Interface Card) with a built-in communication IC. Wireless communication may also be performed using Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark).

[0023] <Internal structure of a smartphone> The mobile terminal B100 includes a control unit B101, a storage unit B102, a memory B103, an input unit B104, an output unit B105, a sensor B106, and a communication unit B107.

[0024] Descriptions of the control unit B101, storage unit B102, memory B103, and input unit B104 of the mobile terminal B100 that are the same as the control unit A101, storage unit A102, memory A103, and input unit A104 of the display device A100 will be omitted. Descriptions of the output unit B105, sensor B106, and communication unit B107 of the mobile terminal B100 that are the same as the output unit A105, sensor A106, and communication unit A107 of the display device A100 will be omitted.

[0025] The terminal configuration has been described assuming an HMD as the display device A100 and a smartphone as the mobile terminal B100, but the above-described terminal configuration is merely an example. The display device A100 is not limited to an HMD, but may be a wearable device such as a handheld device, smart glasses, a goggle-type display, or smart contact lenses, and the terminal configuration may be different. Furthermore, the mobile terminal B100 is not limited to a smartphone, but may be a tablet terminal or a PC, and the mobile terminal configuration may also be different. The mobile terminal B100 may also be a smartwatch.

[0026] <Connection configuration between devices> An information processing system according to a first embodiment of the present invention will be described below with reference to Fig. 2. The information processing system in Fig. 2 includes a display device A100 and a mobile terminal B100.

[0027] 2, the system of this embodiment includes a display device A100 and a mobile terminal B100. The display device A100 and the mobile terminal B100 will be described as having the same hardware configuration as the display device A100 and the mobile terminal B100 shown in FIG.

[0028] The display device A100 includes a Wi-Fi communication unit as the communication unit A107, and is capable of connecting to a local network via a router.

[0029] The mobile terminal B100 also has a Wi-Fi communication unit as the communication unit B107 and is configured to be connectable to a local network via a router.

[0030] Also, the display device A100 and the mobile terminal B100 can communicate via a local network and can communicate all data bidirectionally.

[0031] The above-described connection configuration is merely an example, and different connection methods and connection configurations may be used.

[0032] <Method for obtaining the posture of the HMD> Hereinafter, a method for obtaining the posture information of the display device A100 in the present embodiment will be described.

[0033] There are various methods for obtaining the posture information of the display device A100. For example, it may be a method of calculating and obtaining by self-position estimation such as SLAM (Simultaneous Localization and Mapping) using a camera or an acceleration sensor mounted on the display device A100. In addition, it may be a method of measuring and obtaining by arranging a plurality of external sensors and using position tracking technology.

[0034] The above-described method for obtaining the posture information of the display device A100 is merely an example, and different methods for obtaining the posture information of the display device A100 may be used.

[0035] <Method for obtaining the posture of the mobile terminal> Hereinafter, a method for obtaining the posture information of the mobile terminal B100 from the display device A100 in the present embodiment will be described.

[0036] There are various methods for acquiring the attitude information of the mobile terminal B100 from the display device A100, such as a method of calculating and acquiring the attitude information of the mobile terminal B100 by performing image recognition using a camera mounted on the display device A100. Note that a method of acquiring attitude information from the mobile terminal B100 calculated by self-position estimation such as SLAM using a camera or acceleration sensor mounted on the mobile terminal B100 may also be used. Note that a method of calculating and acquiring the attitude information of the mobile terminal B100 by a distance measurement technique using a wireless communication unit such as Bluetooth or UWB (Ultra-Wide Band) mounted on the display device A100 and the mobile terminal B100 may also be used.

[0037] Any method may be used to acquire the attitude information of the mobile terminal B100 from the display device A100, as long as it is suitable for the configurations of the display device A100 and the mobile terminal B100, or for the combination thereof.

[0038] The above-described method of acquiring the orientation information of the mobile terminal B100 from the display device A100 is merely an example, and the method of acquiring the orientation information of the mobile terminal B100 from the display device A100 may be different.

[0039] <Display device coordinate system> Hereinafter, with reference to FIG. 3, a coordinate system based on the attitude of the display device A100 at a predetermined time in this embodiment will be described.

[0040] In this embodiment, a coordinate system is determined based on the orientation of the display device A100 at a predetermined time. Then, the orientation of an image to be displayed on the display device A100 is determined based on the determined coordinate system and the orientation of the mobile terminal B100. The orientation of the display device A100 at a predetermined time is, for example, the orientation of the display device A100 immediately after an app on the display device A100 is launched, or the orientation of the display device A100 when a display reset operation is performed using a button mounted on the display device A100. In other words, this is orientation information of the display device A100 that serves as a reference for determining the display orientation of an image to be displayed on the display device A100. Note that the coordinates may not be determined based on the orientation of the display device A100, but may be determined based on a physical object such as a predetermined device or marker.

[0041] In this embodiment, the orientation of the display device A100 at a predetermined time is used as a reference, and the direction from the front to the back when the user wears the display device A100 is defined as the Z axis. The direction perpendicular to the Z axis, from the user's feet toward the head, is defined as the Y axis. The following description will be given assuming that the direction perpendicular to the Z axis and the Y axis, from the user's left to right, is defined as the X axis. FIG. 3 is a diagram showing a coordinate system based on the display device A100 at a predetermined time and the orientation of the display device A100 at a predetermined time in this embodiment. In FIG. 3, the front direction of the display device corresponds to the negative direction of the Z axis, and the back direction of the display device corresponds to the positive direction of the Z axis.

[0042] The coordinate system based on the attitude of the display device A100 at a specified point in time described above is defined for the purpose of explaining this embodiment, and therefore the coordinate system based on the attitude of the display device A100 at a specified point in time does not need to be defined, or may be different.

[0043] The coordinate system at a predetermined time point may be defined each time a captured image is acquired (each frame), or may be defined each time the attitude of the display device A100 is acquired.

[0044] The coordinate system at a predetermined time point may be updated at regular intervals, or may be updated when the orientation of the display device A100 has changed by more than a threshold value from the orientation at the time the coordinate system was determined.

[0045] <Determining the orientation of a mobile device> Hereinafter, with reference to Figures 4, 5, and 6, a method for determining whether the orientation of the mobile terminal B100 in this embodiment is vertical (portrait orientation), horizontal (landscape orientation), or flat will be described.

[0046] To determine whether the orientation of the mobile terminal B100 is portrait, landscape, or flat, a coordinate system based on the orientation of the display device A100 at a given time and orientation information of the mobile terminal B100 are used. Here, we assume that the mobile terminal B100 is a smartphone, and define the side with the screen as the front and the side opposite the screen as the back. Also, in the case of a mobile terminal B100 with a camera 401 on the front, when viewed so that the front camera 401 is located above the screen, the upper side is defined as the top side, the lower side as the bottom side, the right side as the right side, and the left side as the left side. Also, in the case of a mobile terminal B100 with a button 402 on the front, when viewed so that the button 402 is located below the screen, the upper side is defined as the top side, the lower side as the bottom side, the right side as the right side, and the left side as the left side.

[0047] The coordinate system based on the attitude of the display device A100 at a predetermined time and the method for acquiring attitude information of the mobile terminal B100 from the display device A100 are as described above.

[0048] As shown in FIG. 4, if the bottom surface of the mobile terminal B100 is perpendicular to the Y axis, the mobile terminal B100 is determined to be in a portrait orientation.

[0049] As shown in FIG. 5, if the side surface of the mobile terminal B100 is perpendicular to the Y axis, the mobile terminal B100 is determined to be in a landscape orientation.

[0050] As shown in FIG. 6, if the front of the mobile terminal B100 is perpendicular to the Y axis, the mobile terminal B100 is determined to be in a flat state.

[0051] However, since the mobile terminal B100 is rarely in a state perpendicular to the axis as described above, in such cases, the mobile terminal B100 determines whether it is in portrait orientation, landscape orientation, or flat orientation based on which of the above-mentioned states it is closest to.

[0052] For example, when the mobile terminal B100 is rotated 90 degrees counterclockwise around the Z axis from the portrait orientation shown in Fig. 4, it changes to the landscape orientation shown in Fig. 5. At this time, the center angle is 45 degrees, so 45 degrees is used as the reference angle, and an angle between 0 degrees and 45 degrees is determined to be portrait orientation, and an angle between 45 degrees and 90 degrees is determined to be landscape orientation. Similarly, by using 45 degrees as the reference angle for the X and Y axes, it is possible to determine whether the orientation of the mobile terminal B100 is portrait, landscape, or flat.

[0053] Furthermore, as mentioned above, if there is only one threshold, when the mobile terminal B100 is in an orientation near the threshold, the state of the mobile terminal B100 will repeatedly switch between portrait, landscape, and flat states in a short period of time. Therefore, it is also possible to set two thresholds. For example, by using different thresholds for changing from portrait to landscape and for changing from landscape to portrait, it is possible to prevent the state from switching multiple times in a short period of time.

[0054] The above-mentioned method of determining whether the orientation of the mobile terminal B100 is portrait, landscape, or flat is merely an example, and the method of determining whether the orientation of the mobile terminal B100 is portrait, landscape, or flat may be different.

[0055] Furthermore, the orientation of the mobile terminal B100 does not have to be limited to three states: portrait, landscape, and flat. For example, if the orientation of the mobile terminal B100 is portrait and the top is facing up and the top is facing down have different meanings, the number of states will be three or more. Depending on the system being configured, the number of states may be less than three. Therefore, it is sufficient to define as many states of the mobile terminal B100 as necessary to suit the system being constructed.

[0056] Here, the state of the mobile terminal B100 is defined using a coordinate system based on the attitude of the display device A100 at a given time. However, the state of the mobile terminal B100 may also be defined based on the attitude of the mobile terminal with respect to the direction of gravity, whether the attitude of the mobile terminal is horizontal with respect to the ground, or which side is the bottom side (the side closest to the ground). For example, if the bottom side is vertical with respect to the direction of gravity or the bottom side is on the bottom side, the mobile terminal is defined as being in a portrait orientation. If the right side or left side is vertical with respect to the direction of gravity or the right side or left side is on the bottom side, the mobile terminal is defined as being in a landscape orientation. Also, if the front and back sides are vertical with respect to the direction of gravity or the front or back side is on the bottom side, the mobile terminal is defined as being in a flat state.

[0057] In addition, whether a smartphone is in portrait, landscape, or flat orientation may be defined based on which direction any of its components, such as the camera or buttons, is perpendicular or parallel to, or in which position relative to which surface.

[0058] <How to determine image display direction> A method for determining the image display direction in this embodiment will be described below.

[0059] As described above, in this embodiment, the display direction of an image displayed on the display device A100 is determined based on the attitude of the display device A100 at a given time, a coordinate system based on that attitude, and the attitude of the mobile terminal B100.

[0060] When the mobile terminal B100 is in a portrait orientation, the image display orientation is determined so that the mobile terminal B100 faces forward and is also in a portrait orientation relative to the orientation of the display device A100 at a given point in time.

[0061] When the mobile terminal B100 is in a landscape orientation, the image display direction is determined so that the mobile terminal B100 faces forward and is landscape orientation relative to the orientation of the display device A100 at a given point in time.

[0062] When the mobile terminal B100 is in a flat position, it is determined whether the bottom surface of the mobile terminal B100 is horizontal or vertical with respect to the X axis. When the bottom surface of the mobile terminal B100 is closer to horizontal than vertical, the image display orientation is determined so that the image faces forward and is oriented vertically relative to the position of the display device A100 at the specified time. When the bottom surface of the mobile terminal B100 is closer to vertical than horizontal, the image display orientation is determined so that the image faces forward and is oriented horizontally relative to the position of the display device A100 at the specified time.

[0063] In this embodiment, the image display direction is determined so that the image faces forward relative to the attitude of the display device A100 at a predetermined time, but a method of determining the image display direction so that the image faces forward relative to the current attitude of the display device A100 may also be used.

[0064] The above-described method for determining the image display direction is merely an example, and the method for determining the image display direction may be different.

[0065] <How to determine the display position of the virtual screen> A method for determining the display position of the virtual screen in this embodiment will be described below.

[0066] In this embodiment, position information is acquired from the orientation information of the mobile terminal B100, and the display position of the virtual screen to be displayed on the display device A100 is determined so that the image virtual screen is displayed above the mobile terminal B100. The method for determining the display position of the virtual screen is merely an example, and other methods for determining the display position of the virtual screen may be used. For example, the virtual screen may be displayed at a position preset by the user. That is, the virtual screen may be displayed so that it follows the user's movement (the movement of the display device A100). In this case, the virtual screen appears to follow the user's movement, but when generating a composite image, the virtual screen is composited so that the virtual screen is displayed at the same position regardless of the background video. Note that the virtual screen may also be displayed at a predetermined position in front of the user regardless of the position of the mobile terminal. That is, the virtual screen may be positioned and displayed so that it remains stationary at a predetermined position in the mixed reality space. In this case, even if the user moves, the virtual screen appears to be fixed in the mixed reality space. However, when generating a composite image, the size and orientation (angle and direction) of the virtual screen are changed so that the virtual screen appears to be displayed at the same position in the background video.

[0067] The display position of the virtual screen may be determined based on a predetermined part of the user, and may be displayed at the position where the user's line of sight is located or near the user's hand, for example.

[0068] <Overview of display device and mobile device operation> Hereinafter, an overview of the operations of the display device A100 and the mobile terminal B100 in this embodiment will be described with reference to FIG.

[0069] 7 is a sequence diagram showing the flow of processing when the display device A100 and the mobile terminal B100 are connected and the screen of the mobile terminal B100 is displayed on the display device A100. The display device A100 and the mobile terminal B100 will be described assuming that they have the same hardware configuration as the display device A100 and the mobile terminal B100 shown in FIG.

[0070] In step S701, when an operation to start an app is performed, the display device A100 executes the processes of steps S702 to S712, which will be described later. The operation to start an app is, for example, a start-up process that is executed when a user instructs it via the input unit A104, or when the display device A100 and the mobile terminal B100 come closer than a certain distance.

[0071] In step S702, the display device A100 requests a remote connection from the mobile terminal B100.

[0072] In step S703, upon receiving the remote connection request from the display device A100 in step S702, the mobile terminal B100 returns a remote connection permission and establishes a connection with the display device A100.

[0073] In step S704, the display device A100 acquires the attitude of the display device A100, which is used when the display device A100 determines the image display direction, that is, the direction in which the virtual screen is displayed, in step S706, which will be described later.

[0074] In step S705, the display device A100 acquires the attitude of the mobile terminal B100, which is used when the display device A100 determines the image display direction in step S706, which will be described later.

[0075] In step S706, the display device A100 determines the image display direction using the image display direction determination method described above, based on the orientation information and coordinate system of the display device A100 acquired in step S704, and the orientation information of the mobile terminal B100 acquired in step S705. Here, the coordinate system is based on the orientation information of the display device A100 acquired in step S704.

[0076] In step S707, the display device A100 determines the image display position, that is, the position where the virtual screen is to be displayed, using the image display position determination method described above, based on the attitude information of the mobile terminal B100 acquired in step S705.

[0077] In step S708, the display device A100 requests an image corresponding to the image display orientation determined in step S706 from the mobile terminal B100. For example, if the image display orientation is portrait, an image for the portrait screen of the mobile terminal B100 is requested, and if the image display orientation is landscape, an image for the landscape screen of the mobile terminal B100 is requested. Note that, although the display device A100 requests an image from the mobile terminal B100 in this example, this is not limiting, and the display device A100 may request information for generating an image. In this case, in the next step S709, the mobile terminal B100 sends information to the display device A100 in accordance with the requested information. The display device A100 also performs information acquisition to acquire the requested information.

[0078] In step S709, the mobile terminal B100 sends the image requested by the display device A100 in step S708 to the display device A100.

[0079] In step S710, the display device A100 displays the image received from the mobile terminal B100 in step S709 in the image display orientation and the image display position determined in step S706 and S707, respectively. Note that, although an image in a predetermined orientation is received from the mobile terminal B100 and displayed as a virtual screen here, this is not limiting. After receiving an image in a predetermined orientation from the mobile terminal B100, the image may be displayed on the virtual screen in the determined image display orientation. For example, when a portrait-oriented image is received from the mobile terminal B100 and displayed as a virtual screen, the image may be displayed as a landscape-oriented virtual screen.

[0080] The processes from step S705 to step S710 are repeatedly executed until an operation to terminate the app is performed. The operation to terminate the app is, for example, a termination process that is executed when a user instructs it via input unit A104 or when display device A100 and mobile terminal B100 are separated by more than a certain distance.

[0081] Next, detailed operations of the display device A100 and the mobile terminal B100 for realizing the above sequence will be described with reference to FIGS.

[0082] 8 is a flowchart of the display device A100 according to this embodiment. This flowchart starts when an application start operation is performed.

[0083] In step S801, the control unit A101 requests a remote connection to the mobile terminal B100 via the communication unit A107. The processing in this step corresponds to the processing in step S702 in FIG.

[0084] In step S802, the control unit A101 determines whether to end the process based on the state of each unit of the display device A100. For example, the control unit A101 ends the process of this flowchart when an end operation is performed via the input unit A104 or when the control unit A101 detects via the sensor A106 that the user has detached the display device A100. If it determines to continue the process, the control unit A101 proceeds to step S803.

[0085] In step S803, the control unit A101 determines whether to execute reset processing based on the state of each unit of the display device A100. For example, the control unit A101 determines to execute reset processing if the reset processing has never been executed since the application was launched or if a reset operation has been performed via the input unit A104. If it is determined to execute reset processing, the control unit A101 proceeds to step S804. If it is determined not to execute reset processing, the control unit A101 proceeds to step S805.

[0086] In step S804, the control unit A101 acquires orientation information of the display device A100 via the sensor A106. The control unit A101 stores the acquired orientation information of the display device A100 in the memory A103 for use in the image display orientation determination process in step S806 and the image display position determination process in step S807, which will be described later. The method for acquiring the orientation information of the display device A100 is as described above. The processing of this step corresponds to the processing of step S704 in FIG. 7.

[0087] In step S805, the control unit A101 acquires posture information of the mobile terminal B100 via the sensor A106 or the communication unit A107. The method for acquiring posture information of the mobile terminal B100 from the display device A100 is as described above. The processing of this step corresponds to the processing of step S705 in FIG. 7.

[0088] In step S806, the control unit A101 determines the image display direction based on the orientation information of the display device A100 acquired in step S804 and the orientation information of the mobile terminal B100 acquired in step S805. The method for determining the image display direction is as described above. The processing in this step corresponds to the processing in step S706 in FIG. 7.

[0089] In step S807, the control unit A101 determines the image display position based on the orientation information of the display device A100 acquired in step S804 and the orientation information of the mobile terminal B100 acquired in step S805. The method for determining the image display position is as described above. The processing of this step corresponds to the processing of step S707 in FIG. 7.

[0090] In step S808, the control unit A101 requests the mobile terminal B100 to send an image corresponding to the image display orientation determined in step S806 via the communication unit A107. If there is no difference between the image information last requested and the image information to be newly requested, the control unit A101 may skip the processing of this step. The processing of this step corresponds to the processing of step S708 in FIG. 7.

[0091] In step S809, the control unit A101 receives the image requested from the mobile terminal B100 in step S808 via the communication unit A107 and stores it in the memory A103. The processing in this step corresponds to the processing in step S709 in FIG.

[0092] In step S810, the control unit A101 displays the image received from the mobile terminal B100 in step S809 on the output unit A105 so that the image display orientation and the image display position determined in step S806 and S807 match. The processing in this step corresponds to the processing in step S710 in FIG. 7.

[0093] 9 is a flowchart of the mobile terminal B100 in this embodiment. This flowchart starts when the mobile terminal B100 receives a connection request from the display device A100.

[0094] In step S901, the control unit B101 determines whether to end the process based on the state of each unit of the mobile terminal B100. For example, the control unit B101 ends the process of this flowchart when an end operation is performed via the input unit B104 or when the control unit B101 detects that an app is to be ended from the display device A100 via the communication unit B107. If it is determined that the process should be continued, the control unit B101 proceeds to step S902.

[0095] In step S902, the control unit B101 determines whether an image request has been received from the display device A100 via the communication unit B107. If the control unit B101 determines that an image request has been received, it executes the process of step S903. If the control unit B101 determines that an image request has not been received, it executes the process of step S904.

[0096] In step S903, the control unit B101 stores the image request information received in step S902 in the memory B103. The image request information is information requested from the display device A100 to the mobile terminal B100 in step S708 in FIG.

[0097] The processing in steps S902 and S903 corresponds to the processing in step S708 in FIG.

[0098] In step S904, the control unit B101 generates an image of the mobile terminal B100 according to the image request information stored in the memory B103 in step S903, and transmits it to the display unit A100 via the communication unit B107. The processing of this step corresponds to the processing of step S709 in FIG.

[0099] As described above, in the system configured with the display device A100 and the mobile terminal B100 of this embodiment, the display direction and display position of the virtual screen displayed on the display device A100 are determined based on the attitude of the display device A100 at a predetermined time and the attitude of the mobile terminal B100. Note that the display direction and display position may also be based on a coordinate system based on the attitude of the display device A100 at the predetermined time.

[0100] This means that when an HMD is used in combination with a mobile device, the screen and position of the mobile device displayed on the HMD will be adjusted according to the state of the mobile device, providing a better user experience.

[0101] An example of a composite image visually recognized by a user in the first embodiment will be described with reference to FIGS. 11(a), 11(b), and 11(c).

[0102] 11(a), a scene is assumed in which the mobile terminal B100 is placed horizontally on the ground and is oriented horizontally as seen by the user. In this case, since the mobile terminal is oriented horizontally as seen by the user, a synthetic image 1101 is generated in which the virtual screen 1102 appears to be displayed horizontally, and is viewed by the user. An object 1103 is a real object.

[0103] 11(b), a scene is assumed in which the mobile terminal B100 is placed sideways against an object 1103. In this case, since the mobile terminal is oriented sideways as seen by the user, a composite image 1111 is generated in which the virtual screen 1112 appears to be displayed sideways, and is viewed by the user.

[0104] 11(c), a scene is assumed in which the mobile terminal B100 is placed vertically against an object 1103. In this case, since the mobile terminal is vertically oriented as seen from the user, a composite image 1121 is generated in which the virtual screen 1122 appears to be displayed horizontally, and is viewed by the user.

[0105] (Second embodiment) In the first embodiment, a method for determining the display direction of an image to be displayed on the display device A100 is described based on the attitude of the display device A100 at a predetermined time, a coordinate system based on the attitude of the display device A100, and the attitude of the mobile terminal B100. In the second embodiment, a case is described in which the display direction of a virtual screen is determined based on setting information of the mobile terminal B100.

[0106] For example, some smartphones have a function that locks the orientation of the display screen to portrait orientation, and some content executed on the smartphone also has a function that locks the display to portrait or landscape orientation.

[0107] When the mobile terminal B100 is in such a state or when an application related to such content is being executed on the mobile terminal B100, it is desirable to determine the display direction of the image to be displayed on the display device A100 taking this into consideration.

[0108] Since this embodiment has many commonalities with the first embodiment, the following description will focus on the parts unique to this embodiment.

[0109] A method for determining the image display direction in this embodiment will be described below.

[0110] As described above, in this embodiment, the display direction of an image to be displayed on the display device A100 is determined taking into consideration the screen settings such as the user settings of the mobile terminal B100 and the application settings for each content.

[0111] If the screen settings of the mobile terminal B100 specify that the screen should be displayed in portrait orientation, the image display direction is determined so that the image is displayed in portrait orientation and directly in front of the user.

[0112] If the screen settings of the mobile terminal B100 specify that the screen should be displayed in landscape orientation, the image display orientation is determined so that the image is displayed in landscape orientation and in front of the user.

[0113] If neither of these is specified in the screen settings of the mobile terminal B100, the image display direction is determined according to the method for determining the image display direction described in the first embodiment.

[0114] In the present embodiment, user settings and application settings for each content are given as examples of screen setting information, but the screen setting information may be other setting information.

[0115] The above-described method for determining the image display direction is merely an example, and the method for determining the image display direction may be different.

[0116] Hereinafter, an overview of the operations of the display device A100 and the mobile terminal B100 in this embodiment will be described with reference to FIG.

[0117] FIG. 10 is a sequence diagram showing the flow of processing when the display device A100 and the mobile terminal B100 are connected and the screen of the mobile terminal B100 is displayed on the display device A100.

[0118] Note that the same reference numerals as those used in FIG. 7 are used to indicate operations similar to those of the display device A100 and the mobile terminal B100 described in the first embodiment, and descriptions thereof will be omitted.

[0119] In step S1001, the mobile terminal B100 acquires screen setting information of the mobile terminal B100, which influences when the display device A100 determines the image display orientation in step S1003, which will be described later. The screen setting information is, for example, information such as whether the screen display of the mobile terminal B100 is locked and fixed to portrait orientation, or whether the display orientation of the content of the app currently running on the mobile terminal B100 is fixed to portrait or landscape orientation.

[0120] In step S1002, the mobile terminal B100 sends the screen setting information acquired in step S1001 to the display device A100.

[0121] The processing of steps S1001 and S1002 does not necessarily have to be performed immediately after step S703 after connection establishment, but may be performed when the screen settings of mobile terminal B100 are updated, or immediately before display device A100 determines the image display orientation, etc. Furthermore, since it is desirable that the latest screen setting information of mobile terminal B100 has been transmitted to display device A100, the processing of steps S1001 and S1002 may be performed every time the screen settings of mobile terminal B100 are updated.

[0122] In step S1003, the display device A100 determines the image display orientation using the image display orientation determination method described above, based on the screen setting information, the orientation information of the display device A100, a coordinate system based on the orientation information of the display device A100, and the orientation information of the mobile terminal B100. Here, the screen setting information was received from the mobile terminal B100 in step S1002, and the orientation information of the display device A100 was acquired in step S704, and the orientation information of the mobile terminal B100 was acquired in step S705.

[0123] As described above, in the system configured by the display device A100 and the mobile terminal B100 of this embodiment, the display direction of the image displayed on the display device A100 is determined taking into consideration the screen settings of the mobile terminal B100.

[0124] In this embodiment, screen setting information is acquired from the mobile terminal B100, but this is not limited to this. Whether the screen is oriented vertically or horizontally may be determined based on the orientation displayed on the screen of the mobile terminal B100 from the image captured by the display device A100.

[0125] This means that when an HMD is used in combination with a mobile device, the screen of the mobile device displayed on the HMD will respond not only to the external state of the mobile device but also to its internal state, providing a better user experience.

[0126] An example of a composite image visually recognized by a user in the second embodiment will be described with reference to FIG.

[0127] 12, a scene is assumed in which mobile terminal B100 is placed vertically against object 1103. Also, a scene is assumed in which a user displays content playing a video to be viewed on a landscape screen on virtual screen 1202. In this case, the mobile terminal is vertically oriented from the user's perspective, but the content to be displayed is a landscape screen, so a composite image 1201 is generated in which virtual screen 1202 appears to be displayed horizontally, and is viewed by the user.

[0128] (Third embodiment) In the first and second embodiments, a method has been described for determining the display direction and display position of a virtual screen (image) to be displayed on the display device A100 based on the attitude of the display device A100 at a predetermined time, a coordinate system based on that attitude, and the attitude of the mobile terminal B100. However, when the display direction or display position of the virtual screen (image) is changed by a user operation, there are cases where it is desirable to display the image accordingly.

[0129] For example, assume that an image is displayed in portrait orientation on the display device A100 according to the image display orientation determination methods of the first and second embodiments. However, if the image orientation is subsequently changed from portrait to landscape by a user operation, it is desirable to continue displaying the image in landscape orientation without following the image display orientation determination methods described in the first and second embodiments.

[0130] Similarly, even if the display position of an image is changed by user operation, it is desirable to continue displaying the image in the current position without following the method of determining the image display position described in the first and second embodiments.

[0131] Since this embodiment has many commonalities with the first and second embodiments, the following description will focus on the unique features of this embodiment.

[0132] A method for determining the image display direction in this embodiment will be described below.

[0133] As described above, in this embodiment, when the image display direction is changed by a user operation, the display direction of the image to be displayed on the display device A100 is determined taking this into consideration.

[0134] The user's operation to change the screen display direction may be performed via the buttons, interfaces, and sensors of the display device A100, or may be performed via the mobile terminal B100.

[0135] When the image display direction is changed by a user operation, the image display direction is determined so as to be the direction specified by the user.

[0136] If the image display direction has not been changed by user operation, or if the image display direction has been changed but the change has been canceled, the image display direction is determined according to the method for determining the image display direction described in the first or second embodiment.

[0137] The above-described method for determining the image display direction is merely an example, and the method for determining the image display direction may be different.

[0138] The method for determining the image display position in this embodiment will be described below.

[0139] As described above, in this embodiment, when the image display position is changed by a user operation, the display position of the image to be displayed on the display device A100 is determined taking this into consideration.

[0140] The user's operation to change the screen display position may be performed via the buttons, interfaces, and sensors of the display device A100, or may be performed via the mobile terminal B100.

[0141] When the image display position is changed by a user operation, the image display position is determined so that it becomes the position specified by the user.

[0142] If the image display position has not been changed by user operation, or if the image display position has been changed but the change has been canceled, the image display position is determined according to the method for determining the image display position described in the first or second embodiment.

[0143] In this embodiment, the image display position is determined so as to be the position specified by the user, but it may also be determined so that the position specified by the user is the relative position with respect to the mobile terminal B100.

[0144] The above-described method for determining the image display position is merely an example, and the method for determining the image display position may be different.

[0145] As described above, in the system configured with the display device A100 and mobile terminal B100 of this embodiment, when the image display direction or display position is changed by user operation, the display direction and display position of the image to be displayed on the display device A100 are determined taking these into consideration.

[0146] This allows for a better user experience when using an HMD in combination with a mobile device, while enabling display that suits the user's preferences.

[0147] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0148] (Fourth embodiment) In the fourth embodiment, a method of determining the orientation of the virtual screen based on information different from that in the first and second embodiments will be described.

[0149] Instead of the setting information of the mobile terminal B100, it may be determined whether the virtual screen is displayed in portrait or landscape orientation based on whether the logo or pattern of the mobile terminal B100 is portrait or landscape.

[0150] It is also possible to determine whether the mobile terminal B100 is oriented vertically or horizontally based on the position of the camera of the mobile terminal B100, and then determine whether the virtual screen is displayed vertically or horizontally.

[0151] Furthermore, instead of using the setting information of the mobile terminal B100, the user may be able to set in advance which orientation of the mobile terminal B100 is portrait or landscape. In this case, if the mobile terminal B100 is close to the portrait orientation that has been set in advance, the virtual screen may be displayed in portrait orientation, and if the mobile terminal B100 is close to the landscape orientation that has been set in advance, the virtual screen may be displayed in landscape orientation.

[0152] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0153] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0154] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0155] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0156] [Configuration 1] image acquisition means for acquiring a captured image; a display control means for controlling a display means to display a composite image obtained by combining the captured image with a virtual screen, the display control means controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation as seen from the user, and to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation as seen from the user. 1. An information processing device comprising:

[0157] [Configuration 2] further comprising an information acquisition means for acquiring information about the virtual screen from the mobile terminal; the display control means controls the display means to combine the captured image with a virtual screen based on the information, and display the combined image on the display means. 2. The information processing device according to configuration 1.

[0158] [Configuration 3] the display control means controls the display means to display the virtual screen in portrait or landscape orientation based on the orientation of the mobile terminal and the orientation of the display means. 3. The information processing device according to configuration 1 or 2.

[0159] [Configuration 4] the display control means controls the display means to display the virtual screen in portrait orientation when the portable terminal is placed horizontally and is oriented vertically as seen from the user, and to display the virtual screen in landscape orientation when the portable terminal is placed horizontally and is oriented horizontally as seen from the user. 4. The information processing device according to configuration 3.

[0160] [Configuration 5] The display device further includes a determination unit that determines a front direction of the display device based on the attitude of the display device at a predetermined time point, the display control means controls the display means to display the virtual screen in portrait orientation when the mobile terminal is oriented vertically relative to the front direction of the display device at the predetermined time point, even if the attitude of the display device changes after the predetermined time point, and to display the virtual screen in landscape orientation when the mobile terminal is oriented horizontally relative to the front direction of the display device at the predetermined time point. 5. The information processing device according to any one of configurations 1 to 4.

[0161] [Configuration 6] the determining means controls the display device to update its front direction when the attitude of the display device at the predetermined time point has changed by a larger amount than a threshold value; 6. The information processing device according to configuration 5.

[0162] [Configuration 7] the display control means displays the virtual screen in portrait orientation when the content to be displayed on the virtual screen is a portrait screen, and displays the virtual screen in landscape orientation when the content to be displayed on the virtual screen is landscape orientation; 7. The information processing device according to any one of configurations 2 to 6.

[0163] [Configuration 8] When the content to be displayed on the virtual screen is a portrait screen, the display control means displays the virtual screen in portrait orientation even if the mobile terminal is in landscape orientation as seen from the user, and when the content to be displayed on the virtual screen is a landscape screen, the display control means displays the virtual screen in landscape orientation even if the mobile terminal is in portrait orientation as seen from the user. 8. The information processing device according to configuration 7.

[0164] [Configuration 9] the information acquisition means acquires information about a portrait-oriented screen from the mobile terminal when the content is a portrait-oriented screen, and acquires information about a landscape-oriented screen from the mobile terminal when the content is a landscape-oriented screen; 9. The information processing device according to configuration 7 or 8.

[0165] [Configuration 10] When the orientation of the screen of the mobile terminal is fixed, the display control means displays the virtual screen in the fixed orientation even if the attitude of the mobile terminal changes. 10. The information processing device according to any one of configurations 2 to 9.

[0166] [Configuration 11] the display control means controls the display of the virtual screen when the mobile terminal and the user are closer than a predetermined distance. 11. The information processing device according to any one of configurations 1 to 10.

[0167] [Configuration 12] the display control means terminates the display of the virtual screen when the mobile terminal and the user move away from each other by a predetermined distance. 12. The information processing device according to any one of configurations 1 to 11.

[0168] [Configuration 13] the display control means displays the virtual screen so that the virtual screen follows the attitude of the display means in accordance with the attitude of the display means. 13. The information processing device according to any one of configurations 1 to 12.

[0169] [Configuration 14] the display control means displays the virtual screen so that the virtual screen remains stationary in the mixed reality space even if the attitude of the display means changes. 14. The information processing device according to any one of configurations 1 to 13.

[0170] [Control method] an image acquisition step of acquiring a captured image; a display control step of controlling the display means to display a composite image obtained by combining the captured image with a virtual screen, the display control step controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation as seen from the user, and to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation as seen from the user; 2. A method for controlling an information processing apparatus comprising:

[0171] [program] A program for causing a computer to function as each of the means of the information processing device described in configuration 1.

[0172] [system] A mobile device, an image acquisition device that acquires a captured image; a display control device that controls a display unit to display a composite image obtained by combining the captured image with a virtual screen, the display control device controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation as seen from the user, and controls the display means to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation as seen from the user. An information processing system comprising:

Claims

1. image acquisition means for acquiring a captured image; a display control means for controlling a display means to display a composite image obtained by combining the captured image with a virtual screen, the display control means controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation as seen from the user, and to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation as seen from the user.

1. An information processing device comprising:

2. further comprising an information acquisition means for acquiring information about the virtual screen from the mobile terminal; the display control means controls the display means to combine the captured image with a virtual screen based on the information, and display the combined image on the display means.

2. The information processing apparatus according to claim 1, wherein:

3. the display control means controls the display means to display the virtual screen in portrait or landscape orientation based on the orientation of the mobile terminal and the orientation of the display means.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the display control means controls the display means to display the virtual screen in portrait orientation when the portable terminal is placed horizontally and is oriented vertically as seen from the user, and to display the virtual screen in landscape orientation when the portable terminal is placed horizontally and is oriented horizontally as seen from the user.

4. The information processing apparatus according to claim 3,

5. The display device further includes a determination unit for determining a front direction of the display device based on the attitude of the display device at a predetermined time point, the display control means controls the display means to display the virtual screen in portrait orientation when the mobile terminal is oriented vertically relative to the front direction of the display device at the predetermined time point, even if the attitude of the display device changes after the predetermined time point, and to display the virtual screen in landscape orientation when the mobile terminal is oriented horizontally relative to the front direction of the display device at the predetermined time point.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. the determining means controls the display device to update its front direction when the attitude of the display device at the predetermined time point has changed by a larger amount than a threshold value; 6. The information processing apparatus according to claim 5,

7. the display control means displays the virtual screen in portrait orientation when the content to be displayed on the virtual screen is a portrait screen, and displays the virtual screen in landscape orientation when the content to be displayed on the virtual screen is landscape orientation; 3. The information processing apparatus according to claim 2, wherein:

8. When the content to be displayed on the virtual screen is a portrait screen, the display control means displays the virtual screen in portrait orientation even if the mobile terminal is in landscape orientation as seen from the user, and when the content to be displayed on the virtual screen is a landscape screen, the display control means displays the virtual screen in landscape orientation even if the mobile terminal is in portrait orientation as seen from the user.

8. The information processing apparatus according to claim 7,

9. the information acquisition means acquires information about a portrait-oriented screen from the mobile terminal when the content is a portrait-oriented screen, and acquires information about a landscape-oriented screen from the mobile terminal when the content is a landscape-oriented screen; 8. The information processing apparatus according to claim 7,

10. When the orientation of the screen of the mobile terminal is fixed, the display control means displays the virtual screen in the fixed orientation even if the attitude of the mobile terminal changes.

3. The information processing apparatus according to claim 2, wherein:

11. the display control means controls the display of the virtual screen when the mobile terminal and the user are closer than a predetermined distance.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

12. the display control means terminates the display of the virtual screen when the mobile terminal and the user move away from each other by a predetermined distance.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

13. the display control means displays the virtual screen so that the virtual screen follows the attitude of the display means in accordance with the attitude of the display means.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

14. the display control means displays the virtual screen so that the virtual screen remains stationary in the mixed reality space even if the attitude of the display means changes.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

15. an image acquisition step of acquiring a captured image; a display control step of controlling the display means to display a composite image obtained by combining the captured image with a virtual screen, the display control step controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation as seen from the user, and to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation as seen from the user; 2. A method for controlling an information processing apparatus comprising:

16. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

17. A mobile device, an image acquisition device that acquires a captured image; a display control device that controls a display unit to display a composite image obtained by combining the captured image with a virtual screen, the display control device controls the display means to display the virtual screen in portrait orientation when the mobile terminal is in portrait orientation as seen from the user, and controls the display means to display the virtual screen in landscape orientation when the mobile terminal is in landscape orientation as seen from the user. An information processing system comprising:

Images