Information processor, control method therefor, and program

Abstract

To provide an information processor capable of appropriately controlling a virtual object according to posture of a user.SOLUTION: An information processor 102 communicates with a display device 101 such as an HMD, and causes the display device 101 to display an image in a virtual space. The information processor 102 determines whether a user wearing the display device 101 is lying down. The information processor 102 changes, on the basis of the result of the determination as to whether the user is lying down, control of a virtual object included in the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] There is known a technology for displaying an image of a virtual space on a display device such as a head-mounted display worn by a user (see, for example, Patent Document 1). One or more virtual objects are displayed on this display device, and the user can move the display positions of these virtual objects by performing a predetermined operation.

[0003] Controls for moving the display position of a virtual object through user operation include, for example, control for moving the virtual object while keeping the horizontal sides of the image of the virtual space displayed on this display device parallel to the horizontal direction of the real space, and control for aligning the horizontal display position of the virtual object directly in front of the user when the user presses a reset button. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-110319 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-described control for moving the display position of a virtual object is basically based on the assumption that the user is standing, and therefore, appropriate control may not be possible when the user is in a position other than standing. For example, when the user is lying down, the horizontal direction of real space is not uniquely determined in the vicinity of the front of the user, and therefore, control based on the horizontal direction of real space as described above cannot be performed. As such, conventionally, it has not been possible to appropriately control a virtual object according to the user's posture.

[0006] The present invention aims to provide an information processing device, a control method for an information processing device, and a program that can appropriately control a virtual object in accordance with the posture of a user. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing device of the present invention is an information processing device that displays an image of a virtual space on a display device worn or held by a user, and is equipped with a determination means that determines whether the user is lying down or not, and a control means that causes the display device to display the image of the virtual space, and is characterized in that the control means changes the control of virtual objects included in the virtual space based on the determination result by the determination means. [Effects of the Invention]

[0008] According to the present invention, it is possible to appropriately control a virtual object in accordance with the user's posture. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing system including an information processing device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of data stored in a data storage unit in FIG. 1; [Figure 3] 10 is a flowchart showing the procedure of a virtual object control process executed by the information processing device of FIG. [Figure 4] FIG. 4 is a diagram for explaining the determination in S302 of FIG. 3. [Figure 5] FIG. 2 is a diagram showing an example of a screen displayed on the display unit of FIG. 1. [Figure 6] FIG. 10 is a diagram showing an example of a setting screen for setting a coordinate system to which a virtual object conforms in the present embodiment. [Figure 7]10 is a flowchart showing another procedure of the virtual object control process executed by the information processing device of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the present invention.

[0011] 1 is a block diagram showing an example of the configuration of an image processing system including an information processing device 102 according to this embodiment. This image processing system is, for example, an MR system that allows a user to experience a mixed reality space (MR space) that combines a real space and a virtual space. In this embodiment, a configuration is described in which a user experiences an MR space by displaying a composite image that combines an image of a virtual space rendered by computer graphics (CG) with an image of a real space.

[0012] As shown in Fig. 1, this image processing system is composed of a display device 101, an information processing device 102, and an operation device 103. In this image processing system, the information processing device 102 and the display device 101 are connected to each other so that they can communicate data with each other. Furthermore, the information processing device 102 and the operation device 103 are connected to each other so that they can communicate data with each other. Note that the data communication may be wired communication or wireless communication.

[0013] In this image processing system, the information processing device 102 generates a composite image by combining an image of real space acquired from the display device 101 with an image of virtual space generated by the information processing device 102. The information processing device 102 outputs the composite image to the display device 101 as a mixed reality image (MR image). Note that in this embodiment, an MR system that displays an MR image (an image obtained by combining an image of real space with an image of virtual space) on the display device 101 will be described as an example, but the image processing system of this embodiment is not limited to an MR system. For example, the image processing system of this embodiment may be a VR (virtual reality) system that displays only an image of virtual space on the display device 101, an AR (augmented reality) system that displays an image of virtual space on the display device 101 with the real space visible through it, or the like.

[0014] The display device 101 is, for example, a head mounted display device (hereinafter referred to as "HMD"). Note that in this embodiment, a non-transmissive HMD will be described as an example of the display device 101, but the display device 101 is not limited to a non-transmissive HMD and may be a transmissive HMD. Furthermore, the display device 101 is not limited to an HMD, but may also be a handheld display (hereinafter referred to as "HHD"). An HHD is a handheld display. In other words, the display device 101 may be a display that a user holds in their hand and peers into like binoculars to observe an image. The display device 101 may also be a display terminal such as a tablet or a smartphone.

[0015] The display device 101 includes a sound recording unit 104 , a sound output unit 105 , an image capturing unit 106 , and a display unit 107 .

[0016] The recording unit 104 captures, as audio data, sounds around the display device 101 and the voice of a user wearing the display device 101 (hereinafter simply referred to as "user"), and outputs this audio data to the information processing device 102. The recording unit 104 may be a directional microphone or a movable pin microphone, depending on the application. The display device 101 may also be equipped with a plurality of such recording units 104.

[0017] The audio output unit 105 outputs audio based on audio data output from the information processing device 102. The audio output unit 105 may be a speaker, an earphone, or a headphone. The display device 101 may also include a plurality of such audio output units 105. In a configuration including a plurality of audio output units 105, each audio output unit 105 may be connected to the information processing device 102 by wire or wirelessly. Alternatively, for example, a device including both the recording unit 104 and the audio output unit 105, such as a wireless earphone with a microphone function, may be configured to communicate with the display device 101 or the information processing device 102.

[0018] The imaging unit 106 captures images of the real space continuously in time series and outputs the captured images of the real space (captured images) to the information processing device 102. The imaging unit 106 may include a stereo camera consisting of two cameras fixed to each other so as to be able to capture images of the real space in the line of sight of the user.

[0019] The display unit 107 displays the MR image output from the information processing device 102. The display unit 107 may include two displays arranged to correspond to the left and right eyes of the user, respectively. In such a configuration, the display corresponding to the user's left eye displays the MR image for the left eye, and the display corresponding to the user's right eye displays the MR image for the right eye.

[0020] The operation device 103 includes a vibration unit 118, an object movement instruction unit 119, and a display reset instruction unit 120. The operation device 103 is, for example, an operation device for video games that can acquire values ​​corresponding to the tilt direction of a stick, values ​​indicating the press state of a button, and the like. However, the operation device 103 is not limited to an operation device held by a user's hand, such as an operation device for video games, but may also be an operation device worn on the user's body, hand, or the like. The operation device 103 may, for example, be a ring-shaped device that can be worn on a user's finger. The operation device 103 may also include multiple buttons or may have a built-in optical trackpad (hereinafter referred to as "OTP"). When the operation device 103 has a built-in OTP, the user can place a finger on the OTP and rub it in any direction to align the pointer with a desired item. The user can then perform a confirmation action, such as confirming the selection of a virtual object, such as a menu item, by pressing a button on the OTP. It is also possible to move a virtual object by pointing to another location while the object is selected, twisting the selected arm or finger, or performing a predetermined operation.

[0021] The vibration unit 118 vibrates to give the user a vibration sensation in accordance with a vibration instruction received from the vibration instruction unit 117 of the information processing device 102. To give the user a vibration sensation, it is desirable for the vibration element to have a short response time. As the vibration element, for example, a piezoelectric element can be used, but other vibration elements such as a linear vibrator (LRA) or an eccentric motor (ERM) can also be used.

[0022] The object movement instruction unit 119 outputs a value according to the user's operation (a value according to the tilt direction of the stick or a value indicating the pressed state of a button) as operation information to the object movement processing unit 116. Note that the movement of the object referred to here may include not only a simple parallel movement of the virtual object but also various affine transformations such as rotation, enlargement, and reduction.

[0023] The display reset instruction unit 120 outputs a display reset instruction to the display reset processing unit 115 in response to a user operation (pressing a button or detection by a sensor such as an OTP).

[0024] The operation device 103 may include a position and orientation calculation unit (not shown) in addition to the vibration unit 118, object movement instruction unit 119, and display reset instruction unit 120. The position and orientation calculation unit calculates (acquires) the position and orientation (position and orientation) of the operation device 103 in a world coordinate system in virtual space. The world coordinate system is a coordinate system equivalent to the real space to which the user belongs. In this embodiment, the position of a virtual object in virtual space is determined based on the user's position and a predetermined axis direction.

[0025] The position and orientation calculation unit outputs information about the calculated position and orientation to the object movement processing unit 116. The position and orientation calculation unit has, for example, a sensor (such as an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor) for calculating the position and orientation of the operation device 103. Note that the position and orientation calculation unit may have a plurality of sensors. Furthermore, the position and orientation calculation unit may include an imaging unit and an imaging processing unit, and may calculate the position and orientation of the operation device 103 by performing Simultaneous Localization and Mapping (SLAM) processing based on feature points of the captured image.

[0026] The position and orientation calculation unit may also calculate the position and orientation of the operation device 103 in conjunction with an optical sensor installed in real space. Note that the position and orientation calculation unit may calculate either the value of the position component or the value of the orientation component, rather than the values ​​of both. The position and orientation calculation unit may also output values ​​(values ​​output from the sensor) and images for calculating the position and orientation of the operation device 103 to the information processing device 102, and the position and orientation calculation unit 114 of the information processing device 102 may calculate the position and orientation of the operation device 103.

[0027] The information processing device 102 has a sound recording processing unit 108, a sound output instruction unit 109, an image synthesis unit 110, a control unit 111, an image generation unit 112, a data storage unit 113, a position and orientation calculation unit 114, a display reset processing unit 115, an object movement processing unit 116, and a vibration instruction unit 117.

[0028] The position and orientation calculation unit 114 calculates the position and orientation of the image capture unit 106 in the world coordinate system. Specifically, the position and orientation calculation unit 114 extracts markers assigned to the world coordinate system from an image of real space captured by the image capture unit 106. Then, the position and orientation calculation unit 114 calculates the position and orientation of the image capture unit 106 in the world coordinate system based on the position and orientation of the extracted markers. Then, the position and orientation calculation unit 114 stores information indicating the calculated position and orientation of the image capture unit 106 (position and orientation information) in the data storage unit 113 via the control unit 111.

[0029] Furthermore, the position and orientation calculation unit 114 calculates the position and orientation of the operation device 103 in the world coordinate system using information about the position and orientation of the operation device 103 acquired from a position and orientation calculation unit (not shown) of the operation device 103 or an image of real space captured by the imaging unit 106. At this time, depending on the calculation method used by the position and orientation calculation unit (not shown) of the operation device 103, a difference (error) may occur between the calculated position and orientation of the operation device 103 and the actual position and orientation. For example, in a method of calculating the position and orientation using a combination of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, errors from the sensors may accumulate, resulting in a calculated position and orientation that has an error different from the actual position and orientation. Alternatively, this method may not be able to calculate the position and orientation of the operation device 103. Furthermore, in a method of calculating the position and orientation using an optical sensor installed in real space, the optical sensor may be blocked by another real object, resulting in a calculated position and orientation that has an error different from the actual position and orientation. Alternatively, this method may not be able to calculate the position and orientation.

[0030] In such a case, the position and orientation calculation unit 114 can extract the markers attached to the operation device 103 from the image of real space captured by the imaging unit 106, and thereby accurately calculate the position and orientation of the operation device 103 based on the position and orientation of the markers. At this time, the position and orientation calculation unit 114 may use all or part of the position and orientation calculation results acquired from a position and orientation calculation unit (not shown) of the operation device 103. However, the calculation method of the position and orientation calculation unit 114 is not limited to a method using markers, and may also be a calculation method using SLAM processing. Furthermore, the orientation may be detected by an orientation sensor unit (not shown) of the display device 101. The orientation sensor unit (not shown) of the display device 101 has an inertial measurement unit (IMU) and outputs information on the orientation of the display device 101 (orientation information) to the information processing device 102.

[0031] The position and orientation calculation unit 114 stores the calculated position and orientation information of the operation device 103 in the data storage unit 113 via the control unit 111 .

[0032] When the operation device 103 does not appear in the image of real space captured by the imaging unit 106, the position and orientation calculation unit 114 calculates the position and orientation of the operation device 103 based on values ​​acquired from a position and orientation calculation unit (not shown) of the operation device 103. At this time, the position and orientation calculation unit 114 may calculate the position and orientation of the operation device 103 based on the values ​​acquired from the position and orientation calculation unit (not shown) of the operation device 103 and other information. For example, when the operation device 103 does not appear in the image of real space, if the operation device 103 includes an acceleration sensor or the like, the position and orientation calculation unit 114 may calculate the position and orientation of the operation device 103 based on the detection result of the acceleration sensor or the like. For example, the position and orientation calculation unit 114 may calculate the current position and orientation of the operation device 103 based on the position of the operation device 103 at a past point in time when the operation device 103 appeared in the image of real space and the amount of movement of the operation device 103 from that point in time calculated from the acceleration. When the operation device 103 is not captured in the image in real space, the position and orientation calculation unit 114 may detect only the orientation of the operation device 103, or may not be able to calculate the position and orientation of the operation device 103.

[0033] The control unit 111 controls the entire information processing device 102. The control unit 111 controls the position and orientation of a UI (graphics) displayed on the display unit 107, for example, based on information about the position and orientation of the operation device 103 stored in the data storage unit 113. The position of the UI is indicated by three-dimensional coordinate information according to a Cartesian coordinate system of three axes, for example, the X-axis, the Y-axis, and the Z-axis. The three-dimensional coordinate information may also be based on a polar coordinate system. If the UI is a virtual ray emitted from the user's hand, the position of the UI is, for example, the start position or the end position of the ray. The orientation of the UI corresponds to the orientation of the UI in a three-dimensional virtual space. If the UI is a ray, the orientation of the UI corresponds to, for example, the extending direction of the ray. In this embodiment, a configuration will be described in which the control unit 111 controls the position and orientation of the UI displayed on the display unit 107 based on information about the position and orientation of the operation device 103, but the present invention is not limited to this configuration. For example, the control unit 111 may control the position and orientation of the UI to be displayed on the display unit 107 based on the information on the position and orientation of the imaging unit 106 stored in the data storage unit 113. Alternatively, the control unit 111 may control the position and orientation of the UI to be displayed on the display unit 107 based on both the information on the position and orientation of the operation device 103 and the information on the position and orientation of the imaging unit 106.

[0034] The control unit 111 reads out information about the position and orientation of the image capture unit 106 that is pre-recorded in the data storage unit 113, and changes the control of a virtual object included in the virtual space based on the information about the position and orientation of the image capture unit 106. The control of the virtual object is, for example, operation control for manipulating the virtual object. For example, when it is determined that the user is lying down based on the information about the position and orientation of the image capture unit 106, the control unit 111 performs control to suppress rotation of the virtual object.

[0035] Here, for example, if the user is standing, even if display device 101 is tilted slightly, the virtual object can be rotated appropriately based on the horizontal direction of real space, and therefore the virtual object may be rotated automatically. On the other hand, if the user is lying down, the horizontal direction of real space is not uniquely determined near the front of the user, and the virtual object cannot be rotated appropriately based on the horizontal direction of real space. For this reason, in this embodiment, when the user is lying down, control unit 111 suppresses the rotation of the virtual object.

[0036] Furthermore, the control unit 111 acquires sound source data from the data storage unit 113, synthesizes the sound source data as necessary, and outputs sound based on this sound source data from the sound output unit 105 of the display device 101 via the sound output instruction unit 109. When controlling the entire information processing device 102, the control unit 111 may perform control in accordance with a voice command. In this case, the control unit 111 analyzes the voice data acquired from the recording unit 104 via the sound recording processing unit 108, and if the voice command is received, performs control in accordance with the command content.

[0037] Furthermore, when the control unit 111 receives a display reset instruction output from the display reset instruction unit 120 via the display reset processing unit 115, the control unit 111 causes the image generation unit 112 to construct a virtual space that takes into account the position and orientation. In this way, the control unit 111 resets the display position of the virtual object that the user wants to see.

[0038] Furthermore, when the control unit 111 receives an object movement instruction output from the object movement instruction unit 119 via the object movement processing unit 116, it causes the image generation unit 112 to construct a virtual space in which the virtual object has been moved to a position corresponding to the object movement instruction. In this way, movement of the virtual object in response to a user operation is realized. Note that the moving virtual object does not have to be a single object, and may be, for example, a group consisting of multiple objects.

[0039] Furthermore, the control unit 111 transmits a vibration instruction to the vibration unit 118 via the vibration instruction unit 117, and gives the user a vibration sensation.

[0040] When the display reset processing unit 115 receives a display reset instruction from the display reset instruction unit 120, it transmits information about the position and orientation of the image capture unit 106, which is pre-recorded in the data storage unit 113, to the image generation unit 112. The image generation unit 112 constructs a virtual space that takes the position and orientation into consideration based on the acquired information about the position and orientation of the image capture unit 106. In this way, the display position of the virtual object that the user wants to view is reset, for example, to a position directly in front of the user in the virtual space. To reset the display position, the user may specify a virtual object and reset only the display position of the specified virtual object. Alternatively, the display positions of a group of predetermined virtual objects may be reset without the user specifying a virtual object. For example, it is conceivable to reset the display positions of a group of items, such as status information or menus, that the user wants to keep in view at all times. Note that if the position of a virtual object changes when the display position is reset, the position information of the virtual object recorded in the data storage unit 113 is also updated via the control unit 111.

[0041] Based on the operation information acquired from the object movement instruction unit 119, the object movement processing unit 116 changes the position information of the data related to each virtual object stored in the data storage unit 113 and stores the changed position information in the data storage unit 113. Thereafter, the object movement processing unit 116 acquires the position information of the data related to each virtual object stored in the data storage unit 113 and the position and orientation information of the imaging unit 106 via the control unit 111, and transmits the acquired information to the image generation unit 112. In this embodiment, the control unit 111 moves the virtual objects by causing the image generation unit 112 to construct a virtual space that takes the position and orientation into consideration based on this information.

[0042] The image generation unit 112 constructs a virtual space based on the virtual space data stored in the data storage unit 113. The virtual space data includes data related to each virtual object that constitutes the virtual space and data related to a light source that illuminates the virtual space. The image generation unit 112 then acquires information on the position and orientation of the image capture unit 106 calculated by the position and orientation calculation unit 114 from the data storage unit 113 via the control unit 111. The image generation unit 112 also acquires information on the position and orientation of the UI controlled by the control unit 111 from the data storage unit 113 via the control unit 111. The image generation unit 112 generates an image of the virtual space according to the position and orientation of the image capture unit 106. At this time, the data related to each virtual object that constitutes the virtual space and stored in the data storage unit 113 includes position information and attribute information, as shown in FIG. 2. In FIG. 2, position information and attribute information are associated with each virtual object. For example, a virtual object called "Object 1" is located at "Position 1" and its attribute information is "A." When attribute information is used, control unit 111 changes display control of virtual objects by referring to the attribute information as well as the position and orientation of image capture unit 106. For example, in this embodiment, attribute information indicating whether movement is restricted is assigned to each virtual object.

[0043] The image synthesis unit 110 synthesizes an image of the virtual space generated by the image generation unit 112 with an image of the real space captured by the imaging unit 106 to generate an MR image. At this time, the image of the virtual space generated by the image generation unit 112 may be an image representing the entire virtual space or an image representing a part of the virtual space. Furthermore, the image synthesis unit 110 may generate an MR image by performing an affine transformation on the image, or may generate an MR image by assigning the image to a parametric curved surface. Then, the image synthesis unit 110 outputs the generated MR image to the display device 101.

[0044] As described above, data storage unit 113 stores various information. Data storage unit 113 includes a RAM, a hard disk drive device, etc. In addition to the information described above as information to be stored in data storage unit 113, data storage unit 113 also stores information, etc., which will be described in this embodiment as known information.

[0045] Furthermore, the image processing system of this embodiment may use the user's hand instead of object movement instruction unit 119 of operation device 103. In this case, object movement processing unit 116 recognizes the movement (gesture) of the user's hand from the image of real space captured by imaging unit 106. Based on the recognized gesture, object movement processing unit 116 changes position information of data related to each virtual object stored in data storage unit 113 via control unit 111 and stores the changed information in data storage unit 113. Thereafter, object movement processing unit 116 transmits the position information of the data related to each virtual object stored in data storage unit 113 and information on the position and orientation of imaging unit 106 to image generation unit 112. Then, control unit 111 moves the virtual objects by causing image generation unit 112 to construct a virtual space taking the position and orientation into consideration.

[0046] Next, a virtual object control process executed by the information processing device 102 will be described.

[0047] Fig. 3 is a flowchart showing the procedure of a virtual object control process executed by the information processing device 102 in Fig. 1. The virtual object control process in Fig. 3 is realized by the control unit 111 executing a program stored in the data storage unit 113.

[0048] 3 , first, in S301, the control unit 111 determines whether or not a virtual object movement event has been detected. In S301, if the information processing device 102 does not receive either a display reset instruction or an object movement instruction from the operation device 103, it is determined that a virtual object movement event has not been detected, and the processing returns to S301. On the other hand, if the information processing device 102 receives a display reset instruction or an object movement instruction from the operation device 103, it is determined that a virtual object movement event has been detected, and the processing proceeds to S302. Note that the control unit 111 determines that the information processing device 102 has received a display reset instruction from the operation device 103, for example, when the control unit 111 receives an instruction from the display reset processing unit 115 to transmit position and orientation information stored in the data storage unit 113 to the image generation unit 112, or when the control unit 111 is notified by the display reset processing unit 115 that a display reset instruction has been received. Furthermore, the control unit 111 determines that the information processing device 102 has received an object instruction from the operation device 103, for example, when the control unit 111 receives an instruction from the object movement processing unit 116 to change the position information of data related to each virtual object stored in the data storage unit 113, or when the control unit 111 is notified by the object movement processing unit 116 that an object movement instruction has been received.

[0049] In S302, the control unit 111 determines whether the user is lying down. In S302, for example, the control unit 111 acquires information on the position and orientation of the image capturing unit 106 calculated by the position and orientation calculation unit 114 from the data storage unit 113, and determines whether the user is lying down based on the acquired information on the position and orientation of the image capturing unit 106. For example, when the pitch angle corresponding to the acquired information on the position and orientation of the image capturing unit 106 is equal to or smaller than a predetermined value θ th If θ2 is equal to or greater than θ2, it is determined in S302 that the user is lying down. th If θ1 is less than 0°, it is determined in S302 that the user is not lying down. th , θ th Although only up to 90 degrees is shown, the angle used to determine lying down may be determined to include a range not shown.

[0050] If it is determined in S302 that the user is lying down, the control unit 111 determines whether the rotation suppression function when lying down is enabled or disabled (S303). In this embodiment, the user can previously set the rotation suppression function when lying down to be enabled (ON) or disabled (OFF) on the setting screen of FIG. 5(a). The value set on this setting screen is stored in the data storage unit 113. The control unit 111 reads the value set on this setting screen from the data storage unit 113 and makes the determination in S303 using the read value. Note that if the user has not previously set the rotation suppression function when lying down to be enabled (ON) or disabled (OFF) on this setting screen, the control unit 111 may read an initial value for the system from the data storage unit 113 and make the determination in S304 using the read value. Note that the rotation referred to here refers to the roll direction among the roll, pitch, and yaw rotation directions.

[0051] If it is determined in S303 that the rotation suppression function for lying down is enabled, the control unit 111 transmits a rotation suppression instruction to the image generation unit 112 to restrict the rotation of the virtual object (S304). The image generation unit 112, which has received the rotation suppression instruction, constructs a virtual space in which the virtual object is moved while restricting rotation, and generates an image of this virtual space. In other words, even if the user attempts to move the virtual object in three-dimensional space while rotating it in the roll direction, in S304 an image of the virtual space is generated that does not reflect the rotation in the roll direction and reflects only the movement in other directions. Then, this processing ends.

[0052] If it is determined in S302 that the user is not lying down, or if it is determined in S303 that the rotation suppression function for lying down is disabled, the process proceeds to S305. In S305, the control unit 111 transmits to the image generation unit 112 a movement instruction to move the virtual object to a position corresponding to the instruction received from the user in S301. Note that this movement instruction does not include an instruction to restrict rotation, such as the rotation suppression instruction. The image generation unit 112 constructs a virtual space in which the virtual object has been moved to a position corresponding to the movement instruction, and generates an image of this virtual space. The process then ends. The image of the virtual space generated in this process is transmitted to the image synthesis unit 110. The image synthesis unit 110 synthesizes the received image of the virtual space with an image of real space captured by the imaging unit 106 to generate an MR image. The MR image is transmitted to the display device 101 and displayed on the display unit 107.

[0053] According to the above-described embodiment, the control of the virtual object is changed based on the determination result of whether the user is lying down, thereby making it possible to appropriately control the virtual object according to the user's posture.

[0054] Furthermore, in the above-described embodiment, the control of the virtual object is an operation control for manipulating the virtual object, and therefore the operation of the virtual object can be appropriately controlled in accordance with the user's posture.

[0055] Furthermore, in the above-described embodiment, when it is determined that the user is lying down, the rotation of the virtual object is suppressed, thereby suppressing the rotation of the virtual object that cannot be controlled as intended by the user due to the user being lying down.

[0056] In the above-described embodiment, control to suppress rotation of a virtual object has been described as an example of virtual object operation control, but this is not limiting. For example, when it is determined that the user is lying down, a predetermined control may be performed to move the virtual object to a position directly in front of the user in the virtual space. Here, for example, if there is a virtual object that is positioned relative to the floor surface according to gravity, like an object in real space, and the user is standing, when the user looks slightly upward, the virtual object does not need to be moved upward to match the user. This is because it is expected that the user will quickly return their gaze to the front. On the other hand, when the user is lying down, if the virtual object is left positioned near the floor surface, the virtual object will not be displayed on the display device 101, which may reduce convenience. In contrast, in this embodiment, when it is determined that the user is lying down, the display position of the virtual object is moved upward, that is, to a position directly in front of the user in the virtual space. This allows the user to easily view the virtual object while lying down.

[0057] In this embodiment, the user may be able to set in advance whether the predetermined control is enabled (ON) or disabled (OFF) on the setting screen of Fig. 5(b), thereby allowing the user's intention to be reflected in the display control of virtual objects when the user is lying down.

[0058] In this embodiment, the predetermined control is, for example, a control to reset the display position of a virtual object to a position directly in front of the user in the virtual space in accordance with an instruction from the user, thereby allowing a user who is lying down to use the reset function for the display position of the virtual object.

[0059] In this embodiment, the control of a virtual object is not limited to operation control for manipulating a virtual object. For example, the control of a virtual object may be display control for displaying a virtual object on the display device 101. In this display control, the coordinate system followed by the virtual object differs depending on whether it is determined that the user is lying down or not. For example, when it is determined that the user is not lying down, the world coordinate system is used, and when it is determined that the user is lying down, the camera coordinate system, which is a coordinate system based on the user's viewpoint in the virtual space, is used. This makes it possible to switch the coordinate system followed by the virtual object depending on the user's posture, thereby appropriately controlling the display of the virtual object.

[0060] Furthermore, in the above-described embodiment, a configuration has been described in which no rotation in the roll direction is reflected in S305, but this configuration is not limited. For example, the rotation angle may be controlled to be slower than when the rotation is not restricted. In such a configuration, the rotation angle may be simply reflected as a rotation angle obtained by dividing the original rotation angle by a predetermined value, or a predetermined angle may be set as a maximum angle of the rotation angle, and the rotation angle may be reflected up to the maximum angle but not beyond the maximum angle.

[0061] In this embodiment, the method of restricting rotation may be changed depending on the position of the target virtual object in three-dimensional space. For example, control is performed such that the rotation restriction is small near the front of the user and large as the virtual object moves away from the front of the user.

[0062] Furthermore, in this embodiment, a configuration may be adopted in which whether or not rotation of a virtual object is suppressed and the degree of rotation suppression are controlled based on attribute information assigned to the virtual object. In such a configuration, image generation unit 112 may read attribute information of a target virtual object from data storage unit 113 via control unit 111. Alternatively, control unit 111 may read attribute information of a target virtual object from data storage unit 113, convert this attribute information into other information indicating whether or not rotation is suppressed and the degree of rotation suppression, and transmit this other information to image generation unit 112. Upon receiving the attribute information or the other information, image generation unit 112 generates an image of a virtual space in which the virtual object has been moved according to the attribute information or the other information.

[0063] In this embodiment, the attribute information may be information indicating a coordinate system in which the virtual object follows. In such a configuration, for example, as shown in the setting screen of FIG. 6(a), the user may be able to set the coordinate system in which the virtual object follows from among the world coordinate system, the local coordinate system, and the camera coordinate system. This allows the user's intention to be reflected in the coordinate system in which the virtual object follows.

[0064] In addition, in this embodiment, for example, as shown in FIG. 6(b), a configuration may be adopted in which a user selects a target virtual object, invokes a setting menu for the selected virtual object, and sets the coordinate system to which the virtual object will conform from the setting menu. In such a configuration, attribute information of the virtual object may be set in advance in the MR space, and the user may change the set attribute information as needed. Furthermore, the virtual object may be assigned unchangeable attribute information. When a virtual object with such attribute information is selected by the user as shown in FIG. 6(b), the setting menu for changing the attribute information of the virtual object is not displayed. At this time, it is preferable to notify the user that the setting menu for changing the attribute information cannot be displayed. Examples of notification methods include displaying a message indicating this on the display unit 107, outputting a sound indicating this from the audio output unit 105, or applying vibrations to the user using the vibration unit 118.

[0065] In this embodiment, the attribute information may be information indicating whether or not it is affected by gravity.

[0066] Fig. 7 is a flowchart showing another procedure of the virtual object control process executed by the information processing device 102 in Fig. 1. Note that the virtual object control process in Fig. 7 is similar to the virtual object control process in Fig. 3 described above, and the following will particularly describe the content that differs from the virtual object control process in Fig. 3 described above. The virtual object control process in Fig. 7, like the virtual object control process in Fig. 3 described above, is also realized by the control unit 111 executing a program stored in the data storage unit 113.

[0067] 7, first, S701, which is the same process as S301 described above, is performed. When a movement event of a virtual object is detected (YES in S701), control unit 111 determines whether or not the virtual object to be moved is affected by gravity (S702). Specifically, control unit 111 reads attribute information of this virtual object from data storage unit 113, and determines whether or not this virtual object is affected by gravity based on the read attribute information. Here, a configuration will be described in which a virtual object affected by gravity is not moved in the vertical direction from its original position.

[0068] If it is determined in S702 that the virtual object to be moved is affected by gravity, the process proceeds to S703. In S703, the control unit 111 transmits to the image generation unit 112 a movement instruction that reflects only the horizontal movement instruction received from the user in S701. For example, if the user tilts the HMD slightly upward, the image generation unit 112 moves the virtual object to a position corresponding to the horizontal movement instruction received from the user in S701, constructs a virtual space in which the virtual object is not moved in the vertical direction, and generates an image of this virtual space. This prevents the generation of an unnatural image of the virtual space, such as one in which a virtual object that is naturally positioned according to gravity appears to be floating in the air. The process then ends. The image of the virtual space generated by this process is transmitted to the image synthesis unit 110. The image synthesis unit 110 synthesizes the received image of the virtual space with an image of the real space captured by the imaging unit 106 to generate an MR image. The MR image is transmitted to the display device 101 and displayed on the display unit 107.

[0069] If it is determined in S702 that the virtual object to be moved is not affected by gravity, steps S704 to S707, which are the same processes as steps S302 to S305 described above, respectively, are performed, and this processing ends.

[0070] 3 and 7, a configuration has been described in which, when a virtual object movement event is detected (YES in S301 or S701), the control of the virtual object is changed based on whether the user is lying down. However, the present invention is not limited to this configuration. For example, it is conceivable that the user does not issue an instruction to move the virtual object, but simply looks up while wearing the HMD, or lies on their back. In such cases, how the virtual object should be moved and displayed is controlled according to the processing from S302 onward or the processing from S702 onward.

[0071] Furthermore, in this embodiment, the movement distance may be limited based on attribute information. In such a configuration, the restriction on the distance need not be equal in all directions, and only upward movement may be restricted, or the restriction on upward movement may be more or less severe than the restriction on other directions. Attribute information indicating such a restriction may be assigned to each virtual object, or the behavior of all virtual objects may be set uniformly as a system setting. Alternatively, the behavior of only virtual objects assigned specific attribute information may be changed in the system setting. For example, the upward movement of a virtual object assigned attribute information that is affected by gravity may be restricted in the system setting.

[0072] Furthermore, in this embodiment, a configuration may be adopted in which a plurality of different pieces of attribute information are added to each virtual object. For example, attribute information indicating the coordinate system set in FIG. 6(b) and attribute information indicating whether or not the virtual object is affected by gravity as described above are added to each virtual object. Note that, as a method for realizing the addition of a plurality of pieces of attribute information with a small amount of data, for example, the attribute information is represented by 1 byte and divided into 2-bit units, so that four pieces of attribute information are independently represented by 1 byte. Note that, although the example shown here shows the attribute information represented by 1 byte or 2 bits, it is of course also easily conceivable to represent the attribute information by multiple bytes or other bits.

[0073] Furthermore, in the above-described embodiment, a configuration in which the information processing device 102 is separate from the display device 101 has been described, but the present invention is not limited to this configuration. For example, the information processing device 102 and the display device 101 may be integrated, that is, the display device 101 may include the information processing device 102. This allows the virtual object to be appropriately controlled in accordance with the user's posture in a configuration in which the information processing device 102 and the display device 101 are integrated.

[0074] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0075] The present invention can also be realized by a process in which a program that realizes one or more functions of the present embodiment is supplied to a system or device via a network or a storage medium, and one or more general-purpose processors ASICs in the computer of the system or device read and execute the program. The present invention can also be realized by a dedicated processor (e.g., an ASIC or FPGA) that realizes one or more functions. Furthermore, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that the term "processor" here refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. Furthermore, the process that realizes the present invention may be performed by a single processor alone, or may be performed by multiple processors located in physically separate locations in cooperation with each other.

[0076] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An information processing device that displays an image of a virtual space on a display device worn or held by a user, comprising a determination means that determines whether the user is lying down or not, and a control means that causes the display device to display the image of the virtual space, wherein the control means changes the control of virtual objects included in the virtual space based on the determination result by the determination means. (Configuration 2) The information processing device according to configuration 1, wherein the control of the virtual object is display control for displaying the virtual object on the display device. (Configuration 3) The information processing device according to Configuration 2, characterized in that in the display control, the coordinate system followed by the virtual object differs when it is determined that the user is lying down and when it is determined that the user is not lying down. (Configuration 4) The information processing device according to Configuration 3, wherein the coordinate system is set by the user. (Configuration 5) The information processing device according to any one of configurations 1 to 4, wherein the control of the virtual object is an operation control for manipulating the virtual object. (Configuration 6) The information processing device described in Configuration 5, characterized in that the control means performs a predetermined control to move the virtual object to a position in front of the user in the virtual space when it is determined that the user is lying down. (Configuration 7) The information processing device according to configuration 6, wherein the predetermined control is a control for resetting the display position of the virtual object to a position directly in front of the user in the virtual space in accordance with an instruction received from the user. (Configuration 8) The information processing device according to any one of configurations 5 to 7, wherein the control means performs control to suppress rotation of the virtual object when it is determined that the user is lying down. (Configuration 9) The information processing device according to any one of configurations 1 to 8, characterized in that attribute information of the virtual object is assigned to the virtual object in advance, and the control means changes control of the virtual object based on the determination result by the determination means and the attribute information. (Configuration 10) The information processing device according to any one of configurations 1 to 9, wherein the information processing device is a device separate from the display device. (Configuration 11) The information processing device according to any one of configurations 1 to 9, characterized in that the information processing device is included in the display device. [Explanation of symbols]

[0077] 101 Display device 107 Display section 102 Information processing equipment 110 Image synthesis unit 111 Control Unit 112 Image generation unit 114 Position and orientation calculation unit 115 Display reset processing unit 116 Object movement processing unit

Claims

1. An information processing device that displays an image of a virtual space on a display device worn or held by a user, a determination means for determining whether the user is lying down; a control means for causing the display device to display an image of the virtual space, The information processing apparatus is characterized in that the control means changes control of virtual objects included in the virtual space based on the determination result by the determination means.

2. 2. The information processing apparatus according to claim 1, wherein the control of the virtual object is display control for displaying the virtual object on the display device.

3. 3. The information processing device according to claim 2, wherein in the display control, the coordinate system to which the virtual object conforms differs depending on whether it is determined that the user is lying down or not.

4. 4. The information processing apparatus according to claim 3, wherein the coordinate system is set by the user.

5. The information processing apparatus according to claim 1 , wherein the control of the virtual object is an operation control for manipulating the virtual object.

6. 6. The information processing device according to claim 5, wherein the control means performs predetermined control to move the virtual object to a position in front of the user in the virtual space when it is determined that the user is lying down.

7. 7. The information processing apparatus according to claim 6, wherein the predetermined control is a control for resetting the display position of the virtual object to a position directly in front of the user in the virtual space in accordance with an instruction received from the user.

8. The information processing apparatus according to claim 5 , wherein the control means performs control to suppress rotation of the virtual object when it is determined that the user is lying down.

9. attribute information of the virtual object is assigned to the virtual object in advance; 2. The information processing apparatus according to claim 1, wherein the control means changes control of the virtual object based on the determination result by the determination means and the attribute information.

10. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus is a device separate from the display device.

11. The information processing device according to claim 1 , wherein the information processing device is included in the display device.

12. A method for controlling an information processing device that displays an image of a virtual space on a display device worn or held by a user, comprising: a determining step of determining whether the user is lying down; a control step of causing the display device to display an image of the virtual space, The control method for an information processing device, wherein the control step changes control of a virtual object included in the virtual space based on the determination result in the determination step.

13. A program that causes a computer to execute a control method for an information processing device that displays an image of a virtual space on a display device worn or held by a user, The control method for the information processing device includes: a determining step of determining whether the user is lying down; a control step of causing the display device to display an image of the virtual space, The control step changes control of a virtual object included in the virtual space based on the determination result in the determination step.

Images