Information processing apparatus, information processing system, information processing program, and information processing method

Abstract

In a state in which occlusion is applied, depending on the arrangement of the virtual object, the visibility of the virtual object may be reduced by being hidden by the real object, and the user experience in the MR space may be impaired.SOLUTION: The information processing apparatus includes an acquisition unit configured to acquire information about an arrangement of a real object in a real space, an image acquisition unit configured to acquire a captured image obtained by capturing an image of the real space in a predetermined direction, and a control unit configured to, in a case where a position of the virtual object in a mixed reality space is a first position at which at least a part of the virtual object is at a position behind the real object or at a position embedded inside the real object, determine whether the position of the virtual object in the mixed reality space is: Control is performed to generate an image of the mixed reality space at a second position that is a position in front of the real object when viewed from the predetermined direction.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device that displays virtual objects to a user in a virtual space or a mixed reality space. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly blends the real world and the virtual world in real time. One example of MR technology is an MR system that uses a video see-through head-mounted display (HMD). In an MR system, a camera built into the HMD captures an image of a subject that roughly matches the subject observed from the user's pupil position. Then, a composite image in which CG (computer graphics, virtual objects) is superimposed on the captured image is presented to the user, allowing the user to experience an MR space (mixed reality space). One technique required to represent virtual objects as if they existed in reality is called occlusion. Occlusion is an image processing technique that calculates the anteroposterior relationship between virtual objects and real objects and renders virtual objects behind real objects as hidden or missing. For example, Patent Document 1 discloses an image processing technique that represents the anteroposterior relationship between virtual objects and real objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-30748 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when occlusion is applied, depending on the placement of a virtual object, the visibility of the virtual object may be reduced by being hidden by a real object, which may impair the user experience in the MR space.

[0005] The present invention has been made in consideration of these points, and aims to improve the user experience of an information processing device user in an MR space. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device comprising: an acquisition means for acquiring information regarding the arrangement of real objects in a real space; an image acquisition means for acquiring an image of the real space in a predetermined direction; and a control means for controlling the generation of an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein the control means controls the generation of an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction, when the position of the virtual object in the mixed reality space is a first position where at least a portion of the virtual object is behind the real object or embedded inside the real object. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the user experience of the information processing device user in the MR space. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating the configuration of an information processing device according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a real space in which a user exists in the first embodiment. [Figure 3]2 is a diagram illustrating a virtual space viewed by a user via an HMD in the first embodiment. FIG. [Figure 4] 10A and 10B are diagrams illustrating an example of a screen when an image in a virtual space is switched to an image in a mixed reality space in the first embodiment. [Figure 5] 3A to 3C are diagrams illustrating an example of a screen on which virtual objects are rearranged in an image of mixed reality space in the first embodiment. [Figure 6] 4 is a flowchart illustrating a detailed operation of the information processing device in the first embodiment. [Figure 7] 10 is a flowchart illustrating a detailed operation of the information processing device in the second modification of the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a confirmation screen for confirming with the user whether or not to rearrange virtual objects in the second modification of the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating a scene in which a virtual window is displayed indoors as a virtual object on a large screen in the second embodiment. [Figure 10] 10A and 10B are diagrams illustrating a case in which the virtual object rearrangement method of the first embodiment is applied to a virtual window 903 in the situation of FIG. 9 according to the second embodiment. [Figure 11] 10A and 10B are diagrams illustrating a case where the virtual object rearrangement method of the second embodiment is applied to a virtual window 903 in the situation of FIG. 9 according to the second embodiment. [Figure 12] FIG. 10 is a diagram illustrating a scene in which a virtual window is displayed indoors as a virtual object on a large screen in the second embodiment. [Figure 13] 12. FIG. 13 is a diagram illustrating a case in which the virtual object rearrangement method of the first embodiment is applied to a virtual window 1203 in the situation of FIG. 12 according to the second embodiment. [Figure 14] 13 is a diagram illustrating a case where the virtual object rearrangement method of the second embodiment is applied to a virtual window 1203 in the situation of FIG. 12 according to the second embodiment. FIG. [Figure 15] FIG. 11 is a diagram illustrating a scene in which three virtual windows are displayed indoors in MR mode in the third embodiment. [Figure 16] FIG. 10 is a diagram illustrating a scene in which three virtual windows have been rearranged according to the rearrangement method of the first embodiment in the third embodiment. [Figure 17] FIG. 15 is a diagram illustrating a scene in which the entire screen is rearranged using a virtual window 1505 as a reference for rearrangement in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) In this embodiment, a use case will be described in which a user wears a head-mounted display (HMD), which is an information processing device worn on the head, and experiences a mixed reality space. The user watches videos of movies, sports, etc. on a virtual display (virtual window) arranged in the mixed reality space. In addition, the description will be made assuming a situation in which the virtual window (virtual object) is relocated when at least a part of the virtual window is embedded in a wall.

[0011] <Internal configuration of information processing device> The internal configuration of an information processing device according to a first embodiment of the present invention will be described below with reference to Fig. 1. In this embodiment, the information processing device A100 will be described assuming that it is an HMD.

[0012] The information processing device A100 includes a control unit A101, a storage unit A102, a memory A103, an input unit A104, an output unit A105, a sensor A106, and a communication unit A107.

[0013] The control unit A101 is a CPU that controls each component of the information processing device A100. The control unit A101 combines a captured image captured by an imaging sensor (camera) with a virtual object to generate a composite image (an image of a mixed reality space). The control unit A101 also generates an image of a virtual space, which is an image that does not include real space and is composed only of CG, based on the captured image captured by the imaging sensor. For example, the control unit A101 identifies a floor and real objects from the captured image and determines the placement of the floor and virtual objects in the virtual space according to the positions of the real objects. The control unit A101 may also acquire information on the distances from two cameras to real objects as distance information and determine the placement of virtual objects based on the distance information. The control unit A101 controls the generated image of the mixed reality space and the image of the virtual space to be displayed on a display such as a liquid crystal panel or an organic EL panel. In addition, when an HMD is assumed as the information processing device A100, the control unit A101 controls the generation of an image for the right eye and an image for the left eye and the display of the respective images. The control unit A101 estimates the position or orientation of the camera (the position or orientation of the information processing device A100) from the captured image. In this embodiment, the position and orientation are estimated using a method for estimating the camera's own position and orientation, such as SLAM (Simultaneous Localization and Mapping). The position and orientation estimation unit 104 may estimate its own position using LiDAR SLAM (LiDAR; Light Detection And Ranging) using a laser, in addition to Visual SLAM, which estimates its own position from a camera image. The position and orientation estimation unit 104 may estimate its own position using Depth SLAM, which uses a ToF sensor (ToF; Time of Flight). Instead of the control unit A101 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0014] The storage unit A102 is an electrically erasable and recordable non-volatile storage medium such as an SSD (Solid State Drive) or flash memory, and stores programs executed by the control unit A101, databases, user settings, etc.

[0015] The memory A103 is, for example, a RAM (Random Access Memory), and is used as a buffer memory for temporarily storing various data, a work area for the control unit A101, and the like.

[0016] The input unit A104 is used to input instructions to the information processing device A100. The input unit A104 includes, for example, a power button for instructing the information processing device A100 to power on / off and operation buttons for instructing screen transitions. Note that the input unit A104 does not necessarily have to be built into the information processing device A100. It may also be configured so that input is made via a communication unit A107, which will be described later.

[0017] The output unit A105 is, for example, a display (display unit) viewed by the user, a GUI (Graphical User Interface) display for interactive operation, or a light-emitting device such as an LED (Light Emitting Diode). The output unit A105 may also include a sound output device such as a speaker, and a vibration device that provides a haptic effect to the user. The output unit A105 may be configured to output via a communication unit A107, which will be described later.

[0018] The sensor A106 is, for example, an imaging sensor that captures surrounding images, or a sensor such as a LiDAR (Light Detection And Ranging) or ToF (Time of Flight) that measures the surrounding conditions. Also, for example, the sensor is, for example, an IMU (Inertial Measurement Unit) or a geomagnetic sensor that measures the attitude and position of the information processing device A100. The sensor A106 does not necessarily need to be built into the information processing device A100.

[0019] The communication unit A107 is, for example, a NIC (Network Interface Card) incorporating a serial bus or parallel bus for connecting to other devices, a connector (RJ45) for connecting to Ethernet, a communication IC, etc. It is also a communication unit for wirelessly connecting to a controller or external devices. The communication unit A107 outputs a weighted wireless signal from an antenna and demodulates a wireless signal received by the antenna to achieve short-range wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). Note that the communication unit A107 may be wired, such as a USB cable (registered trademark), or wireless, such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0020] The terminal configuration has been described assuming that the information processing device A100 is an HMD, but the above-mentioned terminal configuration is merely an example, and the information processing device A100 does not have to be an HMD, and the terminal configuration may also be different. Note that the information processing device A100 may be an information processing device such as a PC (personal computer), tablet terminal, or smartphone connected to an HMD. Alternatively, the information processing device A100 may be an information processing system including each of the components of the information processing device A100 described above.

[0021] <Explanation of situations where virtual objects need to be rearranged> A method for determining whether a virtual object should be rearranged in this embodiment will be described below.

[0022] Whether or not a virtual object should be rearranged is determined using information about the position and size of the virtual object and spatial information such as the positions and sizes of surrounding walls, ceilings, furniture, etc.

[0023] Information about the position and size of the virtual object is grasped by a system or application that controls the display of the virtual object, and the control unit A101 acquires the information about the position and size of the virtual object from the system or application.

[0024] In addition, spatial information such as the positions and sizes of surrounding walls, ceilings, furniture, etc. in the real space must be acquired by capturing images of the surroundings and calculating them through spatial recognition. One method of spatial recognition involves measuring the distance to an object by irradiating it with laser light and measuring the reflected light, and then performing calculations through mapping. Another method involves extracting feature points from an image captured by a camera and combining them with the camera's movement to determine depth, and then making inferences based on the determined depth. Another known method involves using machine learning such as deep learning to estimate depth from an image captured by a camera and then performing calculations based on that. The control unit A101 acquires spatial information of the real space using these methods.

[0025] The control unit A101 calculates the front-to-back relationship between the virtual object and the real object as seen from the HMD in the mixed reality space using the position and size information of the virtual object and the spatial information of the real space acquired by the control unit A101. In other words, it determines which object is in front as seen from the imaging direction of the camera in the mixed reality space.

[0026] If the virtual object is in front, it can be determined that there is no need to rearrange the virtual object because the virtual object will not be obscured by surrounding walls, ceilings, furniture, etc.

[0027] If the virtual object is behind the real object or if the virtual object is hidden by the surrounding walls, ceiling, furniture, etc., it is determined that at least a part of the virtual object will become invisible and that relocation is necessary. In other words, it is determined that relocation is necessary if the virtual object interferes with the surrounding real objects such as the walls, ceiling, furniture, etc. Or, it is determined that relocation is necessary if at least a part of the virtual object is embedded in the surrounding real objects such as the walls, ceiling, furniture, etc.

[0028] The above-described method for determining whether a virtual object should be rearranged is merely an example, and the method for determining whether a virtual object should be rearranged may be different.

[0029] Hereinafter, a real space in which a user exists in the first embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 illustrates a bird's-eye view of the real space in which user 202 exists, assuming a scene in which user 202 exists in room 201 partitioned by a wall. It is assumed that room 201 in which user 202 exists is equipped with a plurality of interior items (real objects) including a television 203, a sofa 204, a table 205, a shelf 206, and a sofa 207. It is also assumed that user 202 is sitting on sofa 207 and facing the television 203.

[0030] Hereinafter, with reference to FIG. 3, a virtual space viewed by a user via an HMD in the first embodiment of the present invention will be described. FIG. 3 illustrates a scene in which an avatar 302, representing the user 202, exists in the virtual space 301, and shows a bird's-eye view of the virtual space in which the avatar 302 exists. The virtual space 301 can be expressed as if it were an expansive space unrelated to real objects. In FIG. 3, the virtual space 301 is assumed to be a space larger than an area 303 corresponding to an area surrounded by walls of the room 201 in the real space in FIG. 2. A virtual object 304 and a virtual window 305 are arranged in the virtual space 301. The virtual object 304 is a virtual object superimposed on the sofa 207 in the room 201 in FIG. 2. The virtual window 305 is a virtual object whose width is larger than the room 201 (area 303) as seen from the user 202. In the virtual space 301, the user 202 can watch a movie, a sport, or other video in a virtual window 305 that is larger than the display that can be placed in the room 201.

[0031] Assume that, in a scene where such a virtual space is being viewed, user 202 switches from an image of the virtual space to an image of a mixed reality space in which virtual window 305 appears as if it were placed in real space.

[0032] Hereinafter, with reference to FIGS. 4(a), 4(b), and 4(c), a description will be given of the screen when switching from an image in virtual space to an image in mixed reality space in the first embodiment of the present invention.

[0033] 4(a) is a bird's-eye view of the mixed reality space in which the user 202 exists, assuming a scene in which the image is switched from the virtual space 301 in FIG. 3 to a mixed reality space image, leaving the virtual window 305. In the mixed reality space 401, the virtual window 305 placed in the virtual space 301 is placed in the real space at the same position and size as it was in the virtual space 301, and the mixed reality space 401 is expressed. Because the virtual window 305 is wider than the room 201 when viewed from the user, it extends beyond the walls of the room 201. In other words, both ends of the virtual window 305 are placed so as to be embedded in the walls of the room 201.

[0034] FIG. 4(b) is a diagram showing an image viewed by a user through an HMD in the scene shown in FIG. 4(a). To accurately represent the positional relationship between virtual objects and real objects, both ends of virtual window 305 are positioned so that they are embedded in the walls of room 201. Therefore, of virtual window 305, virtual window 402 is positioned within the mixed reality space. Furthermore, virtual windows 403 and 404, which correspond to both ends of virtual window 305, are positioned so that they are embedded in the walls in the mixed reality space. In FIG. 4(b), virtual windows 403 and 404, which are embedded in the walls, are also displayed. A face is displayed in virtual window 402, but part of the face is also displayed in virtual window 403, which is embedded in the wall. Note that a virtual object 412 representing the shadow of virtual window 402 is also positioned in the image of mixed reality space 401 so that the user can understand the depth of virtual window 402 in mixed reality space 401. Similarly, virtual object 413, which is the shadow of virtual window 403, and virtual object 414, which is the shadow of virtual window 404, are also placed in the image of mixed reality space 401. That is, control unit A101 combines virtual window 402, virtual window 403, virtual window 404, and virtual object 412, virtual object 413, and virtual object 414, which correspond to the shadows, with the captured image. In this way, an image of mixed reality space 401 is generated. In this way, if the image of mixed reality space 401 is generated so that virtual window 403 and virtual window 404, which are embedded in the wall, are also displayed, it may be difficult for the user to get the feeling that virtual window 305 is actually present in real space.

[0035] FIG. 4(c) is a diagram showing an image viewed by a user through an HMD in the scene shown in FIG. 4(a). In FIG. 4(b), virtual windows 403 and 404, which correspond to both ends of virtual window 305 embedded in the wall of the real space, are displayed in the image of mixed reality space 401. In FIG. 4(c), the virtual object embedded in the wall is not displayed. Here, the only virtual window displayed is virtual object 402, so virtual object 412, which corresponds to the shadow of virtual object 402, is also displayed. A face is displayed in virtual window 402, but part of the face is embedded in the wall, so part of the face is not visible. That is, the control unit A101 combines virtual window 402, virtual object 412, which corresponds to the shadow, with the captured image to generate an image of mixed reality space 401. If the image of mixed reality space 401 is generated in this manner, when the user switches from the virtual space to the mixed reality space, the user may not be able to view the entire virtual window 305 that was being viewed when the image of the virtual space was displayed, which may impair the user experience. For example, when video content is displayed in the virtual window 305, the part that is hidden in the wall may be overlooked.

[0036] <How to rearrange virtual objects> In this way, when a virtual object is displayed as shown in Figures 4(b) and 4(c), the virtual object is displayed in front of the real object. In other words, occlusion is adjusted so that the virtual object is not hidden by the real object. Here, according to the physical laws of real space, if the size of a virtual object in mixed reality space is fixed, the size of the virtual object will appear to increase as it approaches the user. In other words, the size of a virtual object in mixed reality space does not change, but when an image of the mixed reality space is generated, by increasing the size of the virtual object with the same angle of view in the mixed reality space, it appears to be getting closer to the user.

[0037] However, if the size of the virtual object increases when adjusting the occlusion, the area in which the real space is displayed in the image of the mixed reality space viewed by the user will decrease.

[0038] Therefore, when a virtual object is displayed so that it is in front of a real object, the size of the virtual object in the mixed reality space is reduced. In other words, when an image in the mixed reality space is generated, the size of the virtual object in the composite image is not changed, and the virtual object is displayed so that it is in front of the real object. When a virtual object is displayed in the mixed reality space in this way, the size of the virtual object is the same when viewed from the same position before and after adjusting the positional relationship between the real object and the virtual object. Here, before adjusting the positional relationship between the real object and the virtual object, the virtual object is farther from the user, and after adjusting the positional relationship between the real object and the virtual object, the virtual object is closer to the user. However, when viewed from the same distance from each virtual object before and after adjusting the positional relationship between the virtual objects, the size of the virtual object is smaller after adjusting the positional relationship between the virtual objects.

[0039] That is, although the apparent size of the virtual object remains the same from the perspective of a user in the same position, the size of the virtual object in the mixed reality space becomes smaller after the positional relationship of the virtual objects is adjusted. Similarly, with regard to the size of the virtual object synthesized by control unit A101, although the size of the virtual object in the synthesized image remains the same, the size of the virtual object in the mixed reality space becomes smaller after the positional relationship of the virtual objects is adjusted.

[0040] <Example of rearrangement> Hereinafter, with reference to FIGS. 5(a) and 5(b), a screen on which virtual objects are rearranged in an image of mixed reality space according to the first embodiment of the present invention will be described.

[0041] FIG. 5(a) illustrates a scenario in which, when a virtual object is displayed as in FIG. 4(b) or 4(c), the virtual window is translated forward and shrunk in the mixed reality space so that the virtual window's appearance to the user does not change. FIG. 5(a) also illustrates a bird's-eye view of the mixed reality space in which the user 202 exists. In the mixed reality space 501, the virtual object is moved from the position of virtual window 502, which was embedded in the wall, to the position of virtual window 503 in the foreground, and shrunk so that the apparent size to the user does not change. If the virtual object is moved closer to the user while maintaining its original size, the apparent size to the user increases. By moving the virtual object closer to the user while shrinking it, the apparent size to the user can be maintained unchanged. Because a bird's-eye view is assumed here, the size of the virtual object changes before and after rearrangement.

[0042] FIG. 5(b) is an example of a screen viewed by the user 202 in the scene described in FIG. 5(a). That is, FIG. 5(b) is an image of a mixed reality space generated by the control unit A101. In FIG. 5(b), a scene is assumed in which a virtual object is moved closer to the user at the same position and orientation as the user in FIG. 4(b) or 4(c). In FIG. 5(b), a virtual window 503 is positioned closer to the wall of the room than the virtual window in FIG. 4(b) or 4(c) and does not interfere with the wall. That is, at least a portion of the virtual window is not embedded in the wall. Furthermore, the shadow of the virtual window 503 was located at the position of the virtual window shadow 512 before the rearrangement. However, in this image of the mixed reality space, the virtual window shadow 512 is not composited, and a virtual window shadow 513 is drawn according to the rearranged position. In Fig. 5(b), a virtual window of the same size as the virtual window in Fig. 4(b) or 4(c) is synthesized on the image of the mixed reality space generated by the control unit A101. The positional relationship in the mixed reality space has changed. If the distance between the virtual window and the user in Fig. 4(b) is the same as the distance between the virtual window and the user in Fig. 5(b), the size of the virtual window seen by the user will be larger than that in Fig. 4(b).

[0043] By shrinking a virtual object in mixed reality space and moving it forward, a virtual object that was buried in a wall can be made to no longer be buried in the wall. Furthermore, by adjusting the shrinking ratio and movement distance, the appearance of the virtual object can be prevented from changing due to perspective before and after rearrangement. Because the appearance of the virtual object remains unchanged after rearrangement, this method minimizes degradation of the user experience due to virtual object rearrangement compared to other rearrangement methods that change the appearance of the virtual object.

[0044] However, if the user's angle of view does not change, i.e., if the user is in the same position and facing the same direction, the control unit A101 will essentially composite a virtual object of the same size with the captured image before and after rearrangement when generating an image in mixed reality space.

[0045] In this way, the method for rearranging virtual objects according to this embodiment is performed based on information about the position and size of the virtual object and values ​​of surrounding space information and the like.

[0046] In this embodiment, a method for rearranging a virtual object has been described in which the virtual object is scaled down in mixed reality space and moved closer to the user so that the appearance of the virtual object does not change.

[0047] Note that any affine transformation or other method may be used as long as it is a rearrangement method that does not adversely affect the user experience. For example, an image of the virtual object from the user's viewpoint may be generated without changing the size of the virtual object in the mixed reality space, and the image may be composited with the captured image.

[0048] <Virtual object rearrangement flow> Hereinafter, with reference to FIG. 6, the detailed operation of the information processing device in this embodiment will be described.

[0049] 6 is a flowchart of the information processing device A100 according to this embodiment. This flowchart starts when a system or application that controls the display of virtual objects is started.

[0050] In step S601, the control unit A101 determines whether to end the process based on the state of each unit of the information processing device A100. For example, the control unit A101 ends the process of this flowchart when an end operation is performed via the input unit A104. If it is determined that the process should be continued, the control unit A101 proceeds to step S602.

[0051] In step S602, the control unit A101 photographs the surroundings via the sensor A106 and acquires the image. Photographing the surroundings means, for example, photographing an image with a camera or measuring distance with laser light using LiDAR. The photographed information is recorded in the memory A103. Here, the control unit A101 activates the camera to acquire the captured image in real time, and proceeds to step S603.

[0052] In step S603, the control unit A101 recognizes the real space (space recognition) based on the surrounding data captured in step S602. The result of the space recognition performed here is space information such as the positions and sizes of real objects such as surrounding walls, ceilings, and furniture. After acquiring the space information, the control unit A101 proceeds to step S604.

[0053] In step S604, the control unit A101 obtains information about the display position and size of the virtual object from the memory A103, compares the information with the surrounding space information calculated in step S603, and determines whether the virtual object needs to be rearranged. If the control unit A101 determines that the virtual object needs to be rearranged, it proceeds to step S605. If the control unit A101 determines that the virtual object does not need to be rearranged, it proceeds to step S606. While the image of the virtual space is being displayed, the control unit A101 determines that the virtual object does not need to be rearranged because there is no interference with a real object. Furthermore, while the image of the mixed reality space is being displayed, the control unit A101 determines that the virtual object needs to be rearranged if there is interference with a real object or if the virtual object is hidden behind a real object.

[0054] In step S605, the control unit A101 updates the display position and size information of the virtual object stored in the memory A103 according to the above-described method for rearranging the virtual object, and rearranges the virtual object. When at least a part of the virtual object is embedded in (buried in) the real object, the control unit A101 rearranges the virtual object as if it were in front of the real object.

[0055] In step S606, the control unit A101 combines the virtual object read from the memory A103 with the captured image taken by the camera in step S602 to generate an image of mixed reality space. Alternatively, the control unit A101 generates an image of virtual space from the virtual object read from the memory A103. After generating the image of mixed reality space or the image of virtual space, which is an image for display, the control unit A101 proceeds to step S607.

[0056] In step S607, the control unit A101 displays the display image generated in step S606 via the output unit A105, and the process returns to step S601.

[0057] As described above, the information processing device A100 of this embodiment determines whether or not a virtual object needs to be rearranged based on the spatial information obtained by capturing an image of the surroundings and recognizing the space and the positional relationship of the virtual object. If rearrangement is necessary, the information processing device A100 of this embodiment rearranges the virtual object by shrinking it in the mixed reality space and moving it forward.

[0058] This allows the user to view a virtual object without changing its appearance even if the virtual object is relocated, assuming the same angle of view (same position and same viewing direction).This means that virtual objects that were previously invisible due to the surrounding environment can be relocated to a visible position while minimizing degradation of the user experience.

[0059] Note that if the virtual object is a two-dimensional virtual object such as a virtual screen, the thickness of the screen may not be reduced. That is, the vertical and horizontal lengths of the screen may be reduced before and after rearrangement, but the thickness may remain constant before and after rearrangement. Also, if the virtual object is a three-dimensional virtual object, it may be reduced three-dimensionally. Also, a virtual object that casts a shadow over the virtual object may not be displayed. Also, the shadow portion may be displayed in accordance with the reduction ratio and movement amount of the virtual object.

[0060] When switching from an image in virtual space to an image in mixed reality space, virtual objects that are only displayed in virtual space may not be subject to rearrangement, but virtual objects that continue to be displayed in the image in mixed reality space may be subject to rearrangement.

[0061] <Modification 1 of the First Embodiment> As described above, by following the flow in Figure 6, when it is necessary to rearrange a virtual object, an image of the mixed reality space in which the virtual object has been rearranged is displayed without displaying an image in which the virtual object and real object interfere with each other.

[0062] In a modified example, an image of a virtual object interfering with a real object, as shown in FIG. 4(b) or 4(c), may be displayed, followed by an image of a mixed reality space in which the virtual object has been rearranged. In this case, in step S604 of FIG. 6, the control unit A101 determines whether rearrangement of the virtual object is necessary and whether an image of a mixed reality space in which a virtual object interfering with a real object has been displayed once. If the control unit A101 determines that rearrangement of the virtual object is necessary and that an image of a mixed reality space in which a virtual object interfering with a real object has been displayed once, the control unit A101 proceeds to step S605. On the other hand, if the control unit A101 does not determine that rearrangement of the virtual object is necessary and that an image of a mixed reality space in which a virtual object interfering with a real object has not been displayed once, the control unit A101 proceeds to step S606. This allows the user to visually recognize the interference between the virtual object and the real object once and predict that the virtual object will be rearranged. Instead of determining whether an image of a mixed reality space in which a virtual object and a real object interfere with each other has been displayed once, it may be determined whether an image of a mixed reality space in which a virtual object and a real object interfere with each other has been displayed for a predetermined period of time.

[0063] <Modification 2 of the First Embodiment> In another modification 2, after displaying an image in which a virtual object and a real object interfere with each other, if it is determined that the virtual object needs to be rearranged, a screen may be displayed to confirm with the user whether or not to rearrange the virtual object. The flow in this case will be described with reference to FIG.

[0064] 7 is a flowchart of the information processing device A100 according to this embodiment. This flowchart starts when a system or application that controls the display of virtual objects is launched. Here, explanations of steps S601 to S603 and step S606, which are the same as those in the flow of FIG. 6, will be omitted.

[0065] In step S604, the control unit A101 acquires information about the display position and size of the virtual object from the memory A103, compares it with the surrounding space information calculated in step S603, and determines whether rearrangement of the virtual object is necessary. Furthermore, in step S702, which will be described below, the control unit A101 determines whether or not the user has instructed not to rearrange the virtual object. If an instruction not to rearrange the virtual object has been received, the control unit A101 determines that rearrangement of the virtual object is unnecessary; if an instruction not to rearrange the virtual object has not been received, the control unit A101 determines that rearrangement of the virtual object is necessary. If the control unit A101 determines that rearrangement of the virtual object is necessary, the control unit A101 proceeds to step S701; if the control unit A101 determines that rearrangement of the virtual object is unnecessary, the control unit A101 proceeds to step S606.

[0066] In step S701, the control unit A101 determines whether or not the user has issued an instruction to rearrange the virtual objects. If the control unit A101 determines that the user has issued an instruction to rearrange the virtual objects, the control unit A101 proceeds to step S605. If the control unit A101 does not determine that the user has issued an instruction to rearrange the virtual objects, the control unit A101 proceeds to step S702. For example, if the user issues an instruction to rearrange the virtual objects on a confirmation screen generated in the following step S703 for asking the user whether or not to rearrange the virtual objects, the control unit A101 determines that the user has issued an instruction to rearrange the virtual objects. Furthermore, if the user does not issue an instruction to rearrange the virtual objects on a confirmation screen generated in the following step S703 for asking the user whether or not to rearrange the virtual objects, the control unit A101 does not determine that the user has issued an instruction to rearrange the virtual objects.

[0067] Here, an example of a confirmation screen that asks the user whether or not to rearrange the virtual objects will be described with reference to FIGS. 8(a) and 8(b).

[0068] In FIG. 8(a), the portion of the virtual object that is embedded in the wall is also composited with the image of the mixed reality space. For example, the image of FIG. 8(a) may be displayed after the image of the mixed reality space of FIG. 4(b) is displayed. Alternatively, the image of FIG. 8(a) may be displayed without first displaying the image of the mixed reality space of FIG. 4(b). In the image 801 of the mixed reality space of FIG. 8(a), a confirmation screen 802 that asks the user whether or not to rearrange the virtual object is further composited with the image of the mixed reality space of FIG. 4(b). In addition, a virtual object 812 that is a shadow of the confirmation screen 802 and a virtual object 813 that indicates the user's designated position are composited with the image 801 of the mixed reality space. In addition, the confirmation screen 802 displays a button that allows the user to select whether or not to rearrange the virtual window that is embedded in the wall. The user can instruct the rearrangement of the virtual object by moving the virtual object 813 to the "Yes" position and performing a confirmation operation. Furthermore, the user can issue an instruction not to rearrange the virtual object by moving the virtual object 813 to the "No" position and performing a confirm operation.

[0069] In FIG. 8(b), if a virtual object has a portion that is embedded in a wall, the virtual object is not composited into the image of the mixed reality space. Here, a confirmation screen 802 that asks the user whether or not to rearrange the virtual object, a virtual object 812 that is the shadow of the confirmation screen 802, and a virtual object 813 that indicates the user's designated position are composited into the captured image. For example, the image of FIG. 8(a) may be displayed after the image of the mixed reality space of FIG. 4(b) is displayed. Alternatively, the image of FIG. 8(a) may be displayed without first displaying the image of the mixed reality space of FIG. 4(b).

[0070] In step S702, the control unit A101 determines whether or not the user has issued an instruction not to rearrange the virtual objects. If the control unit A101 determines that the user has issued an instruction not to rearrange the virtual objects, the control unit A101 proceeds to step S606. If the control unit A101 does not determine that the user has issued an instruction not to rearrange the virtual objects, the control unit A101 proceeds to step S703. For example, if the user issues an instruction not to rearrange the virtual objects on a confirmation screen generated in the following step S703 to ask the user whether or not to rearrange the virtual objects, the control unit A101 determines that the user has issued an instruction not to rearrange the virtual objects. Furthermore, if the user does not issue an instruction not to rearrange the virtual objects on a confirmation screen generated in the following step S703 to ask the user whether or not to rearrange the virtual objects, the control unit A101 does not determine that the user has issued an instruction not to rearrange the virtual objects. In other words, if the user has not yet issued an instruction to rearrange or not to rearrange the virtual objects on a confirmation screen generated in the following step S703 to ask the user whether or not to rearrange the virtual objects, the control unit A101 proceeds to step S703.

[0071] Furthermore, if the control unit A101 determines that the user has instructed not to rearrange the virtual objects, then in the subsequent step S604, it determines that there is no need to rearrange the virtual objects.

[0072] For example, on the screens of Figures 8(a) and 8(b), the user can issue an instruction to rearrange the virtual object by moving the virtual object 813 to the "Yes" position and performing a confirm operation. Also, the user can issue an instruction not to rearrange the virtual object by moving the virtual object 813 to the "No" position and performing a confirm operation. Also, a case where the user has not yet performed a confirm operation corresponds to a case where the user has not yet issued an instruction to rearrange or an instruction not to rearrange.

[0073] In step S703, the control unit A101 generates a display image on which a confirmation screen for the user is superimposed. The control unit A101 combines the virtual object read from the memory A103, the virtual object corresponding to the confirmation screen for the user, and the captured image captured by the camera in step S602 to generate an image of a mixed reality space. After generating the image of the mixed reality space, which is the display image, the control unit A101 proceeds to step S607. For example, the control unit A101 generates an image of the mixed reality space as shown in FIGS. 8(a) and 8(b) described above as a display image. Note that when it is determined that rearrangement of a virtual object is necessary, if a display image on which a confirmation screen for confirming whether or not to rearrange the virtual object is superimposed is generated, the virtual object that needs to be rearranged may be displayed or not. Furthermore, when the virtual object that needs to be rearranged is also displayed, portions of the virtual object that are hidden or embedded in the real object may not be displayed, or the hidden or embedded portions may be displayed transparently. For example, if a virtual object that needs to be rearranged is also to be displayed, and parts of the virtual object that are hidden by or embedded in a real object are not to be displayed, an image of a mixed reality space is generated in which a confirmation screen is also superimposed on the image of Figure 4(c).

[0074] In addition, when there is a virtual window hidden by a real object, that is, when there is a virtual window placed behind a real object, a confirmation screen may be displayed to confirm whether or not to rearrange the virtual window hidden by the real object.

[0075] This makes it possible to determine whether to rearrange the virtual object in accordance with a user instruction when the virtual object is positioned at a position where at least a part of the virtual object is obscured by a real object.

[0076] (Second embodiment) In the first embodiment, a method for rearranging a virtual object by shrinking it in mixed reality space without changing the appearance of the virtual object and moving it closer to the user has been described. However, depending on the user's position and the arrangement of walls, ceilings, furniture, etc., the rearrangement may result in the virtual object being reduced too much or being moved too far forward, making it appear too close.

[0077] If the virtual object is reduced too much, it may become difficult to perform touch operations, gaze input, ray input, etc. Also, if the virtual object is too close, it may block the view and make it impossible to see the real object in front of you.

[0078] Therefore, in this embodiment, the operation of the information processing device A100 (HMD) when restrictions are placed on the reduction ratio and movement amount when a virtual object is rearranged will be described. Note that this embodiment has many parts in common with the first embodiment, so the description will focus on parts unique to this embodiment.

[0079] A method for determining whether a virtual object should be rearranged in this embodiment will be described below.

[0080] As in the first embodiment, whether to rearrange a virtual object is determined using spatial information such as the position and size of the virtual object and the positions and sizes of surrounding walls, ceilings, furniture, etc. First, using spatial information such as the positions and sizes of the virtual object and surrounding walls, ceilings, furniture, etc., it is calculated which of the virtual object or real object is closer from the perspective of the HMD (or user). In other words, it is calculated which of the virtual object or real object is closer from the perspective of the imaging position.

[0081] If the virtual object is closer to the real object, it can be determined that there is no need to rearrange the virtual object because the virtual object will not be obscured by surrounding walls, ceilings, furniture, etc.

[0082] Next, if the virtual object is behind the real object or if the virtual object is hidden by surrounding walls, ceilings, furniture, etc., a determination is made based on the values ​​of the reduction ratio and movement amount of the virtual object. When attempting to rearrange the virtual object in front of the real object, if both the reduction ratio and movement amount values ​​upon rearrangement do not exceed the limits (thresholds), it is determined that the virtual object should be rearranged. When attempting to rearrange the virtual object in front of the real object, if both the reduction ratio and movement amount values ​​upon rearrangement exceed the limits (thresholds), it is determined that the virtual object should not be rearranged. In other words, if the virtual object is reduced more than the reduction ratio limit or moved more than the movement amount limit upon rearrangement, it is determined that the virtual object should not be rearranged at the position where it was intended to be rearranged. Note that when the virtual object is rearranged, if the reduction ratio is reduced more than the reduction ratio limit or moved more than the movement amount limit, it is only reduced until the reduction ratio reaches the limit and it is only moved until both the movement amount values ​​reach the limit.

[0083] Here, "the reduction rate has reached its limit" means that the virtual object is in the mixed reality space at the reduced size within the set limits, and "the movement amount has reached its limit" means that the virtual object is positioned at the closest position to the HMD within the set limits.

[0084] In the above-described method for determining whether to rearrange a virtual object, if the virtual object is buried in a real object or if the values ​​of the reduction rate and movement amount reach their limits, the virtual object is rearranged to a position where the values ​​of the reduction rate and movement amount reach their limits. In this way, by moving the virtual object forward while shrinking it to its limit, it may be possible to display a part of the virtual object in front of the surrounding walls, ceiling, furniture, etc. Alternatively, the visibility of the virtual object may be improved compared to before the rearrangement.

[0085] The above-described method for determining whether a virtual object should be rearranged is merely an example, and the method for determining whether a virtual object should be rearranged may be different.

[0086] A method for rearranging a virtual object in this embodiment will be described below, taking examples of a rearrangement method in which limitations are placed on the reduction ratio and movement amount when rearranging a virtual object.

[0087] <First Example of Second Embodiment> As a first example, a method for rearranging a virtual object when a limit is placed on the reduction ratio will be described below using Figures 9, 10, and 11. Here, a limit is placed on the reduction ratio, but no limit is placed on the amount of movement. Therefore, the virtual object is placed in front of the real object after being reduced to the limit of the reduction ratio.

[0088] FIG. 9 is a diagram illustrating a scene in which a virtual window is displayed as a virtual object on a large screen indoors. The virtual window 903 is a virtual object that is treated as a target for rearrangement in this embodiment. In FIG. 4 used for the explanation of the first embodiment, the user is using an HMD while sitting on a sofa, and there is a sufficient distance to the wall in front of them. However, in FIG. 9 of this embodiment, the user is using an HMD while sitting at a counter table, and the distance to the wall in front of them is short. In this situation, the virtual window 903 is displayed as if it exists behind the wall of a real object in front of the user.

[0089] Fig. 10 is a diagram illustrating a case where the virtual object rearrangement method of the first embodiment is applied to a virtual window 903 in the situation of Fig. 9. The virtual window 903 is moved so as to be closer to the front wall by the rearrangement, and its size is reduced to approximately one-seventh of its original size. That is, in Fig. 10, it is placed at the position of a virtual window 1003. If the virtual object shrinks too much like this, it may become difficult to perform inputs using touch operations, eye-gaze inputs, or ray operations.

[0090] FIG. 11 is a diagram illustrating a case where the virtual object rearrangement method of the second embodiment is applied to a virtual window 903 in the situation of FIG. 9. Here, a case where the lower limit of the virtual object reduction rate is set to half of the original size is described. The virtual window 903 is still moved by rearrangement so that it is closer to the front wall, but its size is reduced to only half of its original size. The virtual object rearrangement method of this embodiment changes the appearance of the virtual object, but it can prevent the virtual object from shrinking so much that it becomes difficult to perform touch operations, eye-gaze input, ray input, and the like.

[0091] <Second Example of Second Embodiment> As a second example, a method for rearranging a virtual object when a limit is placed on the amount of movement will be described below with reference to Figures 12, 13, and 14. Here, a limit is placed on the amount of movement, but the reduction ratio is adjusted to match the arrangement of the real object.

[0092] Fig. 12 is a diagram illustrating a scene in which a virtual window is displayed indoors on a large screen as a virtual object. A virtual window 1203 is a virtual object that is treated as a target for rearrangement in this embodiment, and is placed in a mixed reality space 1201. Fig. 12 is a diagram showing a bird's-eye view of the mixed reality space 1201 in which a user 1202 exists.

[0093] In Fig. 4 used for explanation in the first embodiment, the user is using the HMD while sitting on a sofa, and there is a sufficient distance between the left and right walls. However, in Fig. 12 of this embodiment, the user is using the HMD in a small room such as a toilet, and the left and right walls are close by.

[0094] FIG. 13 is a diagram illustrating a case where the virtual object rearrangement method of the first embodiment is applied to a virtual window 1203 in the situation of FIG. 12. A virtual window 1303 is a virtual object rearranged in this embodiment and is placed in a mixed reality space 1301. FIG. 13 is a diagram showing a bird's-eye view of the mixed reality space 1301 in which a user 1202 exists. The virtual window 1203 is moved so that it is closer to the left and right walls and the toilet paper holder, and its size is reduced accordingly, and it is placed at the position and size of the virtual window 1303. If the virtual object is rearranged in this way to a position too close to the user, there is a risk that the virtual object will block the user's view and make it impossible to see the real object in front of the user.

[0095] FIG. 14 is a diagram illustrating a case where the virtual object rearrangement method of the second embodiment is applied to a virtual window 1203 in the situation of FIG. 12. Here, a case where the upper limit of the movement amount of the virtual object is set to half the distance from the user is described. A virtual window 1403 is a virtual object rearranged in this embodiment and is arranged in a mixed reality space 1401. FIG. 14 is a diagram illustrating a top view of the mixed reality space 1401 in which a user 1202 exists. The virtual window 1203 is moved only to a position half the distance from the user, which corresponds to the upper limit of the movement amount, and is arranged at the position of the virtual window 1303. The size of the virtual window 1203 is also reduced to a size that will not be obscured by the left and right walls or the toilet paper holder, and is arranged at the size of the virtual window 1303. Although the virtual object rearrangement method of this embodiment changes the appearance of the virtual object, it is possible to prevent the virtual object from obscuring the view of a real object in front of the user.

[0096] As in the two examples described above, the method for rearranging a virtual object in the second embodiment can be determined from information about the position and size of the virtual object, information about the surrounding space, and values ​​such as the reduction ratio and movement amount limit at the time of rearrangement.

[0097] The above-described rearrangement of the virtual object is performed, for example, in the process of step S605 in FIG. 6 or FIG. 7 described in the first embodiment.

[0098] In the second embodiment, a limit (threshold) is set for the amount of movement from the original position of the virtual object, but this is not limiting. Because there is a possibility that the virtual object will block the view and make it impossible to see the real object in front of the user, a limit (threshold) may be set for the position at which the virtual object is rearranged, depending on the distance from the HMD or the user. In other words, even if the virtual object is rearranged, the position at which the virtual object is rearranged will be farther away from the HMD or the user than a predetermined distance.

[0099] The above-described method for rearranging virtual objects is merely an example, and the method for rearranging virtual objects and the restrictions on the reduction ratio and movement amount during rearrangement may be different.

[0100] As described above, in the second embodiment, the operation of the information processing apparatus A100 when limitations are placed on the reduction ratio and movement amount when a virtual object is rearranged has been described.

[0101] By using the method for determining whether to rearrange a virtual object and the method for rearranging a virtual object described in the second embodiment, it is possible to prevent input to the virtual object from becoming difficult and to prevent the virtual object from blocking the view.

[0102] (Third embodiment) In the first and second embodiments, the explanation has been given on the assumption that only one virtual object is displayed, but depending on the use case, it may be possible to simultaneously display and use multiple virtual objects.

[0103] In such use cases, if only the virtual objects that need to be rearranged are rearranged, the relative positions of the virtual objects with respect to other virtual objects will change, confusing the user and causing a deterioration in the user experience.Furthermore, in use cases where multiple virtual objects are displayed, there may be virtual objects that should be excluded from the rearrangement target in consideration of the user experience.

[0104] Therefore, in this embodiment, a method for rearranging virtual objects while suppressing degradation of the user experience when multiple virtual objects are displayed simultaneously will be described.

[0105] Since this embodiment has many commonalities with the first and second embodiments, the following description will focus on the unique features of this embodiment.

[0106] A method for determining whether a virtual object is to be rearranged in this embodiment will be described below.

[0107] In the first and second embodiments, the explanations have been given on the assumption that there is always one virtual object that appears and that it is always subject to rearrangement. In this embodiment, the explanation will be given on the assumption that multiple virtual objects are displayed, and that among these, there are virtual objects that should be excluded from the subject of rearrangement.

[0108] There are several possible methods for determining whether a virtual object is a target for rearrangement. For example, a method in which the user sets whether a virtual object is a target for rearrangement. The user determines for each virtual object whether the virtual object is a target for rearrangement, and this is recorded as a user setting in the memory A103. The information processing device A100 can determine whether a virtual object is a target for rearrangement by checking the user setting recorded in the memory A103.

[0109] Another method is to make the judgment based on the type of virtual object. In many cases, the user experience does not deteriorate even if a virtual window is reduced in size when it is relocated, but reducing a 3DCG displayed at actual size can have an impact on the user experience. For this reason, this method determines that if the type of virtual object is a virtual window, it is a target for relocation, and if it is 3DCG, it is not a target for relocation.

[0110] Another method is to make a judgment based on the situation in which the virtual object is placed. An image of the virtual space is displayed, and whether the virtual object is to be subject to relocation is determined based on whether it was placed in VR mode, which normally does not include the real space in the image, or in MR mode, which displays an image in mixed reality space. In the image of the mixed reality space, the virtual object is placed as if it exists in real space. When a virtual object is placed in MR mode, it is likely that the virtual object is placed based on the real object as the starting point, and if the virtual object is relocated, the virtual object may move to a position that does not match the user's intention. For this reason, a judgment method is used in which virtual objects placed in VR mode are subject to relocation, but virtual objects placed in MR mode are not subject to relocation.

[0111] Although several specific examples have been shown for the method of determining whether a virtual object is to be rearranged, one of these determination methods may be used, or a combination of multiple methods may be used. Furthermore, the above-described determination methods are merely examples, and different methods may be used to determine whether a virtual object is to be rearranged.

[0112] <How to determine whether to reposition a virtual object> A method for determining whether a virtual object should be rearranged in this embodiment will be described below.

[0113] First, from among the plurality of virtual objects being displayed, a virtual object to be treated as a target for rearrangement is identified by the method for determining whether or not the virtual object is a target for rearrangement described above.

[0114] Next, one virtual object to be used as the basis for rearrangement is selected from among the virtual objects to be rearranged. Any virtual object may be selected as the basis for rearrangement, but the behavior at the time of rearrangement will change depending on the virtual object selected. How the behavior at the time of rearrangement changes will be explained with an example in the explanation of the method for rearranging virtual objects, which will be described later.

[0115] A determination is made as to whether the virtual object selected as the reference for rearrangement should be rearranged. The method of determining whether the virtual object should be rearranged is the same as that of the first or second embodiment. If no limitations are imposed on the reduction ratio or movement amount when the virtual object is rearranged, the method of the first embodiment is used, and if limitations are imposed, the method of the second embodiment is used.

[0116] If it is determined that the virtual object selected as the rearrangement reference should be rearranged, it is determined that all virtual objects, including other virtual objects, should be rearranged.If it is determined that the virtual object selected as the rearrangement reference does not need to be rearranged, it is determined that all virtual objects, including other virtual objects, do not need to be rearranged.

[0117] The above-described method for determining whether a virtual object should be rearranged is merely an example, and the method for determining whether a virtual object should be rearranged may be different.

[0118] Hereinafter, a method for rearranging virtual objects in the third embodiment will be described with reference to FIGS.

[0119] Fig. 15 is a diagram illustrating a scene in which three virtual windows are displayed indoors in MR mode. Fig. 15 is a diagram showing a top-down view of a mixed reality space 1501 in which a user 1502 is present and in which, from left to right, virtual windows 1503, 1504, and 1505 are arranged. Virtual window 1503, virtual window 1504, and virtual window 1505 are all treated as candidates for rearrangement.

[0120] In explaining the rearrangement method of this embodiment, for comparison, we will explain the case where the rearrangement method of the first embodiment is applied under the same conditions. Here, for each of multiple virtual objects, it is determined whether there are parts that are hidden by real objects and cannot be seen, or whether there are parts that are buried, and then it is determined whether to rearrange them.

[0121] Fig. 16 is a diagram illustrating a scene in which three virtual windows have been rearranged according to the rearrangement method of the first embodiment. Fig. 16 is a diagram showing a top view of a mixed reality space 1601 in which a user 1502 is present and in which, from left to right, virtual windows 1603, 1504, and 1605 are arranged. In Fig. 16, virtual window 1503 and virtual window 1505 have been rearranged and are now located at the positions of virtual window 1603 and virtual window 1605, respectively.

[0122] Because the left side of virtual window 1503 is embedded in the wall, it is moved forward while shrinking slightly so that it is no longer embedded in the wall. Because virtual window 1503 is moved to the position of virtual window 1603 through rearrangement, it becomes approximately two-thirds its size, and the distance from the user becomes approximately two-thirds of its original distance. Next, virtual window 1504 does not need to be rearranged because it is not embedded in the wall, so it is not reduced in size or moved. Then, virtual window 1505 has its right side embedded in the wall and is hidden by shelf 1606 in the foreground. Therefore, it needs to be further shrunk than virtual window 1503 and moved further forward. Because virtual window 1505 is moved to the position of virtual window 1605 through rearrangement, it becomes approximately half its size, and the distance from the user becomes approximately half of its original distance.

[0123] As described above, when the rearrangement method of the first embodiment is applied individually to each virtual object, if the user is in the same position before and after the rearrangement, the size of the virtual object appears to be unchanged. However, when the rearrangement method of the first embodiment is applied individually to each virtual object, three virtual objects that should have been at the same depth are rearranged to different positions. If the user intentionally places the virtual objects with the same depth, this rearrangement method can be a factor that deteriorates the user experience.

[0124] <Relocation Method in the Third Embodiment> Therefore, in this embodiment, the rearrangement method of other virtual objects is determined in accordance with the rearrangement method of the virtual object selected as the reference.

[0125] Fig. 17 is a diagram illustrating a scene in which an entire space has been rearranged using virtual window 1505 as the reference for rearrangement. Fig. 17 is a diagram showing a top view of mixed reality space 1701 in which user 1502 is present and in which, from left to right, virtual window 1703, virtual window 1704, and virtual window 1705 are arranged. In Fig. 17, virtual window 1503, virtual window 1504, and virtual window 1505 have been rearranged and are now located at the positions of virtual window 1703, virtual window 1704, and virtual window 1705, respectively.

[0126] Since the left side of virtual window 1505 is hidden by the wall, it is shrunk so that it is no longer hidden by the wall, and is moved forward and placed in the position of virtual window 1705. After rearrangement, virtual window 1505 becomes approximately half its size, and the distance from the user is also approximately half of the original distance.

[0127] Virtual windows 1503 and 1504 are moved forward while shrinking by the same amount as virtual window 1505 in accordance with the shrink ratio and movement amount when virtual window 1505 is rearranged, and are thereby placed at the positions of virtual windows 1703 and 1704, respectively.

[0128] In this case, even if virtual window 1504 is selected as the reference, virtual window 1503 and virtual window 1505 are not rearranged and remain embedded in the wall. Also, even if virtual window 1503 is selected as the reference, virtual window 1505 is rearranged and is no longer embedded in the wall, but is embedded in shelf 1606. Therefore, virtual window 1505 is selected as the reference.

[0129] As described above, in this embodiment, by determining the rearrangement method of other virtual objects in accordance with the rearrangement method of the virtual object selected as the reference, it is possible to maintain the relative positional relationships between virtual objects before and after rearrangement.

[0130] The above-described rearrangement of the virtual object is performed, for example, in the process of step S605 in FIG. 6 or FIG. 7 described in the first embodiment.

[0131] As described above, in this embodiment, the operation of the information processing device A100 when multiple virtual objects are displayed simultaneously has been described.

[0132] By using the method described in this embodiment, the relative positional relationships between virtual objects can be maintained before and after rearrangement, and virtual objects can be rearranged while minimizing degradation of the user experience.

[0133] (Fourth embodiment) In the first, second, and third embodiments, methods for rearranging virtual objects according to the respective conditions have been described. However, in order to apply these rearrangement methods while minimizing degradation of the user experience, it is considered desirable to perform the rearrangement gradually over a certain period of time.

[0134] If the HMD completes the relocation of a virtual object within one frame of updating the screen, the user will not notice that the virtual object has been relocated, and the position of the virtual object will differ from the user's perception. This can lead to a poor user experience.

[0135] Therefore, in this embodiment, a method for rearranging virtual objects with animation to prevent the user experience from being adversely affected will be described.

[0136] Since this embodiment has many commonalities with the first to third embodiments, the following description will focus on the unique features of this embodiment.

[0137] A method for rearranging virtual objects with animation in this embodiment will be described below.

[0138] The method for rearranging virtual objects in this embodiment is basically the same as the methods described in the first, second, and third embodiments. The difference is that the rearrangement is performed over time while animating.

[0139] For example, suppose that the HMD switches from VR mode to MR mode due to a user operation.

[0140] When it is determined that a virtual object needs to be rearranged due to a scene change, in this embodiment, the virtual object is rearranged with animation in time with the scene change. The animation may be linear (constant speed), ease-in (gradual acceleration), ease-out (gradual deceleration), ease-in-out (accelerate and decelerate), or the like, as long as it conveys to the user that the virtual object is gradually changing in size or moving. Furthermore, by matching the time it takes for the animation to the time it takes for the scene change, the rearrangement can be performed more naturally.

[0141] The above-described method of rearranging virtual objects with animation is merely an example, and the method of rearranging virtual objects with animation may be different.

[0142] As described above, in this embodiment, the operation of the information processing device A100 when rearranging virtual objects with animation to prevent degradation of the user experience has been described.

[0143] This allows the user to recognize that the virtual object has been rearranged, and there is no discrepancy between the actual position of the virtual object and the user's perception, so the virtual object can be rearranged while minimizing deterioration of the user experience.

[0144] 6, for example, in step S604, if an animation in which a virtual object is being rearranged is in progress, the control unit A101 determines that the virtual object needs to be rearranged until the rearrangement is complete. Furthermore, in step S605, if an animation in which a virtual object is being rearranged is in progress, the control unit A101 rearranges the virtual object in accordance with the animation. Furthermore, in step S606, the control unit A101 combines the virtual object with a captured image of real space in accordance with the arrangement of the virtual object rearranged in step S605, and generates an image of mixed reality space as a display image.

[0145] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0146] In the first, second, third, and fourth embodiments, it is assumed that the virtual object is reduced in size when it is rearranged. However, depending on the size of the virtual object, it may be sufficient to simply move the virtual object closer. Therefore, the position of the virtual object may be adjusted so that it is placed in front of the real object without reducing the virtual object.

[0147] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0148] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0149] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0150] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0151] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0152] The disclosure of this embodiment includes the following configuration, method, and program.

[0153] [Configuration 1] an acquisition means for acquiring information about the arrangement of real objects in a real space; an image acquisition means for acquiring a captured image of the real space in a predetermined direction; a control means for controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object; When the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, the control means controls to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction. 1. An information processing device comprising:

[0154] [Configuration 2] The control means controls the generation of a first mixed reality space image in which the virtual object is placed at the first position in the mixed reality space, and then controls the generation of a second mixed reality space image in which the virtual object is placed at the second position in the mixed reality space. 2. The information processing device according to configuration 1,

[0155] [Configuration 3] The control means controls the generation of an image of a third mixed reality space in which the virtual object is placed at a position between the first position and the second position, after the generation of the image of the first mixed reality space and before the generation of the image of the second mixed reality space. 3. The information processing device according to configuration 2.

[0156] [Configuration 4] When generating an image of the mixed reality space, the control means controls the virtual object to be reduced in size in the mixed reality space from the size at the first position and placed at the second position. 4. The information processing device according to any one of configurations 1 to 3,

[0157] [Configuration 5] The control means reduces the virtual object based on a size of the virtual object in the mixed reality space at the first position. 5. The information processing device according to configuration 4.

[0158] [Configuration 6] The control means generates an image of the mixed reality space in which the virtual object is placed at the second position at a position and size that match the first position and the second position as seen by the user. 6. The information processing device according to configuration 5.

[0159] [Configuration 7] The control means controls a display unit to display the generated image of the mixed reality space, and after displaying an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, controls a display unit to display an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space. 7. The information processing device according to any one of configurations 1 to 6,

[0160] [Configuration 8] The control means controls to display a screen that prompts the user to select whether or not to move the virtual object after displaying the image of the first mixed reality space, and controls to generate an image of the second mixed reality space when the user selects to move the virtual object. 8. The information processing device according to configuration 7.

[0161] [Configuration 9] The control means displays an image of the first mixed reality space when the user selects not to move the virtual object. 9. The information processing device according to configuration 8.

[0162] [Configuration 10] The control means controls a display unit to display the generated image of the mixed reality space, and controls the display unit to not display an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, but to display an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space. 10. The information processing device according to any one of configurations 1 to 9,

[0163] [Configuration 11] The control means controls to display a screen that prompts a user to select whether or not to move the virtual object before displaying the image of the first mixed reality space, and controls to display the image of the second mixed reality space when the user selects to move the virtual object. 11. The information processing device according to configuration 10.

[0164] [Configuration 12] The control means controls the display of the image of the first mixed reality space when the user selects not to move the virtual object. 12. The information processing device according to configuration 11.

[0165] [Configuration 13] The control means generates an image of the mixed reality space in which the virtual object is placed at the second position by performing a predetermined affine transformation on the virtual object. 13. The information processing device according to any one of configurations 1 to 12.

[0166] [Configuration 14] The second location is a location closer to the first location than a predetermined distance, or a location farther from the user's location than a predetermined distance. 14. The information processing device according to any one of configurations 1 to 13,

[0167] [Configuration 15] When the virtual object is reduced, the virtual object is not reduced below a predetermined size. 15. The information processing device according to any one of configurations 4 to 14.

[0168] [Configuration 16] When a plurality of virtual objects exist in the mixed reality space and at least a part of a first virtual object among the plurality of virtual objects is at the first position, the control means controls to determine positions to rearrange the plurality of virtual objects based on the second position in the mixed reality space at which the first virtual object is to be arranged, and to generate an image of the mixed reality space. 16. The information processing device according to any one of configurations 1 to 15.

[0169] [Configuration 17] The control means controls whether or not to place the virtual object at the second position depending on the type of the virtual object. 17. The information processing device according to any one of configurations 1 to 16,

[0170] [Control method] an acquisition step of acquiring information about the arrangement of real objects in a real space; an image acquisition step of acquiring a captured image of the real space in a predetermined direction; a control step of controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object; In the control step, when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, control is performed to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction. 2. A method for controlling an information processing apparatus comprising:

[0171] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17.

[0172] [system] an acquisition device that acquires information about the arrangement of real objects in real space; an image capture device that captures an image of the real space in a predetermined direction; a control device that controls the generation of an image in a mixed reality space that is an image obtained by combining the captured image and a virtual object, When the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, the control device controls to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction. An information processing system comprising:

Claims

1. an acquisition means for acquiring information about the arrangement of real objects in a real space; an image acquisition means for acquiring a captured image of the real space in a predetermined direction; a control means for controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object; When the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, the control means controls to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction.

1. An information processing device comprising:

2. The control means controls the generation of a first mixed reality space image in which the virtual object is placed at the first position in the mixed reality space, and then controls the generation of a second mixed reality space image in which the virtual object is placed at the second position in the mixed reality space.

2. The information processing apparatus according to claim 1, wherein:

3. The control means controls the generation of an image of a third mixed reality space in which the virtual object is placed at a position between the first position and the second position, after the generation of the image of the first mixed reality space and before the generation of the image of the second mixed reality space.

3. The information processing apparatus according to claim 2, wherein:

4. When generating an image of the mixed reality space, the control means controls the virtual object to be reduced in size in the mixed reality space from the size at the first position and placed at the second position.

2. The information processing apparatus according to claim 1, wherein:

5. The control means reduces the virtual object based on a size of the virtual object in the mixed reality space at the first position.

5. The information processing apparatus according to claim 4,

6. The control means generates an image of the mixed reality space in which the virtual object is placed at the second position at a position and size that match the first position and the second position as seen by the user.

6. The information processing apparatus according to claim 5,

7. The control means controls a display unit to display the generated image of the mixed reality space, and after displaying an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, controls a display unit to display an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space.

2. The information processing apparatus according to claim 1, wherein:

8. The control means controls to display a screen that prompts a user to select whether or not to move the virtual object after displaying the image of the first mixed reality space, and controls to generate an image of the second mixed reality space when the user selects to move the virtual object.

8. The information processing apparatus according to claim 7,

9. The control means displays an image of the first mixed reality space when the user selects not to move the virtual object.

9. The information processing apparatus according to claim 8,

10. The control means controls a display unit to display the generated image of the mixed reality space, and controls the display unit to not display an image of a first mixed reality space in which the virtual object is located at the first position in the mixed reality space, but to display an image of a second mixed reality space in which the virtual object is located at the second position in the mixed reality space.

2. The information processing apparatus according to claim 1, wherein:

11. The control means controls to display a screen that prompts a user to select whether or not to move the virtual object before displaying the image of the first mixed reality space, and controls to display the image of the second mixed reality space when the user selects to move the virtual object.

11. The information processing apparatus according to claim 10,

12. The control means controls the display of the image of the first mixed reality space when the user selects not to move the virtual object.

12. The information processing apparatus according to claim 11,

13. The control means generates an image of the mixed reality space in which the virtual object is placed at the second position by performing a predetermined affine transformation on the virtual object.

2. The information processing apparatus according to claim 1, wherein:

14. The second location is a location closer to the first location than a predetermined distance, or a location farther from the user's location than a predetermined distance.

2. The information processing apparatus according to claim 1, wherein:

15. When the virtual object is reduced, the virtual object is not reduced below a predetermined size.

5. The information processing apparatus according to claim 4,

16. When a plurality of virtual objects exist in the mixed reality space and at least a part of a first virtual object among the plurality of virtual objects is at the first position, the control means controls to determine positions to rearrange the plurality of virtual objects based on the second position in the mixed reality space at which the first virtual object is to be arranged, and to generate an image of the mixed reality space.

2. The information processing apparatus according to claim 1, wherein:

17. The control means controls whether or not to place the virtual object at the second position depending on the type of the virtual object.

2. The information processing apparatus according to claim 1, wherein:

18. an acquisition step of acquiring information about the arrangement of real objects in a real space; an image acquisition step of acquiring a captured image of the real space in a predetermined direction; a control step of controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object; In the control step, when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, control is performed to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction.

2. A method for controlling an information processing apparatus comprising:

19. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

20. an acquisition device that acquires information about the arrangement of real objects in real space; an image capture device that captures an image of the real space in a predetermined direction; a control device that controls the generation of an image in a mixed reality space that is an image obtained by combining the captured image and a virtual object, When the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, the control device controls to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction. An information processing system comprising:

Images