Information processing device, information processing system, information processing program, and information processing method

Abstract

This information processing device has an acquisition means for acquiring information about placement of a real object in a real space, and an image acquisition means for acquiring a captured image by capturing an image of the real space in a predetermined direction, wherein, when a position where a virtual object is placed in a mixed reality space is a first position where at least a part of the virtual object is positioned behind the real object or is in a covered position inside the real object, the information processing device performs control to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is a position in front of the real object when viewed from the predetermined direction.

Description

Information processing device, information processing system, information processing program, and information processing method

[0001] The present disclosure relates to an information processing device that displays virtual objects to a user in a virtual space or a mixed reality space.

[0002] In recent years, mixed reality (MR) technology has become known as a technology for seamlessly blending the real world and the virtual world in real time.

[0003] An example of MR technology is an MR system that uses a video see-through head-mounted display (HMD). In the MR system, a camera built into the HMD captures an image of a subject that roughly matches the subject observed from the user's pupil position. Then, a composite image in which CG (computer graphics, virtual objects) is superimposed on the captured image is presented to the user, allowing the user to experience an MR space (mixed reality space). One technique required to represent virtual objects as if they existed in reality is called occlusion. Occlusion is an image processing technique that calculates the anteroposterior relationship between virtual objects and real objects and renders virtual objects behind the real objects in a hidden or missing state. For example, Patent Document 1 discloses an image processing technique that represents the anteroposterior relationship between virtual objects and real objects.

[0004] Japanese Patent Application Laid-Open No. 2020-30748

[0005] However, when occlusion is applied, depending on the placement of a virtual object, the visibility of the virtual object may be reduced by the virtual object being hidden by a real object, which may impair the user experience in the MR space.

[0006] Therefore, the present disclosure has been made in consideration of these points, and aims to improve the user experience in MR space for users of information processing devices.

[0007] One aspect of the present disclosure is an information processing device comprising: an acquisition means for acquiring information regarding the arrangement of real objects in a real space; an image acquisition means for acquiring an image of the real space in a predetermined direction; and a control means for controlling the generation of an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein the control means controls the generation of an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction, when the position at which the virtual object is arranged in the mixed reality space is a first position where at least a portion of the virtual object is behind the real object or embedded inside the real object.

[0008] According to the present disclosure, it is possible to improve the user experience in the MR space of the user of the information processing device.

[0009] 4A is a diagram illustrating the components of an information processing device according to a first embodiment. FIG. 4B is a diagram illustrating a real space in which a user exists according to the first embodiment. FIG. 4C is a diagram illustrating a virtual space viewed by a user through an HMD according to the first embodiment. FIG. 4D is a diagram illustrating a top-down view of a mixed reality space in which a user 202 exists, assuming a scene in which the virtual space 301 in FIG. 3 is switched to an image of a mixed reality space, leaving a virtual window 305, according to the first embodiment. FIG. 4C is a diagram illustrating a scene in which a portion of an image viewed by a user through an HMD in the scene of FIG. 4A is displayed, the portion of an image viewed by a user through an HMD in the scene of FIG. 4A is not displayed, the portion of an image viewed by a user through an HMD in the scene of FIG. 4A according to the first embodiment. FIG. 4D is a diagram illustrating a scene in which, when a virtual object is displayed as in FIG. 4B or FIG. 4C , the virtual window is translated forward and reduced in the mixed reality space so that the virtual window does not appear differently to the user, according to the first embodiment. FIG. 5A is a diagram illustrating an image of the mixed reality space generated by a control unit A101 according to the scene of FIG. 5A according to the first embodiment. FIG. 5D is a flowchart illustrating detailed operations of an information processing device according to the first embodiment. 9 is a flowchart illustrating detailed operations of an information processing device in a second modified example of the first embodiment. FIG. 10 is a diagram illustrating an example of a confirmation screen displayed after a portion of a virtual object embedded in a wall is displayed, for asking a user whether or not to rearrange the virtual object, in the second modified example of the first embodiment. FIG. 11 is a diagram illustrating an example of a confirmation screen displayed before a portion of a virtual object embedded in a wall is displayed, for asking a user whether or not to rearrange the virtual object, in the second modified example of the first embodiment. FIG. 12 is a diagram illustrating a scene in which a virtual window is displayed indoors as a virtual object on a large screen, in the second embodiment. FIG. 13 is a diagram illustrating a case in which the virtual object rearrangement method of the first embodiment is applied to a virtual window 903 in the situation of FIG. 9 in the second embodiment. FIG. 14 is a diagram illustrating a case in which the virtual object rearrangement method of the second embodiment is applied to a virtual window 903 in the situation of FIG. 9 in the second embodiment.12A to 12C are diagrams illustrating a scene in which a virtual window is displayed indoors on a large screen as a virtual object in a second embodiment. FIG. 12B is a diagram illustrating a case in which the virtual object rearrangement method of the first embodiment is applied to a virtual window 1203 in the situation of FIG. 12 in the second embodiment. FIG. 12C is a diagram illustrating a case in which the virtual object rearrangement method of the second embodiment is applied to a virtual window 1203 in the situation of FIG. 12 in the second embodiment. FIG. 12C is a diagram illustrating a scene in which three virtual windows are displayed indoors in MR mode in a third embodiment. FIG. 12D is a diagram illustrating a scene in which the three virtual windows are respectively rearranged according to the rearrangement method of the first embodiment. FIG. 12D is a diagram illustrating a scene in which an entire rearrangement is performed using a virtual window 1505 as a rearrangement reference in the third embodiment.

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the disclosure according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the disclosure, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] First Embodiment In this embodiment, a use case will be described in which a user wears a head-mounted display (HMD), which is an information processing device worn on the head, and experiences a mixed reality space. The user watches videos of movies, sports, etc. on a virtual display (virtual window) arranged in the mixed reality space. In addition, the description will be made assuming a situation in which the virtual window (virtual object) is at least partially embedded in a wall, and the virtual window is rearranged.

[0012] <Internal Configuration of Information Processing Apparatus> The internal configuration of the information processing apparatus according to the first embodiment of the present disclosure will be described below with reference to Fig. 1. In this embodiment, the information processing apparatus A100 will be described assuming that it is an HMD.

[0013] The information processing device A100 includes a control unit A101, a storage unit A102, a memory A103, an input unit A104, an output unit A105, a sensor A106, and a communication unit A107.

[0014] The control unit A101 is a CPU that controls each component of the information processing device A100. The control unit A101 combines a captured image captured by an imaging sensor (camera) with a virtual object to generate a composite image (an image of a mixed reality space). The control unit A101 also generates an image of a virtual space, which is an image that does not include real space and is constructed solely using CG, based on the captured image captured by the imaging sensor. For example, the control unit A101 identifies a floor or real object from the captured image and determines the placement of the floor or virtual object in the virtual space according to the position of the real object. The control unit A101 may also acquire information on the distances from two cameras to real objects as distance information and determine the placement of virtual objects based on the distance information. The control unit A101 controls the generated image of the mixed reality space or the virtual space to be displayed on a display such as a liquid crystal panel or an organic EL panel. Furthermore, if the information processing device A100 is an HMD, the control unit A101 controls the generation of an image for the right eye and an image for the left eye, respectively, and the display of each image on the respective displays. The control unit A101 estimates the position or orientation of the camera (the position or orientation of the information processing device A100) from the captured image. In this embodiment, the position and orientation are estimated using a method for estimating the camera's self-position and orientation, such as SLAM (Simultaneous Localization and Mapping). The position and orientation estimation unit 104 may estimate its self-position using LiDAR SLAM (LiDAR; Light Detection and Ranging) using a laser, in addition to Visual SLAM, which estimates its self-position from a camera image. The position and orientation estimation unit 104 may also estimate its self-position using Depth SLAM, which uses a ToF sensor (ToF; Time of Flight). Instead of the control unit A101 controlling the entire device, the entire device may be controlled by a plurality of hardware devices sharing the processing load.

[0015] The storage unit A102 is an electrically erasable and recordable non-volatile storage medium such as an SSD (Solid State Drive) or flash memory, and stores programs executed by the control unit A101, databases, user settings, etc.

[0016] The memory A103 is, for example, a RAM (Random Access Memory), and is used as a buffer memory for temporarily storing various data, a work area for the control unit A101, and the like.

[0017] The input unit A104 is used to input instructions to the information processing device A100. The input unit A104 includes, for example, a power button for instructing the information processing device A100 to power on / off and operation buttons for instructing screen transitions. Note that the input unit A104 does not necessarily have to be built into the information processing device A100. It may also be configured so that input is made via a communication unit A107, which will be described later.

[0018] The output unit A105 is, for example, a display (display unit) viewed by the user, a GUI (Graphical User Interface) display for interactive operation, or a light-emitting device such as an LED (Light Emitting Diode). The output unit A105 may also include a sound output device such as a speaker, and a vibration device that provides a haptic effect to the user. The output unit A105 may be configured to output via a communication unit A107, which will be described later.

[0019] The sensor A106 is, for example, an imaging sensor that captures surrounding images, or a sensor such as LiDAR (Light Detection and Ranging) or ToF (Time of Flight) that measures the surrounding conditions. The sensor A106 is also, for example, an IMU (Inertial Measurement Unit) or a geomagnetic sensor that measures the attitude and position of the information processing device A100. The sensor A106 does not necessarily need to be built into the information processing device A100.

[0020] The communication unit A107 is, for example, a NIC (Network Interface Card) incorporating a serial bus or parallel bus for connecting to other devices, a connector (RJ45) for connecting to Ethernet, a communication IC, etc. The communication unit A107 is also a communication unit for wirelessly connecting to a controller or external device. The communication unit A107 outputs a weighted wireless signal from an antenna and demodulates a wireless signal received by the antenna to achieve short-range wireless communication in accordance with the IEEE 802.15 standard (so-called Bluetooth (registered trademark)). Note that the communication unit A107 may be wired, such as a USB cable (registered trademark), or wireless, such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0021] The terminal configuration has been described assuming that the information processing device A100 is an HMD, but the above-described terminal configuration is merely an example, and the information processing device A100 does not have to be an HMD, and the terminal configuration may also be different. Note that the information processing device A100 may also be an information processing device such as a PC (personal computer), tablet terminal, smartphone, etc. connected to an HMD. It may also be an information processing system including each of the components of the information processing device A100 described above.

[0022] <Description of Situations in Which Virtual Object Rearrangement is Necessary> Hereinafter, a method of determining whether a virtual object should be rearranged in this embodiment will be described.

[0023] Whether or not a virtual object should be rearranged is determined using information about the position and size of the virtual object and spatial information such as the positions and sizes of surrounding walls, ceilings, furniture, etc.

[0024] Information about the position and size of the virtual object is grasped by a system or application that controls the display of the virtual object, and the control unit A101 acquires the information about the position and size of the virtual object from the system or application.

[0025] Furthermore, spatial information such as the positions and sizes of surrounding walls, ceilings, furniture, etc. in the real space must be acquired by capturing an image of the surroundings and calculating it through spatial recognition. Spatial recognition can be performed, for example, by measuring the distance to an object by irradiating it with laser light and measuring the reflected light, and then performing calculations by mapping it. Another method involves extracting feature points from an image captured by a camera and combining them with the camera's movement to obtain depth, and then making inferences based on the obtained depth. Other known methods include estimating depth from an image captured by a camera using machine learning such as Deep Learning, and then performing calculations based on the depth. The control unit A101 acquires spatial information of the real space using such methods.

[0026] The control unit A101 calculates the front-to-back relationship between the virtual object and the real object as seen from the HMD in the mixed reality space using the position and size information of the virtual object and the spatial information of the real space acquired by the control unit A101. In other words, it determines which object is in front as seen from the imaging direction of the camera in the mixed reality space.

[0027] If the virtual object is in front, it can be determined that there is no need to rearrange the virtual object because the virtual object will not be obscured by surrounding walls, ceilings, furniture, etc.

[0028] If the virtual object is behind the real object or if the virtual object is hidden by the surrounding walls, ceiling, furniture, etc., it is determined that at least a part of the virtual object will become invisible and that relocation is necessary. In other words, it is determined that relocation is necessary if the virtual object interferes with the surrounding real objects such as the walls, ceiling, furniture, etc. Or, it is determined that relocation is necessary if at least a part of the virtual object is embedded in the surrounding real objects such as the walls, ceiling, furniture, etc.

[0029] The above-described method of determining whether a virtual object should be rearranged is merely an example, and the method of determining whether a virtual object should be rearranged may be different.

[0030] Hereinafter, a real space in which a user exists according to the first embodiment of the present disclosure will be described with reference to Fig. 2. Fig. 2 illustrates a bird's-eye view of the real space in which a user 202 exists, with a scene assumed in which a user 202 exists in a room 201 partitioned by a wall. It is assumed that the room 201 in which the user 202 exists is equipped with a plurality of interior objects (real objects) including a television 203, a sofa 204, a table 205, a shelf 206, and a sofa 207. It is also assumed that the user 202 is sitting on the sofa 207 and facing the television 203.

[0031] Hereinafter, with reference to FIG. 3 , a virtual space viewed by a user via an HMD in the first embodiment of the present disclosure will be described. FIG. 3 illustrates a scenario in which an avatar 302, representing the user 202, exists in the virtual space 301, and shows a bird's-eye view of the virtual space in which the avatar 302 exists. The virtual space 301 can be expressed as if it were an expansive space unrelated to real objects. In FIG. 3 , the virtual space 301 is assumed to be a space larger than an area 303 corresponding to the area surrounded by the walls of the room 201 in the real space in FIG. 2 . A virtual object 304 and a virtual window 305 are arranged in the virtual space 301. The virtual object 304 is a virtual object superimposed on the sofa 207 in the room 201 in FIG. 2 . The virtual window 305 is a virtual object whose width is larger than the room 201 (area 303) as viewed from the user 202. In the virtual space 301, the user 202 can watch a movie, a sport, or other video in a virtual window 305 that is larger than the display that can be placed in the room 201.

[0032] Assume that, in a scene where such a virtual space is being viewed, the user 202 switches from the image of the virtual space to an image of the mixed reality space in which the virtual window 305 appears as if it were placed in the real space.

[0033] Hereinafter, with reference to FIGS. 4A, 4B, and 4C, a description will be given of a screen when an image in a virtual space is switched to an image in a mixed reality space in the first embodiment of the present disclosure.

[0034] 4A is a bird's-eye view of the mixed reality space in which the user 202 exists, and illustrates a scenario in which the image is switched from the virtual space 301 in FIG. 3 to a mixed reality space image, leaving the virtual window 305. The scenario is one in which the mixed reality space 401 is expressed by placing the virtual window 305 in the real space at the same position and size as it was in the virtual space 301. Because the virtual window 305 is wider than the room 201 as seen from the user, it extends beyond the walls of the room 201. In other words, the virtual window 305 is positioned so that both ends of the virtual window 305 are embedded in the walls of the room 201.

[0035] FIG. 4B shows an image viewed by a user through an HMD in the scene of FIG. 4A . To accurately represent the positional relationship between the virtual object and the real object, both ends of virtual window 305 are positioned so that they are embedded in the walls of room 201. Therefore, among virtual windows 305, virtual window 402 is positioned in the mixed reality space. Furthermore, virtual windows 403 and 404, which correspond to both ends of virtual window 305, are positioned so that they are embedded in the walls in the mixed reality space. In FIG. 4B , virtual windows 403 and 404, which are embedded in the walls, are also displayed. A face is displayed in virtual window 402, but part of the face is also displayed in virtual window 403, which is embedded in the wall. A virtual object 412, which represents the shadow of virtual window 402, is also positioned in the image of mixed reality space 401 so that the user can understand the depth of virtual window 402 in mixed reality space 401. Similarly, virtual object 413, which is the shadow of virtual window 403, and virtual object 414, which is the shadow of virtual window 404, are also placed in the image of mixed reality space 401. That is, control unit A101 combines virtual window 402, virtual window 403, virtual window 404, and virtual object 412, virtual object 413, and virtual object 414, which correspond to the shadows, with the captured image. In this way, an image of mixed reality space 401 is generated. In this way, if the image of mixed reality space 401 is generated so that virtual window 403 and virtual window 404, which are embedded in the wall, are also displayed, it may be difficult for the user to get the feeling that virtual window 305 is actually present in real space.

[0036] FIG. 4C is a diagram showing an image viewed by a user through an HMD in the scene shown in FIG. 4A . In FIG. 4B , virtual windows 403 and 404, which correspond to both ends of virtual window 305 embedded in the wall of the real space, are displayed in the image of mixed reality space 401. In FIG. 4C , the virtual object embedded in the wall is not displayed. Here, the only virtual window displayed is virtual object 402, so virtual object 412, which corresponds to the shadow of virtual object 402, is also displayed. A face is reflected in virtual window 402, but part of the face is embedded in the wall, so that part of the face is not visible. That is, the control unit A101 combines the virtual window 402 and virtual object 412, which corresponds to the shadow, with the captured image to generate an image of mixed reality space 401. If the image of mixed reality space 401 is generated in this manner, when the user switches from the virtual space to the mixed reality space, the user may not be able to view the entire virtual window 305 that was being viewed when the image of the virtual space was displayed, which may impair the user experience. For example, when video content is displayed in the virtual window 305, the part that is hidden in the wall may be overlooked.

[0037] <Method for rearranging a virtual object> In this way, when a virtual object is displayed as shown in Figures 4B and 4C, the virtual object is displayed so that it is in front of the real object. That is, occlusion is adjusted so that the virtual object is not hidden by the real object. Here, according to the physical laws of real space, if the size of a virtual object in mixed reality space is fixed, the size of the virtual object will be represented as increasing from the user's perspective as it gets closer to the user. That is, although the size of the virtual object in mixed reality space does not change, when an image of the mixed reality space is generated, by increasing the size of the virtual object with the same angle of view in the mixed reality space, the virtual object will appear to be getting closer to the user.

[0038] However, if the size of the virtual object increases when adjusting the occlusion, the area in which the real space is displayed in the image of the mixed reality space viewed by the user will decrease.

[0039] Therefore, when a virtual object is displayed so that it is in front of a real object, the size of the virtual object in the mixed reality space is reduced. In other words, when an image in the mixed reality space is generated, the size of the virtual object in the composite image is not changed, and the virtual object is displayed so that it is in front of the real object. When a virtual object is displayed in the mixed reality space in this way, the size of the virtual object is the same when viewed from the same position before and after adjusting the positional relationship between the real object and the virtual object. Here, before adjusting the positional relationship between the real object and the virtual object, the virtual object is farther from the user, and after adjusting the positional relationship between the real object and the virtual object, the virtual object is closer to the user. However, when viewed from the same distance from each virtual object before and after adjusting the positional relationship between the virtual objects, the size of the virtual object is smaller after adjusting the positional relationship between the virtual objects.

[0040] That is, although the apparent size of the virtual object remains the same from the perspective of a user in the same position, the size of the virtual object in the mixed reality space becomes smaller after the positional relationship of the virtual objects is adjusted. Similarly, with regard to the size of the virtual object synthesized by the control unit A101, although the size of the virtual object in the synthesized image remains the same, the size of the virtual object in the mixed reality space becomes smaller after the positional relationship of the virtual objects is adjusted.

[0041] <Specific Example of Rearrangement> Hereinafter, with reference to FIGS. 5A and 5B, a screen on which virtual objects are rearranged in an image of mixed reality space according to the first embodiment of the present disclosure will be described.

[0042] 5A illustrates a scenario in which, when a virtual object is displayed as in FIGS. 4B and 4C , the virtual window is translated forward and shrunk in the mixed reality space without changing the way the virtual window appears to the user. Also, FIG. 5A illustrates a bird's-eye view of the mixed reality space in which the user 202 exists. In the mixed reality space 501, the virtual object is moved from the position of virtual window 502, which was embedded in the wall, to the position of virtual window 503 in the foreground, and shrunk without changing the size that appears to the user. If the virtual object is moved closer to the user while maintaining its size, the size that appears to the user increases.

[0043] By shrinking the virtual object and bringing it closer to the user, the apparent size to the user can be maintained. In this example, the size of the virtual object changes before and after rearrangement, as a bird's-eye view is assumed.

[0044] FIG. 5B illustrates an example of a screen viewed by the user 202 in the scene described in FIG. 5A . That is, FIG. 5B illustrates an image of a mixed reality space generated by the control unit A101. In FIG. 5B , a scene is assumed in which a virtual object is moved closer to the user at the same position and orientation as the user in FIG. 4B or 4C . In FIG. 5B , a virtual window 503 is positioned closer to the user in the mixed reality space 501 than in FIG. 4B or 4C and does not interfere with the wall of the room. That is, at least a portion of the virtual window is not embedded in the wall. Furthermore, the shadow of the virtual window 503 was located at the position of the virtual window shadow 512 before the rearrangement. However, in this image of the mixed reality space, the virtual window shadow 512 is not composited, and a virtual window shadow 513 is drawn according to the rearranged position. In FIG. 5B , a virtual window of the same size as the virtual window in FIG. 4B or 4C is composited on the image of the mixed reality space generated by the control unit A101. The positional relationship in the mixed reality space has changed. If the distance between the virtual window and the user in FIG. 4B is the same as the distance between the virtual window and the user in FIG. 5B, the size of the virtual window as seen by the user will be larger in FIG. 4B.

[0045] By shrinking a virtual object in mixed reality space and moving it forward, a virtual object that was buried in a wall can be made to no longer be buried in the wall. Furthermore, by adjusting the shrinking ratio and movement distance, the appearance of the virtual object can be prevented from changing due to perspective before and after rearrangement. Because the appearance of the virtual object remains unchanged after rearrangement, this method minimizes degradation of the user experience due to virtual object rearrangement compared to other rearrangement methods that change the appearance of the virtual object.

[0046] However, if the user's angle of view does not change, i.e., if the user is in the same position and facing the same direction, the control unit A101 will essentially composite a virtual object of the same size with the captured image before and after the rearrangement when generating an image in mixed reality space.

[0047] In this way, the method for rearranging virtual objects according to this embodiment is performed based on information about the position and size of the virtual object and values ​​of surrounding space information and the like.

[0048] In this embodiment, a method for rearranging a virtual object has been described in which the virtual object is scaled down in mixed reality space and moved closer to the user so that the appearance of the virtual object does not change.

[0049] Note that any affine transformation or other method may be used as long as it is a rearrangement method that does not adversely affect the user experience. For example, an image of the virtual object from the user's viewpoint may be generated without changing the size of the virtual object in the mixed reality space, and the image may be composited with the captured image.

[0050] <Flow of Virtual Object Rearrangement> Hereinafter, with reference to FIG. 6, detailed operations of the information processing device according to this embodiment will be described.

[0051] 6 is a flowchart of the information processing device A 100 according to this embodiment. This flowchart is started when a system or application that controls the display of virtual objects is started.

[0052] In step S601, the control unit A101 determines whether to end the process based on the status of each unit of the information processing device A100. For example, the control unit A101 ends the process of this flowchart when an end operation is performed via the input unit A104. If it is determined that the process should be continued, the control unit A101 proceeds to step S602.

[0053] In step S602, the control unit A101 captures the surroundings via the sensor A106 and acquires the image. Capturing the surroundings includes, for example, capturing an image with a camera or measuring distance with laser light using LiDAR. The captured information is recorded in the memory A103. Here, the control unit A101 activates the camera to acquire captured images in real time, and then proceeds to step S603.

[0054] In step S603, the control unit A101 recognizes the real space (space recognition) based on the surrounding data captured in step S602. The result of the space recognition performed here is space information such as the positions and sizes of real objects such as surrounding walls, ceilings, and furniture. After acquiring the space information, the control unit A101 proceeds to step S604.

[0055] In step S604, the control unit A101 obtains information about the display position and size of the virtual object from the memory A103, compares it with the surrounding space information calculated in step S603, and determines whether the virtual object needs to be rearranged. If the control unit A101 determines that the virtual object needs to be rearranged, it proceeds to step S605. If the control unit A101 determines that the virtual object does not need to be rearranged, it proceeds to step S606. While the image of the virtual space is being displayed, the control unit A101 determines that the virtual object does not need to be rearranged because there is no interference with a real object. Furthermore, while the image of the mixed reality space is being displayed, the control unit A101 determines that the virtual object needs to be rearranged if there is interference with a real object or if the virtual object is hidden behind a real object.

[0056] In step S605, the control unit A101 updates the display position and size information of the virtual object stored in the memory A103 according to the above-described method for rearranging a virtual object, and rearranges the virtual object. When at least a part of the virtual object is embedded in (buried in) a real object, the control unit A101 rearranges the virtual object as if it were in front of the real object.

[0057] In step S606, the control unit A101 combines the virtual object read from the memory A103 with the captured image taken by the camera in step S602 to generate an image of mixed reality space. Alternatively, the control unit A101 generates an image of virtual space from the virtual object read from the memory A103. After generating the image of mixed reality space or the image of virtual space, which is an image for display, the control unit A101 proceeds to step S607.

[0058] In step S607, the control unit A101 displays the display image generated in step S606 via the output unit A105, and the process returns to step S601.

[0059] As described above, the information processing device A100 of this embodiment determines whether or not rearrangement of a virtual object is necessary based on the spatial information obtained by capturing an image of the surroundings and recognizing the space, and the positional relationship between the virtual object and the spatial information. If rearrangement is necessary, the information processing device A100 of this embodiment rearranges the virtual object by shrinking it in the mixed reality space and moving it forward.

[0060] This allows the user to view a virtual object without changing its appearance even if the virtual object is relocated, assuming the same viewing angle (same position and same viewing direction).This allows virtual objects that were previously invisible due to the surrounding environment to be relocated to a visible position while minimizing degradation of the user experience.

[0061] Note that if the virtual object is a two-dimensional virtual object such as a virtual screen, the thickness of the screen may not be reduced. That is, the vertical and horizontal lengths of the screen may be reduced before and after rearrangement, but the thickness may remain constant before and after rearrangement. Also, if the virtual object is a three-dimensional virtual object, it may be reduced three-dimensionally. Also, a virtual object that casts a shadow over the virtual object may not be displayed. Also, the shadow portion may be displayed in accordance with the reduction ratio and movement amount of the virtual object.

[0062] When switching from an image in virtual space to an image in mixed reality space, virtual objects that are only displayed in virtual space may not be subject to rearrangement, but virtual objects that continue to be displayed in the image in mixed reality space may be subject to rearrangement.

[0063] <First Modification of First Embodiment> As described above, according to the flow of FIG. 6 , when it is necessary to rearrange a virtual object, an image of the mixed reality space in which the virtual object has been rearranged is displayed without displaying an image in which the virtual object and the real object interfere with each other.

[0064] In a modified example, an image in which a virtual object and a real object interfere with each other may be displayed, and then an image in the mixed reality space in which the virtual object has been rearranged may be displayed, as shown in Fig. 4B or 4C. In this case, in step S604 of Fig. 6, the control unit A101 determines whether rearrangement of the virtual object is necessary and whether an image in the mixed reality space in which the virtual object and the real object interfere with each other has been displayed once.

[0065] If the control unit A101 determines that rearrangement of the virtual object is necessary and that an image of the mixed reality space in which the virtual object and the real object interfere with each other has been displayed once, the control unit A101 proceeds to step S605. On the other hand, if the control unit A101 does not determine that rearrangement of the virtual object is necessary and that an image of the mixed reality space in which the virtual object and the real object interfere with each other has been displayed once, the control unit A101 proceeds to step S606. This allows the user to visually recognize once that the virtual object and the real object are interfering with each other, and can predict that the virtual object will be rearranged. Instead of determining whether an image of the mixed reality space in which the virtual object and the real object interfere with each other has been displayed once, the control unit A101 may determine whether an image of the mixed reality space in which the virtual object and the real object interfere with each other has been displayed for a predetermined period of time.

[0066] <Modification 2 of First Embodiment> In another modification 2, after displaying an image in which a virtual object and a real object interfere with each other, if it is determined that the virtual object needs to be rearranged, a screen may be displayed to confirm with the user whether or not to rearrange the virtual object. The flow in this case will be described with reference to FIG. 7 .

[0067] 7 is a flowchart of the information processing device A100 according to this embodiment. This flowchart is started when a system or application that controls the display of virtual objects is launched. Here, explanations of steps S601 to S603 and step S606, which are the same as those in the flow of FIG. 6, will be omitted.

[0068] In step S604, the control unit A101 acquires information about the display position and size of the virtual object from the memory A103, compares the information with the surrounding space information calculated in step S603, and determines whether rearrangement of the virtual object is necessary. Furthermore, in step S702, which will be described below, the control unit A101 determines whether or not the user has instructed not to rearrange the virtual object. If an instruction not to rearrange the virtual object has been received, the control unit A101 determines that rearrangement of the virtual object is unnecessary, and if an instruction not to rearrange the virtual object has not been received, the control unit A101 determines that rearrangement of the virtual object is necessary. If the control unit A101 determines that rearrangement of the virtual object is necessary, the control unit A101 proceeds to step S701. If the control unit A101 determines that rearrangement of the virtual object is unnecessary, the control unit A101 proceeds to step S606.

[0069] In step S701, the control unit A101 determines whether or not the user has issued an instruction to rearrange the virtual objects. If the control unit A101 determines that the user has issued an instruction to rearrange the virtual objects, the control unit A101 proceeds to step S605. If the control unit A101 does not determine that the user has issued an instruction to rearrange the virtual objects, the control unit A101 proceeds to step S702. For example, if the user issues an instruction to rearrange the virtual objects on a confirmation screen generated in the following step S703 for asking the user whether or not to rearrange the virtual objects, the control unit A101 determines that the user has issued an instruction to rearrange the virtual objects. Furthermore, if the user does not issue an instruction to rearrange the virtual objects on a confirmation screen generated in the following step S703 for asking the user whether or not to rearrange the virtual objects, the control unit A101 does not determine that the user has issued an instruction to rearrange the virtual objects.

[0070] An example of a confirmation screen that asks the user whether or not to rearrange the virtual objects will be described with reference to FIGS. 8A and 8B.

[0071] In FIG. 8A , the portion of the virtual object that is embedded in the wall is also composited with the image of the mixed reality space. For example, the image of FIG. 8A may be displayed after the image of the mixed reality space of FIG. 4B is displayed. Alternatively, the image of FIG. 8A may be displayed without first displaying the image of the mixed reality space of FIG. 4B . In the image 801 of the mixed reality space of FIG. 8A , a confirmation screen 802 that asks the user whether or not to rearrange the virtual object is further composited with the image of the mixed reality space of FIG. 4B . Furthermore, a virtual object 812 that is a shadow of the confirmation screen 802 and a virtual object 813 indicating the user's designated position are composited with the image 801 of the mixed reality space. Furthermore, the confirmation screen 802 displays a button that allows the user to select whether or not to rearrange the virtual window that is embedded in the wall. The user can instruct to rearrange the virtual object by moving the virtual object 813 to the "Yes" position and performing a confirmation operation. The user can instruct not to rearrange the virtual object by moving the virtual object 813 to the "No" position and performing a confirmation operation.

[0072] In Fig. 8B , if a virtual object has a portion that is embedded in a wall, the virtual object is not composited into the image of the mixed reality space. Here, a confirmation screen 802 that asks the user whether or not to rearrange the virtual object, a virtual object 812 that is the shadow of the confirmation screen 802, and a virtual object 813 that indicates the position designated by the user are composited into the captured image. For example, the image of Fig. 8A may be displayed after the image of the mixed reality space of Fig. 4B is displayed. Alternatively, the image of Fig. 8A may be displayed without first displaying the image of the mixed reality space of Fig. 4B.

[0073] In step S702, the control unit A101 determines whether the user has issued an instruction not to rearrange the virtual objects. If the control unit A101 determines that the user has issued an instruction not to rearrange the virtual objects, the control unit A101 proceeds to step S606. If the control unit A101 does not determine that the user has issued an instruction not to rearrange the virtual objects, the control unit A101 proceeds to step S703. For example, if the user issues an instruction not to rearrange the virtual objects on a confirmation screen generated in the following step S703 to ask the user whether to rearrange the virtual objects, the control unit A101 determines that the user has issued an instruction not to rearrange the virtual objects. Furthermore, if the user does not issue an instruction not to rearrange the virtual objects on a confirmation screen generated in the following step S703 to ask the user whether to rearrange the virtual objects, the control unit A101 does not determine that the user has issued an instruction not to rearrange the virtual objects. In other words, if the user has not yet issued an instruction to rearrange the virtual objects or an instruction not to rearrange the virtual objects on a confirmation screen generated in the following step S703 to ask the user whether to rearrange the virtual objects, the control unit A101 proceeds to step S703.

[0074] Furthermore, if the control unit A101 determines that the user has instructed not to rearrange the virtual objects, then in the subsequent step S604, the control unit A101 determines that there is no need to rearrange the virtual objects.

[0075] 8A and 8B, the user can issue an instruction to rearrange the virtual object by moving the virtual object 813 to the "Yes" position and performing a confirm operation. The user can also issue an instruction not to rearrange the virtual object by moving the virtual object 813 to the "No" position and performing a confirm operation. A case in which the user has not yet performed a confirm operation corresponds to a case in which the user has not yet issued an instruction to rearrange or an instruction not to rearrange.

[0076] In step S703, the control unit A101 generates a display image on which a confirmation screen for the user is superimposed. The control unit A101 combines the virtual object read from the memory A103, the virtual object corresponding to the confirmation screen for the user, and the captured image captured by the camera in step S602 to generate an image of a mixed reality space. After generating the image of the mixed reality space, which is the display image, the control unit A101 proceeds to step S607. For example, the control unit A101 generates an image of the mixed reality space as shown in FIGS. 8A and 8B described above as a display image. Note that when it is determined that rearrangement of a virtual object is necessary, if a display image on which a confirmation screen for confirming whether or not to rearrange the virtual object is superimposed is generated, the virtual object that needs to be rearranged may be displayed or not displayed. Furthermore, when the virtual object that needs to be rearranged is also displayed, portions of the virtual object that are hidden or embedded in the real object may not be displayed, or the hidden or embedded portions may be displayed transparently. For example, when displaying a virtual object that requires rearrangement, if you do not want to display parts of the virtual object that are hidden by or embedded in a real object, an image of a mixed reality space is generated in which a confirmation screen is also superimposed on the image of Figure 4C.

[0077] In addition, when there is a virtual window hidden by a real object, that is, when there is a virtual window placed behind a real object, a confirmation screen may be displayed to confirm whether or not to rearrange the virtual window hidden by the real object.

[0078] This makes it possible to determine whether to rearrange the virtual object in accordance with a user instruction when the virtual object is positioned at a position where at least a part of the virtual object is obscured by a real object.

[0079] Second Embodiment In the first embodiment, a method for rearranging a virtual object by shrinking it in mixed reality space so as not to change the appearance of the virtual object and moving it closer to the user has been described. However, depending on the user's position and the arrangement of walls, ceilings, furniture, etc., the rearrangement may result in the virtual object being reduced too much, or being moved too far forward, making it too close.

[0080] If the virtual object is reduced too much, it may become difficult to perform touch operations, gaze input, ray input, etc. Also, if the virtual object is too close, it may block the view and make it impossible to see the real object in front of you.

[0081] Therefore, in this embodiment, the operation of the information processing device A100 (HMD) when limitations are placed on the reduction ratio and movement amount when a virtual object is rearranged will be described. Note that this embodiment has many parts in common with the first embodiment, and therefore the description will focus on parts unique to this embodiment.

[0082] A method for determining whether a virtual object should be rearranged in this embodiment will be described below.

[0083] As in the first embodiment, whether to rearrange a virtual object is determined using spatial information such as the position and size of the virtual object and the positions and sizes of surrounding walls, ceilings, furniture, etc. First, using spatial information such as the positions and sizes of the virtual object and surrounding walls, ceilings, furniture, etc., it is calculated which of the virtual object or the real object is closer from the perspective of the HMD (or user). In other words, it is calculated which of the virtual object or the real object is closer from the perspective of the imaging position.

[0084] If the virtual object is closer than the real object, it can be determined that there is no need to rearrange the virtual object because the virtual object will not be obscured by surrounding walls, ceilings, furniture, etc.

[0085] Next, if the virtual object is behind the real object or if the virtual object is obscured by surrounding walls, ceilings, furniture, etc., a determination is made based on the values ​​of the reduction ratio and movement amount of the virtual object. When attempting to rearrange the virtual object in front of the real object, if both the reduction ratio and movement amount values ​​upon rearrangement do not exceed the limits (thresholds), it is determined that the virtual object should be rearranged. When attempting to rearrange the virtual object in front of the real object, if both the reduction ratio and movement amount values ​​upon rearrangement exceed the limits (thresholds), it is determined that the virtual object should not be rearranged. In other words, if the virtual object is reduced more than the reduction ratio limit or moved more than the movement amount limit upon rearrangement, it is determined that the virtual object should not be rearranged at the position where it was originally intended to be rearranged. Note that when the virtual object is reduced more than the reduction ratio limit or moved more than the movement amount limit upon rearrangement, it is only reduced until the reduction ratio reaches the limit and moved until both the movement amount values ​​reach the limit.

[0086] Here, "the reduction ratio value reaching its limit" means that the virtual object is in the most reduced state in the mixed reality space within the set limits, and "the movement amount value reaching its limit" means that the virtual object is positioned in a position closest to the HMD within the set limits.

[0087] In the above-described method for determining whether to rearrange a virtual object, if the virtual object is buried in a real object or if the values ​​of the reduction rate and movement amount reach their limits, the virtual object is rearranged to a position where the values ​​of the reduction rate and movement amount reach their limits. In this way, by moving the virtual object forward while shrinking it to its limit, it may be possible to display a part of the virtual object in front of the surrounding walls, ceiling, furniture, etc. Alternatively, the visibility of the virtual object may be improved compared to before the rearrangement.

[0088] The above-described method of determining whether a virtual object should be rearranged is merely an example, and the method of determining whether a virtual object should be rearranged may be different.

[0089] A method for rearranging a virtual object in this embodiment will be described below, taking examples of a method for rearranging a virtual object when limitations are placed on the reduction ratio and the amount of movement.

[0090] <First Example of Second Embodiment> As a first example, a method for rearranging a virtual object when a limit is placed on the reduction ratio will be described below with reference to Figures 9, 10, and 11. Here, a limit is placed on the reduction ratio, but no limit is placed on the amount of movement. Therefore, the virtual object is placed in front of the real object in a state where it is reduced to the limit of the reduction ratio.

[0091] FIG. 9 is a diagram illustrating a scene in which a virtual window is displayed indoors as a virtual object on a large screen. The virtual window 903 is a virtual object that is treated as a target for relocation in this embodiment. In FIG. 4 used for the explanation of the first embodiment, the user is using the HMD while sitting on a sofa, and there is a sufficient distance to the wall in front of them. However, in FIG. 9 of this embodiment, the user is using the HMD while sitting at a counter table, and the distance to the wall in front of them is short. In this situation, the virtual window 903 is displayed as if it exists behind the wall of a real object in front of the user.

[0092] 10 is a diagram illustrating a case where the virtual object rearrangement method of the first embodiment is applied to a virtual window 903 in the situation of FIG. 9. The virtual window 903 is rearranged so that it is moved in front of the front wall, and its size is reduced to approximately one-seventh of its original size. That is, in FIG. 10, it is placed at the position of virtual window 1003. If the virtual object is reduced too much in this way, it may become difficult to perform inputs using touch operations, eye-gaze inputs, ray inputs, and the like.

[0093] 11 is a diagram illustrating a case where the virtual object rearrangement method of the second embodiment is applied to a virtual window 903 in the situation of FIG. 9 . Here, a case where the lower limit of the virtual object reduction rate is set to half of the original size is described. The virtual window 903 is still moved so as to be closer to the front wall by rearrangement, but its size is reduced to only half of its original size. The virtual object rearrangement method of this embodiment changes the appearance of the virtual object, but it can prevent the virtual object from shrinking so much that it becomes difficult to perform input by touch operation, gaze input, ray input, or the like.

[0094] <Second Example of Second Embodiment> Hereinafter, as a second example, a method for rearranging a virtual object when a limit is placed on the amount of movement will be described with reference to Fig. 12, Fig. 13, and Fig. 14. Here, a limit is placed on the amount of movement, but the reduction ratio is matched to the arrangement of the real object.

[0095] 12 is a diagram illustrating a scene in which a virtual window is displayed indoors on a large screen as a virtual object. A virtual window 1203 is a virtual object to be rearranged in this embodiment, and is placed in a mixed reality space 1201. FIG. 12 is a diagram showing a bird's-eye view of the mixed reality space 1201 in which a user 1202 exists.

[0096] In Fig. 4 used for explanation in the first embodiment, the user is using the HMD while sitting on a sofa, and there is a sufficient distance between the left and right walls. However, in Fig. 12 of this embodiment, the user is using the HMD in a small room such as a toilet, and the distance between the left and right walls is short.

[0097] Fig. 13 is a diagram illustrating a case where the virtual object rearrangement method of the first embodiment is applied to a virtual window 1203 in the situation of Fig. 12. The virtual window 1303 is a virtual object rearranged in this embodiment and is placed in a mixed reality space 1301. Fig. 13 is a diagram illustrating a top-down view of the mixed reality space 1301 in which a user 1202 exists. The virtual window 1203 is moved so that it is closer to the left and right walls and the toilet paper holder, and its size is reduced accordingly, and it is placed at the position and size of the virtual window 1303. If the virtual object is rearranged in this way to a position too close to the user, there is a risk that the virtual object will block the user's view and make it impossible to see the real object in front of the user.

[0098] FIG. 14 is a diagram illustrating a case where the virtual object rearrangement method of the second embodiment is applied to a virtual window 1203 in the situation of FIG. 12 . Here, a case where the upper limit of the movement amount of the virtual object is set to half the distance from the user is described. A virtual window 1403 is a virtual object rearranged in this embodiment and is arranged in a mixed reality space 1401. FIG. 14 is a bird's-eye view of the mixed reality space 1401 in which a user 1202 exists. The virtual window 1203 is rearranged to a position half the distance from the user, which corresponds to the upper limit of the movement amount, and is arranged at the position of the virtual window 1303. Furthermore, the size of the virtual window 1203 is reduced to a size that will not be obscured by the left and right walls or the toilet paper holder, and is arranged at the size of the virtual window 1303. The virtual object rearrangement method of this embodiment changes the appearance of the virtual object, but prevents the virtual object from obscuring the view of a real object in front of the user.

[0099] As in the two examples described above, the method for rearranging a virtual object in the second embodiment can be determined from information about the position and size of the virtual object, information about the surrounding space, and values ​​such as the reduction rate and movement amount limit at the time of rearrangement.

[0100] The above-described rearrangement of the virtual object is performed, for example, in the process of step S605 in FIG. 6 or FIG. 7 described in the first embodiment.

[0101] In the second embodiment, a limit (threshold) is set for the amount of movement from the original position of the virtual object, but this is not limiting. Because there is a possibility that the virtual object will block the view and make the real object in front of the user invisible, a limit (threshold) may be set for the position at which the virtual object is rearranged, depending on the distance from the HMD or the user. In other words, even if the virtual object is rearranged, the position at which the virtual object is rearranged will be farther away from the HMD or the user than a predetermined distance.

[0102] The above-described method for rearranging virtual objects is merely an example, and the method for rearranging virtual objects and the restrictions on the reduction ratio and movement amount during rearrangement may be different.

[0103] As described above, in the second embodiment, the operation of the information processing apparatus A100 when limitations are placed on the reduction ratio and movement amount when a virtual object is rearranged has been described.

[0104] By utilizing the method for determining whether to rearrange a virtual object and the method for rearranging a virtual object described in the second embodiment, it is possible to prevent input to the virtual object from becoming difficult and to prevent the virtual object from blocking the view.

[0105] Third Embodiment In the first and second embodiments, the explanation has been given on the assumption that only one virtual object is displayed. However, depending on the use case, it may be possible to simultaneously display and use multiple virtual objects.

[0106] In such use cases, if only the virtual objects that need to be rearranged are rearranged, the relative positions of the virtual objects with respect to other virtual objects will change, confusing the user and causing a deterioration in the user experience.Furthermore, in use cases where multiple virtual objects are displayed, there may be virtual objects that should be excluded from the rearrangement target in consideration of the user experience.

[0107] Therefore, in this embodiment, a method for rearranging virtual objects while suppressing degradation of the user experience when multiple virtual objects are displayed simultaneously will be described.

[0108] Since this embodiment has many commonalities with the first and second embodiments, the following description will focus on the unique features of this embodiment.

[0109] A method of determining whether a virtual object is to be rearranged in this embodiment will be described below.

[0110] In the first and second embodiments, the explanation has been given on the assumption that there is always one virtual object that appears and that it is always subject to rearrangement. In this embodiment, the explanation will be given on the assumption that a plurality of virtual objects are displayed, and that among these, there is a virtual object that should be excluded from the subject of rearrangement.

[0111] There are several possible methods for determining whether a virtual object is a target for rearrangement. For example, a method in which the user sets whether a virtual object is a target for rearrangement. The user determines for each virtual object whether the virtual object is a target for rearrangement, and this is recorded as a user setting in the memory A103. The information processing device A100 can determine whether a virtual object is a target for rearrangement by checking the user setting recorded in the memory A103.

[0112] Another method is to make the determination based on the type of virtual object. In many cases, the user experience does not deteriorate even if a virtual window or the like is reduced in size when it is relocated, but reducing a 3DCG displayed at actual size may affect the user experience. Therefore, this method determines that if the type of virtual object is a virtual window, it is subject to relocation, and if it is 3DCG, it is not subject to relocation.

[0113] Another method involves making a determination based on the situation in which the virtual object is placed. An image of a virtual space is displayed, and whether the virtual object is to be subject to relocation is determined based on whether the virtual object was placed in VR mode, which typically does not include real space in the image, or in MR mode, which displays an image in mixed reality space. In the image of mixed reality space, the virtual object is placed as if it exists in real space. When a virtual object is placed in MR mode, it is highly likely that the virtual object is placed using a real object as the starting point, and if the virtual object is relocated, the virtual object may move to a position that does not match the user's intention. For this reason, a determination method is used in which virtual objects placed in VR mode are subject to relocation, and virtual objects placed in MR mode are not subject to relocation.

[0114] Although several specific examples have been shown for the method of determining whether a virtual object is to be rearranged, one of these determination methods may be used, or a combination of multiple methods may be used. Furthermore, the above-described determination methods are merely examples, and different methods may be used to determine whether a virtual object is to be rearranged.

[0115] <Method of Determining Whether Virtual Object Should Be Rearranged> Hereinafter, a method of determining whether a virtual object should be rearranged in this embodiment will be described.

[0116] First, from among the plurality of virtual objects being displayed, a virtual object to be treated as a target for rearrangement is identified by the method for determining whether or not the virtual object is a target for rearrangement described above.

[0117] Next, one virtual object is selected as the basis for rearrangement from among the virtual objects to be rearranged. Any virtual object may be selected as the basis for rearrangement, but the behavior at the time of rearrangement will vary depending on the virtual object selected. How the behavior at the time of rearrangement changes will be described below with an example in the description of the method for rearranging virtual objects.

[0118] A determination is made as to whether a virtual object selected as a reference for rearrangement should be rearranged. The method of determining whether a virtual object should be rearranged is the same as that of the first or second embodiment. If no limitations are imposed on the reduction ratio or movement amount when the virtual object is rearranged, the method of the first embodiment is used, and if limitations are imposed, the method of the second embodiment is used.

[0119] If it is determined that the virtual object selected as the rearrangement reference should be rearranged, it is determined that all virtual objects, including other virtual objects, should be rearranged.If it is determined that the virtual object selected as the rearrangement reference does not need to be rearranged, it is determined that all virtual objects, including other virtual objects, do not need to be rearranged.

[0120] The above-described method of determining whether a virtual object should be rearranged is merely an example, and the method of determining whether a virtual object should be rearranged may be different.

[0121] Hereinafter, a method for rearranging virtual objects in the third embodiment will be described with reference to FIGS. 15, 16, and 17. FIG.

[0122] Fig. 15 is a diagram illustrating a scene in which three virtual windows are displayed indoors in MR mode. Fig. 15 is a diagram showing a top-down view of a mixed reality space 1501 in which a user 1502 is present and in which, from left to right, virtual windows 1503, 1504, and 1505 are arranged. Virtual windows 1503, 1504, and 1505 are all treated as candidates for rearrangement.

[0123] In explaining the rearrangement method of this embodiment, for comparison, a case where the rearrangement method of the first embodiment is applied under the same conditions will be explained. Here, for each of a plurality of virtual objects, it is determined whether there are parts that are hidden by real objects and cannot be seen, or whether there are parts that are buried, and then it is determined whether to perform rearrangement.

[0124] Fig. 16 is a diagram illustrating a scene in which three virtual windows have been rearranged according to the rearrangement method of the first embodiment. Fig. 16 is a diagram showing a top view of a mixed reality space 1601 in which a user 1502 is present and in which, from left to right, virtual windows 1603, 1504, and 1605 are arranged. In Fig. 16, the virtual windows 1503 and 1505 have been rearranged and are now located at the positions of virtual windows 1603 and 1605, respectively.

[0125] Because the left side of virtual window 1503 is embedded in the wall, it is moved forward while shrinking slightly so that it is no longer embedded in the wall. Because virtual window 1503 is moved to the position of virtual window 1603 through rearrangement, it becomes approximately two-thirds its size, and the distance from the user becomes approximately two-thirds of its original distance. Next, virtual window 1504 does not need to be rearranged because it is not embedded in the wall, so it is not reduced in size or moved. Then, virtual window 1505 is embedded in the wall on the right side and is hidden by shelf 1606 in the foreground. Therefore, it needs to be further shrunk than virtual window 1503 and moved further forward. Because virtual window 1505 is moved to the position of virtual window 1605 through rearrangement, it becomes approximately half its size, and the distance from the user becomes approximately half of its original distance.

[0126] As described above, when the rearrangement method of the first embodiment is applied individually to each virtual object, if the user is in the same position before and after the rearrangement, the size of the virtual object appears to remain unchanged. However, when the rearrangement method of the first embodiment is applied individually to each virtual object, three virtual objects that should have been at the same depth are rearranged to different positions. If the user intentionally placed the virtual objects with the same depth, this rearrangement method can be a factor that degrades the user experience.

[0127] <Relocation Method in Third Embodiment> In this embodiment, a method for rearranging other virtual objects is determined in accordance with a method for rearranging a virtual object selected as a reference.

[0128] Fig. 17 is a diagram illustrating a scene in which an entire space has been rearranged using virtual window 1505 as the rearrangement reference. Fig. 17 is a diagram showing a top view of mixed reality space 1701 in which user 1502 is present and in which, from left to right, virtual window 1703, virtual window 1704, and virtual window 1705 are arranged. In Fig. 17, virtual window 1503, virtual window 1504, and virtual window 1505 are rearranged and are therefore located at the positions of virtual window 1703, virtual window 1704, and virtual window 1705, respectively.

[0129] Since the left side of virtual window 1505 is hidden by the wall, it is shrunk so that it is no longer hidden by the wall, and is moved forward and placed in the position of virtual window 1705. After rearrangement, virtual window 1505 is reduced to approximately half its size, and the distance from the user is also reduced to approximately half of its original distance.

[0130] Virtual windows 1503 and 1504 are moved forward while shrinking by the same amount as virtual window 1505 in accordance with the shrink ratio and movement amount when virtual window 1505 is rearranged, and are thereby placed at the positions of virtual windows 1703 and 1704, respectively.

[0131] In this case, even if virtual window 1504 is selected as the reference, virtual windows 1503 and 1505 are not rearranged and remain embedded in the wall. Also, even if virtual window 1503 is selected as the reference, virtual window 1505 is rearranged and is no longer embedded in the wall, but is instead embedded in shelf 1606. Therefore, virtual window 1505 is selected as the reference.

[0132] As described above, in this embodiment, by determining the rearrangement method of other virtual objects in accordance with the rearrangement method of the virtual object selected as the reference, it is possible to maintain the relative positional relationships between virtual objects before and after rearrangement.

[0133] The above-described rearrangement of the virtual object is performed, for example, in the process of step S605 in FIG. 6 or FIG. 7 described in the first embodiment.

[0134] As described above, in this embodiment, the operation of the information processing device A100 when multiple virtual objects are displayed simultaneously has been described.

[0135] By using the method described in this embodiment, the relative positional relationships between virtual objects can be maintained before and after rearrangement, and virtual objects can be rearranged while minimizing degradation of the user experience.

[0136] Fourth Embodiment In the first, second, and third embodiments, methods for rearranging virtual objects according to respective conditions have been described. However, in order to apply these rearrangement methods while minimizing degradation of the user experience, it is considered desirable to perform the rearrangement gradually over a certain period of time.

[0137] If the HMD completes the relocation of the virtual object within one frame of updating the screen, the user will not notice that the virtual object has been relocated, and the position of the virtual object will differ from the user's perception. This can lead to a poor user experience.

[0138] Therefore, in this embodiment, a method for rearranging virtual objects with animation to prevent the user experience from being adversely affected will be described.

[0139] Since this embodiment has many commonalities with the first to third embodiments, the following description will focus on the unique features of this embodiment.

[0140] A method for rearranging virtual objects with animation in this embodiment will be described below.

[0141] The method for rearranging virtual objects in this embodiment is basically the same as the methods described in the first, second, and third embodiments, except that the rearrangement is performed over time while animating the objects.

[0142] For example, suppose that the HMD scene is switched from VR mode to MR mode by a user operation.

[0143] When it is determined that a virtual object needs to be rearranged due to a scene change, in this embodiment, the virtual object is rearranged with animation in time with the scene change. The animation may be linear (constant speed), ease-in (gradual acceleration), ease-out (gradual deceleration), ease-in-out (acceleration and deceleration), or the like, as long as it conveys to the user that the virtual object is gradually changing in size or moving. Furthermore, by matching the time it takes for the animation to the time it takes for the scene change, the rearrangement can be performed more naturally.

[0144] The above-described method of rearranging virtual objects with animation is merely an example, and the method of rearranging virtual objects with animation may be different.

[0145] As described above, in this embodiment, the operation of the information processing device A100 when rearranging virtual objects with animation to prevent degradation of the user experience has been described.

[0146] This allows the user to recognize that the virtual object has been rearranged, and there is no discrepancy between the actual position of the virtual object and the user's perception, so the virtual object can be rearranged while minimizing a deterioration in the user experience.

[0147] 6 , for example, in step S604, if the animation in which the virtual object is being rearranged is in progress, the control unit A101 determines that the virtual object needs to be rearranged until the rearrangement is complete. Furthermore, in step S605, if the animation in which the virtual object is being rearranged is in progress, the control unit A101 rearranges the virtual object in accordance with the animation. Furthermore, in step S606, the control unit A101 combines the virtual object with a captured image of real space in accordance with the arrangement of the virtual object rearranged in step S605, to generate an image of mixed reality space as a display image.

[0148] Although the preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present disclosure.

[0149] In the first, second, third, and fourth embodiments, it is assumed that the virtual object is reduced in size when it is rearranged. However, depending on the size of the virtual object, it may be sufficient to simply move the virtual object closer. Therefore, the position of the virtual object may be adjusted so that it is placed in front of the real object without reducing the virtual object.

[0150] (Other Embodiments) The present disclosure can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and a computer (or a control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0151] Although the present disclosure has been described in detail above based on preferred embodiments thereof, the present disclosure is not limited to these specific embodiments, and various forms within the scope of the gist of the present disclosure are also included in the present disclosure. Parts of the above-described embodiments may be combined as appropriate.

[0152] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0153] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0154] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, programmable logic devices, etc.).

[0155] The disclosure of this embodiment includes the following configuration, method, and program.

[0156] [Configuration 1] An information processing apparatus comprising: an acquisition means for acquiring information regarding an arrangement of real objects in a real space; an image acquisition means for acquiring a captured image of the real space in a predetermined direction; and a control means for controlling generation of an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein the control means controls generation of the image of the mixed reality space such that, when a position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction.

[0157] [Configuration 2] The information processing device according to Configuration 1, wherein the control means generates an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, and then controls the generation of an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space.

[0158] [Configuration 3] The information processing device according to Configuration 2, wherein the control means controls the generation of an image of a third mixed reality space in which the virtual object is placed at a position between the first position and the second position, after generating an image of the first mixed reality space and before generating an image of the second mixed reality space.

[0159] [Configuration 4] The information processing device according to any one of Configurations 1 to 3, wherein, when generating an image of the mixed reality space, the control means controls the virtual object to be reduced in size in the mixed reality space from the size at the first position and placed at the second position.

[0160] [Configuration 5] The information processing apparatus according to Configuration 4, wherein the control means reduces the virtual object based on the size of the virtual object in the mixed reality space at the first position.

[0161] [Configuration 6] The information processing device according to Configuration 5, wherein the control means generates an image in the mixed reality space in which the virtual object is placed at the second position at a position and size that match the first position and the second position as seen by the user.

[0162] [Configuration 7] The information processing device according to any one of Configurations 1 to 6, wherein the control means controls a display unit to display the generated image of the mixed reality space, and after displaying an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, controls a display unit to display an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space.

[0163] [Configuration 8] The information processing device according to Configuration 7, wherein the control means controls to display a screen that prompts a user to select whether or not to move the virtual object after displaying an image of the first mixed reality space, and when the user selects to move the virtual object, controls to generate an image of the second mixed reality space.

[0164] [Configuration 9] The information processing device according to Configuration 8, wherein the control means displays an image of the first mixed reality space when the user selects not to move the virtual object.

[0165] [Configuration 10] The information processing device according to any one of Configurations 1 to 9, wherein the control means controls a display unit to display the generated image of the mixed reality space, and controls the display unit to not display an image of a first mixed reality space in which the virtual object is located at the first position in the mixed reality space, but to display an image of a second mixed reality space in which the virtual object is located at the second position in the mixed reality space.

[0166] [Configuration 11] The information processing device according to Configuration 10, wherein the control means controls to display a screen that prompts a user to select whether or not to move the virtual object before displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to display the image of the second mixed reality space.

[0167] [Configuration 12] The information processing device according to Configuration 11, wherein the control means controls the display of the image of the first mixed reality space when the user selects not to move the virtual object.

[0168] [Configuration 13] The information processing device according to any one of Configurations 1 to 12, wherein the control means generates an image of the mixed reality space in which the virtual object is placed at the second position by performing a predetermined affine transformation on the virtual object.

[0169] [Configuration 14] The information processing device according to any one of Configurations 1 to 13, wherein the second location is a location closer than a predetermined distance from the first location or a location farther than a predetermined distance from the user's location.

[0170] [Configuration 15] The information processing device according to any one of configurations 4 to 14, wherein when the virtual object is reduced, the virtual object is not reduced below a predetermined size.

[0171] [Configuration 16] The information processing device described in any one of Configurations 1 to 15, wherein when a plurality of virtual objects exist in the mixed reality space and at least a part of a first virtual object among the plurality of virtual objects is at the first position, the control means controls to determine positions to rearrange the plurality of virtual objects based on the second position in the mixed reality space at which the first virtual object is to be arranged, and generate an image of the mixed reality space.

[0172] [Configuration 17] The information processing device according to any one of Configurations 1 to 16, wherein the control means controls whether or not to place the virtual object at the second position depending on the type of the virtual object.

[0173] [Control method] A control method for an information processing device, comprising: an acquisition step of acquiring information regarding an arrangement of real objects in a real space; an image acquisition step of acquiring a captured image of the real space in a predetermined direction; and a control step of controlling to generate an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein in the control step, when a position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, control is performed to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction.

[0174] [Program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17.

[0175] [System] An information processing system comprising: an acquisition device that acquires information regarding the arrangement of real objects in real space; an image acquisition device that acquires a captured image of the real space in a predetermined direction; and a control device that controls to generate an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein the control device controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object.

[0176] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, to apprise the public of the scope of the present invention, the following claims are appended.

[0177] This application claims priority based on Japanese Patent Application No. 2024-107716, filed July 3, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device comprising: an acquisition means for acquiring information regarding the arrangement of real objects in real space; an image acquisition means for acquiring a captured image of the real space in a predetermined direction; and a control means for controlling the generation of an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein the control means controls the generation of an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object.

2. The information processing device according to claim 1, characterized in that the control means controls the generation of an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, and then generates an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space.

3. The information processing device according to claim 2, characterized in that the control means controls the generation of an image of a third mixed reality space in which the virtual object is placed at a position between the first position and the second position, after generating an image of the first mixed reality space and before generating an image of the second mixed reality space.

4. The information processing device according to claim 1, characterized in that, when generating an image of the mixed reality space, the control means controls the virtual object to be reduced in size in the mixed reality space from the size at the first position and placed at the second position.

5. The information processing device according to claim 4, wherein the control means reduces the virtual object based on the size of the virtual object in the mixed reality space at the first position.

6. The information processing device according to claim 5, characterized in that the control means generates an image of the mixed reality space in which the virtual object is placed at the second position at a position and size that matches the first position and the second position as seen by the user.

7. The information processing device according to claim 1, characterized in that the control means controls the display unit to display the generated image of the mixed reality space, and controls the display unit to display an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space, and then controls the display unit to display an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space.

8. The information processing device according to claim 7, wherein the control means controls to display a screen that prompts the user to select whether or not to move the virtual object after displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to generate an image of the second mixed reality space.

9. The information processing device according to claim 8, wherein said control means displays an image of said first mixed reality space when said user selects not to move said virtual object.

10. The information processing device according to claim 1, characterized in that the control means controls the display unit to display the generated image of the mixed reality space, and controls the display unit to display an image of a second mixed reality space in which the virtual object is placed at the second position in the mixed reality space, without displaying an image of a first mixed reality space in which the virtual object is placed at the first position in the mixed reality space.

11. The information processing device according to claim 10, characterized in that the control means controls to display a screen that prompts the user to select whether or not to move the virtual object before displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to display the image of the second mixed reality space.

12. The information processing device according to claim 11, wherein the control means controls the display of the image of the first mixed reality space when the user selects not to move the virtual object.

13. The information processing device according to claim 1, characterized in that the control means generates an image of the mixed reality space in which the virtual object is placed at the second position by performing a predetermined affine transformation on the virtual object.

14. The information processing device according to claim 1, wherein the second location is a location closer than a predetermined distance from the first location, or a location farther than a predetermined distance from the user's location.

15. The information processing device according to claim 4, wherein when the virtual object is reduced, the virtual object is not reduced below a predetermined size.

16. The information processing device according to claim 1, characterized in that, when a plurality of virtual objects exist in the mixed reality space and at least a portion of a first virtual object among the plurality of virtual objects is at the first position, the control means determines positions to rearrange the plurality of virtual objects based on the second position in the mixed reality space at which the first virtual object is to be arranged, and controls to generate an image of the mixed reality space.

17. The information processing device according to claim 1, wherein said control means controls whether or not to place said virtual object at said second position depending on the type of said virtual object.

18. The information processing device according to claim 1, wherein the real object is a real object that is stationary in the real space.

19. A control method for an information processing device, comprising: an acquisition step of acquiring information regarding the arrangement of real objects in real space; an image acquisition step of acquiring a captured image of the real space in a predetermined direction; and a control step of controlling to generate an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein in the control step, when the position at which the virtual object is arranged in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, control is performed to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction.

20. A program for causing a computer to function as each means of the information processing device according to claim 1.

21. An information processing system comprising: an acquisition device that acquires information regarding the arrangement of real objects in real space; an image acquisition device that acquires an image of the real space in a predetermined direction; and a control device that controls to generate an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object, wherein the control device controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction, when the position at which the virtual object is placed in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object.

Images