Information processing system and information processing program

Abstract

To provide an information processing system and program configured to move a user even in a space within a limited range while displaying a virtual object in the real world or in virtual reality.SOLUTION: An information processing system / apparatus 10 includes: a display unit 15; a space recognition unit 12 capable of recognizing a real space; a tracking unit 13 capable of recognizing the position and orientation of a user; a storage unit 18 which stores data on a virtual space object to be displayed on the display unit 15; and a control unit 11 which arranges the virtual space object on real-space coordinates and causes the display unit to display a virtual object according to the position and orientation of the user. The control unit is configured to: recognize a boundary in the real space; cause, when the user moving in the real space reaches the boundary, the display unit to display display to prompt the user to change the orientation; and causes, when the user moves up to a desired orientation, the display unit to display an image as if the virtual object that has been located in front of the user before the movement is existing in front of the user.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing system and an information processing program for displaying virtual reality or mixed reality. [Background technology]

[0002] Patent Document 1 proposes a three-dimensional display device that uses mixed reality (MR) to perform rehabilitation. Patent Document 1 describes a system that uses mixed reality technology to place virtual numbers in the real world and allows the user to select one of the numbers to perform rehabilitation.

[0003] Patent Document 2 also proposes a rehabilitation system that uses mixed reality. In addition to the number selection rehabilitation described in Patent Document 1, Patent Document 2 describes a rehabilitation program that uses mixed reality technology to display virtual walls, mazes, grids, etc. in the real world, allowing the user to walk through the real space and complete given tasks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-10441 [Patent Document 2] International Publication No. 2020-152779 [Patent Document 3] Japanese Patent Publication No. 2019-020905 [Patent Document 4] Special Publication No. 2019-537087 [Patent Document 5] Japanese Patent Application Laid-Open No. 2017-111515 [Patent Document 6] International Publication No. 2020-203831 [Patent Document 7] International Publication No. 2020-152779 [Patent Document 8] Japanese Patent Publication No. 2021-058389 Summary of the Invention [Problem to be solved by the invention]

[0005] Although Patent Document 1 uses mixed reality, it does not disclose a system in which the user moves. In Patent Document 2, the movement of a user is taken into consideration using mixed reality, but the presence of obstacles and walls in the real world is not taken into consideration.

[0006] Using mixed reality technology to display virtual objects in the real world and allow users to move around can be applied to fields other than rehabilitation. When using mixed reality technology to display virtual objects in the real world and allow a user to move around them, the presence of walls and obstacles in the real world must be taken into consideration. Furthermore, even if there is a wall in the real world, there has never been a system that can transport a user to the virtual world that exists beyond that wall. Therefore, an object of the present invention is to provide a system that uses mixed reality technology to allow a user to move around while displaying virtual objects in the real world.

[0007] Furthermore, even when a virtual reality (VR) is displayed and the user is immersed in the virtual reality, the presence of walls and obstacles must be taken into consideration when moving the user. Therefore, an object of the present invention is to provide a system that uses virtual reality technology to allow a user to move while displaying virtual objects in virtual reality. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention has the following features: The present invention is an information processing system including a display unit capable of displaying images related to mixed reality or virtual reality, a tracking unit capable of recognizing the position and orientation of a user, a storage unit that stores data related to virtual space objects to be displayed on the display unit, and a control unit that displays the virtual objects on the display unit in accordance with the position and orientation of the user recognized by the tracking unit.

[0009] The control unit sets a boundary for the user to move through, and when the user reaches the boundary while moving through real space, the control unit notifies the user to prompt the user to change orientation, and when the user moves to the desired orientation, the control unit displays an image on the display unit as if the virtual object that was in front of the user before the movement started is actually in front of the user.

[0010] In one embodiment, while the user is moving to the desired orientation, the control unit moves the position coordinates of the virtual object in accordance with the user's rotation, and displays an image on the display unit as if the virtual object that was in front of the user before the movement started is actually in front of the user.

[0011] In one embodiment, while the user is moving to the desired orientation, the control unit displays on the display unit an image of a virtual object that was in front of the user before the user started moving, and after the user has moved to the desired orientation, moves the position coordinates of the virtual object that was in front of the user before the user started moving as if it were in front of the user after the movement, and displays the image on the display unit.

[0012] Preferably, the control unit determines whether an obstacle has appeared in front of the user, and if an obstacle has appeared, causes the display unit to display a message instructing the user to stop moving.

[0013] The present invention also provides a program executed by a control unit in an information processing system that includes a display unit capable of displaying images related to mixed reality or virtual reality, a tracking unit capable of recognizing a user's position and orientation, a memory unit that stores data related to virtual space objects to be displayed on the display unit, and a control unit that displays the virtual objects on the display unit in accordance with the user's position and orientation recognized by the tracking unit.

[0014] The information processing program causes the control unit to set a boundary when the user moves, and when the user reaches the boundary while moving in real space, executes a notification to prompt the user to change orientation, and when the user moves to the desired orientation, causes the display unit to display an image as if the virtual object that was in front of the user before the movement started is actually in front of the user. [Effects of the Invention]

[0015] As described above, according to the present invention, it is possible to provide a system and a program that allows a user to move within a limited space while displaying virtual objects in the real world or virtual reality.

[0016] These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram for explaining an overview of the processing executed in an information processing system 10 according to one embodiment of the present invention. [Figure 2] FIG. 2 is a diagram for explaining an outline of the processing executed by the information processing system 10 according to one embodiment of the present invention. [Figure 3] FIG. 3 is a diagram for explaining the processing executed in the information processing system 10 according to one embodiment of the present invention. [Figure 4]FIG. 4 is a diagram for explaining an outline of the processing executed by the information processing system 10 according to one embodiment of the present invention. [Figure 5] FIG. 5 is a diagram for explaining an outline of the processing executed in the information processing system 10 according to one embodiment of the present invention. [Figure 6] FIG. 6 is a diagram for explaining an outline of the processing executed by the information processing system 10 according to one embodiment of the present invention. [Figure 7] FIG. 7 is a block diagram of an information processing system 10 according to an embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart showing the operation of the information processing system 10 according to one embodiment of the present invention. [Figure 9] FIG. 9 is a flowchart showing the operation of the information processing system 10 according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] First, an overview of the processing executed in an information processing system 10 according to one embodiment of the present invention will be described with reference to FIGS. 1 to 6 and 7. FIG. The following explanation focuses on optical see-through mixed reality, but supplementary explanations will be provided for video see-through mixed reality and immersive virtual reality, where appropriate, if the operation differs. 1 to 6, the solid lines in (a) indicate a boundary 200 of the real space (such as a wall of a room), a user 201 positioned in the real space, and a movement space 203 in the real space. 1 to 6, the dashed lines in (a) indicate virtual objects placed in real space. Here, the virtual objects include a road 100, a crosswalk 101, a traffic light 102, a sidewalk 103 on the opposite side, a convenience store 104, and a sidewalk 105 on the near side in the virtual space.

[0019] In the following description, it is assumed that the user 201 is viewing a virtual object using a display unit 15 such as a head-mounted display or smart glasses worn on the head.

[0020] The virtual object is displayed in the real space, which is recognized by the mixed reality (MR) technology. In this case, the display unit 15 used for the mixed reality may be an optical see-through type (transmissive type) or a video see-through type. Furthermore, the virtual object may be displayed in a virtual space using virtual reality (VR) technology. In this case, the display unit 15 used for the virtual reality may be an immersive type. That is, in the present invention, the display unit 15 may be any of an optical see-through type, a video see-through type, and an immersive type.

[0021] An information processing system 10 used in this embodiment has the functional blocks shown in Fig. 7. In Fig. 7, the information processing system 10 includes a control unit 11, a spatial recognition unit 12, a tracking unit 13, an image processing unit 14, a display unit 15, a communication unit 16, an output unit 17, a storage unit 18, and an operation unit 19. The information processing system 10 may be realized as a head-mounted display alone, or may be realized in combination with an external computer device connected by wire or wirelessly to the display unit 15, which is a pair of smart glasses or a head-mounted display.

[0022] The control unit 11 executes programs necessary for the operation of this embodiment and controls various devices.

[0023] The space recognition unit 12 is a device that recognizes the space around the user and represents objects existing in the real space as digital data. The digital data in the real space recognized by the space recognition unit 12 is stored in the storage unit 18 as real space coordinates.

[0024] The tracking unit 13 is a device that can track at least the orientation and position of the user (position tracking). In addition, the tracking unit 13 may be equipped with well-known tracking functions such as eye tracking for tracking the user's line of sight, hand tracking for tracking the user's hand movements, and head tracking for moving the direction of a sound source according to the orientation of the user's head.

[0025] The position of the user tracked by the tracking unit 13 is stored as user position information in the storage unit 18. In addition, the orientation of the user tracked by the tracking unit 13 is stored in the storage unit 18 as user orientation information.

[0026] The communication unit 16 is a device for communicating with the outside. The output unit 17 is a device for outputting sound and the like. The storage unit 18 is a memory or the like. The operation unit 19 is an operation device such as various switches and a touch panel.

[0027] Information about virtual space objects is stored in storage unit 18. The information about the virtual space objects includes at least information about the shape of the virtual space objects and position coordinates of where the virtual space objects are located. Note that the position coordinates of the virtual objects are positions on coordinates representing real space (real space coordinates) in the case of mixed reality, and positions on coordinates representing virtual reality (virtual space coordinates) in the case of virtual reality. In the case of mixed reality, the control unit 11 places a virtual space object on the real space coordinates, and causes the display unit 15 to display the virtual space object based on the user position information and the user orientation information. In the case of virtual reality, the control unit 11 places a virtual object on the virtual reality coordinate system, and causes the display unit 15 to display the virtual space object based on the user position information and the user orientation information.

[0028] When the display unit 15 is an optical see-through type, the real space is visible to the user through the see-through display, and virtual space objects according to the user's position information and user orientation information are displayed on the transparent display, allowing the user to perceive the virtual objects as if they exist in the real space that is visible through the see-through display.

[0029] When the display unit 15 is a video see-through type, the information processing system 10 also includes an imaging unit for capturing a peripheral image, and the control unit 15 combines the virtual space object with the peripheral image captured by the imaging unit and displays it on the display unit 15. This allows the user to perceive the virtual object as if it were present on the image of real space.

[0030] When the display unit 15 is an immersive type, the control unit 15 recognizes boundaries and obstacles in the real space coordinates, then constructs a virtual space on the real space coordinates, places virtual space objects in the virtual space, and displays them on the display unit 15. As a result, even for virtual objects placed in the virtual space, the relationship with boundaries and obstacles in the real space coordinates is taken into consideration, so the control unit 15 can prompt the user to avoid the boundaries and obstacles. The user can recognize that virtual space objects exist in the virtual space.

[0031] 1, the space recognition unit 12 recognizes the space around the user 201, including the boundary 200, and digitizes it. The digitized real space is stored in the storage unit 18 as real space coordinates.

[0032] In the example shown in Figure 1, the task is to cross the crosswalk when the light is green and go to convenience store 104. At this time, the spatial recognition unit 12 can recognize that the user 201 is located at the position shown in Fig. 1. The tracking unit 13 can recognize that the user 201 is facing north (upper side of the paper).

[0033] In the case of mixed reality, the control unit 11 arranges the virtual space objects 100 to 105 on the real space coordinates so that the virtual space objects 100 to 105 are in the direction that the user 201 is facing. The virtual space objects 100 to 105 shown by dashed lines in FIG. 1(a) are the results after arrangement. When virtual reality is used, the control unit 11 places a virtual space object on the virtual space coordinates.

[0034] 1(a), the control unit 11 causes the image processing unit 14 to generate an image aligned with the orientation of the user 201 and displays it on the display unit 15. As a result, as shown in the image 202 in FIG. 1(b), the user 201 can recognize that the virtual space objects 100 to 105 are present in front of him or her. In this case, in the case of an optical see-through type, the user sees through the real world. In the case of a video see-through type, the user sees a display in which virtual space objects are superimposed on an image of the real world. In the case of an immersive type, the user sees virtual space objects 100 to 105 as part of a virtual reality world.

[0035] As shown in FIG. 2(a), assume that the user 201 moves further north from the position in FIG. 1(a). The control unit 11 causes the image processing unit 14 to generate an image corresponding to the user's position and orientation based on the user's position information and user orientation on the real space coordinates, and causes the display unit 15 to display the image.

[0036] In Fig. 2(a), the user 201 is positioned at the position of the sidewalk 105. Therefore, as shown in Fig. 2(b), unlike Fig. 1(b), the display unit 15 used by the user 201 does not display the sidewalk 105 in front, but displays what is ahead of the crosswalk 101.

[0037] Here, the control unit 11 determines that the user 201 has approached the boundary 200 and that it is dangerous to proceed any further. Then, as shown in FIG. 2(c), the control unit 11 causes the display unit 15 to display a message urging the user to stop moving and change direction. For example, the control unit 11 causes the display unit 15 to display "Please turn 180 degrees left."

[0038] Assume that the user 201 changes his / her orientation in accordance with the display in FIG. 2(c). FIG. 3(a) shows the state when the user faces northwest. After receiving an instruction to change direction, in the case of mixed reality (optical see-through type and video see-through type), the control unit 11 also rotates the coordinates of the virtual space object on the real space coordinate in accordance with the change in the user orientation information, as shown by the dashed line in FIG. 3(a). In the case of virtual reality, the control unit 11 rotates the virtual reality space coordinates including the virtual space object. Then, each time a rendering timing arrives, the control unit 11 causes the image processing device 14 to generate an image in accordance with the user orientation information and displays it on the display unit 15.

[0039] 3(b), the image at the time when the user 201 starts to change direction is displayed on the display unit 15. In addition, the control unit 11 causes the display unit 15 to display the remaining rotation angle (here, 135°).

[0040] Similarly, when the user 201 faces west, the coordinates of the virtual space object are updated as shown in Fig. 4(a). Then, as shown in Fig. 4(b), the image at the time when the user 201 starts to change direction is displayed on the display unit 15 as it is. The control unit 11 causes the display unit 15 to display the remaining rotation angle (90°).

[0041] FIG. 5(a) shows the state when the user 201 continues to rotate and faces south. At this time, the coordinates of the virtual space object are updated in the same way. Then, as shown in FIG. 5(b), the image of the user 201 when he started to change direction is displayed on the display unit 15. Then, the control unit 11 causes the display unit 15 to display a message to stop the rotation.

[0042] After the state shown in FIG. 5, the state shown in FIG. 6 is reached. That is, the virtual space object after the user 201 rotates 180° is shown by the dashed line in FIG. 6(a). From the user 201's perspective, this state means that the user 201 can continue walking from the state shown in FIG. 2(a). As shown in FIG. 6(b), the warning on the display unit 15 is erased, and the user 201 can continue walking on the crosswalk 101.

[0043] In the state of Figure 6(a), in the case of mixed reality, the coordinates of the virtual space object are placed in the real space coordinates, but the differences between Figures 2(a) to 5(a) and Figure 6(a) will be explained below. In the states of FIGS. 2(a) to 5(a), the virtual space object rotates horizontally on the real space coordinate system in accordance with the horizontal rotation of the user 201. Once the state shown in FIG. 6(a) is reached, even if the user 201 rotates or moves, the coordinates of the virtual space object do not change and the object remains positioned in the real space coordinates. In the case of virtual reality, the virtual reality space coordinates including the virtual space objects all move in accordance with the user's movements, so when the state shown in Figure 6(a) is reached, the virtual reality space coordinates including the virtual space objects become fixed and do not move in accordance with the user's movements.

[0044] That is, when control unit 11 instructs the user to rotate, the virtual space object rotates in accordance with the user's horizontal rotation, and in the case of mixed reality, is placed in real space coordinates (in the case of virtual reality, the virtual space coordinates move) while rotating. Then, when the user rotates to a predetermined position, the virtual space object is fixed in the real space coordinates in the case of mixed reality (in the case of virtual reality, the virtual space coordinates are fixed), and thereafter, the user's movement and the virtual space object are no longer linked.

[0045] The operation of the program executed by the control unit 11 will be described with reference to FIGS. When the timing for displaying a frame on the display unit 15 arrives (S101), the control unit 11 recognizes the position, shape, etc. of an object existing in the real space based on data from the space recognition unit 12, and generates real space coordinates (S102).

[0046] Next, the control unit 11 determines the boundary in the real space based on the recognized real space coordinates (S103). There are various methods for determining the boundary. For example, the control unit 11 recognizes a predetermined distance in front of an object (for example, a wall, a desk, a chair, etc.) that exists in real space as a boundary.

[0047] There is a possibility that a person, a wheelchair, or the like may suddenly move and appear in the real space, so the control unit 11 determines whether or not an obstacle has appeared between the user and the boundary (S104). If an obstacle appears, the control unit 11 causes the display unit 15 to display a message instructing the user to pause (S110), and waits until the next display frame in S101. The determination as to whether an obstacle appears may not be performed in accordance with the sequence of FIG. 8, but may be constantly monitored by the control unit 11 and executed as an interrupt process.

[0048] If no obstacle appears, the control unit 11 proceeds to the operation of S 105. In the operation of S105, the control unit 11 places a virtual space object. Next, the control unit 11 recognizes the position and orientation of the user in the real space coordinate system based on the data from the tracking unit 13 (S106).

[0049] The control unit 11 determines whether or not the user has reached the boundary of the real space (S107). If the user has reached the boundary, the control unit 11 proceeds to process A (FIG. 9). If the user has not reached the boundary, the control unit 11 renders a virtual space object for the area visible to the user (S108), and causes the display unit 109 to display an image (S109). After S109, the control unit 11 waits until the timing of the next frame arrives.

[0050] 9, the control unit 11 causes the display unit 109 to display an image at the time when the boundary is reached (S201). Then, the control unit 11 causes the display unit 109 to display an image instructing a change in orientation (S202).

[0051] For example, as shown in Figure 9, "Please rotate 180 degrees left." is displayed. The control unit 11 recognizes the position of the user in real space coordinates based on data from the tracking unit 13, and can therefore determine how many degrees of rotation is necessary to allow the user to continue walking in a comfortable direction. Therefore, the control unit 11 instructs the user to rotate to a direction that allows the user to continue walking in a comfortable direction, and the rotation angle is not limited to 180°.

[0052] While the user is rotating, the control unit 11 can continue to recognize the user's position and orientation in real space using data from the tracking unit 13, and can therefore recognize the remaining required rotation angle. Therefore, the control unit 11 causes the display unit 15 to display an image instructing the user to change orientation while indicating how many degrees remain (S203).

[0053] The control unit 11 rotates and positions the virtual object on the real space coordinates in accordance with the user's rotation, renders it, and displays it on the display unit 15 (S204). In the case of virtual reality, the control unit 11 moves the virtual object along with the virtual space coordinates.

[0054] The control unit 11 continues to determine whether or not the user has turned in the desired direction after the instruction (S205), and executes the operations of S203 and S204 until the user has turned in the desired direction after the instruction. When the user faces the direction after the instruction (YES in S205), the control unit 11 causes the display unit 15 to display an image instructing the user to stop changing the direction (S206).

[0055] Based on data from the tracking unit 13, the control unit 11 recognizes that the user's rotation has stopped, and fixes the virtual object to the real space coordinates at the point in time when the rotation has stopped. After that, even if the user moves, the virtual space object does not move within the real space coordinates. In the case of virtual reality, the control unit 11 fixes the virtual space coordinates so that they do not move. Then, the control unit 11 performs rendering, and displays an image aligned with the user's position and orientation on the display unit 15 (S207), and the process returns to process B in FIG. 8.

[0056] Returning to process B in Fig. 8, the control unit 11 recognizes the real space and the position and orientation of the user at the timing of each display frame, and renders an image of the virtual object in the area visible to the user. As a result, the image displayed on the display unit 15 changes every time the user moves, and the user feels as if they are walking among virtual space objects that exist in real space (or virtual space).

[0057] (Variation) In the above embodiment, the virtual object is constantly rendered and displayed while rotating in real space coordinates while the user is rotating, but the present invention does not necessarily require such an operation. For example, when the user reaches a boundary and must rotate, an image immediately before the start of rotation may be stored and displayed on display unit 15, while the remaining rotation angle is displayed, and the virtual object itself is not moved in the real space coordinates. Once the rotation is complete, the virtual object may be repositioned in the real space coordinates, so that the user begins to move.

[0058] In the above embodiment, while the user is rotating, an image of the virtual object before the user starts moving is displayed on the display unit 15. However, while the user is moving, an image of the real space through which the user is passing or an image of the real space or virtual space aligned with the user's orientation may be displayed on the display unit 15, and after the user has moved to the desired orientation, an image of the virtual object before movement may be displayed on the display unit 15.

[0059] In the above embodiment, the spatial recognition unit 12 is used to recognize the real space around the user, and the control unit 11 sets the boundary, but the method of setting the boundary by the control unit 11 is not limited to using the spatial recognition unit 12. For example, the control unit 11 sets a certain range (for example, a rectangular range with the user at the center) from the initial position of the user as a boundary. Then, the control unit 11 recognizes where the user is located within the certain range based on the moving distance and moving direction of the user obtained from the tracking unit 13, and can determine whether the user has reached the boundary. Alternatively, instead of determining a fixed range, the control unit 11 can set boundaries in advance based on the distance and direction in all directions from the user's initial position, and determine that the user has reached the boundary when they move within that distance and in that direction. Therefore, in the present invention, the control unit only needs to set the boundary for the user to move, and various methods can be used to set the boundary, regardless of whether mixed reality or virtual reality is used.

[0060] In the above embodiment, when urging the user to change direction, the control unit 11 displays a message to that effect on the display unit 15, but the control unit 11 may also urge the user to change direction by using voice, a warning sound, vibration, etc. Therefore, when the user reaches a boundary while moving in real space, the control unit 11 may execute a notification (by display, voice, a warning sound, vibration, etc.) urging the user to change direction.

[0061] As described above, according to one embodiment of the present invention, even in a real space with a limited range and a boundary, a virtual object can be placed, and when the user reaches the boundary, the user is prompted to change direction, and after the user changes direction, the user can move while viewing the virtual object placed on the real space coordinates after the change of direction. This makes it possible to provide a system that allows a user to move around even in a limited space while displaying virtual objects in the real world or virtual reality.

[0062] Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit its scope. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. Each of the constituent elements of the invention disclosed in this specification is considered to be an independent, stand-alone invention. Inventions that combine the constituent elements in any manner are also included in the present invention. The specific expressions in this specification are merely examples, and the present invention also includes those that conceptualize these exemplary expressions. [Industrial Applicability]

[0063] The present invention relates to an information processing system and an information processing program, and is industrially applicable. [Explanation of symbols]

[0064] 10 Information Processing Systems 11 Control section 12 Spatial recognition section 13 Tracking Section 14 Image processing section 15 Display 16 Communications Department 17 Output section 18 Memory section 19 Control section 100 Virtual Space Object Roads 101 Virtual Space Object Walkway 102 Virtual Object Traffic Light 103 Sidewalk opposite the virtual space object 104 Virtual Space Object Convenience Store 105 Sidewalk in front of virtual space object 200 Boundaries of Real Space 201 users 202 Images viewed by users 203 Movement space in real space coordinates

Claims

1. a display unit capable of displaying images relating to mixed reality or virtual reality; a tracking unit capable of recognizing the position and orientation of a user; a storage unit that stores data related to a virtual space object to be displayed on the display unit; a control unit that displays the virtual object on the display unit in accordance with the position and orientation of the user recognized by the tracking unit, The control unit setting boundaries within which the user may move; When the user reaches the boundary while moving through real space, a notification is sent to the user prompting the user to change orientation; When the user moves to a desired orientation, an image is displayed on the display unit as if the virtual object that was in front of the user before the user started moving were actually in front of the user.

2. 2. The information processing system according to claim 1, wherein the control unit, while the user is moving to the desired orientation, moves the position coordinates of the virtual object in accordance with the rotation of the user, and displays an image on the display unit as if the virtual object, which was in front of the user before the movement started, is actually in front of the user.

3. 2. The information processing system according to claim 1, wherein the control unit causes the display unit to display an image of the virtual object that was in front of the user before the user started moving while the user is moving to the desired orientation, and after the user has moved to the desired orientation, the control unit moves the position coordinates of the virtual object that was in front of the user before the user started moving as if it were in front of the user after the movement, and displays the image on the display unit.

4. 2. The information processing system according to claim 1, wherein the control unit determines whether an obstacle has appeared in front of the user, and, if the obstacle has appeared, causes the display unit to display a message instructing the user to stop moving.

5. a display unit capable of displaying images relating to mixed reality or virtual reality; a tracking unit capable of recognizing the position and orientation of a user; a storage unit that stores data related to a virtual space object to be displayed on the display unit; a control unit that displays the virtual object on the display unit in accordance with the position and orientation of the user recognized by the tracking unit, The control unit allowing the user to set boundaries for movement; When the user reaches the boundary while moving in real space, a notification is executed to prompt the user to change orientation, and the notification is displayed on the display unit; When the user moves to a desired orientation, the information processing program causes the display unit to display an image as if the virtual object that was in front of the user before the user started moving were actually in front of the user.

Images