Virtual object display device, program thereof, and virtual object display system

Abstract

To provide a virtual object display device which makes it easy to recognize the position and size of a virtual object.SOLUTION: The virtual object display device 1 includes a video generation unit 14 that is connected via a network to a server 5 that stores a spherical video obtained by photographing a real space with a spherical camera and position information of a virtual object, and generates and displays, as a display video, an MR video obtained by mixing the spherical video and a video obtained by photographing the virtual object with the virtual camera arranged at the center of the spherical camera, or a virtual video obtained by photographing the virtual object with the virtual camera arranged at an arbitrary position, and a video switching unit 130 that switches the display video to the MR video or the virtual video based on an operation instruction.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a virtual object display device, a program therefor, and a virtual object display system. [Background technology]

[0002] In recent years, advances in mixed reality technology, which overlays real-world images captured in real space with virtual objects, have made it possible to display images that seamlessly combine images of the real world and virtual objects (hereinafter referred to as MR images). This mixed reality technology allows a user wearing a head-mounted display (HMD) to recognize virtual objects through the images displayed on the HMD.

[0003] Also, a technology has been disclosed in which an HMD is used in a web conference or the like, and one of two users at different locations wears the HMD and shares an MR image displayed on the HMD (see Patent Document 1). In the method of Patent Document 1, only the person wearing the HMD can change the viewpoint, and users not wearing the HMD (web conference attendees) cannot freely manipulate the viewpoint. Therefore, the present inventor has disclosed a method that enables a user who is not wearing an HMD to view and operate a virtual object superimposed on an image of the real world on a monitor (see Non-Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-033575 [Non-patent literature]

[0005] [Non-Patent Document 1] Akiyuki Ihara, "XR Opens Up RX (Research Transformation)," Information Processing Society of Japan, Journal "Information Processing: Digital Practice Corner," Vol. 64, No. 8, August 2023 Summary of the Invention [Problem to be solved by the invention]

[0006] The technique described in Non-Patent Document 1 is advantageous in that even a user who is not wearing an HMD can operate a virtual object in an MR image. However, when a user views a virtual object superimposed on an image of the real world, the virtual object is always displayed in front, making it difficult to recognize the position and size of the virtual object, and there was room for improvement.For example, it was difficult for a user not wearing an HMD to recognize how big the virtual object was, how far away it was, and whether the virtual object was in front of or behind the user wearing the HMD.

[0007] Therefore, an object of the present invention is to provide a virtual object display device and program thereof, as well as a virtual object display system, that allows a user not wearing an HMD to easily recognize the position and size of a virtual object. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the virtual object display device of the present invention uses an object object representing an object and a spherical object representing a celestial camera as virtual objects, is connected via a network to a server that stores celestial spherical images of real space captured by the celestial camera and positional information of the virtual objects, and displays the virtual objects, and is configured to include an image generation unit and an image switching unit.

[0009] In such a configuration, the virtual object display device generates and displays, as a display image, an MR image that combines a celestial sphere image and an image of the virtual object captured by placing a virtual camera at the center of the celestial sphere camera, or a virtual image of the virtual object captured by placing a virtual camera at an arbitrary position, based on the position information of the virtual object, using an image generation unit. By combining a celestial sphere image and an image of a virtual object as an MR image, the user can see through the virtual object in the background even if the virtual object is in the foreground. Furthermore, by displaying a virtual image captured by a virtual camera placed at an arbitrary position, the user can confirm the position and size of a virtual object.

[0010] Then, the virtual object display device switches the display image between the MR image and the virtual image by the image switching unit based on the operation instruction. This allows the virtual object display device to visualize and present the position and size of the virtual object to the user. The virtual object display device can be operated by a virtual object display program that causes a computer to function as each of the above-mentioned units.

[0011] Furthermore, in order to solve the above-mentioned problems, the virtual object display system of the present invention is a virtual object display system in which virtual objects are an object object representing an object and a spherical object representing a celestial camera, a celestial camera placed in real space, a server that stores celestial images of the real space captured by the celestial camera and positional information of the virtual objects, and a virtual object display device, all connected to a network, and the virtual object display device is configured to include an image generation unit and an image switching unit.

[0012] In such a configuration, the virtual object display system stores the celestial sphere image captured by the celestial sphere camera and the position information of the virtual object in the server. The virtual object display device connected to the server via a network generates and displays, as a display image, an MR image that combines a celestial sphere image and an image of the virtual object captured by placing a virtual camera at the center of the celestial sphere camera, or a virtual image of the virtual object captured by placing a virtual camera at an arbitrary position, based on the position information of the virtual object, using an image generation unit. Then, the virtual object display device switches the display image between the MR image and the virtual image by the image switching unit based on the operation instruction. This allows the user of the virtual object display device to confirm the position and size of the virtual object. [Effects of the Invention]

[0013] According to the present invention, it is possible to present a virtual object superimposed on an image of the real world to a user who is not wearing an HMD in a manner that makes it easy to recognize the position and size of the object. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a configuration diagram showing the overall configuration of a virtual object display system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram (part 1) showing an example of an MR image viewed on a monitor by a user not wearing an HMD. [Figure 3] FIG. 2 is a diagram (part 2) showing an example of an MR image viewed on a monitor by a user not wearing an HMD. [Figure 4] FIG. 1 is a diagram showing an example of a virtual image (virtual space image) viewed on a monitor by a user not wearing an HMD. [Figure 5] FIG. 1 is a diagram showing an example of an MR image visually recognized by a user wearing an HMD through the HMD. [Figure 6] FIG. 2 is a block diagram showing the configuration of the virtual object display device of FIG. [Figure 7] FIG. 10 is an explanatory diagram for explaining an image in gaze mode. [Figure 8] FIG. 10 is an explanatory diagram for explaining an image in a tracking mode. [Figure 9] FIG. 2 is a block diagram showing the configuration of the HMD of FIG. [Figure 10] 10 is a flowchart illustrating an operation of the virtual object display device according to the embodiment of the present invention. [Figure 11] 11 is a flowchart showing the operation of the selection process of FIG. 10. [Figure 12] 11 is a flowchart showing the operation of the operation processing of FIG. 10. [Figure 13] 11 is a flowchart showing the operation of the video switching process of FIG. 10. [Figure 14] 11 is a flowchart showing the operation of the line-of-sight change processing of FIG. 10. [Figure 15] 11 is a flowchart showing the operation of the viewpoint change process of FIG. 10. [Figure 16] 11 is a flowchart showing the operation of the mode switching process of FIG. 10. [Figure 17] 11 is a flowchart showing the operation of the MR image generation process of FIG. 10. [Figure 18] 11 is a flowchart showing the operation of the virtual video generation processing of FIG. [Figure 19] FIG. 1 is a diagram showing an example of a conventional MR image. [Figure 20] FIG. 20 is a diagram showing an example of an MR image of the present invention corresponding to FIG. 19. DETAILED DESCRIPTION OF THE INVENTION

[0015] A virtual object display system according to an embodiment of the present invention will be described below with reference to the drawings.

[0016] <<Overall configuration of the virtual object display system>> First, with reference to FIG. 1, the overall configuration of a virtual object display system 100 according to an embodiment of the present invention will be described. The virtual object display system 100 presents an MR image in which a virtual object is drawn superimposed on the real world to a user M1 wearing an HMD3, and also presents an MR image in which a virtual object is drawn superimposed on an image of the real world to a user M2 not wearing an HMD3, from a different viewpoint than that of user M1. The virtual object display system 100 also presents MR images to a user M3 at a remote location who is not wearing the HMD 3, from a viewpoint different from that of the user M1. Here, the "local location" shown in Fig. 1 is a space (MR space) in which a real space and a virtual object are associated with each other. The "remote location" is a real space that is distant from the local location and is not associated with a virtual object.

[0017] The virtual object display system 100 uses an object object representing an object and a spherical object representing a celestial sphere camera 2 as virtual objects, and is configured by connecting the celestial sphere camera 2 placed in real space, a server 5 that stores celestial sphere images of real space captured by the celestial sphere camera 2 and position information of the virtual objects, and a virtual object display device 1 to a network via communication equipment 4. The spherical object indicates the position of the spherical camera 2 by its center position, and is a virtual object on which the spherical image captured by the spherical camera 2 is rendered. In addition, there is a real object in the real space (here, a desk O R ) is assumed to exist. Also, the virtual object display device 1 and the HMD 3 are assumed to store information about the virtual object (data such as three-dimensional structure and texture) in advance.

[0018] The virtual object display device 1 displays, as an MR image, an image of a specified direction of a specified celestial sphere image among images (celestial sphere images) captured by a celestial camera 2 (21, 22) on a monitor MO. At this time, the virtual object display device 1 generates the MR image by blending the celestial sphere image and the image of the virtual object captured by the virtual camera. The virtual object display device 1 also displays on the monitor MO a virtual image (image of the virtual space) obtained by virtually capturing a virtual space associated with the real space from an arbitrary viewpoint.

[0019] The virtual object display device 1 has a function of moving a virtual object, and stores in the server 5 position information when the virtual object is moved. The virtual object display device 1 is connected to a monitor (display device) MO and instruction devices (mouse MS, keyboard KB, etc.) that allow a user (operator) to perform various operations. The virtual object display device 1 acquires a desired celestial sphere image from a server 5 arranged on a network via a communication device 4.

[0020] Here, an example will be described in which two virtual object display devices 1 are installed, one at a local location and one at a remote location. However, the virtual object display device 1 may be installed in one of the local or remote locations, one or more at each of the local or remote locations, or one or more at each of a plurality of remote locations.

[0021] The celestial camera 2 is an omnidirectional camera that captures 360-degree images of the celestial sphere in all directions, up, down, left, and right. The celestial camera 2 sequentially transmits the captured images of the celestial sphere to the server 5 via the communication device 4. The celestial camera 2 may transmit celestial images to the server 5 via a personal computer (PC: not shown) equipped with a communication control unit and then via the communication device 4. Alternatively, the celestial camera 2 may be connected to the virtual object display device 1 and transmit celestial images to the server 5 via the virtual object display device 1 and then via the communication device 4. Here, an example will be described in which two celestial cameras 2 are placed on-site and one in a remote location. However, it is sufficient to place at least one celestial camera 2 in a space where real space and virtual objects are associated.

[0022] The HMD 3 is worn on the head of the user M1 and displays MR images in which virtual objects are drawn over the real world according to the position and orientation of the head of the user M1. The HMD 3 used must have a hand tracking function that allows the user M1 to move virtual objects in the MR images. The HMD 3 stores, in the server 5, position information when a virtual object is moved.

[0023] Furthermore, the HMD 3 may be an optical see-through type device that allows the real world to be viewed directly through a translucent display, or a video see-through type device that captures images of the real world and displays them on a display. Further, although the following description is given assuming that one user wears the HMD 3, multiple users may wear the HMD 3.

[0024] The communication device 4 is a communication device that transmits and receives data between the virtual object display device 1, the celestial camera 2, and the HMD 3 and the server 5. For example, the communication device 4 may be a Wi-Fi router that transmits and receives data wirelessly between the virtual object display device 1, the celestial camera 2, and the HMD 3.

[0025] The server 5 stores the celestial sphere images captured by the celestial camera 2 and the position information of the virtual objects placed in the virtual space. This server 5 can be configured with a general storage device such as a hard disk. The server 5 can be installed at any location connected via a network (for example, on the cloud).

[0026] The server 5 stores the celestial sphere images transmitted from the celestial sphere cameras 2 for each celestial sphere camera 2, and transmits the requested celestial sphere images to the virtual object display device 1 on a frame-by-frame basis. The server 5 also stores position information of virtual objects (virtual object position information) in advance and transmits it to the requested virtual object display device 1 and HMD 3. The position information of a virtual object includes position coordinates (three variables), angle (two variables), scale (one variable), etc. in a three-dimensional virtual space.

[0027] Each virtual object is previously set with individual identification information and information indicating the type of object, such as a physical object or a spherical object. Here, the physical object includes an avatar. An avatar is an alter ego of the user (operator) of the virtual object display device 1 placed in the virtual space.

[0028] When the position of a virtual object is changed by an operation such as moving the virtual object performed by the virtual object display device 1 or the HMD 3, the server 5 receives update information from the virtual object display device 1 or the HMD 3 and updates the stored position information. Although the server 5 stores the celestial sphere image and the position information here, the celestial sphere image and the position information may be stored in different servers.

[0029] With the above configuration, the virtual object display system 100 can switch between presenting to users M2 and M3 who are not wearing HMD3 via the monitor MO an MR image in which a virtual object is semi-transparently synthesized with the real world, and a virtual image in which a virtual object placed in a virtual space is viewed from any viewpoint.

[0030] In this way, the virtual object display system 100 can prevent the virtual object in the MR video from always being overwritten and displayed in front, which makes it easier for the user to visually recognize the position of the virtual object in relation to the front and rear. Furthermore, the virtual object display system 100 can display a virtual object placed in a virtual space from any viewpoint, which makes it easier for the user to grasp the position and size of the virtual object.

[0031] Before describing the configuration and operation of the virtual object display device according to the embodiment of the present invention, we will now describe examples of images displayed on the monitor MO and HMD 3. Note that although the multiple monitors MO differ depending on the viewpoints and directions specified by the users M2 and M3, the configuration of the images is the same. In addition, in figures such as Figure 2, which illustrate examples of images displayed on the monitor MO and HMD 3, objects (including virtual objects) represented by dotted lines indicate that they are drawn semi-transparently, and objects (including virtual objects) represented by solid lines indicate that they are drawn opaquely.

[0032] (Image from monitor) First, referring to FIG. 2, an MR image I generated based on the celestial sphere image captured by the celestial sphere camera 22 is displayed on the monitor MO. V2That is, MR image I V2 is an image generated with the virtual camera's viewpoint set at the center of the celestial spherical camera 22. Note that the virtual camera's viewpoint does not necessarily have to be the center of the celestial spherical camera 22, but can be any position near the center.

[0033] The virtual object display device 1 displays MR images I that combine images of the real world and images of virtual objects captured by a virtual camera. V2 is displayed on the monitor (display device) MO. MR Image I V2 Among them, user M1 wearing HMD3 and desk O R and the celestial camera 21 are real-world objects. All other objects are virtual objects. Here, an object O is used as an example of a virtual object. VT and a spherical object O V1 ,O V3 and,

[0034] Object O VT is a virtual object to be observed and manipulated, which is a three-dimensional structure generated by 3D (three-dimensional) CAD or the like. In addition, MR image I V2 Since the image of the real world and the image of the virtual object are mixed, the object O VT is the desk O, which is the real object on the back side. R can be seen through it.

[0035] Sphere Object O V1 is a virtual object in the shape of a sphere of a predetermined size (for example, 50 cm in diameter) that represents the celestial sphere camera 21. Sphere Object O V3 is a spherical virtual object representing the celestial camera 23 (see FIG. 1) at a remote location, and the user M1 can change the position and size of the object. In addition, the spherical object O V1 ,O V3 The texture is arbitrary, but for example, it can be a texture onto which a celestial sphere image is mapped.

[0036] Sphere Object O V1 is a virtual object whose center is initially set to the center position of the celestial spherical camera 21 in the real world and is assumed not to move. Sphere Object O V3 is a virtual object in which a remote celestial camera 23 is virtually placed in the local real world, and is a movable virtual object. V3 The initial position of the spherical object O is set near the HMD 3. V3 This can prevent you from losing sight of the This MR image I V2 Sphere object O V1 ,O V3 By selecting one of the cameras, the viewpoint is switched to the corresponding celestial spherical camera 21, 23.

[0037] Next, referring to FIG. 3, the MR image I generated based on the celestial sphere image captured by the celestial sphere camera 21 and displayed on the monitor MO is V1 That is, MR image I V1 is an image generated with the center of the celestial sphere camera 21 as the viewpoint of the virtual camera. MR Image I V1 is the same as the MR image I explained in Figure 2, except for the viewpoint position. V2 The contents are the same as those of the sphere object O. V2 is a virtual object whose center is initially set to the center position of the celestial spherical camera 22 in the real world and is assumed not to move. Although the MR image generated from the celestial spherical image captured by the remote celestial spherical camera 23 is not shown, in the configuration example of FIG. 1, only the spherical object is superimposed on the image of the real world.

[0038] Next, referring to Fig. 4, a virtual image I of a virtual object placed in a virtual space viewed from an arbitrary viewpoint is displayed on the monitor MO. VR That is, virtual image I VRis an image captured by a virtual camera in a virtual space from an arbitrary viewpoint away from the viewpoint of the celestial camera 2 (spherical object).

[0039] As shown in Figure 4, the virtual image I VR In this case, a virtual object (object O VT , a spherical object O V1 ,O V2 ,O V3 ) will be displayed. In addition, Virtual Image I VR is an image in which a virtual object is composited against a background of a predetermined color. In this case, the virtual object can be drawn by overwriting the background with the texture of the virtual object itself.

[0040] (HMD footage) Next, referring to Fig. 5, the MR image I displayed on the HMD 3 HMD This article explains: HMD3 is an MR image that overlays the real world or an image of the real world with a virtual object. HMD Display. MR Image I HMD In the desk R The celestial cameras 21 and 22 are objects in the real world. VT , and a spherical object O V1 ,O V2 ,O V3 are virtual objects. Each virtual object is the same as that explained with reference to FIGS. 2 to 4, and therefore the explanation will be omitted.

[0041] The user M1 wearing the HMD 3 can move to any position and change the viewpoint, making it easy to grasp the position of the virtual object. HMD is an image in which the virtual object is overwritten with its original texture.

[0042] <Configuration of virtual object display device> Next, with reference to Fig. 6 (and Fig. 1 as needed), a description will be given of the configuration of the virtual object display device 1. The virtual object display device 1 includes a storage unit ME and a control unit CL.

[0043] The storage unit ME stores information about virtual objects used in the virtual object display system 100, and can be configured from a general storage medium such as a semiconductor memory. Here, the storage unit ME includes a virtual object information storage unit 10.

[0044] The virtual object information storage unit 10 stores structural information of virtual objects. Here, the structural information is information that specifies the structure of the virtual object, such as information about a three-dimensional structure created by 3D CAD. Note that if the virtual object is a moving object, the structural information also includes animation information that indicates the movement.

[0045] The control unit CL performs various control operations of the virtual object display device 1. The control unit CL includes an instruction analysis unit 11, a virtual object control unit 12, a display image control unit 13, and an image generation unit .

[0046] The instruction analysis unit 11 analyzes instructions given by the user. Here, the instruction analysis unit 11 analyzes whether an instruction input from an externally connected instruction device such as a mouse MS is an instruction for a virtual object or an instruction for a displayed image.

[0047] An instruction for a virtual object is an instruction to control the virtual object. In this case, the instruction for a virtual object is information such as button operation (press / release) information of a mouse MC and key operation (press / release) information of a keyboard KB that are assigned in advance to operations on the virtual object, such as selection and operation of the virtual object.

[0048] The instructions for the displayed image are instructions to control the displayed image. Here, the instructions for the displayed image include mouse MC button operation (press / release) information, drag information, keyboard KB key operation (press / release) information, etc., which are assigned in advance to operations on the displayed image, such as switching between MR image and virtual image, changing the line of sight, and switching the display mode. The instruction analysis unit 11 outputs an instruction for a virtual object to the virtual object control unit 12, and outputs an instruction for a display image to the display image control unit 13.

[0049] The virtual object control unit 12 operates the virtual object according to the instruction input from the instruction analysis unit 11 . Here, the virtual object control unit 12 includes a selection unit 120 and an operation unit 121.

[0050] The selection unit 120 receives an instruction to select a virtual object, and then selects the specified virtual object as the operation target. When the left button of the mouse MS is pressed as an instruction to select a virtual object, for example, the selection unit 120 selects a virtual object corresponding to the position of the mouse MS on the monitor MO. The virtual object corresponding to the position of the mouse MS can be detected as the virtual object located closest to the position of the mouse MS by associating the position information of the virtual object within the angle of view of the display image specified by the display image control unit 13.

[0051] It is also possible to associate the virtual object with the virtual object position information of the server 5 and store information indicating the right to monopolize the operation, and the selection unit 120 may perform a process to acquire or release the right when selecting a virtual object.

[0052] The selection unit 120 stores identification information for identifying the selected virtual object in a memory or the like (not shown). If the selected virtual object is a spherical object, the selection unit 120 notifies the display image control unit 13 that the spherical object has been selected, along with the identification information.

[0053] When an instruction to operate a virtual object is input to the operation unit 121, the operation unit 121 performs an operation corresponding to the instruction on the virtual object, with the selected virtual object as the operation target. The operation unit 121 performs operations such as moving a virtual object in the direction of the arrow by pressing an arrow key, rotating a virtual object by pressing an arrow key while holding down the shift key, and enlarging or reducing a virtual object by pressing the " / " key or the "*" key. The operation unit 121 transmits the position information (position coordinates, angle, scale) of the virtual object that is changed in accordance with the operation of this virtual object to the server 5 and updates it.

[0054] The display image control unit 13 and the instruction analysis unit 11 operate the display image in accordance with the instructions input thereto. Here, the display image control unit 13 includes an image switching unit 130 , a line of sight changing unit 131 , a viewpoint line of sight changing unit 132 , and a mode switching unit 133 .

[0055] Upon receiving an instruction to switch the image, the image switching unit 130 switches the displayed image between an MR image or a virtual image (image in a virtual space). In response to an instruction to switch images, the image switching unit 130 changes the position of the virtual camera capturing the image generated by the image generating unit 14 to the center of the spherical object corresponding to the specified celestial spherical camera 2 or to any position in virtual space, thereby switching the displayed image generated by the image generating unit 14. For example, the center of the spherical object may be used as the viewpoint position in the virtual space as the any position in virtual space.

[0056] By positioning the virtual camera at the center of the spherical object, the image generator 14 can generate an MR image as shown in FIG. Furthermore, by setting the position of the virtual camera to any position in the virtual space, the image generating unit 14 can generate a virtual image such as that shown in FIG. Here, when the image switching unit 130 is notified by the virtual object control unit 12 that a spherical object has been selected, the image switching unit 130 sets the center of the spherical object as the position of the virtual camera. The image switching unit 130 outputs to the image generating unit 14 the type of image to be switched (MR image or virtual image) and the identification information of the spherical object.

[0057] When an instruction to change the line of sight direction is inputted during display of the MR video, the line of sight change unit 131 changes the line of sight to the instructed direction. When the MR video is displayed with the center of the spherical object as the viewpoint, the line of sight change unit 131 changes the direction of the line of sight in accordance with the movement of the mouse MS when a predetermined instruction, for example, drag information of the mouse MS, is input. The line of sight change unit 131 outputs the changed line of sight direction to the video generation unit 14.

[0058] When a virtual image is displayed, the viewpoint line of sight changing unit 132 receives an instruction to change the viewpoint position or line of sight direction, and changes the viewpoint position or line of sight direction to the instructed position or direction.

[0059] When a virtual space is being photographed by the virtual camera, viewpoint line of sight changing unit 132 changes the viewpoint position of the virtual camera in response to a predetermined instruction, for example, pressing a predetermined key on the keyboard KB. Also, when a virtual space is being photographed by the virtual camera, viewpoint line of sight changing unit 132 changes the direction of the line of sight of the virtual camera in accordance with the movement of the mouse MS in response to a predetermined operation instruction, for example, drag information of the mouse MS. The viewpoint / line of sight change unit 132 outputs the changed viewpoint position or line of sight direction to the video generation unit 14.

[0060] The mode switching unit 133 functions when an avatar representing the user (operator) is set as a physical object, and switches between various modes for displaying images based on the avatar. The mode switching unit 133 switches modes in response to a predetermined instruction, for example, pressing a predetermined key on the keyboard KB. The modes are stored in a memory or the like (not shown). This mode has three modes.

[0061] The first mode is a mode in which, when an MR image is displayed, the MR image is generated so that the avatar follows the movement of the avatar and is at the center of the displayed image (hereinafter referred to as the gaze mode). In this gaze mode, the line of sight changing unit 131 does not change the line of sight. For example, in the gaze mode, as shown in FIG. 7, an avatar O2, which is an alter ego of the user M2 (FIG. 1), AV2 However, MR image I V1 In the case where it is displayed in avatar AV2 When the avatar O is moved by the operation unit 121, AV2 In this mode, the avatar O is displayed at the center of the image. AV3 is also illustrated.

[0062] The second mode is a mode in which, when a virtual image is displayed, a virtual image is generated in the direction of the avatar's line of sight by following the avatar's movements (hereinafter referred to as tracking mode). For example, in the tracking mode, as shown in Fig. 8, the virtual image I VR In this case, avatar O is the alter ego of user M2 (Fig. 1). AV2 The avatar O is always positioned at the center of the bottom of the image. AV2 This mode places the virtual camera at a fixed position behind the

[0063] The third mode is a mode in which the displayed image does not follow the movement of the avatar (hereinafter referred to as "normal mode"). In this normal mode, the object behaves in the same way as if no avatar were assigned to it. The mode switching unit 133 outputs the type of the switched mode to the video generating unit 14 .

[0064] The image generation unit 14 generates and displays, as a display image, an MR image that combines a celestial sphere image with an image of the virtual object captured by placing a virtual camera at the center of the celestial sphere camera 2, or a virtual image of the virtual object captured by placing a virtual camera at an arbitrary position, based on the position information of the virtual object. The image generating unit 14 includes an MR image generating unit 140 and a virtual image generating unit 141.

[0065] The MR image generating unit 140 generates an MR image by mixing an image of the real world with an image of a virtual object captured by a virtual camera. The MR image generation unit 140 operates in response to an instruction from the image switching unit 130 to switch the displayed image to an MR image.

[0066] The MR video generation unit 140 acquires the virtual object position information and the celestial sphere image corresponding to the spherical object notified by the image switching unit 130 from the server 5 via the communication device 4. Then, the MR video generation unit 140 semi-transparently overlays the acquired celestial sphere image with an image of the virtual object captured by the virtual camera, generates a two-dimensional image in a predetermined initial viewing direction or in the viewing direction notified by the viewing direction changing unit 131, and outputs the image to the monitor MO. Note that the images can be mixed using a common method such as alpha (α) blending. For example, if the colors of the celestial sphere image and the virtual object image are RGB and the transparency of the celestial sphere image is α (0<α<1), the celestial sphere image R1G1B1 and the virtual object image R2G2B2 can be composited as follows: R = R1 × (1-α) + R2 × α, G = G1 × (1-α) + G2 × α, B = B1 × (1-α) + B2 × α. The transparency α can be set to any value, such as 0.4, 0.5, or 0.6.

[0067] The MR image generator 140 sequentially outputs two-dimensional images at a predetermined frame rate. This allows the monitor MO to display MR images with a transparent virtual object, as shown in Figure 2.

[0068] The MR image generating unit 140 normally operates in such an operation mode (normal mode), but when switched to the gaze mode by the mode switching unit 133, generates MR images corresponding to the gaze mode. That is, when the MR video generation unit 140 receives a notification of the gaze mode from the mode switching unit 133 during display of the MR video, the MR video generation unit 140 generates the same MR video as in the normal mode for virtual objects other than the avatar representing the user (operator). Then, the MR video generation unit 140 composites the user's own avatar at the center position of the video. For example, as shown in FIG. 7, when the MR video I V1 In the center of the avatar AV2 This allows the user's avatar to always be at the center of the image even if the line of sight changes, preventing the user from losing sight of their avatar.

[0069] Virtual image generator 141 operates in response to an instruction from image switcher 130 to switch the displayed image to a virtual image. The virtual image generation unit 141 acquires virtual object position information from the server 5 via the communication device 4. Then, the virtual image generation unit 141 overlays the virtual object on a predetermined background at the current viewpoint position and line of sight direction, generates a two-dimensional image, and outputs it to the monitor MO.

[0070] The virtual video generator 141 sequentially outputs two-dimensional images at a predetermined frame rate. It should be noted that virtual image generating unit 141 normally operates in such an operation mode (normal mode), but when switched to the tracking mode by mode switching unit 133, generates a virtual image corresponding to the tracking mode. That is, when the virtual image is displayed, if the tracking mode is notified from the mode switching unit 133, the virtual image generating unit 141 places the virtual camera in the virtual space corresponding to the real world at a position relatively shifted from the position of the avatar behind the user's own avatar so that the user's own avatar is at the center position of the bottom of the captured image. Then, the virtual image generating unit 141 generates an image in which the virtual object is projected onto the virtual camera. For example, as shown in FIG. 8, the virtual image I VR Avatar O at the bottom center position AV2 This allows the avatar to move freely through the virtual space.

[0071] With the configuration described above, the virtual object display device 1 can switch and display, for a user M2 who is not wearing the HMD 3, an MR image in which the image of the virtual object is mixed with the real world via the monitor MO, and a virtual image of the virtual object arranged in the virtual space viewed from an arbitrary viewpoint.

[0072] Note that the virtual object display device 1 can be operated by a program (virtual object display program) that causes a computer to function as each part of the control unit CL described above. For example, the virtual object display device 1 can also be realized by operating a virtual object display program on a PC (personal computer).

[0073] <Configuration of HMD> Next, referring to FIG. 9 (appropriately referring to FIG. 1), the configuration of the HMD 3 will be described. The HMD 3 includes a storage unit ME and a control unit CL. Note that the server 5 in FIG. 9 is the same as the server 5 in FIGS. 1 and 6, but the celestial sphere image not used in the HMD 3 is shown by a dashed line frame.

[0074] The storage unit ME stores information on virtual objects used in the HMD 3 and can be configured with a general storage medium such as a semiconductor memory. Here, the storage unit ME includes a virtual object information storage unit 30. Information used in a general HMD, for example, information for detecting the movement of fingers, etc., is omitted here.

[0075] The virtual object information storage unit 30 stores the structural information of the virtual object. The information stored in this virtual object information storage unit 30 is the same as the information stored in the virtual object information storage unit 10 described in FIG. 6.

[0076] The control unit CL performs various control operations of the HMD 3. The control unit CL includes an HMD main body unit 31 and a virtual object control unit 32.

[0077] The HMD main body 31 displays a virtual object superimposed on real space on a display device (not shown), and presents an MR image to a user wearing the HMD 3. A conventional HMD with a hand tracking function may be used as the HMD main body 31. Here, when the HMD main body 31 detects various actions such as grasping, releasing, or moving a virtual object by the user wearing the HMD 3 using the hand tracking function, the HMD main body 31 outputs the details of the action to the virtual object control unit 32 as an operation instruction. Furthermore, the HMD main body 31 displays a virtual object at a position corresponding to the position information of the virtual object stored in the server 5 on the display device. Hereinafter, the details of the HMD main body 31 will be omitted because they are the same as those of known HMDs.

[0078] The virtual object control unit 32 operates the virtual object according to an instruction input from the HMD main body unit 31. Here, the virtual object control unit 32 includes a selection unit 320 and an operation unit 321 .

[0079] The selection unit 320 receives an input from the HMD main body 31 indicating that a virtual object has been selected (held), and sets the selected virtual object as an object to be operated. The selection unit 320 stores identification information for identifying the selected virtual object in a memory or the like (not shown). The selection unit 320 outputs the identification information of the selected virtual object to the operation unit 321 . Furthermore, when a non-selection (release) of a virtual object is input from the HMD main body 31, the selection unit 320 excludes the virtual object from the objects to be operated. When the selection unit 320 is notified of the release, it deletes the identification information stored in a memory or the like (not shown).

[0080] The operation unit 321 performs operations on virtual objects in response to an operation instruction input from the HMD main body 31. When an operation instruction to move (including rotation, scale change, etc.) a virtual object is input from the HMD main body 31, the operation unit 321 updates the position information of the virtual object stored in the server 5. This allows the HMD 3 to refer to the position information of the updated virtual object stored in the server 5 and display the updated virtual object on a display device (not shown).

[0081] With the above-described configuration, the virtual object control unit 32 of the HMD 3 can allow a user who is not wearing the HMD 3 to operate the same virtual object via the server 5. The virtual object control unit 32 of the HMD 3 can operate a computer in the HMD 3 using a program (virtual object control program) that causes the computer to function as each of the above-mentioned units. This computer may be configured separately from the HMD 3 and connected to the HMD 3 by wire or the like.

[0082] <<Operation of the virtual object display system>> Next, the operation of the virtual object display system 100 according to the embodiment of the present invention will be described with reference to Figures 10 to 18 (and also Figures 1 and 6 as appropriate). Here, the operation of the virtual object display device 1 will be mainly described. This operation is performed in response to an instruction from the user of the virtual object display device 1 as an event, and therefore the order in which the events occur is not limited to this. Also, here, it is assumed that the normal mode is set as the initial state. The virtual object display device 1 executes the following operations at the frame cycle of the image to be displayed, and also executes operations in response to instructions inputted irregularly from the user as events.

[0083] In step S1, the instruction analysis unit 11 determines whether or not the input instruction is an instruction to select a virtual object, which is assigned in advance to pressing the mouse MS or the like. If the instruction is to select a virtual object (Yes in step S1), the selection unit 120 of the virtual object control unit 12 executes a selection process in step S2, and the process proceeds to step S3. If the instruction is not to select a virtual object (No in step S1), the virtual object display device 1 proceeds to step S3.

[0084] Here, the operation of the selection process in step S2 will be described with reference to FIG. In step S20, the selection unit 120 refers to the virtual object position information stored in the server 5 and detects a virtual object in the vicinity of the position where the mouse MS is located. In step S21, identification information for identifying the selected virtual object is stored in a memory or the like (not shown).

[0085] In step S22, the selection unit 120 refers to the server 5 and determines whether the selected virtual object is a spherical object. If the selected virtual object is a spherical object (Yes in step S22), in step S23, the selection unit 120 generates an image switching instruction to switch the displayed image to an MR image in which the viewpoint is shifted to the inside of the selected spherical object. Whether or not this image switching instruction is present is determined in step S5. After generating the video switching instruction in step S23, or if the selected virtual object is not a spherical object (No in step S22), the virtual object display device 1 proceeds to the operation of step S3 (FIG. 10).

[0086] If the instruction is an instruction to operate a virtual object (Yes in step S3), the operation unit 121 of the virtual object control unit 12 executes operation processing in step S4, and the process proceeds to step S5. If the instruction is not an instruction to operate a virtual object (No in step S3), the virtual object display device 1 proceeds to step S5.

[0087] Here, the operation of the operation processing in step S4 will be described with reference to FIG. In step S40, the operation unit 121 performs the instructed operation such as movement, rotation, or scale change on the selected virtual object that is placed at a position corresponding to the virtual object position information stored in the server 5.

[0088] In step S41, the operation unit 121 updates the virtual object position information stored in the server 5 with the position information of the virtual object after the operation, and the operation proceeds to step S5 (FIG. 10).

[0089] If the instruction is to switch the image (Yes in step S5), the image switching unit 130 of the display image control unit 13 executes image switching processing in step S6, and the operation proceeds to step S7. If the instruction is not to switch the image (No in step S5), the virtual object display device 1 proceeds to step S7.

[0090] Here, the operation of the video switching process in step S6 will be described with reference to FIG. In step S60, the video switching unit 130 sets the type of the displayed video to an MR video or a virtual video in response to an instruction. This instruction may be given directly from the outside using a keyboard KB or the like, or may be generated in step S23. In step S61, video switching unit 130 sets the position of the virtual camera to the center of a spherical object, which is the designated virtual object, or to any position in the virtual space, and then proceeds to step S7 (FIG. 10).

[0091] If the instruction is to change the line of sight direction (Yes in step S7), in step S8, the line of sight changing unit 131 or the viewpoint line of sight changing unit 132 of the display image control unit 13 executes line of sight change processing, and the operation proceeds to step S9. If the instruction is not to change the line of sight direction (No in step S7), the virtual object display device 1 proceeds to step S9. Note that the line of sight changing unit 131 executes this processing when the MR image is displayed, and the viewpoint line of sight changing unit 132 executes this processing when the virtual image is displayed.

[0092] Here, the operation of the line of sight change processing in step S8 will be described with reference to FIG. In step S80, the line of sight changing unit 131 or the viewpoint line of sight changing unit 132 determines whether or not the current display mode is the normal mode. If the display mode is the normal mode (Yes in step S80), in step S81, line of sight changing section 131 or viewpoint line of sight changing section 132 sets the line of sight direction of the virtual camera in accordance with the movement of the mouse MS. After the line of sight direction is set in step S81, or if the current display mode is not the normal mode (No in step S80), the virtual object display device 1 proceeds to the operation of step S9 (FIG. 10).

[0093] If the instruction is to change the viewpoint direction (Yes in step S9), in step S10, the viewpoint line of sight changing unit 132 of the display image control unit 13 executes viewpoint change processing, and the operation proceeds to step S11. If the instruction is not to change the viewpoint direction (No in step S9), the virtual object display device 1 proceeds to step S11.

[0094] Here, the operation of the viewpoint change processing in step S10 will be described with reference to FIG. In step S100, the viewpoint line of sight change unit 132 determines whether the currently displayed image is a virtual image or an MR image. When the displayed image is an MR image, the viewpoint is switched by selecting a celestial sphere object.

[0095] If the displayed image is a virtual image ("Virtual Image" in step S100), in step S101, viewpoint line of sight change unit 132 sets the viewpoint position of the virtual camera by pressing a predetermined key on keyboard KB. After the viewpoint position is set in step S101, or if the currently displayed image is an MR image ("MR image" in step S100), the virtual object display device 1 proceeds to step S11 (FIG. 10).

[0096] If the instruction is an instruction to switch modes (Yes in step S11), the mode switching unit 133 of the display image control unit 13 executes mode switching processing in step S12, and the operation proceeds to step S13. If the instruction is not an instruction to switch modes (No in step S11), the virtual object display device 1 proceeds to step S13.

[0097] Here, the operation of the mode switching process in step S12 will be described with reference to FIG. In step S120, the mode switching unit 133 determines the specified mode. Here, if the specified mode is the normal mode ("normal mode" in step S120), in step S121, the mode switching unit 133 sets the current mode to the normal mode.

[0098] Furthermore, if the designated mode is the gaze mode ("gaze mode" in step S120), mode switching unit 133 sets the current mode to the gaze mode in step S122. However, although not shown, if the currently displayed image is a virtual image, the gaze mode is not set.

[0099] Also, if the designated mode is the tracking mode ("tracking mode" in step S120), in step S123, the mode switching unit 133 sets the current mode to the tracking mode. However, although not shown, if the currently displayed image is an MR image, the tracking mode is not set. Thereafter, the virtual object display device 1 proceeds to step S13 (FIG. 10).

[0100] In step S13, the image generator 14 determines whether the currently displayed image is an MR image. If the currently displayed image is an MR image ("MR image" in step S13), the MR image generator 140 executes an MR image generation process in step S14, and the process proceeds to step S16. If the currently displayed image is a virtual image ("Virtual Image" in step S13), in step S15, the virtual image generating unit 141 executes a virtual image generating process, and the process proceeds to step S16.

[0101] Here, the MR image generation process in step S14 will be described with reference to FIG. In step S140, the MR image generation unit 140 acquires the virtual object position information from the server 5 via the communication device 4. In step S141, the MR image generator 140 acquires a celestial sphere image corresponding to the currently selected celestial sphere object from the server 5 via the communication device 4.

[0102] In step S142, the MR video generation unit 140 combines the celestial sphere video acquired in step S142 with the video of the virtual object identified by the virtual object position information acquired in step S141 to generate a display video corresponding to the currently set viewpoint position and line of sight direction. Note that when the gaze mode is set, the MR video generation unit 140 synthesizes the video so that a preset avatar representing the user (operator) is at the center of the video.

[0103] Next, the virtual video generation process in step S15 will be described with reference to FIG. In step S150, the virtual video generator 141 acquires the virtual object position information from the server 5 via the communication device 4.

[0104] In step S151, the virtual image generating unit 141 places a virtual object in a virtual space, projects it onto a virtual camera corresponding to the currently set viewpoint position and line of sight direction, and generates a display image. Note that when the tracking mode is set, the virtual image generating unit 141 synthesizes the captured image so that a preset avatar representing the user (operator) is positioned at the center of the bottom of the captured image.

[0105] Returning to FIG. 10, in step S16, the virtual object display device 1 determines whether or not an end command has been issued by an end button, switch, or the like on a menu screen (not shown). Here, if an instruction to end is not given (No in step S16), the virtual object display device 1 returns to step S1 and continues the operation. On the other hand, if an instruction to end is given (Yes in step S16), the virtual object display device 1 ends its operation. The operation of the HMD 3 is the same as that of the conventional HMD (Non-Patent Document 1) except that the virtual object position information is stored in the server 5, and therefore a description thereof will be omitted.

[0106] Through the operations described above, the virtual object display device 1 can switch between displaying, via the monitor MO, an MR image that combines images of the real world and images of virtual objects, and a virtual image in which virtual objects placed in a virtual space are viewed from any viewpoint, to a user who is not wearing the HMD 3.

[0107] In this way, by mixing images of the real world and images of virtual objects, a user who is not wearing the HMD 3 can see the MR image I shown in Figure 19. V In the present invention, the MR image I in FIG. V You will be able to see the following. In Figure 19, the original desk O R Object O, a virtual object placed on VT is superimposed on the person M, and the object O VT It is difficult to recognize the placement position. On the other hand, in FIG. 20, the object O VT Since the image is mixed with the real world image, the person M in the foreground is not completely overwritten, and the object O is visually VT The placement position of the object can be correctly recognized.

[0108] Although the virtual object display system 100 according to the embodiment of the present invention has been described above, the present invention is not limited to this embodiment. For example, as shown in FIG. 1, the virtual object display system 100 is assumed to be operated only on-site by a user wearing an HMD 3. In the example of FIG. 1, a virtual object at a remote location is displayed on-site as a spherical object O. V3 (See Figure 2) However, the user wearing the HMD 3 may be present both locally and remotely, and may place a movable virtual object at the remote location, and may move the virtual object at the remote location into the local virtual space by performing a preset operation. This makes it possible to create the effect of virtual objects from a remote location appearing to pop up in the local virtual space. [Explanation of symbols]

[0109] 100 Virtual Object Display System 1 Virtual object display device 10 Virtual object information storage unit 11 Instruction analysis section 12 Virtual object control section 120 Selection Section 121 Operation section 13 Display image control unit 130 Video switching section 131 Line of Sight Change Unit 132 Viewpoint change section 133 Mode switching section 14 Image generation unit 140 MR image generation section 141 Virtual image generation unit 2. Celestial Camera 3 HMD 30 Virtual object information storage unit 31 HMD main body 32 Virtual object control section 320 Selection Section 321 Operation section 4. Communication equipment 5 Server

Claims

1. A virtual object display device that displays virtual objects including an object object representing an object and a spherical object representing a celestial camera, the virtual object display device being connected via a network to a server that stores a celestial sphere image captured by the celestial camera of real space and position information of the virtual objects, an image generating unit that generates and displays, as a display image, an MR image that combines the celestial sphere image with an image of the virtual object captured by a virtual camera placed at the center of the celestial sphere camera, or a virtual image that captures the virtual object by a virtual camera placed at an arbitrary position, based on the position information of the virtual object; an image switching unit that switches the displayed image between the MR image and the virtual image based on an operation instruction; A virtual object display device comprising:

2. The server stores a plurality of celestial sphere images captured by a plurality of celestial sphere cameras, 2. The virtual object display device according to claim 1, wherein the image switching unit switches the displayed image to one of the MR images corresponding to the celestial sphere images or the virtual image based on the operation instruction.

3. a line-of-sight changing unit that changes the line-of-sight direction to a specified direction when the MR image is displayed; a viewpoint / line-of-sight changing unit that changes a viewpoint position or line-of-sight direction to a designated position or direction when the virtual image is displayed; The virtual object display device according to claim 1 , further comprising:

4. The physical objects include an avatar that represents the operator, a mode in which the MR image is generated so that the avatar is at the center of the displayed image while following the movement of the avatar when the MR image is displayed; a mode in which, when the virtual image is displayed, a virtual image is generated in the line of sight of the avatar by following the movement of the avatar; a mode in which the displayed image does not follow the movement of the avatar; a mode switching unit for switching between The virtual object display device according to claim 1 , wherein the image generating unit generates the display image in a mode selected by the mode switching unit.

5. A virtual object display program for causing a computer to function as the virtual object display device according to any one of claims 1 to 4.

6. A virtual object display system including a physical object representing a physical object and a spherical object representing a celestial sphere camera, a celestial sphere camera disposed in a real space, a server that stores a celestial sphere image of the real space captured by the celestial sphere camera and positional information of the virtual object, and a virtual object display device connected to a network, The virtual object display device includes: an image generating unit that generates and displays, as a display image, an MR image that combines the celestial sphere image with an image of the virtual object captured by a virtual camera placed at the center of the celestial sphere camera, or a virtual image that captures the virtual object by a virtual camera placed at an arbitrary position, based on the position information of the virtual object; an image switching unit that switches the displayed image between the MR image and the virtual image based on an operation instruction; A virtual object display system comprising:

7. a head-mounted display connected to the network that displays the virtual object as an MR image superimposed on a real space; 7. The virtual object display system according to claim 6, wherein the head-mounted display has a hand tracking function and updates the position information of the virtual object stored in the server in accordance with the operator's movements.

Images