Virtual object display device, program for same, and virtual object display system

Abstract

A virtual object display device (1) comprises a video generation unit (14) that: uses, as a virtual object, a physical body object representing a physical body and a spherical object representing an omnidirectional camera; is connected via a network to a server (5) that stores an omnidirectional video obtained by capturing the real space by the omnidirectional camera and the position information of the virtual object; and on the basis of the position information of the virtual object, generates and displays as a display video an MR video obtained by mixing the omnidirectional video with a video obtained by capturing the virtual object by a virtual camera disposed at the center of the omnidirectional camera, or a virtual video obtained by capturing the virtual object by the virtual camera disposed at an arbitrary position, and a video switching unit (130) that switches the display video between the MR video and the virtual video on the basis of an operation instruction.

Description

Virtual object display device and program thereof, and virtual object display system

[0001] The present invention relates to a virtual object display device, a program therefor, and a virtual object display system.

[0002] In recent years, mixed reality (MVR) technology has advanced, superimposing real-world images captured in real space onto virtual objects, making it possible to display images that seamlessly blend images of the real world and virtual objects (hereinafter referred to as MR images). With this MVR technology, a user wearing a head-mounted display (hereinafter referred to as HMD) can recognize virtual objects through images displayed on the HMD.

[0003] Furthermore, a technology has been disclosed in which an HMD is used in a web conference or the like, and one of two users at different locations wears the HMD and shares an MR image displayed on the HMD (see Patent Document 1). In the method of Patent Document 1, only the user wearing the HMD can change the viewpoint, and users not wearing the HMD (web conference attendees) cannot freely manipulate the viewpoint. Therefore, the present inventor has disclosed a method that allows users not wearing an HMD to view and manipulate a virtual object superimposed on an image of the real world on a monitor (see Non-Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2017-033575

[0005] Akiyuki Ihara, "XR Pioneering RX (Research Transformation)," Information Processing Society of Japan, Journal "Information Processing: Digital Practice Corner," Vol. 64, No. 8, August 2023

[0006] The technique described in Non-Patent Document 1 is advantageous in that it allows even a user not wearing an HMD to operate a virtual object in an MR image. However, when the user views a virtual object superimposed on an image of the real world, the virtual object is always displayed in front, making it difficult to recognize the position and size of the virtual object, and there is room for improvement. For example, it is difficult for a user not wearing an HMD to recognize the size of the virtual object, how far away it is, and whether the virtual object is in front of or behind the user wearing the HMD.

[0007] Therefore, an object of the present invention is to provide a virtual object display device and program thereof, as well as a virtual object display system, that allows a user not wearing an HMD to easily recognize the position and size of a virtual object.

[0008] In order to solve the above-mentioned problems, the virtual object display device of the present invention uses an object object representing an object and a spherical object representing a celestial camera as virtual objects, is connected via a network to a server that stores celestial spherical images of real space captured by the celestial camera and positional information of the virtual objects, and displays the virtual objects, and is configured to include an image generation unit and an image switching unit.

[0009] In this configuration, the virtual object display device uses an image generation unit to generate and display, as a display image, an MR image that combines a celestial sphere image with an image of the virtual object captured by a virtual camera placed at the center of the celestial sphere camera, or a virtual image that captures the virtual object with a virtual camera placed at an arbitrary position, based on the position information of the virtual object. By combining the celestial sphere image with the image of the virtual object as an MR image, the user can see through the virtual object even if it is in the foreground. Furthermore, by displaying a virtual image captured by a virtual camera placed at an arbitrary position, the user can check the position and size of the virtual object.

[0010] The virtual object display device then switches the displayed image between the MR image and the virtual image based on the operation instruction by the image switching unit. This allows the virtual object display device to visualize and present the position and size of the virtual object to the user. The virtual object display device can be operated by a virtual object display program that causes a computer to function as each of the above-mentioned units.

[0011] Furthermore, in order to solve the above-mentioned problems, the virtual object display system of the present invention is a virtual object display system in which virtual objects are an object object representing an object and a spherical object representing a celestial camera, a celestial camera placed in real space, a server that stores celestial images of the real space captured by the celestial camera and positional information of the virtual objects, and a virtual object display device, all connected to a network, and the virtual object display device is configured to include an image generation unit and an image switching unit.

[0012] In this configuration, the virtual object display system has a server that stores celestial sphere images captured by a celestial camera and positional information about the virtual objects. The virtual object display device, connected to the server via a network, generates and displays, as a display image, an MR image that combines the celestial sphere image with an image captured by a virtual camera positioned at the center of the celestial sphere camera, or a virtual image captured by a virtual camera positioned at an arbitrary position, based on the positional information about the virtual objects. The virtual object display device then switches the display image between the MR image and the virtual image based on an operation instruction, using an image switching unit. This allows a user of the virtual object display device to confirm the position and size of the virtual object.

[0013] According to the present invention, it is possible to present a virtual object superimposed on an image of the real world to a user who is not wearing an HMD in a manner that makes it easy to recognize the position and size of the object.

[0014] 1 is a configuration diagram showing the overall configuration of a virtual object display system according to an embodiment of the present invention. FIG. 1 is a diagram (part 1) showing an example of an MR image viewed on a monitor by a user not wearing an HMD. FIG. 2 is a diagram (part 2) showing an example of an MR image viewed on a monitor by a user not wearing an HMD. FIG. 3 is a diagram showing an example of a virtual image (virtual space image) viewed on a monitor by a user not wearing an HMD. FIG. 4 is a diagram showing an example of an MR image viewed through the HMD by a user wearing an HMD. FIG. 5 is a block configuration diagram showing the configuration of the virtual object display device of FIG. 1. FIG. 6 is an explanatory diagram for explaining an image in a gaze mode. FIG. 7 is an explanatory diagram for explaining an image in a tracking mode. FIG. 8 is a block configuration diagram showing the configuration of the HMD of FIG. 1. FIG. 9 is a flowchart showing the operation of a virtual object display device according to an embodiment of the present invention. FIG. 10 is a flowchart showing the operation of a selection process of FIG. 10. FIG. 11 is a flowchart showing the operation of an operation process of FIG. 10. FIG. 11 is a flowchart showing the operation of an image switching process of FIG. 10. FIG. 12 is a flowchart showing the operation of a line of sight change process of FIG. 10. FIG. 12 is a flowchart showing the operation of a viewpoint change process of FIG. 10. FIG. 13 is a flowchart showing the operation of a mode switching process of FIG. 10. FIG. 14 is a flowchart showing the operation of an MR image generation process of FIG. 10. FIG. 15 is a flowchart showing the operation of a virtual image generation process of FIG. FIG. 20 is a diagram showing an example of an MR image according to the present invention corresponding to FIG. 19.

[0015] A virtual object display system according to an embodiment of the present invention will be described below with reference to the drawings.

[0016] <<Overall Configuration of Virtual Object Display System>> First, the overall configuration of a virtual object display system 100 according to an embodiment of the present invention will be described with reference to Fig. 1. The virtual object display system 100 displays an MR image in which a virtual object is drawn superimposed on the real world to a user M wearing an HMD 3. １ and presenting the image to a user M who is not wearing the HMD 3. ２ The MR image, which depicts a virtual object overlaid on the real-world image, is then displayed to the user M. １ The virtual object display system 100 also presents the virtual object from a different viewpoint than the user M in a remote location who is not wearing the HMD 3. ３ Also, user M１ The MR image is presented from a different viewpoint than the actual location. Here, the "local location" shown in Fig. 1 is a space (MR space) in which real space and virtual objects are associated. The "remote location" is a real space that is distant from the actual location and is not associated with virtual objects.

[0017] The virtual object display system 100 uses an object object representing an object and a spherical object representing a celestial camera 2 as virtual objects, and is configured by connecting the celestial camera 2 placed in real space, a server 5 that stores celestial images of the real space captured by the celestial camera 2 and position information of the virtual objects, and a virtual object display device 1 to a network via a communication device 4. The spherical object indicates the position of the spherical camera 2 by its center position, and is a virtual object on which the spherical image captured by the spherical camera 2 is rendered. Note that the real space contains a real object (here, a desk O Ｒ ) is assumed to exist. The virtual object display device 1 and the HMD 3 are assumed to store information about the virtual object (data such as three-dimensional structure and texture) in advance.

[0018] The virtual object display device 1 includes a celestial camera 2 (2 １ , 2 ２ The virtual object display device 1 displays, on the monitor MO, an image of a specified celestial sphere image from a specified direction among images (celestial sphere images) captured by a virtual camera as an MR image. At this time, the virtual object display device 1 generates the MR image by blending the celestial sphere image with an image of a virtual object captured by a virtual camera. The virtual object display device 1 also displays, on the monitor MO, a virtual image (image of the virtual space) virtually captured from an arbitrary viewpoint of a virtual space associated with real space.

[0019] The virtual object display device 1 has a function for moving a virtual object, and stores position information when the virtual object is moved in a server 5. The virtual object display device 1 is connected to a monitor (display device) MO and instruction devices (mouse MS, keyboard KB, etc.) that allow a user (operator) to perform various operations. The virtual object display device 1 obtains a desired celestial sphere image from the server 5 located on a network via a communication device 4.

[0020] Here, an example will be described in which two virtual object display devices 1 are installed, one at a local location and one at a remote location. However, the virtual object display device 1 may be installed in one of the local or remote locations, one or more at each of the local or remote locations, or one or more at each of a plurality of remote locations.

[0021] The celestial camera 2 is a celestial camera that captures 360-degree celestial images in all directions, up, down, left, and right. The celestial camera 2 sequentially transmits the captured celestial images to the server 5 via the communication device 4. The celestial camera 2 may transmit the celestial images to the server 5 via the communication device 4 via a personal computer (PC: not shown) equipped with a communication control unit. Alternatively, the celestial camera 2 may be connected to the virtual object display device 1 and transmit the celestial images to the server 5 via the virtual object display device 1 and the communication device 4. Here, an example will be described in which two celestial cameras 2 are installed locally and one is installed remotely. However, it is sufficient to install at least one celestial camera 2 in a space where real space and virtual objects are associated.

[0022] The HMD 3 is １ The user M １ The HMD 3 displays an MR image in which a virtual object is drawn and superimposed on the real world according to the position and direction of the user's head. １ The HMD 3 has a hand tracking function for moving a virtual object in an MR image. The HMD 3 stores position information of the virtual object when it is moved in the server 5.

[0023] The HMD 3 may be an optical see-through type device that allows the real world to be viewed directly through a semi-transparent display, or a video see-through type device that captures images of the real world and displays the captured images on a display. Although the following description is given assuming that one user is wearing the HMD 3, multiple users may wear the HMD 3.

[0024] The communication device 4 is a communication device that transmits and receives data between the virtual object display device 1, the celestial camera 2, and the HMD 3 and the server 5. For example, the communication device 4 may be a Wi-Fi router that transmits and receives data wirelessly between the virtual object display device 1, the celestial camera 2, and the HMD 3.

[0025] The server 5 stores the celestial sphere images captured by the celestial camera 2 and the position information of the virtual objects placed in the virtual space. This server 5 can be configured with a general storage device such as a hard disk. The server 5 can be installed in any location connected to a network (for example, on the cloud).

[0026] The server 5 stores the celestial sphere images sent from the celestial camera 2 for each celestial camera 2, and transmits the requested celestial sphere images frame by frame to the virtual object display device 1. The server 5 also stores position information of the virtual object (virtual object position information) in advance, and transmits the requested information to the virtual object display device 1 and the HMD 3. The position information of the virtual object includes position coordinates (three variables), angle (two variables), scale (one variable), etc. in the three-dimensional virtual space.

[0027] Each virtual object is previously assigned individual identification information and information indicating the type of object, such as a physical object or a spherical object. Here, the physical object includes an avatar. An avatar is an alter ego of the user (operator) of the virtual object display device 1 placed in the virtual space.

[0028] When the position of a virtual object is changed by an operation such as moving the virtual object performed by the virtual object display device 1 or the HMD 3, the server 5 receives update information from the virtual object display device 1 or the HMD 3 and updates the stored position information. Note that, although the server 5 stores the celestial sphere image and the position information here, the celestial sphere image and the position information may be stored in different servers.

[0029] With the above configuration, the virtual object display system 100 can display the virtual object to the user M who is not wearing the HMD 3. ２ , M ３Through the monitor MO, it is possible to switch between presenting an MR image in which a virtual object is semi-transparently synthesized in the real world, and a virtual image in which a virtual object placed in a virtual space is viewed from any viewpoint.

[0030] In this way, the virtual object display system 100 can prevent a virtual object in an MR image from always being displayed in front of other objects. This makes it easier for the user to visually recognize the position of the virtual object, such as its front-to-back relationship. Furthermore, the virtual object display system 100 can display a virtual object placed in a virtual space from any viewpoint. This makes it easier for the user to grasp the position and size of the virtual object.

[0031] Before describing the configuration and operation of the virtual object display device according to the embodiment of the present invention, an example of an image displayed on the monitor MO and the HMD 3 will be described. ２ , M ３ 2 and other drawings that illustrate examples of images displayed on the monitor MO and HMD 3, objects (including virtual objects) represented by dotted lines are drawn semi-transparently, and objects (including virtual objects) represented by solid lines are drawn opaquely.

[0032] (Monitor Image) First, referring to FIG. 2, the image of the celestial camera 2 displayed on the monitor MO is ２ MR image I generated based on the celestial spherical image Ｖ２ That is, the MR image I Ｖ２ is Celestial Camera 2 ２ The image is generated with the center of the celestial sphere as the viewpoint of the virtual camera. ２ It does not have to be at the center, but may be at any position near the center.

[0033] The virtual object display device 1 displays an MR image I that combines an image of the real world with an image of a virtual object captured by a virtual camera. Ｖ２ The image is displayed on the monitor (display device) MO. Ｖ２ Among them, user M wearing HMD3１ , desk O Ｒ and Celestial Camera 2 １ are real-world objects. All other objects are virtual objects. Here, an example of a virtual object is object O ＶＴ and a spherical object O Ｖ１ , O Ｖ３ and,

[0034] Object O ＶＴ is a virtual object to be observed and operated, which is a three-dimensional structure generated by, for example, 3D (three-dimensional) CAD. Ｖ２ Since the image of the real world and the image of the virtual object are mixed, ＶＴ is the desk O, which is the real object on the back side. Ｒ can be seen through it.

[0035] Sphere object O Ｖ１ is Celestial Camera 2 １ The spherical object O is a virtual object in the shape of a sphere having a predetermined size (for example, a diameter of 50 cm) that represents the object O. Ｖ３ is a remote celestial camera 2 ３ (see FIG. 1) is a spherical virtual object representing the user M １ The position and size of the spherical object O can be changed. Ｖ１ , O Ｖ３ The texture is arbitrary, but for example, it can be a texture onto which a celestial sphere image is mapped.

[0036] Sphere object O Ｖ１ By default, the center is the celestial sphere camera 2 in the real world. １ The spherical object O is set at the center of the virtual object O and is assumed not to move. Ｖ３ is a remote celestial camera 2 ３ The spherical object O is a virtual object virtually placed in the real world and is a movable virtual object. Ｖ３ The initial position of the user M is set to the vicinity of the HMD 3. １ is a spherical object O Ｖ３This MR image I can prevent the patient from losing sight of the MRI image. Ｖ２ Sphere object O Ｖ１ , O Ｖ３ By selecting the corresponding celestial camera 2 １ , 2 ３ The perspective will switch to.

[0037] Next, referring to FIG. 3, the celestial camera 2 displayed on the monitor MO １ MR image I generated based on the celestial spherical image Ｖ１ That is, the MR image I Ｖ１ is Celestial Camera 2 １ The image is generated with the center of the image as the viewpoint of the virtual camera. Ｖ１ is the same as the MR image I explained in Figure 2, except for the viewpoint position. Ｖ２ The content of the spherical object O is the same as that of the spherical object O. Ｖ２ By default, the center is the celestial sphere camera 2 in the real world. ２ It is a virtual object that is set at the center of the celestial sphere and is assumed not to move. ３ Although the MR image generated from the celestial spherical image is not shown, in the configuration example of FIG. 1, only the spherical object is superimposed on the image of the real world.

[0038] Next, referring to FIG. 4, a virtual image I is displayed on the monitor MO, in which a virtual object placed in a virtual space is viewed from an arbitrary viewpoint. ＶＲ That is, the virtual image I ＶＲ is an image of the virtual space captured by a virtual camera at an arbitrary viewpoint away from the viewpoint of the celestial camera 2 (spherical object).

[0039] As shown in FIG. ＶＲ In this case, a virtual object ( ＶＴ , spherical object O Ｖ１ , O Ｖ２ , O Ｖ３ ) will be displayed. ＶＲis an image in which a virtual object is composited against a background of a predetermined color. In this case, the virtual object can be drawn by overwriting the background with the texture of the virtual object itself.

[0040] (Image of HMD) Next, referring to FIG. 5, the MR image I displayed on the HMD 3 ＨＭＤ The HMD 3 displays an MR image I in which the real world or an image of the real world is superimposed on a virtual object. ＨＭＤ Display MR image I ＨＭＤ Among them, desk O Ｒ and Celestial Camera 2 １ , 2 ２ is a real-world object. ＶＴ , and a spherical object O Ｖ１ , O Ｖ２ , O Ｖ３ are virtual objects. Each virtual object is the same as that described with reference to FIGS. 2 to 4, and therefore a description thereof will be omitted.

[0041] In addition, the user M wearing the HMD 3 １ The viewpoint can be changed by moving to any position, making it easy to grasp the position of the virtual object. ＨＭＤ is an image in which the virtual object is overwritten with its original texture.

[0042] <Configuration of Virtual Object Display Device> Next, with reference to Fig. 6 (and also with reference to Fig. 1 as appropriate), a description will be given of the configuration of the virtual object display device 1. The virtual object display device 1 includes a storage unit ME and a control unit CL.

[0043] The storage unit ME stores information about virtual objects used in the virtual object display system 100, and can be configured with a general storage medium such as a semiconductor memory. Here, the storage unit ME includes a virtual object information storage unit 10.

[0044] The virtual object information storage unit 10 stores structural information of virtual objects. Here, the structural information is information that specifies the structure of the virtual object, such as information about a three-dimensional structure created by 3D CAD. If the virtual object moves, the structural information also includes animation information that indicates the movement.

[0045] The control unit CL performs various control operations of the virtual object display device 1. The control unit CL includes an instruction analysis unit 11, a virtual object control unit 12, a display image control unit 13, and an image generation unit .

[0046] The instruction analysis unit 11 analyzes instructions given by the user. Here, the instruction analysis unit 11 analyzes whether an instruction input from an externally connected instruction device such as a mouse MS is an instruction for a virtual object or an instruction for a displayed image.

[0047] An instruction for a virtual object is an instruction to control the virtual object. In this case, the instruction for a virtual object is information such as button operation (press / release) information of the mouse MC and key operation (press / release) information of the keyboard KB that are assigned in advance to operations on the virtual object, such as selection and operation of the virtual object.

[0048] The instructions for the displayed image are instructions for controlling the displayed image. Here, the instructions for the displayed image include mouse MC button operation (press / release) information, drag information, keyboard KB key operation (press / release) information, etc., which are assigned in advance to operations for the displayed image, such as switching between MR image and virtual image, changing the line of sight, and switching display modes. The instruction analysis unit 11 outputs instructions for the virtual object to the virtual object control unit 12, and outputs instructions for the displayed image to the displayed image control unit 13.

[0049] The virtual object control unit 12 performs operations on virtual objects in accordance with instructions input from the instruction analysis unit 11. Here, the virtual object control unit 12 includes a selection unit 120 and an operation unit 121.

[0050] The selection unit 120 is configured to operate a specified virtual object when an instruction to select the virtual object is input. For example, when the left button of the mouse MS is pressed as an instruction to select the virtual object, the selection unit 120 selects the virtual object corresponding to the position of the mouse MS on the monitor MO. The virtual object corresponding to the position of the mouse MS can be detected as the virtual object located closest to the position of the mouse MS by associating the position information of the virtual object within the angle of view of the display image specified by the display image control unit 13.

[0051] It is also possible to associate the virtual object with the virtual object position information of the server 5 and store information indicating the right to monopolize operation, and the selection unit 120 may perform processing to acquire or release that right when selecting a virtual object.

[0052] The selection unit 120 stores identification information for identifying the selected virtual object in a memory or the like (not shown). If the selected virtual object is a spherical object, the selection unit 120 notifies the display image control unit 13 that the spherical object has been selected, along with the identification information.

[0053] The operation unit 121 receives an instruction to operate a virtual object and performs an operation corresponding to the instruction on the selected virtual object as the operation target. For example, the operation unit 121 performs operations such as moving the virtual object in the direction of the arrow by pressing an arrow key, rotating the virtual object by pressing an arrow key while holding down the shift key, and enlarging or reducing the virtual object by pressing the " / " key or the "*" key. The operation unit 121 transmits position information (position coordinates, angle, scale) of the virtual object that changes in accordance with the operation of the virtual object to the server 5 and updates it.

[0054] The display image control unit 13 performs operations on the display image based on instructions input from the instruction analysis unit 11. Here, the display image control unit 13 includes an image switching unit 130, a line of sight changing unit 131, a viewpoint line of sight changing unit 132, and a mode switching unit 133.

[0055] Upon receiving an instruction to switch images, the image switching unit 130 switches the displayed image between an MR image or a virtual image (image in virtual space). In response to the instruction to switch images, the image switching unit 130 changes the position of the virtual camera that captures the image generated by the image generation unit 14 to the center of the spherical object corresponding to the specified celestial camera 2 or to any position in virtual space, thereby switching the displayed image generated by the image generation unit 14. For example, the center of the spherical object may be used as the viewpoint position in virtual space as the any position in virtual space.

[0056] By positioning the virtual camera at the center of the spherical object, the image generation unit 14 can generate an MR image such as that shown in FIG. 2. Furthermore, by positioning the virtual camera at any position in the virtual space, the image generation unit 14 can generate a virtual image such as that shown in FIG. 4. Here, when the image switching unit 130 is notified by the virtual object control unit 12 that a spherical object has been selected, the image switching unit 130 sets the center of the spherical object as the position of the virtual camera. The image switching unit 130 outputs the type of image to be switched (MR image or virtual image) and identification information of the spherical object to the image generation unit 14.

[0057] The line-of-sight changing unit 131 changes the line of sight to the specified direction when an instruction to change the line of sight direction is input when an MR image is displayed. When an MR image is displayed with the center of a spherical object as the viewpoint, the line-of-sight changing unit 131 changes the line of sight direction in accordance with the movement of the mouse MS when a predetermined instruction, for example, drag information of the mouse MS, is input. The line-of-sight changing unit 131 outputs the changed line of sight direction to the image generating unit 14.

[0058] When a virtual image is displayed, the viewpoint line of sight changing unit 132 receives an instruction to change the viewpoint position or line of sight direction, and changes the viewpoint position or line of sight direction to the instructed position or direction.

[0059] The viewpoint line of sight changing unit 132 changes the viewpoint position of the virtual camera in response to a predetermined instruction, for example, pressing a predetermined key on the keyboard KB, while the virtual camera is capturing an image of the virtual space. Furthermore, the viewpoint line of sight changing unit 132 changes the line of sight direction of the virtual camera in accordance with the movement of the mouse MS in response to a predetermined operation instruction, for example, drag information of the mouse MS, while the virtual camera is capturing an image of the virtual space. The viewpoint line of sight changing unit 132 outputs the changed viewpoint position or line of sight direction to the image generation unit 14.

[0060] The mode switching unit 133 functions when an avatar representing the user (operator) is set for the physical object, and switches between various modes for displaying images based on the avatar. The mode switching unit 133 switches modes in response to a predetermined instruction, for example, pressing a predetermined key on the keyboard KB. The modes are stored in a memory or the like (not shown). There are three modes:

[0061] The first mode is a mode (hereinafter referred to as "gazing mode") in which, when an MR image is displayed, an MR image is generated so that the avatar follows the movement of the avatar and is positioned at the center of the displayed image. In this gaze mode, the line of sight changing unit 131 does not change the line of sight. For example, in the gaze mode, as shown in FIG. 7, ２ (Figure 1) Avatar O as an alter ego ＡＶ２ However, MRI image I Ｖ１ In the case where it is displayed in avatar O ＡＶ２ When the avatar O is moved by the operation unit 121, ＡＶ２ In this mode, the user M is displayed at the center of the image. ３ (Figure 1) Avatar O as an alter ego ＡＶ３ is also illustrated.

[0062] The second mode is a mode (hereinafter referred to as tracking mode) in which, when a virtual image is displayed, a virtual image is generated in the line of sight of the avatar by following the movement of the avatar. For example, in the tracking mode, as shown in FIG. ＶＲ In this case, user M ２(Figure 1) Avatar O as an alter ego ＡＶ２ Avatar O is always positioned at the center of the bottom of the image. ＡＶ２ This mode places the virtual camera at a fixed position behind the

[0063] The third mode is a mode in which the displayed image does not follow the movement of the avatar (hereinafter referred to as normal mode). In this normal mode, the same operation as when no avatar is set for the physical object is performed. The mode switching unit 133 outputs the type of the switched mode to the image generation unit 14.

[0064] The image generation unit 14 generates and displays, as a display image, an MR image that combines a celestial sphere image with an image of the virtual object captured by a virtual camera placed at the center of the celestial sphere camera 2, or a virtual image of the virtual object captured by a virtual camera placed at an arbitrary position, based on the position information of the virtual object. The image generation unit 14 includes an MR image generation unit 140 and a virtual image generation unit 141.

[0065] The MR image generating unit 140 generates an MR image by mixing an image of the real world with an image of a virtual object captured by a virtual camera. The MR image generating unit 140 operates in response to an instruction from the image switching unit 130 to switch the displayed image to an MR image.

[0066] The MR image generation unit 140 acquires the virtual object position information and the celestial sphere image corresponding to the spherical object notified by the image switching unit 130 from the server 5 via the communication device 4. The MR image generation unit 140 then semi-transparently overlays the acquired celestial sphere image with an image of the virtual object captured by the virtual camera, generates a two-dimensional image in a predetermined initial viewing direction or in the viewing direction notified by the viewing direction change unit 131, and outputs the image to the monitor MO. Note that a common method such as alpha (α) blending can be used to blend the images. For example, if the colors of the celestial sphere image and the image of the virtual object are RGB and the transparency of the celestial sphere image is α (0<α<1), the celestial sphere image R １ G １ B １ and virtual object image R ２ G ２ B ２ and R = R１ ×(1-α)+R ２ ×α, G = G １ ×(1-α)+G ２ ×α, B=B １ × (1-α) + B ２ The transparency α may be set to any value such as 0.4, 0.5, 0.6, or the like.

[0067] The MR image generator 140 sequentially outputs two-dimensional images at a predetermined frame rate, thereby enabling the monitor MO to display an MR image with a virtual object visible through it, as shown in FIG.

[0068] The MR image generating unit 140 normally operates in this operation mode (normal mode), but when the mode switching unit 133 switches to the gaze mode, it generates an MR image corresponding to the gaze mode. That is, when the MR image generating unit 140 is notified of the gaze mode by the mode switching unit 133 while an MR image is being displayed, it generates MR images similar to those in the normal mode for virtual objects other than the avatar representing the user (operator). Then, the MR image generating unit 140 composites its own avatar at the center of the image. For example, as shown in FIG. 7, when an MR image I Ｖ１ In the center of the ＡＶ２ This allows the user's avatar to always be at the center of the image even if the line of sight changes, preventing the user from losing sight of their avatar.

[0069] The virtual image generation unit 141 operates in response to an instruction to switch the displayed image to a virtual image from the image switching unit 130. The virtual image generation unit 141 acquires virtual object position information from the server 5 via the communication device 4. Then, the virtual image generation unit 141 overlays a virtual object on a predetermined background at the current viewpoint position and line of sight direction, generates a two-dimensional image, and outputs it to the monitor MO.

[0070] The virtual image generating unit 141 sequentially outputs two-dimensional images at a predetermined frame rate. Note that the virtual image generating unit 141 normally operates in such an operation mode (normal mode), but when the mode switching unit 133 switches to the tracking mode, it generates a virtual image corresponding to the tracking mode. That is, when the tracking mode is notified from the mode switching unit 133 during display of a virtual image, the virtual image generating unit 141 places a virtual camera in a virtual space corresponding to the real world, behind the user's own avatar, at a position relatively shifted from the position of the avatar so that the user's own avatar is at the center position at the bottom of the captured image. Then, the virtual image generating unit 141 generates an image in which a virtual object is projected onto the virtual camera. For example, as shown in FIG. 8, a virtual image I ＶＲ Avatar O is placed at the bottom center of ＡＶ２ This allows the avatar to move freely through the virtual space.

[0071] With the above-described configuration, the virtual object display device 1 can display the virtual object to a user M who is not wearing the HMD 3. ２ The monitor MO can be used to switch between displaying an MR image in which images of virtual objects are mixed into the real world, and a virtual image in which virtual objects placed in a virtual space are viewed from any viewpoint.

[0072] The virtual object display device 1 can be operated by a program (virtual object display program) that causes a computer to function as each part of the control unit CL. For example, the virtual object display device 1 can be realized by running a virtual object display program on a PC (personal computer).

[0073] <Configuration of HMD> Next, the configuration of the HMD 3 will be described with reference to Fig. 9 (and Fig. 1 as appropriate). The HMD 3 includes a memory unit ME and a control unit CL. Note that the server 5 in Fig. 9 is the same as the server 5 in Figs. 1 and 6, but celestial spherical images that are not used by the HMD 3 are indicated by dashed lines.

[0074] The storage unit ME stores information about virtual objects used in the HMD 3, and can be configured with a general storage medium such as a semiconductor memory. Here, the storage unit ME includes a virtual object information storage unit 30. Note that a description of information used in a general HMD, such as information for detecting finger movements, will be omitted here.

[0075] The virtual object information storage unit 30 stores structural information of virtual objects. The information stored in the virtual object information storage unit 30 is the same as the information stored in the virtual object information storage unit 10 described with reference to FIG.

[0076] The control unit CL performs various control operations of the HMD 3. The control unit CL includes an HMD main body 31 and a virtual object control unit 32.

[0077] The HMD main body 31 superimposes a virtual object on real space and displays it on a display device (not shown), thereby presenting an MR image to a user wearing the HMD 3. A conventional HMD with a hand tracking function may be used as the HMD main body 31. Here, when the HMD main body 31 detects various actions, such as grasping, releasing, or moving a virtual object, by the user wearing the HMD 3 using the hand tracking function, the HMD main body 31 outputs the details of the action as an operation instruction to the virtual object control unit 32. Furthermore, the HMD main body 31 displays a virtual object at a corresponding position on the display device based on position information of the virtual object stored in the server 5. The details of the HMD main body 31 are similar to those of known HMDs, and therefore will not be described in detail below.

[0078] The virtual object control unit 32 operates the virtual object in response to an instruction input from the HMD main body unit 31. Here, the virtual object control unit 32 includes a selection unit 320 and an operation unit 321.

[0079] The selection unit 320, upon receiving an input from the HMD main body 31 indicating that a virtual object has been selected (grasped), sets the selected virtual object as an operation target. The selection unit 320 stores identification information for identifying the selected virtual object in a memory or the like (not shown). The selection unit 320 outputs the identification information of the selected virtual object to the operation unit 321. Furthermore, upon receiving an input from the HMD main body 31 indicating that the virtual object has been deselected (released), the selection unit 320 excludes the virtual object from the operation targets. When notified of the release, the selection unit 320 deletes the identification information stored in a memory or the like (not shown).

[0080] The operation unit 321 performs operations on virtual objects in response to operation instructions input from the HMD main body 31. When an operation instruction to move (including rotation, scale change, etc.) a virtual object is input from the HMD main body 31, the operation unit 321 updates the position information of the virtual object stored in the server 5. This allows the HMD 3 to refer to the updated position information of the virtual object stored in the server 5 and display the updated virtual object on a display device (not shown).

[0081] With the configuration described above, the virtual object control unit 32 of the HMD 3 can allow a user who is not wearing the HMD 3 to operate the same virtual object via the server 5. The virtual object control unit 32 of the HMD 3 can operate a computer in the HMD 3 using a program (virtual object control program) that causes the computer to function as each of the above-mentioned units. This computer may also be configured separately from the HMD 3 and connected to the HMD 3 by wire or the like.

[0082] <<Operation of Virtual Object Display System>> Next, with reference to FIGS. 10 to 18 (and also with reference to FIGS. 1 and 6 as appropriate), the operation of the virtual object display system 100 according to an embodiment of the present invention will be described. Here, the operation of the virtual object display device 1 will be mainly described. This operation is performed in response to an instruction from the user of the virtual object display device 1 as an event, and therefore the order in which events occur is not limited to the above. Also, it is assumed here that the normal mode is set as the initial state. The virtual object display device 1 performs the following operations at the frame cycle of the video to be displayed, and also performs operations in response to instructions inputted irregularly from the user as events.

[0083] In step S1, the instruction analysis unit 11 determines whether the input instruction is an instruction to select a virtual object that has been assigned in advance to pressing the mouse MS, etc. If the instruction is an instruction to select a virtual object (Yes in step S1), in step S2 the selection unit 120 of the virtual object control unit 12 executes a selection process and proceeds to step S3. If the instruction is not an instruction to select a virtual object (No in step S1), the virtual object display device 1 proceeds to step S3.

[0084] 11, the operation of the selection process in step S2 will be described. In step S20, the selection unit 120 detects a virtual object in the vicinity of the position of the mouse MS by referring to the virtual object position information stored in the server 5. In step S21, identification information for identifying the selected virtual object is stored in a memory or the like (not shown).

[0085] In step S22, the selection unit 120 refers to the server 5 and determines whether the selected virtual object is a spherical object. If the selected virtual object is a spherical object (Yes in step S22), in step S23, the selection unit 120 generates an image switching instruction to switch the displayed image to an MR image in which the viewpoint is shifted to the inside of the selected spherical object. In step S5, it is determined whether this image switching instruction is present. After generating the image switching instruction in step S23, or if the selected virtual object is not a spherical object (No in step S22), the virtual object display device 1 proceeds to operation in step S3 ( FIG. 10 ).

[0086] If the instruction is an instruction to operate a virtual object (Yes in step S3), the operation unit 121 of the virtual object control unit 12 executes operation processing in step S4, and the process proceeds to step S5. If the instruction is not an instruction to operate a virtual object (No in step S3), the virtual object display device 1 proceeds to step S5.

[0087] The operation of the operation processing in step S4 will now be described with reference to Fig. 12. In step S40, the operation unit 121 executes the instructed operation, such as movement, rotation, or scale change, on the selected virtual object that is placed at a position corresponding to the virtual object position information stored in the server 5.

[0088] In step S41, the operation unit 121 updates the virtual object position information stored in the server 5 with the position information of the virtual object after the operation, and proceeds to step S5 (FIG. 10).

[0089] If the instruction is to switch the image (Yes in step S5), the image switching unit 130 of the display image control unit 13 executes image switching processing in step S6, and the operation proceeds to step S7. If the instruction is not to switch the image (No in step S5), the virtual object display device 1 proceeds to step S7.

[0090] Here, the operation of the image switching process in step S6 will be described with reference to Fig. 13. In step S60, the image switching unit 130 sets the type of displayed image to an MR image or a virtual image in response to an instruction. This instruction may be given directly from an external device such as a keyboard KB, or may be generated in step S23. In step S61, the image switching unit 130 sets the position of the virtual camera to the center of a spherical object, which is the specified virtual object, or to any position in the virtual space, and proceeds to step S7 (Fig. 10).

[0091] If the instruction is to change the line of sight direction (Yes in step S7), in step S8, the line of sight changing unit 131 or the viewpoint line of sight changing unit 132 of the display image control unit 13 executes line of sight changing processing, and the operation proceeds to step S9. If the instruction is not to change the line of sight direction (No in step S7), the virtual object display device 1 proceeds to step S9. Note that the line of sight changing unit 131 executes this processing when the MR image is displayed, and the viewpoint line of sight changing unit 132 executes this processing when the virtual image is displayed.

[0092] Here, the operation of the line of sight change processing in step S8 will be described with reference to FIG. 14. In step S80, the line of sight change unit 131 or the viewpoint line of sight change unit 132 determines whether the current display mode is the normal mode. If the display mode is the normal mode (Yes in step S80), in step S81, the line of sight change unit 131 or the viewpoint line of sight change unit 132 sets the line of sight direction of the virtual camera in accordance with the movement of the mouse MS. After setting the line of sight direction in step S81, or if the current display mode is not the normal mode (No in step S80), the virtual object display device 1 proceeds to step S9 ( FIG. 10 ).

[0093] If the instruction is to change the viewpoint direction (Yes in step S9), in step S10, the viewpoint line of sight changing unit 132 of the display image control unit 13 executes viewpoint change processing, and the operation proceeds to step S11. If the instruction is not to change the viewpoint direction (No in step S9), the virtual object display device 1 proceeds to step S11.

[0094] The viewpoint change process in step S10 will now be described with reference to Fig. 15. In step S100, the viewpoint line of sight change unit 132 determines whether the currently displayed image is a virtual image or an MR image. If the displayed image is an MR image, the viewpoint is switched by selecting a celestial sphere object.

[0095] If the displayed image is a virtual image ("Virtual Image" in step S100), in step S101, the viewpoint line of sight change unit 132 sets the viewpoint position of the virtual camera by pressing a predetermined key on the keyboard KB. After the viewpoint position is set in step S101, or if the currently displayed image is an MR image ("MR Image" in step S100), the virtual object display device 1 proceeds to operation in step S11 (FIG. 10).

[0096] If the instruction is to switch modes (Yes in step S11), the mode switching unit 133 of the display image control unit 13 executes mode switching processing in step S12, and the operation proceeds to step S13. If the instruction is not to switch modes (No in step S11), the virtual object display device 1 proceeds to step S13.

[0097] 16, the operation of the mode switching process in step S12 will be described. In step S120, the mode switching unit 133 determines the specified mode. If the specified mode is the normal mode ("normal mode" in step S120), in step S121, the mode switching unit 133 sets the current mode to the normal mode.

[0098] Furthermore, if the designated mode is the gaze mode ("gaze mode" in step S120), the mode switching unit 133 sets the current mode to the gaze mode in step S122. However, although not shown, if the currently displayed image is a virtual image, the gaze mode is not set.

[0099] Furthermore, if the specified mode is the tracking mode ("tracking mode" in step S120), the mode switching unit 133 sets the current mode to the tracking mode in step S123. However, although not shown, if the currently displayed image is an MR image, the tracking mode is not set. Thereafter, the virtual object display device 1 proceeds to step S13 (FIG. 10).

[0100] In step S13, the image generator 14 determines whether the currently displayed image is an MR image. If the currently displayed image is an MR image ("MR image" in step S13), the MR image generator 140 executes an MR image generation process in step S14, and the process proceeds to step S16. If the currently displayed image is a virtual image ("virtual image" in step S13), the virtual image generator 141 executes a virtual image generation process in step S15, and the process proceeds to step S16.

[0101] 17 , the MR image generation process in step S14 will be described. In step S140, the MR image generation unit 140 acquires virtual object position information from the server 5 via the communication device 4. In step S141, the MR image generation unit 140 acquires a celestial sphere image corresponding to the currently selected celestial sphere object from the server 5 via the communication device 4.

[0102] In step S142, the MR image generator 140 combines the celestial sphere image acquired in step S142 with the image of the virtual object identified by the virtual object position information acquired in step S141 to generate a display image corresponding to the currently set viewpoint position and line of sight direction. Note that when the gaze mode is set, the MR image generator 140 combines the images so that a preset avatar representing the user (operator) is at the center of the image.

[0103] Next, the virtual image generation process in step S15 will be described with reference to Fig. 18. In step S150, the virtual image generation unit 141 acquires virtual object position information from the server 5 via the communication device 4.

[0104] In step S151, the virtual image generating unit 141 places a virtual object in a virtual space, projects it onto a virtual camera corresponding to the currently set viewpoint position and line of sight direction, and generates a display image. Note that when the tracking mode is set, the virtual image generating unit 141 synthesizes the captured image so that a preset avatar representing the user (operator) is positioned at the center of the bottom of the captured image.

[0105] 10 , in step S16, the virtual object display device 1 determines whether or not an end command has been issued by an end button, switch, or the like on a menu screen (not shown). If an end command has not been issued (No in step S16), the virtual object display device 1 returns to step S1 and continues operation. On the other hand, if an end command has been issued (Yes in step S16), the virtual object display device 1 ends operation. Note that the operation of the HMD 3 is the same as that of the conventional device (Non-Patent Document 1) except that virtual object position information is stored in the server 5, and therefore a description thereof will be omitted.

[0106] Through the operations described above, the virtual object display device 1 can switch between displaying, via the monitor MO, an MR image that combines images of the real world and images of virtual objects, and a virtual image of a virtual object placed in a virtual space as seen from any viewpoint, to a user who is not wearing the HMD 3.

[0107] In this way, by mixing the image of the real world with the image of the virtual object, a user who is not wearing the HMD 3 can see the MR image I shown in FIG. Ｖ In the present invention, the MR image I in FIG. Ｖ In FIG. 19, the desk O Ｒ The object O is a virtual object placed on ＶＴ is superimposed on the person M, and the object O ＶＴ On the other hand, in FIG. 20, it is difficult to recognize the placement position of the object O ＶＴ Since the image is mixed with the real world image, the person M in the foreground is not completely overwritten, and the object O ＶＴ The placement position of the object can be correctly recognized.

[0108] Although the virtual object display system 100 according to the embodiment of the present invention has been described above, the present invention is not limited to this embodiment. For example, as shown in FIG. 1, the virtual object display system 100 is assumed to be operated by a user wearing an HMD 3 only at the local site. In the example of FIG. 1, a virtual object at a remote location is displayed as a spherical object O. Ｖ３ (See FIG. 2 ). However, a user wearing the HMD 3 may be present both locally and remotely and place a movable virtual object at the remote location. Then, the virtual object at the remote location may be moved into the local virtual space by a preset operation. This allows an effect in which the virtual object at the remote location appears to pop up in the local virtual space.

[0109] REFERENCE SIGNS LIST 100 Virtual object display system 1 Virtual object display device 10 Virtual object information storage unit 11 Instruction analysis unit 12 Virtual object control unit 120 Selection unit 121 Operation unit 13 Display image control unit 130 Image switching unit 131 Line of sight change unit 132 Viewpoint line of sight change unit 133 Mode switching unit 14 Image generation unit 140 MR image generation unit 141 Virtual image generation unit 2 Celestial camera 3 HMD 30 Virtual object information storage unit 31 HMD main body unit 32 Virtual object control unit 320 Selection unit 321 Operation unit 4 Communication device 5 Server

Claims

1. A virtual object display device that displays virtual objects, where the virtual objects are an object object representing an object and a spherical object representing a celestial camera, and is connected via a network to a server that stores celestial spherical images of real space captured by a celestial camera and positional information of the virtual objects, comprising: an image generation unit that generates and displays, as a display image, an MR image that combines the celestial spherical image with an image of the virtual object captured by a virtual camera placed at the center of the celestial camera, or a virtual image of the virtual object captured by a virtual camera placed at an arbitrary position, based on the positional information of the virtual object; and an image switching unit that switches the display image between the MR image and the virtual image based on an operation instruction.

2. The virtual object display device according to claim 1, characterized in that the server stores a plurality of celestial sphere images captured by a plurality of celestial sphere cameras, and the image switching unit switches the displayed image to one of a plurality of MR images corresponding to the plurality of celestial sphere images or the virtual image based on the operation instruction.

3. The virtual object display device according to claim 1, further comprising: a line of sight change unit that changes the line of sight direction to a specified direction when the MR image is displayed; and a viewpoint line of sight change unit that changes the viewpoint position or line of sight direction to a specified position or direction when the virtual image is displayed.

4. The virtual object display device according to claim 1, further comprising a mode switching unit that switches between: a mode in which the physical object includes an avatar that represents the operator, and that generates the MR image so that the avatar is at the center of the displayed image when the MR image is displayed, following the movement of the avatar; a mode in which the virtual image is displayed so that the virtual image is generated in the line of sight of the avatar, following the movement of the avatar; and a mode in which the displayed image does not follow the movement of the avatar; and the image generation unit generates the displayed image in the mode switched by the mode switching unit.

5. A virtual object display program for causing a computer to function as the virtual object display device according to any one of claims 1 to 4.

6. A virtual object display system in which virtual objects are object objects representing objects and spherical objects representing a celestial camera, a celestial camera placed in real space, a server that stores celestial images of the real space captured by the celestial camera and positional information of the virtual objects, and a virtual object display device, all connected to a network, wherein the virtual object display device comprises: an image generation unit that generates and displays, as a display image, an MR image that combines the celestial image with an image of the virtual object captured by a virtual camera placed at the center of the celestial camera, based on the positional information of the virtual object, or a virtual image of the virtual object captured by a virtual camera placed at an arbitrary position; and an image switching unit that switches the display image between the MR image and the virtual image based on an operation instruction.

7. The virtual object display system according to claim 6, further comprising a head-mounted display connected to the network that displays the virtual object as an MR image superimposed on real space, wherein the head-mounted display has a hand tracking function and updates the position information of the virtual object stored in the server in accordance with the operator's movements.

Images