Virtual display system

Abstract

In virtual display in a virtual space, a virtual display system and method are provided that improve the reality of the venue and enhance the user's sense of immersion. 【Solution means】A virtual display system that displays an image of a virtual space on goggles 301 worn by a user based on data sent from a server 100. The server prepares, as image data for virtual display, a venue image representing a virtual space that is the venue for virtual display and a booth image representing an exhibitor's booth as high-resolution still images, respectively. Also, information-providing images such as a promotional video to be displayed to the user at the booth are prepared as high-resolution videos. Separately from these images, an object video representing a movable object moving within the virtual space, for example, a video of people walking back and forth in the venue, is prepared as a low-resolution video. By transmitting and superimposing these data on the user terminal 300 for display, the reality of the virtual display, such as people walking back and forth, is improved.

Description

Technical Field

[0001] The present invention relates to a virtual display system that presents a virtual space to a user.

Background Art

[0002] Recently, technologies for holding exhibitions and other events using virtual spaces have attracted attention. Patent Document 1 discloses a system that presents data in a virtual space to a browsing user in a state where it can be browsed by a three-dimensional viewer such as a head-mounted display. The browsing user can browse virtual exhibits while moving within the virtual space. When the browsing user makes a request, an explanation about the virtual exhibit is displayed in characters or the like, or output as audio. Patent Document 2 discloses a technique for realizing a virtual space in which an object is arranged at an arbitrary position by arranging an image obtained by projecting a three-dimensional model in which an object is arranged within a virtual section, using a full-sphere panoramic image representing the virtual space as a background image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When performing virtual display in a virtual space, it is important to improve the reality and give a sense of immersion to the users who visit the virtual display. At a real exhibition, a lively atmosphere at the venue, such as a large number of people coming and going, may have a secondary effect of enhancing the interest in the exhibition. In virtual display as well, such secondary effects are expected for improving reality and immersion. However, in the conventional virtual display system, the improvement of reality and immersion has not yet been fully considered. Even if, in the prior art, the background image representing the entire venue is prepared not as a still image but as a moving image representing a state where many people are coming and going, it is possible to realize an atmosphere close to the actual venue. However, in this case, it is necessary to distribute the moving image to the terminal used by the user, and problems such as communication speed and traffic occur. On the contrary, if the moving image is made of low resolution in consideration of the communication speed etc., the entire venue will become a rough image, which may rather reduce the reality. Such problems are not limited to the case of holding an exhibition in a virtual space, but are common problems when presenting a virtual space to a user for various purposes such as tourism, other nature observations, art museums, museums, event venues, factory tours and other facility visits. An object of the present invention is to solve such problems and provide a technique for improving reality and immersion when presenting a virtual space.

Means for Solving the Problems

[0005] The present invention is a virtual display system for electronically performing a virtual display in a virtual space, a virtual data storage unit that stores virtual data representing the virtual space and the virtual display, and a movable object whose position in the virtual space can move and object data representing its movement; a viewing information input unit that inputs viewing information for specifying the viewing field of the user in the virtual space who views the virtual display; and a user display control unit that, based on the viewing information, uses the virtual data and the object data to display an image of the virtual display on a user output device used by the user. The user display control unit displays a background image as a still image representing the virtual space and the virtual display based on the virtual data, and A virtual display system can be configured to superimpose and display an object video prepared at a resolution lower than that of the background image based on the object data in front of the background image in a state where objects other than the movable object are transparent.

[0006] According to the present invention, by displaying an image of a virtual space and a virtual display, a user can be made to recognize that they are browsing a virtual display within the virtual space. Here, in the present invention, a background image representing a virtual space or a virtual display is prepared as a still image, and an object video representing a movable object is superimposed and displayed in front of the still image. The movable object is, for example, people moving around in the virtual space, characters, balloons or other floating objects floating in the virtual space, and so on. By doing so, the reality of the virtual space viewed by the user can be improved, and it becomes possible to give the user a sense of immersion. Moreover, in the present invention, the background image is prepared at a high resolution, and the object video is prepared at a low resolution. By doing so, it is possible to avoid deterioration in the quality of the image of the virtual space or the virtual display while suppressing the data volume transmitted to the user output device. Note that the resolutions of the background image and the object video are relative, and specific resolutions can be arbitrarily determined in consideration of image quality, communication speed to the user output device, and so on.

[0007] In the present invention, the visual field information can be, for example, the position of the viewpoint, the line-of-sight direction, the field angle, etc. within the virtual space. Only a part of these may be input. For example, the virtual display system may input the position of the viewpoint as visual field information, and the user output device may specify the line-of-sight direction, the field angle, etc. for display. Also, some information may be set as a preset fixed value. In the present invention, the virtual space and the virtual display may be two-dimensional, but by making them three-dimensional, a more immersive display can be achieved. Also, the virtual space and the virtual display may be provided by means of CG (computer graphics), or may be provided by pasting textures of still images or videos created from photographs, videos, and CG of actual displays or the like. The virtual space and the virtual display can be provided by various other methods. The virtual display can be used to explain the content of products or services like a so-called exhibition, and can also cover various contents such as the scenery of tourist destinations, art museums, expositions, event venues, etc. The movable object means an object whose position can move within the virtual space. In addition to the people moving around in the virtual space described above, any movable object can be used, such as animals, birds, rain, snow, clouds, the flow of waterfalls or rivers. Also, even if the position of the tree itself does not move, for example, when its branches and leaves move like the tree swaying in the wind, the whole can be regarded as a movable object.

[0008] The user output device may be a display such as a personal computer, a tablet, or a smartphone, or may utilize goggles or glasses worn on the head. In particular, when using goggles or glasses, there is an advantage of further enhancing the sense of immersion.

[0009] In the present invention, the user display control unit may respectively transmit the data for displaying the background image and the data for displaying the object video to the user output device, and cause the user output device to perform the superimposed display.

[0010] This mode is a mode in which a background image and an object video are separately transmitted to a user output device and superimposed on each other. By doing so, the user output device can easily realize various uses such as caching and repeatedly using the object video, caching and selectively using multiple types of object videos, or displaying a still image instead of the object video when the communication state is unstable. However, the display in the present invention does not exclude a mode in which, in a virtual exhibition system, video data in which a background image and an object image are superimposed is distributed to a user output device for display.

[0011] In the present invention, The user display control unit may limit the display of the object video to a predetermined area where the movable object exists in the virtual space.

[0012] By doing so, the data amount of the object video can be further reduced. The predetermined area can be arbitrarily determined according to the movable object. For example, when people passing by are used as movable objects, the object video may be prepared by omitting the space above a certain height in the virtual space. As this method, a method of preparing object video data limited to a predetermined area, a method of setting all to be completely transparent while preparing dummy data also outside the predetermined area, etc. can be considered.

[0013] In the present invention, The user display control unit may repeatedly display the object video of a predetermined length.

[0014] By doing so, the data capacity of the object video can be further suppressed.

[0015] In the present invention, The virtual data storage unit stores a plurality of types of the object data, The user display control unit may switch any of the object data and display the object video.

[0016] In this aspect, since a plurality of types of object videos can be switched and displayed, it is possible to prevent the user from getting used to or bored with the object video, and it is possible to maintain the reality. The types of object videos to be prepared can be arbitrarily determined. It is not necessary for the plurality of types of object videos to have the same length, and they may have different lengths. Also, the selection can be made in various ways. You may select according to a predetermined order or a predetermined rule, or you may select randomly.

[0017] Note that, in either the aspect of repeatedly using the object video or the aspect of selecting and using a plurality of types of object videos, it is not always necessary to play the object video from the beginning, and you may appropriately select the part to play, such as playing from the middle.

[0018] Also, in either the aspect of repeatedly using the object video or the aspect of selecting and using a plurality of types of object videos, The user display control unit may apply one or both of fade-in and fade-out to the object video and display it.

[0019] By doing so, it is possible to alleviate the sense of discomfort at the seam when shifting from the playback of one video to the playback of the next video. Note that fade-in and fade-out are not essential, and only one of them may be applied, or neither of them may be applied. When neither fade-in nor fade-out is applied, for example, the sense of discomfort at the seam may be reduced by preparing so that the last frame of one video and the first frame constitute a continuous image.

[0020] In the present invention, The user display control unit may realize a display in a state where parts other than the movable object are transparent by setting an alpha channel representing the parts to be made transparent in the data for displaying the object video.

[0021] To make the background image visible, various methods can be used to make part of the object video transparent. As one of the above aspects, it is a method of setting an alpha channel as transparency information in the object video itself. By doing so, the object video and the transparency information can be integrated, and there is an advantage that no deviation occurs between the two. However, the transparency is not limited to such a method. For example, separately from the object video, a mask data may be prepared by setting a place to be made transparent. Since the place to be made transparent also moves according to the movement of the movable object, the mask data is also prepared as time-series data. In this case, every time a deviation of a predetermined amount or more occurs between the object video and the mask data, the two may be forcibly synchronized.

[0022] In the present invention, a transmittance may be set for the movable object. For example, if a transmittance that blends with the background and is visible is set in a range greater than 0% and less than 100% for a movable object located farther away from the viewpoint, the visibility is reduced, so the movement of the movable object can be prevented from giving the user a sense of discomfort. Also, a movable object in a predetermined area including a virtual display may be made transparent. By doing so, although there may be a risk of a sense of discomfort with respect to the movement of the movable object, there is an advantage that the visibility of the virtual display can always be ensured.

[0023] In the present invention, the virtual data storage unit further stores video data for displaying an information-providing video for providing information to the user, and the user display control unit may display the information-providing video at a predetermined position in the virtual space on the back side of the object video.

[0024] An information-providing video refers to, for example, a video that is displayed for presenting products or the like to a user within a booth in a virtual exhibition, a video that distributes events, venue layouts, programs, etc. within a virtual space, and the like. As a method of displaying these information-providing videos within a virtual space, a method can be considered in which the information-providing video is played separately from the background image, subjected to an affine transformation or the like, and pasted at a display position according to the user's visual field information. By doing so, for example, a state can be realized in which the information-providing video is displayed as if it were on a panel within a booth or a guide board within a venue. However, in the present invention, since there is an object video, if this is superimposed between the background image and the information-providing video, it may result in a very uncomfortable display as if a movable object is sandwiched between a wall and a panel. Therefore, if the information-providing video is displayed on the back side of the object video as in the above aspect, it is possible to suppress the occurrence of such discomfort.

[0025] In the present invention, when the object video cannot be normally displayed on the user output device, the user display control unit may stop the display of the object video.

[0026] The case where the object video cannot be normally displayed refers to, for example, a case where the communication speed between the virtual exhibition system and the user output device is slow, a case where the data of the object video cannot be smoothly transmitted due to poor communication, or a case where it has not been sufficiently cached yet. According to the above aspect, in such a case, since the display of the object video is stopped, it is possible to avoid a state where the display is delayed until the data of the object video is cached to a certain extent in the user output device, or the display is stuck due to poor communication of the object video. The stop of the display of the object video may be temporarily stopped until the caching is completed in the user output device, or the display of the object video itself may be completely abandoned. Also, when not displaying the object video, only the virtual space and virtual display may be displayed, or instead of the object video, a still image representing a movable object may be displayed.

[0027] In the present invention, The position of the user in the virtual space may be restricted to a preset default position.

[0028] By doing so, virtual display can be provided efficiently, and the user can also browse efficiently. In the above aspect, the view information at the specified position may be arbitrary, or the view information may be restricted such as only in the front.

[0029] The present invention does not necessarily have to have all of the above-described features, and some of them may be omitted or combined as appropriate. It may be configured as a virtual display method for executing the above-described virtual display by a computer, or may be configured as a computer program for causing a computer to perform such a method. Further, it may be configured as a computer-readable recording medium on which such a computer program is recorded.

Brief Description of the Drawings

[0030]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0031] Hereinafter, taking the case of holding an exhibition in a virtual space as an example, embodiments of the present invention will be described separately in the following items. A. System Configuration: B. Display of Virtual Space and Virtual Exhibition: C. Display of Object Video D. Display Control Process for Users: E. Effects and Modification Examples:

[0032] A. System Configuration: FIG. 1 is an explanatory diagram showing the configuration of a virtual exhibition system in an embodiment. It is a system in which a plurality of exhibitors set up virtual booths in a virtual space and provide virtual exhibitions of products or services to users. The virtual exhibition system is configured by connecting a server 100 that provides its main functions, exhibitor terminals 200 used by exhibitors of each booth, and user terminals 300 used by users who visit the booths, via a network NE such as the Internet. As the exhibitor terminals 200 and user terminals 300, various devices connectable to the network NE such as personal computers, tablets, and smartphones can be used. Also, the functions of the server 100 may be provided by a plurality of units. A plurality of exhibitor terminals 200 and user terminals 300 can also be connected.

[0033] The exhibitor terminals 200 and user terminals are provided with the functions shown in the figure. The transmission / reception units 211 and 311 function to transmit and receive information via the network NE. Examples of the information include characters, images, videos, and voices. Also included are various instructions from exhibitors and users to the virtual exhibition system. The input / output units 212 and 312 control devices for inputting and outputting information. In this embodiment, it is assumed that the operator of the exhibitor wears the goggles 201 and the user also wears the goggles 301. Various images provided by the virtual exhibition system are displayed on these goggles 201 and 301, and voices are output. Also, a microphone is incorporated, and voices can be input. When the operator inputs to the virtual exhibition system, devices such as the keyboard and mouse of the exhibitor terminal 200 are used. When the user inputs, the touch panel of the user terminal 300 is used. The input / output can take various forms, not limited to such examples. The applications 213 and 313 provide functions such as causing the above-described input / output to be performed based on the information transmitted from the virtual exhibition system. They may be configured as software unique to the virtual exhibition system or may use general-purpose software such as a browser. In this embodiment, as will be described later, an object video representing a movable object moving in the virtual space is superimposed on a background image (still image) representing the virtual space and the virtual exhibition and displayed. The applications 213 and 313 receive data such as the background image and the object video, generate an image in which these are superimposed, and function to display it on the goggles 201 and 301 via the input / output units 212 and 312.

[0034] The server 100 is provided with the functions shown in the figure. The virtual data storage unit 101 stores various data (hereinafter sometimes collectively referred to as "virtual data") for realizing the virtual space and the virtual exhibition. In this embodiment, the virtual data representing the entire venue where the virtual exhibition is held is prepared by the organizer of the virtual exhibition or the operator of the virtual exhibition system. The virtual data representing each booth is prepared by the exhibitor. The virtual data includes both still images and videos. For example, a still image representing a building that serves as a venue for a virtual exhibition can be an image corresponding to a full sphere, i.e., a so-called 360-degree equirectangular image, which can be prepared using CG or actual shooting. Also, an image for a virtual exhibition such as a booth can be generated, for example, by preparing a three-dimensional model and projecting it. The image for the virtual exhibition may be integrated with the image representing the venue. In addition, for example, in the venue, a two-dimensional information-providing video that provides information such as event information, venue layout, and various programs to the user, or an information-providing video (so-called promotion video) that provides information on products, etc. at each booth may be prepared. The user data storage unit 102 stores information about users who visit each booth. The exhibitor data storage unit 103 stores information about exhibitors who exhibit at each booth. The material data storage unit 104 stores data such as materials and videos distributed to users at each booth. The specific content and structure of these data will be described later.

[0035] The virtual space allocation unit 111 allocates a virtual space to the user. In this embodiment, when the user visits each booth, the virtual space is set as a space unique to that user. By doing so, the user can freely look around the booth, and it becomes possible for the exhibitor's operator and the user to communicate as if only that user exists in the booth. The data management unit 112 manages the input and output of data in the virtual data storage unit 101, the user data storage unit 102, the exhibitor data storage unit 103, and the material data storage unit 104. The transmission / reception unit 113 transmits and receives information to and from the exhibitor terminal 200 and the user terminal 300 via the network NE.

[0036] The display control unit 121 for exhibitors generates and transmits image data to be displayed on the exhibitor terminal 200. Information for generating the image may be transmitted to the exhibitor terminal 200 to cause the generation of the image data. The guidance information input unit 122 inputs guidance information for guiding products or services from the exhibitor terminal 200. The guidance information can be input by an operator at the exhibitor terminal 200 in the form of characters, voices, etc. By transmitting the input guidance information to the user terminal 300, communication between the operator and the user is realized. The information providing unit 123 provides various information regarding the exhibition results to the exhibitor. For example, information such as the time the user stayed at the booth, the number of users who visited the booth, and the types of materials the user obtained at the booth can be mentioned. Analysis results based on this information may also be included. The push-type guidance unit 124 functions to allow the operator to set the position of the user, the line-of-sight direction, etc. By doing so, it is also possible to guide the user inside the booth as if the operator were leading the user. Also, when the operator is unable to handle it, the server 100 may automatically guide the user according to the pre-prepared data.

[0037] The display control unit 131 for users generates and transmits image data (hereinafter referred to as "user images") to be displayed on the user terminal 300. Information for generating the image may be transmitted to the user terminal 300 to cause the generation of the image data. The position setting unit 132 sets the position of the user within the virtual space. In this embodiment, the positions where the user can move are preset, and by the user selecting any one of them, the user can move within the virtual space. The user may be able to move freely within the virtual space. The visual field information input unit 133 inputs visual field information from the user terminal 300. The visual field information is information such as the viewpoint position, the line-of-sight direction, and the field of view angle for generating the user image. In this embodiment, the viewpoint position of the user, that is, the position information of the user, is input as the visual field information, and display data is prepared regardless of the line-of-sight direction and the field of view angle. The line-of-sight direction and the field of view angle are input at the exhibitor terminal 200 and the user terminal 300, and display data corresponding thereto is generated and displayed on the goggles 201 and 301. The material distribution unit 134 transmits the data of the materials distributed within the booth according to the instructions of the user. The data of the materials is downloaded to the user terminal 300 or the like.

[0038] The virtual exhibition system is not limited to such a configuration and can take various configurations. It is not necessary to have all the illustrated functions, and some of them may be omitted. Also, it may have functions other than those illustrated. Note that each of the functions described above is constructed software-wise in this embodiment, but it is also possible to construct part or all of them hardware-wise.

[0039] Figure 2 is an explanatory diagram showing the data structure. In the exhibitor data storage unit 103, an exhibitor ID, which is an identifier for each exhibitor, is assigned, and information of the exhibitor such as the exhibitor name, contact information, virtual data, inviter, and visitor is stored. For the convenience of the data structure, the exhibitor ID is also assigned to the organizer of the virtual exhibition or the operator of the virtual exhibition system. The virtual data information indicates the location of the 3D model, photo / video, texture, and other data that constitute the exhibitor's booth. In this embodiment, since the virtual data is stored in the virtual data storage unit 101 with a virtual data ID as an identifier, the virtual data ID can be stored as the virtual data information. The inviter information is the information of the users invited by the exhibitor to the booth. As will be described later, since each user is assigned a user ID as an identifier, the user ID can be stored as the inviter information. The visitor information is the information of the users who visited the booth. Similar to the inviter, the user ID is stored.

[0040] In the user data storage unit 102, information of the user such as name, contact information, visit log, material log, and group is stored for each user ID. The visit log is the history of the visited booths. It can be recorded by the time of visiting the booth, the time of leaving, etc. Also, the booth can be specified by the exhibitor ID. The data log is the history of materials obtained by downloading and can be recorded using a unique ID attached to the materials or the like. The group information stores the user IDs of the users who make up the user and the group. In this embodiment, when a booth is visited, the booth is allocated as a virtual space unique to the user, but by specifying a group, it is also possible to allocate it as a virtual space unique to the group. The group information is preferably editable by the user as appropriate.

[0041] The virtual data storage unit 101 stores virtual data, object data, and video data. All of these data exist in the virtual data, including data prepared by the organizer of the virtual exhibition or the operator of the virtual exhibition system, and data prepared by each exhibitor. The entity that prepared the data is distinguished by the "exhibitor ID". Virtual data is data that constitutes a virtual space and a virtual exhibition. Virtual data prepared by the organizer or the like includes a 3D model, photos, videos, textures, etc. that represent the entire virtual space of the venue where the virtual exhibition is held. Virtual data prepared by each exhibitor includes booth data and the like. The virtual data is stored with a virtual data ID attached to each. When an exhibitor participates in multiple booths, a plurality of virtual data IDs will be assigned. The same applies when an exhibitor or the like prepares multiple types of virtual data, such as when preparing virtual data of the venue according to time zones such as morning, noon, and night. The virtual data information stores booth data, exhibit data, viewpoint data, guide data, browsing logs, and the like. Booth data is data such as a 3D model, photos, videos, textures, etc. that make up the booth. Photos or videos taken of the actual booth may be used to represent the booth. Exhibit data is 3D data, 2D image data, video data, etc. of the exhibits in the booth. In addition to providing the exhibits in CG, it may also be provided by pasting photos or videos taken of the exhibits. Viewpoint data stores the positions within the booth where the user can move. In this embodiment, as will be described later, the user cannot move completely freely within the booth, but can selectively move to preset locations. The viewpoint data is data representing these movable positions. The guide data is three-dimensional image data representing the person who guides the user within the booth, i.e., data of a so-called avatar. In this embodiment, the guide does not move freely within the booth, but is fixedly displayed at a specified position. The guide data also stores information on the position where the guide is to be displayed. The browsing log is the history of users who have visited the booth. The visit time and departure time are recorded for each user. The content of the communication between the user and the operator may also be recorded. In addition, when the operator is absent, data for automatically guiding the booth to the user may be prepared. For example, voice guidance data, data for controlling the user's viewpoint and line of sight, data for instructing the enlargement of panels within the booth, the playback of videos, etc. can be considered. It is not necessary to prepare all the data exemplified here, and they may be appropriately omitted or the like.

[0042] Object data is movable object data within the virtual space whose position within the virtual space can move and object data representing its movement. Similar to the virtual data, an object data ID, which is an identifier, is assigned to each object data, and an exhibitor ID representing the entity that prepared the data is also assigned. As the movable object data, data representing the movable object and its movement are prepared. In this embodiment, the movable object data is prepared as video data of a predetermined length for the movable object to move within the virtual space. For example, CG or live-action videos representing people moving back and forth within the virtual space can be used. Instead of the above-described mode, as the movable object data, for example, a three-dimensional model representing the movable object and data defining its movement may be prepared. Viewpoint data is data representing the viewpoint position corresponding to the movable object data. In the case of this embodiment, as described above, since the positions where the user can move are limited and the movable object data is prepared for each viewpoint, data for associating with the user's position is prepared as the viewpoint data.

[0043] Video data is data of an information-providing video for providing information to the user. Examples include videos for providing event information in the virtual space and promotional videos for providing information about products, etc. inside the booth. Similar to the virtual data, the video data is also assigned a video data ID as an identifier, and an exhibitor ID representing the entity that prepared the data is also assigned. The video data can be prepared, for example, as two-dimensional video data. The display position data is the position information in the virtual space where the information-providing video is displayed.

[0044] The virtual space allocation data is data generated by the virtual space allocation unit 111 when the user visits the booth. As shown in the figure, it associates the virtual data ID for specifying the booth, the exhibitor ID for specifying the exhibitor, and the user ID for specifying the user. When the user visits in a group, a plurality of user IDs existing in the group are associated. The user's questions, etc. are conveyed only to the operator of the associated exhibitor ID, and conversely, the operator's answers are conveyed only to the associated users. By doing so, the user can have a closed communication with the operator without worrying about leakage to other users. The virtual space allocation data is generated for each user or group. The virtual data IDs may overlap. By doing so, the booth display can be provided to multiple users while providing a unique virtual space for each user. When multiple users visit a booth, multiple virtual space allocation data are set for a single exhibitor ID. In such a case, the operator will select one of the visiting users and communicate with them. The conversation flag is a flag indicating whether communication with the operator is possible. When the operator selects one of the visiting users, the conversation flag is turned on in the virtual space allocation data for that user, and the conversation flags for other virtual space allocation data are turned off. By doing so, even when multiple users visit the booth, the above-described closed communication can be realized. In addition, when preparing multiple exhibitor terminals 200, multiple operators may share the work of handling users. In such a case, the flag may be omitted.

[0045] B. Display of Virtual Space and Virtual Exhibition FIG. 3 is an explanatory diagram showing a method for generating a displayed image. As shown in FIG. 3(a), in this embodiment, the virtual space representing the venue is configured by pasting a 360-degree equirectangular image of a photo / video representing the venue on the inner surface of a spherical virtual space S. Also, as shown in the figure, the booth B is configured with virtual data by the exhibitor using a 3D model or the like and is in a state of being arranged within the virtual space. The user image can be generated by perspective projection according to visual field information such as the viewpoint VP, the line-of-sight direction VL, and the viewing angle VA within the virtual space. The viewpoint VP and the line-of-sight direction VL can be determined using 3D coordinates within the virtual space. The viewing angle VA may be fixed or adjustable by the user. For the sake of illustration and explanation, in FIG. 3, the viewpoint VP is shown as being located outside the virtual space S of the venue, but in reality, the viewpoint VP will be located at the center of the virtual space S. The booth B may also be created by synthesizing a photo of the actual booth or an image generated by perspective projection into the 360-degree equirectangular image of the virtual space. Hereinafter, the images representing the virtual space and the booth may also be collectively referred to as background images.

[0046] FIG. 3(b) is an example of a background image in this embodiment. The state where a 360-degree equirectangular image of a booth generated by perspective projection is synthesized with a 360-degree equirectangular image of an exhibition hall is illustrated. By attaching this image to the inner surface of the entire sphere, virtual space and virtual exhibition can be visually recognized for any line of sight of the user.

[0047] FIG. 4 is an explanatory diagram showing an example of an image (1) displayed to the user. An image IM0 of the venue and an image IM1 depicting the booth three-dimensionally are shown. Also, a guide G guiding the booth is displayed. A person PM inside the booth is a humanoid fixed model drawn to express a sense of presence. The guide G and the person PM are not objects that move within the virtual space. Therefore, in this embodiment, they are displayed as still images instead of object videos. By preparing still images of a plurality of poses, they may be displayed with changed poses as if they were videos. Exhibits hung on the wall are panels (still images) represented by still images, videos, etc. This video is one of the information-providing videos provided to the user and is displayed using the video data described in FIG. 2. VP1 drawn on the floor represents a viewpoint position where the user can move. A floor plan W1 of the booth is displayed in the upper left of the user image. An enlarged view of the floor plan W1 is shown above the figure. In this embodiment, the user cannot move freely inside the booth but selectively moves to a preset viewpoint position. The floor plan W1 illustrates the viewpoint positions and the field of view where the user can move. The movable viewpoint positions are indicated by camera icons. The user image shown in FIG. 4 represents the state of being viewed from the viewpoint position VP0 like the field of view VA. The circular viewpoint position VP1 represented on the floor in the user image corresponds to the viewpoint position VP1 represented in the lower right of the floor plan W1. When the user clicks or selects an icon of any camera, the viewpoint position moves. The circular viewpoint position VP1 drawn on the floor in the user image may be selected. FIG. 5 shows an image when the viewpoint position VP2 is selected.

[0048] FIG. 5 is an explanatory diagram showing an example of an image (2) displayed to the user. It can be seen from the plan view W1 shown in the upper left that the viewpoint position VP2 is selected in FIG. 4, and it is an image in a state of viewing the wall side. A display is drawn on the wall, and the video MV is automatically played silently, giving a sense of presence as an exhibition. When the user clicks the icon IC1, the video MV is enlarged and played with sound. The video may be played over the entire user image. A panel is displayed beside it. When the user clicks the icon IC2 under the panel, the panel is enlarged and displayed. When the user clicks the icon IC3, the material can be downloaded.

[0049] C. Display of Object Video Next, the method for displaying the object video in this embodiment will be described. FIG. 6 is an explanatory diagram showing a method for displaying an object video. FIG. 6(a) shows the background image described in FIG. 3. As shown in FIG. 6(c), the background image is composed of a 360-degree equirectangular image showing the entire venue (referred to as the venue image) and an image generated by perspective projection of each booth (referred to as the booth image). In this embodiment, both are generated as high-resolution still images. They may be combined as the background image, or they may be superimposed and displayed. In addition to these background images, as described with reference to FIGS. 4 and 5, an information-providing image for providing information to the user is also prepared. In the example of FIG. 6(a), the promotional video displayed on the panel placed in front of the booth is the information-providing image. The information-providing image is a high-resolution video.

[0050] FIG. 6(b) illustrates an object video. Here, images of people MO1, MO2, etc. as movable objects moving back and forth in the virtual exhibition are used as the object video. To avoid complication of the figure, although not labeled in the figure, the other people shown also correspond to movable objects. The object video is a video in which these movable objects MO1, MO2, etc. walk in the virtual space. In addition, in FIG. 6(b), an object video is created in a state of being superimposed on the background image, but the background image may be omitted, and only movable objects MO1, MO2, etc. may be used as the object video. As will be described later, the object video is displayed superimposed on the background image. In the object video, portions other than the movable objects are set to a transparent state so that the background image can be visually recognized when superimposed. Such a setting can be performed by setting an alpha channel that determines the transmittance in the video. Note that a transmittance may be set for the movable objects. For example, if a transmittance that blends with the background is set within a range greater than 0% and less than 100% for movable objects located farther away from the viewpoint, the movement of the movable objects can be prevented from giving the user a sense of discomfort due to a decrease in visibility. Also, movable objects in a predetermined area including the virtual display may be made transparent. By doing so, although there may be a sense of discomfort with respect to the movement of the movable objects, there is an advantage that the visibility of the virtual display can always be ensured.

[0051] In this embodiment, the object video is prepared with a lower resolution than the background image and the information-providing image. By doing so, the data volume of the object video can be suppressed. Not only the resolution but also the frame rate may be lowered. In this embodiment, the object video is prepared limited to the range indicated by the area A in the figure. This is because, since the people passing by are used as the movable objects, there are no movable objects in the upper space. By doing so, the data volume of the object video can be further reduced. Instead of the method of preparing the object video limited only to the area A, a method may be taken in which the entire object video is prepared as white monochromatic data or the like in areas other than the area A, and then a transmittance that becomes completely transparent is set for the entire area other than the area A.

[0052] Fig. 6(c) shows the order in which each image is superimposed. The arrow indicates the side closer to the viewpoint from the background. In this embodiment, starting from the background side towards the front, the venue image, booth image, information-providing image, and object video are superimposed in this order. By doing so, the information-providing image is displayed in front of the background images (venue image, booth image), and the user can view the information-providing image without discomfort. Also, the object video is displayed in front of the background images (venue image, booth image), and the user can view the state where the movable object is moving within the virtual space. Also, in the example of Fig. 6(c), the movable object is displayed in front of the information-providing image. By doing so, for example, in Fig. 6(a), the movable object can be viewed without discomfort as it crosses the front of the panel. If the order of the information-providing image and the object image is swapped, the movable object may be displayed as if it slips between the panel and the information-providing image displayed thereon, which may result in a very uncomfortable image. In this embodiment, by setting the order as shown in Fig. 6(c), such discomfort is avoided. The superimposition order shown in Fig. 6(c) is just an example. For example, it may be set such that the information-providing image is displayed in front of the movable object. Note that the user may be able to set the display / non-display of the object video and the information-providing image, as well as the superimposition order, etc.

[0053] Fig. 7 is an explanatory diagram showing a display example with an object video. Fig. 7(a) is a display example without an object video. A promotion video is displayed on the panel as the information-providing video PV. Person P1 is displayed as a still image instead of a video. Fig. 7(b) shows a state where an object video is superimposed. Videos of people such as the movable object MO10 walking are displayed. By displaying people walking like this, the reality of the virtual exhibition can be improved. The movable object is not limited to a person. FIG. 7(c) shows an example in which, instead of a person, the decoration in the venue is displayed as the movable object MO11. The movable object MO11 is displayed in a floating state up and down, left and right, etc. within the venue. By displaying the movable object MO11 as a decoration in this way, the reality can be improved and the promotional effect of a booth or the like can also be enhanced. In addition to this, various other things can be displayed as the movable object.

[0054] D. Display control process for users: Next, the process for realizing the display exemplified in FIGS. 6 and 7 will be described. FIG. 8 is a flowchart of the display control process for users. It is a process executed by the user display control unit 131 (see FIG. 1) of the virtual exhibition system, and hardware-wise, it is a process executed by the CPU of the server 100 of the virtual exhibition system.

[0055] When the process starts, the server 100 inputs the user's visual field information (step S11). In this embodiment, as the visual field information, the position information specified by the user is input. The server 100 generates a venue image and a booth image according to the position information (step S12). High-resolution 360-degree equirectangular images are prepared respectively. The booth image can be generated by perspective projection of the 3D model of the booth. In this embodiment, since the viewpoints are limited to certain fixed positions in advance, the venue image and the booth image may be generated for each viewpoint in advance and stored in advance. In this case, the process of step S12 can be achieved by reading out the data corresponding to the position information.

[0056] Next, the server 100 prepares video data for the information-providing image (step S13). The pre-stored video data may be read out.

[0057] The server 100 also prepares object video data (step S14). In this embodiment, the video data prepared for each viewpoint is read out. When the three-dimensional model of the movable object and its movement are stored as data, as the process of step S14, the movable object may be moved in the virtual space, and object video data may be generated by perspective projection. In this case, although there is a load for generating the video, there is an advantage that it is easy to prepare object videos of various variations.

[0058] When each data can be prepared in this way, the server 100 transmits each data to the user terminal 300 (step S15). As will be described later, the user terminal 300 will receive and display this data.

[0059] However, the transmission of these data does not always proceed smoothly. Therefore, in this embodiment, it is determined whether the communication speed exceeds the reference speed (step S16). If the reference speed is not reached, it is determined that there is an obstacle to the display on the user terminal 300, and alternative still image data of the object video is prepared (step S17), and the alternative still image data is transmitted to the user terminal (step S18). The alternative still image may be any frame of the object video cut out.

[0060] By doing so, the data capacity to be transmitted to the user terminal 300 can be suppressed, and the display on the user terminal 300 can be performed smoothly. The reference speed in the above process can be arbitrarily determined. Instead of the communication speed, for example, based on the data capacity of the data that has been transmitted to the user terminal and the required time, the time required to transmit the data to be transmitted in step S11 is estimated, and it may be determined whether this required time exceeds a predetermined reference time. Also, when transmitting the alternative still image data of the object video, the transmission of the object video may be stopped or may continue. When continuing, after the transmission and reception of the object video are completed, the object video can be displayed, so there is an advantage that the reality of the virtual exhibition can be improved.

[0061] FIG. 9 is a flowchart of image display processing. This is processing for performing display on the user terminal 300 based on the data sent from the virtual exhibition system according to FIG. 8. This processing is mainly executed by the application 313 of the user terminal 300, and is a process that the CPU of the user terminal 300 repeatedly executes in terms of hardware.

[0062] When starting the process, the user terminal 300 inputs the line-of-sight direction (step S21). The line-of-sight direction may be indicated by an operation of the user terminal 300, or the movement of the goggles 301 may be detected to input the line-of-sight direction. Also, it may be possible to input the viewing angle, so-called zoom, according to the line-of-sight direction.

[0063] Next, the user terminal 300 performs an affine transformation on the video data according to the line-of-sight direction (step S22). By doing so, the video data is converted into an image in a state where it is displayed at a predetermined position such as the wall or panel of the booth in the virtual space.

[0064] Then, the user terminal 300 determines whether the object video is in a state where it can be displayed (step S23). If the data of the object video is cached to the extent that it can be displayed as a video, it is determined that it can be displayed.

[0065] When the object video can be displayed, the user terminal 300 superimposes and displays the venue image, booth image, information-providing image, and object video (step S25). By doing so, an image in the state shown in FIG. 6(c) can be displayed.

[0066] On the other hand, when the object video cannot be displayed, the user terminal 300 superimposes and displays an alternative still image of the object video together with the venue image, booth image, and information-providing image (step S24). Also in this case, by displaying the alternative still image instead of the object video, people moving in the virtual space and the like can be displayed as still images, and it becomes possible to make the user feel the atmosphere of the virtual exhibition to some extent.

[0067] The user terminal 300 repeatedly executes the above processing. Therefore, if the user changes the line-of-sight direction, corresponding display can be realized. Also, even if an alternative still image is displayed while the cache of the object video is not completed (step S24), after the cache is completed, it can be switched to the display of the object video (step S25).

[0068] FIG. 10 is an explanatory diagram showing a reproduction mode of an object video. FIG. 10(a) shows an example in which an alternative still image is initially displayed, and then the object video is displayed after the cache of the object video is completed. As shown in the figure, the object video is repeatedly displayed. Note that when communication is performed smoothly, the object video may be repeatedly displayed without displaying the first alternative still image.

[0069] FIG. 10(b) is an example in which, similar to FIG. 10(a), an alternative still image is displayed and then the object video is repeatedly displayed, but illustrates a case where fade-in and fade-out effects are applied to the object video for reproduction. When simply repeatedly displaying the object video as in FIG. 10(a), there is a possibility of giving a sense of discomfort such that the movable object suddenly disappears and appears at another location at the boundary where the reproduction ends and the next reproduction starts. In contrast, such a sense of discomfort can be suppressed by applying fade-in and fade-out effects.

[0070] FIG. 10(c) shows an example in which, compared to FIG. 10(b), the fade-in and fade-out of the object video to be reproduced are overlapped and cross-faded for reproduction. By doing so, the sense of discomfort at the boundary of the object video can be further reduced.

[0071] Figure 10(d) shows an example of selectively using multiple types of object videos. In this example, after the alternative still image, object video 1 is played, and then object video 2 is played. At the transition between the two, an example of cross-fading is shown, but it may be simply joined as in Figure 10(a), or only fade-in and fade-out effects may be applied as in Figure 10(b). By using multiple types of object videos as shown in Figure 10(d), there is an advantage that boredom with repetitive object videos can be suppressed, and it becomes easier to maintain the reality of the exhibition. The object videos are not limited to two types, and even more types may be selectively used.

[0072] In Figures 8 and 9, examples of superimposing object videos on the user terminal 300 were shown, but as shown below as a modification example, these processes may be performed on the server 100. Figure 11 is a flowchart of the display control process for the user as a modification example. When the process starts, the server 100 inputs the user's visual field information (step S31), generates a venue image and a booth image (step S32), and generates an information-providing image (step S33). The processes up to this point are the same as steps S11 to S13 in Figure 8. Next, the server 100 determines whether the communication speed with the user terminal 300 exceeds the reference speed (step S34). If it exceeds the reference speed, the object video is played (step S36). On the other hand, if it does not exceed the reference speed, alternative still image data of the object video is generated (step S35). Then, the server 100 superimposes these data and transmits them to the user terminal (step S37). However, each data is not transmitted as a single synthesized video, but as data of each layer synchronized so as to be finally superimposed and displayed, as shown in Figure 6(c). In the embodiment (Figures 8 and 9), the object video is transmitted and received separately from the background image, and the user terminal 300 reads and superimposes them separately, whereas in the modification example, they are transmitted in a synchronized state. Even with such a modification example, the same display as in the embodiment can be realized. According to the modification example, there is also an advantage that the load on the user terminal 300 is lighter than in the embodiment. In the modification example (FIG. 11), in order to reduce the load of the superimposing process, the server 100 can also take a mode of superimposing and then transmitting to the user terminal.

[0073] E. Effects and Modification Examples: According to the virtual display system of the embodiment described above, by superimposing and displaying an object video on the virtual display, the reality of the virtual display can be improved and the user's sense of immersion can be enhanced. Also, by setting the still image representing the virtual space and the virtual display to high resolution and the object video to a lower resolution than that, such a display can be realized while suppressing the data volume.

[0074] The various features described above do not necessarily have to include all of them, and some may be omitted or combined to form a configuration. Also, the present invention can form various modification examples other than the system for holding an exhibition in a virtual space. For example, if the scenery of a tourist destination etc. is configured as a virtual display instead of a booth, it can be configured as a system for sightseeing within a virtual space. In addition to this, it is possible to provide various virtual displays such as art museums and museums.

Industrial Applicability

[0075] The present invention can be used to present a virtual space to a user.

Explanation of Reference Numerals

[0076] 100 Server 101 Virtual Data Storage Unit 102 User Data Storage Unit 103 Exhibitor Data Storage Unit 104 Material Data Storage Unit 111 Virtual Space Allocation Unit 112 Data Management Unit 113 Transmission / Reception Unit 121 Exhibitor Display Control Unit 122 Guide Information Input Unit 123 Information Provision Unit 124 Push-Type Guide Unit 131 User Display Control Unit 132 Position Setting Unit 133 Visual Field Information Input Unit 134 Material Distribution Unit 200 Exhibitor Terminal 201 Google Glass 211 Transmission / Reception Unit 212 Input / Output Unit 213 Application 300 User Terminal 301 Google Glass 311 Transmission / Reception Unit 312 Input / Output Unit 313 Application

Claims

1. A virtual display system for electronically performing a virtual display in a virtual space, comprising: a virtual data storage unit that stores virtual data representing the virtual space and the virtual display, movable objects whose positions in the virtual space can move, and object data representing the movement thereof; a viewing information input unit that inputs viewing information for specifying the viewing field of a user viewing the virtual display in the virtual space; a user display control unit that, based on the viewing information, uses the virtual data and the object data to display an image of the virtual display on a user output device used by the user; wherein the user display control unit: displays a background image as a still image representing the virtual space and the virtual display based on the virtual data, and a virtual display system that superimposes and displays an object video prepared at a resolution lower than that of the background image on the front of the background image in a state where other than the movable object is transparent, based on the object data.

2. The virtual display system according to claim 1, wherein the user display control unit: transmits data for displaying the background image and data for displaying the object video to the user output device respectively, and causes the user output device to perform the superimposed display.

3. The virtual display system according to claim 1, wherein the user display control unit limits the display of the object video to a predetermined area where the movable object exists in the virtual space.

4. The virtual display system according to claim 1, wherein the user display control unit repeatedly displays the object video having a predetermined length.

5. The virtual display system according to claim 1, wherein the virtual data storage unit stores a plurality of types of the object data, and the user display control unit switches any of the object data and displays the object video.

6. The virtual display system according to claim 4 or 5, wherein the user display control unit performs one or both of fade-in and fade-out on the object video for display.

7. The virtual display system according to claim 1, A virtual display system in which the user display control unit realizes a display in a state where portions other than the movable object are transparent by setting an alpha channel representing the portions to be transparent in data for displaying the object video.

8. The virtual display system according to claim 1, wherein the virtual data storage unit further stores video data for displaying an information-providing video for providing information to the user, and the user display control unit displays the information-providing video at a predetermined position within the virtual space on the back side of the object video.

9. The virtual display system according to claim 1, wherein the user display control unit stops displaying the object video when the object video cannot be normally displayed on the user output device.

10. The virtual display system according to claim 1, wherein the position of the user within the virtual space is restricted to a preset default position.

11. A virtual display method for performing a virtual display electronically configured in a virtual space, including steps implemented by a computer: storing the virtual space, virtual data representing the virtual display, a movable object whose position within the virtual space can move, and object data representing the movement thereof; inputting view information for specifying the view of the user within the virtual space who views the virtual display; and a user display control step of displaying an image of the virtual display on a user output device used by the user using the virtual data and the object data based on the view information, wherein the user display control step displays a background image as a still image representing the virtual space and the virtual display based on the virtual data, and superimposes and displays, in front of the background image, an object video prepared at a lower resolution than the background image based on the object data in a state where portions other than the movable object are transparent. Virtual display method.

Images