Virtual exhibition system

Abstract

To provide a virtual exhibition system and method which enhances reality of an exhibition site and enhances immersive feeling of users at virtual exhibition in a virtual space.SOLUTION: A virtual exhibition system allows a goggle 301 worn by a user to display an image of a virtual space on the basis of data transmitted from a server 100. The server prepares, with high resolution still images, venue images representing a virtual space being a venue of virtual exhibition, and booth images representing booths of exhibitors, and also prepares, with high resolution moving images, information providing images such as promotion moving images to be displayed to users at the booths. Besides these images, the server prepares, as low resolution moving images, object moving images representing moving objects moving in the virtual space, for example, moving images of people passing through the venue. These pieces of data are transmitted to a user terminal 300, and by allowing the user terminal to display these pieces of data while superposing them, the reality of the virtual exhibition is enhanced such that the people are passing by.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a virtual exhibition system that presents a virtual space to a user. [Background technology]

[0002] Recently, technology for holding exhibitions and other events in virtual spaces has been attracting attention. Patent Document 1 discloses a system that presents data in a virtual space to a viewing user in a viewable manner using a three-dimensional viewer such as a head-mounted display. The viewing user can view virtual exhibits while moving around in the virtual space. If the viewing user requests it, an explanation of the virtual exhibits is displayed in text or output as audio. Patent Document 2 discloses a technology for realizing a virtual space in which an object is placed at an arbitrary position by using a celestial sphere panoramic image representing the virtual space as a background image and arranging an image in which a three-dimensional model in which an object is placed within a virtual section is projected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-133109 [Patent Document 2] Patent No. 6852295 Summary of the Invention [Problem to be solved by the invention]

[0004] When creating a virtual exhibition in a virtual space, it is important to enhance the sense of reality and provide a sense of immersion to users who visit the exhibition. In real-world exhibitions, the lively atmosphere of the venue, with many people coming and going, can have the secondary effect of increasing interest in the exhibition. Increasing the sense of reality and immersion is also expected to have this secondary effect in virtual exhibitions. However, conventional virtual exhibition systems have not yet given sufficient consideration to improving realism and immersion. Even with conventional technology, if the background image showing the entire venue were prepared as a video showing a large number of people coming and going, rather than a still image, it would be possible to create an atmosphere close to that of a real venue. However, in this case, it would be necessary to distribute the video to the terminals used by users, which would cause problems with communication speed and traffic. Conversely, if the video resolution were set low to take communication speed and other factors into consideration, the entire venue would appear grainy, which could actually reduce the realism. This problem is not limited to cases where an exhibition is held in a virtual space, but is a common problem when a virtual space is exhibited to users for various purposes, such as sightseeing or other nature observation, or touring art galleries, museums, event venues, factories, and other facilities. The present invention aims to solve this problem and provide technology that improves the realism and immersion when exhibiting a virtual space. [Means for solving the problem]

[0005] The present invention provides A virtual exhibition system that displays a virtual exhibition electronically constructed in a virtual space, a virtual data storage unit that stores virtual data representing the virtual space and the virtual exhibit, as well as object data representing movable objects whose positions can be moved within the virtual space and their movements; a view information input unit for inputting view information that specifies a view in the virtual space of a user viewing the virtual exhibit; a user display control unit that uses the virtual data and the object data based on the view information to display an image of the virtual exhibit on a user output device used by the user; The user display control unit a background image as a still image representing the virtual space and the virtual exhibit based on the virtual data; The virtual exhibition system can display an object video prepared based on the object data at a lower resolution than the background image, superimposed on top of the background image while allowing everything except the movable object to be seen.

[0006] According to the present invention, by displaying images of a virtual space and a virtual exhibit, a user can perceive the virtual exhibit as being viewed within the virtual space. Here, in the present invention, a background image representing the virtual space or virtual exhibit is prepared as a still image, and an object video representing a movable object is superimposed on top of the background image. Examples of movable objects include people and characters moving through the virtual space, balloons floating within the virtual space, and other floating objects. This improves the realism of the virtual space as perceived by the user, providing a sense of immersion for the user. Furthermore, in the present invention, the background image is prepared in high resolution and the object video is prepared in low resolution, which reduces the amount of data sent to the user output device while avoiding degradation of the image quality of the virtual space and virtual exhibit. The resolution of the background image and object video is relative, and the specific resolution can be determined arbitrarily based on the image quality and the communication speed to the output device for the user.

[0007] In the present invention, the field of view information may be the position of the viewpoint, the line of sight direction, the angle of view, etc. in the virtual space. Only a part of these may be input. For example, a virtual exhibition system may input the position of the viewpoint as field of view information, and the user output device may specify the line of sight direction, the angle of view, etc. and perform display. Furthermore, some of the information may be preset fixed values. In the present invention, the virtual space and virtual exhibit may be two-dimensional, but by making them three-dimensional, a more realistic exhibit can be created. The virtual space and virtual exhibit may also be provided using CG (computer graphics), or by attaching textures of still images or videos created using CG of real exhibits. The virtual space and virtual exhibit may also be provided in various other ways. The virtual exhibits may be those that explain the contents of products or services, such as exhibitions, as well as various other contents such as scenery of tourist spots, art museums, expositions, and event venues. A movable object is an object whose position can move within a virtual space. In addition to the previously described people moving around within the virtual space, any movable object can be used, such as animals, birds, rain, snow, clouds, waterfalls, and flowing rivers. Also, even if the position of a tree itself does not move, such as when a tree sways in the wind, if its branches and leaves move, the entire tree can be considered a movable object.

[0008] The user output device may be a display of a personal computer, tablet, smartphone, or the like, or may be goggles or glasses worn on the head, etc. In particular, the use of goggles or glasses has the advantage of further enhancing the sense of realism.

[0009] In the present invention, The user display control unit Data for displaying the background image and data for displaying the object moving image may be transmitted to the user output device, respectively, and the user output device may be caused to perform the superimposed display.

[0010] In this mode, the background image and the object video are transmitted separately to the user output device and then superimposed on each other. This has the advantage that the user output device can easily cache the object video and use it repeatedly, cache multiple types of object video and use them separately, or display a still image in place of the object video when the communication state is unstable. However, the display in the present invention does not exclude a mode in which a background image and an object image are superimposed on each other and distributed to a user output device for display in a virtual exhibition system as a set of moving image data.

[0011] In the present invention, The user display control unit may display the object video in a limited area within the virtual space where the movable object exists.

[0012] By doing this, the amount of data for the object video can be further reduced. The predetermined area can be determined arbitrarily depending on the movable object. For example, if people passing by are used as movable objects, object video can be prepared by omitting space above a certain height in the virtual space. Possible methods for this include preparing object video data limited to the predetermined area, or preparing dummy data for areas other than the predetermined area, all of which can be set to be completely transparent.

[0013] In the present invention, The user display control section may repeatedly display the object moving image for a predetermined length.

[0014] By doing so, the data volume of the object video can be further reduced.

[0015] In the present invention, the virtual data storage unit stores a plurality of types of the object data, The user display control section may switch any one of the object data to display the object moving image.

[0016] In this aspect, since it is possible to switch between displaying a plurality of types of object videos, it is possible to prevent the user from becoming accustomed to or bored with the object videos, and it is possible to maintain realism. The types of object videos to be prepared can be determined arbitrarily. The multiple types of object videos do not need to be the same length, and may be of different lengths. The selection can also be done in various ways. The selection may be made in a predetermined order or according to a predetermined rule, or may be made randomly.

[0017] In either case where an object video is used repeatedly or where multiple types of object video are selected and used, it is not necessary to play the object video from the beginning, and the part to be played may be selected as appropriate, such as playing it from the middle.

[0018] In addition, in both the mode in which an object video is repeatedly used and the mode in which a plurality of types of object video are selected and used, The user display control section may display the object moving image by applying one or both of a fade-in and a fade-out to the object moving image.

[0019] This reduces the awkwardness felt when one video ends and the next video starts. Note that fade-in and fade-out are not essential, and either one may be applied, or neither may be applied. If neither fade-in nor fade-out is applied, for example, the last frame and the first frame of a video may be prepared to form a continuous image, thereby reducing the sense of incongruity at the joint.

[0020] In the present invention, The user display control unit may set an alpha channel representing the portion to be transparent in the data for displaying the object video, thereby realizing a display in which everything other than the movable object is transparent.

[0021] There are various methods for making part of the object video transparent so that the background image can be seen. One of the methods described above is to set an alpha channel as transparency information in the object video itself. This has the advantage that the object video and the transparency information can be integrated, preventing misalignment between them. However, the transparency is not limited to this method. For example, the location to be transparent may be set and mask data may be prepared separately from the object video. Since the location to be transparent will also move as the movable object moves, the mask data will also be prepared as time-series data. In this case, the object video and mask data may be forcibly synchronized each time a discrepancy of a predetermined amount or more occurs between them.

[0022] In the present invention, a transparency may be set for a movable object. For example, if a transparency is set for a movable object located farther away from the viewpoint, in a range greater than 0% and less than 100%, so that the object appears to blend with the background, the visibility is reduced, thereby reducing the possibility that the movement of the movable object will cause discomfort to the user. Furthermore, movable objects located in a predetermined area including the virtual exhibit may be made transparent. While this may cause discomfort due to the movement of the movable object, it has the advantage of always ensuring the visibility of the virtual exhibit.

[0023] In the present invention, The virtual data storage unit further stores video data for displaying an information providing video that provides information to the user, The user display control section may display the informational video at a predetermined position in the virtual space, behind the object video.

[0024] Examples of information providing videos include videos displayed to users in booths in virtual exhibitions to present products, etc., and videos that deliver events, venue layouts, programs, etc. within virtual spaces. One way to display these informational videos in virtual space is to play the informational videos separately from the background image, apply affine transformation to them, and paste them in a display position that corresponds to the user's field of view. By doing this, it is possible to realize, for example, an informational video being displayed as if it were on a panel in a booth or on a guide board in a venue. However, in the present invention, since an object video exists, if this is superimposed between the background image and the informational video, it may result in a very unnatural display, as if the movable object is sandwiched between a wall and a panel. Therefore, by displaying the informational video behind the object video, as in the above embodiment, it is possible to prevent this unnatural display.

[0025] In the present invention, The user display control section may stop display of the object moving image when the object moving image cannot be displayed normally on the user output device.

[0026] Examples of situations in which the object video cannot be displayed normally include when the communication speed between the virtual exhibition system and the user output device is slow, when the data for the object video cannot be transmitted smoothly due to communication problems, when the data has not yet been sufficiently cached, etc. According to the above aspect, in such cases, the display of the object video is stopped, thereby avoiding situations such as delays in display until the data for the object video has been cached to a certain extent on the user output device, or display disruptions due to communication problems for the object video. Display of the object video may be stopped temporarily until caching is completed in the user output device, or display of the object video may be completely abandoned. Furthermore, when object animation is not displayed, only the virtual space and virtual exhibits may be displayed, or still images showing movable objects may be displayed instead of object animation.

[0027] In the present invention, The position of the user in the virtual space may be restricted to a preset position.

[0028] In this way, the virtual exhibition can be efficiently provided and the user can efficiently view it. In the above aspect, the view information at the specified position may be arbitrary, or the view information may be restricted to only the front.

[0029] The present invention does not necessarily have to include all of the above-described features, and some of them may be omitted or combined as appropriate. The above-described virtual exhibition may be configured as a virtual exhibition method executed by a computer, or as a computer program for causing a computer to perform such a method. Furthermore, the present invention may be configured as a computer-readable recording medium on which such a computer program is recorded. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is an explanatory diagram showing the configuration of a virtual exhibition system in an embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating a data structure. [Figure 3] FIG. 10 is an explanatory diagram showing a method for generating an image to be displayed. [Figure 4] FIG. 1 is an explanatory diagram showing an example image (1) displayed to the user. [Figure 5] FIG. 10 is an explanatory diagram showing an example image (2) displayed to the user. [Figure 6] FIG. 10 is an explanatory diagram showing a method for displaying an object moving image. [Figure 7] FIG. 10 is an explanatory diagram showing a display example with an object moving image. [Figure 8]10 is a flowchart of a user display control process. [Figure 9] 10 is a flowchart of an image display process. [Figure 10] FIG. 10 is an explanatory diagram showing a playback mode of an object moving image. [Figure 11] 10 is a flowchart of a modified example of a user display control process. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, an embodiment of the present invention will be described in the following sections, taking as an example a case where an exhibition is held in a virtual space. A. System Configuration: B. Display of Virtual Spaces and Virtual Exhibits: C. Displaying object videos D. Display control process for users: E. Effects and Modifications:

[0032] A. System Configuration: 1 is an explanatory diagram showing the configuration of a virtual exhibition system according to an embodiment of the present invention, in which multiple exhibitors set up virtual booths in a virtual space and provide users with virtual exhibitions of their products or services. The virtual exhibition system is composed of a server 100 that provides its main functions, exhibitor terminals 200 used by exhibitors at each booth, and user terminals 300 used by users visiting the booths, all connected via a network NE such as the Internet. The exhibitor terminals 200 and user terminals 300 can be various devices that can be connected to a network NE, such as personal computers, tablets, and smartphones. The functions of the server 100 may also be provided by multiple devices. Multiple exhibitor terminals 200 and user terminals 300 can also be connected.

[0033] The exhibitor terminal 200 and the user terminal are provided with the functions shown in the figure. The transmitter / receiver 211 and the transmitter / receiver 311 function to transmit and receive information via the network NE. The information includes text, images, videos, and sounds. It also includes various instructions from exhibitors and users to the virtual exhibition system. The input / output units 212 and 312 control devices that input and output information. In this embodiment, the exhibitor's operator wears goggles 201, and the user also wears goggles 301. These goggles 201 and 301 display various images and output audio provided by the virtual exhibition system. They also have built-in microphones, allowing audio input. When the operator makes input to the virtual exhibition system, they use devices such as a keyboard and mouse on the exhibitor terminal 200. When the user makes input, they use the touch panel on the user terminal 300. Input and output are not limited to the above example, and various other modes are possible. The application 213 and the application 313 provide functions such as performing the above-mentioned input and output based on information transmitted from the virtual exhibition system. They may be configured as software specific to the virtual exhibition system, or general-purpose software such as a browser may be used. In this embodiment, as will be described later, a background image (still image) representing the virtual space and virtual exhibition is superimposed with an object video or the like representing a movable object moving within the virtual space. The applications 213 and 313 receive data such as the background image and the object video, generate an image in which these are superimposed, and display it on the goggles 201 and 301 via the input / output units 212 and 312.

[0034] The server 100 is provided with the functions shown in the figure. The virtual data storage unit 101 stores various data (hereinafter sometimes collectively referred to as "virtual data") for realizing a virtual space and a virtual exhibition. In this embodiment, the virtual data representing the entire venue where the virtual exhibition is held is prepared by the organizer of the virtual exhibition or the operator of the virtual exhibition system. The virtual data representing each booth is prepared by the exhibitor. Virtual data includes both still images and videos. For example, a still image representing a building or the like that will serve as the venue for a virtual exhibition can be an image that corresponds to the entire celestial sphere, a so-called 360-degree equirectangular image, and can be prepared using computer graphics or live action. Images of virtual exhibitions such as booths can be generated, for example, by preparing a three-dimensional model and projecting it. Images of virtual exhibitions may be integrated with images representing the venue. In addition, for example, two-dimensional informational videos that provide users with information on events, venue layouts, various programs, etc. at the venue, and informational videos (so-called promotional videos) that provide information on products, etc. at each booth may be prepared. The user data storage unit 102 stores information about users who visit each booth. The exhibitor data storage unit 103 stores information about the exhibitors who will be exhibiting at each booth. The material data storage unit 104 stores data such as materials and videos to be distributed to users at each booth. The specific contents and structures of these data will be described later.

[0035] The virtual space allocation unit 111 allocates a virtual space to a user. In this embodiment, when a user visits a booth, the virtual space is set as a space unique to that user. This allows the user to freely look around the booths and to communicate with the exhibitor's operators as if the user were the only person present in the booth. The data management unit 112 manages the input and output of data to and from the virtual data storage unit 101 , the user data storage unit 102 , the exhibitor data storage unit 103 , and the material data storage unit 104 . The transmitting / receiving unit 113 transmits and receives information to and from the exhibitor terminal 200 and the user terminal 300 via the network NE.

[0036] The exhibitor display control unit 121 generates and transmits image data to be displayed on the exhibitor terminal 200. Information for generating an image may be transmitted to the exhibitor terminal 200, causing the exhibitor terminal 200 to generate the image data. The guide information input unit 122 inputs guide information for introducing products or services from the exhibitor terminal 200. The operator can input the guide information in the form of text, voice, etc. at the exhibitor terminal 200. The input guide information is sent to the user terminal 300, thereby realizing communication between the operator and the user. The information providing unit 123 provides various information regarding the performance of the exhibit to the exhibitor. For example, the information includes the time spent by users at the booth, the number of users who visited the booth, and the types of materials acquired by users at the booth. The information providing unit 123 may also include the results of an analysis based on this information. The push-type guidance unit 124 has a function that allows the operator to set the user's position, line of sight, etc. This allows the operator to guide the user around the booth as if he or she were taking them there. Also, if the operator is unable to respond, the server 100 may automatically guide the user according to pre-prepared data.

[0037] The user display control unit 131 generates and transmits image data (hereinafter referred to as "user image") to be displayed on the user terminal 300. Information for generating an image may be transmitted to the user terminal 300 to cause the image data to be generated. The position setting unit 132 sets the position of the user in the virtual space. In this embodiment, positions to which the user can move are set in advance, and the user can move within the virtual space by selecting one of these positions. Alternatively, the user may be allowed to move freely within the virtual space. The view information input unit 133 inputs view information from the user terminal 300. View information is information such as the viewpoint position, line of sight direction, and angle of view used to generate a user image. In this embodiment, the user's viewpoint position, i.e., user position information, is input as view information, and display data is prepared regardless of the line of sight direction or angle of view. The line of sight direction and angle of view are input at the exhibitor terminal 200 and the user terminal 300, and display data corresponding to these is generated and displayed on the goggles 201 and 301. The material distribution unit 134 transmits data of materials distributed in the booth in accordance with instructions from the user. The data of the materials is downloaded to the user terminal 300 or the like.

[0038] The virtual exhibition system is not limited to the above configuration and can have various other configurations. It is not necessary to have all of the functions shown in the figure, and some may be omitted. In addition, the virtual exhibition system may have functions other than those shown in the figure. Although the above-described functions are implemented as software in this embodiment, some or all of them may be implemented as hardware.

[0039] FIG. 2 is an explanatory diagram showing the data structure. The exhibitor data storage unit 103 assigns an exhibitor ID, which is an identifier, to each exhibitor, and stores exhibitor information such as exhibitor name, contact details, virtual data, invitees, visitors, etc. For the sake of convenience in data structure, an exhibitor ID is also assigned to the organizer of the virtual exhibition or the operator of the virtual exhibition system. The virtual data information indicates the location of the 3D model, photos, videos, textures, and other data that make up the exhibitor's booth. In this embodiment, the virtual data is stored in the virtual data storage unit 101 with a virtual data ID, which is an identifier, so the virtual data ID can be stored as the virtual data information. The guest information is information about users invited by the exhibitor to their booth. As will be explained later, each user is assigned a user ID, which serves as an identifier, so the guest information can be stored as the user ID. Visitor information is information about users who visited the booth. As with invitees, the user ID is stored.

[0040] The user data storage unit 102 stores user information such as name, contact details, visit log, material log, and group for each user ID. A visit log is a history of booths visited. It can record the time you visited a booth, the time you left, etc. Also, a booth can be identified by its exhibitor ID. The material log is a history of materials acquired by downloading. It can be recorded using a unique ID attached to the material. The group information stores the user IDs of the user and the other users who make up the group. In this embodiment, when a user visits a booth, the booth is assigned as a virtual space specific to the user. However, by specifying a group, it is also possible to assign a virtual space specific to the group. It is preferable that the group information be editable by the user as needed.

[0041] The virtual data storage unit 101 stores virtual data, object data, and video data. For each type of data, there are data prepared by the organizer of the virtual exhibition or the operator of the virtual exhibition system, and data prepared by each exhibitor. The entity that prepared the data is identified by an "exhibitor ID." Virtual data is data that makes up a virtual space and a virtual exhibition. The virtual data prepared by organizers and others includes 3D models, photos, videos, textures, etc. that represent the virtual space of the entire venue where the virtual exhibition will be held. The virtual data prepared by each exhibitor includes booth data, etc. Each piece of virtual data is stored with a virtual data ID. If an exhibitor exhibits multiple booths, multiple virtual data IDs will be assigned. The same applies when exhibitors prepare multiple types of virtual data, such as virtual data for the venue according to time periods such as morning, noon, and night. As virtual data information, booth data, exhibit data, viewpoint data, guide data, viewing log, etc. are stored. Booth data includes data such as 3D models, photos, videos, and textures that make up the booth. Booths may also be represented using photos and videos of actual booths. Exhibit data includes 3D data, 2D image data, video data, etc. of exhibits in the booth. In addition to providing CG of the exhibits, it may also be provided by pasting photographs or videos of the exhibits. The viewpoint data stores the positions to which the user can move within the booth. In this embodiment, as will be described later, the user is not allowed to move completely freely within the booth, but can selectively move to pre-set positions. The viewpoint data is data that represents these positions to which the user can move. The guide data is three-dimensional image data, so-called avatar data, that represents a person guiding a user in a booth. In this embodiment, the guide is not free to move around the booth, but is fixed in a specific position. The guide data also stores information about the position where the guide is to be displayed. The viewing log is a history of users who have visited the booth. The time of visit and the time of departure are recorded for each user. The content of communication between the user and the operator may also be recorded. In addition, data may be prepared to automatically guide users to the booth when an operator is not present, such as voice guidance data, data to control the user's viewpoint and line of sight, and data to instruct the user to enlarge panels in the booth or play videos. It is not necessary to prepare all of the data exemplified here, and some may be omitted as appropriate.

[0042] Object data is object data that represents movable objects whose position can move within a virtual space and their movements. As with virtual data, each piece of object data is assigned an object data ID as an identifier, and also an exhibitor ID that represents the entity that prepared the data. The movable object data is prepared as data representing a movable object and its movement. In this embodiment, the movable object data is prepared as video data of a predetermined length showing the movable object moving within a virtual space. For example, CG or live-action video representing people moving through the virtual space can be used. Alternatively to the above embodiment, the movable object data may be prepared as, for example, a three-dimensional model representing the movable object and data defining its movement. The viewpoint data is data that represents a viewpoint position corresponding to the movable object data. In this embodiment, as described above, the positions to which the user can move are limited, and movable object data is prepared for each viewpoint, so data for associating the viewpoint data with the user's position is prepared.

[0043] Video data is informational video data for providing information to users, such as videos that provide event information in a virtual space, or promotional videos that provide product information in a booth. Like virtual data, video data is assigned a video data ID as an identifier, and also an exhibitor ID that represents the entity that prepared the data.Video data can be prepared, for example, as two-dimensional video data.Display position data is information about the position in the virtual space where the informational video is displayed.

[0044] The virtual space allocation data is data generated by the virtual space allocation unit 111 when a user visits a booth. As shown in the figure, a virtual data ID that identifies the booth is associated with an exhibitor ID that identifies the exhibitor, and a user ID that identifies the user. When users visit in groups, multiple user IDs in the group are associated. Questions from users are conveyed only to the operator of the associated exhibitor ID, and conversely, the operator's responses are conveyed only to the associated user. This allows users to communicate privately with the operator without worrying about their information being leaked to other users. Virtual space allocation data is generated for each user or group. Virtual data IDs can be duplicated. This allows booth exhibits to be provided to multiple users, while still providing each user with a unique virtual space. When multiple users visit a booth, multiple virtual space allocation data are set for one exhibitor ID. In such cases, the operator selects one of the visiting users to communicate with. The conversation flag is a flag that indicates whether or not the user can converse with the operator. When the operator selects one of the visiting users, the conversation flag in the virtual space allocation data for that user is turned on, and the conversation flags in the other virtual space allocation data are turned off. This makes it possible to achieve the closed communication described above even when multiple users are visiting a booth. If a plurality of exhibitor terminals 200 are prepared, the users may be served by a plurality of operators. In such a case, the flag may be omitted.

[0045] B. Display of Virtual Spaces and Virtual Exhibits: Fig. 3 is an explanatory diagram showing a method for generating the displayed image. As shown in Fig. 3(a), in this embodiment, the virtual space representing the venue is constructed by pasting 360-degree equirectangular images of photos and videos representing the venue onto the inner surface of a spherical virtual space S. As shown in the figure, the exhibitor has created virtual data for Booth B using a three-dimensional model or the like, and the Booth B is placed in a virtual space. The user image can be generated in the virtual space by perspective projection according to field of view information such as the viewpoint VP, line of sight direction VL, and angle of view VA. The viewpoint VP and line of sight direction VL can be determined using three-dimensional coordinates in the virtual space. The angle of view VA may be fixed or may be adjustable by the user. For the sake of convenience in illustration and explanation, in FIG. 3 the viewpoint VP is shown as being located outside the virtual space S of the venue, but in reality the viewpoint VP is located at the center of the virtual space S. Booth B may also be created by combining a photograph of the actual booth or an image generated by perspective projection with a 360-degree equirectangular image of the virtual space. Hereinafter, the images representing the virtual space and the booths may be collectively referred to as background images.

[0046] 3(b) shows an example of a background image in this embodiment. This example illustrates a state in which a 360-degree equirectangular image of a booth generated by perspective projection is synthesized with a 360-degree equirectangular image representing an exhibition hall. By pasting this image onto the inner surface of a celestial sphere, the virtual space and virtual exhibits can be viewed from any user's line of sight.

[0047] FIG. 4 is an explanatory diagram showing an example image (1) displayed to the user. An image IM0 of the venue and an image IM1 depicting a booth in three dimensions are displayed. Also displayed is a guide G guiding the user around the booth. The person PM in the booth is a fixed humanoid model drawn to create a sense of realism. The guide G and the person PM are not objects that move within the virtual space. Therefore, in this embodiment, they are displayed as still images rather than as object animations. Still images of multiple poses may be prepared so that the poses can be changed to appear as if they were an animation. The exhibits hanging on the wall are panels (still images) displayed as still images, videos, etc. This video is one of the information-providing videos provided to users, and is displayed using the video data explained in Figure 2. VP1 drawn on the floor represents the user's adjustable viewpoint position. A floor plan W1 of the booth is displayed in the upper left of the user image. An enlarged view of the floor plan W1 is shown at the top of the figure. In this embodiment, the user is not able to move freely within the booth, but can selectively move to a pre-set viewpoint position. The floor plan W1 illustrates the viewpoint positions and field of view to which the user can move. The movable viewpoint positions are indicated by a camera icon. The user image shown in Figure 4 represents the view as seen from viewpoint position VP0, as in field of view VA. The circular viewpoint position VP1 shown on the floor in the user image corresponds to viewpoint position VP1 shown in the lower right of the floor plan W1. When the user selects one of the camera icons by clicking on it, the viewpoint position moves. The user may select a circular viewpoint position VP1 drawn on the floor in the user image. Figure 5 shows the image when viewpoint position VP2 is selected.

[0048] 5 is an explanatory diagram showing an example image (2) displayed to the user. From the plan view W1 shown in the upper left, it can be seen that the image is of the state in which the viewpoint position VP2 in FIG. 4 is selected and the viewer is looking toward the wall. A display is drawn on the wall, and a video MV is automatically played silently, creating a sense of realism like an exhibition. When the user clicks on icon IC1, the video MV is enlarged and played with sound. The video can also be played over the entire user image. A panel is displayed next to it. When users click on the icon IC2 below the panel, the panel will be enlarged. When users click on the icon IC3, they can download the materials.

[0049] C. Displaying object videos Next, a method for displaying an object moving image in this embodiment will be described. FIG. 6 is an explanatory diagram showing a method for displaying object videos. FIG. 6(a) shows the background image described in FIG. 3. As shown in FIG. 6(c), the background image is composed of a 360-degree equirectangular image showing the entire venue (referred to as a venue image) and images generated by perspectively projecting each booth (referred to as booth images). In this embodiment, both are generated as high-resolution still images. The two may be combined to form a background image, or the two may be displayed superimposed on each other. In addition to these background images, informational images that provide information to users are also prepared, as explained in Figures 4 and 5. In the example of Figure 6(a), the informational image is a promotional video displayed on a panel placed in front of the booth. The informational image is a high-resolution video.

[0050] Figure 6(b) shows an example of an object video. Here, images of people MO1, MO2, etc., as movable objects moving around the virtual exhibit, are used as object videos. To avoid cluttering the illustration, other displayed people also correspond to movable objects, although they are not given reference numbers in the illustration. The object video is a video of these movable objects MO1, MO2, etc., walking around the virtual space. Also, in FIG. 6(b), the object video is created with the object superimposed on the background image, but the background image may be omitted and only the movable objects MO1, MO2, etc. may be used as the object video. As will be described later, object videos are displayed superimposed on a background image. In order to allow the background image to be seen when superimposed, the parts of the object video other than the movable objects are set to be transparent. This setting can be achieved by setting an alpha channel, which determines the transparency, to the video. It is also possible to set a transparency for movable objects. For example, if a transparency is set for a movable object located farther away from the viewpoint, in the range between 0% and 100%, so that the object appears to blend with the background, the visibility is reduced, thereby reducing the possibility that the movement of the movable object will cause discomfort to the user. Also, movable objects located in a specified area including the virtual exhibit may be made transparent. While this may cause discomfort when the movable object moves, it has the advantage of always ensuring the visibility of the virtual exhibit.

[0051] In this embodiment, the object video is prepared at a lower resolution than the background image and the informational image. This reduces the data volume of the object video. In addition to the resolution, the frame rate may also be reduced. In this embodiment, the object video is prepared within a range limited to area A in the figure. This is because the people passing by are treated as movable objects, and no movable objects exist in the space above. This makes it possible to further reduce the data volume of the object video. Note that instead of preparing the object video only for area A, a method may be used in which the entire object video is prepared as monochrome white data for the area other than area A, and a transparency is set to make the entire area other than area A completely transparent.

[0052] Figure 6(c) shows the order in which each image is superimposed. The arrow indicates the side closer to the viewpoint from the background. In this embodiment, the venue image, booth image, informational image, and object video are superimposed in this order from the background to the foreground. By doing this, the informational image is displayed in front of the background image (venue image, booth image), allowing the user to view the informational image without any sense of incongruity. In addition, the object video is displayed in front of the background image (venue image, booth image), allowing the user to view the movable object moving within the virtual space. In the example of FIG. 6(c), the movable object is displayed in front of the informational image. By doing so, for example, in FIG. 6(a), the movable object is visually perceived as moving across the front of the panel without creating a sense of incongruity. If the order of the informational image and the object image were reversed, the movable object would appear to pass through the gap between the panel and the informational image displayed thereon, potentially creating a very unnatural image. In this embodiment, this sense of incongruity is avoided by setting the order as shown in FIG. 6(c). The order of superimposition shown in FIG. 6(c) is an example, and for example, the information provision image may be set to be displayed in front of the movable object. The user may be able to set whether to display or hide the object video and information provision image, the order of superimposition, and the like.

[0053] Fig. 7 is an explanatory diagram showing a display example accompanied by an object video. Fig. 7(a) is a display example without an object video. A promotional video is displayed on the panel as an information provision video PV. Person P1 is displayed as a still image, not as a video. Figure 7(b) shows the state when object videos are superimposed. Videos of people walking, such as the movable object MO10, are displayed. Displaying people coming and going in this way can improve the realism of the virtual exhibit. Movable objects are not limited to people. Figure 7(c) shows an example in which, instead of people, decorations within the venue are displayed as movable objects MO11. The movable objects MO11 are displayed floating up and down, left and right, etc. within the venue. Displaying movable objects MO11 as decorations in this way not only improves realism, but also enhances the advertising effect of booths and the like. Various other movable objects can also be displayed.

[0054] D. Display control process for users: Next, a process for realizing the display shown in FIGS. 6 and 7 will be described. 8 is a flowchart of the user display control process. This process is executed by the user display control unit 131 (see FIG. 1) of the virtual exhibition system, and in terms of hardware, it is executed by the CPU of the server 100 of the virtual exhibition system.

[0055] When the process starts, the server 100 inputs the visibility information of the user (step S11). In this embodiment, the visibility information is the position information designated by the user. The server 100 generates venue images and booth images according to the location information (step S12). High-resolution 360-degree equirectangular images are prepared for each. The booth images can be generated by perspectively projecting a 3D model of the booth. In this embodiment, since the viewpoints are limited to certain locations in advance, venue images and booth images can be generated for each viewpoint and stored in advance. In this case, the processing of step S12 simply requires reading out data according to the location information.

[0056] Next, the server 100 prepares video data for the information provision image (step S13) by reading out pre-stored video data.

[0057] The server 100 also prepares object video data (step S14). In this embodiment, video data prepared for each viewpoint is read out. If a three-dimensional model of a movable object and its movement are stored as data, the movable object may be moved in virtual space and object video data may be generated by perspective projection in the process of step S14. In this case, although the load of generating the video is increased, it has the advantage of making it easier to prepare a variety of object videos.

[0058] Once each piece of data is prepared in this way, the server 100 transmits each piece of data to the user terminal 300 (step S15). As will be described later, the user terminal 300 receives this data and displays it.

[0059] However, the transmission of this data does not always go smoothly. Therefore, in this embodiment, it is determined whether the communication speed exceeds a reference speed (step S16), and if it does not reach the reference speed, it is determined that there is a problem with display on the user terminal 300, and alternative still image data for the object video is prepared (step S17), and the alternative still image data is transmitted to the user terminal (step S18). The alternative still image can be obtained by extracting any frame from the object video.

[0060] By doing so, the amount of data transmitted to the user terminal 300 can be reduced, and display on the user terminal 300 can be performed smoothly. The reference speed in the above process can be determined arbitrarily. Instead of the communication speed, for example, the time required to transmit the data to be transmitted in step S11 may be estimated based on the amount of data that has been transmitted to the user terminal and the required time, and it may be determined whether this required time exceeds a predetermined reference time. Furthermore, when transmitting the substitute still image data for the object moving image, the transmission of the object moving image may be stopped or continued. If the transmission of the object moving image is continued, the object moving image can be displayed after the transmission and reception of the object moving image is completed, which has the advantage of improving the reality of the virtual exhibition.

[0061] Fig. 9 is a flowchart of the image display process. This process displays on the user terminal 300 based on the data sent from the virtual exhibition system in Fig. 8. This process is primarily executed by the application 313 of the user terminal 300, and in terms of hardware, it is repeatedly executed by the CPU of the user terminal 300.

[0062] When the process starts, the user terminal 300 inputs the line of sight direction (step S21). The line of sight direction may be specified by operating the user terminal 300, or the line of sight direction may be input by detecting the movement of the goggles 301. Also, the angle of view, or so-called zoom, may be input according to the line of sight direction.

[0063] Next, the user terminal 300 performs an affine transformation on the video data according to the line of sight (step S22). In this way, the video data is transformed into an image displayed at a predetermined position, such as on a booth wall or a panel, in the virtual space.

[0064] Then, the user terminal 300 determines whether the object video is in a state in which it can be displayed (step S23). If the data of the object video has been cached to an extent that it can be displayed as a video, it is determined that it can be displayed.

[0065] If the object video can be displayed, the user terminal 300 displays the venue image, the booth image, the information provision image, and the object video in a superimposed manner (step S25). In this way, the image shown in FIG. 6(c) can be displayed.

[0066] On the other hand, if the object video cannot be displayed, the user terminal 300 superimposes and displays a substitute still image for the object video along with the venue image, booth image, and information image (step S24). In this case, too, by displaying a substitute still image in place of the object video, it is possible to display still images of people moving around in the virtual space, allowing the user to get a sense of the atmosphere of the virtual exhibition to some extent.

[0067] The user terminal 300 repeatedly executes the above process. Therefore, if the user changes the line of sight, a corresponding display can be realized. Furthermore, even if a substitute still image is displayed while caching of the object video is not complete (step S24), once caching is complete, it is possible to switch to displaying the object video (step S25).

[0068] FIG. 10 is an explanatory diagram showing the playback mode of an object moving image. 10(a) shows an example in which an alternative still image is initially displayed, and then the object video is displayed after caching of the object video is completed. As shown in the figure, the object video is displayed repeatedly. Note that if communication is proceeding smoothly, the object video may be displayed repeatedly without first displaying the alternative still image.

[0069] Figure 10(b) is an example of repeatedly displaying an object video after displaying a substitute still image, similar to Figure 10(a), but illustrates a case where the object video is played back with fade-in and fade-out effects applied. If the object video is simply displayed repeatedly as in Figure 10(a), there is a risk of the movable object suddenly disappearing and reappearing in a different location at the transition where playback ends and the next playback begins, creating a sense of incongruity. However, by applying fade-in and fade-out effects, this sense of incongruity can be reduced.

[0070] Compared to Figure 10(b), Figure 10(c) shows an example in which the object videos are played back with a cross-fade, with the fade-in and fade-out overlapping. This makes it possible to further reduce the sense of incongruity at the joins between the object videos.

[0071] Figure 10(d) shows an example in which multiple types of object videos are used. In this example, after the alternative still image, object video 1 is played, and then object video 2 is played. An example of a crossfade is shown at the joint between the two, but it is also possible to simply join them as in Figure 10(a), or to simply apply a fade-in / fade-out effect as in Figure 10(b). As shown in Figure 10(d), using multiple types of object videos has the advantage of preventing viewers from getting bored with repetitive object videos and making it easier to maintain the realism of the exhibition. The number of object videos is not limited to two, and more types may be used.

[0072] 8 and 9 show an example in which the object video is superimposed at the user terminal 300, but as shown as a modified example below, this processing may be performed at the server 100. 11 is a flowchart of a modified example of the user display control process. When the process starts, the server 100 inputs the user's field of view information (step S31), generates a venue image and a booth image (step S32), and generates an information provision image (step S33). The process up to this point is the same as steps S11 to S13 in FIG. 8. Next, the server 100 determines whether the communication speed with the user terminal 300 exceeds a reference speed (step S34), and if it does, plays back the object moving image (step S36). On the other hand, if it does not exceed the reference speed, substitute still image data for the object moving image is generated (step S35). The server 100 then superimposes these pieces of data and transmits them to the user terminal (step S37). However, the respective pieces of data are not transmitted as a single composite video, but are transmitted as synchronized layer data so that they are ultimately superimposed and displayed, as shown in Fig. 6(c). In the embodiment (Figs. 8 and 9), the object video is transmitted and received separately from the background image, and the user terminal 300 reads and superimposes each piece individually, whereas in the modified example, the object video is transmitted in a synchronized state. In this way, the same display as in the embodiment can be realized with the modified example. The modified example has the advantage that the load on the user terminal 300 is lighter than in the embodiment. In the modified example (FIG. 11), in order to reduce the load of the superimposing process, it is also possible to adopt a mode in which the server 100 superimposes the images and then transmits them to the user terminal.

[0073] E. Effects and Modifications: According to the virtual exhibition system of the embodiment described above, by superimposing and displaying object videos on the virtual exhibit, it is possible to improve the realism of the virtual exhibit and enhance the user's sense of immersion. Furthermore, by using high-resolution still images representing the virtual space and virtual exhibit and lower-resolution object videos, it is possible to realize such display while suppressing data volume.

[0074] It is not necessary to have all of the various features described above, and some may be omitted or combined. Furthermore, the present invention can be configured into various modified forms other than a system for holding an exhibition in a virtual space. For example, if the virtual exhibit is a landscape of a tourist spot instead of a booth, it can be configured as a system for sightseeing in a virtual space. In addition, it is possible to provide various virtual exhibits such as those of art galleries and museums. [Industrial Applicability]

[0075] The present invention can be used to present a virtual space to a user. [Explanation of symbols]

[0076] 100 servers 101 Virtual Data Storage Unit 102 User data storage unit 103 Exhibitor Data Storage Unit 104 Document Data Storage Unit 111 Virtual Space Allocation Department 112 Data Management Department 113 Transmitter / Receiver 121 Exhibitor display control unit 122 Guidance information input section 123 Information Provision Department 124 Push-type guide unit 131 User display control unit 132 Position setting section 133 Visibility information input unit 134 Materials Distribution Department 200 exhibitor terminals 201 Goggles 211 Transmitter / Receiver 212 Input / output section 213 Applications 300 User Terminals 301 Goggles 311 Transmitting and Receiving Unit 312 Input / output section 313 Applications

Claims

1. A virtual exhibition system that displays a virtual exhibition electronically constructed in a virtual space, a virtual data storage unit that stores virtual data representing the virtual space and the virtual exhibit, as well as object data representing movable objects whose positions can be moved within the virtual space and their movements; a view information input unit for inputting view information that specifies a view in the virtual space of a user viewing the virtual exhibit; a user display control unit that uses the virtual data and the object data based on the view information to display an image of the virtual exhibit on a user output device used by the user; The user display control unit a background image as a still image representing the virtual space and the virtual exhibit based on the virtual data; A virtual exhibition system that displays an object video prepared based on the object data at a lower resolution than the background image, superimposed on the front of the background image with everything except the movable object being transparent.

2. 2. The virtual exhibition system according to claim 1, The user display control unit A virtual exhibition system that transmits data for displaying the background image and data for displaying the object video to the user output device, respectively, and causes the user output device to perform the superimposed display.

3. 2. The virtual exhibition system according to claim 1, The user display control unit is a virtual exhibition system that displays the object video in a limited area within the virtual space where the movable object exists.

4. 2. The virtual exhibition system according to claim 1, The user display control unit is a virtual exhibition system that repeatedly displays the object video of a predetermined length.

5. 2. The virtual exhibition system according to claim 1, the virtual data storage unit stores a plurality of types of the object data, The user display control unit switches any one of the object data to display the object video.

6. 6. A virtual exhibition system according to claim 4 or 5, The user display control unit displays the object video by applying either or both of a fade-in and a fade-out.

7. 2. The virtual exhibition system according to claim 1, The user display control unit is a virtual exhibition system that realizes a transparent display of everything except the movable object by setting an alpha channel representing the part to be transparent in the data for displaying the object video.

8. 2. The virtual exhibition system according to claim 1, The virtual data storage unit further stores video data for displaying an information providing video that provides information to the user, The user display control unit displays the informational video at a predetermined position in the virtual space, behind the object video.

9. 2. The virtual exhibition system according to claim 1, The user display control unit stops display of the object video when the object video cannot be displayed normally on the user output device.

10. 2. The virtual exhibition system according to claim 1, A virtual exhibition system in which the user's position within the virtual space is restricted to a preset position.

11. A virtual exhibition method for creating an electronically constructed virtual exhibition in a virtual space, comprising: The steps that a computer will implement are: storing virtual data representing the virtual space and the virtual exhibit, and object data representing movable objects whose positions can be moved within the virtual space and their movements; inputting view information that identifies a view in the virtual space of a user viewing the virtual exhibit; a user display control step of displaying an image of the virtual exhibition on a user output device used by the user using the virtual data and the object data based on the view information, The user display control step includes: a background image as a still image representing the virtual space and the virtual exhibit based on the virtual data; a step of superimposing an object video, which is prepared based on the object data at a resolution lower than that of the background image, on the front of the background image in a state where everything other than the movable object is transparent, and displaying the object video. Virtual exhibition methods.

Images