Virtual space image generation system and virtual space image generation program

Abstract

To increase the degree of freedom in setting a virtual viewpoint in a system that generates a virtual space image including a real image. [Solution] A virtual space image generation system (1) comprising a virtual space configuration data storage unit (12), a plurality of photographing units (52) arranged to surround a subject, a virtual space generation unit (21) that generates a virtual space from the virtual space configuration data, a virtual body position determination unit (22) that determines the position of a virtual body in the virtual space according to predetermined criteria, a virtual viewpoint setting unit (23) that sets a virtual viewpoint in the virtual space, an image acquisition unit (24) that acquires an image of the subject using a first photographing unit that captures the subject at an angle of view closest to the angle of view at which the virtual viewpoint captures the virtual body, an image display processing unit (24) that displays the image of the subject at the position of the virtual body, facing the virtual viewpoint directly, and a virtual space image generation unit (27) that generates an image capturing the virtual space including the image of the subject from the virtual viewpoint.

Description

[Technical Field]

[0001] The present invention relates to a technique for generating an image capturing a virtual space incorporating an image captured by a camera or the like. [Background technology]

[0002] A variety of VR games have been developed in which players can control a player character in a virtual space. In these VR games, the player wears VR goggles, and an image of the virtual space seen from the player character's point of view is displayed on the VR goggles' display. This allows the player to enjoy the game while feeling immersed.

[0003] In VR games, it is common to distribute gameplay images during and after gameplay so that people other than the player can enjoy the progress of the game. In this case, if the image displayed on the display unit of the VR goggles (i.e., an image capturing the virtual space from the player character's viewpoint) is simply distributed, the distributed image tends to be monotonous. Therefore, a technology is used in which a virtual viewpoint is set at a position different from the player character's viewpoint, and a virtual viewpoint image is generated that captures the virtual space, including the player character, from that virtual viewpoint.

[0004] Patent Document 1 describes a technology in which an image of a player is captured by a camera in real space and then projected onto a transparent virtual screen placed in a virtual space. With this technology, by placing the virtual screen directly opposite the virtual viewpoint, an image can be obtained that makes it appear as if the player is in the virtual space. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7699785 specification [Patent Document 1] Patent No. 7634856 specification [Non-patent literature]

[0006] [Non-Patent Document 1] Steven M. Seitz, Charles R. Dyer, "View morphing", SIGGRAPH '96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 01 August 1996, Pages 21-30. Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 sets a virtual viewpoint so that it captures an image projected on a virtual screen at the same angle of view as the camera captures the subject in real space (distance from the subject and direction from which the subject is captured), thereby obtaining a virtual viewpoint image that looks natural. With this technology, when the camera is moved or its orientation is changed in real space, the virtual viewpoint is changed accordingly. However, since it is physically difficult to perform processing that requires operations that significantly change the position or orientation of the camera, such as switching from an angle of view that captures the subject from the front to an angle that captures the subject from the back, there is a problem in that the degree of freedom in setting the virtual viewpoint is low.

[0008] The problem that the present invention aims to solve is to increase the degree of freedom in setting the virtual viewpoint from which a virtual space is captured in a system that generates an image capturing a virtual space that incorporates images captured by a camera or the like. [Means for solving the problem]

[0009] In order to solve the above problems, the virtual space image generation system according to the present invention comprises: a storage unit in which configuration data of the virtual space is stored; a plurality of imaging units arranged to surround a subject in a photography space; a virtual space generation unit that generates a virtual space based on the configuration data of the virtual space; a virtual body position determination unit that determines a position of a virtual body corresponding to the subject in the virtual space according to a predetermined criterion; a virtual viewpoint setting unit that receives a predetermined input and sets a virtual viewpoint in the virtual space; an image acquisition unit that acquires an image of the subject using a first image capture unit among the plurality of image capture units that captures the subject in the imaging space at an angle of view closest to an angle of view at which the virtual viewpoint captures the virtual object; an image display processing unit that displays the image of the subject acquired by the image acquisition unit at the position of the virtual body so as to face the virtual viewpoint; a virtual space image generation unit that generates an image capturing the virtual space including an image of the subject from the virtual viewpoint; The present invention is characterized by comprising:

[0010] In order to solve the above problems, the virtual space image generation program according to the present invention provides: A memory section that stores the configuration data of the virtual space Equipped with A plurality of imaging units arranged to surround the subject in the imaging space The computer connected to a virtual space generation unit that generates a virtual space based on the configuration data of the virtual space; a virtual body position determination unit that determines a position of a virtual body corresponding to the subject in the virtual space according to a predetermined criterion; a virtual viewpoint setting unit that receives a predetermined input and sets a virtual viewpoint in the virtual space; an image acquisition unit that acquires an image of the subject using a first image capture unit among the plurality of image capture units that captures the subject in the imaging space at an angle of view closest to an angle of view at which the virtual viewpoint captures the virtual object; an image display processing unit that displays the image of the subject acquired by the image acquisition unit at the position of the virtual body so as to face the virtual viewpoint; a virtual space image generation unit that generates an image capturing the virtual space including an image of the subject from the virtual viewpoint; The present invention is characterized in that it operates as a

[0011] The photographing space may be a real space or a virtual space (a virtual space separate from the virtual space). When the photographing space is a real space, the subject is, for example, a person or a sculpted object, and when the photographing space is a virtual space, the subject is, for example, a three-dimensional model made of high-resolution polygons.

[0012] The criteria by which the subject position determination unit determines the position of the virtual body corresponding to the subject in the virtual space may be determined appropriately depending on the application. For example, when the present invention is used in a VR game, the initial position of the player character at the start of game play may be set as the initial position of the virtual body, and the position of the virtual body in the virtual space may be changed in response to the player's operation of the player character.

[0013] The virtual viewpoint setting unit may, for example, receive an input from a user specifying the position of the virtual viewpoint and set the virtual viewpoint at the specified position. Alternatively, when the present invention is used in a multiplayer game, the virtual viewpoint setting unit may receive an input of the position of a player (another player) other than the player who is the subject, and set the virtual viewpoint at the position of the other player. [Effects of the Invention]

[0014] In the virtual space image generation system and the virtual space image generation program according to the present invention, a plurality of image capture units are arranged in advance in a shooting space so as to surround a subject, and configuration data of the virtual space is stored in advance in a storage unit.

[0015] When using the virtual space image generation system and the virtual space image generation program according to the present invention, the virtual body position determination unit determines the position of the virtual body corresponding to the subject in the virtual space based on a predetermined criterion, and then the virtual viewpoint setting unit receives a predetermined input and sets a virtual viewpoint in the virtual space.

[0016] Once the position of the virtual body in the virtual space is determined and the virtual viewpoint is set, the image acquisition unit acquires an image of the subject using a first image capture unit among the multiple image capture units that captures the subject in the captured space at an angle of view closest to the angle of view at which the virtual viewpoint captures the subject at the virtual position.The image display processing unit then displays the acquired image at the position of the virtual body corresponding to the subject in the virtual space, facing the virtual viewpoint.The virtual space image generation unit then generates an image capturing the virtual space including the image of the subject.

[0017] In the virtual space image generation system and virtual space image generation program according to the present invention, a plurality of image capture units are arranged to surround a subject in a shooting space, and an image of the subject is acquired by one of the image capture units that captures the subject in the shooting space at an angle of view closest to the angle of view at which the virtual viewpoint captures the virtual body. In this way, with the present invention, the angle of view at which the subject is captured can be switched simply by selecting one of the plurality of image capture units arranged in the shooting space, thereby increasing the degree of freedom in setting the virtual viewpoint. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram showing the overall configuration of a game system that is an embodiment of a virtual space image generation system according to the present invention. [Figure 2] FIG. 2 is a diagram showing the main configuration of a photography system in the game system of the present embodiment. [Figure 3] An example of the configuration of a photography system in real space. [Figure 4] FIG. 2 is a diagram showing the main configuration of a control and processing device in the game system of this embodiment. [Figure 5] 3A to 3C are diagrams illustrating a surface image according to the present embodiment. [Figure 6]3A and 3B are diagrams illustrating a state in which a virtual body A, a virtual body B, and a second virtual viewpoint are positioned in a virtual space in this embodiment. [Figure 7] FIG. 1 is a diagram illustrating a camera that captures an image of a real subject in an imaging system. DETAILED DESCRIPTION OF THE INVENTION

[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a virtual space image generating system and a virtual space image generating program according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0020] FIG. 1 shows the overall configuration of a virtual space image generation system 1 of this embodiment. This virtual space image generation system 1 has multiple (1 to n) imaging systems 50 (50a to 50n) and a control and processing device 10. The imaging systems 50a to 50n are installed in an amusement facility such as a game center, for example. The multiple imaging systems 50a to 50n may be installed in different locations, or some or all of them may be installed in the same location. The multiple imaging systems 50a to 50n are each connected to the control and processing device 10 via a network.

[0021] Fig. 2 shows the main configuration of the imaging system 50. Fig. 3 shows an example of the configuration of the imaging system 50 installed in real space. The configuration described below is common to the imaging systems 50a to 50n. In the following description, when describing components that belong to different imaging systems (50a, 50b, etc.), reference numerals with suffixes such as a, b, etc. are used, and when describing components that are common to the imaging systems 50a to 50n without any particular distinction, reference numerals are used without suffixes.

[0022] The photography system 50 is equipped with a display unit 51 having 24 displays (first display 51-1 to twenty-fourth display 51-24) arranged to surround a user (corresponding to the subject in the present invention). The 24 displays 51-1 to 51-24 have the same shape and are arranged to form a 24-sided polygon in plan view. LED panels can be suitably used for the displays 51-1 to 51-24. Hereinafter, the space surrounded by the 24 displays 51-1 to 51-24 will be referred to as the booth.

[0023] At the boundary between two adjacent display bodies (e.g., first display body 51-1 and second display body 51-2), a camera group (e.g., first camera group 52-1) consisting of cameras (e.g., first upper camera 52-1U, first middle camera 52-1M, and first lower camera 52-1L) located at the top, middle, and bottom rows, respectively, is arranged. Since 24 boundary areas are formed by the 24 display bodies 51-1 to 51-24, the entire photography system 50 is arranged with the first camera group 52-1 to the 24th camera group 52-24 (72 cameras in total), which constitute the photography unit 52. Each camera is arranged so that the center of its angle of view faces the center of the booth.

[0024] A movement detection unit 53 is provided on the bottom (floor) of the booth. In this embodiment, the movement detection unit 53 is an omni-directional treadmill, which detects the direction and amount of movement of the user as they walk on the floor. Although an omni-directional treadmill is used in this embodiment, other devices that use various sensors to detect the direction and amount of movement of the user may also be used as the movement detection unit 53.

[0025] Also located within the booth are a sound collection unit (microphone) 54 that collects the user's voice emitted within the booth, and a sound generation unit (speaker) 55 that outputs the collected voice at a predetermined position within the virtual space (not shown in FIG. 3). Furthermore, the photography system 50 is equipped with a controller 56 that allows the user to perform appropriate operations. FIG. 3 shows an example of a gunfight between users A and B in the virtual space image generation system 1 of this embodiment, with the users holding gun-shaped controllers 56 within the booth.

[0026] FIG. 4 shows the main configuration of the control and processing device 10.

[0027] The control / processing device 10 includes a storage unit 11. The storage unit 11 includes a virtual space configuration data storage unit 12, a photography system data storage unit 13, an object data storage unit 14, an effect data storage unit 15, and a photography data storage unit 16.

[0028] The virtual space configuration data storage unit 12 stores data that configures a virtual space. This virtual space may be, for example, a representation of cityscapes or scenery of tourist destinations around the world, or a game space where various games are played. The data that configures the virtual space may include information that changes the state of the virtual space over time, such as weather changes or scene changes at predetermined times.

[0029] The imaging system data storage unit 13 stores information about the imaging systems 50a to 50n connected to the control and processing device 10. Specifically, the information stored includes information for identifying each imaging system 50 (such as name and unique ID), the location where each imaging system 50 is installed, the operating status of each imaging system 50, and information about the placement of displays and cameras in each imaging system 50.

[0030] The object data storage unit 14 stores configuration data of objects, which are external elements such as non-player characters, items, and buildings that appear in the virtual space based on a predetermined algorithm. The configuration data of the objects may include three-dimensional model data, movement data, and voice data for each character. The three-dimensional model data of the objects is a three-dimensional model composed of a similar number of polygons (e.g., tens of thousands to one hundred million) to those used in conventional VR games and the like.

[0031] The effect data storage unit 15 stores data on effects to be generated in the virtual space under predetermined conditions. For example, if the virtual space is a simulation of a tourist destination, effects may include rain or snow falling at predetermined times, or fireworks launched at predetermined times and locations. These effects may be generated in response to predetermined input operations via the input unit or predetermined actions by the user (which may include operations on the controller 63). For example, if the virtual space is a game space, effects may include sparks or gunfire emitted from the tip of a gun when a player character fires, or flames emitted by a monster non-player character, which are primarily generated when the actions of a player character or non-player character satisfy predetermined conditions. The effect data may include both image data and audio data.

[0032] The photographed data storage unit 16 stores image data of the virtual space generated by a virtual space image generating unit 27 (to be described later) and sound data of the virtual space.

[0033] The control and processing device 10 includes, as functional blocks, a virtual space generation unit 21, a virtual body position determination unit 22, a virtual viewpoint setting unit 23, a plane image display processing unit 24, an object display processing unit 25, an effect generation unit 26, a virtual space image generation unit 27, and an imaging control unit 28. The control and processing device 10 can be configured, for example, as a general personal computer or a cloud server, and these functional blocks are realized by executing a dedicated program (a virtual space image generation program) pre-installed on a processor. In addition, an input unit including a keyboard, a mouse, etc., and a display unit (separate from the display unit 51 of the imaging system 60) including a liquid crystal display, etc., are connected to the control and processing device 10.

[0034] The virtual space generation unit 21 reads out the configuration data of the virtual space stored in the virtual space configuration data storage unit 12 and generates the virtual space.

[0035] The virtual body position determination unit 22 determines the position in virtual space of a virtual body corresponding to the user of this system (corresponding to the subject in the present invention). The initial position of the virtual body is determined in advance, and the virtual body position determination unit 22 changes the position of the virtual body in virtual space in response to input of data acquired by the movement detection unit 53 of each imaging system 50. Note that the virtual body is represented by a plane image 63 (described later), and is different from a three-dimensional model generally used to represent a player character in virtual space.

[0036] The virtual viewpoint setting unit 23 sets a first virtual viewpoint at the position of a virtual body corresponding to the user in each imaging system 50. The virtual viewpoint setting unit 23 also sets a second virtual viewpoint in response to a predetermined input operation via the input unit or the like. The second virtual viewpoint is set as needed, for example, for the purpose of creating or distributing a video recording the user's experience of the virtual space in the virtual space image generation system 1.

[0037] The plane image display processing unit 24 arranges the image of the user photographed by the photographing system 50 as a plane image 63 at the position of the virtual body in the virtual space. This plane image 63 is arranged as a plane image 63 facing directly toward each of the first virtual viewpoint (the position of the virtual body that is the user in the other photographing system 50) and the second virtual viewpoint (if set; the same applies below). In other words, the plane images 63 of one subject are arranged at the position of the virtual body corresponding to that subject, in the same number as the number of virtual viewpoints excluding the first virtual viewpoint of that virtual body (except when multiple virtual viewpoints are set at the same position).

[0038] The surface image 63 is, for example, an image of the player character displayed on a flat screen, with the screen cut out along the outline of the image (also referred to as an "IVB: interactive visual board"). This IVB is virtually placed and can be changed to any shape at any time. Although it is referred to as a screen here, the display form of the surface image 63 is arbitrary and may be other forms. In either case, the surface image 63 itself is two-dimensional and appears like a thin plate when viewed from the side. However, by capturing an image of the user with sufficient resolution in real space and positioning the surface image 63 directly facing the first and second virtual viewpoints, as shown in FIG. 5, from the first and second virtual viewpoints, it is perceived as a three-dimensional model of the player character made up of high-polygons and having a high-resolution texture.

[0039] Here, an example is described in which a flat surface image 63 that has no thickness in the optical axis direction of the camera is placed, but an image that has thickness in the optical axis direction may also be used. Specifically, for example, a camera or sensor that can acquire distance information along with the image may be used to create an image that has distance information for each pixel (an image of the user's surface that has unevenness as it is).

[0040] The object display processing unit 25 reads out the data stored in the object data storage unit 14, places three-dimensional models of objects such as non-player characters at predetermined positions in the virtual space, and operates them according to a predetermined algorithm.

[0041] The effect generation unit 26 generates effects at predetermined timing and positions based on the effect data stored in the effect data storage unit 15. For example, effects such as rain, snow, fireworks, etc. are generated at predetermined timing and positions.

[0042] Furthermore, the effect generation unit 26 generates an effect at a predetermined position in the virtual space when a predetermined action is performed by a user or a non-player character, which is the subject of the shot. For example, in a scene in which a gunfight takes place in the virtual space, user A fires at virtual body B or a non-player character corresponding to another user B, and the shot hits the target, sparks are emitted from the tip of the gun in the face image 63 of user A, a bullet trail is generated from the tip toward virtual body B or the non-player character, and blood splatters and the like are generated at the hit position in the face image 63 of virtual body B and the three-dimensional model of the non-player character. Conversely, in a scene in which user B or a non-player character fires at a player character and the shot hits the target, sparks are emitted from the tip of the gun in the face image of virtual body B and the tip of the gun held by the non-player character, a bullet trail is generated from the tip toward virtual body A, and blood splatters and the like are generated at the hit position in the face image 63 of user A. The algorithms for determining whether a bullet has hit the target and generating effects can be any suitable algorithm used in conventional VR games or the algorithm described in Patent Document 2.

[0043] The virtual space image generation unit 27 acquires images capturing the virtual space in all directions from the first virtual viewpoint, transmits the images to the photographing system 50, and stores the images in the photographed data storage unit 16. The images acquired here include surface images 63 arranged in the virtual space (however, only those enabled for the first virtual viewpoint). In addition, sound data of the virtual space is collected at the position of the first virtual viewpoint, and is transmitted to the photographing system 50 and stored in the photographed data storage unit 16.

[0044] The shooting control unit 28 acquires an image capturing the virtual space located in a specific direction (typically the front) from the second virtual viewpoint, and creates a gameplay video. The created gameplay video is stored in the shooting data storage unit 16.

[0045] Here, the plane image 63 arranged in the virtual space will be described in detail.

[0046] FIG. 6 shows a planar view of the virtual space. In this example, users A and B use imaging systems 50a and 50b, respectively, and virtual bodies A and B corresponding to users A and B, who are subjects, are located in the virtual space. A second virtual viewpoint is set separately from the positions of virtual bodies A and B to generate an image capturing the virtual space. In this state, a plane image 63a facing the first virtual viewpoint (first virtual viewpoint B) corresponding to virtual body B and a plane image 63b corresponding to the second virtual viewpoint are located at the position of virtual body A. However, plane image 63a is enabled only for first virtual viewpoint B, and plane image 63b is enabled only for the second virtual viewpoint. Therefore, plane image 63b does not appear in the image capturing the virtual space from first virtual viewpoint B, and plane image 63a does not appear in the image capturing the virtual space from the second virtual viewpoint. Similarly, a plane image 63c facing the first virtual viewpoint (first virtual viewpoint A) corresponding to virtual body A and a plane image 63d corresponding to the second virtual viewpoint are arranged at the position of virtual body B. Plane image 63c is enabled only for first virtual viewpoint A, and plane image 63d is enabled only for the second virtual viewpoint. Therefore, plane image 63d does not appear in the image capturing the virtual space from first virtual viewpoint A, and plane image 63c does not appear in the image capturing the virtual space from the second virtual viewpoint.

[0047] FIG. 7 shows the positional relationship between the virtual body A, the virtual body B, and the second virtual viewpoint shown in FIG. 6 superimposed on a floor plan of the booth of the imaging system 50a used by the user A.

[0048] Twenty-four displays (first display 51a-1 to twenty-fourth display 51a-24) and twenty-four camera groups 52a (first camera group 52a-1 to 52a-24; only some of the camera groups are shown in FIG. 7) are arranged around user A, who uses the imaging system 50a. Each of the twenty-four displays (first display 51a-1 to twenty-fourth display 51a-24) displays an image of the virtual space captured in all directions from the first virtual viewpoint of virtual body A, generated by the virtual space image generator 27. This allows user A to visually recognize the situation around the first viewpoint of virtual body A in the virtual space. In the virtual space, virtual body B is located in the direction of display body 51a-10 as seen from virtual body A corresponding to user A, and this display body 51a-10 displays an image of the virtual space including a surface image 63c of user B. As described above, a surface image 63d of user B is also generated in the virtual space, but since the surface image 63d is only enabled for the second virtual viewpoint, it is not displayed on the display of the imaging system 50a.

[0049] In the virtual space, it is the ninth camera group 52a-9 that captures user A in the booth at an angle closest to the angle (capture angle) at which virtual body B captures virtual body A. The plane image display processing unit 24 identifies the ninth camera group 52a-9 based on the information about the imaging system 50a stored in the imaging system data storage unit 13 and the positional relationship between virtual body A and virtual body B in the virtual space, and outputs a control signal to capture an image of user A. In response to this control signal, the imaging system 50a captures an image of user A with each of the three cameras (the ninth upper camera 52a-9U, the ninth middle camera 52a-9M, and the ninth lower camera 52a-9L) that make up the ninth camera group 52a-9, and transmits the data of the captured images to the control and processing device 10.

[0050] When the control / processing device 10 receives the three pieces of image data, the surface image display processor 24 generates a single image by combining the three images and displays it as a surface image 63a at the position of virtual body A. This surface image 63a is enabled only for the first virtual viewpoint of virtual body B. The three images are combined so that the vertical angular spread of user A captured by each camera falls within a predetermined range. Typically, an image of the upper part of user A is extracted from the image data of the ninth upper cameras 52a-9U, an image of the central part of user A from the image data of the ninth middle cameras 52a-9M, and an image of the lower part of user A from the image data of the ninth lower cameras 52a-9L, and then combined. This processing allows for an image with minimal distortion to be obtained. The surface image display processor 24 repeatedly executes a series of processes at a predetermined cycle (e.g., 60 times per second): identifying the positional relationship of each virtual body in the virtual space; identifying the camera that captures the user in the imaging system 50 used by the user corresponding to the virtual body; and displaying a surface image 63 based on the composite image from the received image data.

[0051] The same processing is performed for the second virtual viewpoint, where a composite image is generated from image data of user A captured by each of the three cameras (sixth upper camera 52a-6U, ninth middle camera 52a-6M, and ninth lower camera 52a-6L) constituting the sixth camera group 52a-6 of the imaging system 50, and a surface image 63b is displayed in the virtual space. This surface image 63d is enabled only for the second virtual viewpoint, and does not appear in the image of the virtual space captured from the first virtual viewpoint of virtual body B. Only the surface image 63a, and not the surface image 63b, is displayed on the display constituting the display unit 51b of the imaging system 50b used by user B.

[0052] In the above, a case where only one camera group is used to capture an image of the user has been described, but depending on the number and arrangement of the camera groups, there may be a large difference between the capture angle at which virtual body B captures virtual body A and the actual capture angle at which each camera captures the user, which may result in an unnatural appearance of the surface image 63. For example, if 24 camera groups are arranged at equal intervals in the booth as in the above embodiment, the angles at which each camera group captures the user will differ by 15 degrees, and the difference between the capture angle and the capture angle will be a maximum of 7.5 degrees.

[0053] In such a case, it is advisable to generate a composite image from image data of the user captured by two camera groups positioned on either side of the direction capturing virtual body B from virtual body A, and display the surface image 63. Specifically, for example, the following processing can be used to prevent the surface image 63 from looking unnatural.

[0054] The plane image display processing unit 24 identifies the direction (capture angle) at which the first virtual viewpoint of the virtual body B in the virtual space captures the virtual body A, and transmits this information to the photographing system 50a. The photographing system 50a compares the capture angle received from the plane image display processing unit 24 with the angle (photographing angle) at which each camera group of the photographing system 50a captures the user A.

[0055] If the difference between the capture angle and the shooting angle of any of the 24 cameras is less than a predetermined angle (for example, 3 degrees), user A is photographed by three cameras (upper camera, middle camera, and lower camera) belonging to the camera group that captures user A at the capture angle closest to the reception angle, and the data of a composite image obtained by combining the three images is sent to the control and processing device 10. The control and processing device 10 generates a plane image 63 from the received composite image data and places it in the virtual space. This is the same process as described above.

[0056] If the difference between the capture angle and the shooting angle of any of the 24 camera groups is equal to or greater than a predetermined angle, two camera groups (two camera groups positioned on either side of the direction in which virtual body B is captured from virtual body A) that capture user A at a shooting angle that is smaller than the capture angle are identified. In the example of Fig. 7, the ninth camera group 52a-9 and the tenth camera group 52a-10 are identified for the first virtual viewpoint of virtual body B, and the sixth camera group 52a-6 and the seventh camera group 52a-7 are identified for the second virtual viewpoint.

[0057] Next, user A is photographed using three cameras (top camera, middle camera, and bottom camera) belonging to each of the two identified camera groups, and the three images obtained from each camera group are combined into one to create two composite image data, which is then transmitted to the control and processing device 10. In the control and processing device 10, the plane image display processing unit 24 further combines the received two composite image data using a predetermined image processing algorithm or a pre-prepared image processing AI to generate an image capturing user A at a capture angle, and generates a plane image 63 from that image and places it in the virtual space. As a technology for performing such processing, for example, the one described in Non-Patent Document 1 (View Morphing) can be used.

[0058] In this embodiment, a plurality of camera groups 52 are arranged to surround the user in real space. Images of the user are acquired by the camera group 52 capturing the user at angles of view closest to the angles of view at which the first and second virtual viewpoints capture the virtual body, and are displayed in virtual space as plane image 63. Alternatively, a composite image is created from images of the user captured by two camera groups 52 capturing the user at angles of view close to the angles of view at which the first and second virtual viewpoints capture the virtual body, and is displayed in virtual space as plane image 63. In this embodiment, the angle of view at which the user (subject) is captured can be easily switched simply by switching between the plurality of camera groups 52 arranged in real space, which allows for flexible response to movement of the first virtual viewpoint that occurs in conjunction with movement of another virtual body (player character) in virtual space. Furthermore, the degree of freedom in setting the second virtual viewpoint can be increased.

[0059] The above embodiment is merely an example and can be modified as appropriate in accordance with the spirit of the present invention.

[0060] In the above embodiment, a user (a person playing in a virtual space) is photographed in real space, but it is also possible to photograph something other than a person (for example, an animal or a structure). This allows an animal or structure that exists in real space to be incorporated into and displayed in a virtual space. In this case, since the subject does not need to check the situation in the virtual space, the display body 51 may be a simple wall surface (of course, the configuration described in the above embodiment may be used as is).

[0061] Furthermore, although the above embodiment has described the imaging system 50 installed in real space, it may also be installed in a virtual space (a virtual space dedicated to imaging, separate from the virtual space described in the above embodiment). In that case, the imaging units described with reference to FIGS. 2 and 3 may be virtually positioned to surround the subject. The subject in the virtual space is, for example, a three-dimensional model (a static object such as a sculpted object, or a moving object that operates according to a predetermined algorithm, such as a non-player character) made of high-definition polygons. Since static objects and non-player characters do not usually need to check their surroundings in the virtual space, when photographing them, the display 51 in the above embodiment may be a simple wall surface.

[0062] Placing such a high-resolution three-dimensional model in the virtual space of the above embodiment places a heavy load on the display processing of the three-dimensional model. In contrast, by adopting a configuration in which the three-dimensional model is photographed using a virtually installed photography system and the surface image is displayed in the virtual space, it is possible to display a high-resolution object without significantly increasing the load on the display processing of the virtual space of the above embodiment.

[0063] In the above embodiment, the imaging systems 50a to 50n have the same configuration, but the configuration may be different for each imaging system 50. For example, the number and types of displays 51 and cameras 52 may be different.

[0064] In the above embodiment, a single composite image is generated from images of the user taken by the upper, middle, and lower cameras. However, it is also possible to use only images of the user taken by one camera. However, if an image is taken at a wide angle using only one camera, distortion is likely to occur at the edges. Therefore, it is preferable to use a camera group consisting of multiple cameras (two, four, or more) as in the above embodiment.

[0065] In the above embodiment, data of images captured by a plurality of cameras is transmitted to the control and processing device 10, and the plane image display processing unit 24 generates a composite image from that data, but a composite image may also be generated by providing a composite image generation unit in the photographing system 50. In addition, the configuration may be changed so that some of the functions of the control and processing device 10 are executed in the photographing system 50. [Industrial Applicability]

[0066] More than 40 years have passed since digital games began to replace analog games such as board games. In conventional digital games, it was common sense that the player playing the game and the character acting in the game space were clearly separated. For example, in action games, children choose one of a pre-prepared character set and control that character. Similarly, in role-playing games, players do not play their own avatars, but rather progress through a fantasy world while training adventurers based on a pre-prepared story.

[0067] Until now, there have been very few games in which players can enter a virtual world and control an avatar that projects their own image. This invention is a technology that captures the player's own image, makes it possible for it to appear in the game space in real time, and operate as an avatar.

[0068] In the past, creating a realistic video avatar required the expensive creation of a detailed three-dimensional model, and the enormous load associated with its creation and display processing made it difficult to process in real time. Therefore, conventional methods have been limited to simply combining images of the player, such as capturing a realistic photo of the player's face and simply pasting it onto a three-dimensional model. In contrast, the present invention is an innovative technology that enables players to appear in a virtual space in real time, enabling a more intuitive and realistic gameplay experience in the field of playing games that unfold in a virtual space or experiencing experiences that are difficult to experience in real life.

[0069] [Aspect] It will be apparent to those skilled in the art that the above-described exemplary embodiments are examples of the following aspects.

[0070] (Section 1) A virtual space image generation system according to one aspect of the present invention includes: a storage unit in which configuration data of the virtual space is stored; a plurality of imaging units arranged to surround a subject in a photography space; a virtual space generation unit that generates a virtual space based on the configuration data of the virtual space; a virtual body position determination unit that determines a position of a virtual body corresponding to the subject in the virtual space according to a predetermined criterion; a virtual viewpoint setting unit that receives a predetermined input and sets a virtual viewpoint in the virtual space; an image acquisition unit that acquires an image of the subject using a first image capture unit among the plurality of image capture units that captures the subject in the imaging space at an angle of view closest to an angle of view at which the virtual viewpoint captures the virtual object; an image display processing unit that displays the image of the subject acquired by the image acquisition unit at the position of the virtual body so as to face the virtual viewpoint; a virtual space image generation unit that generates an image capturing the virtual space including an image of the subject from the virtual viewpoint; The present invention is characterized by comprising:

[0071] (Section 8) A virtual space image generation program according to another aspect of the present invention includes: A memory section that stores the configuration data of the virtual space Equipped with A plurality of imaging units arranged to surround the subject in the imaging space The computer connected to a virtual space generation unit that generates a virtual space based on the configuration data of the virtual space; a virtual body position determination unit that determines a position of a virtual body corresponding to the subject in the virtual space according to a predetermined criterion; a virtual viewpoint setting unit that receives a predetermined input and sets a virtual viewpoint in the virtual space; an image acquisition unit that acquires an image of the subject using a first image capture unit among the plurality of image capture units that captures the subject in the imaging space at an angle of view closest to an angle of view at which the virtual viewpoint captures the virtual object; an image display processing unit that displays the image of the subject acquired by the image acquisition unit at the position of the virtual body so as to face the virtual viewpoint; a virtual space image generation unit that generates an image capturing the virtual space including an image of the subject from the virtual viewpoint; The present invention is characterized in that it operates as a

[0072] The photographing space may be a real space or a virtual space (a virtual space separate from the virtual space). When the photographing space is a real space, the subject is, for example, a person or a sculpted object, and when the photographing space is a virtual space, the subject is, for example, a three-dimensional model made of high-resolution polygons.

[0073] The criteria by which the subject position determination unit determines the position of the virtual body corresponding to the subject in the virtual space may be determined appropriately depending on the application. For example, when the virtual space image generation system according to paragraph 1 and the virtual space image generation program according to paragraph 8 are used in a VR game, the initial position of the player character at the start of game play may be set as the initial position of the virtual body, and the position of the virtual body in the virtual space may be changed in response to the player's operation of the player character.

[0074] The virtual viewpoint setting unit may, for example, receive an input from a user specifying the position of the virtual viewpoint and set the virtual viewpoint at the specified position. Alternatively, when the virtual space image generation system according to paragraph 1 and the virtual space image generation program according to paragraph 8 are used in a multiplayer game, the virtual viewpoint setting unit may receive an input of the position of a player (another player) other than the player who is the subject, and set the virtual viewpoint at the position of the other player.

[0075] In the virtual space image generation system according to paragraph 1 and the virtual space image generation program according to paragraph 8, a plurality of imaging units are arranged in advance in a shooting space so as to surround a subject. Also, configuration data of the virtual space is stored in advance in a storage unit.

[0076] When using the virtual space image generation system according to paragraph 1 and the virtual space image generation program according to paragraph 8, the virtual body position determination unit determines the position of the virtual body corresponding to the subject in the virtual space based on a predetermined criterion. Then, the virtual viewpoint setting unit sets a virtual viewpoint in the virtual space based on a predetermined input operation.

[0077] Once the position of the virtual body in the virtual space is determined and the virtual viewpoint is set, the image acquisition unit operates a first image capture unit, among the multiple image capture units, that captures the subject in the captured space at an angle of view closest to the angle of view at which the virtual viewpoint captures the subject at the virtual position, to acquire an image of the subject.The image display processing unit then displays the acquired image at the position of the virtual body corresponding to the subject in the virtual space, facing the virtual viewpoint.The virtual space image generation unit then generates an image capturing the virtual space including the image of the subject.

[0078] In the virtual space image generation system according to paragraph 1 and the virtual space image generation program according to paragraph 8, a plurality of camera units are arranged to surround a subject in a shooting space, and an image of the subject is acquired by one of the camera units that captures the subject in the shooting space at an angle of view closest to the angle of view at which the virtual viewpoint captures the virtual body. In this way, in the virtual space image generation system according to paragraph 1 and the virtual space image generation program according to paragraph 8, the angle of view at which the subject is captured can be switched simply by selecting one of the camera units arranged in the shooting space, thereby increasing the degree of freedom in setting the virtual viewpoint.

[0079] (Section 2) The virtual space image generation system according to paragraph 2 is the virtual space image generation system according to paragraph 1, The aforementioned Multiple The imaging unit is installed in real space, and further, a display unit that is arranged to surround the real subject in the real space and displays an image of the virtual space captured from the position of the virtual body; Equipped with.

[0080] The virtual space image generation system according to paragraph 2 allows a subject in real space to view a virtual space without having to wear a headset or the like.

[0081] (Section 3) The virtual space image generation system according to paragraph 3 is the virtual space image generation system according to paragraph 2, The display unit is composed of a plurality of display bodies, Each of the plurality of image capturing units is disposed at a boundary of the plurality of display bodies.

[0082] By using the virtual space image generation system according to paragraph 3, the display unit and the photographing unit can be arranged while efficiently using the photographing space.

[0083] (Section 4) The virtual space image generation system according to paragraph 4 is a virtual space image generation system according to any one of paragraphs 1 to 3, Each of the above Multiple a first imaging system and a second imaging system each having an imaging unit and a plurality of display bodies; In the first imaging system, the virtual viewpoint setting unit sets the virtual viewpoint at a position of the virtual body corresponding to the subject in the first imaging system; the virtual space image generation unit generates an image capturing the virtual space including an image of the subject captured by the second imaging system; The images generated by the virtual space image generation unit are displayed on the plurality of display units.

[0084] By using the virtual space image generation system according to paragraph 4, multiple users can use the virtual space image generation system simultaneously and communicate with each other.

[0085] (Section 5) The virtual space image generation system according to paragraph 5 is a virtual space image generation system according to any one of paragraphs 1 to 4, The aforementioned Multiple The imaging unit is provided in a second virtual space different from the virtual space.

[0086] The aforementioned MultipleThe photographing unit can be configured to photograph a three-dimensional model created with high-definition polygons and placed in the second virtual space, thereby significantly reducing the processing load related to the main virtual space compared to when such a three-dimensional model is displayed in the main virtual space in the virtual space image generation system.

[0087] (Section 6) The virtual space image generation system according to paragraph 6 is a virtual space image generation system according to any one of paragraphs 1 to 5, each of the plurality of photographing units includes a camera group consisting of a plurality of cameras arranged in a vertical direction; The image acquisition unit acquires an image of the subject by combining data acquired by each of the plurality of cameras.

[0088] By using the virtual space image generation system according to paragraph 6, it is possible to display an image of a real subject with little distortion in a virtual space.

[0089] (Section 7) The virtual space image generation system according to paragraph 7 is a virtual space image generation system according to any one of paragraphs 1 to 6, When a difference between a capture angle at which the virtual viewpoint captures the virtual object and a shooting angle at which the first shooting unit captures the real object is larger than a predetermined reference, the image acquisition unit further acquiring an image of the subject by a second imaging unit that captures the subject in the imaging space at an angle of view that is second closest to an angle of view at which the virtual viewpoint captures the virtual object; An image of the subject captured at the capture angle is generated by combining the image of the subject acquired by the first image capturing unit and the image of the actual subject acquired by the second image capturing unit.

[0090] By using the virtual space image generation system according to paragraph 7, an image of a subject captured at a capture angle at which a virtual viewpoint captures a virtual body can be displayed in virtual space, thereby avoiding any visual discomfort. [Explanation of symbols]

[0091] 1...Virtual space image generation system 10...Control and processing device 11...Storage section 12...Virtual space configuration data storage unit 13...Photography system data storage unit 14...Object data storage unit 15...Effect data storage section 16...Photography data storage unit 21...Virtual space generation unit 22...Virtual body position determination unit 23...Virtual viewpoint setting section 24...Picture image display processing unit 25...Object display processing section 26...Effect generation section 27...Virtual space image generation unit 28...Shooting control unit 50, 50a, 50b, ... 50n... Shooting system 51...Display section 51-1~51-24…Display body 52...Photography Department 52-1~52-24...First camera group 52-1U~52-24U...1st upper camera 52-1M~52-24M...1st Middle Camera 52-1L~52-24L...1st lower camera 53...Movement detection unit 54…Sound collection section 55...Phonetic section 56...Controller 63, 63a to 63d... Surface images

Claims

1. a storage unit in which configuration data of the virtual space is stored; a plurality of imaging units arranged to surround a subject in a photography space; a virtual space generation unit that generates a virtual space based on the configuration data of the virtual space; a virtual body position determination unit that determines a position of a virtual body corresponding to the subject in the virtual space according to a predetermined criterion; a virtual viewpoint setting unit that receives a predetermined input and sets a virtual viewpoint in the virtual space; an image acquisition unit that acquires an image of the subject using a first image capture unit among the plurality of image capture units that captures the subject in the imaging space at an angle of view closest to an angle of view at which the virtual viewpoint captures the virtual body; an image display processing unit that displays the image of the subject acquired by the image acquisition unit at the position of the virtual body so as to face the virtual viewpoint; a virtual space image generation unit that generates a first image capturing the virtual space including an image of the subject from the virtual viewpoint; A virtual space image generation system comprising:

2. the plurality of imaging units are provided in real space, moreover, a display unit that is disposed to surround the subject in the real space and that displays a second image of the virtual space captured from the position of the virtual body; The virtual space image generation system according to claim 1 , comprising:

3. The display unit is composed of a plurality of display bodies, Each of the plurality of image capturing units is disposed at a boundary of the plurality of display bodies.

3. The virtual space image generation system according to claim 2, wherein:

4. a first image capturing system and a second image capturing system, each of which has the plurality of image capturing units, the display unit, and the virtual viewpoint setting unit; a virtual viewpoint setting unit of the first image capturing system sets the virtual viewpoint at a position of the virtual body corresponding to a subject in the second image capturing system; a virtual viewpoint setting unit of the second image capturing system sets the virtual viewpoint at a position of the virtual body corresponding to the subject in the first image capturing system; a display unit of the first image capturing system displays a second image including an image of the subject captured in the second image capturing system, the second image capturing system capturing the virtual space from the position of the virtual body in the first image capturing system; A display unit of the second image capturing system displays a second image including an image of the subject captured in the first image capturing system, the second image capturing system capturing the virtual space from the position of the virtual body in the second image capturing system.

3. The virtual space image generation system according to claim 2.

5. The virtual space image generation system according to claim 1 , wherein the plurality of image capturing units are provided in a second virtual space different from the virtual space.

6. each of the plurality of photographing units includes a camera group consisting of a plurality of cameras arranged in a vertical direction; The image acquisition unit acquires an image of the subject by combining data acquired by each of the plurality of cameras.

2. The virtual space image generation system according to claim 1.

7. When a difference between a capture angle at which the virtual viewpoint captures the virtual object and a shooting angle at which the first shooting unit captures the subject is greater than a predetermined reference, the image acquisition unit further acquiring an image of the subject by a second imaging unit that captures the subject in the imaging space at an angle of view that is second closest to the angle of view at which the virtual viewpoint captures the virtual object; An image of the subject captured at the capture angle is generated by combining the image of the subject captured by the first image capturing unit and the image of the subject captured by the second image capturing unit.

2. The virtual space image generation system according to claim 1.

8. A memory section that stores the configuration data of the virtual space Equipped with A plurality of imaging units arranged to surround the subject in the imaging space The computer connected to a virtual space generation unit that generates a virtual space based on the configuration data of the virtual space; a virtual body position determination unit that determines a position of a virtual body corresponding to the subject in the virtual space according to a predetermined criterion; a virtual viewpoint setting unit that receives a predetermined input and sets a virtual viewpoint in the virtual space; an image acquisition unit that acquires an image of the subject using a first image capture unit among the plurality of image capture units that captures the subject in the imaging space at an angle of view closest to an angle of view at which the virtual viewpoint captures the virtual body; an image display processing unit that displays the image of the subject acquired by the image acquisition unit at the position of the virtual body so as to face the virtual viewpoint; a virtual space image generation unit that generates an image capturing the virtual space including an image of the subject from the virtual viewpoint; A virtual space image generation program characterized by causing the program to operate as follows.

Images