Imaging system

Abstract

An imaging system (1) comprises: an imaging information storage unit (22) that has stored therein data of an image of an object imaged by using a camera in a real space and position information indicating the positional relationship between the object and the camera when the object is imaged by using the camera; a virtual space generation unit (341) that generates a virtual space by using display data that is prepared in advance and simulates a space in which the content is imaged; a virtual screen display processing unit (342) which disposes a transmission-type virtual screen (62) in the virtual space; an image display processing unit (36) that reads the data of the image of the object from the imaging information storage unit and displays the image on the virtual screen; and a virtual imaging unit (37) that captures, from a prescribed position determined on the basis of the position information in the virtual space, an image including the virtual screen together with a virtual space visible through the virtual screen.

Description

Shooting System

[0001] The present invention relates to a photography system used to create content such as movies and television programs.

[0002] Filming systems using chromakey compositing are widely used to create content such as movies and television programs. In a filming system using chromakey compositing, a camera captures a performer, such as an actor, against a background colored a predetermined color, known as a green screen, and then extracts an image of the performer's area by excluding the area of ​​the predetermined color. The extracted image is then combined with a separately prepared image using computer graphics (CG) or the like to create the content.

[0003] Recently, a technology called virtual production has been proposed (see, for example, Patent Document 1). In virtual production, an image is projected onto a display consisting of multiple LED light sources, and a camera captures actors performing against this background. This system simultaneously captures the actors and the background, eliminating the need for image compositing, allowing content to be created in a short amount of time. This system displays images with a sense of depth on the display, and by changing the display mode of the image depending on the relative positions of the camera and display, it is possible to create content that makes it appear as if actors are performing in real space.

[0004] JP 2024-35420 A

[0005] In virtual production, displays must be positioned to cover the entire area seen by the camera, which necessitates the use of large, expensive screens equipped with numerous LED light sources in order to capture images of performers and other subjects from various angles.

[0006] The problem to be solved by the present invention is to provide a technology that can acquire, at low cost, images of an object captured at various angles of view.

[0007] The photography system of the present invention, which has been developed to solve the above-mentioned problems, is characterized by comprising: a photography information storage unit that stores image data of an object photographed by a camera in real space and positional information that represents the positional relationship between the object and the camera when the object is photographed by the camera; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates the space in which content is photographed; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that reads image data of the object from the photography information storage unit and displays the image on the virtual screen; and a virtual photography unit that photographs an image including the virtual screen, together with the virtual space that is visible through the virtual screen, from a predetermined position in the virtual space that is determined based on the positional information.

[0008] In the imaging system according to the present invention, a shooting information storage unit stores image data of an object captured by a camera in real space and position information indicating the relative positions of the object and the object when the object was captured by the camera. The object captured here is, for example, an actor or other performer. The image of the object is, for example, an extracted image of the object obtained by capturing the object against a conventional green screen. Display data that serves as the background of the content is also prepared in advance. The virtual space generation unit generates a virtual space using the display data. The virtual screen display processing unit places a transparent virtual screen in the virtual space, and the image display processing unit displays the image of the object on the virtual screen. The transparent virtual screen is a virtual object (screen) that functions as a reference plane for setting the imaging plane (focus) of the virtual capture unit in the virtual space and is transparent except for the area where the image is displayed. The virtual capture unit captures an image including the virtual screen, along with the virtual space visible through the virtual screen, from a predetermined position determined based on the position information. In this way, the imaging system according to the present invention acquires an image incorporating the image of the object captured in real space into the virtual space. With the imaging system of the present invention, it is possible to capture an object in real space against, for example, a conventional green screen, eliminating the need for a large, expensive display with numerous LED light sources, as in virtual production. Furthermore, the virtual space that serves as the background can be freely designed, and the size and position of the virtual screen placed therein can be easily changed. Therefore, with the imaging system of the present invention, it is possible to capture images of an object captured at various angles of view at low cost.

[0009] 1 is a diagram illustrating the configuration of the main components of an embodiment of a photography system according to the present invention. FIG. 1 is an example of an image obtained by photographing with a wide-angle lens of a camera in the photography system of this embodiment. FIG. 2 is a diagram schematically illustrating a state in which a performer is photographed with a wide-angle lens of a camera in the photography system of this embodiment. FIG. 3 is a diagram illustrating a state in which a performer is photographed with a narrow-angle lens of a camera in the photography system of this embodiment. FIG. 4 is an example of an image obtained by photographing with a narrow-angle lens of a camera in the photography system of this embodiment. FIG. 5 is a diagram illustrating a state in which an extracted image of a performer is projected onto a virtual screen placed in a virtual space in the photography system of this embodiment. FIG. 6 is a diagram illustrating a modified example of the camera angle of view acquisition unit in the photography system of this embodiment. FIG. 7 is a diagram illustrating an example in which a performer is photographed with multiple cameras in a studio in the photography system of this embodiment. FIG. 8 is a diagram illustrating an example in which multiple sets of virtual cameras and virtual screens are arranged in a virtual space in the photography system of this embodiment. FIG. 9 is a diagram illustrating an example in which a virtual screen is tilted forward and backward in the photography system of this embodiment. FIG. 10 is a diagram illustrating an example in which a background screen is arranged in the photography system of this embodiment. FIG. 11 is a diagram illustrating an example of a first virtual space that can be used in the photography system of this embodiment.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an imaging system according to the present invention will be described below with reference to the drawings.

[0011] Fig. 1 shows the main configuration of a photography system 1 of this embodiment. The photography system 1 of this embodiment has a photography device 10, a control and processing device 20, and a player terminal 50. The photography device 10, the control and processing device 20, and the player terminal 50 are connected to each other so that they can communicate with each other via a wired or wireless network 9. Although Fig. 1 shows only one player terminal 50, there may be multiple player terminals 50.

[0012] The filming device 10 is placed in a studio 100 where a performer 14, who is the subject of filming, performs. The filming device 10 includes a background member 11 called a green screen, which is colored a predetermined color (e.g., green), a camera 12, a position sensor 13, and a sound collection unit 15. The performer 14 also wears earphones 16 as an audio output unit. In this embodiment, earphones 16 are used, but speakers may also be placed in the studio 100. Furthermore, a screen 17 as a display unit is placed in the studio 100 at a position outside the field of view of the camera 12, and an image captured by a virtual camera 61 provided in a virtual space 60, which will be described later, is displayed on the screen 17 in real time.

[0013] As shown in FIG. 2 , the background member 11 has a grid of reference points (grid points 111). In this embodiment, seven lattice points 111 are provided in both the horizontal and vertical directions (a total of 49 lattice points). The spacing and number of the lattice points 111 can be changed as desired. In this embodiment, the reference points are provided in a grid pattern; however, they may be provided two-dimensionally and regularly on the surface of the background member 11, and may be provided in a honeycomb pattern, for example. The camera 12 captures an actor or performer 14 with the background member 11 as the background. The position sensor 13 in this embodiment is an infrared LiDAR (Light Detection and Ranging) sensor that emits infrared light from its main body 134 to the surrounding area and measures the positions of the reference points 131 to 133 by receiving reflected light from reference points 131 and 132 provided at two locations on the camera 12 (e.g., the tip of the lens and the main body of the camera 12) and reference point 133 provided at one location on the performer 14 (e.g., the nape of the neck). For example, stickers that reflect infrared light are attached to the reference points 131 to 133. The positions and number of the reference points 131 to 133 are also examples and can be changed as appropriate as long as the above functions can be achieved.

[0014] The control / processing device 20 includes a storage unit 21. The storage unit 21 includes a shooting information storage unit 22, a virtual space display data storage unit 23, a character data storage unit 24, and a shooting data storage unit 25. The shooting information storage unit 22 stores shooting information including image data of an object captured in real space, the posture of the camera 12 when the image was captured, and information correlating the camera 12 with the positional information of the performer 14. Note that the concept of an image in this specification includes not only still images but also moving images (e.g., moving images composed of still images at 60 frames per second). The virtual space display data storage unit 23 stores in advance time-series display data in a 3D (three-dimensional) virtual space simulating various scenes to be used in content created by the shooting system 1. This display data also includes audio data such as sound effects. The character data storage unit 24 stores display data of the 3D avatar of the player character (PC) operated by the player, and time-series display data of the 3D avatar of a non-player character (NPC) not operated by the player. As described above, since the non-player characters are not operated by the player, their actions during filming of the video content are determined in advance according to a scenario, and the time-series display data (including audio data) is stored in the character data storage unit 24.

[0015] The control / processing device 20 includes, as functional blocks, an image extraction unit 31, a camera angle of view acquisition unit 32, a position information acquisition unit 33, a virtual space generation unit 341, a virtual screen display processing unit 342, a character display processing unit 35, an image display processing unit 36, a virtual shooting unit 37, and an audio processing unit 38. The operation of each of these units will be described later. The actual entity of the control / processing device 20 is, for example, a general-purpose personal computer, and is realized by executing a pre-installed dedicated program (content creation program 30) on a processor. The control / processing device 20 may be located in an appropriate location inside or outside the studio 100. Alternatively, a cloud server provided on a network may be used as the control / processing device 20. In addition, an input unit 41 including a keyboard, a mouse, etc., and a display unit 42 including a liquid crystal display, etc., are connected to the control / processing device 20.

[0016] The player terminal 50 includes a memory unit 51. The memory unit 51 stores information about the player using the player terminal 50 (such as the player's ID information). The player terminal 50 includes an input operation unit 52, an audio input unit 53, an audio output unit 54, and a display unit 55. The player terminal 50 may be, for example, a head-mounted display equipped with a motion capture or position sensor that functions as the input operation unit 52. The player terminal 50 can be installed in any location. In other words, the player can control the player character from any location inside or outside the studio 100.

[0017] Next, a procedure for creating video content using the imaging system 1 of this embodiment will be described.

[0018] When creating video content, the background space of the content is set as a 3D virtual space in advance, and time-series data for generating a 3D virtual space that changes over time in accordance with a scenario is created and stored in the virtual space display data storage unit 23. Additionally, time-series display data for 3D avatars of player characters appearing in the content, as well as 3D avatars of non-player characters, are stored in the character data storage unit 24. Player characters are, for example, characters who appear in the content and have contact with performers, such as through conversation. Non-player characters are, for example, so-called mob characters that appear as a crowd in the background of the content, enemy characters that act based on a predetermined algorithm and fight performers or player characters, or objects positioned at predetermined locations in the virtual space. When there are multiple player characters, information associating each player character with information about the player terminal 50 (e.g., terminal ID information) used by the player controlling that player character is also stored. Hereinafter, for convenience, the display form of non-player characters, which are objects, will also be referred to as avatars.

[0019] Once the above preparations are complete, preparations for filming the video content begin. The preparations for filming the video content are carried out both in the studio 100 set up in the real space and in the virtual space 60.

[0020] First, the preparations for filming the performer 14 in a studio 100 set in real space will be described. A background member 11 called a green screen is placed in the studio 100. Reference points 131 and 132 of the LiDAR sensor serving as the position sensor 13 are set on the lens and body of the camera 12, respectively. Furthermore, a reference point 133 of the LiDAR sensor is set at a predetermined position on the performer 14. In this embodiment, infrared light-reflecting stickers are attached to the lens and body of the camera 12 and to the neck of the performer 14. A LiDAR sensor body 134 is placed at a predetermined position in the studio 100 (e.g., a corner of the ceiling). The operating principle of LiDAR is well known, and therefore a detailed description will be omitted here. Furthermore, a sound collection unit 15 for collecting sounds emitted by the performer 14 and a screen 17 are placed at positions outside the angle of view of the camera 12 in the studio 100. Furthermore, the performer 14 wears earphones 16.

[0021] The camera 12 photographs the performer 14 standing in an initial position with the background member 11 as the background, and acquires image data of the performer 14. The image data of the performer 14 acquired by the camera 12 is stored in the photographing information storage unit 22 together with a timestamp indicating the time when the image was acquired (time of photographing, time elapsed since the start of the performance, etc.). In addition, sounds made by the performer 14 during the performance are collected by the sound collection unit 15 and stored in the photographing information storage unit 22.

[0022] Figures 3 and 4 show examples of photographing the performer 14 in the studio 100. Figure 3 shows the state in which the performer 14 is photographed with the wide-angle lens of the camera 12, and Figure 4 shows the state in which the performer 14 is photographed with the narrow-angle lens of the camera 12. When the performer 14 is photographed with the wide-angle lens of the camera 12 as in Figure 3, an image capturing the entire body of the performer 14 is obtained, as shown in Figure 2. When the performer 14 is photographed with the narrow-angle lens of the camera 12 as in Figure 4, an enlarged image capturing a portion of the performer 14 (the upper body in this example) is obtained, as shown in Figure 5. As shown in Figures 2 and 5, the grid points 111 provided on the background member 11 are also photographed in the image of the performer 14.

[0023] In the control / processing device 20, the image extraction unit 31 reads out the image data stored in the shooting information storage unit 22. Then, the image data is extracted excluding a predetermined color area (the green screen portion, including the grid point 111) contained in the image. This generates an image (extracted image) from which the performer 14 has been cut out. The generated extracted image is also stored in the shooting information storage unit 22.

[0024] The camera angle of view acquisition unit 32 calculates the number of grid points 111 included in the image stored in the shooting information storage unit 22 (in this embodiment, the number of grid points 111 arranged vertically and the number of grid points 111 arranged horizontally. This number includes the number of grid points 111 hidden by the performer 14). Then, based on the number of grid points 111, the angle of view (wide angle, narrow angle, etc.) of the lens of the camera 12 when the image was captured is calculated. If the positions of the camera 12 and the performer 14 do not change, the greater the number of grid points included in the image, the wider the lens of the camera 12; and the fewer the number of grid points included in the image, the narrower the lens of the camera 12 (zoomed state). Furthermore, the camera angle of view acquisition unit 32 determines the positions of reference points 131 and 132 set on the camera 12 based on the output signal from the position sensor 13, and determines the attitude (elevation angle θ) of the camera 12 from the relative positional relationship between the two.

[0025] Based on the output signal from the position sensor 13, the position information acquisition unit 33 determines the positions of the reference point 131 (or / and 132) set on the camera 12 and the reference point 133 set on the performer 14, and determines the positional relationship between them. The information on the positional relationship between them is information on the direction in which the camera 12 captures the performer 14 and information on the distance between the camera 12 and the performer 14 (i.e., vector information). The information on the camera attitude acquired by the camera angle of view acquisition unit 32 and the information on the positional relationship between the camera 12 and the performer 14 acquired by the position information acquisition unit 33 are each stored in the shooting information storage unit 22 together with a timestamp indicating the time when the information was acquired.

[0026] The above-mentioned processing by the image extraction unit 31, the camera angle of view acquisition unit 32, and the position information acquisition unit 33 is performed in real time while the performer 14 is being filmed in the studio 100.

[0027] Next, preparation for shooting in the virtual space will be described.

[0028] The virtual space generation unit 341 first reads, from the virtual space display data storage unit 23, time-series display data of the 3D virtual space in which the video content is shot, and generates a virtual space based on the display data at the start time. Furthermore, the virtual screen display processing unit 342 places a transparent virtual screen 62 at a predetermined initial position within the virtual space, as illustrated in FIG. 6 . The transparent virtual screen 62 is a virtual object (screen) that can function as a reference surface when setting the imaging plane (focus) of the virtual imaging unit 37 in the virtual space 60 and is transparent except for the area onto which the image is projected. Because the virtual screen 62 is a virtual object, player characters and non-player characters can freely pass through the virtual screen 62.

[0029] The character display processing unit 35 reads the display data of the player character avatars and the non-player character avatars and places them at their initial positions in the virtual space. The virtual shooting unit 37 places a virtual camera 61 in the virtual space 60 based on the information on the positional relationship between the camera 12 and the performer 14 obtained by the position information acquisition unit 33. Specifically, the position of the performer 14 is matched to the position of the virtual screen 62, and using this as a reference, the relative positional relationship between the camera 12 and the performer 14 is reflected in the positional relationship between the virtual screen 62 and the virtual camera 61 to determine the position of the virtual camera 61. In this embodiment, the virtual camera 61 is positioned directly facing the virtual screen 62. In other words, the virtual camera 61 and the virtual camera 61 are positioned so that the optical axis of the virtual camera 61 is perpendicular to the surface of the virtual screen 62. The size of the virtual screen 62 is changed appropriately depending on the size of the extracted image 64 of the performer 14 displayed on the virtual screen 62.

[0030] The image display processing unit 36 ​​displays the extracted image 64 of the performer 14 extracted by the image extraction unit 31 on the virtual screen 62. In this embodiment, the image display processing unit 36 ​​projects the extracted image 64 of the performer 14 from the position of the virtual camera 61 toward the virtual screen 62, thereby displaying the extracted image 64 of the performer 14 on the virtual screen 62. The relative positional relationship between the virtual camera 61 and the virtual screen 62 reflects the positional relationship between the camera 12 and the screen 17. Therefore, by projecting the extracted image 64 of the performer 14 from the virtual camera 61 toward the virtual screen 62, an image capturing the performer 14 at the same angle of view as that captured by the camera 12 in real space can be displayed on the virtual screen 62. However, the method of displaying the extracted image 64 of the performer 14 on the virtual screen 62 is not limited to this, and various methods can be used, such as transmitting pixel data constituting the extracted image 64 to the projection screen 62. In this way, a real-life image of the performer 14 captured in real space is incorporated into the virtual space.

[0031] As described above, the virtual screen 62 is a transparent type, so as shown in Fig. 6, the virtual camera 61 simultaneously captures an extracted image 64 of the performer 14 projected onto the virtual screen 62 and the virtual space 60 located in the background of the virtual screen 62. The virtual shooting unit 37 uses the virtual camera 61 to shoot images of the virtual space 60 at a predetermined frame rate (e.g., 60 frames per second). The data of the shot images is stored in the shooting data storage unit 25 and is also displayed in real time on the screen 17 of the studio 100.

[0032] The virtual space generation unit 341 generates an image capturing the virtual space from the position of the player character 63 placed at an initial position, and transmits the image to the player terminal 50 used by the player operating the player character 63. The received image is displayed on the display unit 55 of the player terminal 50. Although only one player character 63 is shown in FIG. 6, multiple player characters 63 may exist simultaneously. As shown in FIG. 6, when the player character 63 is located behind the virtual screen 62, a silhouette of an extracted image 64 of the performer 14 is displayed on the display unit 55. When the player character 63 is located on the front side of the virtual screen 62 (the side where the virtual camera 61 is located), an image of the extracted image 64 of the performer 14 viewed from the front is displayed on the display unit 55.

[0033] After preparations for filming are completed in the studio 100 set up in the real space and the virtual space 60 as described above, the director commands the start of filming by a predetermined action (e.g., giving a voice command to start filming in the studio 100 and simultaneously pressing a filming start button via the input unit 41). In response to this, the performer 14 in the studio 100 begins performing according to a pre-prepared scenario. The performance of the performer 14 includes vocalizations, and the vocalizations of the performer 14 are collected by the sound collection unit 15 and transmitted to the control / processing device 20 (corresponding to the function of the first sound collection unit in the present invention). In the control / processing device 20, the audio processing unit 38 outputs the audio received from the sound collection unit 15 as audio emanating from a predetermined position on the virtual screen 62 in the virtual space 60 (e.g., the center of gravity of the extracted image 64) (corresponding to the function of the first audio output unit in the present invention).

[0034] Furthermore, the virtual space generation unit 341 and the character display processing unit 35 read the time-series display data of the virtual space and the time-series display data of the non-player characters from the virtual space display data storage unit 23 and the character data storage unit 24, respectively, and sequentially reflect (play) them in the virtual space 60. This display data also includes audio data, and the audio data is played as audio emanating from the position of the non-player character in the virtual space.

[0035] At the player terminal 50, the player controls the player character 63 via the input operation unit 52 and also makes sounds according to a prepared scenario. The actions input by the player via the input operation unit 52 are sequentially transmitted to the control / processing device 20 and reflected in the actions of the player character 63 in the virtual space. In addition, sounds made by the player are collected by the audio input unit 53 and sequentially transmitted to the control / processing device 20. In the control / processing device 20, the audio processing unit 38 outputs the received sounds as sounds made from the position of the player character 63 in the virtual space 60.

[0036] The audio processing unit 38 collects audio from within the virtual space 60 at a predetermined position on the virtual screen 62 (e.g., the center of gravity of the extracted image 64) (corresponding to the function of the second audio collection unit in the present invention). The collected audio is transmitted to the imaging device 10 and output from earphones 16 worn by the performer 14 (corresponding to the function of the second audio output unit in the present invention). This allows the performer 14 to visually confirm the state of the virtual space 60 on the screen 17 and to perform their performance while listening to audio emitted in the virtual space 60 and captured at the position of the virtual screen 62 through the earphones 16. The audio processing unit 38 also collects audio from within the virtual space 60 at the position of the player character 63 in the virtual space 60. The collected audio is transmitted to the player terminal 50 used by the player operating the player character 63 and output from the audio output unit 54. This allows the player operating the player character 63 to view the state within the virtual space 60 from the viewpoint of the player character 63 on the display unit 55, and to move the player character 63 while listening to the sound (including the sound of the performer 14 emitted from a predetermined position on the virtual screen 62) emitted in the virtual space 60 and picked up at the position of the player character 63 on the sound output unit 54.

[0037] During filming, the positions of the camera 12 and the performer 14 are monitored by the position information acquisition unit 33 based on the output signal from the position sensor 13. When the camera 12 or the performer 14 moves (changes their position and / or posture), the position information acquisition unit 33 in the control / processing device 20 detects this information. The position of the camera 12 is associated with the position of the virtual camera 61, and when the position information acquisition unit 33 detects the movement of the camera 12, the virtual shooting unit 37 moves the virtual camera 61 placed in the virtual space 60 in accordance with the movement of the camera 12. The position of the performer 14 is associated with the position of the virtual screen 62, and when the position information acquisition unit 33 detects the movement of the performer 14, the virtual screen display processing unit 342 moves the virtual screen 62 placed in the virtual space 60 in accordance with the movement of the performer 14.

[0038] The angle of view of the camera 12 is also monitored by the camera angle of view acquisition unit 32. Changes in the angle of view of the camera 12 are detected based on the number of lattice points included in the image captured by the camera 12. The information on the angle of view of the camera 12 acquired by the camera angle of view acquisition unit 32 is reflected in the angle of view of the virtual camera 61 by the virtual capturing unit 37. For example, if the angle of view of the camera 12 is changed so that the performer 14 is captured in a magnified manner, the angle of view of the virtual camera 61 is also changed in the virtual space so that the extracted image of the performer 14 displayed on the virtual screen 62 is captured in a zoomed manner. Therefore, the same state of the performer 14 as captured by the camera 12 in real space is captured by the virtual camera 61 in the virtual space.

[0039] When the performer 14 has completed his or her performance based on a prepared scenario, the director instructs the end of filming by a predetermined action (for example, by giving an audio command to end filming in the studio 100 and simultaneously pressing the end filming button via the input unit 41). This causes the virtual space generation unit 341 and the character display processing unit 35 to end playback of the time-series display data of the virtual space 60 and the time-series display data of the non-player character avatar, respectively. Furthermore, the player's operation of the player character 63 on the player terminal 50 is terminated. The virtual shooting unit 37 saves the image data acquired by the virtual camera 61 at a predetermined frame rate in the shooting data storage unit 25 as a single video file.

[0040] In the filming system 1 of this embodiment, image data of the performer 14 performing in a real-space studio 100 is displayed on a transparent virtual screen 62 disposed in a virtual space 60, thereby enabling the creation of video content incorporating a live-action image of the performer 14 in the virtual space. The filming system 1 of this embodiment simply captures an object against a green screen in the real-space studio 100 and obtains an extracted image 64 by extracting the image of the performer 14 using conventional processing. This eliminates the need for a large, expensive display with multiple LED light sources, as in virtual production. Furthermore, because the background of the video content is a virtual space, the position and size of the virtual screen 62 onto which the live-action extracted image 64 is projected can be easily changed to match the angle of view of the camera 12, the position of the performer 14, and the relative positions of the camera 12 and the performer 14. Therefore, video of an object captured at various angles of view can be obtained at low cost.

[0041] The filming system 1 of this embodiment can create video content in which a player character 63 and non-player characters operated by a player using a player terminal 50 appear together with a performer 14 who is filmed in live action.

[0042] Previously, when combining images or videos of performers with a separately prepared background image, the only way to create a composite was to combine a layer of the performer's image, created as a single image file, with the background image. This meant that, like combining a cel drawing of a character with a background image in the production of classic animation, the composite was merely a composite of two independent images. This meant that, for example, the performer's line of sight or posture would not match the background image, creating a sense of incongruity in the composite image.

[0043] In contrast, in the filming system 1 of this embodiment, images captured by a virtual camera 61 located in a virtual space 60 are displayed in real time on a screen 17 located in the studio 100. Furthermore, the sound emitted by the performer 14 is reflected in the virtual space, and the sound in the virtual space is output from the performer's 14's earphones 16 at the position of a virtual screen 62 located in the virtual space. This allows interactive communication (e.g., conversation) between the performer 14 and a player character 63 or non-player character located in the virtual space 60. Furthermore, the shadow of the performer 14 can be made to appear in the virtual space 60 in real time, and the sound emitted by the player character 63 or the like in the virtual space 60 can be heard by the performer 14 at a volume corresponding to the distance between the player character 63 and the performer 14. This allows the performer 14 to perform a scene while experiencing the scenario. Furthermore, since post-processing such as image compositing is not required, the cost of such processing can be reduced.

[0044] In the above embodiment, a case where video content is created by filming the performances of the performers 14 and the player characters 63 has been described, but a similar configuration can also be used to play a game in a virtual space and film the play scenes. Furthermore, when playing a game, a configuration with even greater interactivity than the above embodiment can be used.

[0045] For example, in a battle game, when an attack made by the player character 63 hits the extracted image 64 of the performer 14, a predetermined effect (e.g., blood splatter) can be generated at the location on the virtual screen 62 where the attack hit. Alternatively, conversely, when the performer 14 attacks the player character 63, an attack can be generated against the player character 63 from the display position of the extracted image 64 of the performer 14 on the virtual screen 62 (e.g., an image of a bullet being fired from the position of the extracted image 64 in response to a gun shot made by the performer 14 is generated). Furthermore, a predetermined effect (e.g., darkening the display of the virtual space 60) can be generated not only between the player character 63 and the performer 14, but also in the virtual space 60 in response to a predetermined action made by the performer 14.

[0046] Such effect data may be stored in an effect data storage unit provided in advance in the storage unit 21 of the control / processing device 20, for example. Alternatively, the effect data may be stored in the virtual space display data storage unit 23 or the character data storage unit 24. Furthermore, the control / processing device 20 may include, as functional blocks, an effect generation unit (first effect generation unit) that reads effect data from the storage unit when a motion detection unit (first motion detection unit, e.g., a motion sensor) detects that the performer 14 has performed a predetermined motion and generates an effect such as a hit or an attack on the virtual screen 62 or the virtual space 60, and an effect generation unit (second effect generation unit) that reads effect data from the storage unit when a motion detection unit (second motion detection unit) detects that the player character 63 has performed a predetermined motion on the extracted image 64 of the target object through operation by the player via the player terminal and generates an effect on the extracted image on the virtual screen 62. Alternatively, the functions of these effect generation units may be performed by the virtual space generation unit 341 or the virtual screen display processing unit 342.

[0047] Except for the fact that the effects are executed via the virtual screen 62, the algorithms for generating the effects are similar to those used in conventional games played in virtual space, and therefore, the techniques used in those games may be used as appropriate. Here, a battle game has been used as an example, but a similar configuration can also be suitably used when creating video content for battle scenes. Of course, these effects are not limited to battle games and battle scenes, and a similar configuration can be used in the production of games other than battle games and video content other than battle scenes, with the necessary effects prepared in advance.

[0048] The above embodiment has been described as a specific example of a preferred embodiment of the imaging system 1 according to the present invention, and can be modified as appropriate in accordance with the spirit of the present invention.

[0049] In the above embodiment, lattice points 111 are provided on the background member 11, and the angle of view (wide angle, narrow angle) of the camera 12 is determined based on the number of lattice points 111 contained in the image captured by the camera 12, but the angle of view of the camera 12 may also be determined by other methods.

[0050] For example, by machine learning training data that identifies each part (head, hands, feet, etc.) contained in an image of a human (performer, etc.) photographed at various angles of view, a trained model can be constructed that outputs the angle of view of the camera at the time the image was taken in response to an input image of a human, and a classifier equipped with this trained model can be used as a camera angle of view acquisition unit.

[0051] Alternatively, as shown in Figure 7, a reference object of a predetermined size may be placed on a specific part of the performer (face, hands, feet, etc.), and a camera angle of view acquisition unit may be used to determine the camera angle of view based on the size of the reference object 71 (e.g., a sticker) included in the image of the performer. The reference object may be a circular sticker or accessory with a diameter of several centimeters. Alternatively, a sticker attached to the performer 14 may be used as the reference point 133 of the position sensor 13.

[0052] In the above embodiment, one performer 14 is photographed against the background member 11, but multiple performers 14 may be photographed as shown in Fig. 7. In this case, an extracted image 64 is created for each performer 14 and projected onto the virtual screen 62.

[0053] In the above embodiment, image data obtained by actually photographing the performers in the studio 100 is projected in real time onto the virtual screen 62 arranged in the virtual space, but image data of the performers may also be obtained in advance. In that case, along with the time-series data of the performer's images, information on the position and angle of view of the camera 12, the position of the performer 14, and the positions of both may be created as time-series data and stored in the shooting information storage unit 22. When creating video content, when playing back the time-series display data of the virtual space and the time-series display data of the characters in the virtual space 60, the time-series data of the performer's images, etc. stored in the shooting information storage unit 22 may be projected onto the virtual screen 62 in the virtual space 60, aligning the positions of the timestamps.

[0054] In the above embodiment, the performer 14 is photographed against a background member 11 (such as a green screen) placed in the studio 100, but the performer 14 can also be photographed without using the background member 11, and an extracted image of the performer 14 can be obtained from the photograph. Specifically, for example, the performer 14 can be photographed by focusing the camera 12 on the performer 14, and only the portion in focus can be extracted from the obtained image to obtain an extracted image. Using this method, it is possible to obtain an extracted image from an image of a performer such as an actor photographed outdoors (location shooting) without using a green screen, and to create video content that combines this extracted image with a virtual space in real time.

[0055] Furthermore, the configuration described below can be added to the above embodiment.

[0056] In the above embodiment, the performer 14 was photographed with one camera 12 and an extracted image 64 of the performer 14 was displayed on one virtual screen 62, but the performer 14 may be photographed with multiple cameras 12 and the extracted image 64 of the performer 14 obtained by each camera 12 may be displayed on a virtual screen 62 associated with each camera 12, and an image of the virtual space 60 including the extracted image 64 may be photographed with multiple virtual cameras 61.

[0057] Fig. 8 shows a state in which performers 141 and 142 are photographed by multiple cameras 121 to 123 in studio 100, in a plan view of studio 100. Fig. 9 shows a plan view of virtual space 60, in which multiple virtual cameras 611 to 613 and virtual screens 621 to 623 are arranged in virtual space 60.

[0058] As shown in Fig. 8, in studio 100, camera 121 simultaneously captures both of two performers 141 and 142 from the front at a wide angle. Camera 122 and camera 123 each capture one of performers 141 and 142 at a narrow angle from an oblique direction. Also, as shown in Fig. 9, in virtual space 60, a virtual camera 611 and a virtual screen 621 corresponding to camera 121 are arranged, a virtual camera 612 and a virtual screen 622 corresponding to camera 122 are arranged, and a virtual camera 613 and a virtual screen 623 corresponding to camera 123 are arranged. While studio 100 uses one background component 11, in virtual space 60, virtual screens 621 to 623 are arranged so as to directly face the virtual cameras 611 to 613, respectively.

[0059] 9 illustrates three sets of virtual cameras 611-613 and virtual screens 621-623. Each virtual camera 611-613 captures only the image displayed on the virtual screen 621-623 associated with that virtual camera. For example, virtual camera 612 captures only the extracted image 64 of performer 142 displayed on virtual screen 622, and does not capture the extracted images 64 of performers 141 and 142 displayed on virtual screens 621 and 623. This configuration eliminates mutual interference between the multiple virtual cameras 611-613 and the virtual screens 621-623 (the reflection of extracted images 64 displayed on other virtual screens). Since the virtual screen 62 used in this embodiment is an object virtually placed in the virtual space 60, as described above, such switching can be easily performed by appropriately configuring the settings in advance. Here, we have explained an example in which performers 141 and 142 performing simultaneously are photographed by multiple cameras 121 to 123, and virtual cameras 611 to 613 and virtual screens 621 to 623 corresponding to each camera 121 to 123 are arranged, but this can be changed as appropriate, such as photographing performer 14 performing individually with multiple cameras 12.

[0060] As described above, by shooting the performers 141 and 142 in parallel using the multiple cameras 121 to 123 and using the virtual cameras 611 to 613 and virtual screens 621 to 623 corresponding to the cameras 121 to 123, it is possible to instantly switch the angle of view for shooting the performances of the performers 141 and 142 (for example, an angle of view that captures both performers 141 and 142, an angle of view that shoots a close-up of the performer 141, or an angle of view that shoots a close-up of the performer 142). When creating video content, the director selects one of the multiple virtual cameras 611 to 613 using a predetermined input operation, and video content can be created in real time by combining images shot by the virtual cameras 611 to 613 selected at each point in time. Images taken by virtual cameras 611 to 613 selected by the director may also be displayed on screen 17. However, since screen 17 displays images for performer 14 to check the state of the interior of the virtual space, it is preferable to display images taken by a predetermined virtual camera that takes images of virtual space 60 at a wide angle (virtual camera 611 in the above example). Images taken by unselected virtual camera 61 may also be stored in storage unit 21. This allows video content to be created in real time, and then some of the images that make up the video content to be changed to images taken by unselected virtual cameras 61.

[0061] Here, the case where three cameras 121 to 123 are used has been described, but the number of cameras 12, virtual cameras 61, and virtual screens 62 can be changed as desired. Furthermore, if the background of the virtual space 60 is uniform (for example, the entire space is the ocean and there are no player characters or the like), only one set of virtual camera 61 and virtual screen 62 may be placed in the virtual space 60, and extracted images 64 of the performer 14 acquired by the multiple cameras 121 to 123 may be displayed on the virtual screen 62 in a switching manner.

[0062] In the above embodiment, the positional relationship between the camera 12 and the background member 11 in real space is reflected as is in the virtual camera 61 and the virtual screen 62 in virtual space, but this may be changeable. Specifically, for example, when the director performs an input operation through the input unit 41 of the control / processing device 20, the virtual screen display processing unit 342 may be configured to tilt the virtual screen 62 forward or backward from a reference position (the position of the virtual screen 62 that directly reflects the positional relationship between the camera 12 and the background member 11 in real space) or to change the surface shape of the virtual screen 62 (for example, to curve it).

[0063] Figure 10 shows an example in which the top of the virtual screen 62 is tilted forward and backward. The center of Figure 10 shows the virtual screen 62 placed at the reference position, the top shows an example in which the top of the virtual screen 62 is tilted forward (towards where the virtual camera 61 is located), and the bottom shows an example in which the top of the virtual screen 62 is tilted backward. As can be seen from the figure, by adopting the above configuration, it is possible to create video content in which an extracted image 64, which is an image of the actual performer 14 that has been arbitrarily modified, such as to make the performer 14 appear taller than in reality, is incorporated into a virtual space.

[0064] In the above embodiment, the virtual space generation unit 341 generated the entire virtual space as a three-dimensional space. However, generating the entire infinitely expanding virtual space as a three-dimensional space would result in a large processing load. Therefore, a space that is more than a predetermined distance away from the virtual camera 61 may be projected as a two-dimensional image onto a background screen positioned at that distance. For example, as shown in FIG. 11 , a hemispherical background screen 65 may be positioned around the position of the virtual camera 61, and an image of the space that is more than the predetermined distance away may be projected onto the hemispherical background screen 65. FIG. 11 illustrates a virtual space with a ground plane, so a hemispherical background screen 65 is used. However, if the virtual space is in the air or underwater, a spherical background screen 65 may be used. While FIG. 11 illustrates a hemispherical background screen, other shapes, such as a flat screen, may also be used. However, if the background screen has an edge, such as a flat screen, the edge is positioned outside the field of view of the virtual camera 61.

[0065] With the above configuration, it is not necessary to generate all of the three-dimensional objects to be placed in the virtual space, thereby reducing the processing load on the virtual space generation unit 341. When a viewer views an image captured by the virtual camera 61 or video content composed of the image, the viewer does not need to closely check objects located far from the virtual camera 61. Therefore, by appropriately determining the predetermined distance in consideration of the resolution of the image acquired by the virtual camera 61, the processing load on the virtual space generation unit 341 can be reduced without causing any sense of incongruity in the image or video.

[0066] In the above embodiment, the virtual space generation unit 341 generates one virtual space and creates video content by taking pictures in that virtual space using the virtual camera 61, but it is also possible to create a configuration using multiple virtual spaces. Specifically, video content is created using two virtual spaces that have the same virtual screen 62, player character 63, non-player characters, other objects, background configuration, etc. but different resolutions.

[0067] Specifically, a first virtual space is used in which avatars and objects are placed with low resolution (e.g., fewer polygons) and no surface texture is displayed, and a second virtual space is used in which avatars and objects are placed with higher resolution (e.g., more polygons) than the first virtual space and with surface texture displayed. Figure 12 shows an example of the first virtual space. In this configuration, first, in the first virtual space, the projection unit 36 ​​projects an extracted image 64 of the performer 14 onto the virtual screen 62, and the virtual shooting unit 37 shoots the virtual space. Then, from the obtained data, parameter information representing the positional information of the virtual camera 61 and virtual screen 62 in the virtual space, the position and posture of the player character 63, and the position and posture of the non-player characters is acquired. Once this information is acquired in the first virtual space, the virtual shooting unit 37 shoots the player character 63 and other objects in the second virtual space with polygons and textures of sufficient resolution required for the video content, using the parameter information acquired during shooting in the first virtual space, and creates (renders) the video content. The timing of creating the video content in the second virtual space is arbitrary. That is, as soon as parameter information is acquired in the first virtual space, video content may be created in the second virtual space, or video content may be acquired in the second virtual space after the parameter information necessary for creating video content is acquired in the first virtual space.

[0068] In the above embodiment, the virtual space generation unit 341 and the character display processing unit 35 only need to display player characters, non-player characters, etc. at low resolution, which reduces the processing load during shooting. When projecting a captured image of the performer 14 onto the virtual screen 62 in real time, if the processing load for generating the virtual space or processing the display data for the player character 63 is large, the performer 14 or player must wait until these processes are complete, which places a heavy burden on the performer 14 or player. In contrast, in the above embodiment, shooting is performed in the first virtual space, where the load for generating the virtual space or processing the display data for the player character 63 is small, which reduces the burden on the performer 14 or player.

[0069] The imaging system according to the present invention can be used to create various types of video, such as movies, television programs, commercial films (CM), games, and virtual reality (VR) content.

[0070] Aspects It will be apparent to those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0071] (Item 1) An imaging system according to one aspect of the present invention comprises: an imaging information storage unit that stores image data of an object captured by a camera in real space and positional information indicating the positional relationship between the object and the camera when the object is captured by the camera; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates the space in which content is captured; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that reads image data of the object from the imaging information storage unit and projects the image onto the virtual screen; and a virtual imaging unit that captures an image including the virtual screen, together with the virtual space as seen through the virtual screen, from a predetermined position in the virtual space that is determined based on the positional information.

[0072] In the imaging system according to paragraph 1, a shooting information storage unit stores image data of an object captured in real space by a real imaging unit and position information indicating the positional relationship between the object and the real imaging unit when the object was captured. The object captured here is, for example, an actor or other performer. The image of the object is obtained, for example, by capturing the object against a conventional green screen. Display data that will serve as the background of the content is also prepared in advance. A virtual space generation unit generates a virtual space using the display data. A virtual screen display processing unit places a transparent virtual screen in the virtual space, and an image display processing unit displays an image of the object on the virtual screen. A transparent virtual screen is a virtual object (screen) that is transparent except for the area where the image is displayed. The virtual capturing unit then captures an image including the virtual screen, along with the virtual space visible through the virtual screen, from a predetermined position determined based on the position information. In this way, the imaging system according to the present invention obtains an image in which the image of the object captured in real space is incorporated into the virtual space. In the imaging system according to paragraph 1, it is possible to capture an object in real space against, for example, a conventional green screen, eliminating the need for a large, expensive display with numerous LED light sources as in virtual production. Furthermore, the virtual space that serves as the background can be freely designed, and the size and position of the virtual screen placed therein can be easily changed. Therefore, the imaging system according to paragraph 1 can inexpensively capture images of an object captured at various angles of view.

[0073] (Clause 2) The photography system according to clause 2 is the photography system according to clause 1, wherein the photography information storage unit further stores data of an image of the object in association with information on the position of the object in the real space, and the virtual screen display processing unit further changes the position of the virtual screen based on the information on the position of the object.

[0074] In the imaging system according to the second aspect, the movement of the performer in the real space can be reflected in the virtual space.

[0075] (Clause 3) The photographing system according to clause 3 is the photographing system according to clause 1 or 2, wherein the virtual screen display processing unit changes the attitude or shape of the virtual screen in response to a predetermined input operation.

[0076] The photography system according to paragraph 3 makes it possible to create video content by making arbitrary changes to images of actual objects, such as making the objects appear larger (taller) than they actually are, and incorporating them into a virtual space.

[0077] (Clause 4) The photography system according to paragraph 4 is the photography system according to any one of paragraphs 1 to 3, wherein the photography information storage unit further stores data of the image of the object in association with information on the position of the camera in the real space, and the virtual photography unit changes the predetermined position according to the information on the position of the camera.

[0078] (Clause 5) The photography system according to clause 5 is the photography system according to any one of clauses 1 to 4, wherein the photography information storage unit further stores information on the angle of view of the camera in the real space in association with data on the image of the object, and the virtual photography unit changes the angle of view during photography in accordance with the information on the angle of view of the camera.

[0079] The photographing system according to the fourth and fifth aspects can perform photographing that reflects the position and angle of view of the camera when the image of the object is photographed.

[0080] (Clause 6) A photography system according to clause 6 is the photography system according to any one of clauses 1 to 5, wherein the virtual screen is placed directly opposite the virtual photography unit.

[0081] In the photography system according to paragraph 6, the virtual screen is positioned corresponding to the virtual photography unit, i.e., the optical axis of the virtual photography unit is positioned perpendicular to the virtual screen. The closer the optical axis of the virtual photography unit is to being parallel to the virtual screen, the less depth is perceived in the image of the object. However, by adopting the positioning described in paragraph 6, it becomes easier to perceive the depth of the object projected onto the virtual screen.

[0082] (Clause 7) The photography system according to clause 7 is the photography system according to any one of clauses 1 to 6, further comprising: a camera that photographs an object in real space; and an image extraction unit that extracts an area of ​​the object from image data of the object photographed by the camera, and the image display processing unit displays the extracted image of the object extracted by the image extraction unit on the virtual screen in real time.

[0083] In the imaging system according to paragraph 7, video content can be created in real time while imaging an object.

[0084] (Clause 8) The photography system according to clause 8 is the photography system according to any one of clauses 1 to 7, further comprising: a first sound collection unit attached to the object; and a first sound output unit that outputs sound data collected by the first sound collection unit from a predetermined position on the virtual screen.

[0085] In the photography system according to paragraph 8, if the object is a human or other sound-emitting entity, the sound emitted from the object can be recognized by the player controlling the player character in the virtual space.

[0086] (Clause 9) The photography system according to clause 9 is the photography system according to any one of clauses 1 to 8, further comprising: a second sound collection unit provided at a predetermined position on the virtual screen; and a second sound output unit attached to the object and outputting sound data collected by the second sound collection unit.

[0087] (Clause 10) The photography system according to clause 10 is the photography system according to any one of clauses 1 to 9, wherein the subject is a human being, and further comprises: a first movement detection unit that detects a predetermined movement by the subject; and a first effect generation unit that, when the predetermined movement is detected by the first movement detection unit, generates an effect associated with the predetermined movement in the virtual space or on the virtual screen.

[0088] (Clause 11) The photography system according to clause 11 is the photography system according to any one of clauses 1 to 10, further comprising: a player terminal that controls a player character that moves in the virtual space; a second movement detection unit that detects a predetermined movement of the player character operated through the player terminal with respect to the image of the object displayed on the virtual screen; and a second effect generation unit that, when the predetermined movement is detected by the second movement detection unit, generates an effect associated with the predetermined movement on the virtual screen at the position where the image of the object is displayed.

[0089] In the photography system according to paragraph 1, when the subject is a human, communication can be achieved between the human subject and the player character in the virtual space via the virtual screen. Furthermore, the human subject can interact with the virtual space. That is, starting from the virtual screen, an effect pre-associated with a predetermined action (such as vocalization or movement) by the human subject can be generated in the virtual space. Furthermore, an effect pre-associated with a predetermined action (such as vocalization or movement) by the player character in the virtual screen can be generated on the virtual screen.

[0090] Specifically, for example, the filming system according to paragraph 9 allows a player to perform an action while listening to sounds generated in the virtual space. The filming system according to paragraph 10 allows for greater interactivity with the player who controls the player character in the virtual space. Furthermore, the filming system according to paragraph 11 allows for greater interactivity between the player character operating in the virtual space and the object.

[0091] (Article 12) The photography system according to paragraph 12 is the photography system according to any one of paragraphs 1 to 11, further comprising a plurality of pairs of the virtual screen and the virtual photography unit associated with the virtual screen.

[0092] In the photography system according to paragraph 12, data of images of an object photographed in real space by multiple cameras (images of the same object photographed from different directions and / or angles of view, or images of different objects) is projected onto different virtual screens, and images of the virtual space including these images are photographed by a virtual photography unit associated with the virtual screens, thereby making it possible to obtain a wide variety of video content.

[0093] (Clause 13) The photography system according to clause 13 is the photography system according to any one of clauses 1 to 12, further comprising: a background screen provided in the virtual space at a position spaced a predetermined distance from the virtual photography unit; and the virtual space generation unit displays a space in the virtual space that is at least the predetermined distance from the virtual photography unit as a two-dimensional image on the background screen.

[0094] In the photography system according to the thirteenth aspect, it is not necessary to generate all three-dimensional objects to be placed in the virtual space, and therefore the processing load on the virtual space generation unit can be reduced.

[0095] (Clause 14) The photography system according to clause 14 is the photography system according to any one of clauses 1 to 13, wherein the virtual space generation unit generates a first virtual space and a second virtual space having a higher display resolution than the first virtual space, the first virtual space and the second virtual space are provided with the virtual screen and the image display processing unit, respectively, and the virtual photography unit photographs the first virtual space to obtain parameter information of objects present in the first virtual space, and then reflects the parameter information in the second virtual space and photographs the second virtual space.

[0096] In the imaging system according to paragraph 14, since imaging in the first virtual space requires only displaying objects at low resolution, the processing load on the virtual space generation unit can be reduced and parameter information and other information necessary for creating content can be efficiently acquired. Furthermore, by creating content by reflecting parameter information in the second virtual space, video content can be efficiently created in a high-resolution virtual space.

[0097] 1...Filming system 100...Studio 10...Filming device 11...Background member 111...Grid point 12, 121 to 123...Camera 13...Position sensor 131 to 133...Reference point 134...Main body 14, 141, 142...Performer 15...Sound collection unit 16...Earphones 17...Screen 20...Control / processing device 21...Memory unit 22...Filming information memory unit 23...Virtual space display data memory unit 24...Character data memory unit 25...Filming data memory unit 30...Content creation program 31...Image extraction unit 32...Camera angle of view acquisition unit 33...Position information acquisition unit 341...Virtual space generation unit 342...Virtual screen display processing unit 35...Character display processing unit 36...Projection unit 37...Virtual shooting unit 38...Audio processing unit 41...Input unit 42...Display unit 50...Player terminal 51...Memory unit 52... Input operation unit 53... Audio input unit 54... Audio output unit 55... Display unit 60... Virtual space 61, 611 to 613... Virtual cameras 62, 621 to 623... Virtual screen 63... Player character 64... Extracted image of actor projected on virtual screen 71... Reference object 9... Network

Claims

1. A photography system comprising: a photography information storage unit that stores image data of an object photographed in real space by a camera and positional information indicating the relative positions of the object and the camera when the object was photographed; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates the space in which content is photographed; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that reads image data of the object from the photography information storage unit and displays the image on the virtual screen; and a virtual photography unit that photographs an image including the virtual screen, together with the virtual space as seen through the virtual screen, from a predetermined position in the virtual space that is determined based on the positional information.

2. The photography system of claim 1, wherein the photography information storage unit further stores data of the image of the object in association with information on the position of the object in the real space, and the virtual screen display processing unit further changes the position of the virtual screen based on the information on the position of the object.

3. The photographing system according to claim 1, wherein the virtual screen display processing section changes the attitude or shape of the virtual screen in response to a predetermined input operation.

4. The photography system of claim 1, wherein the photography information storage unit further stores information on the position of the camera in the real space in association with data on the image of the object, and the virtual photography unit changes the predetermined position according to the information on the position of the camera.

5. The photography system of claim 1, wherein the photography information storage unit further stores information on the angle of view of the camera in the real space in association with data on the image of the object, and the virtual photography unit changes the angle of view during photography in accordance with the information on the angle of view of the camera.

6. The imaging system according to claim 1, wherein the virtual screen is placed directly opposite the virtual imaging unit.

7. The photographing system according to claim 1, further comprising: a camera that photographs an object in real space; and an image extraction unit that extracts an area of ​​the object from image data of the object photographed by the camera, wherein the image display processing unit displays the extracted image of the object extracted by the image extraction unit on the virtual screen in real time.

8. The photographing system of claim 1, further comprising: a first sound collection unit attached to the object; and a first sound output unit that outputs sound data collected by the first sound collection unit from a predetermined position on the virtual screen.

9. The photographing system described in claim 1, further comprising: a second sound collection unit provided at a predetermined position on the virtual screen; and a second sound output unit attached to the object and outputting sound data collected by the second sound collection unit.

10. The photographing system of claim 1, wherein the object is a human being, and further comprising: a first motion detection unit that detects a predetermined motion by the object; and a first effect generation unit that, when the predetermined motion is detected by the first motion detection unit, generates an effect associated with the predetermined motion in the virtual space or on the virtual screen.

11. The photography system of claim 1 further comprises: a player terminal for operating a player character operating in the virtual space; a second movement detection unit for detecting a predetermined movement of the player character operated through the player terminal with respect to the image of the object displayed on the virtual screen; and a second effect generation unit for, when the predetermined movement is detected by the second movement detection unit, generating an effect associated with the predetermined movement at the position on the virtual screen where the image of the object is displayed.

12. The photography system according to claim 1, characterized in that it comprises a plurality of pairs each consisting of the virtual screen and the virtual photography unit associated with the virtual screen.

13. The photography system of claim 1 further comprises a background screen provided in the virtual space at a position spaced a predetermined distance from the virtual photography unit, and the virtual space generation unit displays a space in the virtual space that is spaced at least the predetermined distance from the virtual photography unit as a two-dimensional image on the background screen.

14. The photography system of claim 1, characterized in that: the virtual space generation unit generates a first virtual space and a second virtual space having a higher display resolution than the first virtual space; the first virtual space and the second virtual space are provided with the virtual screen and the image display processing unit, respectively; and the virtual photography unit photographs the first virtual space to obtain parameter information of objects present in the first virtual space, and then reflects the parameter information in the second virtual space and photographs the second virtual space.

Images