Photography system

Abstract

Provided is a technique capable of acquiring, at low cost, images of an object captured at various shooting angles. 【Solution means】A shooting information storage unit (22) that stores data of an image of an object photographed by a camera in real space and position information representing the positional relationship between the two when the object is photographed by the camera; a virtual space generation unit (341) that generates a virtual space using display data prepared in advance and simulating the space where the content is photographed; a virtual screen display processing unit (342) that arranges a transparent virtual screen (62) in the virtual space; an image display processing unit (36) that reads out the data of the image of the object from the shooting information storage unit and displays the image on the virtual screen; and a virtual shooting unit (37) that, in the virtual space, shoots an image including the virtual screen, together with the virtual space visible through the virtual screen, from a predetermined position determined based on the position information. A shooting system (1) comprising:

Description

Technical Field

[0001] The present invention relates to a shooting system used for creating content such as movies and TV programs.

Background Art

[0002] In order to create content such as movies and TV programs, a shooting system using chroma key compositing is widely used. In a shooting system using chroma key compositing, an actor such as an actor is photographed with a camera against a so-called green screen colored with a predetermined color, and an image of the actor's area is extracted excluding the area of the predetermined color from the photograph. Then, the extracted image is combined with an image prepared separately by computer graphics (CG) or the like to create content.

[0003] Recently, a technique called virtual production has been proposed (for example, Patent Document 1). In virtual production, an image is projected onto a display composed of a large number of LED light sources, and an actor performing against the image is photographed with a camera. In this system, since the actor and the background are photographed at the same time, there is no need to perform an image compositing process, and content can be created in a short time. In this system, an image with a sense of depth is displayed on the display, and by changing the display mode of the image on the display according to the positional relationship between the camera and the display, content can be created as if the actor is performing in real space.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In virtual production, it is necessary to arrange the display so as to cover the entire area shown in the camera. Therefore, in order to acquire images of an object such as a performer captured at various angles of view, there has been a problem that a large and expensive screen having a huge number of LED light sources must be used.

[0006] The problem to be solved by the present invention is to provide a technique capable of acquiring at low cost images of an object captured at various angles of view.

Means for Solving the Problem

[0007] The imaging system according to the present invention made to solve the above problems includes: an imaging information storage unit that stores data of an image of an object captured by a camera in real space and position information representing the positional relationship between the two when the object is captured by the camera; a virtual space generation unit that generates a virtual space using display data prepared in advance and simulating the space where the content is captured; a virtual screen display processing unit that arranges a transmissive virtual screen in the virtual space; an image display processing unit that reads out the data of the image of the object from the imaging information storage unit and displays the image on the virtual screen; a virtual imaging unit that, in the virtual space, captures an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen and is characterized by comprising.

Effect of the Invention

[0008] In the imaging system according to the present invention, in the shooting information storage unit, data of an image of an object photographed by a camera in the real space and position information representing the positional relationship between the two when the object is photographed by the camera are stored. Here, the object to be photographed is, for example, an actor such as an actor. The image of the object is, for example, an image obtained by extracting a part necessary for synthesis from an image obtained by photographing the object against a green screen as in the conventional case (extracted image of the object). Further, display data serving as the background of the content is prepared in advance. The virtual space generation unit generates a virtual space using the display data. The virtual screen display processing unit arranges a transmissive virtual screen in the virtual space, and the image display processing unit displays the image of the object on the virtual screen. The transmissive virtual screen is a virtual object (screen) that functions as a reference plane when setting the imaging surface (focus) of the virtual photographing unit in the virtual space and is transparent outside the area where the image is displayed. The virtual photographing unit photographs an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen. In the imaging system according to the present invention, in this way, an image in which the image of the object photographed in the real space is incorporated into the virtual space is obtained. In the imaging system according to the present invention, in the real space, for example, the object may be photographed against a green screen used conventionally, and it is not necessary to use a large and expensive display having a large number of LED light sources as in virtual production. Further, the virtual space serving as the background can be freely designed, and the size and position of the virtual screen arranged therein can be easily changed. Therefore, in the imaging system according to the present invention, images of the object captured at various angles of view can be obtained at low cost.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0010] Embodiments of the photographing system according to the present invention will be described below with reference to the drawings.

[0011] FIG. 1 shows the main configuration of the imaging system 1 of the present embodiment. The imaging system 1 of the present embodiment includes an imaging device 10, a control / processing device 20, and a player terminal 50. The imaging device 10, the control / processing device 20, and the player terminal 50 are connected to be communicable with each other via a wired or wireless network 9. Although only one player terminal 50 is shown in FIG. 1, a plurality of player terminals 50 may be provided.

[0012] The imaging device 10 is disposed in a studio 100 where a performer 14, who is an imaging target, performs. The imaging device 10 includes a background member 11 called a green screen colored with a predetermined color (for example, green), a camera 12, a position sensor 13, and a sound collection unit 15. Further, an earphone 16 as a voice output unit is attached to the performer 14. Although the earphone 16 is used in the present embodiment, a speaker may be disposed in the studio 100 or the like. Furthermore, a screen 17 as a display unit is disposed at a position outside the field of view of the camera 12 in the studio 100, and an image captured by a virtual camera 61 provided in a virtual space 60 described later is displayed thereon in real time.

[0013] As shown in Fig. 2, the background member 11 is provided with grid-like reference points (grid points 111). In this embodiment, seven grid points 111 (a total of 49 grid points) are provided in the horizontal and vertical directions, respectively. The intervals and numbers of the grid points 111 can be arbitrarily changed. In this embodiment, the reference points are provided in a grid pattern, but the reference points may be provided two-dimensionally and regularly on the surface of the background member 11, and may be provided in a honeycomb pattern or the like. The camera 12 photographs an actor 14 or the like with the background member 11 as the background. The position sensor 13 of this embodiment is an infrared LiDAR (Light Detection and Ranging) sensor, which irradiates infrared light to the surroundings from the main body 134, and receives the reflected light from the reference points 131 and 132 provided at two locations of the camera 12 (for example, the tip of the lens and the main body of the camera 12) and the reference point 133 provided at one location of the actor 14 (for example, the neck muscle) to measure the positions of the respective reference points 131 to 133. For example, seals that reflect infrared light are attached to the reference points 131 to 133. The positions and numbers of the reference points 131 to 133 are also examples, and can be appropriately changed as long as the above functions can be achieved.

[0014] The control / processing device 20 includes a storage unit 21. In the storage unit 21, a photographed information storage unit 22, a virtual space display data storage unit 23, a character data storage unit 24, and a photographed data storage unit 25 are provided. In the photographed information storage unit 22, photographed information including data of an image of an object acquired in the real space, the posture of the camera 12 when the image was photographed, and the position information of the camera 12 and the performer 14 is stored. Note that the concept of an image in the present specification includes not only still images but also moving images (for example, a moving image composed of still images at 60 frames / second). In the virtual space display data storage unit 23, time-series display data in a 3D (three-dimensional) virtual space simulating various scenes, which is used for the content created in the photographing system 1, is stored in advance. This display data includes audio data such as sound effects. In the character data storage unit 24, time-series display data such as display data of a 3D avatar of a player character (PC) operated by a player and display data of a 3D avatar of a non-player character (NPC) not operated by the player is stored. As described above, since the non-player character is not operated by the player, the actions during the shooting of the video content are determined in advance according to the scenario, and the time-series display data (including audio data) is stored in the character data storage unit 24.

[0015] The control and processing device 20 includes, as functional blocks, an image extraction unit 31, a camera angle acquisition unit 32, a position information acquisition unit 33, a virtual space generation unit 341, a virtual screen display processing unit 342, a character display processing unit 35, an image display processing unit 36, a virtual photography unit 37, and an audio processing unit 38. The operations of these units will be described later. The entity of the control and processing device 20 is, for example, a general personal computer, which is realized by a processor executing a dedicated program (content creation program 30) installed in advance. The control and processing device 20 may be arranged at an appropriate location inside or outside the studio 100. Alternatively, as the control and processing device 20, a cloud server provided on a network may be used. Further, an input unit 41 composed of a keyboard, a mouse, etc. and a display unit 42 composed of a liquid crystal display, etc. are connected to the control and processing device 20.

[0016] The player terminal 50 includes a storage unit 51. Information of the player who uses the player terminal 50 (such as the player's ID information) is stored in the storage unit 51. The player terminal 50 includes an input operation unit 52, an audio input unit 53, an audio output unit 54, and a display unit 55. For the player terminal 50, for example, a head-mounted display equipped with a motion capture or a position sensor that functions as the input operation unit 52 can be used. The player terminal 50 can be provided at an arbitrary location. That is, the player can operate the player character from an arbitrary location inside or outside the studio 100.

[0017] Next, a procedure for creating video content using the shooting system 1 of this embodiment will be described.

[0018] When creating video content, first set the space that will serve as the background of the content in a 3D virtual space, create time-series data for generating a 3D virtual space that changes over time according to a scenario, and save it in the virtual space display data storage unit 23. Also, save time-series display data such as the display data of the 3D avatar of the player character that appears in the content and the 3D avatar of the non-player character in the character data storage unit 24. A player character is, for example, a character that appears in the content and has contact such as talking to an actor. A non-player character is, for example, a so-called mob character that appears as a crowd in the background of the content, an enemy character that operates based on a predetermined algorithm and fights against an actor or a player character, or an object placed at a predetermined position in the virtual space. When there are multiple player characters, information associated with the information of the player terminal 50 used by the player who operates each player character (such as the ID information of the terminal) is also saved for each player character. Hereinafter, for the sake of convenience, the display mode of a non-player character that is an object is also referred to as an avatar.

[0019] When the above preparations are completed, the preparation for shooting video content is started. The preparation for shooting video content is carried out in both the studio 100 provided in the real space and the virtual space 60.

[0020] First, the preparation for photographing the performer 14 in the studio 100 set in the real space will be described. In the studio 100, a background member 11 called a green screen is placed. Reference points 131 and 132 of the LiDAR sensor as the position sensor 13 are set on the lens and main body of the camera 12, respectively. In addition, a reference point 133 of the LiDAR sensor is set on a predetermined position of the performer 14. In this embodiment, stickers that reflect infrared light are attached to the lens and main body of the camera 12 and the neck of the performer 14. A main body 134 of the LiDAR sensor is placed at a predetermined position (for example, a corner of the ceiling) of the studio 100. The operating principle of LiDAR is conventionally known, so a detailed description will be omitted here. Furthermore, a sound collection unit 15 that collects the sound emitted by the performer 14 and a screen 17 are placed at a position outside the angle of view of the camera 12 in the studio 100. Furthermore, the performer 14 is made to wear earphones 16.

[0021] The camera 12 photographs the performer 14 standing in an initial position with the background member 11 as the background, and acquires image data of the performer 14. The image data of the performer 14 acquired by the camera 12 is stored in the shooting information storage unit 22 together with a timestamp indicating the time when the image was acquired (time of shooting, time elapsed since the start of the performance, etc.). In addition, sounds made by the performer 14 during the performance are collected by the sound collection unit 15 and stored in the shooting information storage unit 22.

[0022] 3 and 4 show examples of photographing the performer 14 in the studio 100. FIG. 3 shows the state in which the performer 14 is photographed with the wide-angle lens of the camera 12, and FIG. 4 shows the state in which the performer 14 is photographed with the narrow-angle lens of the camera 12. When the performer 14 is photographed with the wide-angle lens of the camera 12 as in FIG. 3, an image capturing the entire body of the performer 14 is obtained as shown in FIG. 2. When the performer 14 is photographed with the narrow-angle lens of the camera 12 as in FIG. 4, an enlarged image capturing a part of the performer 14 (the upper body in this example) is obtained as shown in FIG. 5. As shown in FIGS. 4 and 5, the image of the performer 14 also captures the lattice points 111 provided on the background member 11.

[0023] In the control and processing device 20, the image extraction unit 31 reads out the image data stored in the photographing information storage unit 22. Then, it extracts the image data of the image excluding the area of a predetermined color (the part of the green background, including the grid points 111) included in the image. As a result, an image (extracted image) in which the part of the performer 14 is cut out is generated. The generated extracted image is also stored in the photographing information storage unit 22.

[0024] The camera angle acquisition unit 32 obtains the number of grid points 111 included in the image stored in the photographing information storage unit 22 (in this embodiment, the number of grid points 111 arranged vertically and the number of grid points 111 arranged horizontally. This number includes the number of grid points 111 hidden by the performer 14). Then, based on the number of grid points 111, it obtains the angle of view (wide angle, narrow angle, etc.) of the lens of the camera 12 when the image was taken. When there is no change in the positions of the camera 12 and the performer 14, the larger the number of grid points included in the image, the wider the lens of the camera 12, and the smaller the number of grid points included in the image, the narrower the lens of the camera 12 (in a zoomed state). Also, the camera angle acquisition unit 32 determines the positions of the reference points 131 and 132 set for the camera 12 based on the output signal from the position sensor 13, and determines the attitude (elevation angle θ) of the camera 12 from the relative positional relationship between the two.

[0025] The position information acquisition unit 33 determines the positions of the reference point 131 (or / and 132) set for the camera 12 and the reference point 133 set for the performer 14 based on the output signal from the position sensor 13, and determines their positional relationship. The information on their positional relationship is information on the direction in which the camera 12 captures the performer 14 and information on the distance between the camera 12 and the performer 14 (i.e., vectorial information). The information on the attitude of the camera acquired by the camera angle acquisition unit 32 and the information on the positional relationship between the camera 12 and the performer 14 acquired by the position information acquisition unit 33 are each stored in the photographing information storage unit 22 together with a time stamp indicating the time when the information was acquired.

[0026] The processes performed by the above-described image extraction unit 31, camera angle acquisition unit 32, and position information acquisition unit 33 are executed in real time during the shooting of the performer 14 in the studio 100.

[0027] Next, the shooting preparation in the virtual space will be described.

[0028] First, the virtual space generation unit 341 reads out the time-series display data of the 3D virtual space where the video content is shot from the virtual space display data storage unit 23, and generates a virtual space based on the display data at the start time point. Further, as illustrated in FIG. 6, the virtual screen display processing unit 342 arranges a transmissive virtual screen 62 at a predetermined initial position inside the virtual space. Note that the transmissive virtual screen 62 is a virtual object (screen) that can function as a reference plane when setting the imaging surface (focus) of the virtual photographing unit 37 in the virtual space 60, and the area other than the area where the image is projected is transparent. Since the virtual screen 62 is a virtual object, the player character and the non-player character can freely pass through the virtual screen 62.

[0029] The character display processing unit 35 reads out the display data of the avatar of the player character and the avatar of the non-player character, and arranges them at the initial positions in the virtual space. The virtual photographing unit 37 arranges a virtual camera 61 in the virtual space 60 based on the information on the positional relationship between the camera 12 and the performer 14 obtained by the position information acquisition unit 33. Specifically, the position of the performer 14 is made to correspond to the position of the virtual screen 62, and based on this, the relative positional relationship between the camera 12 and the performer 14 is reflected in the positional relationship between the virtual screen 62 and the virtual camera 61 to determine the position of the virtual camera 61. In the present embodiment, the virtual camera 61 is arranged facing the virtual screen 62. That is, the optical axis of the virtual camera 61 and the surface of the virtual screen 62 are arranged perpendicular to each other. Further, the size of the virtual screen 62 is appropriately changed according to the size of the extracted image 64 of the performer 14 displayed on the virtual screen 62.

[0030] The image display processing unit 36 displays the extracted image 64 of the performer 14 extracted by the image extraction unit 31 on the virtual screen 62. The image display processing unit 36 of the present embodiment displays the extracted image 64 of the performer 14 on the virtual screen 62 by projecting the extracted image 64 of the performer 14 from the position of the virtual camera 61 toward the virtual screen 62. Since the relative positional relationship between the virtual camera 61 and the virtual screen 62 reflects the positional relationship between the camera 12 and the screen 17, by projecting the extracted image 64 of the performer 14 from the virtual camera 61 toward the virtual screen 62, an image capturing the performer 14 at the same angle of view as that captured by the camera 12 in the real space can be displayed on the virtual screen 62. However, the method of displaying the extracted image 64 of the performer 14 on the virtual screen 62 is not limited to this, and various methods can be adopted, such as transmitting the pixel data constituting the extracted image 64 to the projection screen 62. In this way, the actual image of the performer 14 captured in the real space is incorporated into the virtual space.

[0031] As described above, since the virtual screen 62 is of a transmissive type, as shown in FIG. 6, the virtual camera 61 simultaneously captures the extracted image 64 of the performer 14 projected on the virtual screen 62 and the virtual space 60 located in the background of the virtual screen 62. The virtual imaging unit 37 captures an image of the virtual space 60 at a predetermined frame rate (for example, 60 frames per second) by the virtual camera 61. The data of the captured image is stored in the captured data storage unit 25 and is displayed on the screen 17 of the studio 100 in real time.

[0032] The virtual space generation unit 341 generates an image capturing the virtual space from the position of the player character 63 placed at the initial position, and transmits it to the player terminal 50 used by the player who operates the player character 63. On the player terminal 50, the received image is displayed on the display unit 55. Although only one player character 63 is shown in FIG. 6, there may be a plurality of player characters 63 existing simultaneously. As shown in FIG. 6, when the player character 63 is located behind the virtual screen 62, the silhouette of the extracted image 64 of the performer 14 is displayed on the display unit 55. When the player character 63 is located on the front side of the virtual screen 62 (the side where the virtual camera 61 is located), an image of the extracted image 64 of the performer 14 as seen from the front is displayed on the display unit 55.

[0033] After the shooting preparations are completed in the studio 100 and the virtual space 60 provided in the real space as described above, the director instructs the start of shooting by a predetermined operation (for example, instructing the start of shooting by voice in the studio 100 and simultaneously pressing the shooting start button through the input unit 41). In response to this, in the studio 100, the performer 14 starts acting according to a pre-prepared scenario. The performance of the performer 14 includes those that emit sound, and the sound emitted by the performer 14 is collected by the sound collection unit 15 and transmitted to the control and processing device 20 (corresponding to the function of the first sound collection unit in the present invention). In the control and processing device 20, the sound processing unit 38 outputs the sound received from the sound collection unit 15 as sound emitted from a predetermined position of the virtual screen 62 in the virtual space 60 (for example, the center of gravity position of the extracted image 64) (corresponding to the function of the first sound output unit in the present invention).

[0034] In addition, the virtual space generation unit 341 and the character display processing unit 35 each read out the time-series display data of the virtual space and the time-series display data of the non-player character from the virtual space display data storage unit 23 and the character data storage unit 24, respectively, and sequentially reflect (play back) them in the virtual space 60. This display data also includes audio data, and the audio data is played back as audio emitted from the position of the non-player character in the virtual space.

[0035] On the player terminal 50, the player operates the player character 63 through the input operation unit 52 and emits sounds according to a pre-prepared scenario. The operations input by the player through the input operation unit 52 are sequentially transmitted to the control and processing device 20 and reflected in the operations of the player character 63 in the virtual space. Also, the sounds emitted by the player are collected by the audio input unit 53 and sequentially transmitted to the control and processing device 20. In the control and processing device 20, the audio processing unit 38 outputs the received audio as audio emitted from the position of the player character 63 in the virtual space 60.

[0036] The voice processing unit 38 collects voices in the virtual space 60 at a predetermined position of the virtual screen 62 (for example, the center-of-gravity position of the extracted image 64) (corresponding to the function of the second sound collection unit in the present invention). The collected voices are transmitted to the imaging device 10 and output from the earphone 16 worn by the performer 14 (corresponding to the function of the second voice output unit in the present invention). Thereby, the performer 14 can visually recognize the state in the virtual space 60 on the screen 17, and perform acting while listening to the voices uttered in the virtual space 60 and captured at the position of the virtual screen 62 through the earphone 16. Further, the voice processing unit 38 also collects voices in the virtual space at the position of the player character 63 in the virtual space 60. The collected voices are transmitted to the player terminal 50 used by the player who operates the player character 63 and output from the voice output unit 54. Thereby, the player who operates the player character 63 can also visually recognize the state in the virtual space 60 from the viewpoint of the player character 63 by the display unit 55, and operate the player character 63 while listening to the voices uttered in the virtual space 60 and captured at the position of the player character 63 (including the voice of the performer 14 emitted from the predetermined position of the virtual screen 62) through the voice output unit 54.

[0037] During imaging, the positions of the camera 12 and the performer 14 are monitored by the position information acquisition unit 33 based on the output signal from the position sensor 13. When the camera 12 or the performer 14 moves (changes its position and / or posture), the control and processing device 20 detects the information by the position information acquisition unit 33. The position of the camera 12 is associated with the position of the virtual camera 61. When the movement of the camera 12 is detected by the position information acquisition unit 33, the virtual imaging unit 37 moves the virtual camera 61 arranged in the virtual space 60 in accordance with the movement of the camera 12. The position of the performer 14 is associated with the position of the virtual screen 62. When the movement of the performer 14 is detected by the position information acquisition unit 33, the virtual screen display processing unit 342 moves the virtual screen 62 arranged in the virtual space 60 in accordance with the movement of the performer 14.

[0038] Also, the angle of view of the camera 12 is monitored by the camera angle-of-view acquisition unit 32. The change in the angle of view of the camera 12 is detected based on the number of grid points included in the image captured by the camera 12. The information on the angle of view of the camera 12 acquired by the camera angle-of-view acquisition unit 32 is reflected in the angle of view of the virtual camera 61 by the virtual imaging unit 37. For example, when the camera 12 changes its angle of view to capture the performer 14 in an enlarged manner, in the virtual space as well, the angle of view of the virtual camera 61 is changed so as to capture the extracted image of the performer 14 projected on the virtual screen 62 in a zoomed manner. Therefore, the same state as the performer 14 captured by the camera 12 in the real space is captured by the virtual camera 61 in the virtual space.

[0039] When the performance, etc. of the performer 14 based on a pre-prepared scenario is completed, the director instructs the end of shooting by a predetermined operation (for example, instructing the end of shooting by voice in the studio 100 and simultaneously pressing the shooting end button through the input unit 41). As a result, the virtual space generation unit 341 and the character display processing unit 35 each end the reproduction of the time-series display data of the virtual space 60 and the reproduction of the time-series display data of the avatars of the non-player characters. Also, on the player terminal 50, the operation of the player character 63 by the player is ended. The virtual imaging unit 37 stores the data of the images acquired by the virtual camera 61 at a predetermined frame rate as one video file in the shooting data storage unit 25.

[0040] In the imaging system 1 of the present embodiment, by displaying the data of the image of the performer 14 performing in the studio 100 provided in the real space on the transmissive virtual screen 62 arranged in the virtual space 60, video content incorporating the real image of the performer 14 into the virtual space can be created. In the imaging system 1 of the present embodiment, in the studio 100 in the real space, an object may be photographed with a green screen as the background, and an extraction image 64 obtained by extracting the image of the performer 14 by a conventionally known process may be acquired. There is no need to use a large and expensive display having a large number of LED light sources as in virtual production. Further, the background of the video content is the virtual space, and the position and size of the virtual screen 62 on which the real extraction image 64 is projected can be easily changed according to the angle of view of the camera 12, the position of the performer 14, and the relative positional relationship between the camera 12 and the performer 14. Therefore, videos capturing an object at various angles of view can be acquired at low cost.

[0041] In the imaging system 1 of the present embodiment, video content in which a player character 63 and a non-player character operated by the player by the player terminal 50 and the performer 14 photographed in real are co-starred can be created.

[0042] Conventionally, when creating a combination of an image or video of a performer and a separately prepared background image or the like, only the layer of the image of the performer created as one image file was combined with the background image. That is, for example, when combining a cell drawing depicting a character or the like and a background image in the production of classical animation, it was merely a composite of two independent images. Therefore, for example, the line of sight or posture of the performer may not match the background image, resulting in a sense of incongruity in the synthesized image.

[0043] In contrast, in the imaging system 1 of the present embodiment, an image captured by a virtual camera 61 arranged in the virtual space 60 is displayed in real time on a screen 17 arranged in the studio 100. Also, the voice emitted by the performer 14 is reflected in the virtual space, and the voice in the virtual space is output from the earphone 16 of the performer 14 at the position of the virtual screen 62 arranged in the virtual space. Therefore, an interactive exchange (such as conversation) can be carried out between the performer 14 and the player character 63 or non-player character located in the virtual space 60. Also, the shadow of the performer 14 can be made to appear in the virtual space 60 in real time, or the voice emitted by the player character 63 or the like in the virtual space 60 can be made audible to the performer 14 at a volume corresponding to the distance between the player character 63 and the performer 14. Therefore, the performer 14 can perform acting while experiencing the scenario. Also, since post-processing such as image synthesis is unnecessary, the cost associated with such processing can also be reduced.

[0044] In the above embodiment, the case of creating video content by photographing the acting of the performer 14 or the player character 63 has been described. However, the same configuration as described above can also be adopted for playing a game in the virtual space and photographing the play scene. And when playing the game, a configuration with even higher interactivity than the above embodiment can be adopted.

[0045] For example, in a battle game, when an attack made by the player character 63 hits the extracted image 64 of the performer 14, it can be configured to generate a predetermined effect (e.g., blood splatter) at the location where the attack hits on the virtual screen 62. Alternatively, conversely, when the performer 14 attacks the player character 63, it can be configured to cause an attack on the player character 63 from the display position of the extracted image 64 of the performer 14 on the virtual screen 62 (e.g., generate an image of firing a bullet from the position of the extracted image 64 in response to the shooting action by the performer 14). Furthermore, not only between the player character 63 and the performer 14, but also in response to the performer 14 performing a predetermined action, it can be configured to cause a predetermined effect (e.g., fade out the display of the virtual space 60) in the virtual space 60.

[0046] Data of such effects can be, for example, stored in an effect data storage section provided in advance in the storage section 21 of the control and processing device 20. Alternatively, it may be stored in the virtual space display data storage section 23 or the character data storage section 24. Also, as a functional block of the control and processing device 20, at the timing when the motion detection section (first motion detection section. For example, a motion sensor) detects that the performer 14 has performed a predetermined action, the effect data is read from the storage section, and an effect generation section (first effect generation section) that generates effects such as hits or attacks on the virtual screen 62 or in the virtual space 60, or at the timing when the motion detection section (second motion detection section) detects that the player has performed a predetermined action on the extracted image 64 of the object by the player character 63 through an operation using the player terminal, the effect data is read from the storage section, and an effect generation section (second effect generation section) that generates an effect on the extracted image on the virtual screen 62 may be provided. Alternatively, the functions of these effect generation sections may be executed by the virtual space generation section 341 or the virtual screen display processing section 342.

[0047] Except for the point that the effect is executed via the virtual screen 62, the algorithm itself for generating the effect is the same as that of a conventional game implemented in a virtual space. Therefore, the technologies used in those can be appropriately used. Here, a battle game was used as an example for explanation, but the same configuration can be preferably used when creating video content for a battle scene. Of course, such effects are not limited to battle games or battle scenes. In the production of games other than battle games or video content other than battle scenes, after preparing the necessary effects in advance, the same configuration as above can be adopted.

[0048] The above embodiment has described a preferred embodiment of the imaging system 1 according to the present invention with a specific example, and it can be appropriately changed along the gist of the present invention.

[0049] In the above embodiment, the grid points 111 are provided on the background member 11, and the angle of view (wide angle, narrow angle) of the camera 12 is determined based on the number of grid points 111 included in the image captured by the camera 12. However, the angle of view of the camera 12 may be determined by other methods.

[0050] For example, by machine learning teacher data that identifies each part (head, hands, feet, etc.) included in images of humans (actors, etc.) taken at various angles of view, a learned model that outputs the angle of view of the camera when the image was taken in response to the input of an image of a human being is constructed, and an identifier equipped with this learned model can be used as the camera angle acquisition unit.

[0051] Alternatively, as shown in FIG. 7, a reference object of a predetermined size is arranged in advance at a predetermined part (face, hands, feet, etc.) of the actor, and a camera angle acquisition unit that determines the camera angle based on the size of the reference object 71 (for example, a sticker) included in the image of the actor can also be used. As the reference object, a circular sticker with a diameter of several cm, an accessory, etc. can be used. Alternatively, a sticker attached to the actor 14 may be used as the reference point 133 of the position sensor 13 as the reference object.

[0052] In the above embodiment, one performer 14 was photographed with the background member 11 as the background. However, as shown in FIG. 7, a plurality of performers 14 may be photographed. In that case, an extracted image 64 may be created for each performer 14 and projected onto the virtual screen 62.

[0053] In the above embodiment, the image data obtained by actually photographing a performer in the studio 100 was projected in real time onto the virtual screen 62 arranged in the virtual space. However, the image data of the performer may be acquired in advance. In that case, together with the time-series data of the performer's image, the position and angle of view of the camera 12, the position of the performer 14, and the information on the positions of both may also be created as time-series data and stored in the photographing information storage unit 22. Then, when creating video content, when playing back the time-series display data of the virtual space and the time-series display data of the character in the virtual space 60, the time-series data of the performer's image and the like stored in the photographing information storage unit 22 may be projected onto the virtual screen 62 in the virtual space 60 by matching the position of the time stamp.

[0054] In the above embodiment, the performer 14 was photographed with the background member 11 (such as a green screen) arranged in the studio 100 as the background. However, it is also possible to photograph the performer 14 without using the background member 11 and obtain an extracted image of the performer 14 therefrom. Specifically, for example, the performer can be photographed with the focus of the camera 12 adjusted to the performer 14, and the extracted image can be obtained by extracting only the in-focus portions from the obtained image. By using such a method, an extracted image can be obtained from an image of an actor such as an actor photographed outdoors (location shooting) without using a green screen, and video content combined with the virtual space in real time can be created.

[0055] Also, the following configuration can be added to the above embodiment.

[0056] In the above-described embodiment, the extraction image 64 of the performer 14 obtained by photographing the performer 14 with one camera 12 is displayed on one virtual screen 62. However, the performer 14 may be photographed with a plurality of cameras 12, and the extraction images 64 of the performer 14 obtained by each camera 12 may be displayed on the virtual screens 62 associated with the respective cameras 12. Further, an image of the virtual space 60 including the extraction image 64 may be photographed with a plurality of virtual cameras 61.

[0057] Fig. 8 shows, in plan view of the studio 100, a state in which the performers 141 and 142 are photographed with a plurality of cameras 121 to 123 in the studio 100. Fig. 9 shows, in plan view of the virtual space 60, a plurality of virtual cameras 611 to 613 and virtual screens 621 to 623 arranged in the virtual space 60.

[0058] As shown in Fig. 8, in the studio 100, the camera 121 simultaneously captures both of the two performers 141 and 142 from the front with a wide angle. The cameras 122 and 123 respectively capture one of the performers 141 and 142 from an oblique direction with a narrow angle. Also, as shown in Fig. 9, in the virtual space 60, a virtual camera 611 and a virtual screen 621 corresponding to the camera 121 are arranged, a virtual camera 612 and a virtual screen 622 corresponding to the camera 122 are arranged, and a virtual camera 613 and a virtual screen 623 corresponding to the camera 123 are arranged. Although there is one background member 11 used in the studio 100, in the virtual space 60, the virtual screens 621 to 623 are arranged so as to face the respective virtual cameras 611 to 613.

[0059] In Fig. 9, three sets of virtual cameras 611 to 613 and virtual screens 621 to 623 are illustrated. In each of the virtual cameras 611 to 613, only the images displayed on the virtual screens 621 to 623 associated with the virtual camera are captured. For example, in the virtual camera 612, only the extracted image 64 of the performer 142 displayed on the virtual screen 622 is captured, and the extracted images 64 of the performers 141 and 142 displayed on the virtual screens 621 and 623 are not captured. By adopting such a configuration, mutual interference (the reflection of the extracted image 64 displayed on another virtual screen) between the plurality of virtual cameras 611 to 613 and the virtual screens 621 to 623 can be eliminated. Since the virtual screen 62 used in the present embodiment is an object virtually arranged in the virtual space 60 as described above, such switching can be easily performed by appropriately setting in advance. Here, an example in which the performers 141 and 142 performing the performance simultaneously are captured by the plurality of cameras 121 to 123, and the virtual cameras 611 to 613 and the virtual screens 621 to 623 corresponding to the respective cameras 121 to 123 are arranged has been described. However, it can be appropriately changed, such as capturing the performer 14 performing the performance individually by the plurality of cameras 12.

[0060] As described above, by simultaneously photographing the performers 141 and 142 with the plurality of cameras 121 to 123 and using the virtual cameras 611 to 613 and the virtual screens 621 to 623 corresponding to the respective cameras 121 to 123, it is possible to instantaneously switch the shooting angle (for example, an angle that captures both performers 141 and 142, an angle that shoots performer 141 in close-up, an angle that shoots performer 142 in close-up) for photographing the performances of the performers 141 and 142. When creating video content, the director can select any one of the plurality of virtual cameras 611 to 613 by a predetermined input operation, and can create in real time video content that combines the images photographed by the virtual cameras 611 to 613 selected at each time point. Although the images photographed by the virtual cameras 611 to 613 selected by the director may also be displayed on the screen 17, since the screen 17 is for displaying an image for the performer 14 to confirm the internal state of the virtual space, it is preferable to display an image photographed by a virtual camera (virtual camera 611 in the above example) that is predetermined and shoots the virtual space 60 at a wide angle. Further, it is preferable to also store the images photographed by the non-selected virtual camera 61 in the storage unit 21. Thereby, after creating video content in real time, a part of the images constituting the video content can be changed to the images photographed by the non-selected virtual camera 61.

[0061] Here, the case of using three cameras 121 to 123 etc. has been described, but the numbers of the cameras 12, the virtual cameras 61, and the virtual screens 62 can be arbitrarily changed. Further, when the background of the virtual space 60 is uniform (for example, the entire space is the sea and there are no player characters etc.), only one set of the virtual camera 61 and the virtual screen 62 may be arranged in the virtual space 60, and the extracted images 64 of the performer 14 acquired by the plurality of cameras 121 to 123 may be switched and displayed on the virtual screen 62.

[0062] In the above-described embodiment, the positional relationship between the camera 12 and the background member 11 in the real space is directly reflected in the virtual camera 61 and the virtual screen 62 in the virtual space, but this may be made changeable. Specifically, for example, when the director performs an input operation through the input unit 41 of the control and processing device 20, the virtual screen display processing unit 342 tilts the virtual screen 62 back and forth from the reference position (the position of the virtual screen 62 that directly reflects the positional relationship between the camera 12 and the background member 11 in the real space), or changes the surface shape of the virtual screen 62 (for example, makes it curved).

[0063] FIG. 10 shows an example in which the upper part of the virtual screen 62 is tilted back and forth. The center of FIG. 10 shows the virtual screen 62 placed at the reference position. The upper row shows an example in which the upper end of the virtual screen 62 is tilted forward (the side where the virtual camera 61 is located), and the lower row shows an example in which the upper end of the virtual screen 62 is tilted backward. As can be seen from the figure, by adopting the above configuration, video content can be created in which an extracted image 64 with an arbitrary change made to the image of the actual performer 14, such as making the performer 14 appear taller than actual, is incorporated into the virtual space.

[0064] In the above-described embodiment, the virtual space generation unit 341 generates the entire virtual space as a three-dimensional space. However, generating the entire infinitely expanding virtual space as a three-dimensional space increases the processing load. Therefore, a space that is at a predetermined distance or more away from the virtual camera 61 may be projected as a two-dimensional image onto a background screen arranged at that distance. For example, as shown in FIG. 11, a hemispherical background screen 65 is arranged centered on the position of the virtual camera 61, and an image of the space that is at the predetermined distance or more away can be projected thereon. Since FIG. 11 shows a virtual space where the ground exists, a hemispherical background screen 65 is used. However, when the virtual space is in the air, water, etc., a spherical background screen 65 may be used. Note that in FIG. 11, a hemispherical background screen is shown, but it is not limited to a hemispherical shape, and other shaped background screens such as a flat plate may be used. However, in the case of a shape having an edge such as a flat plate, the background screen is arranged so that the edge is located outside the field of view of the virtual camera 61.

[0065] In the above configuration, since it is not necessary to generate all the three-dimensional objects arranged in the virtual space, the processing load on the virtual space generation unit 341 can be reduced. When a viewer views an image captured by the virtual camera 61 or video content composed of the image, the viewer does not carefully check an object located far away from the virtual camera 61. Therefore, by appropriately determining the above-mentioned predetermined distance in consideration of the resolution of the image acquired by the virtual camera 61, etc., the processing load on the virtual space generation unit 341 can be reduced without causing a sense of incongruity in the image or video.

[0066] In the above embodiment, the virtual space generation unit 341 generates one virtual space and creates video content by shooting using the virtual camera 61 in the virtual space. However, a configuration using a plurality of virtual spaces can also be adopted. Specifically, video content is created using two virtual spaces in which the virtual screen 62, the player character 63, the non-player character, other objects, the configuration of the background, etc. are the same but their resolutions are different.

[0067] Specifically, a first virtual space in which avatars, objects, etc. with low resolution (e.g., few polygons) and with the display of surface textures omitted are arranged, and a second virtual space with higher resolution (e.g., more polygons) than the first virtual space and in which avatars, objects, etc. with textures displayed on the surface are arranged are used. FIG. 12 shows an example of the first virtual space. In this mode, first, in the first virtual space, the projection unit 36 projects the extracted image 64 of the performer 14 onto the virtual screen 62, and the virtual photographing unit 37 executes photographing of the virtual space. Then, from the obtained data, parameter information representing the position information of the virtual camera 61 and the virtual screen 62, the position and orientation of the player character 63, the position and orientation of the non-player character, etc. in the virtual space is acquired. When this information is obtained in the first virtual space, the virtual photographing unit 37 uses the parameter information acquired during photographing in the first virtual space to execute photographing and create (render) video content in the second virtual space in which the player character 63, etc. are arranged with polygons and textures of sufficient resolution required for the video content. The timing for creating video content in the second virtual space is arbitrary. That is, as soon as the parameter information is obtained in the first virtual space, the video content may be created in the second virtual space in sequence, or after the parameter information necessary for creating the video content is obtained in the first virtual space, the video content may be obtained in the second virtual space..

[0068] In the above-described aspect, since the virtual space generation unit 341 and the character display processing unit 35 may display the player character, non-player character, etc. at a low resolution, the processing load during shooting can be suppressed. When projecting the image of the performer 14 onto the virtual screen 62 in real time, if the processing load when generating the virtual space or processing the display data of the player character 63 is large, the performer 14 and the player have to wait until those processes are completed, imposing a large burden on the performer 14 and the player. In contrast, in the above-described aspect, since shooting is performed in the first virtual space with a small load for generating the virtual space or processing the display data of the player character 63, the burden on the performer 14 and the player can be reduced.

Industrial Applicability

[0069] The imaging system according to the present invention can be used to create various types of videos such as movies, television programs, commercial films (CM), games, virtual reality (VR) contents, and the like.

[0070] [Aspect] It is obvious to those skilled in the art that the above-described exemplary embodiments are specific examples of the following aspects.

[0071] (Item 1) An imaging system according to an aspect of the present invention includes an imaging information storage unit that stores data of an image of an object photographed by a camera in a real space and position information representing the positional relationship between the two when the object is photographed by the camera, a virtual space generation unit that generates a virtual space using display data prepared in advance and simulating the space where the content is photographed, a virtual screen display processing unit that arranges a transmissive virtual screen in the virtual space, an image display processing unit that reads out the data of the image of the object from the imaging information storage unit and projects the image onto the virtual screen In the virtual space, a virtual photographing unit that photographs an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen is provided.

[0072] In the imaging system according to the first aspect, in the photographing information storage unit, data of an image of an object photographed by the actual imaging unit in the real space and position information representing the positional relationship between the two when the object is photographed by the actual imaging unit are stored. Here, the object to be photographed is, for example, an actor such as an actor. The image of the object is obtained, for example, by photographing the object against a green screen as in the conventional manner. Also, display data serving as the background of the content is prepared in advance. The virtual space generation unit generates a virtual space using the display data. The virtual screen display processing unit arranges a transparent virtual screen in the virtual space, and the image display processing unit displays the image of the object on the virtual screen. The transparent virtual screen refers to a virtual object (screen) that is transparent except for the portion where the image is displayed. Then, the virtual photographing unit photographs an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen. In the imaging system according to the present invention, in this way, an image in which the image of the object photographed in the real space is incorporated into the virtual space is obtained. In the imaging system according to the first aspect, in the real space, for example, the object may be photographed against a green screen conventionally used, and there is no need to use a large and expensive display having a large number of LED light sources like virtual production. Also, the virtual space serving as the background can be freely designed, and the size and position of the virtual screen arranged therein can be easily changed. Therefore, in the imaging system according to the first aspect, it is possible to inexpensively obtain images of the object captured at various angles of view.

[0073] (Second aspect) The imaging system according to the second aspect is the imaging system according to the first aspect, wherein in the photographing information storage unit, further, information on the position of the object in the real space is associated with and stored in the data of the image of the object. The virtual screen display processing unit further changes the position of the virtual screen based on the information on the position of the object.

[0074] In the imaging system according to the second aspect, the movement of the performer in the real space can be reflected in the virtual space.

[0075] (Article 3) The imaging system according to the third aspect is the imaging system according to the first or second aspect, wherein the virtual screen display processing unit receives a predetermined input operation and changes the posture or shape of the virtual screen.

[0076] In the imaging system according to the third aspect, it is possible to create video content incorporated into the virtual space by adding arbitrary changes to the image of the actual object, such as making the object appear larger (taller) than in reality.

[0077] (Article 4) The imaging system according to the fourth aspect is the imaging system according to any one of the first to third aspects, wherein the imaging information storage unit further stores the information on the position of the camera in the real space associated with the data of the image of the object, and the virtual imaging unit changes the predetermined position according to the information on the position of the camera.

[0078] (Article 5) The imaging system according to the fifth aspect is the imaging system according to any one of the first to fourth aspects, wherein the imaging information storage unit further stores the information on the angle of view of the camera in the real space associated with the data of the image of the object, and the virtual imaging unit changes the angle of view at the time of imaging according to the information on the angle of view of the camera.

[0079] In the imaging system according to the fourth aspect and the imaging system according to the fifth aspect, it is possible to perform imaging reflecting the position and angle of view of the camera when the image of the object is captured.

[0080] (Item 6) The imaging system according to Item 6 is the imaging system according to any one of Items 1 to 5, wherein the virtual screen is arranged facing the virtual imaging unit.

[0081] In the imaging system according to Item 6, the virtual screen is arranged corresponding to the virtual imaging unit, that is, the optical axis of the virtual imaging unit is arranged perpendicular to the virtual screen. The closer the optical axis of the virtual imaging unit is to being parallel to the virtual screen, the less the depth of the image of the object is felt. However, by adopting the arrangement described in Item 6, it becomes easier to make the object projected on the virtual screen feel depth.

[0082] (Item 7) The imaging system according to Item 7 is the imaging system according to any one of Items 1 to 6, further comprising a camera that captures an object in the real space, and an image extraction unit that extracts the region of the object from the image data of the object captured by the camera and is provided with The image display processing unit displays the extracted image of the object extracted by the image extraction unit on the virtual screen in real time.

[0083] In the imaging system according to Item 7, video content can be created in real time while capturing an object.

[0084] (Item 8) The imaging system according to Item 8 is the imaging system according to any one of Items 1 to 7, further comprising a first sound collection unit attached to the object, and a first sound output unit that outputs the audio data collected by the first sound collection unit from a predetermined position on the virtual screen and is provided with

[0085] In the imaging system according to Item 8, when the object is a person or the like who emits sound, the sound emitted from the object can be recognized by a player who operates a player character in the virtual space.

[0086] (Item 9) The imaging system according to Item 9 is the imaging system according to any one of Items 1 to 8, and furthermore, a second sound collection unit provided at a predetermined position of the virtual screen, and a second sound output unit that is attached to the object and outputs data of the sound collected by the second sound collection unit are provided.

[0087] (Item 10) The imaging system according to Item 10 is the imaging system according to any one of Items 1 to 9, and the object is a human, and furthermore, a first motion detection unit that detects a predetermined motion by the object, and a first effect generation unit that generates an effect associated with the predetermined motion on the virtual space or the virtual screen when the predetermined motion is detected by the first motion detection unit are provided.

[0088] (Item 11) The imaging system according to Item 11 is the imaging system according to any one of Items 1 to 10, and furthermore, a player terminal that operates a player character operating in the virtual space, and a second motion detection unit that detects a predetermined motion of the player character with respect to the image of the object displayed on the virtual screen by an operation through the player terminal, and a second effect generation unit that generates an effect associated with the predetermined motion at a position on the virtual screen where the image of the object is displayed when the predetermined motion is detected by the second motion detection unit is provided with.

[0089] In the imaging system according to claim 1, when the object is a human, communication can be achieved with a player character in a virtual space via a virtual screen. Further, it is possible to exert an influence on the virtual space from the human who is the object. That is, starting from the virtual screen, in response to a predetermined action (such as vocalization or movement) by the human who is the object, an effect pre-associated with the predetermined action can be generated in the virtual space. Also, in response to a predetermined action (such as vocalization or movement) by a player character within the virtual screen, an effect pre-associated with the predetermined action can also be generated on the virtual screen.

[0090] Specifically, for example, in the imaging system according to claim 9, acting can be performed while listening to voices generated in the virtual space. Also, in the imaging system according to claim 10, the interactivity with a player who operates a player character within the virtual space can be further enhanced. Furthermore, in the imaging system according to claim 11, the interactivity with respect to the object from a player character operating within the virtual space can be further enhanced.

[0091] (Claim 12) The imaging system according to claim 12 is the imaging system according to any one of claims 1 to 11, comprising a plurality of sets each consisting of the virtual screen and the virtual imaging unit associated with the virtual screen.

[0092] In the imaging system according to claim 12, data of images of an object captured by a plurality of cameras in the real space (images of the same object captured from different directions and / or angles of view, or images of different objects) are projected onto different virtual screens, and an image of the virtual space including them is captured by the virtual imaging unit associated with the virtual screen, whereby highly variable video content can be obtained.

[0093] (Item 13) The imaging system according to Item 13 is the imaging system according to any one of Items 1 to 12, and further, a background screen provided at a position in the virtual space that is separated from the virtual imaging unit by a predetermined distance is provided, the virtual space generation unit displays, on the background screen, as a two-dimensional image, a space in the virtual space that is separated from the virtual imaging unit by the predetermined distance or more.

[0094] In the imaging system according to Item 13, since it is not necessary to generate all the three-dimensional objects arranged in the virtual space, the processing load on the virtual space generation unit can be reduced.

[0095] (Item 14) The imaging system according to Item 14 is the imaging system according to any one of Items 1 to 13, and the virtual space generation unit generates a first virtual space and a second virtual space having a higher display resolution than the first virtual space, the virtual screen and the image display processing unit are respectively provided in the first virtual space and the second virtual space, after the virtual imaging unit acquires parameter information of an object existing in the first virtual space by imaging the first virtual space, the virtual imaging unit reflects the parameter information in the second virtual space and then images the second virtual space.

[0096] In the imaging system according to Item 14, in the imaging by the first virtual space, it is only necessary to display the object at a low resolution, so the processing load on the virtual space generation unit can be reduced and parameter information and the like necessary for efficiently creating content can be acquired. Also, by reflecting the parameter information in the second virtual space to create content, video content can be efficiently created in a virtual space with high resolution.

Explanation of Reference Numerals

[0097] 1... Imaging system 100... Studio 10… Photographing device 11… Background member 111… Grid points 12, 121~123… Cameras 13… Position sensor 131~133… Reference points 134… Body 14, 141, 142… Performers 15… Sound collection unit 16… Earphones 17… Screen 20… Control and processing device 21… Memory unit 22… Photographing information memory unit 23… Virtual space display data memory unit 24… Character data memory unit 25… Photographing data memory unit 30… Content creation program 31… Image extraction unit 32… Camera field of view acquisition unit 33… Position information acquisition unit 341… Virtual space generation unit 342… Virtual screen display processing unit 35… Character display processing unit 36… Projection unit 37… Virtual photographing unit 38… Audio processing unit 41… Input unit 42… Display unit 50… Player terminal 51… Memory unit 52… Input operation unit 53… Audio input unit 54… Audio output unit 55… Display unit 60… Virtual space 61, 611~613… Virtual cameras 62, 621~623… Virtual screens 63… Player character 64… Extracted image of the performer projected on the virtual screen 71… Reference object 9… Network

Claims

1. A camera that captures an object in real space; an image extraction unit that extracts a region of the object from image data of the object captured by the camera; a position information acquisition unit that acquires position information representing a positional relationship between the camera and the object; a virtual space generating unit that generates a virtual space using display data that is prepared in advance and that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that displays an image of the object extracted by the image extraction unit on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space seen through the virtual screen; In an imaging system comprising: In response to a predetermined action by a player character or a non-player character in the virtual space, an effect that is associated in advance with the predetermined action is generated on the virtual screen. An imaging system characterized by:

2. A camera that captures an object in real space; an image extraction unit that extracts a region of the object from image data of the object captured by the camera; a position information acquisition unit that acquires position information representing a positional relationship between the camera and the object; a virtual space generating unit that generates a virtual space using display data that is prepared in advance and that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that displays an image of the object extracted by the image extraction unit on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space seen through the virtual screen; A first sound collecting unit attached to the target object; a first sound output unit that outputs the sound data collected by the first sound collection unit from a predetermined position on the virtual screen; An imaging system comprising:

3. A camera that captures an object in real space; an image extraction unit that extracts a region of the object from image data of the object captured by the camera; a position information acquisition unit that acquires position information representing a positional relationship between the camera and the object; a virtual space generating unit that generates a virtual space using display data that is prepared in advance and that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that displays an image of the object extracted by the image extraction unit on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space seen through the virtual screen; A second sound collecting unit provided at a predetermined position on the virtual screen; a second sound output unit that is attached to the object and outputs data of the sound collected by the second sound collection unit; An imaging system comprising:

4. the subject is a human being; moreover, A first motion detection unit that detects a predetermined motion of the object; a first effect generating unit that generates an effect associated with the predetermined motion in the virtual space or on the virtual screen when the predetermined motion is detected by the first motion detecting unit; 4. The imaging system according to claim 1, further comprising:

5. moreover, a second action detection unit that detects a predetermined action of the player character with respect to the image of the object displayed on the virtual screen; a second effect generating unit that generates an effect associated with the predetermined action at a position on the virtual screen where the image of the object is displayed when the predetermined action is detected by the second action detecting unit; The imaging system according to claim 1 , further comprising:

6. The position information acquisition unit further acquires information on a position of the object in the real space, The virtual screen display processing unit further changes the position of the virtual screen based on the information on the position of the object.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

7. The virtual screen display processing unit changes the attitude or shape of the virtual screen in response to a predetermined input operation.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

8. The position information acquisition unit further acquires information on the position of the camera in the real space, The virtual photographing unit changes the predetermined position in accordance with information on the position of the camera.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

9. moreover, a camera angle of view acquisition unit that acquires information on the angle of view of the camera in the real space; Equipped with The virtual photographing unit changes the angle of view during photographing in accordance with information on the angle of view of the camera.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

10. 4. The photographing system according to claim 1, wherein the virtual screen is disposed directly opposite the virtual photographing unit.

11. The virtual screen and the virtual shooting unit corresponding to the virtual screen are provided in a plurality of pairs.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

12. moreover, A background screen provided in the virtual space at a predetermined distance from the virtual shooting unit. Equipped with The virtual space generation unit displays a space in the virtual space that is separated from the virtual shooting unit by at least the predetermined distance on the background screen as a two-dimensional image.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

13. the virtual space generation unit generates a first virtual space and a second virtual space having a display resolution higher than that of the first virtual space; The first virtual space and the second virtual space are provided with the virtual screen and the image display processing unit, The virtual photographing unit photographs the first virtual space to obtain parameter information of an object present in the first virtual space, and then reflects the parameter information in the second virtual space and photographs the second virtual space.

4. The imaging system according to claim 1, wherein the imaging system further comprises:

Images