Photographic system

Abstract

A photography system (1) includes a photography information storage section (22) that stores data of an image of an object photographed by a camera in a real space and position information indicating a positional relationship of both when the object is photographed by the camera; a virtual space generation section (341) that generates a virtual space using display data of a space for photographing content prepared in advance; a virtual screen display processing section (342) that configures a virtual screen (62) of a transmission type in the virtual space; an image display processing section (36) that reads out the data of the image of the object from the photography information storage section and displays the image on the virtual screen; and a virtual photography section (37) that photographs the image including the virtual screen and the virtual space visible through the virtual screen together from a prescribed position determined based on the position information in the virtual space.

Description

Technical Field

[0001] This invention relates to a photographic system for producing films, television programs, and other content. Background Technology

[0002] Chromatic keying systems are widely used in the production of films, television programs, and other content. In chroma keying systems, actors and other performers are filmed against a green screen (colored with a predetermined color) as a background. The areas of the predetermined color are then removed to extract the performer's image. This extracted image is then composited with images prepared separately using computer graphics (CG) to create the content.

[0003] Recently, a technology called virtual production has been proposed (e.g., Patent Document 1). In virtual production, images are projected onto a display composed of numerous LED light sources, which are then used as a background for a performance filmed by a camera. Because the performer and the background are filmed simultaneously, no image compositing is required, allowing content to be created in a short time. Furthermore, the system displays images with a sense of depth on the display, and the way the images are displayed is changed according to the positional relationship between the camera and the display, thereby creating content that makes it appear as if the performer is performing in real space.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2024-35420 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] In virtual production, the monitor needs to be configured to cover the entire area projected onto the camera. This presents a problem: to capture images of performers and other objects from various angles, large and expensive screens with a vast number of LED light sources must be used.

[0009] The problem to be solved by the present invention is to provide a technology that can acquire images of objects from various perspectives at low cost.

[0010] Solution for solving the problem

[0011] The photographic system of the present invention, which was made to solve the above problems, is characterized by having:

[0012] The photographic information storage unit stores data of images of objects captured by a camera in real space, as well as positional information indicating the positional relationship between the two objects when the camera captured the image.

[0013] The virtual space generation department uses pre-prepared display data that simulates the space used for filming to generate virtual space;

[0014] A virtual screen display processing unit, which configures a transmissive virtual screen in the virtual space;

[0015] An image display processing unit reads image data of the object from the photographic information storage unit and displays the image on the virtual screen; and

[0016] A virtual photography unit that captures an image of the virtual screen and the virtual space visible through the virtual screen from a predetermined position determined based on the location information within the virtual space.

[0017] The effects of the invention

[0018] In the camera system of the present invention, the photographic information storage unit stores data of images of objects captured by the camera in real space, as well as positional information indicating the positional relationship between the objects and the camera when they are captured. Here, the object being photographed is, for example, an actor or performer. The image of the object is obtained by extracting the necessary parts for compositing from an image obtained conventionally with a green screen as the background (extracted image of the object). Additionally, display data as the background for the content is prepared in advance. The virtual space generation unit uses this display data to generate a virtual space. The virtual screen display processing unit configures a transmissive virtual screen in the virtual space, and the image display processing unit displays the image of the object on the virtual screen. A transmissive virtual screen is a virtual object (screen) that functions as a reference plane when setting the camera surface (focus) of the virtual camera unit in the virtual space, and is transparent except for the area displaying the image. The virtual camera unit captures the image including the virtual screen and the virtual space visible through the virtual screen from a predetermined position determined based on the positional information. In the camera system of the present invention, an image obtained by incorporating the image of the object captured in real space into the virtual space is thus acquired. In the photographic system of this invention, in real space, the subject can be photographed against a background such as a green screen, as is commonly used, without the need for large and expensive displays with numerous LED light sources, as is required for virtual production. Furthermore, the virtual space serving as the background can be freely designed, and the size and position of the virtual screens configured within it can be easily changed. Therefore, in the photographic system of this invention, images of the subject captured from various angles can be obtained at low cost. Attached Figure Description

[0019] Figure 1 This is a structural diagram of the main parts of one embodiment of the photographic system involved in the present invention.

[0020] Figure 2 This is an example of an image obtained by taking a picture using the wide-angle lens of a camera in the photography system of this embodiment.

[0021] Figure 3 This diagram schematically illustrates the state of a performer being captured by the wide-angle lens of a camera in the photographic system of this embodiment.

[0022] Figure 4 This diagram illustrates the state of a performer captured by a camera's narrow-angle lens in the photographic system of this embodiment.

[0023] Figure 5 This is an example of an image obtained by shooting with a narrow-angle lens of a camera in the photography system of this embodiment.

[0024] Figure 6 This diagram illustrates the state in which an extracted image of a performer is projected onto a virtual screen configured in a virtual space within the photography system of this embodiment.

[0025] Figure 7 This is a diagram illustrating a modified example of the camera perspective acquisition unit in the photography system of this embodiment.

[0026] Figure 8 This diagram illustrates an example of filming performers in a studio using multiple cameras within the photography system of this embodiment.

[0027] Figure 9 This diagram illustrates an example of configuring multiple virtual cameras and virtual screens in a virtual space within the photography system of this embodiment.

[0028] Figure 10 This is a diagram illustrating an example of tilting a virtual screen forward and backward in the photography system of this embodiment.

[0029] Figure 11 This diagram illustrates an example of a background screen configured in the photography system of this embodiment.

[0030] Figure 12 This is a diagram illustrating an example of a first virtual space that can be used in the photography system of this embodiment. Detailed Implementation

[0031] Hereinafter, embodiments of the photographic system according to the present invention will be described with reference to the accompanying drawings.

[0032] Figure 1 The main structural components of the photography system 1 of this embodiment are shown. The photography system 1 of this embodiment includes a photography device 10, a control / processing device 20, and a player terminal 50. The photography device 10, the control / processing device 20, and the player terminal 50 are connected to each other via a wired or wireless network 9 in a manner capable of communication. Figure 1 Only one player terminal 50 is shown, but there can be multiple player terminals 50.

[0033] The camera device 10 is installed in a studio 100 where the performer 14, the subject of the filming, performs. The camera device 10 includes a background component 11, called a green screen, which is colored in a predetermined color (e.g., green), a camera 12, a position sensor 13, and a sound collection unit 15. Additionally, the performer 14 wears headphones 16, which serve as the sound output unit. While headphones 16 are used in this embodiment, speakers or similar devices could also be installed in the studio 100. Furthermore, a screen 17, serving as a display unit, is installed in the studio 100, outside the field of view of the camera 12, where images captured in real-time by the virtual camera 61 installed in the virtual space 60 (described later) are displayed.

[0034] like Figure 2 As shown, a grid-like set of reference points (grid points 111) is provided on the background component 11. In this embodiment, seven grid points 111 are provided in the horizontal direction and seven in the vertical direction (a total of 49 grid points). The spacing and number of grid points 111 can be arbitrarily changed. In this embodiment, the reference points are set in a grid shape, but the reference points only need to be arranged in a two-dimensional and regular manner on the surface of the background component 11, or the reference points can be set in a honeycomb pattern, etc. The camera 12 uses the background component 11 as a background to film the actors and other performers 14. The position sensor 13 in this embodiment is an infrared LiDAR (Light Detection and Ranging) sensor, which illuminates infrared light from the main body 134 and receives reflected light from reference points 131 and 132 provided at two parts of the camera 12 (e.g., the front end of the lens and the main body of the camera 12) and reference point 133 provided at one part of the performer 14 (e.g., the neck), thereby measuring the position of each reference point 131 to 133. For example, stickers reflecting infrared light can be affixed to reference points 131-133. The position and number of reference points 131-133 are just one example; they can be appropriately changed as long as the above function is achieved.

[0035] The control / processing device 20 includes a storage unit 21. The storage unit 21 houses a photography information storage unit 22, a virtual space display data storage unit 23, a character data storage unit 24, and a photography data storage unit 25. The photography information storage unit 22 stores photography information, including data on images of objects captured in real space, the posture of the camera 12 when capturing the image, and information corresponding to the positions of the camera 12 and the performer 14. Furthermore, the concept of an image in this specification includes not only still images but also moving images (e.g., moving images composed of still images at 60 frames per second). The virtual space display data storage unit 23 stores, in advance, time-series display data in a 3D (three-dimensional) virtual space simulating various scenes to be used in the content produced in the photography system 1. This display data also includes sound data such as sound effects. The character data storage unit 24 stores time-series display data of the 3D avatar of the player character (PC) operated by the player, and 3D avatars of non-player characters (NPCs) not operated by the player. As mentioned above, the non-player character is not operated by the player, so its actions during the recording of the video content are predetermined according to the script, and the display data (including sound data) of this time sequence is stored in the character data storage unit 24.

[0036] The control / processing device 20 includes an image extraction unit 31, a camera view acquisition unit 32, a position information acquisition unit 33, a virtual space generation unit 341, a virtual screen display processing unit 342, a character display processing unit 35, an image display processing unit 36, a virtual photography unit 37, and a sound processing unit 38 as functional blocks. The operation of each of these units will be described later. The physical entity of the control / processing device 20 is, for example, a general personal computer, implemented by a processor executing a pre-installed dedicated program (content creation program 30). The control / processing device 20 can be installed in a suitable location, either inside or outside the studio 100. Alternatively, a cloud server set up on a network can also be used as the control / processing device 20. Furthermore, the control / processing device 20 is connected to an input unit 41 consisting of a keyboard, mouse, etc., and a display unit 42 consisting of a liquid crystal display, etc.

[0037] The player terminal 50 includes a storage unit 51. The storage unit 51 stores information about the player using the player terminal 50 (such as player ID information). The player terminal 50 includes an input operation unit 52, a sound input unit 53, a sound output unit 54, and a display unit 55. For example, a head-mounted display equipped with motion capture and position sensors can be used for the player terminal 50, which functions as the input operation unit 52. The player terminal 50 can be placed in any location. That is, the player can operate the player character from anywhere inside or outside the studio 100.

[0038] Next, the process of creating image content using the photography system 1 of this embodiment will be described.

[0039] When creating video content, the space serving as the background of the content is pre-defined as a 3D virtual space. Time-series data for generating the 3D virtual space, which changes over time according to a script, is created and stored in the virtual space display data storage unit 23. Additionally, time-series display data for the 3D virtual images of player characters appearing in the content, as well as the 3D virtual images of non-player characters, are pre-stored in the character data storage unit 24. Player characters are, for example, characters who appear in the content and interact with the performers, such as through dialogue. Non-player characters are, for example, background characters appearing as extras in the background of the content, enemy characters who fight against performers or player characters based on pre-determined algorithms, or objects positioned at designated locations within the virtual space. In the case of multiple player characters, information (such as terminal ID information) about the player terminal 50 used by the player operating that player character is also stored for each player character. Hereinafter, for convenience, the display method of non-player characters as objects will also be referred to as virtual images.

[0040] Once the above preparations are complete, preparations for shooting the video content begin. These preparations take place in both a physical studio 100 and a virtual space 60.

[0041] First, the preparations for filming performer 14 in a studio 100 set up in a real space will be described. A background component 11, referred to as a green screen, is arranged in the studio 100. Reference points 131 and 132, which serve as position sensors 13, are pre-set on the lens and main body of the camera 12, respectively. Additionally, reference point 133 of the LiDAR sensor is pre-set at a predetermined position of the performer 14. In this embodiment, stickers reflecting infrared light are affixed to the lens and main body of the camera 12, as well as to the neck of the performer 14. The main body 134 of the LiDAR sensor is arranged at a predetermined position in the studio 100 (e.g., a corner of the ceiling). The operating principle of LiDAR is known in the past, so detailed descriptions are omitted here. Furthermore, a screen 17 and a sound collection unit 15 for collecting the sound emitted by the performer 14 are arranged outside the field of view of the camera 12 in the studio 100. The performer 14 is also provided with headphones 16.

[0042] Camera 12 captures an image of performer 14 standing in the initial position against the background component 11, obtaining image data of performer 14. The image data of performer 14 captured by camera 12, along with a timestamp indicating the time the image was captured (shooting time, elapsed time since the start of the performance, etc.), is saved to the photography information storage unit 22. Additionally, the sounds emitted by performer 14 during the performance are collected by sound collection unit 15 and saved to the photography information storage unit 22.

[0043] Figure 3 and Figure 4 An example of filming performer 14 in studio 100 is shown. Figure 3 The image shows the performer 14 being filmed using the wide-angle lens of camera 12. Figure 4 The image shows the performer 14 being filmed using the narrow-angle lens of camera 12. When... Figure 3 When using the wide-angle lens of camera 12 to film performer 14, such as Figure 2 As shown, an image capturing the full body of performer 14 is obtained. Additionally, when... Figure 4 When filming performer 14 with the narrow-angle lens of camera 12, such as Figure 5 As shown, a magnified image capturing a portion of performer 14 (upper body in this example) is obtained. Figure 2 and Figure 5 As shown, in the image of performer 14, grid points 111 set on background component 11 are also captured.

[0044] In the control / processing device 20, the image extraction unit 31 reads the image data stored in the photographic information storage unit 22. Then, it extracts the image data containing all areas except for the specified color regions (the green screen portion, including the grid dots 111). This generates an image (extracted image) that captures a portion of the performer 14. The generated extracted image is also stored in the photographic information storage unit 22.

[0045] The camera perspective acquisition unit 32 calculates the number of grid points 111 contained in the image stored in the photographic information storage unit 22 (in this embodiment, the number of grid points 111 arranged in the vertical direction and the number of grid points 111 arranged in the horizontal direction. This number includes the number of grid points 111 hidden by the performer 14). Then, based on the number of grid points 111, the perspective (wide-angle, narrow-angle, etc.) of the camera 12 lens when capturing the image is determined. When the positions of the camera 12 and the performer 14 do not change, if the number of grid points contained in the image is large, the lens of the camera 12 is wide-angle; if the number of grid points contained in the image is small, the lens of the camera 12 is narrow-angle (in the zoomed state). In addition, the camera perspective acquisition unit 32 determines the position of the reference points 131 and 132 set on the camera 12 based on the output signal from the position sensor 13, and determines the posture (elevation angle θ) of the camera 12 according to the relative positional relationship between the two.

[0046] The position information acquisition unit 33 determines the positions of reference point 131 (or / and 132) set on camera 12 and reference point 133 set on performer 14 based on the output signal from position sensor 13, and determines the positional relationship between the two. The positional relationship information includes the orientation of camera 12 when capturing performer 14 and the distance information between camera 12 and performer 14 (i.e., vector information). The camera pose information acquired by camera view acquisition unit 32 and the positional relationship information between camera 12 and performer 14 acquired by position information acquisition unit 33 are stored together with a timestamp indicating the time when the information was acquired in the photography information storage unit 22.

[0047] The processing described above by the image extraction unit 31, the camera view acquisition unit 32, and the position information acquisition unit 33 is performed in real time during the filming of the performer 14 in the studio 100.

[0048] Next, the preparation for photography in virtual space will be explained.

[0049] The virtual space generation unit 341 first reads the time-series display data of the 3D virtual space used for capturing image content from the virtual space display data storage unit 23, and generates the virtual space based on the display data at its start time. Additionally, as... Figure 6As illustrated, the virtual screen display processing unit 342 positions a transmissive virtual screen 62 at a predetermined initial position within the virtual space. Furthermore, the transmissive virtual screen 62 refers to a virtual object (screen) within the virtual space 60 that functions as a reference surface for setting the camera plane (focus) of the virtual camera unit 37, and is transparent except for the area where the image is projected. Since the virtual screen 62 is a virtual object, both player characters and non-player characters can freely pass through it.

[0050] The character display processing unit 35 reads the display data of the virtual images of the player character and the virtual images of the non-player character, and positions them in their initial positions within the virtual space. The virtual camera unit 37, based on the positional relationship between the camera 12 and the performer 14 obtained by the position information acquisition unit 33, positions the virtual camera 61 in the virtual space 60. Specifically, the position of the performer 14 is aligned with the position of the virtual screen 62, and based on this, the relative positional relationship between the camera 12 and the performer 14 is reflected in the positional relationship between the virtual screen 62 and the virtual camera 61 to determine the position of the virtual camera 61. In this embodiment, the virtual camera 61 and the virtual screen 62 are positioned directly opposite each other. That is, they are configured such that the optical axis of the virtual camera 61 is perpendicular to the surface of the virtual screen 62. Furthermore, the size of the virtual screen 62 is appropriately changed according to the size of the extracted image 64 of the performer 14 displayed on the virtual screen 62.

[0051] The image display processing unit 36 ​​displays the extracted image 64 of the performer 14, extracted by the image extraction unit 31, on the virtual screen 62. In this embodiment, the image display processing unit 36 ​​displays the extracted image 64 of the performer 14 on the virtual screen 62 by projecting it from the position of the virtual camera 61 toward the virtual screen 62. The relative positional relationship between the virtual camera 61 and the virtual screen 62 reflects the positional relationship between the camera 12 and the screen 17. Therefore, by projecting the extracted image 64 of the performer 14 from the virtual camera 61 toward the virtual screen 62, an image obtained by capturing the performer 14 from the same perspective as the image captured by the camera 12 in real space can be displayed on the virtual screen 62. However, the method of displaying the extracted image 64 of the performer 14 on the virtual screen 62 is not limited to this; various methods, such as sending pixel data constituting the extracted image 64, can be used to the projection screen 62. In this way, the actual image of the performer 14 captured in real space is incorporated into the virtual space.

[0052] As mentioned above, the virtual screen 62 is transmissive, therefore... Figure 6As shown, the virtual camera 61 simultaneously captures the extracted image 64 of the performer 14 projected onto the virtual screen 62 and the virtual space 60 located in the background of the virtual screen 62. The virtual photography unit 37 captures images of the virtual space 60 at a predetermined frame rate (e.g., 60 frames per second) using the virtual camera 61. The captured image data is stored in the photography data storage unit 25 and displayed in real time on the screen 17 of the studio 100.

[0053] The virtual space generation unit 341 generates an image captured from the virtual space at the position of the player character 63, which is positioned at its initial location, and sends it to the player terminal 50 used by the player operating the player character 63. The received image is displayed on the display unit 55 in the player terminal 50. Figure 6 Only one player character (63) is displayed, but sometimes multiple player characters (63) exist simultaneously. Furthermore, as... Figure 6 As shown, when the player character 63 is located on the back side of the virtual screen 62, a silhouette of the extracted image 64 of the performer 14 is displayed on the display unit 55. When the player character 63 is located on the front side of the virtual screen 62 (the side where the virtual camera 61 is located), an image obtained by viewing the extracted image 64 of the performer 14 from the front is displayed on the display unit 55.

[0054] After the filming preparations are completed in both the real-world studio 100 and the virtual space 60 as described above, the director instructs the start of filming through prescribed actions (e.g., initiating filming with a voice command in the studio 100, and simultaneously pressing the start filming button via the input unit 41). In response, the performer 14 begins performing in the studio 100 according to a pre-prepared script. The performer 14's performance includes emitting sounds, which are collected by the sound collection unit 15 and sent to the control / processing device 20 (equivalent to the function of the first sound collection unit in this invention). In the control / processing device 20, the sound processing unit 38 outputs the sound received from the sound collection unit 15 as sound emanating from a predetermined position (e.g., the center of gravity of the extracted image 64) on the virtual screen 62 within the virtual space 60 (equivalent to the function of the first sound output unit in this invention).

[0055] In addition, the virtual space generation unit 341 and the character display processing unit 35 read the time-series display data of the virtual space and the time-series display data of the non-player character from the virtual space display data storage unit 23 and the character data storage unit 24, respectively, and reflect (reproduce) them sequentially into the virtual space 60. The display data also includes sound data, which is reproduced as sound emitted from the position of the non-player character in the virtual space.

[0056] In the player terminal 50, the player uses the input control unit 52 to instruct the player character 63 to perform actions and emit sounds according to a pre-prepared script. The actions input by the player through the input control unit 52 are sequentially sent to the control / processing device 20, reflecting in the actions of the player character 63 within the virtual space. Additionally, the sounds emitted by the player are collected by the sound input unit 53 and sequentially sent to the control / processing device 20. In the control / processing device 20, the sound processing unit 38 outputs the received sounds as sound emanating from the position of the player character 63 within the virtual space 60.

[0057] The sound processing unit 38 collects sound within the virtual space 60 at a predetermined location on the virtual screen 62 (e.g., the center of gravity of the extracted image 64) (equivalent to the function of the second sound collection unit in this invention). The collected sound is sent to the camera device 10 and output from the headphones 16 worn by the performer 14 (equivalent to the function of the second sound output unit in this invention). Thus, the performer 14 can perform while visually confirming the state within the virtual space 60 through the screen 17 and listening to the sound emitted in the virtual space 60 and captured at the location on the virtual screen 62 through the headphones 16. Additionally, the sound processing unit 38 also collects sound within the virtual space at the location of the player character 63 within the virtual space 60. The collected sound is sent to the player terminal 50 used by the player operating the player character 63 and output from the sound output unit 54. Thus, the player operating the player character 63 can visually confirm the state within the virtual space 60 from the player character 63's perspective via the display unit 55, and listen to the sounds emitted in the virtual space 60 and captured at the player character 63's position via the sound output unit 54 (including the performer 14's voice emitted from a designated position on the virtual screen 62), while simultaneously instructing the player character 63 to perform actions.

[0058] During filming, the position information acquisition unit 33 monitors the positions of the camera 12 and the performer 14 based on the output signal from the position sensor 13. When the camera 12 or the performer 14 moves (changes position and / or posture), the position information acquisition unit 33 detects this information in the control / processing device 20. The position of the camera 12 corresponds to the position of the virtual camera 61. When the position information acquisition unit 33 detects the movement of the camera 12, the virtual camera unit 37 moves the virtual camera 61, which is positioned in the virtual space 60, in coordination with the movement of the camera 12. The position of the performer 14 corresponds to the position of the virtual screen 62. When the position information acquisition unit 33 detects the movement of the performer 14, the virtual screen display processing unit 342 moves the virtual screen 62, which is positioned in the virtual space 60, in coordination with the movement of the performer 14.

[0059] In addition, the camera perspective acquisition unit 32 monitors the perspective of camera 12. Changes in the perspective of camera 12 are detected based on the number of grid points contained in the image captured by camera 12. The perspective information of camera 12 acquired by camera perspective acquisition unit 32 is reflected in the perspective of virtual camera 61 by virtual photography unit 37. For example, when camera 12 changes its perspective to capture performer 14 in a magnified manner, the perspective of virtual camera 61 also changes in virtual space, so that the extracted image of performer 14 projected on virtual screen 62 is captured by zoom. Therefore, the virtual camera 61 captures the same state of performer 14 in virtual space as the camera 12 captures in real space.

[0060] When performer 14 finishes performing based on a pre-prepared script, the director instructs the filming to end via a prescribed action (e.g., an audio signal to end filming in studio 100, and pressing the end-of-filming button via input unit 41). Consequently, the virtual space generation unit 341 and the character display processing unit 35 respectively end the reproduction of the time-series display data of the virtual space 60 and the time-series display data of the virtual image of the non-player character. Furthermore, the player's operation on the player character 63 ends in the player terminal 50. The virtual photography unit 37 saves the image data acquired by the virtual camera 61 at a prescribed frame rate as a motion image file to the photography data storage unit 25.

[0061] In the photography system 1 of this embodiment, the image of the performer 14 performing in a studio 100 set in real space is displayed on a transmissive virtual screen 62 configured in a virtual space 60, thereby enabling the creation of video content that incorporates live-action images of the performer 14 into the virtual space. In the photography system 1 of this embodiment, the object is filmed against a green screen in a studio 100 in real space, and the extracted image 64 obtained by extracting the image of the performer 14 is acquired using conventional processing, eliminating the need for large and expensive displays with numerous LED light sources as in virtual production. Furthermore, since the background of the video content is virtual space, the position and size of the virtual screen 62 projecting the live-action extracted image 64 can be easily changed in accordance with the camera 12's perspective, the performer 14's position, and the relative positional relationship between the camera 12 and the performer 14. Therefore, images of objects captured from various perspectives can be obtained at low cost.

[0062] In the photography system 1 of this embodiment, it is possible to create video content in which a player character 63, which is operated by the player terminal 50, and a non-player character perform together with a performer 14 who is filmed live.

[0063] Previously, when creating content that combined images of performers, motion graphics, and separately prepared background images, the performer's image, which was created as a single image file, was simply combined with the background image. In other words, it was like combining cels depicting characters with a background image in traditional animation production—it was merely compositing two separate images. Therefore, sometimes the performer's gaze, posture, and the background image did not match, resulting in a sense of disharmony in the composite image.

[0064] In contrast, in the photography system 1 of this embodiment, images captured by a virtual camera 61 positioned in the virtual space 60 are displayed in real-time on a screen 17 positioned in the studio 100. Furthermore, the voice emitted by the performer 14 is reflected in the virtual space, and the sound within the virtual space at the location of the virtual screen 62 positioned in the virtual space is output from the performer 14's headphones 16, thus enabling interactive communication (dialogue, etc.) between the performer 14 and the player character 63 and non-player characters located in the virtual space 60. Additionally, the performer 14's shadow can appear in the virtual space 60 in real-time, or the voices emitted by the player character 63, etc., in the virtual space 60 can be heard by the performer 14 at a volume appropriate to the distance between the player character 63 and the performer 14. Therefore, the performer 14 can experience the script firsthand while performing. Furthermore, since post-processing such as image compositing is unnecessary, the costs associated with such processing can be reduced.

[0065] In the above embodiments, the case of creating video content by filming the performance of performer 14 and player character 63 was described. However, the same structure can also be used to film game scenes in virtual space. Moreover, when playing games, a structure that further improves interactivity compared to the above embodiments can be used.

[0066] For example, in a combat game, it can be configured such that when the player character 63's attack hits the extracted image 64 of the performer 14, a predetermined special effect (e.g., blood splatter) is generated on the part of the virtual screen 62 where the attack hits. Alternatively, it can be configured such that when the performer 14 attacks the player character 63, an attack is generated on the player character 63 from the display position of the extracted image 64 of the performer 14 on the virtual screen 62 (e.g., receiving the performer 14's firing action and generating an image of a bullet being fired from the position of the extracted image 64). Furthermore, it can be configured such that not only between the player character 63 and the performer 14, but also when the performer 14 performs a predetermined action, the virtual space 60 produces a predetermined special effect (e.g., the display of the virtual space 60 dims).

[0067] For such special effects data, it is sufficient to pre-set a special effects data storage unit in the storage unit 21 of the control / processing device 20 and store it therein. Alternatively, it can be stored in the virtual space display data storage unit 23 or the character data storage unit 24. In addition, as a functional block of the control / processing device 20, it is sufficient to provide a special effects generation unit (first special effects generation unit) that reads the special effects data from the storage unit and generates hit, attack, and other special effects on the virtual screen 62 and in the virtual space 60 when the motion detection unit (first motion detection unit, such as a motion sensor) detects that the performer 14 has performed a predetermined action, and a special effects generation unit (second special effects generation unit) that reads the special effects data from the storage unit and generates special effects on the extracted image 64 of the object when the motion detection unit (second motion detection unit) detects that the player character 63 has performed a predetermined action on the extracted image 64 of the object through the player's operation via the player terminal, and generates special effects on the extracted image on the virtual screen 62. Alternatively, the functions of these special effects generation units can be performed by the virtual space generation unit 341 and the virtual screen display processing unit 342.

[0068] Aside from utilizing a virtual screen 62 to execute special effects, the algorithms used to generate these effects are the same as those used in previous games implemented in virtual space. Therefore, it is sufficient to use the techniques employed in those games appropriately. Here, a battle game has been used as an example, but the same structure can ideally be used when creating video content for battle scenes. Of course, such special effects are not limited to battle games or battle scenes; the same structure can be employed in the production of other games and video content other than battle scenes, provided the desired special effects are prepared in advance.

[0069] In the above embodiments, a preferred embodiment of the photographic system 1 of the present invention has been described by way of a specific example, and modifications can be made appropriately in accordance with the spirit of the present invention.

[0070] In the above embodiment, grid points 111 are set in the background component 11, and the viewing angle (wide angle, narrow angle) of the camera 12 is determined based on the number of grid points 111 contained in the image captured by the camera 12, but the viewing angle of the camera 12 can also be determined by other methods.

[0071] For example, by performing machine learning on training data that identifies the various parts (head, hands, feet, etc.) contained in images of people (performers, etc.) taken from various angles, a fully learned model can be pre-built to output the camera's perspective when the image is taken as input to an image of a person, and the recognizer equipped with the fully learned model can be used as a camera perspective acquisition unit.

[0072] Or, it can also be like Figure 7As shown, a reference object of predetermined size is placed in advance on a specified part of the performer (face, hands, feet, etc.), and a camera perspective acquisition unit determines the camera's perspective based on the size of the reference object 71 (e.g., a sticker) included in an image of the performer. As a reference object, a circular sticker, ornament, or the like with a diameter of a few centimeters can be used. Alternatively, a sticker that serves as a reference point 133 for the position sensor 13 and is attached to the performer 14 can also be used as a reference object.

[0073] In the above embodiment, a performer 14 is photographed against the background component 11, but it can also be done as follows: Figure 7 Multiple performers 14 are filmed as shown. In this case, simply create an extracted image 64 for each performer 14 and project it onto a virtual screen 62.

[0074] In the above embodiment, image data obtained by actually filming the performers in the studio 100 is projected in real time onto the virtual screen 62 configured in the virtual space. However, the performers' image data can also be obtained in advance. In this case, along with the time-series data of the performers' images, the position and viewpoint of the camera 12, the position of the performers 14, and information on the positions of both are also pre-created as time-series data and stored in the photography information storage unit 22. Then, when creating video content, when reproducing the time-series display data of the virtual space and the time-series display data of the characters in the virtual space 60, the time-series data of the performers' images stored in the photography information storage unit 22, etc., are projected onto the virtual screen 62 in the virtual space 60 in conjunction with the timestamp position.

[0075] In the above embodiment, the performer 14 is filmed against a background component 11 (green screen, etc.) configured in the studio 100. However, it is also possible to film the performer 14 without using the background component 11 and obtain an extracted image of the performer 14 from it. Specifically, for example, it is also possible to film the performer 14 by focusing the camera 12 on the performer 14 and extract only the focused area from the obtained image to obtain an extracted image. If such a method is used, it is possible to obtain an extracted image from images obtained by filming actors or other performers outdoors (location shooting) without using a green screen and to create video content by combining the extracted image with virtual space in real time.

[0076] In addition, the structure described below can also be added to the above embodiments.

[0077] In the above embodiment, the extracted image 64 of the performer 14 obtained by shooting the performer 14 with a single camera 12 is displayed on a virtual screen 62. However, multiple cameras 12 can be used to shoot the performer 14, and the extracted images 64 of the performer 14 obtained by each camera 12 are displayed on the virtual screen 62 corresponding to each camera 12. In addition, multiple virtual cameras 61 are used to capture images of the virtual space 60 containing the extracted images 64.

[0078] exist Figure 8 The image shows a bird's-eye view of studio 100, illustrating the actions of performers 141 and 142 filmed within studio 100 using multiple cameras 121-123. Additionally, in... Figure 9 The diagram above shows a plurality of virtual cameras 611-613 and virtual screens 621-623 configured in the virtual space 60.

[0079] like Figure 8 As shown, in studio 100, camera 121 simultaneously captures both performers 141 and 142 from the front with a wide angle. Cameras 122 and 123 capture one of performers 141 and 142 from an oblique angle with narrow angles, respectively. Additionally, as... Figure 9 As shown, in the virtual space 60, a virtual camera 611 and a virtual screen 621 corresponding to camera 121 are configured, a virtual camera 612 and a virtual screen 622 corresponding to camera 122 are configured, and a virtual camera 613 and a virtual screen 623 corresponding to camera 123 are configured. The background component 11 used in the photography studio 100 is a single element, but in the virtual space 60, each virtual screen 621~623 is configured to face the virtual camera 611~613 respectively.

[0080] exist Figure 9The diagram illustrates three sets of virtual cameras 611-613 and virtual screens 621-623, but each virtual camera 611-613 only captures images displayed on the virtual screens 621-623 corresponding to that virtual camera. For example, virtual camera 612 only captures the extracted image 64 of performer 142 displayed on virtual screen 622, and does not capture the extracted images 64 of performers 141 and 142 displayed on virtual screens 621 and 623. By adopting such a structure, the mutual interference between multiple virtual cameras 611-613 and virtual screens 621-623 (the projection of extracted images 64 displayed on other virtual screens) can be eliminated. As described above, the virtual screen 62 used in this embodiment is an object virtually configured in the virtual space 60, so such switching can be easily performed by making appropriate pre-settings. Here, an example is described using multiple cameras 121-123 to film performers 141 and 142 performing simultaneously, and configuring virtual cameras 611-613 and virtual screens 621-623 corresponding to each camera 121-123. However, it can be changed appropriately, for example, using multiple cameras 12 to film performers 14 performing individually.

[0081] As described above, by using multiple cameras 121-123 to film performers 141 and 142 in parallel, and using virtual cameras 611-613 and virtual screens 621-623 corresponding to each camera 121-123, the perspective for filming the performance of performers 141 and 142 can be switched instantly (e.g., capturing the perspectives of both performers 141 and 142, a close-up of performer 141, and a close-up of performer 142). During the production of the video content, the director can select any one of the multiple virtual cameras 611-613 through specified input operations to create video content in real time by combining images captured by the selected virtual cameras 611-613 at various points in time. Images captured by virtual cameras 611-613 selected by the director can also be displayed on screen 17, but since screen 17 displays images for the performer 14 to confirm the state of the interior of the virtual space, it is preferable to display images captured by a pre-determined virtual camera (virtual camera 611 in the above example) that captures the virtual space 60 with a wide angle. Additionally, it is preferable to also save images captured by the unselected virtual camera 61 in storage unit 21. Thus, after the video content is produced in real time, a portion of the images constituting that video content can be changed to images captured by the unselected virtual camera 61.

[0082] Here, the case of using three cameras 121-123, etc., is described, but the number of cameras 12, virtual cameras 61, and virtual screens 62 can be arbitrarily changed. In addition, when the background of the virtual space 60 is uniform (for example, the entire space is the sea, and there are no player characters, etc.), only one set of virtual cameras 61 and virtual screens 62 can be configured in the virtual space 60, and the extracted images 64 of the performer 14 obtained by multiple cameras 121-123 can be switched and displayed on the virtual screens 62.

[0083] In the above embodiments, the positional relationship between the camera 12 and the background member 11 in the real space is reflected in the virtual camera 61 and the virtual screen 62 in the virtual space, but it is also possible to modify it. Specifically, for example, when the director performs an input operation through the input unit 41 of the control / processing device 20, the virtual screen display processing unit 342 can tilt the virtual screen 62 forward or backward from the reference position (the position of the virtual screen 62 that reflects the positional relationship between the camera 12 and the background member 11 in the real space) or change the surface shape of the virtual screen 62 (e.g., bend it).

[0084] Figure 10 An example is shown where the upper part of the virtual screen 62 is tilted forward and backward. Figure 10 The diagram shows the virtual screen 62 positioned at a reference point in the center. The upper part shows an example where the top of the virtual screen 62 is tilted forward (to the side where the virtual camera 61 is located), and the lower part shows an example where the top of the virtual screen 62 is tilted backward. As can be seen from this diagram, by employing the above structure, it is possible to create video content by incorporating an extracted image 64—obtained by arbitrarily altering the image of the actual performer 14 to make the performer 14 appear taller than they actually are—into virtual space.

[0085] In the above embodiment, the virtual space generation unit 341 generates the entire virtual space as a three-dimensional space. However, if the entire virtual space with infinite width is generated as a three-dimensional space, the processing load increases. Therefore, it is also possible to project a space separated from the virtual camera 61 by a predetermined distance as a two-dimensional image onto a background screen positioned at that distance. For example, as... Figure 11 As shown, a hemispherical background screen 65 can be configured with the position of the virtual camera 61 as the center, and the image of the space separated by a specified distance can be projected onto it. Figure 11 Since it's a virtual space existing on the ground, a hemispherical background screen 65 is used. However, if the virtual space is in the air, underwater, or similar locations, a spherical background screen 65 is sufficient. Furthermore, in... Figure 11The background screen is set to a hemispherical shape, but is not limited to a hemispherical shape; other shapes such as a flat screen can also be used. However, in the case of a shape with edges, such as a flat screen, the background screen is configured so that its edges are outside the field of view of the virtual camera 61.

[0086] In the above structure, it is no longer necessary to generate all three-dimensional objects configured in the virtual space, thus reducing the processing load of the virtual space generation unit 341. When the viewer visually confirms the image captured by the virtual camera 61, or the image content composed of the image, they will not carefully examine objects located far from the virtual camera 61. Therefore, by taking into account the resolution of the image acquired by the virtual camera 61 and appropriately determining the aforementioned distance in advance, the processing load of the virtual space generation unit 341 can be reduced without causing a sense of incongruity in the image or video.

[0087] In the above embodiment, the virtual space generation unit 341 generates a virtual space, and image content is created in this virtual space by using the virtual camera 61 to capture images. However, it can also be configured to use a structure that uses multiple virtual spaces. Specifically, image content is created using two virtual spaces with the same structure such as virtual screen 62, player character 63, non-player character, other objects, and background, but with different resolutions.

[0088] Specifically, a first virtual space is used, which is configured with low resolution (e.g., few polygons) and omits the display of surface textures, and a second virtual space is configured with higher resolution (e.g., more polygons) and displays textures on the surface. Figure 12 An example of a first virtual space is shown. In this method, firstly, in the first virtual space, the projection unit 36 ​​projects the extracted image 64 of the performer 14 onto the virtual screen 62, and the virtual camera unit 37 performs filming in the virtual space. Then, based on the obtained data, parameter information such as the position information of the virtual camera 61 and the virtual screen 62 in the virtual space, the position and posture of the player character 63, and the position and posture of the non-player character are acquired. When this information is obtained in the first virtual space, the virtual camera unit 37 performs filming in a second virtual space, in which the player character 63 and others are configured with polygons and textures of sufficient resolution required for the image content, using the parameter information acquired during filming in the first virtual space, and produces (renders) image content. The timing of producing image content in the second virtual space is arbitrary. That is, image content can be produced sequentially in the second virtual space as parameter information is acquired in the first virtual space, or image content can be acquired in the second virtual space after the parameter information required for producing image content is obtained in the first virtual space.

[0089] In the above-described method, the virtual space generation unit 341 and the character display processing unit 35 only need to display the player character and non-player character at a low resolution, thus reducing the processing load during filming. When projecting the image obtained by filming the performer 14 onto the virtual screen 62 in real time, if the processing load for generating the virtual space or processing the display data of the player character 63 is high, the performer 14 and player must remain idle until these processes are completed, placing a heavy burden on them. In contrast, in the above-described method, filming is conducted in a first virtual space where the load for generating the virtual space or processing the display data of the player character 63 is low, thus reducing the burden on the performer 14 and player.

[0090] Industrial availability

[0091] The photographic system involved in this invention can be used to produce various images such as movies, television programs, commercial films (CM), games, and virtual reality (VR) content.

[0092] [Way]

[0093] It will be apparent to those skilled in the art that the above exemplary embodiments are specific examples of the following methods.

[0094] (Item 1)

[0095] One aspect of the present invention relates to a photographic system comprising:

[0096] The photographic information storage unit stores data of images of objects captured by a camera in real space, as well as positional information indicating the positional relationship between the two objects when the camera captured the image.

[0097] The virtual space generation department uses pre-prepared display data that simulates the space used for filming to generate virtual space;

[0098] A virtual screen display processing unit, which configures a transmissive virtual screen in the virtual space;

[0099] An image display processing unit reads image data of the object from the photographic information storage unit and projects the image onto the virtual screen; and

[0100] A virtual photography unit that captures an image of the virtual screen and the virtual space visible through the virtual screen from a predetermined position determined based on the location information within the virtual space.

[0101] In the camera system described in claim 1, the photographic information storage unit stores data of an image of an object captured by a real-world camera in real space, as well as positional information indicating the positional relationship between the object and the camera when the real-world camera captured the object. Here, the object being photographed is, for example, an actor or performer. The image of the object is, for example, an image obtained by photographing the object against a green screen, as is common practice. Additionally, display data for the background of the content is prepared in advance. The virtual space generation unit uses this display data to generate a virtual space. The virtual screen display processing unit configures a transmissive virtual screen in the virtual space, and the image display processing unit displays the image of the object on the virtual screen. A transmissive virtual screen is a virtual object (screen) that is transparent except for the portion displaying the image. Then, the virtual camera unit captures an image containing the virtual screen and the virtual space visible through the virtual screen from a predetermined position determined based on the positional information. In the camera system according to this invention, an image obtained by incorporating an image of an object captured in real space into a virtual space is acquired in this way. In the photographic system described in item 1, the subject can be photographed against a backdrop of, for example, a green screen, in real space, without the need for large and expensive displays with numerous LED light sources, as is required in virtual production. Furthermore, the virtual space serving as the background can be freely designed, and the size and position of the virtual screens within it can be easily changed. Therefore, in the photographic system described in item 1, images of the subject captured from various angles can be obtained inexpensively.

[0102] (Item 2)

[0103] The photographic system mentioned in item 2 is the same as the photographic system mentioned in item 1.

[0104] The photographic information storage unit also stores information about the location of the object in the real space, corresponding to the image data of the object.

[0105] The virtual screen display processing unit also changes the position of the virtual screen based on the position information of the object.

[0106] The photography system described in item 2 is able to reflect the movement of performers in real space into virtual space.

[0107] (Item 3)

[0108] The photographic system referred to in item 3 is the same as the photographic system referred to in item 1 or item 2.

[0109] The virtual screen display processing unit accepts specified input operations to change the posture or shape of the virtual screen.

[0110] The photographic system mentioned in item 3 can produce image content by making the object appear larger (taller) than it actually is, applying arbitrary changes to the image of the actual object, and then incorporating it into virtual space.

[0111] (Item 4)

[0112] The photographic system referred to in item 4 is any of the photographic systems referred to in items 1 through 3.

[0113] The photographic information storage unit also stores information about the camera's position in the real space, corresponding to the image data of the object.

[0114] The virtual camera unit changes the specified position based on the position information of the camera.

[0115] (Item 5)

[0116] The photographic system referred to in item 5 is any of the photographic systems referred to in items 1 through 4.

[0117] The photographic information storage unit also stores information about the camera's perspective in the real space, corresponding to the image data of the object.

[0118] The virtual camera unit changes the shooting angle based on the camera's perspective information.

[0119] In both the photographic systems mentioned in item 4 and item 5, it is possible to capture images that reflect the position and angle of the camera when shooting the object.

[0120] (Item 6)

[0121] The photographic system referred to in item 6 is any of the photographic systems referred to in items 1 through 5.

[0122] The virtual screen is positioned directly opposite the virtual camera unit.

[0123] In the photographic system described in item 6, the virtual screen and the virtual camera unit are configured correspondingly, that is, the optical axis of the virtual camera unit is configured to be perpendicular to the virtual screen. The closer the optical axis of the virtual camera unit is to being parallel to the virtual screen, the more difficult it is to perceive the depth of the object's image. However, by adopting the configuration described in item 6, it is easier to perceive the depth of the object projected onto the virtual screen.

[0124] (Item 7)

[0125] The photographic system referred to in item 7 is one that, in any of the photographic systems referred to in items 1 through 6, further comprises:

[0126] A camera, which captures images of objects in real space; and

[0127] The image extraction unit extracts a region of the object from the image data of the object captured by the camera.

[0128] The image display processing unit displays the extracted image of the object extracted by the image extraction unit on the virtual screen in real time.

[0129] The photographic system mentioned in item 7 is capable of simultaneously capturing images of objects and creating video content in real time.

[0130] (Item 8)

[0131] The photographic system referred to in item 8 is one that, in any of the photographic systems referred to in items 1 to 7, further comprises:

[0132] A first sound collecting unit, which is installed on the object; and

[0133] The first sound output unit outputs the sound data collected by the first sound collection unit from a designated position on the virtual screen.

[0134] In the photography system described in item 8, when the object is a person or other object that makes a sound, it enables the player operating the player character in the virtual space to recognize the sound made from the object.

[0135] (Item 9)

[0136] The photographic system referred to in item 9 is one that, in any of the photographic systems referred to in items 1 through 8, further comprises:

[0137] A second sound collection unit is positioned at a predetermined location on the virtual screen; and

[0138] The second sound output unit is installed on the object and outputs the sound data collected by the second sound collection unit.

[0139] (Item 10)

[0140] The photographic system referred to in item 10 is any of the photographic systems referred to in items 1 through 9.

[0141] The object in question is a person.

[0142] The photographic system also features:

[0143] A first motion detection unit detects a predetermined motion performed by the object; and

[0144] When the first motion detection unit detects the predetermined motion, the first special effects generation unit generates special effects in the virtual space or on the virtual screen that correspond to the predetermined motion.

[0145] (Item 11)

[0146] The photographic system referred to in item 11 is one that, in any of the photographic systems referred to in items 1 through 10, further comprises:

[0147] A player terminal, which operates the player character who performs actions in the virtual space;

[0148] The second motion detection unit detects pre-determined actions of the player character in response to an image of an object displayed on the virtual screen, based on operations performed via the player's terminal; and

[0149] When the predetermined action is detected by the second motion detection unit, the second special effects generation unit generates a special effect corresponding to the predetermined action at the position on the virtual screen where the image of the object is displayed.

[0150] In the photography system described in item 1, when the subject is a person, communication can be achieved between the person and the player character in the virtual space via a virtual screen. Furthermore, it is possible to influence the virtual space from the perspective of the person. That is, starting from the virtual screen, based on the prescribed actions (voices, movements, etc.) of the person, pre-established special effects corresponding to those actions can be generated in the virtual space. Similarly, based on the prescribed actions (voices, movements, etc.) of the player character within the virtual screen, pre-established special effects corresponding to those actions can also be generated on the virtual screen.

[0151] Specifically, for example, in the camera system described in item 9, it is possible to perform while listening to sounds generated in the virtual space. Furthermore, in the camera system described in item 10, interactivity with players operating player characters within the virtual space can be further improved. And, in the camera system described in item 11, interactivity between player characters performing actions within the virtual space and objects can be further improved.

[0152] (Item 12)

[0153] The photographic system referred to in item 12 is any of the photographic systems referred to in items 1 through 11.

[0154] It has multiple groups consisting of the virtual screen and the virtual camera unit corresponding to the virtual screen.

[0155] In the photographic system described in item 12, images of objects captured by multiple cameras in real space (images obtained by shooting the same object from different directions and / or perspectives, and images obtained by shooting different objects) are projected onto different virtual screens, and images of the virtual space containing these images are captured by a virtual camera unit corresponding to the virtual screen, thereby enabling the acquisition of diverse and rich image content.

[0156] (Item 13)

[0157] The photographic system referred to in item 13 is any of the photographic systems referred to in items 1 through 12.

[0158] It also includes a background screen, which is positioned within the virtual space at a predetermined distance from the virtual camera area.

[0159] The virtual space generation unit displays the space within the virtual space that is separated from the virtual photography unit by a predetermined distance as a two-dimensional image on the background screen.

[0160] In the photographic system described in item 13, it is no longer necessary to generate all the three-dimensional objects configured in the virtual space, thus reducing the processing load of the virtual space generation unit.

[0161] (Item 14)

[0162] The photographic system referred to in item 14 is any of the photographic systems referred to in items 1 through 13.

[0163] The virtual space generation unit generates a first virtual space and a second virtual space with a display resolution higher than that of the first virtual space.

[0164] The virtual screen and the image display processing unit are respectively set up for the first virtual space and the second virtual space.

[0165] After obtaining parameter information of objects existing in the first virtual space by photographing the first virtual space, the virtual photography unit reflects the parameter information in the second virtual space to photograph the second virtual space.

[0166] In the photographic system described in item 14, during shooting using the first virtual space, objects can be displayed at a low resolution, thus reducing the processing load on the virtual space generation unit and efficiently acquiring parameter information required for content creation. Furthermore, by creating content by reflecting parameter information in the second virtual space, image content can be efficiently created in a high-resolution virtual space.

[0167] Explanation of reference numerals in the attached figures

[0168] 1: Photography system; 100: Studio; 10: Photography equipment; 11: Background components; 111: Grid points; 12, 121~123: Cameras; 13: Position sensors; 131~133: Reference points; 134: Main subject; 14, 141, 142: Performers; 15: Sound collection unit; 16: Headphones; 17: Screen; 20: Control / processing device; 21: Storage unit; 22: Photography information storage unit; 23: Virtual space display data storage unit; 24: Character data storage unit; 25: Photography data storage unit; 30: Content production program; 31: Image extraction unit; 32: Camera perspective acquisition unit; 3 3: Location information acquisition unit; 341: Virtual space generation unit; 342: Virtual screen display processing unit; 35: Character display processing unit; 36: Projection unit; 37: Virtual photography unit; 38: Sound processing unit; 41: Input unit; 42: Display unit; 50: Player terminal; 51: Storage unit; 52: Input operation unit; 53: Sound input unit; 54: Sound output unit; 55: Display unit; 60: Virtual space; 61, 611~613: Virtual camera; 62, 621~623: Virtual screen; 63: Player character; 64: Extracted image of the performer projected onto the virtual screen; 71: Reference object; 9: Network.

Claims

1. A photographic system, characterized in that, have: The photographic information storage unit stores data of images of objects captured by a camera in real space, as well as positional information indicating the positional relationship between the two objects when the camera captured the image. The virtual space generation department uses pre-prepared display data that simulates the space used for filming to generate virtual space; A virtual screen display processing unit, which configures a transmissive virtual screen in the virtual space; An image display processing unit reads image data of the object from the photographic information storage unit and displays the image on the virtual screen; as well as A virtual photography unit that captures an image of the virtual screen and the virtual space visible through the virtual screen from a predetermined position determined based on the location information within the virtual space.

2. The photographic system according to claim 1, characterized in that, The photographic information storage unit also stores information about the location of the object in the real space, corresponding to the image data of the object. The virtual screen display processing unit also changes the position of the virtual screen based on the position information of the object.

3. The photographic system according to claim 1, characterized in that, The virtual screen display processing unit accepts specified input operations to change the posture or shape of the virtual screen.

4. The photographic system according to claim 1, characterized in that, The photographic information storage unit also stores information about the camera's position in the real space, corresponding to the image data of the object. The virtual camera unit changes the specified position based on the position information of the camera.

5. The photographic system according to claim 1, characterized in that, The photographic information storage unit also stores information about the camera's perspective in the real space, corresponding to the image data of the object. The virtual camera unit changes the shooting angle based on the camera's perspective information.

6. The photographic system according to claim 1, characterized in that, The virtual screen is positioned directly opposite the virtual camera unit.

7. The photographic system according to claim 1, characterized in that, It also has: A camera, which captures images of objects in real space; and The image extraction unit extracts a region of the object from the image data of the object captured by the camera. The image display processing unit displays the extracted image of the object extracted by the image extraction unit on the virtual screen in real time.

8. The photographic system according to claim 1, characterized in that, It also has: A first sound collecting unit, which is installed on the object; and The first sound output unit outputs the sound data collected by the first sound collection unit from a designated position on the virtual screen.

9. The photographic system according to claim 1, characterized in that, It also has: A second sound collection unit is positioned at a predetermined location on the virtual screen; and The second sound output unit is installed on the object and outputs the sound data collected by the second sound collection unit.

10. The photographic system according to claim 1, characterized in that, The object in question is a person. The photographic system also features: A first motion detection unit detects a predetermined motion performed by the object; and When the first motion detection unit detects the predetermined motion, the first special effects generation unit generates special effects in the virtual space or on the virtual screen that correspond to the predetermined motion.

11. The photographic system according to claim 1, characterized in that, It also has: A player terminal, which operates the player character who performs actions in the virtual space; The second motion detection unit detects actions based on pre-determined actions of the player character in response to an image of an object displayed on the virtual screen, performed through the player's terminal. as well as When the predetermined action is detected by the second motion detection unit, the second special effects generation unit generates a special effect corresponding to the predetermined action at the position on the virtual screen where the image of the object is displayed.

12. The photographic system according to claim 1, characterized in that, It has multiple groups consisting of the virtual screen and the virtual camera unit corresponding to the virtual screen.

13. The photographic system according to claim 1, characterized in that, It also includes a background screen, which is positioned within the virtual space at a predetermined distance from the virtual camera area. The virtual space generation unit displays the space within the virtual space that is separated from the virtual photography unit by a predetermined distance as a two-dimensional image on the background screen.

14. The photographic system according to claim 1, characterized in that, The virtual space generation unit generates a first virtual space and a second virtual space with a display resolution higher than that of the first virtual space. The virtual screen and the image display processing unit are respectively set up for the first virtual space and the second virtual space. After obtaining parameter information of objects existing in the first virtual space by photographing the first virtual space, the virtual photography unit reflects the parameter information in the second virtual space to photograph the second virtual space.

Images