Content Creation Program

Abstract

To provide a technique capable of acquiring a video image of an object captured at various angles of view at low cost.SOLUTION: An imaging system (1) includes: an imaging information storage part (22) that stores data of an image of an object captured by a camera in a real space and position information indicating a positional relationship between the image and the position information when the object is captured by the camera; a virtual space generation part (341) that generates a virtual space using a display data that simulates a space in which content is captured, which is prepared in advance; a virtual screen display processing part (342) that arranges a transmissive virtual screen (62) in the virtual space; an image display processing part (36) that reads the data of the image of the object from the imaging information storage part and displays the image on the virtual screen; and a virtual imaging part (37) that captures the image including the virtual screen together with the virtual space viewed through the virtual screen from a predetermined position determined on the basis of position information in the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a photography system used to create content such as movies and television programs. [Background technology]

[0002] Filming systems using chromakey compositing are widely used to create content such as movies and television programs. In a filming system using chromakey compositing, a performer such as an actor is photographed with a camera against a background colored a predetermined color, known as a green screen, and an image of the performer's area is extracted by excluding the area of ​​the predetermined color. The extracted image is then composited with a separately prepared image using computer graphics (CG) or the like to create the content.

[0003] Recently, a technology called virtual production has been proposed (see, for example, Patent Document 1). In virtual production, an image is projected onto a display consisting of many LED light sources, and a camera captures actors performing against this background. This system simultaneously captures the actors and the background, eliminating the need for image compositing, and allowing content to be created in a short amount of time. This system displays images with a sense of depth on the display, and by changing the way the images are displayed on the display depending on the relative positions of the camera and display, it is possible to create content that makes it seem as if actors are performing in real space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-35420 Summary of the Invention [Problem to be solved by the invention]

[0005] In virtual production, displays must be positioned to cover the entire area seen by the camera, which necessitates the use of large, expensive screens equipped with numerous LED light sources in order to capture images of performers and other subjects from various angles.

[0006] The problem to be solved by the present invention is to provide a technology that can acquire, at low cost, images of an object captured at various angles of view. [Means for solving the problem]

[0007] The imaging system according to the present invention, which is made to solve the above problems, comprises: an image information storage unit that stores image data of an object captured by a camera in real space and position information that indicates the positional relationship between the object and the camera when the object is captured by the camera; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that reads out image data of the object from the photographing information storage unit and displays the image on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen; The present invention is characterized by comprising: [Effects of the Invention]

[0008] In the imaging system according to the present invention, a shooting information storage unit stores image data of an object captured by a camera in real space and position information indicating the relative positions of the object and the object when the object was captured by the camera. The object captured here is, for example, a performer such as an actor. The image of the object is, for example, an extracted image of the object obtained by capturing the object against a conventional green screen, from which a portion necessary for compositing is extracted (an extracted image of the object). Display data that will serve as the background of the content is also prepared in advance. A virtual space generation unit generates a virtual space using the display data. A virtual screen display processing unit places a transparent virtual screen in the virtual space, and an image display processing unit displays an image of the object on the virtual screen. A transparent virtual screen is a virtual object (screen) that functions as a reference plane for setting the imaging plane (focus) of the virtual capture unit in the virtual space and is transparent except for the area where the image is displayed. The virtual capture unit captures an image including the virtual screen, along with the virtual space visible through the virtual screen, from a predetermined position determined based on the position information. In this way, the imaging system according to the present invention acquires an image in which the image of the object captured in real space is incorporated into the virtual space. With the imaging system of the present invention, it is possible to capture an object in real space against, for example, a conventional green screen, eliminating the need for a large, expensive display with numerous LED light sources, as in virtual production. Furthermore, the virtual space that serves as the background can be freely designed, and the size and position of the virtual screen placed therein can be easily changed. Therefore, with the imaging system of the present invention, it is possible to capture images of an object captured at various angles of view at low cost. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the configuration of a main part of an embodiment of an imaging system according to the present invention; [Figure 2] 10 shows an example of an image obtained by photographing with a wide-angle lens of a camera in the photographing system of this embodiment. [Figure 3]FIG. 2 is a diagram showing a state in which a performer is photographed with a wide-angle lens of a camera in the photographing system of the present embodiment. [Figure 4] 10A and 10B are diagrams showing a state in which a performer is photographed with a narrow-angle lens of a camera in the photographing system of the present embodiment. [Figure 5] 10 is an example of an image obtained by photographing with a narrow-angle lens of a camera in the photographing system of this embodiment. [Figure 6] 10A and 10B are diagrams illustrating a state in which an extracted image of a performer is projected onto a virtual screen placed in a virtual space in the imaging system of the present embodiment. [Figure 7] 10A and 10B are diagrams illustrating a modified example of the camera angle of view acquisition unit in the imaging system of the present embodiment. [Figure 8] 1 is a diagram illustrating an example in which a performer is photographed by a plurality of cameras in a studio in the photographing system of the present embodiment. [Figure 9] 10A and 10B are diagrams illustrating an example in which a plurality of sets of virtual cameras and virtual screens are arranged in a virtual space in the imaging system of the present embodiment. [Figure 10] 10A and 10B are diagrams illustrating an example in which the virtual screen is tilted forward and backward in the imaging system of the present embodiment. [Figure 11] 10A and 10B are diagrams illustrating an example in which a background screen is arranged in the imaging system of the present embodiment. [Figure 12] FIG. 3 is a diagram illustrating an example of a first virtual space that can be used in the imaging system of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an imaging system according to the present invention will be described below with reference to the drawings.

[0011] Fig. 1 shows the main configuration of a photography system 1 of this embodiment. The photography system 1 of this embodiment has a photography device 10, a control and processing device 20, and a player terminal 50. The photography device 10, the control and processing device 20, and the player terminal 50 are connected to each other so that they can communicate with each other via a wired or wireless network 9. Although Fig. 1 shows only one player terminal 50, there may be multiple player terminals 50.

[0012] The filming device 10 is placed in a studio 100 where a performer 14, who is the subject of filming, performs. The filming device 10 includes a background member 11 called a green screen, which is colored a predetermined color (e.g., green), a camera 12, a position sensor 13, and a sound collection unit 15. The performer 14 also wears earphones 16 as an audio output unit. In this embodiment, earphones 16 are used, but speakers may also be placed in the studio 100. Furthermore, a screen 17 as a display unit is placed in the studio 100 at a position outside the field of view of the camera 12, and an image captured by a virtual camera 61 provided in a virtual space 60, which will be described later, is displayed on the screen 17 in real time.

[0013] As shown in FIG. 2, the background member 11 has a grid of reference points (grid points 111). In this embodiment, seven lattice points 111 are provided in both the horizontal and vertical directions (a total of 49 lattice points). The spacing and number of the lattice points 111 can be changed as desired. In this embodiment, the reference points are provided in a grid pattern; however, the reference points may be provided two-dimensionally and regularly on the surface of the background member 11, and may be provided in a honeycomb pattern, for example. The camera 12 captures an actor 14 with the background member 11 as the background. The position sensor 13 in this embodiment is an infrared LiDAR (Light Detection and Ranging) sensor that irradiates the surroundings with infrared light from a main body 134 and receives reflected light from reference points 131 and 132 provided at two locations on the camera 12 (e.g., the tip of the lens and the main body of the camera 12) and a reference point 133 provided at one location on the actor 14 (e.g., the nape of the neck), thereby measuring the positions of the reference points 131 to 133. For example, stickers that reflect infrared light are attached to the reference points 131 to 133. The positions and number of the reference points 131 to 133 are also just an example and can be changed as appropriate as long as the above functions can be achieved.

[0014] The control / processing device 20 includes a storage unit 21. The storage unit 21 includes a shooting information storage unit 22, a virtual space display data storage unit 23, a character data storage unit 24, and a shooting data storage unit 25. The shooting information storage unit 22 stores shooting information including data on images of objects captured in real space, the posture of the camera 12 when the images were captured, and information associating the positions of the camera 12 and the performer 14. Note that the concept of an image in this specification includes not only still images but also moving images (e.g., moving images composed of still images at 60 frames per second). The virtual space display data storage unit 23 stores in advance time-series display data in a 3D (three-dimensional) virtual space simulating various scenes, which will be used for content created by the shooting system 1. This display data also includes audio data such as sound effects. The character data storage unit 24 stores time-series display data such as 3D avatar display data for the player character (PC) operated by the player and 3D avatars for non-player characters (NPCs) not operated by the player. As described above, since non-player characters are not operated by the player, their actions during filming of the video content are determined in advance according to a scenario, and the time-series display data (including audio data) is stored in the character data storage unit 24.

[0015] The control and processing device 20 includes, as functional blocks, an image extraction unit 31, a camera angle of view acquisition unit 32, a position information acquisition unit 33, a virtual space generation unit 341, a virtual screen display processing unit 342, a character display processing unit 35, an image display processing unit 36, a virtual shooting unit 37, and an audio processing unit 38. The operation of each of these units will be described later. The actual entity of the control and processing device 20 is, for example, a general-purpose personal computer, and is realized by executing a dedicated program (content creation program 30) pre-installed on the processor. The control and processing device 20 may be located in an appropriate location inside or outside the studio 100. Alternatively, a cloud server provided on a network may be used as the control and processing device 20. An input unit 41 including a keyboard, a mouse, etc., and a display unit 42 including a liquid crystal display, etc., are also connected to the control and processing device 20.

[0016] The player terminal 50 includes a memory unit 51. The memory unit 51 stores information about the player using the player terminal 50 (such as the player's ID information). The player terminal 50 includes an input operation unit 52, an audio input unit 53, an audio output unit 54, and a display unit 55. The player terminal 50 may be, for example, a head-mounted display equipped with a motion capture or position sensor that functions as the input operation unit 52. The player terminal 50 can be installed in any location. In other words, the player can control the player character from any location inside or outside the studio 100.

[0017] Next, a procedure for creating video content using the imaging system 1 of this embodiment will be described.

[0018] When creating video content, the background space of the content is set as a 3D virtual space in advance, and time-series data for generating a 3D virtual space that changes over time in accordance with a scenario is created and stored in the virtual space display data storage unit 23. Additionally, time-series display data for 3D avatars of player characters appearing in the content and 3D avatars of non-player characters are stored in the character data storage unit 24. A player character is, for example, a character who appears in the content and interacts with the performer, such as through conversation. A non-player character is, for example, a mob character that appears as a crowd in the background of the content, an enemy character that moves based on a predetermined algorithm and fights the performer or player character, or an object positioned at a predetermined location in the virtual space. When there are multiple player characters, information associating each player character with information (such as terminal ID information) of the player terminal 50 used by the player controlling that player character is also stored. Hereinafter, for convenience, the display mode of a non-player character, which is an object, will also be referred to as an avatar.

[0019] Once the above preparations are complete, preparations for filming the video content begin. The preparations for filming the video content are carried out in both the studio 100 set up in the real space and the virtual space 60.

[0020] First, the preparations for filming the performer 14 in a studio 100 set in real space will be described. A background member 11 called a green screen is placed in the studio 100. Reference points 131 and 132 of a LiDAR sensor serving as a position sensor 13 are set on the lens and body of the camera 12, respectively. Furthermore, a reference point 133 of the LiDAR sensor is set at a predetermined position on the performer 14. In this embodiment, infrared light-reflecting stickers are attached to the lens and body of the camera 12 and to the neck of the performer 14. A LiDAR sensor body 134 is placed at a predetermined position in the studio 100 (for example, a corner of the ceiling). The operating principle of LiDAR is well known, so a detailed description will be omitted here. Furthermore, a sound collection unit 15 for collecting sounds emitted by the performer 14 and a screen 17 are placed at positions outside the angle of view of the camera 12 in the studio 100. Furthermore, the performer 14 wears earphones 16.

[0021] Camera 12 photographs performer 14 standing in an initial position with background member 11 as the background, and acquires image data of performer 14. The image data of performer 14 acquired by camera 12 is stored in shooting information storage unit 22 together with a timestamp indicating the time when the image was acquired (time of shooting, time elapsed since the start of the performance, etc.). In addition, sounds made by performer 14 during the performance are collected by sound collection unit 15 and stored in shooting information storage unit 22.

[0022] 3 and 4 show examples of photographing the performer 14 in the studio 100. FIG. 3 shows the state in which the performer 14 is photographed with the wide-angle lens of the camera 12, and FIG. 4 shows the state in which the performer 14 is photographed with the narrow-angle lens of the camera 12. When the performer 14 is photographed with the wide-angle lens of the camera 12 as in FIG. 3, an image capturing the entire body of the performer 14 is obtained as shown in FIG. 2. When the performer 14 is photographed with the narrow-angle lens of the camera 12 as in FIG. 4, an enlarged image capturing a portion of the performer 14 (in this example, the upper body) is obtained as shown in FIG. 5. As shown in FIGS. 4 and 5, the grid points 111 provided on the background member 11 are also photographed in the image of the performer 14.

[0023] In the control and processing device 20, the image extraction unit 31 reads out the image data stored in the shooting information storage unit 22. Then, it extracts the image data excluding the area of ​​a predetermined color contained in the image (the green screen portion, including the grid point 111). This generates an image (extracted image) from which the performer 14 has been cut out. The generated extracted image is also stored in the shooting information storage unit 22.

[0024] The camera angle of view acquisition unit 32 calculates the number of grid points 111 included in the image stored in the shooting information storage unit 22 (in this embodiment, the number of grid points 111 arranged vertically and the number of grid points 111 arranged horizontally. This number includes the number of grid points 111 hidden by the performer 14). Then, based on the number of grid points 111, the angle of view (wide angle, narrow angle, etc.) of the lens of the camera 12 when the image was captured is calculated. If there is no change in the positions of the camera 12 and the performer 14, the greater the number of grid points included in the image, the wider the lens of the camera 12, and the fewer the number of grid points included in the image, the narrower the lens of the camera 12 (zoomed state). Furthermore, the camera angle of view acquisition unit 32 determines the positions of reference points 131 and 132 set on the camera 12 based on the output signal from the position sensor 13, and determines the attitude (elevation angle θ) of the camera 12 from the relative positional relationship between the two.

[0025] Based on the output signal from the position sensor 13, the position information acquisition unit 33 determines the positions of the reference point 131 (or / and 132) set on the camera 12 and the reference point 133 set on the performer 14, and determines the positional relationship between them. The information on the positional relationship between them is information on the direction in which the camera 12 captures the performer 14 and information on the distance between the camera 12 and the performer 14 (i.e., vector information). The information on the camera attitude acquired by the camera angle of view acquisition unit 32 and the information on the positional relationship between the camera 12 and the performer 14 acquired by the position information acquisition unit 33 are each stored in the shooting information storage unit 22 together with a timestamp indicating the time when the information was acquired.

[0026] The above-mentioned processing by the image extraction unit 31, the camera angle of view acquisition unit 32, and the position information acquisition unit 33 is executed in real time while the performer 14 is being photographed in the studio 100.

[0027] Next, preparation for shooting in the virtual space will be described.

[0028] The virtual space generation unit 341 first reads, from the virtual space display data storage unit 23, time-series display data of the 3D virtual space in which the video content is shot, and generates a virtual space based on the display data at the start point. Furthermore, the virtual screen display processing unit 342 places a transparent virtual screen 62 at a predetermined initial position within the virtual space, as illustrated in FIG. 6. The transparent virtual screen 62 is a virtual object (screen) that can function as a reference surface when setting the imaging plane (focus) of the virtual imaging unit 37 in the virtual space 60, and is transparent except for the area onto which the image is projected. Because the virtual screen 62 is a virtual object, player characters and non-player characters can freely pass through the virtual screen 62.

[0029] The character display processing unit 35 reads the display data of the player character avatar and the non-player character avatar, and places them at their initial positions within the virtual space. The virtual shooting unit 37 places a virtual camera 61 in the virtual space 60 based on the information on the positional relationship between the camera 12 and the performer 14 obtained by the position information acquisition unit 33. Specifically, the position of the performer 14 is matched to the position of the virtual screen 62, and based on this, the relative positional relationship between the camera 12 and the performer 14 is reflected in the positional relationship between the virtual screen 62 and the virtual camera 61 to determine the position of the virtual camera 61. In this embodiment, the virtual camera 61 is positioned directly facing the virtual screen 62. In other words, the virtual camera 61 and the virtual camera 61 are positioned so that the optical axis of the virtual camera 61 is perpendicular to the surface of the virtual screen 62. The size of the virtual screen 62 is changed appropriately depending on the size of the extracted image 64 of the performer 14 displayed on the virtual screen 62.

[0030] The image display processing unit 36 ​​displays the extracted image 64 of the performer 14 extracted by the image extraction unit 31 on the virtual screen 62. In this embodiment, the image display processing unit 36 ​​projects the extracted image 64 of the performer 14 from the position of the virtual camera 61 toward the virtual screen 62, thereby displaying the extracted image 64 of the performer 14 on the virtual screen 62. The relative positional relationship between the virtual camera 61 and the virtual screen 62 reflects the positional relationship between the camera 12 and the screen 17. Therefore, by projecting the extracted image 64 of the performer 14 from the virtual camera 61 toward the virtual screen 62, an image of the performer 14 captured at the same angle of view as that captured by the camera 12 in real space can be displayed on the virtual screen 62. However, the method of displaying the extracted image 64 of the performer 14 on the virtual screen 62 is not limited to this; various methods can be used, such as transmitting pixel data constituting the extracted image 64 to the projection screen 62. In this way, a real-life image of the performer 14 captured in real space is incorporated into the virtual space.

[0031] As described above, the virtual screen 62 is a transparent type, so as shown in Fig. 6, the virtual camera 61 simultaneously captures an extracted image 64 of the performer 14 projected onto the virtual screen 62 and a virtual space 60 positioned in the background of the virtual screen 62. The virtual shooting unit 37 uses the virtual camera 61 to shoot images of the virtual space 60 at a predetermined frame rate (for example, 60 frames per second). The data of the shot images is stored in the shooting data storage unit 25 and is also displayed in real time on the screen 17 of the studio 100.

[0032] The virtual space generation unit 341 generates an image capturing the virtual space from the position of the player character 63 placed at an initial position, and transmits the image to the player terminal 50 used by the player operating the player character 63. The received image is displayed on the display unit 55 of the player terminal 50. Although only one player character 63 is shown in FIG. 6, multiple player characters 63 may exist at the same time. Note that, as shown in FIG. 6, when the player character 63 is located behind the virtual screen 62, a silhouette of an extracted image 64 of the performer 14 is displayed on the display unit 55. When the player character 63 is located on the front side of the virtual screen 62 (the side where the virtual camera 61 is located), an image of the extracted image 64 of the performer 14 viewed from the front is displayed on the display unit 55.

[0033] After preparations for filming are completed in the studio 100 set up in the real space and the virtual space 60 as described above, the director issues a command to start filming by a predetermined action (for example, giving a voice command to start filming in the studio 100 and simultaneously pressing a filming start button via the input unit 41). In response to this, the performer 14 in the studio 100 begins performing according to a pre-prepared scenario. The performance of the performer 14 includes vocalizations, and the vocalizations of the performer 14 are collected by the sound collection unit 15 and transmitted to the control and processing device 20 (corresponding to the function of the first sound collection unit in the present invention). In the control and processing device 20, the audio processing unit 38 outputs the audio received from the sound collection unit 15 as audio emanating from a predetermined position on the virtual screen 62 in the virtual space 60 (for example, the center of gravity of the extracted image 64) (corresponding to the function of the first audio output unit in the present invention).

[0034] Furthermore, the virtual space generation unit 341 and the character display processing unit 35 read the time-series display data of the virtual space and the time-series display data of the non-player characters from the virtual space display data storage unit 23 and the character data storage unit 24, respectively, and sequentially reflect (play) them in the virtual space 60. This display data also includes audio data, and the audio data is played as audio emanating from the position of the non-player character in the virtual space.

[0035] In the player terminal 50, the player controls the player character 63 via the input operation unit 52 and also makes sounds according to a prepared scenario. The actions input by the player via the input operation unit 52 are sequentially transmitted to the control and processing device 20 and reflected in the actions of the player character 63 in the virtual space. In addition, the sounds made by the player are collected by the sound input unit 53 and sequentially transmitted to the control and processing device 20. In the control and processing device 20, the sound processing unit 38 outputs the received sounds as sounds made from the position of the player character 63 in the virtual space 60.

[0036] The audio processing unit 38 collects audio from within the virtual space 60 at a predetermined position on the virtual screen 62 (for example, the center of gravity of the extracted image 64) (corresponding to the function of the second audio collection unit in the present invention). The collected audio is transmitted to the imaging device 10 and output from earphones 16 worn by the performer 14 (corresponding to the function of the second audio output unit in the present invention). This allows the performer 14 to visually confirm the state of the virtual space 60 on the screen 17 and to perform their acting while listening to audio emitted in the virtual space 60 and captured at the position of the virtual screen 62 through the earphones 16. The audio processing unit 38 also collects audio from within the virtual space 60 at the position of the player character 63 in the virtual space 60. The collected audio is transmitted to the player terminal 50 used by the player operating the player character 63 and output from the audio output unit 54. This allows the player operating the player character 63 to view the state within the virtual space 60 from the player character's 63's perspective on the display unit 55, and to move the player character 63 while listening to the sound emitted in the virtual space 60 and captured at the position of the player character 63 on the sound output unit 54 (including the sound of the performer 14 emitted from a specified position on the virtual screen 62).

[0037] During filming, the positions of the camera 12 and the performer 14 are monitored by the position information acquisition unit 33 based on the output signal from the position sensor 13. When the camera 12 or the performer 14 moves (changes their position and / or posture), the position information acquisition unit 33 in the control and processing device 20 detects this information. The position of the camera 12 is associated with the position of the virtual camera 61, and when the position information acquisition unit 33 detects the movement of the camera 12, the virtual filming unit 37 moves the virtual camera 61 placed in the virtual space 60 in accordance with the movement of the camera 12. The position of the performer 14 is associated with the position of the virtual screen 62, and when the position information acquisition unit 33 detects the movement of the performer 14, the virtual screen display processing unit 342 moves the virtual screen 62 placed in the virtual space 60 in accordance with the movement of the performer 14.

[0038] The angle of view of camera 12 is also monitored by camera angle of view acquisition unit 32. Changes in the angle of view of camera 12 are detected based on the number of grid points included in the image captured by camera 12. Information about the angle of view of camera 12 acquired by camera angle of view acquisition unit 32 is reflected in the angle of view of virtual camera 61 by virtual capture unit 37. For example, if the angle of view of camera 12 is changed so that it captures a magnified image of performer 14, the angle of view of virtual camera 61 is also changed in virtual space so that it captures a zoomed-in image of performer 14 displayed on virtual screen 62. Therefore, the same state of performer 14 as captured by camera 12 in real space is captured by virtual camera 61 in virtual space.

[0039] When the performer 14 has completed his / her performance based on a prepared scenario, the director instructs the end of filming by a predetermined action (for example, by giving a voice command to end filming in the studio 100 and simultaneously pressing the filming end button via the input unit 41). This causes the virtual space generation unit 341 and the character display processing unit 35 to end playback of the time-series display data of the virtual space 60 and the time-series display data of the non-player character avatar, respectively. Furthermore, the player's operation of the player character 63 on the player terminal 50 ends. The virtual filming unit 37 saves the image data acquired by the virtual camera 61 at a predetermined frame rate in the filming data storage unit 25 as a single video file.

[0040] In the filming system 1 of this embodiment, image data of the performer 14 performing in a real-space studio 100 is displayed on a transparent virtual screen 62 arranged in a virtual space 60, thereby enabling the creation of video content incorporating a live-action image of the performer 14 in the virtual space. The filming system 1 of this embodiment simply captures an object against a green screen in the real-space studio 100 and obtains an extracted image 64 by extracting the image of the performer 14 using conventional processing. This eliminates the need for a large, expensive display with multiple LED light sources, as in virtual production. Furthermore, because the background of the video content is a virtual space, the position and size of the virtual screen 62 onto which the live-action extracted image 64 is projected can be easily changed to match the angle of view of the camera 12, the position of the performer 14, and the relative positions of the camera 12 and the performer 14. Therefore, video of an object captured at various angles of view can be obtained at low cost.

[0041] The filming system 1 of this embodiment can create video content in which the player character 63 and non-player characters operated by the player using the player terminal 50 appear together with the performer 14 who is filmed in live action.

[0042] Previously, when combining images or videos of performers with a separately prepared background image, the only way to create a composite was to combine a layer of the performer's image, created as a single image file, with the background image. This meant that, like combining a cel drawing of a character with a background image in the production of classic animation, the composite was merely a composite of two independent images. This meant that, for example, the performer's line of sight or posture would not match the background image, creating a sense of incongruity in the composite image.

[0043] In contrast, in the filming system 1 of this embodiment, images captured by a virtual camera 61 located in a virtual space 60 are displayed in real time on a screen 17 located in the studio 100. Furthermore, the sound emitted by the performer 14 is reflected in the virtual space, and the sound in the virtual space is output from the earphones 16 of the performer 14 at the position of a virtual screen 62 located in the virtual space. This allows interactive communication (conversation, etc.) between the performer 14 and a player character 63 or non-player character located in the virtual space 60. Furthermore, the shadow of the performer 14 can be made to appear in the virtual space 60 in real time, and the sound emitted by the player character 63 or the like in the virtual space 60 can be heard by the performer 14 at a volume corresponding to the distance between the player character 63 and the performer 14. Therefore, the performer 14 can perform his / her performance while experiencing the scenario. Furthermore, since post-processing such as image compositing is not required, the cost of such processing can be reduced.

[0044] In the above embodiment, a case where video content is created by filming the performances of the performers 14 and the player characters 63 has been described, but a similar configuration can also be used to play a game in a virtual space and film the play scenes. When playing a game, a configuration with even higher interactivity than the above embodiment can be used.

[0045] For example, in a battle game, when an attack made by the player character 63 hits the extracted image 64 of the performer 14, a predetermined effect (e.g., blood splatter) can be generated at the location on the virtual screen 62 where the attack hit. Alternatively, conversely, when the performer 14 attacks the player character 63, an attack can be generated against the player character 63 from the display position of the extracted image 64 of the performer 14 on the virtual screen 62 (e.g., an image of a bullet being fired from the position of the extracted image 64 in response to a gun shot made by the performer 14 is generated). Furthermore, it is also possible to generate a predetermined effect not only between the player character 63 and the performer 14, but also in the virtual space 60 in response to a predetermined action made by the performer 14 (e.g., darkening the display of the virtual space 60).

[0046] Such effect data may be stored in an effect data storage unit provided in advance in the storage unit 21 of the control / processing device 20, for example. Alternatively, the effect data may be stored in the virtual space display data storage unit 23 or the character data storage unit 24. Furthermore, the control / processing device 20 may include, as functional blocks, an effect generation unit (first effect generation unit) that reads effect data from the storage unit when a motion detection unit (first motion detection unit, e.g., a motion sensor) detects that the performer 14 has performed a predetermined motion and generates an effect such as a hit or an attack on the virtual screen 62 or the virtual space 60, and an effect generation unit (second effect generation unit) that reads effect data from the storage unit when a motion detection unit (second motion detection unit) detects that the player character 63 has performed a predetermined motion on the extracted image 64 of the target object through an operation by the player via the player terminal and generates an effect on the extracted image on the virtual screen 62. Alternatively, the functions of these effect generation units may be performed by the virtual space generation unit 341 or the virtual screen display processing unit 342.

[0047] Except for the fact that the effects are executed via the virtual screen 62, the algorithms for generating the effects are similar to those used in conventional games played in virtual space, and therefore the techniques used in those games may be used as appropriate. Here, a battle game has been used as an example, but a similar configuration can also be suitably used when creating video content for battle scenes. Of course, these effects are not limited to battle games and battle scenes, and a similar configuration can be used in the production of games other than battle games and video content other than battle scenes, with the necessary effects prepared in advance.

[0048] The above embodiment has been described as a specific example of a preferred embodiment of the photography system 1 according to the present invention, and can be modified as appropriate in accordance with the spirit of the present invention.

[0049] In the above embodiment, lattice points 111 are provided on the background member 11, and the angle of view (wide angle, narrow angle) of the camera 12 is determined based on the number of lattice points 111 contained in the image captured by the camera 12, but the angle of view of the camera 12 may also be determined by other methods.

[0050] For example, by machine learning training data that identifies each part (head, hands, feet, etc.) contained in an image of a human (performer, etc.) photographed at various angles of view, a trained model can be constructed that outputs the angle of view of the camera at the time the image was taken in response to an input image of a human, and a classifier equipped with this trained model can be used as a camera angle of view acquisition unit.

[0051] Alternatively, as shown in Fig. 7, a reference object of a predetermined size may be placed on a specific part of the performer (face, hands, feet, etc.), and a camera angle of view acquisition unit may be used that determines the camera angle of view based on the size of the reference object 71 (e.g., a sticker) included in the image of the performer. The reference object may be a circular sticker or accessory with a diameter of several centimeters. Alternatively, a sticker attached to the performer 14 may be used as the reference point 133 of the position sensor 13.

[0052] In the above embodiment, one performer 14 was photographed against the background member 11, but multiple performers 14 may also be photographed as shown in Figure 7. In this case, an extracted image 64 is created for each performer 14 and projected onto the virtual screen 62.

[0053] In the above embodiment, image data obtained by actually photographing the performers in studio 100 is projected in real time onto virtual screen 62 arranged in virtual space, but performer image data may also be obtained in advance. In that case, along with the time-series data of the performer's images, information on the position and angle of view of camera 12, the position of performer 14, and the positions of both may also be created as time-series data and stored in shooting information storage unit 22. When creating video content, when playing back the time-series display data of the virtual space and the time-series display data of the characters in virtual space 60, the time-series data of the performer's images, etc. stored in shooting information storage unit 22 may be projected onto virtual screen 62 in virtual space 60, aligning the positions of the timestamps.

[0054] In the above embodiment, the performer 14 was photographed against a background member 11 (such as a green screen) placed in the studio 100, but it is also possible to photograph the performer 14 without using the background member 11 and obtain an extracted image of the performer 14 from the photograph. Specifically, for example, the performer 14 is photographed by focusing the camera 12 on the performer 14, and only the portion that is in focus is extracted from the obtained image to obtain an extracted image. Using this method, it is possible to obtain an extracted image from an image of a performer such as an actor photographed outdoors (location shooting) without using a green screen, and to create video content that combines this extracted image with a virtual space in real time.

[0055] Furthermore, the configuration described below can be added to the above embodiment.

[0056] In the above embodiment, the performer 14 was photographed with one camera 12 and an extracted image 64 of the performer 14 was displayed on one virtual screen 62, but the performer 14 may be photographed with multiple cameras 12 and the extracted image 64 of the performer 14 obtained by each camera 12 may be displayed on a virtual screen 62 associated with each camera 12, and an image of the virtual space 60 including the extracted image 64 may be photographed with multiple virtual cameras 61.

[0057] Fig. 8 shows a state in which performers 141 and 142 are photographed by multiple cameras 121 to 123 in studio 100, in a plan view of studio 100. Fig. 9 also shows a plan view of virtual space 60, in which multiple virtual cameras 611 to 613 and virtual screens 621 to 623 are arranged in virtual space 60.

[0058] As shown in Fig. 8, in studio 100, camera 121 simultaneously captures both of two performers 141 and 142 from the front at a wide angle. Camera 122 and camera 123 each capture one of performers 141 and 142 at a narrow angle from an oblique direction. Also, as shown in Fig. 9, in virtual space 60, a virtual camera 611 and a virtual screen 621 corresponding to camera 121 are arranged, a virtual camera 612 and a virtual screen 622 corresponding to camera 122 are arranged, and a virtual camera 613 and a virtual screen 623 corresponding to camera 123 are arranged. Although only one background component 11 is used in studio 100, in virtual space 60, virtual screens 621 to 623 are arranged so as to directly face the virtual cameras 611 to 613, respectively.

[0059] FIG. 9 illustrates three sets of virtual cameras 611-613 and virtual screens 621-623. Each of the virtual cameras 611-613 captures only the image displayed on the virtual screen 621-623 associated with that virtual camera. For example, the virtual camera 612 captures only the extracted image 64 of the performer 142 displayed on the virtual screen 622, and does not capture the extracted images 64 of the performers 141 and 142 displayed on the virtual screens 621 and 623. This configuration can eliminate mutual interference between the multiple virtual cameras 611-613 and the virtual screens 621-623 (reflection of the extracted images 64 displayed on other virtual screens). As described above, the virtual screen 62 used in this embodiment is an object virtually placed in the virtual space 60. Therefore, by appropriately configuring the settings in advance, such switching can be easily performed. Here, we have explained an example in which performers 141 and 142 performing simultaneously are photographed by multiple cameras 121 to 123, and virtual cameras 611 to 613 and virtual screens 621 to 623 corresponding to each camera 121 to 123 are arranged, but this can be modified as appropriate, such as photographing performer 14 performing individually with multiple cameras 12.

[0060] As described above, by shooting performers 141 and 142 in parallel using multiple cameras 121 to 123 and using virtual cameras 611 to 613 and virtual screens 621 to 623 corresponding to each camera 121 to 123, it is possible to instantly switch the angle of view for shooting the performances of performers 141 and 142 (for example, an angle of view that captures both performers 141 and 142, an angle of view that shoots performer 141 in close-up, or an angle of view that shoots performer 142 in close-up). When creating video content, the director selects one of the multiple virtual cameras 611 to 613 using a predetermined input operation, and video content can be created in real time by combining images shot by the virtual cameras 611 to 613 selected at each point in time. Images taken by virtual cameras 611 to 613 selected by the director may also be displayed on screen 17. However, since screen 17 displays images for performer 14 to check the state of the interior of the virtual space, it is preferable to display images taken by a predetermined virtual camera that takes images of virtual space 60 at a wide angle (virtual camera 611 in the above example). It is also preferable to store images taken by non-selected virtual cameras 61 in storage unit 21. In this way, after video content is created in real time, some of the images that make up the video content can be changed to images taken by non-selected virtual cameras 61.

[0061] Here, the case where three cameras 121 to 123 are used has been described, but the numbers of cameras 12, virtual cameras 61, and virtual screens 62 can be changed as desired. Furthermore, if the background of the virtual space 60 is uniform (for example, the entire space is the sea and there are no player characters or the like), only one set of virtual camera 61 and virtual screen 62 may be placed in the virtual space 60, and extracted images 64 of the performer 14 acquired by the multiple cameras 121 to 123 may be displayed on the virtual screen 62 in a switching manner.

[0062] In the above embodiment, the positional relationship between the camera 12 and the background member 11 in real space is reflected as is in the virtual camera 61 and the virtual screen 62 in virtual space, but this may be changeable. Specifically, for example, when the director performs an input operation through the input unit 41 of the control and processing device 20, the virtual screen display processing unit 342 may be configured to tilt the virtual screen 62 forward or backward from a reference position (the position of the virtual screen 62 that directly reflects the positional relationship between the camera 12 and the background member 11 in real space) or to change the surface shape of the virtual screen 62 (for example, to curve it).

[0063] Figure 10 shows an example in which the top of virtual screen 62 is tilted forward and backward. The center of Figure 10 shows virtual screen 62 placed at the reference position, the top row shows an example in which the top edge of virtual screen 62 is tilted forward (towards where virtual camera 61 is located), and the bottom row shows an example in which the top edge of virtual screen 62 is tilted backward. As can be seen from the figure, by adopting the above configuration, it is possible to create video content in which extracted image 64, which is an image of actual performer 14 that has been arbitrarily modified, such as to make performer 14 appear taller than in reality, is incorporated into a virtual space.

[0064] In the above embodiment, the virtual space generation unit 341 generated the entire virtual space as a three-dimensional space. However, generating the entire infinitely expanding virtual space as a three-dimensional space would increase the processing load. Therefore, a space that is more than a predetermined distance away from the virtual camera 61 may be projected as a two-dimensional image onto a background screen placed at that distance. For example, as shown in FIG. 11 , a hemispherical background screen 65 may be placed around the position of the virtual camera 61, and an image of the space more than the predetermined distance away may be projected onto the hemispherical background screen 65. FIG. 11 illustrates a virtual space with a ground plane, so the hemispherical background screen 65 is used. However, if the virtual space is in the air or underwater, a spherical background screen 65 may be used. While FIG. 11 illustrates a hemispherical background screen, other shapes, such as a flat screen, may also be used. However, if the background screen has an edge, such as a flat screen, it is positioned so that the edge is outside the field of view of the virtual camera 61.

[0065] With the above configuration, it is not necessary to generate all three-dimensional objects to be placed in the virtual space, thereby reducing the processing load on the virtual space generation unit 341. When a viewer views an image captured by the virtual camera 61 or video content composed of the image, the viewer does not need to closely check objects located far from the virtual camera 61. Therefore, by appropriately determining the above-mentioned predetermined distance in consideration of the resolution of the image acquired by the virtual camera 61, the processing load on the virtual space generation unit 341 can be reduced without causing any discomfort in the image or video.

[0066] In the above embodiment, the virtual space generation unit 341 generates one virtual space, and video content is created in that virtual space by shooting using the virtual camera 61, but it is also possible to use a configuration using multiple virtual spaces. Specifically, video content is created using two virtual spaces that have the same virtual screen 62, player character 63, non-player characters, other objects, background configuration, etc., but different resolutions.

[0067] Specifically, a first virtual space is used in which avatars and objects with low resolution (e.g., fewer polygons) and no surface texture are placed, and a second virtual space is used in which avatars and objects with higher resolution (e.g., more polygons) than the first virtual space are placed and texture is displayed on the surface. Figure 12 shows an example of the first virtual space. In this configuration, a projection unit 36 ​​first projects an extracted image 64 of the performer 14 onto a virtual screen 62 in the first virtual space, and a virtual shooting unit 37 captures the virtual space. From the resulting data, parameter information is acquired, representing the positional information of the virtual camera 61 and virtual screen 62 in the virtual space, the position and posture of the player character 63, and the position and posture of the non-player characters. Once this information is acquired in the first virtual space, the virtual shooting unit 37 captures the player character 63 and other objects with polygons and textures of sufficient resolution required for the video content in a second virtual space, using the parameter information acquired during shooting in the first virtual space, and creates (renders) the video content. The timing of creating the video content in the second virtual space is arbitrary. That is, as soon as parameter information is acquired in the first virtual space, video content may be created in the second virtual space, or the video content may be acquired in the second virtual space after the parameter information necessary for creating video content is acquired in the first virtual space.

[0068] In the above embodiment, the virtual space generation unit 341 and the character display processing unit 35 only need to display the player characters and non-player characters at low resolution, which reduces the processing load during shooting. When projecting a captured image of the performer 14 onto the virtual screen 62 in real time, if the processing load for generating the virtual space or processing the display data for the player character 63 is large, the performer 14 or player must wait until these processes are complete, which places a heavy burden on the performer 14 or player. In contrast, in the above embodiment, shooting is performed in the first virtual space, where the load for generating the virtual space or processing the display data for the player character 63 is small, which reduces the burden on the performer 14 or player. [Industrial Applicability]

[0069] The imaging system according to the present invention can be used to create various types of video, such as movies, television programs, commercial films (CM), games, and virtual reality (VR) content.

[0070] [Aspect] It will be apparent to those skilled in the art that the above-described exemplary embodiments are examples of the following aspects.

[0071] (Section 1) An imaging system according to one aspect of the present invention includes: an image information storage unit that stores image data of an object captured by a camera in real space and position information that indicates the positional relationship between the object and the camera when the object is captured by the camera; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that reads out image data of the object from the photographing information storage unit and projects the image onto the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen; Equipped with.

[0072] In the imaging system according to paragraph 1, a shooting information storage unit stores image data of an object captured in real space by a real imaging unit and position information indicating the positional relationship between the object and the real imaging unit when the object was captured. The object captured here is, for example, an actor or other performer. The image of the object is obtained by capturing the object against a conventional green screen, for example. Display data that will serve as the background of the content is also prepared in advance. A virtual space generation unit generates a virtual space using the display data. A virtual screen display processing unit places a transparent virtual screen in the virtual space, and an image display processing unit displays an image of the object on the virtual screen. A transparent virtual screen is a virtual object (screen) that is transparent except for the area where the image is displayed. The virtual capturing unit then captures an image including the virtual screen, along with the virtual space visible through the virtual screen, from a predetermined position determined based on the position information. In this way, the imaging system according to the present invention acquires an image in which the image of the object captured in real space is incorporated into the virtual space. In the imaging system according to paragraph 1, it is possible to photograph an object in real space against, for example, a conventional green screen, and there is no need to use a large, expensive display with many LED light sources as in virtual production. Furthermore, the virtual space that serves as the background can be freely designed, and the size and position of the virtual screen placed there can also be easily changed. Therefore, the imaging system according to paragraph 1 can inexpensively capture images of an object captured at various angles of view.

[0073] (Section 2) The photography system according to paragraph 2 is the photography system according to paragraph 1, The photographing information storage unit further stores data of the image of the object in association with information on the position of the object in the real space, The virtual screen display processing unit further changes the position of the virtual screen based on the information on the position of the object.

[0074] In the filming system according to paragraph 2, the movement of the performer in the real space can be reflected in the virtual space.

[0075] (Section 3) The photography system according to paragraph 3 is a photography system according to paragraph 1 or 2, The virtual screen display processing unit changes the attitude or shape of the virtual screen in response to a predetermined input operation.

[0076] The photography system according to paragraph 3 makes it possible to create video content by making arbitrary changes to images of actual objects, such as making the objects appear larger (taller) than they actually are, and incorporating them into a virtual space.

[0077] (Section 4) The photography system according to paragraph 4 is a photography system according to any one of paragraphs 1 to 3, The photographing information storage unit further stores data of the image of the object in association with information on the position of the camera in the real space, The virtual photographing unit changes the predetermined position in accordance with information about the position of the camera.

[0078] (Section 5) The imaging system according to paragraph 5 is an imaging system according to any one of paragraphs 1 to 4, The photographing information storage unit further stores data of the image of the object in association with information on the angle of view of the camera in the real space, The virtual photographing unit changes the angle of view during photographing in accordance with information about the angle of view of the camera.

[0079] The photographing system according to paragraph 4 and the photographing system according to paragraph 5 can perform photographing that reflects the position and angle of view of the camera when the image of the object is photographed.

[0080] (Section 6) The photography system according to paragraph 6 is a photography system according to any one of paragraphs 1 to 5, The virtual screen is placed directly opposite the virtual shooting unit.

[0081] In the photography system according to paragraph 6, the virtual screen is positioned corresponding to the virtual photography unit, i.e., the optical axis of the virtual photography unit is positioned perpendicular to the virtual screen. The closer the optical axis of the virtual photography unit is to the virtual screen, the less depth is perceived in the image of the object. However, by adopting the positioning described in paragraph 6, it becomes easier to perceive the depth of the object projected onto the virtual screen.

[0082] (Section 7) The imaging system according to paragraph 7 is the imaging system according to any one of paragraphs 1 to 6, further comprising: a camera that captures an image of an object in real space; an image extraction unit that extracts a region of the object from image data of the object photographed by the camera; Equipped with The image display processing unit displays the extracted image of the object extracted by the image extraction unit on the virtual screen in real time.

[0083] The imaging system according to paragraph 7 can create video content in real time while imaging an object.

[0084] (Section 8) The imaging system according to paragraph 8 is the imaging system according to any one of paragraphs 1 to 7, further comprising: A first sound collection unit attached to the object; a first audio output unit that outputs the audio data collected by the first audio collection unit from a predetermined position on the virtual screen; Equipped with.

[0085] In the photography system according to paragraph 8, if the object is a human or other sound-emitting entity, the sound emitted from the object can be recognized by the player operating the player character in the virtual space.

[0086] (Section 9) The imaging system according to paragraph 9 is an imaging system according to any one of paragraphs 1 to 8, moreover, a second sound collection unit provided at a predetermined position on the virtual screen; a second sound output unit attached to the object and outputting sound data collected by the second sound collection unit; Equipped with.

[0087] (Section 10) The imaging system according to paragraph 10 is an imaging system according to any one of paragraphs 1 to 9, the subject is a human; moreover, a first motion detection unit that detects a predetermined motion by the object; a first effect generating unit that generates an effect associated with the predetermined action in the virtual space or on the virtual screen when the predetermined action is detected by the first action detecting unit; Equipped with.

[0088] (Section 11) The imaging system according to paragraph 11 is the imaging system according to any one of paragraphs 1 to 10, further comprising: a player terminal for controlling a player character operating in the virtual space; a second action detection unit that detects a predetermined action of the player character with respect to the image of the object displayed on the virtual screen, the action being performed through the player terminal; a second effect generating unit that, when the predetermined action is detected by the second action detecting unit, generates an effect associated with the predetermined action at a position on the virtual screen where the image of the object is displayed; Equipped with.

[0089] In the photography system according to paragraph 1, when the target is a human, communication can be achieved between the human and the player character in the virtual space via the virtual screen. Furthermore, the human target can also interact with the virtual space. That is, starting from the virtual screen, an effect pre-associated with a predetermined action (utterance, movement, etc.) by the human target can be generated in the virtual space. Furthermore, an effect pre-associated with a predetermined action (utterance, movement, etc.) by the player character in the virtual screen can be generated on the virtual screen.

[0090] Specifically, for example, the filming system according to paragraph 9 allows a player to perform while listening to sounds generated in the virtual space. The filming system according to paragraph 10 further enhances interactivity with the player who controls the player character in the virtual space. Furthermore, the filming system according to paragraph 11 further enhances interactivity between the player character operating in the virtual space and the object.

[0091] (Section 12) The imaging system according to paragraph 12 is the imaging system according to any one of paragraphs 1 to 11, The system includes a plurality of pairs each consisting of the virtual screen and the virtual shooting unit associated with the virtual screen.

[0092] In the photography system according to paragraph 12, data of images of an object photographed in real space by multiple cameras (images of the same object photographed from different directions and / or angles of view, or images of different objects) is projected onto different virtual screens, and images of the virtual space including these images are photographed by a virtual photography unit associated with the virtual screens, thereby making it possible to obtain a wide variety of video content.

[0093] (Section 13) The imaging system according to paragraph 13 is the imaging system according to any one of paragraphs 1 to 12, further comprising: a background screen provided in the virtual space at a predetermined distance from the virtual shooting unit; Equipped with The virtual space generation unit displays a space in the virtual space that is spaced at least the predetermined distance from the virtual shooting unit as a two-dimensional image on the background screen.

[0094] In the photography system according to paragraph 13, it is not necessary to generate all three-dimensional objects to be placed in the virtual space, so the processing load on the virtual space generation unit can be reduced.

[0095] (Section 14) The imaging system according to paragraph 14 is an imaging system according to any one of paragraphs 1 to 13, the virtual space generation unit generates a first virtual space and a second virtual space having a higher display resolution than the first virtual space; the virtual screen and the image display processing unit are provided in the first virtual space and the second virtual space, respectively; The virtual photographing unit photographs the first virtual space to obtain parameter information of objects present in the first virtual space, and then reflects the parameter information in the second virtual space and photographs the second virtual space.

[0096] In the photography system according to paragraph 14, photography in the first virtual space only requires displaying objects at low resolution, thereby reducing the processing load on the virtual space generation unit and efficiently acquiring parameter information and other information necessary for content creation. Furthermore, by creating content by reflecting parameter information in the second virtual space, video content can be efficiently created in a high-resolution virtual space. [Explanation of symbols]

[0097] 1. Shooting system 100...Studio 10...Photographing equipment 11...Background material 111...Lattice point 12, 121~123...Camera 13...Position sensor 131~133…Reference point 134...Main body 14, 141, 142...Performers 15…Sound collection section 16...Earphones 17...Screen 20...Control and processing device 21...Storage section 22... Shooting information storage section 23...Virtual space display data storage unit 24...Character data storage unit 25...Shooting data storage unit 30...Content Creation Program 31...Image extraction unit 32...Camera angle of view acquisition unit 33...Position information acquisition unit 341...Virtual space generation unit 342...Virtual screen display processing unit 35...Character display processing unit 36…Projection section 37...Virtual Photography Department 38...Audio processing unit 41...Input section 42...Display section 50...Player terminal 51...Storage section 52...input operation section 53...Audio input section 54...Audio output section 55...Display section 60...Virtual space 61, 611-613...Virtual Camera 62, 621-623...Virtual screen 63...Player character 64…Extracted image of the performer projected onto the virtual screen 71...Reference object 9. Network

Claims

1. A computer connected to a camera that photographs an object in real space, an image extraction unit that extracts a region of the object from image data of the object photographed by the camera; a position information acquisition unit that acquires position information representing a positional relationship between the camera and the object; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that displays the image of the object extracted by the image extraction unit on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen; A program that operates as In response to a predetermined action by a player character or a non-player character in the virtual space, an effect associated with the predetermined action is generated on the virtual screen. A content creation program comprising:

2. A computer connected to a camera that photographs an object in real space and a first sound collection unit attached to the object, an image extraction unit that extracts a region of the object from image data of the object photographed by the camera; a position information acquisition unit that acquires position information representing a positional relationship between the camera and the object; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that displays the image of the object extracted by the image extraction unit on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen; a first sound output unit that outputs the sound data collected by the first sound collection unit from a predetermined position on the virtual screen; A content creation program characterized by causing the program to operate as follows.

3. A computer connected to a camera that photographs an object in real space and a second audio output unit attached to the object, an image extraction unit that extracts a region of the object from image data of the object photographed by the camera; a position information acquisition unit that acquires position information representing a positional relationship between the camera and the object; a virtual space generation unit that generates a virtual space using pre-prepared display data that simulates a space in which the content is to be shot; a virtual screen display processing unit that places a transparent virtual screen in the virtual space; an image display processing unit that displays the image of the object extracted by the image extraction unit on the virtual screen; a virtual photographing unit that photographs, in the virtual space, an image including the virtual screen from a predetermined position determined based on the position information, together with the virtual space visible through the virtual screen; a second sound collecting section provided at a predetermined position on the virtual screen; A program that operates as The sound data collected by the second sound collection unit is output from the second sound output unit. A content creation program comprising:

4. the subject is a human; Furthermore, the computer a first motion detection unit that detects a predetermined motion of the object; a first effect generation unit that generates an effect associated with the predetermined action in the virtual space or on the virtual screen when the predetermined action is detected by the first action detection unit; 4. The content creation program according to claim 1, wherein the program is operated as follows:

5. Furthermore, the computer a second action detection unit that detects a predetermined action of the player character with respect to the image of the object displayed on the virtual screen; a second effect generation unit that, when the second action detection unit detects the predetermined action, generates an effect associated with the predetermined action at a position on the virtual screen where the image of the object is displayed.

2. The content creation program according to claim 1, wherein the program is operated as follows:

6. The position information acquisition unit further acquires information about the position of the object in the real space, The virtual screen display processing unit further changes the position of the virtual screen based on the information on the position of the object.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

7. The virtual screen display processing unit changes the attitude or shape of the virtual screen in response to a predetermined input operation.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

8. The position information acquisition unit further acquires information about the position of the camera in the real space, The virtual photographing unit changes the predetermined position in accordance with information about the position of the camera.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

9. Further, the computer: a camera angle of view acquisition unit that acquires information about the angle of view of the camera in the real space; and operate as The virtual photographing unit changes the angle of view at the time of photographing in accordance with information about the angle of view of the camera.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

10. The virtual screen is placed directly opposite the virtual shooting unit.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

11. The computer is operated so as to have a plurality of pairs each consisting of the virtual screen and the virtual shooting unit associated with the virtual screen.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

12. Operate the computer so that a background screen is provided in the virtual space at a position spaced a predetermined distance from the virtual shooting unit, The virtual space generation unit displays a space in the virtual space that is separated from the virtual shooting unit by the predetermined distance or more on the background screen as a two-dimensional image.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

13. the virtual space generation unit generates a first virtual space and a second virtual space having a higher display resolution than the first virtual space; the virtual screen and the image display processing unit are provided in the first virtual space and the second virtual space, respectively; The virtual photographing unit photographs the first virtual space to obtain parameter information of an object present in the first virtual space, and then reflects the parameter information in the second virtual space and photographs the second virtual space.

4. The content creation program according to claim 1, wherein the content creation program is a program for creating a content.

Images