Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that enable suitable rendering of video including a person. This information processing device includes: a placement control unit that selects, from among a plurality of candidates, a placement position of a person in a virtual space on the basis of the position of a subject in camera video acquired by a first camera that acquires images of the subject against a background of a display that displays display video including the person; and a rendering unit that renders video of the virtual space as seen from a virtual viewpoint corresponding to a viewpoint of the first camera so as to generate display video. The present technology can be applied to, for example, an imaging system used in virtual production.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and particularly to an information processing device, an information processing method, and a program that are capable of suitably rendering video including people.

[0002] Recently, virtual production (in-camera VFX), a filming technique using large displays, has become popular in the filming of movies and dramas.

[0003] In virtual production, a subject placed in front of a display is shot in the foreground against an image displayed on the display. For example, the image displayed on the display would be the background image of a subject shot from a virtual viewpoint corresponding to the viewpoint of a camera in a virtual space. This image is rendered based on the camera's viewpoint, angle of view, etc.

[0004] When filming movies or TV dramas, the lead actor is sometimes filmed against a background of passersby or crowds, often played by multiple actors known as extras. Filming using virtual production also involves filming the lead actor against a background of footage displayed on a screen, including extras.

[0005] For example, Patent Document 1 describes a technique for synthesizing a live-action video of a person and a stage video created using CG in association with each other.

[0006] Japanese Patent Application Laid-Open No. 2020-72415

[0007] When there are a large number of extras, rendering footage that includes the extras places a heavy load on the rendering device, which can make it difficult to render the footage in real time based on the camera's viewpoint, angle of view, etc.

[0008] The present technology has been made in consideration of such circumstances, and makes it possible to suitably render video including people.

[0009] An information processing device according to one aspect of the present technology includes a placement control unit that selects a placement position of a person in a virtual space from among a plurality of candidates based on the position of the subject in a camera image acquired by a first camera that photographs the subject against a background of a display that displays a display image including the person, and a rendering unit that renders an image of the virtual space as seen from a virtual viewpoint corresponding to the viewpoint of the first camera to generate the display image.

[0010] An information processing method according to one aspect of the present technology selects a position for placing the person in a virtual space from among multiple candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a background of a display that displays a display image including the person, and generates the display image by rendering an image of the virtual space as seen from a virtual viewpoint corresponding to the viewpoint of the camera.

[0011] A program according to one aspect of the present technology causes a computer to select a position for placing the person in a virtual space from among multiple candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a display that displays a display image including the person in the background, and to perform a process of rendering an image of the virtual space as seen from a virtual viewpoint corresponding to the viewpoint of the camera to generate the display image.

[0012] In one aspect of the present technology, a position for placing the person in a virtual space is selected from among multiple candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a background of a display that displays a display image including the person, and an image of the virtual space as seen from a virtual viewpoint corresponding to the viewpoint of the camera is rendered to generate the display image.

[0013] 9 is a diagram illustrating an overview of a filming system to which the present technology is applied. FIG. 1 is a diagram illustrating an example of camera footage captured by a main camera. FIG. 2 is a diagram illustrating an example of a state during filming. FIG. 3 is a diagram illustrating another example of camera footage captured by the main camera. FIG. 4 is a diagram illustrating an example of footage showing a digital actor. FIG. 5 is a diagram illustrating an example of a conventional display of a digital actor. FIG. 6 is a diagram illustrating an example of a display of a digital actor according to the present technology. FIG. 7 is a block diagram illustrating an example of the functional configuration of an information processing device. FIG. 8 is a flowchart illustrating processing performed by an information processing device having the configuration shown in FIG. 9. FIG. 10 is a diagram illustrating an example of sub-camera placement. FIG. 11 is a diagram illustrating an example of main camera footage. FIG. 12 is a diagram illustrating a studio appearance when no sub-cameras are installed. FIG. 13 is a flowchart illustrating person detection processing performed in step S2 of FIG. 9. FIG. 14 is a first diagram illustrating an example of adjustment of the number of candidates for placement positions of a digital actor in accordance with the proportion of the area occupied by an actor in the main camera footage. FIG. 15 is a second diagram illustrating an example of adjustment of the number of candidates for placement positions of a digital actor in accordance with the proportion of the area occupied by an actor in the main camera footage. FIG. 16 is a diagram illustrating an example of a candidate placement position map for a digital actor. FIG. 17 is a diagram illustrating another example of a candidate placement position map for a digital actor. FIG. 18 is a flowchart illustrating digital actor placement processing performed in step S3 of FIG. FIG. 1 is a diagram showing an example of candidate placement positions for a digital actor that are set in accordance with objects that are placed in a virtual space. FIG. 2 is a first diagram illustrating a method for selecting a placement position for a digital actor from a plurality of candidates. FIG. 3 is a second diagram illustrating a method for selecting a placement position for a digital actor from a plurality of candidates. FIG. 4 is a diagram illustrating an example of a method for selecting a digital actor. FIG. 5 is a diagram showing an example of placement of digital actors in accordance with the type of digital actor data. FIG. 6 is a diagram showing examples of high polygon and low polygon 3D models. FIG. 7 is a first diagram showing an example of placement of two-dimensional live-action video. FIG. 8 is a second diagram showing an example of placement of two-dimensional live-action video. FIG. 9 is a block diagram showing an example of the configuration of computer hardware.

[0014] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the imaging system 2. Configuration and operation of the information processing device

[0015] 1. Overview of Imaging System FIG. 1 is a diagram illustrating an overview of an imaging system 1 to which the present technology is applied.

[0016] 1 is a system used for shooting, for example, virtual production (in-camera VFX). The shooting system 1 is composed of a main camera 11, a video storage device (not shown) that stores camera images captured by the main camera 11, a wall-type display 12, and an information processing device 13 (not shown) that controls the display 12.

[0017] The display 12 is configured, for example, by an LED (Light Emitting Diode) display and is placed in a real space such as a studio. The display 12 displays, for example, an image of a virtual space created by CG (Computer Graphics) as a display image. The photographer P uses the main camera 11 to photograph the subject, the motorcycle M1, with the display image displayed on the display 12 as the background.

[0018] FIG. 2 is a diagram showing an example of a camera image captured by the main camera 11. As shown in FIG.

[0019] 2, the camera image captured by the main camera 11 makes the motorcycle M1 appear as if it were actually present in the virtual space shown in the displayed image. In this way, by using the photography system 1, the photographer P can capture camera images in the studio in which the space shown in the displayed image appears to be expanding in the background of the motorcycle M1.

[0020] FIG. 3 is a diagram showing an example of a state during shooting.

[0021] As shown in FIG. 3, during shooting, for example, the entire display image is displayed on the entire display 12.

[0022] 3, when the main camera 11 captures an image from the right side of the centrally located motorcycle M1, a portion of the displayed image is displayed superimposed on the entire displayed image in the imaging area A1, which is an area on the display 12 included in the imaging range of the main camera 11. In the imaging area A1, for example, an image of a portion that would be the background when the image is captured from the right side of the motorcycle M1 in the virtual space shown by the displayed image is cut out from the entire displayed image and displayed.

[0023] 3, when the main camera 11 captures an image from the left side of the motorcycle M1 located in the center, a portion of the displayed image is displayed in the imaging area A1 so as to be superimposed on the entire displayed image. For example, in the imaging area A1, an image of a portion that would be the background when the image is captured from the left side of the motorcycle M1 in the virtual space shown by the displayed image is cut out from the entire displayed image and displayed.

[0024] In this way, when taking a photograph, the photographer P takes a photograph while successively changing the viewpoint, angle of view, focus position, aperture value, etc. of the main camera 11.

[0025] The information processing device 13 tracks the camerawork (viewpoint and angle of view) of the main camera 11 and controls the position on the display 12 where the displayed image is displayed according to the camerawork of the main camera 11. Specifically, the information processing device 13 detects the shooting area based on the position, attitude, settings, etc. of the main camera 11 and controls the displayed image to be displayed in the shooting area. Hereinafter, the shooting area within the display surface of the display 12 is also referred to as the inner frustum (area within the angle of view).

[0026] In FIG. 3, for ease of explanation, the imaging area A1 is shown surrounded by a thick line, but in reality, the line surrounding the imaging area is not displayed.

[0027] When filming a movie or drama, a lead actor may be filmed against a background of passersby, crowds, etc., in which multiple actors called extras play roles. In the filming system 1 of the present technology, a video including the extras is displayed as a display video on the display 12, and the lead actor as a subject is filmed against the background of the display video including the extras.

[0028] FIG. 4 is a diagram showing another example of a camera image captured by the main camera 11. In FIG.

[0029] The camera footage obtained by filming the lead actor Ac1 with a display image including extras as the background will appear as if the lead actor Ac1 is walking down a street with many passersby, as shown in Figure 4. In this way, by filming using the filming system 1, the cameraman P can film a camera footage in which many extras appear in the background in a studio where only the lead actor Ac1 is actually present.

[0030] Hereinafter, the extras displayed on the display 12 will also be referred to as digital actors. Note that the display image including the digital actors is displayed, for example, in an inner frustum.

[0031] FIG. 5 is a diagram illustrating an example of a video showing a digital actor.

[0032] As a representation of the digital actor, for example, two-dimensional live-action footage, volumetric footage, and digital human footage are displayed, as shown in FIG.

[0033] A two-dimensional live-action image is an image obtained by actually filming actors (extras) against a background such as a green screen, and then removing the background.

[0034] Volumetric video is free-viewpoint video that allows for free movement of the viewpoint (position and posture), and is obtained, for example, by using volumetric capture technology to actually film actors (extras) against a green screen or other background. Volumetric capture technology generates a 3D model of the subject from video shot from multiple viewpoints, and then generates video of the 3D model as seen from any viewpoint, thereby providing video from any viewpoint.

[0035] Digital human video is a computer-generated video of a digital human, and is a free viewpoint video that allows the viewpoint to be moved freely. Digital humans are characters that reproduce the appearance, movements, and emotional expressions of a human.

[0036] When two-dimensional live-action video is displayed to represent digital actors, the two-dimensional live-action video is rendered on a plane placed within a virtual space. Because the actors (digital actors) appearing in the two-dimensional live-action video are rendered on a two-dimensional plane, the perspective of the digital actors may appear unnatural depending on the viewpoint of the displayed video (the viewpoint of the main camera 11). Furthermore, when two-dimensional live-action video showing multiple actors is displayed, the distance between the digital actors and the placement positions of each digital actor cannot be adjusted.

[0037] Therefore, when 2D live-action footage is displayed as a representation of a digital actor, there is limited freedom in camerawork during virtual production, and there is also limited freedom in positioning the digital actor in the virtual space. Because 2D live-action footage is footage of an actor that was actually filmed, the digital actor looks natural. Because the viewpoint cannot be moved with 2D live-action footage, the data size and rendering load are small.

[0038] Volumetric video allows for a shift in viewpoint. Therefore, when volumetric video is used to represent a digital actor, the degree of freedom in camerawork during virtual production and the degree of freedom in positioning the digital actor in the virtual space are greater than when 2D live-action video is used. The digital actor appears more unnatural than when 2D live-action video is used. The data size and rendering load are greater than when 2D live-action video is used.

[0039] Lighting can be freely changed with digital human footage. Therefore, when digital human footage is displayed as a representation of a digital actor, the degree of freedom in camerawork during virtual production and the degree of freedom in placing the digital actor in virtual space are greater than when volumetric footage is displayed. Digital humans often lack lifelikeness and realism, and their movements are often unnatural, making the digital actor appear more unnatural than when volumetric footage is displayed. The data size and rendering load are greater than when volumetric footage is displayed.

[0040] Fig. 6 shows an example of a conventional digital actor display. Fig. 6 illustrates an example in which a lead actor Ac11 and supporting actors Ac12 to Ac14 are filmed as subjects in a studio while a display image including digital actors DAc1 to DAc3 is displayed on a display 12. The same applies to Fig. 7.

[0041] Conventionally, digital actors are positioned at predetermined positions in a virtual space. Therefore, as shown in Figure 6, from the viewpoint of main camera 11, supporting actors Ac12 to Ac14 may appear to overlap digital actors DAc1 to DAc3, respectively. In other words, digital actors DAc1 to DAc3 may not appear very clearly in the camera image of main camera 11.

[0042] Furthermore, when there are a large number of digital actors, rendering the display image places a heavy load on the information processing device 13 that performs the rendering, and it may become difficult to render the display image in real time based on the viewpoint, angle of view, etc. of the main camera 11.

[0043] One embodiment of the present technology has been conceived with a focus on the above points, and proposes a technology that can place the minimum number of digital actors in a virtual space and reduce the load on the information processing device 13. In the information processing device 13 of the present technology, a placement position of the digital actor in the virtual space is selected from among multiple candidates based on the position of a subject in a camera image acquired by a main camera 11 that captures the subject against a background of a display that displays a display image including the digital actor (person), and a display image is generated by rendering an image of the virtual space as seen from a virtual viewpoint corresponding to the viewpoint of the main camera 11.

[0044] FIG. 7 is a diagram showing an example of a display of a digital actor according to the present technology.

[0045] 7, the information processing device 13 of the present technology automatically controls the positions of the digital actors DAc1 to DAc3 in the virtual space so that they do not overlap with the lead actor Ac11 and supporting actors Ac12 to Ac14 when viewed from the viewpoint of the main camera 11. Therefore, it is possible to effectively display the digital actors.

[0046] When digital human images are displayed as digital actors, extras do not need to be in the studio, and there is no need to film extras in advance, which makes it possible to reduce performance fees and production costs.

[0047] 2. Configuration and Operation of Information Processing Device FIG. 8 is a block diagram showing an example of the functional configuration of the information processing device 13. As shown in FIG.

[0048] As shown in FIG. 8, the information processing device 13 includes a camera image acquisition unit 31, a person detection unit 32, a placement position candidate setting unit 33, a placement control unit 34, and a rendering unit 35.

[0049] The camera image acquisition unit 31 acquires camera images from cameras arranged in the studio and supplies them to the person detection unit 32. In the studio, a sub-camera (second camera) that is a camera different from the main camera 11 is arranged in addition to the main camera 11 (first camera). The sub-camera may be attached to the main camera 11 or installed in a position that overlooks the studio, for example.

[0050] The person detection unit 32 detects people (actors) appearing in the camera footage supplied from the camera footage acquisition unit 31, and supplies the person detection results to the placement position candidate setting unit 33 and the placement control unit 34. Specifically, the person detection unit 32 detects the position of people in the camera footage. Note that the person detection unit 32 can also analyze the camera footage to acquire subject information indicating the characteristics of people appearing in the camera footage, such as height and gender.

[0051] The placement position candidate setting unit 33 acquires shooting information, which is information related to shooting by the main camera 11. The shooting information includes, for example, information indicating the internal state of the main camera 11, such as the angle of view, focus position, and aperture value, and information indicating the viewpoint of the main camera 11. The placement position candidate setting unit 33 places the virtual main camera 11 in the virtual space based on the shooting information, and calculates the shooting range of the virtual main camera 11 in the virtual space.

[0052] The placement position candidate setting unit 33 sets candidate placement positions for the digital actor within the shooting range of the virtual main camera 11 in the virtual space. The placement position candidate setting unit 33 generates a placement position candidate map, which is information indicating candidate placement positions for the digital actor in the virtual space, and supplies this to the placement control unit 34 together with the shooting information.

[0053] Based on the subject information acquired by the person detection unit 32, the placement control unit 34 determines placement parameters related to the method for selecting the placement position of the digital actor and data selection parameters related to the method for selecting the digital actor to be placed in the virtual space.

[0054] The placement parameters are information that specifies the density of the crowd made up of digital actors, the randomness of the crowd placement, the number of people in the crowd, the distance between the actors in the studio and the digital actors in the virtual space, the spacing between the actors in the studio and the digital actors in the camera footage, etc. The data selection parameters are information that specifies the direction the digital actors are facing, the gender of the digital actors, the age of the digital actors, the color of the digital actors' clothing, etc. The placement parameters and data selection parameters may also be input by a user of the filming system 1, such as a cinematographer or director.

[0055] The placement control unit 34 selects a placement position for the digital actor from among the multiple candidates set by the placement position candidate setting unit 33, based on the person detection results by the person detection unit 32 and the shooting information supplied by the placement position candidate setting unit 33. For example, the placement control unit 34 selects a placement position such that the digital actor does not overlap with people in the studio when viewed from the virtual main camera 11.

[0056] Furthermore, the placement control unit 34 selects a placement position that is in line with the placement parameters from among the multiple candidates set by the placement position candidate setting unit 33. For example, when shooting a scene with a large crowd of people, a placement position close to the actors in the studio is selected, and when shooting a scene with actors in the street, a placement position a little distance away from the actors in the studio is selected.

[0057] The placement control unit 34 acquires digital actor data according to the data selection parameters. The digital actor data is data used to display digital actors (data representing digital actors) and includes 2D live-action footage, 3D actor models generated using volumetric capture technology, and 3D digital human models. The digital actor data is associated with metadata indicating the type of digital actor data, the number of meshes in the 3D model, and the digital actor's height, gender, age, etc.

[0058] The placement control unit 34 can acquire digital actor data that conforms to the data selection parameters by referencing the metadata based on the data selection parameters.

[0059] The 2D live-action footage and 3D models of actors generated using volumetric capture technology may be content obtained through prior filming, or may be real-time content obtained through filming in parallel with virtual production filming.

[0060] The placement control unit 34 places the digital actor in the virtual space. Specifically, the placement control unit 34 places the 2D live-action video, the 3D model of the actor, and the 3D model of the digital human in the virtual space.

[0061] The rendering unit 35 places the display surface of the virtual display 12 in the virtual space and renders the display image by projecting the virtual space as seen from the virtual main camera 11 (virtual viewpoint) onto the display surface of the virtual display 12. Here, rendering of volumetric images and digital human images is also performed in accordance with the lighting in the virtual space. Note that it is not necessary to adjust the lighting of the 2D live-action images.

[0062] The rendering unit 35 supplies the display image to the display 12, causing the display image to be displayed.

[0063] Next, the processing performed by the information processing device 13 having the above configuration will be described with reference to the flowchart of FIG.

[0064] In step S1, the camera image acquisition unit 31 acquires camera images from cameras installed in the studio.

[0065] In step S2, the information processing device 13 performs a person detection process. In the person detection process, people appearing in the camera image captured by the main camera 11 are detected, and a map of candidate positions for placing digital actors is generated. Details of the person detection process will be described later with reference to FIG. 13.

[0066] In step S2, detection of a person appearing in the camera image captured by the main camera 11 is performed based on, for example, the camera image captured by the sub-camera.

[0067] Fig. 10 is a diagram schematically illustrating an example of the arrangement of sub-cameras. In the example of Fig. 10, a movie or drama is shot by main camera 11 with display 12 as the background and lead actor Ac21 as the subject. Hereinafter, the camera image captured by main camera 11 will also be referred to as "main camera image," and the camera image captured by the sub-camera will also be referred to as "sub camera image."

[0068] As shown in A of Fig. 10, the sub-camera 51 captures an overhead view of the entire studio. As shown in B of Fig. 10, the sub-camera image P1 captured by the sub-camera 51 shows the entire body of the lead actor Ac21 as seen from diagonally above and to the left.

[0069] The sub-camera 51 is calibrated so that the information processing device 13 can always grasp its positional relationship with the main camera 11. Therefore, the information processing device 13 can identify the position of the lead actor Ac21 in the main camera image by projecting the position of the lead actor Ac21 detected in the sub-camera image P1 onto the main camera image.

[0070] Generally, because the main camera image is considered to have high image quality, it takes time to detect a person appearing in the main camera image based on the main camera image. By identifying the position of a person in the main camera image based on the sub-camera image, which has lower image quality than the main camera image, the position of the person can be detected faster than when the position of the person is detected based on the main camera image, and the digital actor can be displayed without delay.

[0071] The position of a person in the camera image is indicated by a bounding box, skeletal posture information, facial feature points, etc. The position of a person in the main camera image may be indicated by feature points that are commonly captured in the main camera image and the sub-camera image. A person captured in the camera image may be detected using a learning model acquired by machine learning or the like.

[0072] It is also possible to install multiple sub-cameras 51 in the studio, and detect the three-dimensional positions of people in the studio based on the sub-camera images captured by these sub-cameras 51.

[0073] The sub-camera 51 may be configured as a non-visible light camera such as a ToF (Time Of Flight) camera, an IR (Infrared) camera, a SWIR (Short Wavelength Infrared) camera, a thermal camera, etc. A plurality of different types of sub-cameras 51, such as a visible light camera, a non-visible light camera, or a LiDAR (Light Detection And Ranging) camera, may be arranged in the studio.

[0074] 9, in step S3, the information processing device 13 performs a digital actor placement process. The digital actor placement process determines the placement position of the digital actor, and the digital actor is placed in the virtual space. Details of the digital actor placement process will be described later with reference to FIG. 18.

[0075] The person detection process in step S2 and the digital actor placement process in step S3 may be performed, for example, for each frame of the main camera image, sub-camera image, display image, etc., or may be performed only when starting to shoot a scene that makes up a movie or drama.

[0076] In step S4, the rendering unit 35 renders the display image and displays the display image on the display 12. The main camera 11 takes an image of the actors in the studio as subjects, with the display image as the background.

[0077] FIG. 11 is a diagram showing an example of a main camera image.

[0078] In the example of A in Fig. 11, the lead actor Ac22 is shown on the left side of the main camera image, and the two digital actors DAc11 and DAc12 are shown on the right side. By the above-described processing, it is possible to position the digital actors DAc11 and DAc12 in positions that do not overlap with the lead actor Ac22 when viewed from the viewpoint of the main camera 11, as shown in A in Fig. 11.

[0079] If the size of the digital actors is not adjusted, there is a possibility that digital actor DAc13 will appear to be significantly larger than lead actor Ac22 when viewed from the viewpoint of main camera 11, as shown in B of Fig. 11. In the information processing device 13 of the present technology, not only the position of a person in the main camera video but also subject information such as the person's height is acquired, and therefore, as shown in A of Fig. 11, the sizes of digital actors DAc11 and DAc12 can be adjusted to match the height of lead actor Ac22.

[0080] Although the above description has been given of an example in which the position of a person in the main camera image is detected based on the sub camera image, the position of a person in the main camera image may also be detected based on the main camera image.

[0081] FIG. 12 is a diagram showing a typical state of the studio when the sub-camera 51 is not installed.

[0082] As shown in A of Figure 12, when a movie or drama is shot using the main camera 11 with the display 12 as the background and the lead actor Ac21 as the subject, the main camera image P2 shot by the main camera 11 as shown in B of Figure 12 shows, for example, the upper body of the lead actor Ac21 viewed from the front, at a position slightly to the left of the center.

[0083] If sub-camera 51 is not installed, information processing device 13 detects the position where a person appears in main camera image P2 based on main camera image P2. Since main camera image P2 has empty spaces to the left and right of lead actor Ac21, digital actors are placed to fill those spaces, for example.

[0084] Next, the person detection process performed in step S2 of FIG. 9 will be described with reference to the flowchart of FIG.

[0085] In step S21, the person detection unit 32 detects actors (people) appearing in the camera image.

[0086] In step S22, the person detection unit 32 determines whether or not there is only one actor in the camera image.

[0087] If it is determined in step S22 that there is not just one actor in the camera image, then in step S23, the person detection unit 32 calculates the distance between the actors in the camera image. Information indicating the distance between the actors is used as subject information. Then, the process proceeds to step S24. On the other hand, if it is determined in step S22 that there is just one actor in the camera image, then step S23 is skipped and the process proceeds to step S24.

[0088] In step S24, the person detection unit 32 analyzes the camera image to obtain subject information indicating the actor's height, gender, clothing color, etc. Note that the subject information may include a motion vector, which is information indicating the magnitude and direction of the actor's movement.

[0089] When the actor's motion vectors are analyzed, a digital actor is displayed that moves in accordance with the actor's movements, for example, a digital actor moving in the same direction as the actor is moving, or a digital actor moving in the opposite direction to the actor's movement.

[0090] The metadata of the digital actor data includes information indicating the direction in which the digital actor will move. By referencing the metadata, the information processing device 13 can select and place a digital actor from multiple candidates that corresponds to the movement of the actor in the studio.

[0091] If extras are filmed in parallel with virtual production filming, and two-dimensional live-action footage is obtained, for example, the direction in which the extras will move can be obtained appropriately by analyzing this two-dimensional live-action footage. If extras are filmed in parallel with virtual production filming, the motion vectors of actors in the studio where the virtual production filming is taking place can be presented to the extras. By moving in the same direction and amount as the actors, the extras can be displayed as digital actors moving in accordance with the actors' movements.

[0092] The movement of the digital actor may be controlled in accordance with the movement of the actor (subject). For example, if the actor is walking left as viewed from the viewpoint of the main camera 11, the movement of the digital actor may be controlled to walk right.

[0093] In step S25, the placement position candidate setting unit 33 sets candidates for placement positions of the digital actor in the virtual space, and generates a map of candidate placement positions for the digital actor.

[0094] Specifically, the placement position candidate setting unit 33 places the virtual main camera 11 in the virtual space based on the shooting information, and calculates the shooting range within the virtual space of the virtual main camera 11. The placement position candidate setting unit 33 sets candidates for the placement position of the digital actor within the shooting range of the virtual main camera 11. The placement position candidate setting unit 33 also adjusts the number of candidates for the placement position of the digital actor depending on the proportion of the area in the main camera image that is occupied by the actor.

[0095] 14 and 15 are diagrams illustrating an example of adjusting the number of candidates for placement positions of a digital actor in accordance with the proportion of the area in the main camera image that is occupied by the actor.

[0096] First, let us assume that a main camera image P11 showing the upper body of lead actor Ac21 is captured, as shown in A of Fig. 14. In this case, two candidate placement positions Op1 and Op2 for the digital actor are set in the virtual space, as shown in B of Fig. 14. The candidate placement positions Op1 and Op2 for the digital actor are set on the far side of the display 12 when viewed from the viewpoint of the virtual main camera 11.

[0097] Next, assume that main camera image P12 is captured, showing the entire body of lead actor Ac21, as shown in A of Fig. 15. In this case, the proportion of the area in which lead actor Ac21 is shown in main camera image P12 is smaller than the proportion of the area in which lead actor Ac21 is shown in main camera image P11. Therefore, in the virtual space, four candidate placement positions Op11 to Op14 for the digital actor are set, as shown in B of Fig. 15. The candidate placement positions Op11 to Op14 for the digital actor are set on the far side of display 12 when viewed from the viewpoint of virtual main camera 11.

[0098] Thus, the smaller the proportion of the area in the main camera image that is occupied by actors, in other words, the larger the shooting area within the display surface of display 12, the more digital actors need to be displayed. When the camera's viewpoint or angle of view changes, placement position candidate setting unit 33 can increase or decrease the number of candidates for placement positions of digital actors in real time.

[0099] Alternatively, the user may input candidate placement positions for the digital actors in the virtual space and the areas in which the digital actors are to be placed. In such cases, the placement position candidate setting unit 33 generates a map of candidate placement positions for the digital actors based on the information input by the user.

[0100] A map of possible placement positions for the digital actor may also be presented to the user. The map of possible placement positions for the digital actor presented to the user may be an overhead image of the virtual space, or an image of the virtual space as seen from the viewpoint of the virtual main camera 11.

[0101] FIG. 16 is a diagram showing an example of a map of possible placement positions for digital actors.

[0102] 16, the placement position candidate map shows a bird's-eye view of the virtual space. This placement position candidate map shows the virtual main camera 11, the avatar of lead actor Ac21, and virtual objects. The placement position candidate map also shows three placement position candidates for the digital actor using bounding boxes B1 to B3.

[0103] By viewing the candidate placement position map, the user can confirm that if the digital actor is placed at the position indicated by the bounding boxes B1 to B3, the background will be filled in to some extent with extras and objects, and a well-balanced main camera image will be captured in which the extras are not concentrated in one place.

[0104] FIG. 17 is a diagram showing another example of a map of possible placement positions for digital actors.

[0105] 17, the placement position candidate map shows an image of the virtual space as seen from the viewpoint of the virtual main camera 11. This placement position candidate map shows the avatar of lead actor Ac21 and virtual objects. The placement position candidate map also shows three candidate placement positions for the digital actor as bounding boxes B1 to B3.

[0106] It is also possible for the user to adjust the candidate placement positions for the digital actor while viewing the candidate placement position map described above.

[0107] After the above-described arrangement position candidate map is generated in step S25 of FIG. 13, the process returns to step S2 of FIG. 9, and the subsequent processes are carried out.

[0108] Next, the digital actor placement process performed in step S3 of FIG. 9 will be described with reference to the flowchart of FIG.

[0109] In step S41, the placement position candidate setting unit 33 updates (sets) candidates for the placement position of the digital actor based on at least one of the positions and shapes of objects other than the digital actor that are placed in the virtual space.

[0110] FIG. 19 shows examples of candidates for placement positions of a digital actor set in accordance with objects placed in a virtual space.

[0111] In some cases, objects such as vehicles, buildings, mountains, slopes, and rivers are placed within the virtual space. In the example of A in Fig. 19, a vehicle object Obj1 is placed within the virtual space. In this case, in the example of A in Fig. 19, object Obj1 is placed to the right of lead actor Ac21 when viewed from the viewpoint of virtual main camera 11, and therefore candidate placement positions Op31 to Op33 for digital actors are placed to the left of lead actor Ac21.

[0112] In the main camera image P21 taken against the background of a display image in which digital actors DAc41 to DAc43 are placed in candidates Op31 to Op33, object Obj1 and digital actors DAc41 to DAc43 are shown without overlapping, as shown in B of Figure 19.

[0113] In this way, the placement position candidate setting unit 33 adjusts the placement position candidates so that the object and the digital actor do not overlap. Also, if a placement position candidate for a digital actor is set above an object and the contact surface of the object with the digital actor is not flat, the placement position candidate setting unit 33 adjusts the height of the placement position candidate taking into account the shape of the contact surface.

[0114] Returning to FIG. 18, in step S42, the placement control unit 34 selects a placement position for the digital actor from among the multiple candidates shown in the placement position candidate map, based on the shooting information and the position of the actor in the main camera image.

[0115] A method for selecting a placement location for a digital actor from among multiple candidates will now be described with reference to Figures 20 and 21.

[0116] As shown on the left side of FIG. 20, it is assumed that candidates for placement positions are set so that 16 digital actors DA51 to DA66 are placed in the virtual space.

[0117] Main camera image P31, taken against the background of an image of a virtual space in which digital actors DA51 to DA66 are positioned, shows some of the digital actors DA51 to DA66 and lead actor Ac21, as shown in the lower right part of Fig. 20. In other words, digital actors (DAc51, DAc55, DAc56, DAc60, DAc61) positioned outside the shooting range of virtual main camera 11 in the virtual space, as well as lead actor Ac21, other digital actors, and digital actors hidden by other objects when viewed from the viewpoint of virtual main camera 11, are not shown in main camera image P31.

[0118] As mentioned above, when there are a large number of digital actors, rendering the display image places a heavy load on the information processing device 13 that performs the rendering, and it may become difficult to render the display image in real time based on the viewpoint, angle of view, etc. of the main camera 11.

[0119] Therefore, the placement control unit 34 acquires the occlusion area and does not place the digital actor in a position that is included in the occlusion area. In other words, the placement control unit 34 selects, from among the candidate placement positions for the digital actor, a placement position that is not included in the occlusion area as the placement position for the digital actor. The occlusion area is an area in virtual space that is hidden by the lead actor Ac21, other digital actors, and other objects when viewed from the viewpoint of the virtual main camera 11.

[0120] In addition, placement positions that are not included in the shooting range of the main camera 11 within the virtual space are excluded from the candidates for placement positions of the digital actor by the placement position candidate setting unit 33, for example, when a placement position candidate map is generated.

[0121] When main camera 11 photographs lead actor Ac21 from the front, the area enclosed by the gray ellipse in A of Fig. 21 becomes the occlusion area. Therefore, among the candidate placement positions (DAc52 to DAc54, DAc57 to DAc59, DAc62 to DAc65) included in the shooting range of main camera 11 in the virtual space, digital actors are not placed at the positions of digital actors DAc63 and DAc64 included in the occlusion area.

[0122] Furthermore, when main camera 11 captures lead actor Ac21 from diagonally in front, the area enclosed by the gray ellipse in B of Fig. 21 becomes the occlusion area. Therefore, among the candidate placement positions (DAc52 to DAc54, DAc57 to DAc59, DAc62 to DAc64) included in the shooting range of main camera 11 in the virtual space, digital actors will not be placed at the positions of digital actors DAc57, DAc62, and DAc64 included in the occlusion area.

[0123] Even if the digital actor is not positioned in the occlusion area, the digital actor will appear in a well-balanced manner in the main camera images P32 and P33, as shown in the lower right portions of Figures 21A and 21B, respectively.

[0124] In this way, each time the viewpoint of the main camera 11 changes (translation, panning, tilting), the placement control unit 34, for example, calculates an occlusion area and determines the placement position of the digital actor based on the shooting information and the actor (person) detection results by the person detection unit 32. The occlusion area may be calculated based on mask information of the person in the main camera image, or may be calculated based on a bounding box surrounding the person.

[0125] Information indicating occlusion areas may be presented to the user, and information indicating occlusion areas may be retained during virtual production filming to serve as a guide map for the user to place digital actors in the virtual space.

[0126] By placing digital actors only in positions that are not included in occlusion regions, the information processing device 13 can make the most of limited rendering resources.

[0127] 18 , in step S43, the placement control unit 34 selects a digital actor to be placed at the placement position selected from the candidates in accordance with the data selection parameters and the subject information. For example, digital actors may be selected so that two-dimensional live-action footage, volumetric footage, and digital human footage are displayed in a predetermined proportion, digital actors wearing clothing of a predetermined color, or digital actors may be selected so that the gender ratio of extras is a predetermined ratio.

[0128] FIG. 22 is a diagram illustrating an example of a method for selecting a digital actor.

[0129] If the lead actor Ac21 is wearing, for example, white clothing, then digital actors DAc81 to DAc84 are placed in the virtual space wearing clothing of a different color from that of the clothing worn by the lead actor Ac21. In Figure 22, the digital actors DAc81 to DAc84 are shown in gray, indicating that the color of the clothing worn by digital actors DAc81 to DAc84 is different from the color of the clothing worn by lead actor Ac21.

[0130] It should be noted that a digital actor wearing clothes of the same color as the clothes worn by the lead actor Ac21 may be placed in the virtual space.

[0131] In this way, the placement control unit 34 can refer to the metadata associated with the digital actor data based on the data selection parameters and the subject information, and select a digital actor that suits the scene.

[0132] 18, in step S44, the placement control unit 34 determines the priority of the digital actors based on, for example, the type of digital actor data. The higher the priority, the closer the digital actor is to be placed in the virtual space to the virtual main camera 11 (i.e., the closer the actor in the studio).

[0133] FIG. 23 shows an example of the placement of digital actors according to the type of digital actor data.

[0134] When filming using virtual production, for example, when creating content in which naturalness is a priority, digital actor DAc91 displayed as a volumetric image is placed closest to the virtual main camera 11 (leading actor Ac21) in the virtual space, as shown in A of Figure 23. Furthermore, digital actor DAc92 displayed as a 2D live-action image is placed second closest to the virtual main camera 11 in the virtual space, and digital actor DAc93 displayed as a digital human image is placed farthest from the virtual main camera 11 in the virtual space. Here, the priority order is from digital actor DAc91 to DAc93.

[0135] In main camera image P41, which was taken against the background of a display image including digital actors DAc91 to DAc93, digital actor DAc91 appears the largest and digital actor DAc93 appears the smallest of digital actors DAc91 to DAc93, as shown in B of Fig. 23. Because digital actor DAc93 appears small, the unnaturalness of the digital human is less noticeable.

[0136] For example, when filming extras is done in parallel with filming using a virtual production, the digital actor displayed using the digital actor data obtained by filming the extras is positioned closest to the virtual main camera 11 in the virtual space.

[0137] Furthermore, the priority of digital actors may be determined based on the number of meshes (number of polygons, data size) of the 3D model that represents the shape of the digital actor.

[0138] As shown in FIG. 24, the LOD (Level Of Detail) of a 3D model generated by volumetric capture technology or a 3D model of a digital human can be adjusted.

[0139] A high-polygon 3D model Mo1 is shown in A of Fig. 24, and a low-polygon 3D model Mo2 is shown in B of Fig. 24. The high-polygon 3D model Mo1 is a 3D model rendered with high resolution using a large number of meshes (polygons). The low-polygon 3D model Mo2 is a 3D model rendered with a smaller number of meshes (polygons) than the 3D model Mo1.

[0140] For example, the placement control unit 34 sets a high priority to digital actors whose shapes are represented by high-polygon 3D models (3D model data sizes are large) and a low priority to digital actors whose shapes are represented by low-polygon 3D models (3D model data sizes are small). In this way, the placement control unit 34 can position and display high-resolution digital actors in positions close to the virtual main camera 11 in the virtual space, thereby maintaining high quality of the overall displayed image.

[0141] Returning to FIG. 18, in step S45, the placement control unit 34 places the digital actors in the virtual space according to priority.

[0142] 25 and 26 are diagrams showing examples of the arrangement of two-dimensional real-life images.

[0143] As described above, the 2D live-action image of the digital actor is rendered on a plane placed in the virtual space, and the plane on which the 2D live-action image is rendered is positioned so as to always face the viewpoint of the virtual main camera 11 in the virtual space.

[0144] For example, as shown in the order of A in Figure 25, B in Figure 25, and C in Figure 26, when the viewpoint of the main camera 11 changes from viewpoint V1 to viewpoint V2, and from viewpoint V2 to viewpoint V3, the rectangular two-dimensional live-action image P51 is positioned so that it always faces directly toward the viewpoint of the main camera 11.

[0145] When the two-dimensional live-action image P51 is placed in a virtual space with its orientation fixed, as shown in Figure 26D, for example, in the main camera image P74 taken from a viewpoint on the right side of the two-dimensional live-action image P51, the perspective of the two-dimensional live-action image P51 (digital actor) becomes unnatural.

[0146] On the other hand, when the two-dimensional live-action image P51 is positioned in the virtual space so that it always faces directly toward the viewpoint of the virtual main camera 11, the sense of perspective of the two-dimensional live-action image P51 (digital actor) is guaranteed in all of the main camera images P71 to P73 taken from viewpoints V1 to V3, respectively, as shown in the lower right parts of Figures 25A, 25B, and 26C.

[0147] After the digital actor is placed in the virtual space in step S45 of FIG. 18, processing returns to step S3 of FIG. 9 and subsequent processing is performed.

[0148] Through the above processing, the information processing device 13 of the present technology can suitably render video including digital actors by arranging the digital actors in a well-balanced manner and rendering the minimum number of digital actors required.

[0149] Because the digital actor data is created as 3D data, the information processing device 13 of the present technology can instantly (in real time) edit the size, orientation, movement, etc. of the digital actor in response to user input, the viewpoint of the main camera 11, etc., even during filming by virtual production. Because a display image is generated by placing a 3D model of the digital actor in a three-dimensional virtual space, the information processing device 13 can easily control the lighting of the digital actor by controlling the light source information of the virtual space. In this way, the information processing device 13 can generate a suitable display image in real time in response to user input, the viewpoint of the main camera 11, etc.

[0150] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0151] FIG. 27 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0152] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0153] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0154] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0155] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0156] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0157] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0158] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0159] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0160] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0161] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0162] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0163] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0164] (1) An information processing device comprising: a placement control unit that selects a placement position of the person in a virtual space from among multiple candidates based on a position of the subject in a camera image acquired by a first camera that captures the subject against a background of a display that displays a display image including the person; and a rendering unit that generates the display image by rendering an image of the virtual space seen from a virtual viewpoint corresponding to the viewpoint of the first camera. (2) The information processing device described in (1), wherein the placement control unit selects, as the placement position of the person, a placement position that is not included in an area in the virtual space that is hidden by the subject when viewed from the virtual viewpoint. (3) The information processing device described in (2), wherein the placement control unit selects, as the placement position of the person, a placement position that is not included in an area in the virtual space that is hidden by another person or object placed in the virtual space when viewed from the virtual viewpoint. (4) The information processing device described in any of (1) to (3), further comprising: a setting unit that sets multiple candidates for the placement position of the person based on shooting information related to shooting by the first camera. (5) The information processing device according to (4), wherein the setting unit sets candidates for the placement position of the person based on at least one of the position and shape of an object other than the person placed in the virtual space. (6) The information processing device according to (4) or (5), wherein the setting unit adjusts the number of candidates for the placement position of the person based on the proportion of an area in which the subject occupies in the camera image acquired by the first camera. (7) The information processing device according to any of (4) to (6), wherein the setting unit generates information to be presented to a user, which indicates candidates for the placement position of the person. (8) The information processing device according to any of (1) to (7), wherein the placement control unit places at least one of live-action video of the person, a 3D model generated based on multiple live-action videos of the person taken from multiple viewpoints, and a 3D model of the person created by CG, as data indicating the person, at the placement position of the person.(9) The information processing device according to (8), wherein the placement control unit, when placing multiple people in the virtual space, determines a priority order for each of the multiple people, and places the person at a placement position closer to the virtual viewpoint as the priority order increases. (10) The information processing device according to (9), wherein the placement control unit determines the priority order for the people based on a type of data representing the people. (11) The information processing device according to (9) or (10), wherein the placement control unit increases the priority order for the people as the data size of a 3D model representing the people increases. (12) The information processing device according to any of (8) to (11), wherein the placement control unit places live-action video of the people so that the people are always directly facing the virtual viewpoint in the virtual space. (13) The information processing device according to any of (1) to (12), further comprising a detection unit that detects a position of the subject in the camera video acquired by the first camera. (14) The information processing device according to (13), wherein the detection unit identifies a position of the subject in the camera image acquired by the first camera based on the camera image acquired by the first camera or the camera image acquired by a second camera different from the first camera. (15) The information processing device according to (13) or (14), wherein the detection unit analyzes the camera image acquired by the first camera to acquire subject information indicating characteristics of the subject. (16) The information processing device according to (15), wherein the placement control unit selects the person to be placed in the virtual space from among a plurality of candidates based on the subject information or information input by a user. (17) The information processing device according to (16), wherein the detection unit acquires information indicating a color of clothing of the subject as the subject information, and the placement control unit selects the person wearing clothing of a color corresponding to the color of the clothing of the subject as the person to be placed in the virtual space.(18) The information processing device according to (16) or (17), wherein the detection unit acquires information indicating movement of the subject as the subject information, and the placement control unit selects the person who moves in accordance with the movement of the subject as the person to be placed in the virtual space. (19) An information processing method comprising: selecting a placement position of the person in a virtual space from among a plurality of candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a background of a display that displays a display image including the person, and rendering an image of the virtual space seen from a virtual viewpoint corresponding to the viewpoint of the camera to generate the display image. (20) A program for causing a computer to execute processes of: selecting a placement position of the person in a virtual space from among a plurality of candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a background of a display that displays a display image including the person, and rendering an image of the virtual space seen from a virtual viewpoint corresponding to the viewpoint of the camera to generate the display image.

[0165] REFERENCE SIGNS LIST 1 Photography system, 11 Main camera, 12 Display, 13 Information processing device, 31 Camera image acquisition unit, 32 Person detection unit, 33 Placement position candidate setting unit, 34 Placement control unit, 35 Rendering unit

Claims

1. An information processing device comprising: a placement control unit that selects a placement position of a person in a virtual space from among multiple candidates based on the position of the subject in a camera image acquired by a first camera that photographs the subject against a display that displays a display image including the person in the background; and a rendering unit that renders an image of the virtual space seen from a virtual viewpoint corresponding to the viewpoint of the first camera to generate the display image.

2. The information processing device according to claim 1, wherein the placement control unit selects, as the placement position of the person, a placement position that is not included in an area in the virtual space that is hidden by the subject when viewed from the virtual viewpoint.

3. The information processing device according to claim 2, wherein the placement control unit selects, as the placement position of the person, a placement position that is not included in an area in the virtual space that is hidden by other people or objects placed in the virtual space when viewed from the virtual viewpoint.

4. The information processing device according to claim 1, further comprising a setting unit that sets a plurality of candidates for the position of the person based on photographing information relating to photographing by the first camera.

5. The information processing device according to claim 4, wherein the setting unit sets candidates for the placement position of the person based on at least one of the position and shape of an object other than the person placed in the virtual space.

6. The information processing device according to claim 4, wherein the setting unit adjusts the number of candidates for the position of the person based on the proportion of the area in which the subject appears within the camera image acquired by the first camera.

7. The information processing device according to claim 4, wherein the setting unit generates information to be presented to a user, the information indicating candidates for placement positions of the person.

8. The information processing device according to claim 1, wherein the placement control unit places at least one of the following data representing the person at the placement position: live-action footage of the person; a 3D model generated based on multiple live-action footage of the person taken from multiple viewpoints; and a 3D model of the person created using CG.

9. The information processing device according to claim 8, wherein when placing multiple people in the virtual space, the placement control unit determines a priority for each of the multiple people, and places the person at a placement position closer to the virtual viewpoint as the priority increases.

10. The information processing device according to claim 9, wherein the placement control unit determines the priority of the person based on the type of data indicating the person.

11. The information processing device according to claim 9, wherein the placement control unit assigns a higher priority to a person as the data size of the 3D model representing the person increases.

12. The information processing device according to claim 8, wherein the placement control unit places the live-action video of the person so that the person always faces the virtual viewpoint in the virtual space.

13. The information processing device according to claim 1, further comprising a detection unit that detects the position of the subject within the camera image acquired by the first camera.

14. The information processing device described in claim 13, wherein the detection unit identifies the position of the subject in the camera image acquired by the first camera based on the camera image acquired by the first camera or the camera image acquired by a second camera different from the first camera.

15. The information processing device according to claim 13, wherein the detection unit analyzes the camera image acquired by the first camera to acquire subject information indicating characteristics of the subject.

16. The information processing device according to claim 15, wherein the placement control unit selects the person to be placed in the virtual space from among a plurality of candidates based on the subject information or information input by a user.

17. The information processing device described in claim 16, wherein the detection unit acquires information indicating the color of the subject's clothing as the subject information, and the placement control unit selects the person wearing clothing of a color corresponding to the color of the subject's clothing as the person to be placed in the virtual space.

18. The information processing device according to claim 16, wherein the detection unit acquires information indicating the movement of the subject as the subject information, and the placement control unit selects the person who moves in accordance with the movement of the subject as the person to be placed in the virtual space.

19. An information processing method comprising: selecting a position for a person in a virtual space from among multiple candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a display that displays a display image including the person in the background; and rendering an image of the virtual space as seen from a virtual viewpoint corresponding to the viewpoint of the camera to generate the display image.

20. A program for causing a computer to execute the following process: selecting a position for a person in a virtual space from among multiple candidates based on the position of the subject in a camera image acquired by a camera that photographs the subject against a display that displays a display image including the person in the background; and rendering an image of the virtual space seen from a virtual viewpoint corresponding to the viewpoint of the camera to generate the display image.

Images