Information processing apparatus, information processing method, and program

Abstract

The present technology relates to an information processing apparatus, an information processing method, and a program that enable appropriate rendering of a video including a person. The information processing apparatus includes an arrangement control unit that selects an arrangement position of a person in a virtual space from among a plurality of candidates based on a position of a subject in a camera video acquired by a first camera that captures the subject with a display as a background, the display displaying a display video including the person, and a rendering unit that renders a video of the virtual space viewed from a virtual viewpoint corresponding to a viewpoint of the first camera to generate the display video. The present technology can be applied to, for example, an imaging system for virtual production.

Description

Technical Field

[0001] This technology relates to information processing apparatus, information processing methods and programs, and more particularly to information processing apparatus, information processing methods and programs capable of appropriately rendering videos including people. Background Technology

[0002] In recent years, virtual production (In-camera VFX), a shooting technique that uses large displays, has been widely adopted in the production of films and television series.

[0003] In virtual production, filming is performed using video displayed on a monitor as the background and a subject positioned in front of the monitor as the foreground. The monitor displays, for example, the portion of the video used as the background when the subject is filmed from a virtual viewpoint corresponding to the viewpoint of the camera device in a virtual space. This video is rendered based on the viewpoint, angle, etc., of the camera device.

[0004] In film and television production, the main actors can be filmed against a backdrop of extras, who act as passersby or other people in the crowd. Similarly, in virtual production, the main actors can be filmed against a backdrop of video displayed on a monitor, including extras.

[0005] For example, Patent Document 1 describes a technique for synthesizing live-action video of people in association with stage video created by computer graphics (CG). Citation List Patent documents

[0006] Patent Document 1: JP 2020-72415 A Summary of the Invention Technical issues

[0007] When there are many extras, rendering videos that include extras will increase the load on the rendering device, which may make it difficult to render the video in real time based on the viewpoint and angle of the camera device.

[0008] This technology was developed in view of such circumstances, and its purpose is to enable the appropriate rendering of videos that include people. Solution to the problem

[0009] An information processing apparatus according to one aspect of the present technology includes: a placement control unit configured to select, from a plurality of candidates, a placement position of a person in a virtual space based on the position of a subject in a video captured by a first camera device, the first camera device being configured to capture the subject against a display screen, the display screen being configured to display a display video including the person; and a rendering unit configured to render a video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the first camera device to generate a display video.

[0010] According to one aspect of the present technology, the information processing method includes: selecting, from multiple candidates, the placement position of a person in a virtual space based on the position of a subject in a video captured by a camera device; the camera device is configured to capture the subject against a display screen as a background; the display screen is configured to display a video including the person; and rendering the video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device to generate a display video.

[0011] A procedure according to one aspect of the present technology causes a computer to perform the following operations: selecting, from a plurality of candidates, the arrangement position of a person in a virtual space based on the position of a subject in a video captured by a camera device, the camera device being configured to capture the subject against a display screen, the display screen being configured to display a display video including the person; and rendering the video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device to generate a display video.

[0012] According to one aspect of the present technology, based on the position of the subject in the video captured by the camera device, the arrangement position of a person in a virtual space is selected from multiple candidates, the camera device is configured to capture the subject against a display as a background, the display is configured to display a display video including the person; and the video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device is rendered to generate a display video. Attached Figure Description

[0013] Figure 1 This diagram is used to describe an overview of a shooting system that utilizes this technology.

[0014] Figure 2 This is an example diagram showing video footage captured by the main camera device.

[0015] Figure 3 This is an example diagram showing the state during filming.

[0016] Figure 4 This is another example of a video recording taken by the main camera device.

[0017] Figure 5This is a diagram used to describe an example of a video displaying digital actors.

[0018] Figure 6 This is a diagram illustrating a display example of digital actors in the prior art.

[0019] Figure 7 This is a diagram illustrating a display example of a digital actor using this technology.

[0020] Figure 8 This is a block diagram illustrating an example of the functional configuration of an information processing device.

[0021] Figure 9 It is used to describe a composition having, for example Figure 8 The flowchart shows the process performed by the information processing device configured in the configuration.

[0022] Figure 10 This is a schematic diagram illustrating an example of the arrangement of a secondary camera device.

[0023] Figure 11 This is an example diagram showing the video from the main camera device.

[0024] Figure 12 This diagram schematically illustrates the state of a photography studio without a secondary camera installed.

[0025] Figure 13 It is used to describe in Figure 9 The flowchart for the character detection process performed in step S2.

[0026] Figure 14 This is the first diagram, used to illustrate an example of adjusting the number of candidates for the placement of digital actors based on the proportion of the area occupied by the actors in the video from the main camera setup.

[0027] Figure 15 This is the second figure, an example used to describe the number of candidates for adjusting the placement of digital actors based on the proportion of the area occupied by the actors in the main camera's video.

[0028] Figure 16 This is an example diagram showing a candidate map of the placement positions for digital actors.

[0029] Figure 17 This is another example of a map showing candidate locations for the placement of digital actors.

[0030] Figure 18 It is used to describe in Figure 9 The flowchart for the digital actor placement process performed in step S3.

[0031] Figure 19 This is a diagram showing an example of candidate placement positions for digital actors based on objects arranged in virtual space.

[0032] Figure 20 This is the first diagram used to describe a method for selecting the placement of digital actors from multiple candidates.

[0033] Figure 21 This is the second figure, used to describe a method for selecting the placement of digital actors from multiple candidates.

[0034] Figure 22 This is a diagram illustrating an example of a method for selecting digital actors.

[0035] Figure 23 This is a diagram illustrating an example of arranging digital actors based on the type of digital actor data.

[0036] Figure 24 This is a diagram showing examples of high-polygon and low-polygon 3D models.

[0037] Figure 25 The first image shows an example of the layout of a two-dimensional live-action video.

[0038] Figure 26 This is the second image, showing an example of the arrangement of a two-dimensional live-action video.

[0039] Figure 27 This is a block diagram illustrating an example of a computer's hardware configuration. Detailed Implementation

[0040] The following section describes the implementation methods used to achieve this technology. The descriptions will proceed in the following order. 1. Overview of the shooting system 2. Configuration and operation of information processing devices

[0041] 1. Overview of the shooting system

[0042] Figure 1 This diagram is used to describe an overview of a shooting system that utilizes this technology.

[0043] Figure 1 The shooting system 1 is, for example, a system for in-camera VFX shooting. The shooting system 1 includes a main camera device 11, a video storage device (not shown) for storing video of the camera device captured by the main camera device 11, a wall display 12, and an information processing device 13 (not shown) for controlling the display 12.

[0044] The display 12 is, for example, a light-emitting diode (LED) display and is set in a real space such as a photography studio. The display 12 displays, for example, video of a virtual space created by computer graphics (CG). The cameraman P uses the main camera device 11 to film the motorcycle M1 as the subject against the background of the video displayed on the display 12.

[0045] Figure 2 This is a diagram showing an example of video captured by the main camera device 11.

[0046] like Figure 2 As shown, the video captured by the main camera device 11 is a video in which the motorcycle M1 appears to exist in a virtual space represented by the displayed video. Therefore, by using the shooting system 1, the cameraman P can capture a video in the studio in which the space appearing in the displayed video seems to extend into the background of the motorcycle M1.

[0047] Figure 3 This is an example diagram showing the state during filming.

[0048] like Figure 3 As shown, during the recording, for example, the entire display video is displayed on the entire display 12.

[0049] like Figure 3 As shown in the upper part, when the main camera device 11 is shooting at the motorcycle M1 located in the center from the right, a portion of the display video is displayed in the shooting area A1 in a manner superimposed on the entire display video. The shooting area A1 is the area on the display 12 that is included within the shooting range of the main camera device 11. In the shooting area A1, for example, the portion of the video that serves as the background in the virtual space shown in the display video when shooting the motorcycle M1 from the right is cropped from the overall display video and displayed.

[0050] like Figure 3 As shown in the lower part, when the main camera 11 is shooting from the left towards the motorcycle M1 positioned in the center, a portion of the display video is displayed in the shooting area A1 in a manner that overlays the entire display video. In the shooting area A1, for example, the portion of the video that serves as the background in the virtual space shown in the display video when shooting the motorcycle M1 from the left is cropped from the overall display video and displayed.

[0051] Therefore, during filming, the cameraman P takes pictures while changing the viewpoint, angle, focus position, aperture value, etc. of the main camera device 11 in sequence.

[0052] The information processing device 13 tracks the photographic operations (viewpoint and angle) of the main camera device 11, and controls the position of the video displayed on the display 12 based on the photographic operations of the main camera device 11. Specifically, the information processing device 13 detects the shooting area based on the position, posture, settings, etc. of the main camera device 11, and controls the display video displayed in the shooting area. In the following text, the shooting area within the display surface of the display 12 is also referred to as the internal viewing cone (region within the viewing angle).

[0053] Note that in Figure 3 In the image, the shooting area A1 is circled with a thick line for clarity; however, in practice, the line circling the shooting area is not displayed.

[0054] In the shooting of movies or TV series, the main actors can be filmed against a background of multiple actors playing passersby, crowds, etc., referred to as extras. In the shooting system 1 of this technology, the video including the extras is displayed as a display video on the monitor 12, and the main actors, as the subjects, are filmed against the background of this display video including the extras.

[0055] Figure 4 This is a diagram showing another example of video footage captured by the main camera device 11.

[0056] Video obtained by filming the main actor Ac1 against a background of a display video including extras, for example, is as follows: Figure 4 The video shown depicts the main actor Ac1 seemingly walking on a street with multiple passersby. Therefore, by using the shooting system 1, the cameraman P was able to shoot a video with a large number of extras appearing in the background in a studio where only the main actor Ac1 was actually present.

[0057] In the following text, the extras displayed on monitor 12 will also be referred to as digital actors. Note that the display video, including the digital actors, is displayed on, for example, the internal viewing cone.

[0058] Figure 5 This is a diagram used to describe an example of a video displaying digital actors.

[0059] like Figure 5 As shown, for example, 2D live-action video, volumetric video, and digital human video are displayed as representations of digital actors.

[0060] Two-dimensional live-action video is obtained by shooting actual actors (extras) against a green background and then removing the background.

[0061] Volumetric video is free-viewpoint video that allows for free movement of the viewpoint (position and orientation), and is acquired, for example, by using volumetric capture techniques relative to actual filmed actors (extras) such as a green background. Volumetric capture techniques are those that generate a 3D model of the subject from video shot from multiple viewpoints, and provide video from a free-viewpoint by generating a video of the 3D model that can be viewed from any desired viewpoint.

[0062] Digital human videos are videos of digital humans created using CG, and are free-viewpoint videos that allow for free movement of the viewpoint. Digital humans are characters created by reproducing a person's posture, movements, and emotional expressions.

[0063] When displaying 2D live-action video as representation of digital actors, the 2D live-action video is rendered on a plane arranged in virtual space. Because the actors (digital actors) appearing in the 2D live-action video are rendered on a 2D plane, the perspective of the digital actors sometimes appears unnatural, depending on the viewpoint of the displayed video (the viewpoint of the main camera device 11). Furthermore, when displaying 2D live-action video with multiple actors, it is impossible to adjust the distance between the digital actors or the placement of each digital actor.

[0064] Therefore, when displaying 2D live-action video as a representation of digital actors, the freedom of camera operation in virtual production is low, and the freedom of placement of digital actors in virtual space is also low. 2D live-action video is video obtained by actually filming actors, and therefore the digital actors look very natural. In 2D live-action video, the viewpoint cannot move, and therefore the data volume and rendering load are small.

[0065] In the case of volumetric video, the viewpoint can move. Therefore, when displaying volumetric video as a representation of digital actors, the freedom of camera operation and the freedom of arranging digital actors in virtual space are greater compared to displaying 2D live-action video. However, the digital actors appear more unnatural compared to displaying 2D live-action video. The data volume and rendering load are also greater compared to displaying 2D live-action video.

[0066] In the case of digital human videos, lighting can be freely changed. Therefore, when displaying digital human videos as representations of digital actors, the freedom of camera operation and the placement of digital actors in virtual space are greater in virtual production shooting compared to displaying volumetric videos. However, in the case of digital humans, many digital humans lack human-like qualities and realism, and their movements are often unnatural; therefore, digital actors appear more unnatural compared to displaying volumetric videos. The data volume and rendering load are also greater compared to displaying volumetric videos.

[0067] Figure 6This is a diagram illustrating a display example of digital actors in the prior art. Figure 6 An example is shown where, in a studio setting, with the video display including digital actors DAc1 to DAc3 shown on monitor 12, the main actor Ac11 and supporting actors Ac12 to Ac14 are filmed as subjects. The same applies to... Figure 7 .

[0068] In existing technology, digital actors are positioned in predetermined locations within a virtual space. Therefore, as... Figure 6 As shown, when viewed from the perspective of the main camera device 11, supporting actors Ac12 to Ac14 may overlap with digital actors DAc1 to DAc3, respectively. In other words, digital actors DAc1 to DAc3 may not appear significantly in the video feed of the main camera device 11.

[0069] Furthermore, when there are a large number of digital actors, rendering the display video will place a heavy load on the information processing device 13 that performs the rendering, which may make it difficult to render the display video in real time based on the viewpoint and angle of the main camera device 11.

[0070] One embodiment of this technology is conceived with the above points in mind, and proposes a technique that enables the placement of a minimum necessary number of digital actors in virtual space while reducing the load on the information processing device 13. In the information processing device 13 of this technology: based on the position of the subject in the video captured by the main camera device 11, the placement position of the digital actors (characters) in virtual space is selected from multiple candidates, wherein the main camera device 11 is configured to capture the subject against a display screen as a background, the display screen being configured to display a display video including the digital actors; and the video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the main camera device 11 is rendered to generate the display video.

[0071] Figure 7 This is a diagram illustrating a display example of a digital actor using this technology.

[0072] More specifically, such as Figure 7 As shown, the information processing device 13 of this technology automatically controls the positions of digital actors DAc1 to DAc3 in virtual space, so that when viewed from the perspective of the main camera device 11, digital actors DAc1 to DAc3 do not overlap with the main actor Ac11 and supporting actors Ac12 to Ac14. Therefore, digital actors can be displayed effectively.

[0073] When digital human videos are used as representations of digital actors, there is no need for extras in the studio or to pre-shoot extras, thus reducing appearance fees, production costs, etc.

[0074] 2. Configuration and operation of information processing devices

[0075] Figure 8 This is a block diagram illustrating an example of the functional configuration of the information processing device 13.

[0076] like Figure 8 As shown, the information processing device 13 includes a video acquisition unit 31 for a camera device, a person detection unit 32, a candidate setting unit for placement position 33, a placement control unit 34, and a rendering unit 35.

[0077] The video acquisition unit 31 acquires video from the camera device located in the studio and provides it to the person detection unit 32. In the studio, a secondary camera device (secondary camera device), distinct from the primary camera device 11 (first camera device) described above, is installed alongside the primary camera device 11. The secondary camera device is attached to the primary camera device 11 or, for example, installed at a position overlooking the studio.

[0078] The person detection unit 32 detects persons (actors) appearing in the video from the camera device provided by the camera device video acquisition unit 31, and provides the person detection results to the placement position candidate setting unit 33 and the placement control unit 34. Specifically, the person detection unit 32 detects the position of persons in the camera device video. Note that the person detection unit 32 can also analyze the camera device video to obtain subject information indicating the characteristics (such as height and gender) of persons appearing in the camera device video.

[0079] The placement candidate setting unit 33 acquires shooting information, i.e., information related to shooting via the main camera device 11. The shooting information includes, for example, information indicating the internal states of the main camera device 11, such as viewing angle, focus position, and aperture value, as well as information indicating the viewpoint of the main camera device 11. Based on the shooting information, the placement candidate setting unit 33 places the virtual main camera device 11 in a virtual space and calculates the shooting range of the virtual main camera device 11 in the virtual space.

[0080] The placement candidate setting unit 33 sets up candidate placement positions for digital actors within the shooting range of the virtual main camera device 11 in the virtual space. The placement candidate setting unit 33 generates a placement candidate map as information indicating candidate placement positions for digital actors in the virtual space, and provides the placement candidate map along with shooting information to the placement control unit 34.

[0081] Based on the subject information obtained by the person detection unit 32, the placement control unit 34 determines placement parameters related to the method of selecting the placement position of digital actors and data selection parameters related to the method of selecting digital actors to be placed in virtual space.

[0082] The setup parameters specify information such as the density of the crowd composed of digital actors, the randomness of the crowd arrangement, the number of people in the crowd, the distance between the studio actors and digital actors in virtual space, and the spacing between the studio actors and digital actors in the video feed from the camera device. The data selection parameters specify information such as the orientation of the digital actors, their gender, age, and the color of their costumes. Note that both the setup parameters and data selection parameters can be input by the user of the shooting system 1 (such as a cameraman or producer).

[0083] Based on the detection results of the person by the person detection unit 32 and the shooting information provided by the placement position candidate setting unit 33, the placement control unit 34 selects the placement position of the digital actor from multiple candidates set by the placement position candidate setting unit 33. For example, the placement control unit 34 selects a placement position such that when viewed from the virtual master camera device 11, the digital actor does not overlap with the people in the studio.

[0084] Furthermore, the placement control unit 34 selects a placement position from multiple candidates set by the placement position candidate setting unit 33 based on placement parameters. For example, when shooting a scene with a crowded crowd, a placement position closer to the actors in the studio is selected, while when shooting a scene where the actors are in a town, a placement position slightly farther away from the actors in the studio is selected.

[0085] The control unit 34 acquires digital actor data based on data selection parameters. Digital actor data is used to display digital actors (data representing digital actors) and includes 2D live-action video, 3D actor models generated through volumetric capture technology, and 3D digital human models. The digital actor data is associated with metadata indicating: the type of digital actor data; multiple meshes of the 3D model; the digital actor's height, gender, and age; and so on.

[0086] The deployment control unit 34 can obtain digital actor data corresponding to the data selection parameters by referring to metadata based on the data selection parameters.

[0087] Note that the 2D live-action video and the 3D models of the actors generated through volumetric capture technology can be content acquired through previous shooting or real-time content acquired through shooting in parallel with virtual production shooting.

[0088] The deployment control unit 34 places each digital actor in the virtual space. Specifically, it places the 2D live-action video, the actor's 3D model, and the digital human's 3D model in the virtual space.

[0089] The rendering unit 35 arranges the display surface of the virtual display 12 in a virtual space and projects the virtual space viewed from the virtual master camera device 11 (virtual viewpoint) onto the display surface of the virtual display 12, thereby rendering the displayed video. Here, the rendering of volumetric video or digital human video is also performed based on the lighting in the virtual space. Note that the lighting for two-dimensional live-action video does not need to be adjusted.

[0090] The rendering unit 35 provides the display video to the display 12 and causes the display 12 to display the display video.

[0091] Next, we will refer to Figure 9 The flowchart in the diagram describes the processing performed by the information processing device 13, which has a configuration as described above.

[0092] In step S1, the video acquisition unit 31 of the camera device acquires video from the camera device set in the photography studio.

[0093] In step S2, the information processing device 13 performs person detection processing. In this process, people appearing in the video captured by the main camera device 11 are detected, and a candidate map of digital actor placement positions is generated. Note that the following will refer to... Figure 13 Describe the details of the character detection and processing.

[0094] In step S2, for example, people appearing in the video captured by the main camera 11 are detected based on the video captured by the secondary camera.

[0095] Figure 10 This is a schematic diagram illustrating an example of the arrangement of a secondary camera device. Figure 10 In the example, it is assumed that the main camera device 11 is filming a movie or TV series with the display 12 as the background and the main actor Ac21 as the subject. In the following text, the video of the camera device captured by the main camera device 11 will also be referred to as the main camera device video, and the video of the camera device captured by the secondary camera device will also be referred to as the secondary camera device video.

[0096] like Figure 10 As shown in Figure A, the secondary camera device 51 views and films the entire studio from above. Figure 10 As shown in B, in the sub-camera video P1 captured by the sub-camera device 51, the full body of the main actor Ac21 is shown when viewed from the upper left.

[0097] The secondary camera device 51 is calibrated so that the information processing device 13 can always identify the positional relationship between the secondary camera device 51 and the main camera device 11. Thus, the information processing device 13 can identify the position of the main actor Ac21 in the main camera device video by projecting the position of the main actor Ac21 detected in the secondary camera device video P1 onto the main camera device video.

[0098] Typically, main camera video is considered to have high image quality, and therefore detecting people appearing in the main camera video based on it takes time. By identifying the position of people in the main camera video based on secondary camera video, which has lower image quality than the main camera video, the position of people can be detected much faster than based on the main camera video, and digital actors can be displayed without delay.

[0099] The position of each person in the camera-equipped video is indicated by bounding boxes, skeletal pose information, facial feature points, etc. The position of each person in the main camera-equipped video can be indicated by feature points that co-occur in both the main and secondary camera-equipped videos. People appearing in the camera-equipped video can be detected using a learning model acquired through machine learning or similar methods.

[0100] Note that multiple secondary camera devices 51 can be installed in the studio, and the three-dimensional position of people in the studio can be detected based on the videos captured by these secondary camera devices 51 respectively.

[0101] The secondary camera device 51 may include a non-visible light camera such as a time-of-flight (ToF) camera, an infrared (IR) camera, a short-wave infrared (SWIR) camera, or a thermal camera. Multiple different types of secondary camera devices 51, such as visible light cameras, invisible light cameras, and laser detection and ranging (LiDAR), can be installed in the studio.

[0102] Return to Figure 9 In step S3, the information processing device 13 performs digital actor placement processing. This process determines the placement positions of the digital actors and places them in virtual space. The following will refer to... Figure 18 Describe the details of the digital actor setup and processing.

[0103] Note that the character detection processing in step S2 and the digital actor placement processing in step S3 can be performed for each frame of the main camera video, the secondary camera video, the display video, etc., or they can be performed, for example, only when shooting a scene that constitutes a movie or TV series begins.

[0104] In step S4, the rendering unit 35 renders the display video and displays it on the monitor 12. The main camera device 11 uses the display video as a background to film the actors in the studio, who are the subjects.

[0105] Figure 11 This is an example diagram showing the video from the main camera device.

[0106] exist Figure 11 In example A, the main actor Ac22 appears in the left portion of the main camera's video, and two digital actors DAc11 and DAc12 appear in the right portion. Through the processing described above, as... Figure 11 As shown in A, digital actors DAc11 and DAc12 can be positioned so that they do not overlap with the main actor Ac22 when viewed from the perspective of the main camera device 11.

[0107] Without adjusting the size of the digital actor, when viewed from the perspective of the main camera 11, the size of the digital actor DAc13 may appear very large compared to the size of the main actor Ac22, such as... Figure 11 As shown in B. In the information processing apparatus 13 of this technology, not only is the position of the person in the video of the main camera device acquired, but also subject information such as the height of the person is acquired, so that the size of the digital actors DAc11 and DAc12 can be adjusted according to the height of the main actor Ac22, such as Figure 11 As shown in A.

[0108] The example of detecting the position of a person in the video of the main camera based on the video of the secondary camera has been described above; however, it is also possible to detect the position of a person in the video of the main camera based on the video of the main camera.

[0109] Figure 12 This diagram schematically illustrates the state of the studio without the auxiliary camera device 51 installed.

[0110] like Figure 12 As shown in Figure A, when the main camera device 11 is shooting a movie or TV series with the display 12 as the background and the main actor Ac21 as the subject, in the main camera device video P2 shot by the main camera device 11, the upper body of the main actor Ac21, seen from the front, appears in, for example, slightly to the left of the center. Figure 12 As shown in B.

[0111] Without the secondary camera device 51 installed, the information processing device 13 detects the position of the person appearing in the main camera device video P2 based on the main camera device video P2. In the main camera device video P2, since the left and right sides of the main actor Ac21 are empty, digital actors are arranged, for example, to fill that area.

[0112] Next, refer to Figure 13 The flowchart is in Figure 9 The character detection process performed in step S2 is described below.

[0113] In step S21, the person detection unit 32 detects actors (people) appearing in the video of the camera device.

[0114] In step S22, the character detection unit 32 determines whether the number of actors appearing in the video of the camera device is one.

[0115] If, in step S22, it is determined that the number of actors appearing in the video from the camera device is not one, the person detection unit 32 calculates the spacing between the actors in the video from the camera device in step S23. The information indicating the spacing between the actors is used as subject information. Next, the process proceeds to step S24. On the other hand, if, in step S22, it is determined that the number of actors appearing in the video from the camera device is one, the processing in step S23 is skipped, and the process proceeds to step S24.

[0116] In step S24, the person detection unit 32 analyzes the video from the camera device and acquires subject information indicating each actor's height, gender, clothing color, etc. Note that motion vectors, i.e., information indicating the amplitude and direction of the actor's movement, can also be used as subject information for analysis.

[0117] When analyzing an actor's motion vector, a digital actor is displayed that moves in accordance with the actor's motion. For example, a digital actor moving in the same direction as the actor's movement or moving in the opposite direction is displayed.

[0118] The metadata of the digital actor data includes information indicating the direction of movement of the digital actor. The information processing device 13 can select a digital actor from multiple candidates that corresponds to the movement of the actor in the studio by referring to the metadata, and then arrange the digital actor.

[0119] When filming extras in parallel with virtual production shooting and acquiring, for example, 2D live-action video, the direction of movement of the extras can be obtained in real time by analyzing the 2D live-action video. When filming extras in parallel with virtual production shooting, the motion vectors of the actors in the virtual production studio can be presented to the extras. The extras move according to the direction and amount of movement of the actors, enabling a display where the digital actors move in unison with the actors' movements.

[0120] The movement of a digital actor can be controlled based on the movement of the actor (subject). For example, if the actor is seen walking to the left from the viewpoint of the main camera 11, the digital actor's movement is controlled to walk to the right.

[0121] In step S25, the placement candidate setting unit 33 sets candidate placement positions for digital actors in the virtual space and generates a candidate placement position map for digital actors.

[0122] Specifically, the placement candidate setting unit 33 places the virtual main camera device 11 in the virtual space based on the shooting information, and calculates the shooting range of the virtual main camera device 11 in the virtual space. The placement candidate setting unit 33 sets candidate placement positions for digital actors within the shooting range of the virtual main camera device 11. Furthermore, the placement candidate setting unit 33 adjusts the number of candidate placement positions for digital actors based on the proportion of the area where the actors appear in the main camera device video.

[0123] Figure 14 and Figure 15 This is a diagram illustrating an example of adjusting the number of candidates for the placement of digital actors based on the proportion of the area occupied by the actors in the main camera's video feed.

[0124] First, such as Figure 14 As shown in Figure A, assume that the main camera device video P11 is used to film the upper body of the main actor Ac21. In this case, as... Figure 14 As shown in Figure B, two candidate positions, Op1 and Op2, are set in the virtual space for arranging digital actors. When viewed from the viewpoint of the virtual master camera device 11, the candidate positions, Op1 and Op2, are set facing backward in the display 12.

[0125] Next, as Figure 15 As shown in A, assume that the main camera video P12 captures the full-body appearance of the main actor Ac21. In this case, the proportion of the area in the main camera video P12 where the main actor Ac21 appears is smaller than the proportion of the area in the main camera video P11 where the main actor Ac21 appears. Therefore, as Figure 15 As shown in B, four candidate positions, Op11 to Op14, are set in the virtual space for arranging digital actors. When viewed from the viewpoint of the virtual master camera device 11, the candidate positions, Op11 to Op14, are arranged facing the rear in the display 12.

[0126] Therefore, the smaller the proportion of the area where the actors appear in the main camera's video, in other words, the larger the shooting area on the display surface of the monitor 12, the more digital actors need to be displayed. The placement candidate setting unit 33 can increase or decrease the number of candidate placement positions for digital actors in real time when the viewpoint or angle of the camera changes.

[0127] Note that the candidate locations for the digital actors in the virtual space, as well as the areas where the digital actors should be placed, can be input by the user. In these cases, the location candidate setting unit 33 generates a candidate map of the digital actors' locations based on the user-input information.

[0128] In addition, a candidate map of digital actor placement locations can be presented to the user. The candidate map of digital actor placement locations presented to the user can be a video overlooking the virtual space, or a video of the virtual space viewed from the viewpoint of the virtual master camera device 11.

[0129] Figure 16 This is an example diagram showing a candidate map of the placement positions for digital actors.

[0130] exist Figure 16 In the example, a video view overlooking virtual space is shown as a candidate map for placement positions. This candidate map shows the virtual main camera setup 11, the avatar of the main actor Ac21, and virtual objects. Furthermore, in the candidate map, three candidate placement positions for the digital actors are indicated by bounding boxes B1 to B3.

[0131] By viewing the candidate placement map, users can confirm that if digital actors are placed in the positions indicated by bounding boxes B1 to B3, well-balanced main camera setup video can be captured—the background is partially filled with extras and objects, and the extras are not concentrated in one position.

[0132] Figure 17 This is another example of a map showing candidate locations for the placement of digital actors.

[0133] exist Figure 17 In the example, a video of a virtual space viewed from the viewpoint of the virtual master camera device 11 is shown as a candidate map for placement positions. This candidate map shows the avatar of the main actor Ac21 and virtual objects. Furthermore, in the candidate map, three candidate placement positions for the digital actors are indicated by bounding boxes B1 to B3.

[0134] Note that users can adjust the candidate placement locations for digital actors while viewing the placement location candidate map described above.

[0135] exist Figure 13After generating the candidate map of layout locations as described above in step S25, return to... Figure 9 Step S2 is performed, and subsequent processing is carried out.

[0136] Next, refer to Figure 18 The flowchart is in Figure 9 The digital actor placement process performed in step S3 is described below.

[0137] In step S41, the placement candidate setting unit 33 updates (sets) the placement candidate of the digital actor based on at least one of the positions or shapes of objects other than the digital actor placed in the virtual space.

[0138] Figure 19 This is a diagram showing an example of candidate placement positions for digital actors based on objects arranged in virtual space.

[0139] In virtual space, there exist objects such as vehicles, buildings, mountains, hills, and rivers. Figure 19 In the example in A, the vehicle object Obj1 is arranged in virtual space. In this case, Figure 19 In the example in A, when viewed from the viewpoint of the virtual master camera device 11, object Obj1 is positioned to the right rear of the main actor Ac21, and therefore, the candidate positions of digital actors Op31 to Op33 are positioned to the left rear of the main actor Ac21.

[0140] like Figure 19 As shown in B, in the main camera video P21, which is shot against the backdrop of a display video showing digital actors DAc41 to DAc43 positioned at candidates Op31 to Op33, the object Obj1 and the digital actors DAc41 to DAc43 do not overlap.

[0141] Therefore, the placement position candidate setting unit 33 adjusts the placement position candidates so that the object and the digital actor do not overlap. Furthermore, when a candidate placement position for the digital actor is set above the object and the contact surface between the object and the digital actor is not planar, the placement position candidate setting unit 33 takes into account the shape of the contact surface to adjust the height of the placement position candidate.

[0142] Return to Figure 18 In step S42, the arrangement control unit 34 selects the arrangement position of the digital actor from multiple candidates shown in the arrangement position candidate map based on the shooting information and the position of the actor in the video of the main camera device.

[0143] Now refer to Figure 20 and Figure 21Describes a method for selecting the placement of digital actors from multiple candidates.

[0144] like Figure 20 As shown on the left, assume that candidate placement positions have been set so that 16 digital actors DA51 to DA66 can be placed in virtual space.

[0145] In the main camera setup video P31, filmed against the backdrop of a virtual space featuring digital actors DA51 to DA66, such as Figure 20 As shown in the lower right portion, some of the digital actors DA51 to DA66 and the main actor Ac21 appear. In other words, digital actors (DAc51, DAc55, DAc56, DAc60, DAc61) placed outside the shooting range of the virtual main camera device 11 in virtual space, as well as digital actors obscured by the main actor Ac21, other digital actors, and other objects when viewed from the perspective of the virtual main camera device 11, do not appear in the main camera device video P31.

[0146] As described above, when rendering and displaying video with a large number of digital actors, the load on the information processing device 13 that performs the rendering increases, which may make it difficult to render and display video in real time based on the viewpoint, angle, etc. of the main camera device 11.

[0147] Therefore, the placement control unit 34 acquires the occlusion area and does not place the digital actor in the location included in the occlusion area. In other words, the placement control unit 34 selects from the candidate placement positions of the digital actor a placement position that is not included in the occlusion area as the placement position of the digital actor. The occlusion area is the area in the virtual space that is obscured by the main actor Ac21, other digital actors, and other objects when viewed from the perspective of the virtual main camera device 11.

[0148] Note that, for example, when generating a candidate map of placement locations, the placement location candidate setting unit 33 excludes placement locations in the virtual space that are not within the shooting range of the main camera device 11 from the candidates for placement locations of the digital actors.

[0149] When the main camera 11 is filming the main actor Ac21 from the front... Figure 21 The area enclosed by the gray ellipse in A is the occlusion area. Therefore, in the virtual space, among the candidate placement positions (DAc52 to DAc54, DAc57 to DAc59 and DAc62 to DAc65) included in the shooting range of the main camera device 11, no digital actors are placed at the positions of digital actors DAc63 and DAc64, which are included in the occlusion area.

[0150] Furthermore, when the main camera 11 is filming the main actor Ac21 from the front along a diagonal direction, Figure 21 In B, the area enclosed by the gray ellipse is the occlusion area. Therefore, in the virtual space, among the candidate placement positions (DAc52 to DAc54, DAc57 to DAc59 and DAc62 to DAc64) included in the shooting range of the main camera device 11, no digital actors are placed at the positions of digital actors DAc57, DAc62 and DAc64 that are included in the occlusion area.

[0151] Even without placing digital actors in obstructed areas, the digital actors appear in a well-balanced manner in the main camera setup's video P32 and P33, such as... Figure 21 The lower right portions of A and B are shown.

[0152] Therefore, whenever the viewpoint of the main camera device 11 changes (pan, tilt, or zoom), the placement control unit 34 calculates the occlusion area and determines the placement position of the digital actor, for example, based on the shooting information and the detection results of the actor (person) by the person detection unit 32. The occlusion area can be calculated based on the mask information of the person in the main camera device video, or it can be calculated based on the bounding box surrounding the person.

[0153] Note that information indicating obscured areas can also be presented to the user. Furthermore, to utilize this information as a guide map when users place digital actors in the virtual space, information indicating obscured areas during virtual production filming can be saved.

[0154] By placing digital actors only in locations not included in the occlusion area, the information processing device 13 can make the most of limited rendering resources.

[0155] Return to Figure 18 In step S43, the placement control unit 34 selects digital actors to be placed in the placement positions selected from the candidates based on the data selection parameters and the subject information. For example, digital actors are selected to display the 2D live-action video, volumetric video, and digital human video at a predetermined ratio, digital actors wearing clothing of a predetermined color are selected, or digital actors are selected to make the gender ratio of the extras a predetermined ratio.

[0156] Figure 22 This is a diagram illustrating an example of a method for selecting digital actors.

[0157] In the case where the main actor Ac21 is wearing white clothing, for example, digital actors DAc81 to DAc84, dressed in a different color than the main actor Ac21's clothing, are arranged in virtual space. Figure 22In the image, digital actors DAc81 to DAc84 are shown in gray, indicating that the color of the costumes worn by digital actors DAc81 to DAc84 is different from the color of the costume worn by the main actor Ac21.

[0158] Note that digital actors wearing costumes of the same color as the main actor Ac21 can also be placed in virtual space.

[0159] Therefore, the deployment control unit 34 can select digital actors suitable for the scene by referring to metadata associated with the digital actor data, based on data selection parameters and subject information.

[0160] Return to Figure 18 In step S44, the arrangement control unit 34 determines the priority of digital actors, for example, based on the type of digital actor data. Digital actors with higher priority are placed in the virtual space closer to the virtual master camera device 11 (closer to the actors in the studio).

[0161] Figure 23 This is a diagram illustrating an example of arranging digital actors based on the type of digital actor data.

[0162] In virtual filming, when prioritizing content that feels natural, such as... Figure 23 As shown in Figure A, digital actor DAc91, represented by volumetric video, is positioned in the virtual space closest to the virtual main camera device 11 (main actor Ac21). Furthermore, digital actor DAc92, represented by 2D live-action video, is positioned in the virtual space second closest to the virtual main camera device 11, and digital actor DAc93, represented by digital human video, is positioned in the virtual space farthest from the virtual main camera device 11. Here, priority decreases from digital actor DAc91 to digital actor DAc93.

[0163] In the main camera device video P41, which was shot against the backdrop of display videos including digital actors DAc91 to DAc93, such as Figure 23 As shown in B, among digital actors DAc91 to DAc93, digital actor DAc91 appears the largest, and digital actor DAc93 appears the smallest. When digital actor DAc93 appears small, the unnaturalness of the digital human is not noticeable.

[0164] For example, in the case of shooting extras in parallel with virtual production shooting, the digital actors obtained by shooting the extras are positioned in the virtual space closest to the virtual main camera device 11.

[0165] The priority of digital actors can be determined based on the number of grids (the number of polygons or the amount of data in the 3D model) of each 3D model representing the shape of the digital actor.

[0166] like Figure 24 As shown, the level of detail (LOD) of 3D models generated through volumetric capture technology and 3D models of digital humans can be adjusted.

[0167] Figure 24 A shows a high-polygon 3D model Mo1, and Figure 24 B shows the low-polygon 3D model Mo2. The high-polygon 3D model Mo1 is a 3D model rendered in high definition using a large number of meshes (polygons). The low-polygon 3D model Mo2 is a 3D model rendered using a smaller number of meshes (polygons) compared to 3D model Mo1.

[0168] For example, the placement control unit 34 sets high priority for digital actors with shapes represented by high-polygon 3D models (3D models with large data volume) and low priority for digital actors with shapes represented by low-polygon 3D models (3D models with small data volume). Thus, the placement control unit 34 places and displays digital actors rendered in high definition in the virtual space near the virtual main camera device 11, thereby maintaining high image quality throughout the displayed video.

[0169] Return to Figure 18 In step S45, the arrangement control unit 34 arranges the digital actors in the virtual space according to priority.

[0170] Figure 25 and Figure 26 This is a diagram showing an example of the layout of a two-dimensional live-action video.

[0171] As described above, the two-dimensional live-action video of the digital actors is rendered on a plane arranged in virtual space. The plane on which the two-dimensional live-action video is rendered is arranged in virtual space to always face the viewpoint of the virtual master camera device 11.

[0172] For example, such as Figure 25 A, Figure 25 B Figure 26 As shown in C, when the viewpoint of the main camera device 11 changes from viewpoint V1 to viewpoint V2 and from viewpoint V2 to viewpoint V3, the two-dimensional real-shot video P51 with a rectangular shape is arranged to always face the viewpoint of the main camera device 11.

[0173] When the two-dimensional live-action video P51 is arranged in a virtual space with its orientation fixed, for example... Figure 26As shown in D, in the main camera device video P74, which is shot from a viewpoint to the right of the 2D live-action video P51, the sense of distance of the 2D live-action video P51 (digital actor) becomes unnatural.

[0174] On the other hand, when the two-dimensional live-action video P51 is arranged in virtual space so that it always faces the viewpoint of the virtual main camera device 11, such as Figure 25 A, Figure 25 B Figure 26 As shown in the lower half of C, the sense of distance in the two-dimensional live-action video P51 (digital actor) is ensured in the main camera device videos P71 to P73, which are shot from viewpoints V1 to V3 respectively.

[0175] exist Figure 18 After arranging the digital actors in the virtual space in step S45, the process returns to... Figure 9 Step S3 is performed, and subsequent processing is carried out.

[0176] Through the above processing, the information processing device 13 of this technology can appropriately render a video including digital actors by rendering the minimum necessary number of digital actors while arranging them in a well-balanced manner.

[0177] When creating digital actor data based on 3D data, the information processing device 13 of this technology can immediately (in real-time) edit the size, orientation, and movements of each digital actor based on user input, the viewpoint of the main camera device 11, etc., even during virtual production and filming. When generating display video by arranging 3D models of digital actors in a three-dimensional virtual space, the information processing device 13 can easily control the lighting of the digital actors by controlling the light source information of the virtual space. Therefore, the information processing device 13 can generate suitable display video in real-time based on user input, the viewpoint of the main camera device 11, etc.

[0178] Computer configuration examples

[0179] The series of processes described above can be executed by hardware or by software. When these processes are executed by software, the program constituting the software is installed from a program recording medium onto a computer with embedded dedicated hardware, a general-purpose personal computer, or the like.

[0180] Figure 27 This is a block diagram illustrating an example configuration of computer hardware that performs the series of processes described above through a program.

[0181] The central processing unit (CPU) 501, read-only memory (ROM) 502 and random access memory (RAM) 503 are interconnected via bus 504.

[0182] The input / output interface 505 is also connected to the bus 504. Input units 506, including a keyboard and mouse, and output units 507, including a display and speakers, are connected to the input / output interface 505. Additionally, storage units 508, including hard disks and non-volatile memory, communication units 509, including network interfaces, and drivers 510 that drive removable media 511 are connected to the input / output interface 505.

[0183] In a computer configured as described above, for example, CPU 501 loads a program stored in storage unit 508 into RAM 503 via input / output interface 505 and bus 504, and executes the program to perform a series of processes as described above.

[0184] The program executed by the CPU 501 is recorded on removable medium 511, or provided via wired or wireless transmission media such as local area network, Internet and digital broadcast, and is installed in storage unit 508.

[0185] Note that a program executed by a computer may be a program that performs processing sequentially in the order described in this specification, or it may be a program that performs processing in parallel or at necessary time points (e.g., when called).

[0186] Note that in this specification, "system" refers to a collection of multiple components (devices, modules (parts), etc.), and it is not important whether all components are housed in the same housing. Therefore, "system" can refer to both multiple devices housed in a separate housing and connected via a network, and multiple modules housed in a single housing.

[0187] Note that the effects described in this article are merely examples and not limitations, and other effects may occur.

[0188] The implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the spirit of this technology.

[0189] For example, this technology can have a cloud computing configuration, in which one function is shared and processed collaboratively by multiple devices via a network.

[0190] Furthermore, each step described in the flowchart above can be performed by a single device or can be performed by multiple devices.

[0191] Furthermore, in cases where a single step includes multiple processing operations, the multiple processing operations included in that single step can be performed by a single device, or can be performed by multiple devices in a distributed manner.

[0192] Configuration combination examples

[0193] Note that this technology can also have the following configurations.

[0194] (1) An information processing device, comprising: A placement control unit is configured to: select a placement position for a person in a virtual space from multiple candidates based on the position of a subject in a video captured by a first camera device; the first camera device is configured to capture the subject against a display background; and the display is configured to display a video including the person. The rendering unit is configured to render video of a virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the first camera device to generate a display video.

[0195] (2) The information processing apparatus according to (1), wherein, The placement control unit is configured to select a placement position in the virtual space that is not included in the area obscured by the subject when viewed from a virtual viewpoint, as the placement position of the character.

[0196] (3) The information processing apparatus according to (2), wherein, The placement control unit is configured to select a placement location in the virtual space that is not included in the area obscured by other characters or objects placed in the virtual space when viewed from the virtual viewpoint, as the placement location of the character.

[0197] (4) The information processing apparatus according to any one of (1) to (3) further comprises: The setting unit is configured to set multiple candidates for the placement of the person based on shooting information related to the shooting by the first camera device.

[0198] (5) The information processing apparatus according to (4), wherein, The setting unit is configured to set multiple candidates for the placement position of the character based on at least one of the position and shape of objects other than the character placed in the virtual space.

[0199] (6) The information processing apparatus according to (4) or (5), wherein, The setting unit is configured to adjust the number of multiple candidates for the placement of the person based on the proportion of the area in which the subject appears in the video obtained by the first camera device.

[0200] (7) The information processing apparatus according to any one of (4) to (6), wherein, The setting unit is configured to generate information to be presented to the user, indicating multiple candidates for the placement of the character.

[0201] (8) The information processing apparatus according to any one of (1) to (7), wherein, The placement control unit is configured to place at least one of the following as data representing a person at the placement position of the person: live-action video obtained by shooting the person, a 3D model generated based on multiple live-action video items obtained by shooting the person from multiple viewpoints, and a 3D model of the person created by CG.

[0202] (9) The information processing apparatus according to (8), wherein, The placement control unit is configured to: when placing multiple characters in a virtual space, determine the priority of each character among the multiple characters, and place the character with the higher the priority in a placement position closer to the virtual viewpoint.

[0203] (10) The information processing apparatus according to (9), wherein, The deployment control unit is configured to determine the priority of a person based on the type of data representing that person.

[0204] (11) The information processing apparatus according to (9) or (10), wherein, The control unit is configured such that the larger the amount of data in the 3D model used to represent a character, the higher the character's priority.

[0205] (12) The information processing apparatus according to any one of (8) to (11), wherein, The arrangement control unit is configured to arrange the live-action video obtained by filming people so that the live-action video is always facing the virtual viewpoint in virtual space.

[0206] (13) The information processing apparatus according to any one of (1) to (12) further comprises: The detection unit is configured to detect the position of the subject in the video captured by the first camera device.

[0207] (14) The information processing apparatus according to (13), wherein, The detection unit is configured to determine the position of the subject in the video captured by the first camera device based on the video captured by the first camera device or the video captured by the second camera device, which is different from the first camera device.

[0208] (15) The information processing apparatus according to (13) or (14), wherein, The detection unit is configured to analyze the video captured by the first camera device to obtain subject information indicating the features of the subject.

[0209] (16) The information processing apparatus according to (15), wherein, The placement control unit is configured to select, from multiple candidates, the person to be placed in the virtual space based on the subject information or information input by the user.

[0210] (17) The information processing apparatus according to (16), wherein, The detection unit is configured to: acquire information indicating the color of the clothing of the subject, as subject information, and The placement control unit is configured to select a person wearing clothing in a color that corresponds to the color of the subject's clothing as the person to be placed in the virtual space.

[0211] (18) The information processing apparatus according to (16) or (17), wherein, The detection unit is configured to acquire information indicating the motion of the subject, as subject information, and The placement control unit is configured to select a character that moves in unison with the movement of the subject as the character to be placed in the virtual space.

[0212] (19) An information processing method, comprising: Based on the position of the subject in the video captured by the camera device, the arrangement position of a person in virtual space is selected from multiple candidates. The camera device is configured to capture the subject against a display screen, and the display screen is configured to display a video including the person. Render the video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device to generate a display video.

[0213] (20) A program for causing a computer to perform the following operations: Based on the position of the subject in the video captured by the camera device, the arrangement position of a person in virtual space is selected from multiple candidates. The camera device is configured to capture the subject against a display screen, and the display screen is configured to display a video including the person. Render the video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device to generate a display video. List of reference numerals

[0214] 1. Shooting system; 11. Main camera device; 12. Display; 13. Information processing device; 31. Video acquisition unit of camera device; 32. Person detection unit; 33. Candidate setting unit for placement position; 34. Placement control unit; 35. Rendering unit.

Claims

1. An information processing apparatus, comprising: A placement control unit is configured to: select, from multiple candidates, a placement position for a person in a virtual space based on the position of a subject in a video captured by a first camera device, wherein the first camera device is configured to capture the subject against a display screen, and the display screen is configured to display a video including the person; and The rendering unit is configured to render a video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the first camera device to generate the display video.

2. The information processing apparatus according to claim 1, wherein, The placement control unit is configured to select a placement position in the virtual space that is not included in the area obscured by the subject when viewed from the virtual viewpoint, as the placement position of the character.

3. The information processing apparatus according to claim 2, wherein, The placement control unit is configured to select a placement position in the virtual space that is not included in an area obscured by other characters or objects placed in the virtual space when viewed from the virtual viewpoint, as the placement position of the character.

4. The information processing apparatus according to claim 1, further comprising: The setting unit is configured to set multiple candidates for the placement of the person based on shooting information related to the shooting by the first camera device.

5. The information processing apparatus according to claim 4, wherein, The setting unit is configured to set multiple candidates for the placement position of the character based on at least one of the position and shape of objects other than the character arranged in the virtual space.

6. The information processing apparatus according to claim 4, wherein, The setting unit is configured to adjust the number of multiple candidates for the placement of the person based on the proportion of the area where the subject appears in the video obtained by the first camera device.

7. The information processing apparatus according to claim 4, wherein, The setting unit is configured to generate information to be presented to the user, the information indicating multiple candidates for the placement of the character.

8. The information processing apparatus according to claim 1, wherein, The placement control unit is configured to place at least one of the following as data representing the person at the placement position of the person: a live-action video obtained by shooting the person, a 3D model generated based on multiple live-action videos obtained by shooting the person from multiple viewpoints, and a 3D model of the person created by CG.

9. The information processing apparatus according to claim 8, wherein, The placement control unit is configured to: when placing multiple characters in the virtual space, determine the priority of each character among the multiple characters, and the higher the priority, the closer the character is to the virtual viewpoint.

10. The information processing apparatus according to claim 9, wherein, The arrangement control unit is configured to determine the priority of a person based on the type of data representing that person.

11. The information processing apparatus according to claim 9, wherein, The placement control unit is configured such that the larger the amount of data in the 3D model used to represent the character, the higher the priority of the character.

12. The information processing apparatus according to claim 8, wherein, The placement control unit is configured to arrange the live-action video obtained by filming the person so that the live-action video is always facing the virtual viewpoint in the virtual space.

13. The information processing apparatus according to claim 1, further comprising: A detection unit is configured to detect the position of the subject in the video captured by the first camera device.

14. The information processing apparatus according to claim 13, wherein, The detection unit is configured to determine the position of the subject in the video captured by the first camera device based on the video captured by the first camera device or the video captured by a second camera device different from the first camera device.

15. The information processing apparatus according to claim 13, wherein, The detection unit is configured to analyze the video footage acquired by the first camera device to obtain subject information indicating the characteristics of the subject.

16. The information processing apparatus according to claim 15, wherein, The placement control unit is configured to select, from multiple candidates, a person to be placed in the virtual space based on the subject information or information input by the user.

17. The information processing apparatus according to claim 16, wherein, The detection unit is configured to: acquire information indicating the color of the clothing of the subject, as subject information, and The placement control unit is configured to select a person wearing clothing of a color corresponding to the color of the subject's clothing as the person to be placed in the virtual space.

18. The information processing apparatus according to claim 16, wherein, The detection unit is configured to: acquire information indicating the movement of the subject, as subject information, and The placement control unit is configured to select a character that moves in accordance with the movement of the subject as the character to be placed in the virtual space.

19. An information processing method, comprising: Based on the position of the subject in the video captured by the camera device, the arrangement position of the person in the virtual space is selected from multiple candidates. The camera device is configured to capture the subject against the background of the display, and the display is configured to display a video including the person. as well as The video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device is rendered to generate the display video.

20. A program for causing a computer to perform the following processes: Based on the position of the subject in the video captured by the camera device, the arrangement position of a person in a virtual space is selected from multiple candidates. The camera device is configured to capture the subject against a display screen, and the display screen is configured to display a video including the person. The video of the virtual space viewed from a virtual viewpoint corresponding to the viewpoint of the camera device is rendered to generate the display video.

Images