Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device has an image generation model, a quality analysis unit, and a photographing support unit. The image generation model reconstructs an emitted radiance field and infers an image of a new viewpoint. The quality analysis unit estimates the quality of an inference image on the basis of an accuracy factor that affects inference accuracy. The photographing support unit outputs, to a display device, support information pertaining to at least either one of the inference accuracy and the quality of the inference image in the photographing of a virtual production in which the inference image is used as a background image.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] A video production technique using Neural Radiance Fields (NeRF) has been attracting attention. NeRF is a technology that infers images from any viewpoint (new viewpoint) using an image generation model called the "continuous volumetric scene function." Because NeRF can generate photorealistic inference images, it is expected to be used as a background asset in virtual production (VP).

[0003] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, COMMUNICATIONS OF THE ACM 99, JANUARY 2022, VOL. 65, NO. 1

[0004] The quality of NeRF inference images may vary depending on the viewpoint and image position due to the shooting method of the images used to train the image generation model and the optical characteristics of the subject. Therefore, when shooting a VP with NeRF inference images as the background, camerawork must be considered to ensure that the inference images are displayed clearly. However, this judgment is difficult for staff (such as studio directors and cameramen) who are unfamiliar with the characteristics of the image generation model.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that provide photography support to achieve appropriate camerawork.

[0006] According to the present disclosure, there is provided an information processing device including an image generation model that reconstructs a radiance field to infer an image from a new viewpoint, a quality analysis unit that estimates the quality of the inferred image based on an accuracy factor that affects the inference accuracy, and a shooting support unit that outputs support information related to at least one of the inference accuracy and the quality of the inferred image to a display device in shooting virtual production using the inferred image as a background image.The present disclosure also provides an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium storing a program that causes a computer to realize the information processing of the information processing device.

[0007] 1 is a diagram explaining NeRF technology. FIG. 1 is a diagram explaining NeRF technology. FIG. 1 is a diagram explaining application of NeRF technology to VP. FIG. 2 is a diagram explaining an example of a process for creating an image generation model. FIG. 3 is a diagram explaining an example of an inference process for an image from a new viewpoint. FIG. 4 is a diagram explaining issues when shooting a VP. FIG. 5 is a diagram explaining issues when shooting a VP. FIG. 6 is a diagram explaining an example of an information processing device. FIG. 7 is a diagram explaining an example of a UI that presents the quality of an NeRF inference image. FIG. 8 is a diagram explaining an example of a guide image that indicates a range of appropriate camera angles. FIG. 9 is a diagram explaining quality prediction based on the density of shooting viewpoints. FIG. 10 is a diagram explaining quality prediction based on the range of shooting viewpoints. FIG. 11 is a diagram explaining quality prediction based on the reproducibility of images for each shooting viewpoint. FIG. 12 is a diagram explaining quality prediction based on the number of shooting viewpoints for each coordinate point. FIG. 13 is a diagram explaining quality prediction based on the number of shooting viewpoints for each coordinate point. FIG. 14 is a diagram explaining quality prediction based on the number of shooting viewpoints for each coordinate point. FIG. 15 is a diagram explaining an example of the hardware configuration of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Background] [1-1. NeRF technology] [1-2. Application of NeRF technology to VP] [1-3. Issues when photographing VP] [2. Photography support method of the present disclosure] [3. UI for presenting quality of NeRF inference image] [4. Quality analysis method] [4-1. Quality prediction based on density of photographing viewpoints] [4-2. Quality prediction based on range of photographing viewpoints] [4-3. Quality prediction based on reproducibility of image for each photographing viewpoint] [5. Example of hardware configuration] [6. Effects]

[0010] [1. Background] [1-1. NeRF Technology] FIGS. 1 and 2 are diagrams illustrating NeRF technology.

[0011] NeRF is a technology that uses a group of images of a target scene captured from multiple viewpoints as input and acquires a 3D representation through AI-based processing. The target scene refers to the scene that is the subject of inference by NeRF. NeRF generates (infers) images from new viewpoints by optimizing a "continuous volumetric scene function" using sparse viewpoint images as input.

[0012] Hereinafter, the "continuous volumetric scene function" will be referred to as an "image generation model MD" (see FIG. 4). The viewpoint image of the actual camera CM used for learning to optimize the image generation model MD will be referred to as an "image IM". Ｃ " The virtual camera CM generated by the image generation model MD Ｖ The viewpoint image is called "inference image IM" Ｇ " Image IM Ｃ The viewpoint of the real camera CM when shooting is referred to as the "shooting viewpoint SV." Ｖ This perspective is referred to as "New Perspective NV."

[0013] The image generation model MD is expressed as a deep network that takes 5D coordinates as input and outputs volume density and viewpoint-dependent radiance. 5D coordinates are expressed as the spatial position of the viewpoint (x, y, z) and the line of sight (θ, φ). The image generation model MD infers the image of the new viewpoint NV by referencing the 5D coordinate volume density and viewpoint-dependent radiance along the ray L emanating from the new viewpoint NV and projecting them onto the image using a volume rendering method.

[0014] Each pixel in an image is represented by one ray L. The value of the ray L (pixel value) is calculated by volume rendering. Specifically, radiance and volume density are obtained from each coordinate point CP in the inference space SP along the ray L. The inference space SP refers to a three-dimensional space that is the inference target of the image generation model MD. The inference space SP includes all subjects SB included in the target scene. Multiple coordinate points CP (3D points) are set in the inference space SP. The value of the ray L is obtained by adding up the radiance and volume density at each coordinate point CP.

[0015] To optimize the image generation model MD, we first use the image IM of the target scene. Ｃ (RGB image) and image IM Ｃ The pose and intrinsic of the real camera CM corresponding to the above are required as input. Pose refers to the shooting viewpoint SV and shooting posture (line of sight). Intrinsic refers to the angle of view and lens distortion. Hereinafter, pose and intrinsic will be referred to as camera parameters. For example, the camera parameters are calculated from the image using an external tool called COLMAP.

[0016] The image generation method using NeRF requires fewer images than the existing 3D capture method. Ｃ This method can generate photorealistic new viewpoint images robustly. It also has the advantage of being able to naturally represent viewpoint-dependent changes in brightness and color due to light reflection, as well as semi-transparent objects.

[0017] [1-2. Application of NeRF Technology to VP] FIG. 3 is a diagram for explaining application of NeRF technology to VP.

[0018] VP is a video production technique that synthesizes live-action images and CG images in real time. A large display called an LED wall is installed in the studio. The large display WD projects background images generated by CG onto a large screen made up of LED panels. The background images are shot together with real objects (such as performers AC and props) in the studio. The background images are taken with a cinema camera CM. Ｒ It changes in accordance with the movement of the subject, and you can get a complete, composited package just by taking a photo.

[0019] Cinema camera commercial Ｒ The movement of the camera is detected by a tracker. The tracker is an infrared sensor installed on the ceiling of the studio or a cinema camera. Ｒ Cinema camera commercials using the IMU (Inertial Measurement Unit) installed in Ｒ The tracker tracks the position of the camera in the studio in real time. Ｒ In addition to the position of Cinema Camera CM Ｒ Camera parameters such as orientation, focus, and lens distortion are also recorded as tracking information.

[0020] The tracking information is transmitted to the renderer, which then renders the virtual camera CM in the 3DCG space. Ｖ A real-world cinema camera commercial Ｒ The 3DCG space refers to the three-dimensional virtual space that serves as the background. Virtual camera CM Ｖ is a virtual camera installed in the 3DCG space. Ｖ The shooting process is done by a virtual camera Ｖ This is performed as a process of generating a new viewpoint image using the camera parameters of the virtual camera CM. Ｖ The new viewpoint image acquired in step 2 is acquired as a background image.

[0021] Virtual camera commercial Ｖ The camera parameters are based on the cinema camera CM Ｒ The virtual camera CM is controlled based on the tracking information. Ｖ is a real-world cinema camera commercial. ＲWhen the camera parameters change, the camera moves in the same direction. Ｒ The background image changes in real time according to the movement of the object. Distortion correction is performed on the background image according to the shape of the large display WD. The distortion-corrected background image is then displayed on the large display WD.

[0022] The background image is shot with the real object in the studio in the foreground. Cinema camera commercial Ｒ The background image seen from the virtual camera commercial Ｖ It matches the background of the 3D CG space seen from the cinema camera commercial. Ｒ By simply taking a photo, you can obtain a complete package in which the actual object and background image are properly combined.

[0023] In the studio, there are many staff members ST (see FIG. 8) involved in the filming. In the studio, inference images IM are stored for the staff members ST. Ｇ A viewer VW is installed to preview the virtual camera commercial. Ｖ Manipulating camera parameters to generate inference images from various viewpoints Ｇ can be confirmed.

[0024] The flow of video production using NeRF technology will be explained using Figures 4 and 5. Figure 4 is a diagram showing an example of the process of creating an image generation model MD. Figure 5 is a diagram showing an example of the process of inferring an image from a new viewpoint NV.

[0025] First, a target scene to be inferred is photographed from multiple viewpoints SV. The captured images are analyzed, and each image IM Ｃ Camera parameters P Ｃ The camera parameters P Ｃ is used as input, and the image IM Ｃ The "continuous volumetric scene function" is optimized by learning with the output "x" as the vector. The optimized "continuous volumetric scene function" is obtained as the image generation model MD.

[0026] Cinema camera commercial ＲBased on the camera parameters of the virtual camera CM Ｖ Camera parameters P Ｖ The image generation model MD and the virtual camera CM are obtained. Ｖ Camera parameters P Ｖ Based on this, the camera parameters P Ｖ Inference image IM of new viewpoint NV according to Ｇ is obtained. Ｇ After distortion correction is performed if necessary, the image is displayed on the large display WD in the studio.

[0027] [1-3. Issues in VP Imaging] FIGS. 6 and 7 are diagrams for explaining issues in VP imaging.

[0028] The generation of viewpoint images by NeRF is performed by generating an inference image IM according to the position of the new viewpoint NV and the type of subject SB. Ｇ This is because the quality of the image IM used for learning can vary. Ｃ The inventors' investigation has revealed that this is due to the shooting method and the characteristics of the subject SB.

[0029] For example, in the example of Figure 6, the area enclosed by the frame is distorted as if smoke is rising from the image. This is thought to be because this area is hidden behind other objects, and sufficient photographic data could not be obtained. In the example of Figure 7, the bottom of the pot is dull, and the image IM taken from the same viewpoint is Ｃ According to the inventor's investigation, it has been found that a glossy and reflective subject SB cannot be reproduced sufficiently during inference.

[0030] In order to capture high-quality images in VP photography, inference images IM Ｇ Camera work is required that takes into account the angle at which the image is displayed clearly. However, it is difficult for staff ST (studio directors, cameramen, etc.) who are unfamiliar with the characteristics of the image generation model MD to make such judgments. As a result, it may take time to adjust the camera work in the studio, or images may be taken from viewpoints of poor quality for the image generation model MD being used.

[0031] [2. Shooting Support Method of the Present Disclosure] The present disclosure has been made in consideration of the above-mentioned problems. In the present disclosure, the characteristics of the image generation model MD (such as inference quality for each viewpoint and inference quality according to subject characteristics) are analyzed in advance, and the analysis results are presented to the staff ST. This supports the design of camerawork that does not conspicuously affect poor quality.

[0032] Specifically, the shooting viewpoint SV of the image group used for learning the image generation model MD is presented, and the characteristics of the image generation model MD are analyzed to determine the quality of each viewpoint and the inferred image IM. Ｇ It is also possible to automatically calculate and suggest desirable camera work.

[0033] The above process makes it easier for the staff ST to understand the characteristics of the image generation model MD. Ｇ By understanding the quality of the image, the inference image IM Ｇ It becomes easier to design camera work that matches the inference image IM. Ｇ Avoid shooting from viewpoints that result in poor quality, and shoot from a viewpoint range with good quality. Ｒ We have taken measures such as moving the camera, and using a cinema camera commercial to prevent poor quality images from being captured. Ｒ It is expected that this will enable the camera to control the orientation and zoom of the camera, resulting in higher quality footage being captured in a shorter amount of time.

[0034] The above-described photographing support is performed by an information processing device such as a server. Fig. 8 is a diagram showing an example of an information processing device 1. The information processing device 1 includes, for example, an image generation unit 10, a quality analysis unit 20, a photographing support unit 30, and a storage unit 40.

[0035] The storage unit 40 stores learning data for the image generation model MD. The learning data includes a group of images of the target scene used for learning the optimization of the image generation model MD, and the camera parameters P when the group of images were captured. Ｃ The image group includes a plurality of images IM obtained by photographing the target scene from different photographing viewpoints SV by a real camera CM. ＣThe storage unit 40 stores the image IM Ｃ For each, the camera parameters P Ｃ and image IM Ｃ The camera parameters P are stored in association with the Ｃ is the pose information for each image IM Ｃ It includes information on the shooting viewpoint SV of the real camera CM linked to the image.

[0036] The image generation unit 10 generates an inference image IM using the image generation model MD. Ｇ The image generation model MD reconstructs the radiance field by volume rendering and infers the image of the new viewpoint NV. The new viewpoint NV means a viewpoint arbitrarily designated by the staff ST or a viewpoint automatically calculated by the system as a desired camera work. The new viewpoint NV is generated by using the camera parameters P Ｃ A viewpoint different from the shooting viewpoint SV of the actual camera CM shown in Ｃ It can be set as a perspective that is not tied to any

[0037] In the present disclosure, the image generation model MD is based on the "continuous volumetric scene function" used in NeRF. However, the image generation model MD is not limited to this. The image generation model MD can be based on the images IM acquired from multiple shooting viewpoints SV. Ｃ It is sufficient if the method is trained based on the above and reconstructs the radiance field to infer an image of a new viewpoint NV.

[0038] The quality analysis unit 20 calculates the inferred image IM based on accuracy factors that affect the inference accuracy of the image generation model MD. Ｇ The accuracy factor includes the position of the shooting viewpoint of the images used to train the image generation model MD and the optical characteristics of the subject SB captured in the images. The quality analysis unit 20 estimates the quality of the images included in the training data and the camera parameters P Ｃ The position information of the shooting viewpoint obtained from the is used for quality analysis. The analysis results are used as support information for shooting support.

[0039] For example, the camera parameter P Ｃprovides information about the range and density of the shooting viewpoints SV when the learning images were captured. For positions that are far outside the range of the shooting viewpoints SV, or positions where the density of the shooting viewpoints SV is low even within the range of the shooting viewpoints SV, the inference image IM Ｇ The quality of the inferred image IM is expected to be poor. The optical properties of the object SB provide information about the magnitude of scattering and reflection caused by the object SB. As shown in FIG. 7, for a glossy object SB, the quality of the inferred image IM Ｇ The quality of the image is predicted to be poor. Obtaining such quality predictions makes it easier to design appropriate camerawork.

[0040] The photographing support unit 30 generates an inference image IM Ｇ In the shooting of VP with the background image, the inference accuracy of the image generation model MD and the inference image IM Ｇ The information about at least one of the qualities is output to a display device as support information. The support information can be displayed on various display devices. Examples of the display device include a large display WD in the studio and various viewers VW. Cinema camera commercial Ｒ The viewfinder, preview monitor MT, or AR (Augmented Reality) glasses GL worn by the staff ST can also be treated as display devices that display support information. The shooting support unit 30 controls these display devices to display the support information.

[0041] 3. UI for Presenting the Quality of an Inferred Image of NeRF An example of support information will be described below. Ｇ 10 is a diagram illustrating an example of a UI (User Interface) that presents the quality of the image.

[0042] In the example of FIG. 9, the display device on which the support information is displayed is the inference image IM Ｇ The photographing support unit 30 displays the inference image IM in the viewer VW. Ｇ As a result, the photographing support unit 30 outputs support information (quality information Q) regarding the quality of the inference image IM on the viewer VW. ＧThe viewer VW is used to generate and check an image from an arbitrary viewpoint. The staff ST can generate an inference image IM from an arbitrary viewpoint, regardless of the image displayed on the large display WD. Ｇ can be generated and confirmed on the viewer VW.

[0043] For example, on the viewer VW, each image IM used in training the image generation model MD is displayed. Ｃ 9, the shooting viewpoints SV are imaged and displayed. In the example of FIG. 9, the shooting posture of the real camera CM is shown in each shooting viewpoint SV. On the viewer VW, a numerical value indicating the reproducibility of each shooting viewpoint SV is displayed as quality information Q. The reproducibility of the shooting viewpoint SV is the quality of the image IM of the real camera CM acquired at the shooting viewpoint SV. Ｃ is the inferred image IM inferred from the same viewpoint as the shooting viewpoint SV. Ｇ This means the degree to which the

[0044] The photographing support unit 30 generates an inference image IM Ｇ The imaging support unit 30 can also indicate ranges of good quality and ranges of bad quality within the inference image IM. Ｇ The photographing support unit 30 can display such markers, frames, and colors as the quality information Q on the display device.

[0045] Staff ST operates input devices such as a mouse to create a virtual camera commercial. Ｖ The viewpoint of the inference image IM displayed in the viewer VW can be changed. Ｇ changes according to the operation of the input device. The staff ST changes the viewpoint and Ｇ You can see where the quality is poor.

[0046] The quality information Q can also be displayed on a display device other than the viewer VW. For example, a large display WD that displays a background image can be used as a display device that displays the support information. The photography support unit 30 can output the quality information Q to the large display WD and display a UI showing the quality information Q on the large display WD. The photography support unit 30 can output the quality information Q to the large display WD as the quality information Q. Ｇ The quality of the inference image IM is displayed numerically. The display may be a display of the numerical value itself or a display such as a color bar indicating the numerical range. Ｇ If there are any areas of poor quality within the image, these areas may be displayed as rectangles or shaded.

[0047] The display device that displays the support information is a cinema camera commercial that shoots VPs. Ｒ The photographing support unit 30 can output the quality information Q to the viewfinder or preview monitor MT and display the quality information Q superimposed on the viewfinder or preview monitor MT.

[0048] The AR glasses GL worn by the director or cameraman who shoots the VP can also be used as a display device to display the support information. The shooting support unit 30 can output the quality information Q to the AR glasses GL and display the quality information Q on the AR glasses GL.

[0049] The quality information Q can be displayed as shown in Fig. 9. The schematic diagram of the shooting viewpoint SV and the position of the markers shown in Fig. 9 can be displayed as the inference image IM to be displayed. Ｇ viewpoint (Cinema camera commercial) Ｒ The viewpoint is adjusted according to the viewpoint of the AR glass GL.

[0050] A guide image indicating the range of camera angles at which the background image looks clear may be displayed as quality information Q for the AR glasses GL worn by the photographer. Fig. 10 is a diagram showing an example of a guide image GI indicating the range of appropriate camera angles. The AR glasses GL display markers MK around the staff member ST based on the quality information Q acquired from the photography support unit 30. The photography support unit 30 can indicate the photography range at which the background image looks clear based on the presence, size, color, etc. of the markers MK.

[0051] 4. Quality Analysis Method 4-1. Quality Prediction Based on the Density of Shooting Viewpoints An example of a quality analysis method will be described below. Fig. 11 is a diagram for explaining quality prediction based on the density of shooting viewpoints SV.

[0052] The quality analysis unit 20 performs a performance analysis of the NeRF algorithm in advance and calculates the inference image IM. Ｇ The analysis targets not the image generation model MD itself but the inference algorithm performed by the image generation model MD (performance determined by the structure of the "continuous volumetric scene function").

[0053] For example, the system developer prepares multiple sets of training data for evaluation that have different densities of shooting viewpoints SV. The system developer acquires multiple image generation models for evaluation trained with the different training data for evaluation, and generates inference images IM for each of the image generation models for evaluation. Ｇ The quality of the inference image IM is evaluated as the sparser the shooting viewpoint SV. Ｇ The quality of the image will be worse. Ｇ There is a correlation between the quality of the image and the density of the shooting viewpoints SV, and the relationship between the two can be quantified.

[0054] The storage unit 40 stores a group of images used in learning the image generation model MD, and the camera parameters P Ｃ The quality analysis unit 20 stores the camera parameters P ＣThe quality analysis unit 20 acquires the shooting viewpoint SV (density distribution of the shooting viewpoint SV) of the image group from the image group. Ｇ Based on the relationship with the quality of the image, an inferred image IM for each input viewpoint to the image generation model MD is generated from the shooting viewpoint SV of the image group. Ｇ The quality analysis unit 20 generates a quality prediction map QM indicating the prediction result as an inference image IM. Ｇ The quality information is acquired as support information (quality information Q) regarding the quality of the product.

[0055] The quality prediction map QM is a new viewpoint NV to be inferred and an inference image IM. Ｇ The quality of the inference is obtained as distribution data of the inference quality, which specifies the relationship between an index that quantifies the quality of the inference and the quality of the shooting viewpoints SV. The shooting support unit 30 compares the density of the shooting viewpoints SV of the target scene to be actually used with the index that was quantified in advance, and evaluates in stages whether the density of the shooting viewpoints SV is sufficient or insufficient. Based on the evaluation results, the shooting support unit 30 presents quality information Q as shown in FIG.

[0056] 4-2. Quality Prediction Based on the Range of the Shooting Viewpoint FIG. 12 is a diagram illustrating quality prediction based on the range of the shooting viewpoint SV.

[0057] Inference image IM for a viewpoint range with abundant learning data Ｇ On the other hand, if the viewpoint of the image group in the training data is far removed, the inferred image IM Ｇ The quality of the image group is significantly deteriorated. Ｃ The quality analysis unit 20 determines an appropriate range of the input viewpoint to be input to the image generation model MD based on the range of the shooting viewpoint SV acquired from the quality analysis unit 20. The quality analysis unit 20 acquires the determined appropriate range as quality information Q.

[0058] For example, the quality analysis unit 20 may calculate the camera parameters P Ｃ A viewpoint map VM indicating the range of the shooting viewpoint SV is generated from each image IM included in the image group. ＣThe quality analysis unit 20 acquires the coordinates of all the shooting viewpoints SV and determines the position of the center of gravity as the center of gravity viewpoint. The coordinates are set as coordinates within the inference space SP, for example.

[0059] The quality analysis unit 20 projects each shooting viewpoint onto the xy plane and calculates the shooting viewpoint SV that is farthest from the center of gravity viewpoint in the +x direction, -x direction, +y direction, and -y direction. The quality analysis unit 20 determines that the area outside the shooting viewpoint SV that is farthest from the center of gravity viewpoint is outside the shooting range. The quality analysis unit 20 projects each shooting viewpoint onto the yz plane and the zx plane and performs similar processing. As a result, a cube-shaped range is acquired as the range of the shooting viewpoint SV. The quality analysis unit 20 determines the acquired cube-shaped range as the appropriate range. The shooting support unit 30 can display the appropriate range on a display device using a marker MK shown in FIG. 10 or the like.

[0060] 4-3. Quality Prediction Based on Image Reproducibility for Each Shooting Viewpoint FIG. 13 is a diagram illustrating quality prediction based on image reproducibility for each shooting viewpoint SV.

[0061] As shown in FIG. Ｇ The quality of the image is affected by the optical characteristics of the subject SB. Depending on the subject SB, sufficient reproducibility may not be obtained during inference due to the influence of gloss, etc. Ｃ is calculated using a method such as PnP, but depending on the shooting viewpoint SV, accurate calculation results may not be obtained, and sufficient reproducibility may not be obtained during inference. Ｃ Inference is performed on the image from the same perspective as the original image IM Ｃ and inference image IM Ｇ Reproducibility can be quantified by quantifying the difference between the

[0062] For example, the quality analysis unit 20 may use the camera parameter P Ｃ From the image IM used to train the image generation model MD, Ｃ The quality analysis unit 20 acquires the image IM of the shooting viewpoint SV for each shooting viewpoint SV used to shoot the image group. ＣThe inferred image IM is inferred from the same viewpoint as the shooting viewpoint SV. Ｇ The quality analysis unit 20 compares the image IM based on the comparison result. Ｃ The quality analysis unit 20 acquires the detected reproducibility as quality information Q at the shooting viewpoint SV. The shooting support unit 30 can display subjects SB with low reproducibility on the display device using markers or colors as shown in FIG.

[0063] The reproducibility of the subject SB also depends on the number of viewpoints from which the subject SB is captured. The quality of areas with few viewpoints in the learning process, such as the edges of the image or areas with occlusions, tends to be poor. Blurring often occurs in such areas. To address this issue, volume rendering can be used to generate the inferred image IM. Ｇ When generating the coordinate point CP used in the calculation, the number of shooting viewpoints SV from which the coordinate point CP is shot is analyzed and quantified.

[0064] 14 to 16 are diagrams for explaining quality prediction based on the number of shooting viewpoints for each coordinate point CP.

[0065] The quality analysis unit 20 counts the number of shooting viewpoints SV from which images of the coordinate points CP are captured for each coordinate point CP in the inference space SP, and acquires the distribution of the number of shooting viewpoints in the inference space SP as quality information Q. The shooting support unit 30 can display subjects SB with low reproducibility on the display device using markers or coloring as shown in Fig. 9.

[0066] In the example of Fig. 14, subject A is photographed from seven shooting viewpoints SV. The coordinate point CP of subject A can be seen from the four central shooting viewpoints SV. From the leftmost shooting viewpoint SV, the coordinate point CP is outside the angle of view of the real camera CM, so it is not captured. From the two shooting viewpoints SV on the right, the coordinate point CP is also not captured because it is blocked by subject B in the foreground. Therefore, the number of shooting viewpoints SV from which the coordinate point CP is captured is four.

[0067] Similarly, in the example of FIG. 15, the coordinate point CP Ａ The number of shooting viewpoints SV for shooting is two, and the coordinate point CP ＢThe number of shooting viewpoints SV from which the subject is photographed is five. The example in Fig. 16 shows the relationship between the distance from the center-of-gravity viewpoint and the number of shooting viewpoints. The number of shooting viewpoints SV for a central subject that is close to the center-of-gravity viewpoint is large. The number of shooting viewpoints SV for a subject at an end that is far from the center-of-gravity viewpoint is small.

[0068] The quality analysis unit 20 calculates the number of shooting viewpoints and the inference image IM Ｇ The relationship between the quality of the inferred image IM is analyzed. Ｇ The photographing support unit 30 can display the quantified quality on the display device using the numerical values, markers, or colors shown in FIG.

[0069] 5. Example of Hardware Configuration FIG. 17 is a diagram illustrating an example of the hardware configuration of the information processing device 1. As shown in FIG.

[0070] The information processing of the information processing device 1 is realized by, for example, a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0071] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0072] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0073] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0074] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0075] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0076] For example, when the computer 1000 functions as the information processing device 1 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may acquire these programs from another device via an external network 1550.

[0077] [6. Effects] The information processing device 1 has an image generation model MD, a quality analysis unit 20, and a photography support unit 30. The image generation model MD reconstructs a radiance field to infer an image of a new viewpoint NV. The quality analysis unit 20 calculates an inferred image IM based on accuracy factors that affect the inference accuracy. Ｇ The photographing support unit 30 estimates the quality of the inference image IM Ｇ In VP photography with the background image, the inference accuracy and the inference image IM Ｇ and outputs support information relating to at least one of the qualities of the above to a display device. In the information processing method disclosed herein, the processing of the information processing device 1 is executed by a computer 1000. The computer-readable non-transitory storage medium disclosed herein stores a program that causes the computer 1000 to realize the processing of the information processing device 1.

[0078] According to this configuration, the inference accuracy of the image generation model MD or the inference image IM Ｇ Based on the quality of the image generation model MD, shooting support is provided to achieve appropriate camerawork. Therefore, even studio directors and cameramen who are unfamiliar with the characteristics of the image generation model MD can achieve good shooting results.

[0079] The accuracy factor includes the position of the viewpoint of the images used to train the image generation model MD.

[0080] According to this configuration, quality information is presented according to the details of the learning.

[0081] The quality analysis unit 20 calculates the density of the shooting viewpoints SV acquired in advance and the inference image IM. ＧBased on the relationship with the quality of the image, an inferred image IM for each input viewpoint to the image generation model MD is generated from the shooting viewpoint SV of the image group. Ｇ The quality analysis unit 20 obtains a quality prediction map QM indicating the predicted results as support information.

[0082] This configuration provides an appropriate quality prediction according to the shooting viewpoint SV during learning, making it easier to select appropriate camera work.

[0083] The quality analysis unit 20 calculates the camera parameters P Ｃ The quality analysis unit 20 determines an appropriate range of the input viewpoint to be input to the image generation model MD based on the range of the shooting viewpoint SV acquired from the image generation model MD. The quality analysis unit 20 acquires the determined appropriate range as support information.

[0084] According to this configuration, high-quality images can be captured by capturing images within the appropriate range.

[0085] The quality analysis unit 20 analyzes the image of the shooting viewpoint SV used to shoot the image group by dividing the image of the shooting viewpoint SV into an inferred image IM inferred from the same viewpoint as the shooting viewpoint SV. Ｇ The quality analysis unit 20 detects the reproducibility of the entire image or each location on the basis of the comparison result. The quality analysis unit 20 compares the detected reproducibility with the inference image IM at the shooting viewpoint SV. Ｇ This information is obtained as information about the quality of the product.

[0086] According to this configuration, camerawork that can accurately reproduce the subject SB can be recognized.

[0087] The quality analysis unit 20 counts the number of shooting viewpoints SV from which the coordinate points CP are captured for each coordinate point CP in the space to be inferred (the inference space SP). The quality analysis unit 20 acquires the distribution of the number of shooting viewpoints in the inference space SP as support information.

[0088] According to this configuration, the inference image IM is generated based on the number of shooting viewpoints SV. Ｇ By selecting camerawork that increases the number of shooting viewpoints SV, high-quality images can be captured.

[0089] For example, the display device may display an inference image IM ＧThe photographing support unit 30 displays the inference image IM in the viewer VW. Ｇ The support information regarding the quality of the inference image IM is output on the viewer VW. Ｇ At the same time, a UI showing support information is displayed.

[0090] According to this configuration, appropriate camera work can be recognized on the viewer VW.

[0091] For example, the display device is a large display WD that displays a background image. The photographing support unit 30 displays an inference image IM on the large display WD. Ｇ The support information relating to the quality of the image is output, and a UI showing the support information is displayed on the large display WD.

[0092] According to this configuration, it is possible to select appropriate camera work while directly viewing the background image to be photographed.

[0093] For example, the display device may be a cinema camera for filming a VP. Ｒ The photographing support unit 30 displays the inference image IM on the viewfinder or preview monitor MT. Ｇ The support information relating to the quality of the image is output and displayed superimposed on the viewfinder or the preview monitor MT.

[0094] This configuration allows the cameraman to recognize appropriate camera work.

[0095] For example, the display device is an AR glass GL worn by a director or a cameraman who shoots a video projection. The shooting support unit 30 displays an inference image IM on the AR glass GL. Ｇ and displays the support information on the AR glass GL.

[0096] This configuration allows the director and cameraman to recognize appropriate camera work.

[0097] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0098] [Additional Notes] The present technology can also be configured as follows. (1) An information processing device comprising: an image generation model that reconstructs a radiance field to infer an image from a new viewpoint; a quality analysis unit that estimates the quality of the inferred image based on an accuracy factor that affects the inference accuracy; and a shooting support unit that outputs support information related to at least one of the inference accuracy and the quality of the inferred image to a display device in shooting virtual production using the inferred image as a background image. (2) The information processing device described in (1) above, wherein the accuracy factor includes a position of a shooting viewpoint of a group of images used to train the image generation model. (3) The information processing device described in (2) above, wherein the quality analysis unit predicts the quality of the inferred image for each input viewpoint to the image generation model from the shooting viewpoint of the group of images based on a relationship between the density of shooting viewpoints acquired in advance and the quality of the inferred image, and obtains a quality prediction map showing the predicted results as the support information related to the quality of the inferred image. (4) The information processing device according to (2) or (3), wherein the quality analysis unit determines an appropriate range of input viewpoints to be input to the image generation model based on a range of shooting viewpoints acquired from camera parameters of the group of images, and acquires the determined appropriate range as the support information for the quality of the inferred images. (5) The information processing device according to any one of (2) to (4), wherein the quality analysis unit, for each shooting viewpoint used to capture the group of images, compares the image from the shooting viewpoint with the inferred image inferred from the same viewpoint as the shooting viewpoint, detects reproducibility of the image as a whole or for each location based on the comparison result, and acquires the detected reproducibility as information on the quality of the inferred image at the shooting viewpoint. (6) The information processing device according to any one of (2) to (5), wherein the quality analysis unit tallies the number of shooting viewpoints from which the coordinate point in a space to be inferred was captured, and acquires a distribution of the number of shooting viewpoints within the space as the support information for the quality of the inferred images.(7) The information processing device according to any one of (1) to (6), wherein the display device is a viewer for previewing the inference image, and the shooting support unit outputs the support information regarding the quality of the inference image to the viewer and causes a UI indicating the support information to be displayed on the viewer together with the inference image. (8) The information processing device according to any one of (1) to (6), wherein the display device is a large display that displays the background image, and the shooting support unit outputs the support information regarding the quality of the inference image to the large display and causes a UI indicating the support information to be displayed on the large display. (9) The information processing device according to any one of (1) to (6), wherein the display device is a viewfinder or a preview monitor of a cinema camera that shoots the virtual production, and the shooting support unit outputs the support information regarding the quality of the inference image to the viewfinder or the preview monitor and causes the support information to be superimposed and displayed on the viewfinder or the preview monitor. (10) The information processing device according to any one of (1) to (6), wherein the display device is AR glasses worn by a director or cameraman shooting the virtual production, and the shooting support unit outputs the support information related to the quality of the inference image to the AR glasses and causes the AR glasses to display the support information. (11) An information processing method executed by a computer, comprising: reconstructing a radiance field to infer an image from a new viewpoint, estimating the quality of the inference image based on an accuracy factor that affects the inference accuracy, and outputting support information related to at least one of the inference accuracy and the quality of the inference image to a display device in shooting a virtual production using the inference image as a background image.(12) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: reconstruct a radiance field to infer an image from a new viewpoint; estimate the quality of the inferred image based on accuracy factors that affect the inference accuracy; and output support information regarding at least one of the inference accuracy and the quality of the inferred image to a display device when shooting virtual production using the inferred image as a background image.

[0099] 1 Information processing device 20 Quality analysis unit 30 Filming support unit CM Ｒ Cinema camera CP Coordinate point GL AR glasses MD Image generation model IM Ｃ Image IM Ｇ Inference image MT Preview monitor P Ｃ Camera parameters QM Quality prediction map SB Subject SV Shooting viewpoint VW Viewer WD Large display

Claims

1. An information processing device having an image generation model that reconstructs a radiance field to infer an image from a new viewpoint; a quality analysis unit that estimates the quality of the inferred image based on accuracy factors that affect the inference accuracy; and a shooting support unit that outputs support information regarding at least one of the inference accuracy and the quality of the inferred image to a display device in shooting virtual production in which the inferred image is used as a background image.

2. The information processing device according to claim 1, wherein the accuracy factor includes a position of a shooting viewpoint of a group of images used in training the image generation model.

3. The information processing device described in claim 2, wherein the quality analysis unit predicts the quality of the inferred image for each input viewpoint to the image generation model from the shooting viewpoint of the image group based on the relationship between the density of shooting viewpoints acquired in advance and the quality of the inferred image, and obtains a quality prediction map showing the predicted results as the support information regarding the quality of the inferred image.

4. The information processing device of claim 2, wherein the quality analysis unit determines an appropriate range of input viewpoints to be input to the image generation model based on the range of shooting viewpoints obtained from the camera parameters of the image group, and obtains the determined appropriate range as the support information regarding the quality of the inference image.

5. The information processing device described in claim 2, wherein the quality analysis unit compares, for each shooting viewpoint used to capture the group of images, the image from the shooting viewpoint with the inferred image inferred from the same viewpoint as the shooting viewpoint, detects the reproducibility of the entire image or each location based on the comparison result, and obtains the detected reproducibility as information regarding the quality of the inferred image at the shooting viewpoint.

6. The information processing device according to claim 2, wherein the quality analysis unit counts the number of shooting viewpoints from which each coordinate point in the space to be inferred was photographed, and obtains a distribution of the number of shooting viewpoints within the space as the support information regarding the quality of the inference image.

7. The information processing device according to claim 1, wherein the display device is a viewer for previewing the inference image, and the shooting support unit outputs the support information regarding the quality of the inference image to the viewer and displays a UI showing the support information together with the inference image on the viewer.

8. The information processing device of claim 1, wherein the display device is a large display that displays the background image, and the shooting support unit outputs the support information regarding the quality of the inference image to the large display and displays a UI showing the support information on the large display.

9. The information processing device according to claim 1, wherein the display device is a viewfinder or preview monitor of a cinema camera used to shoot the virtual production, and the shooting support unit outputs the support information regarding the quality of the inference image to the viewfinder or preview monitor and displays the support information superimposed on the viewfinder or preview monitor.

10. The information processing device according to claim 1, wherein the display device is AR glasses worn by a director or cameraman who shoots the virtual production, and the shooting support unit outputs the support information regarding the quality of the inference image to the AR glasses and causes the AR glasses to display the support information.

11. An information processing method executed by a computer, comprising: reconstructing a radiance field to infer an image of a new viewpoint; estimating the quality of the inferred image based on an accuracy factor that affects the inference accuracy; and outputting support information regarding at least one of the inference accuracy and the quality of the inferred image to a display device in shooting virtual production in which the inferred image is used as a background image.

12. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following steps: reconstruct a radiance field to infer an image from a new viewpoint; estimate the quality of the inferred image based on an accuracy factor that affects the inference accuracy; and output support information regarding at least one of the inference accuracy and the quality of the inferred image to a display device when shooting virtual production in which the inferred image is used as a background image.

Images