Free-viewpoint image generation device, method, and program

Abstract

In an embodiment of the present invention, when parameter information including the size and the viewpoint position of a free-viewpoint image to be rendered is input, a line-of-sight direction vector passing from the viewpoint position through a plurality of pixels of image data in a rendering space stored in advance is generated for each of the pixels. Next, for each of the line-of-sight direction vectors, a correction value for the line-of-sight direction vector is derived on the basis of a vector represented by the viewpoint position, the line-of-sight direction vector, and the distance from the viewpoint position, and values representing radiance fields at a plurality of sample points set on the line-of-sight direction vector are calculated on the basis of the distances from the viewpoint position to the sample points and the correction values of the line-of-sight direction vectors. In addition, the colors of pixels corresponding to the line-of-sight direction vectors are estimated by aggregating the values representing the radiance fields at the plurality of calculated sample points, and a free-viewpoint image is generated on the basis of the position coordinates of the plurality of pixels and the estimated values of the colors.

Description

Free viewpoint image generation device, method, and program

[0001] One aspect of the present invention relates to a free viewpoint image generation device, method, and program for generating an image from any different viewpoint based on a group of images captured from a plurality of viewpoints.

[0002] One known technique for generating an image from a different arbitrary viewpoint based on a group of images captured from multiple viewpoints is NeRF (Neural Radiance Fields), described in Non-Patent Document 1. NeRF is a technique in which an area including a drawing target, called a drawing space, is represented by a rectangular parallelepiped, a sphere, or the like, and the luminance field (color distribution and density) within the drawing space is modeled using a neural network.

[0003] In NeRF, for example, the rendering space is represented by a sphere with center coordinates (0,0,0) and radius Rw. Then, camera parameters are specified when rendering, and the image is calculated for each pixel based on the luminance field. More specifically, as shown in Figure 7, for example, a projection function K, a rotation matrix A, a translation vector o, and a pixel coordinate p of the image are specified as camera parameters, and a line from the translation vector o passing through the pixel coordinate p of the image is taken as the line of sight, and the direction vector d of this line of sight is calculated. Then, a starting point t is placed on the line of sight. ｎ and the end point t ｆ and calculates an estimate of the color C(p) of pixel coordinate p by aggregating the luminance field. The luminance field is represented by a color distribution and a density, which are respectively calculated using a neural network NN. ｃ , N.N. σ The neural network is trained in advance based on a set of images with known camera parameters.

[0004] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi and Ren Ng. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, ECCV, 2020, LNCS 12346, pp. 405-421, 2020, Internet <https: / / doi.org / 10.1007 / 978-3-030-58452-8 24>

[0005] However, NeRF requires not only images but also camera parameters. Because the camera parameters are estimated from the images, they may contain errors. If there is an error in the camera parameters, this error is reflected in the line-of-sight direction vector d, and the position of the object in the image is shifted due to the influence of the error in the line-of-sight direction vector d. This reduces the resolution of the generated free-viewpoint image, resulting in a blurred image. The reduction in image resolution becomes greater the longer the distance to the object, i.e., the more distant the object.

[0006] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a technique for reducing errors in the line of sight direction and improving the resolution of free viewpoint images.

[0007] In order to solve the above problem, one aspect of the free viewpoint image generating device or method according to the present invention performs a free viewpoint image generating process as follows.

[0008] That is, first, a plurality of image data captured from a plurality of viewpoints in a rendering space is acquired and stored. Then, when parameter information including the size and viewpoint position of a free-viewpoint image to be rendered is input, a gaze direction vector passing through each of the plurality of pixels in the image data from the viewpoint position included in the parameter information is generated for each of the plurality of pixels. Next, for each of the gaze direction vectors, a correction value for the gaze direction vector is calculated based on a vector represented by the viewpoint position, the gaze direction vector, and a distance from the viewpoint position. Values ​​representing luminance fields at a plurality of sample points set on the gaze direction vector are calculated based on the distance from the viewpoint position to the sample point and the correction value for the gaze direction vector. Then, for each of the gaze direction vectors, the calculated values ​​representing luminance fields at the plurality of sample points are aggregated to estimate the color of the pixel corresponding to the gaze direction vector. A free-viewpoint image is generated based on the position coordinates of the plurality of pixels and the estimated color values.

[0009] According to one aspect of the present invention, even if the parameters contain errors, the influence of the errors can be reduced by correcting the gaze direction, thereby making it possible to generate high-resolution free-viewpoint images. This effect becomes more pronounced when there is a mixture of images in which the object is close and images in which the object is far away, and when there is a mixture of images in which the gaze directions to the object are various.

[0010] That is, according to one aspect of the present invention, it is possible to provide a technique for reducing errors in the line of sight direction and improving the resolution of a free viewpoint image.

[0011] FIG. 1 is a block diagram showing an example of a hardware configuration of a free viewpoint image generation device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a software configuration of a free viewpoint image generation device according to an embodiment of the present invention. FIG. 3 is a flowchart showing an example of a processing procedure and processing content of a neural network learning process executed by a control unit of the free viewpoint image generation device shown in FIG. 2 in a learning phase. FIG. 4 is a flowchart showing an example of a processing procedure and processing content of a free viewpoint image generation process executed by a control unit of the free viewpoint image generation device shown in FIG. 2 in a drawing phase. FIG. 5 is a diagram for explaining an example of an error occurring in a gaze direction vector. FIG. 6A is a diagram for explaining another example of an error occurring in a gaze direction vector. FIG. 6B is a diagram for explaining another example of an error occurring in a gaze direction vector. FIG. 7 is a diagram for explaining an example of a free viewpoint image generation method using NeRF.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One Embodiment] (Configuration Example) FIGS. 1 and 2 are block diagrams showing an example of the hardware configuration and software configuration, respectively, of a free viewpoint image generating device according to one embodiment of the present invention.

[0014] The free viewpoint image generation device 1 according to an embodiment is configured by, for example, a personal computer. Note that the free viewpoint image generation device 1 may also be configured by a server computer installed on the web or on the cloud.

[0015] The free viewpoint image generation device 1 includes a control unit 10 that uses a hardware processor such as a central processing unit (CPU), and a storage unit having a program storage unit 20 and a data storage unit 30, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 40 are connected to the control unit 10 via a bus 50.

[0016] An input device 2 and a display device 3 are connected to the input / output I / F unit 40. The input device 2 is composed of, for example, a keyboard and a mouse, and is mainly used to input control commands and various parameters. The input parameters will be described later. The display device 3 is used to display information related to various operations of the device and information necessary for input operations, as well as to display a free viewpoint image generated by the control unit 10.

[0017] Furthermore, in order to input image data to be used for learning and drawing, for example, multiple cameras (not shown) arranged in the drawing space are connected to the input / output I / F unit 40. Note that the image data in the drawing space may be acquired from an external storage device, or may be acquired via a network from another terminal device or a server computer on the web or cloud.

[0018] The program storage unit 20 is configured by combining, for example, a nonvolatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various controls according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0019] The data storage unit 30 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and the storage area is provided with a parameter storage unit 31, an image storage unit 32, and a learning model storage unit 33 as main storage units related to one embodiment of the present invention.

[0020] The parameter storage unit 31 is used to store control parameters used in the process of generating a free viewpoint image and camera parameters for rendering.

[0021] The image storage unit 32 is used to store image data groups for learning and drawing. The image data groups are made up of a plurality of image data captured from a plurality of viewpoints in a drawing space.

[0022] The learning model storage unit 33 stores learning models for estimating the color distribution and density of the luminance field, and a learning model for gaze correction. Each learning model is created by training a neural network in advance.

[0023] The control unit 10 has the processing functions necessary to implement one embodiment, including an image generation control processing unit 11, an input information acquisition processing unit 12, an output information output processing unit 13, a luminance field aggregation processing unit 14, a line of sight correction processing unit 15, and a luminance field calculation processing unit 16.

[0024] Each of the processing units 11 to 16 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 20. Note that some or all of the processing units 11 to 16 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0025] The image generation control processing unit 11 executes neural network learning processing and free viewpoint image generation processing by controlling the luminance field aggregation processing unit 14, the line of sight correction processing unit 15, and the luminance field calculation processing unit 16, which will be described later. An example of these processes will be described in detail in the operation example.

[0026] The input information acquisition processing unit 12 acquires the control commands and various parameters inputted to the input device 2 via the input / output I / F unit 40, and stores the parameters in the parameter storage unit 31. The input information acquisition processing unit 12 also acquires a group of image data of the drawing space inputted from a camera or an external storage device (not shown) via the input / output I / F unit 40, and stores the group of image data in the image storage unit 32.

[0027] The output information output processing unit 13 outputs the free viewpoint image data generated in the control unit 10 from the input / output I / F unit 40 to the display device 3 .

[0028] During the learning phase, under the control of the image generation control processing unit 11, the luminance field aggregation processing unit 14 generates gaze samples based on the position coordinates of pixels in the given image data, and for each generated gaze sample, aggregates values ​​representing the color distribution and density of the luminance field calculated by the luminance field calculation processing unit 16 to estimate the color of the pixel.

[0029] Furthermore, in the drawing phase, the luminance field aggregation processing unit 14 generates gaze samples for each pixel of each image data based on camera parameters including the viewpoint of the drawing target under the control of the image generation control processing unit 11. Then, for each of the generated gaze samples, values ​​representing the color distribution and density of the luminance field calculated by the luminance field calculation processing unit 16 are aggregated to estimate the color of the pixel.

[0030] The line-of-sight correction processing unit 15 calculates a line-of-sight correction value for a plurality of sample points on each line-of-sight sample generated by the luminance field aggregation processing unit 14. A learning model is used to calculate the line-of-sight correction value.

[0031] The luminance field calculation processing unit 16 calculates the color distribution and density of the luminance field for a plurality of sample points on the line-of-sight sample, reflecting the line-of-sight correction value calculated by the line-of-sight correction processing unit 15. A learning model is used to calculate the color distribution and density.

[0032] (Example of Operation) Next, an example of operation of the free viewpoint image generation device 1 configured as above will be described.

[0033] (1) Learning Phase FIG. 3 is a flowchart showing the procedure and content of the neural network learning process executed by the control unit 10 of the free viewpoint image generation device 1 in the learning phase.

[0034] In the following description, it is assumed that the learning model memory unit 33 pre-stores data representing a neural network for creating a learning model that estimates the color distribution and density of the luminance field, and data representing a neural network for creating a learning model for gaze correction.

[0035] (1-1) Input of Learning Data When a learning phase is designated by inputting a control command, the control unit 10 of the free viewpoint image generation device 1, under the control of the input information acquisition processing unit 12, first, in step S10, acquires a plurality of image data I obtained by capturing images of the drawing space from a plurality of viewpoints (M locations) using a plurality of cameras or external storage devices, etc., arranged in the drawing space. m (m=1, . . . , M) is acquired via the input / output I / F unit 40, and the acquired image data I m is stored in the image storage unit 32.

[0036] Next, in step S11, the input information acquisition processing unit 12 acquires the image data I m The known camera parameters corresponding to the projection function K are acquired via the input / output I / F unit 40 and stored in the parameter storage unit 31. m , rotation matrix A m , translation vector o m Includes:

[0037] The input information acquisition processing unit 12 also acquires control parameters input by, for example, a system administrator via the input device 2 via the input / output I / F unit 40. The acquired control parameters are then stored in the parameter storage unit 31. The control parameters include the radius R of the drawing space, w The number of model learning steps S and the number of sample points N set on the line-of-sight sample are specified.

[0038] (1-2) Generation of Gaze Samples Next, under the control of the image generation control processing unit 11, the control unit 10 of the free viewpoint image generation device 1 first randomly selects one piece of image data from the image storage unit 32 in step S13, and then randomly selects one pixel from the selected image data in step S14. The image generation control processing unit 11 then provides the coordinates p of the pixel of the selected image data to the luminance field aggregation processing unit 14 together with the camera parameters stored in the parameter storage unit 31.

[0039] In step S15, based on the given coordinates p of the pixel, the projection function K of the camera parameters, the rotation matrix A, and the translation vector o, the luminance field aggregation processing unit 14 generates a line of sight sample of a straight line passing from the viewpoint o represented by the translation vector to the coordinates P of the pixel. The line of sight sample is represented as a direction vector d, and the direction vector d is expressed by the following equation:

[0040]

[0041] (1-3) Gaze Correction Next, in step S17, under the control of the gaze correction processing unit 15, the control unit 10 of the free viewpoint image generation device 1 performs a process of correcting the gaze of the above gaze sample using a neural network for gaze correction stored in the learning model memory unit 33.

[0042] That is, the line-of-sight correction processing unit 15 first calculates the line of sight o, the line-of-sight direction vector d, and the distance ∥r i A vector u with the elements of i is generated as follows, where # represents the concatenation of vectors.

[0043]

[0044] Next, the line of sight correction processing unit 15 calculates the vector u i Neural network NN for gaze correction ａ Neural network NN for gaze correction ａ For example, as shown below, the neural network has one hidden layer and the activation function is ReLU. Note that various neural network configurations and learning methods are known, and detailed explanations will be omitted here.

[0045]

[0046] where η i is the line of sight correction value, w １ , b １ , w ２ , b ２ is the weight of the training target, and ReLU is expressed as follows:

[0047]

[0048] The line of sight correction processing unit 15 is the neural network NN ａ The line of sight correction value η i is provided to the luminance field calculation processing unit 16.

[0049] (1-3) Calculation of Luminance Field Under the control of the luminance field calculation processing unit 16, in step S18, the line of sight correction value η i and the viewpoint o of the line of sight sample, the line of sight direction vector d, and the radius R of the drawing space. w Based on this, a neural network NN ｃ , N.N. σ Using this, we estimate the color distribution and density of the luminance field as follows:

[0050] That is, the luminance field calculation processing unit 16 first calculates the starting point t ｎ , end point t ｆ is calculated as follows:

[0051]

[0052] At the same time, the luminance field calculation processing unit 16 calculates the luminance field at each sample point t ｉ (i=1, . . . , N) are sampled so as to satisfy the following relationship:

[0053]

[0054] Then, the luminance field calculation processing unit 16 calculates the luminance field at each point t ｉ , the value of the luminance field, i.e., the color distribution c ｉ , density σ ｉ , distance r from the viewpoint ｉ , the next pixel t ｉ＋１Interval δ ｉ is calculated as follows:

[0055]

[0056] The line-of-sight correction processing unit 15 and the luminance field calculation processing unit 16 calculate the line-of-sight sample points t ｉ (i=1, . . . , N), the line of sight correction value η i and the estimation process of the color distribution c of the luminance field in step S18. ｉ and density σ ｉ Then, when the series of estimation processes described above for all N sample points is completed, the process moves from step S16 to step S19.

[0057] In step S19, the luminance field aggregation processing unit 14 calculates the color distribution c of the luminance field at all N sample points estimated by the luminance field calculation processing unit 16. ｉ and density σ ｉ are aggregated as follows to calculate the estimated value C^(p) of the color of the pixel corresponding to the gaze sample. As a result, the gaze correction value η i An estimate C^(p) of the color of the pixel reflected by the pixel is obtained.

[0058]

[0059] Then, the luminance field aggregation processing unit 14 passes the calculated estimated value C^(p) of the color of the pixel corresponding to the gaze sample to the image generation control processing unit 11.

[0060] In step S20, the image generation control processing unit 11 calculates the difference value between the corrected color of the pixel calculated by the luminance field summarization processing unit 14 and the original color before correction as follows.

[0061]

[0062] Then, based on the calculated difference value of the color of the pixel, the image generation control processing unit 11 adjusts the color distribution of the luminance field, the density, and each neural network NN for line of sight correction so as to reduce the difference value. ｃ , N.N. σ , N.N. ａ Learn.

[0063] Each neural network NN based on the color of the selected pixel ｃ , N.N. σ , N.N. ａ When the learning process is completed, the control unit 10 of the free viewpoint image generation device 1 randomly selects an unselected pixel of the selected image data or a pixel of another image data in steps S13 and S14, and generates a gaze sample passing through the selected pixel. Then, a series of processes in steps S16 to S20 are performed on the gaze sample, and each neural network NN ｃ , N.N. σ , N.N. ａ Learn about the following.

[0064] The above learning process is repeatedly executed until the number of steps S set in advance is reached. When the number of times the learning process has been executed reaches the number of steps S, the control unit 10 of the free viewpoint image generation device 1 performs the following steps: ｃ , N.N. σ , N.N. ａ The learning process for the neural network NN is completed, and the process proceeds to step S21. ｃ , N.N. σ , N.N. ａ The model parameters are stored in the learning model storage unit 33 as learned model data.

[0065] (2) Drawing Phase FIG. 4 is a flowchart showing an example of the processing procedure and processing content of the free viewpoint image generation processing executed by the control unit 10 of the free viewpoint image generation device 1 in the drawing phase.

[0066] (2-1) Input of parameters When image data corresponding to an arbitrary viewpoint in the drawing space is generated by the free viewpoint image generation device 1, the user inputs, for example, on the input device 2, the camera parameters corresponding to the arbitrary viewpoint, namely, the projection function K, the rotation matrix A, and the translation vector o, and further inputs the width W and height H, which are parameters that represent the size of the image to be generated.

[0067] In response to this, the control unit 10 of the free viewpoint image generation device 1, under the control of the input information acquisition processing unit 12, acquires the camera parameters and image size parameters input by the input device 2 via the input / output I / F unit 40 in step S30.

[0068] (2-2) Generation of gaze samples The control unit 10 of the free viewpoint image generation device 1 then, under the control of the image generation control processing unit 11, gives an instruction to the luminance field aggregation processing unit 14 to generate gaze samples based on the above-mentioned input camera parameters and image size parameters.

[0069] Upon receiving the generation instruction, in step S32, the luminance field aggregation processing unit 14 generates gaze samples that pass through the pixels from the translation vector o representing the viewpoint for all pixels (W × H) of the image defined by the image size parameters width W and height H. The gaze direction vector d representing the gaze sample is expressed by Equation 1 described in the learning phase.

[0070] (2-3) Line-of-sight correction Next, in step S34, under the control of the line-of-sight correction processing unit 15, the control unit 10 of the free viewpoint image generation device 1 calculates a line-of-sight correction value η i The process of estimating is carried out as follows:

[0071] That is, the line-of-sight correction processing unit 15 first calculates the line-of-sight sample by calculating the viewpoint o, the line-of-sight direction vector d, and the distance ∥r i A vector u with the elements of i Then, the generated vector u i Neural network NN for gaze correction ａ As a result, the neural network NN for gaze correction is ａ From the above vector u i The corresponding line of sight correction value η i is estimated and output.

[0072] (2-4) Calculation of Luminance Field Subsequently, in step S35, under the control of the luminance field calculation processing unit 16, the control unit 10 of the free viewpoint image generation device 1 calculates the line-of-sight correction value η i The color distribution c of the luminance field reflecting ｉ and density σ ｉ The process of estimating is performed as follows.

[0073] That is, the line of sight correction value η estimated by the line of sight correction processing unit 15 i and the viewpoint o of the gaze sample, the gaze direction vector d, and the radius Rw of the drawing space. The trained neural network NN ｃ , N.N. σ Then, the trained neural network NN ｃ , N.N. σ From the above, the line of sight correction value η i The color distribution c of the luminance field reflecting ｉ and density σ ｉ Get.

[0074] The control unit 10 of the free viewpoint image generating device 1 calculates the line-of-sight correction value η for the line-of-sight sample. i The estimation process and the correction value η i The color distribution c of the luminance field reflecting ｉ and density σ ｉ The process of estimating the line of sight correction value η is repeatedly executed for a predetermined number of samples N. i and the color distribution c of the luminance field ｉ and density σ ｉ When the estimation process is completed, the control unit 10 of the free viewpoint image generation apparatus 1 proceeds from step S33 to step S36.

[0075] (2-4) Calculation of Pixel Color In step S36, under the control of the luminance field aggregation processing unit 14, the control unit 10 of the free viewpoint image generation device 1 calculates the color distribution c of the luminance field at each sample point on the line of sight sample calculated by the luminance field calculation processing unit 16. ｉ and density σ ｉare aggregated to calculate an estimate C^(p) of the color of the pixel corresponding to that gaze sample.

[0076] (2-5) Image Generation The control unit 10 of the free viewpoint image generation device 1 calculates the line-of-sight correction value η for each pixel in steps S32 to S36 described above. i The process of estimating the color after reflecting the above is repeatedly executed for all pixels. Then, when the process of estimating the color for all pixels is completed, the process moves from step S31 to step S37.

[0077] In step S37, under the control of the image generation control processing unit 11, the control unit 10 of the free viewpoint image generation device 1 generates image data corresponding to the specified viewpoint o based on the estimated color values ​​C^(p) of all pixels calculated by the luminance field aggregation processing unit 14. Then, under the control of the output information output processing unit 13, the control unit 10 outputs the image data generated by the image generation control processing unit 11 to the display device 3 from the input / output I / F unit 40.

[0078] (Effects) Generally, when camera parameters are estimated from an image, errors are included. The influence of these errors appears as errors in the gaze direction. For example, as shown in Figure 5, when the distance from the viewpoint is short, the correct gaze direction vector (dashed line) d １ The error of gaze direction ε １ Therefore, the line of sight vector (dashed line) d when the distance is long ２ The error of gaze direction ε ２ In other words, the more distant an object is, the more it is affected by the above error.

[0079] In addition, since the density varies greatly near the object, the degree of influence of the error also changes depending on the direction of the line of sight relative to the object. For example, as shown in FIG. 6A, ３ is the line of sight direction θ when the line of sight is perpendicular to the object BT ３ Deviation δ ３ As a result of the error in density Za caused by ３ is the line of sight direction θ when the line of sight is oblique to the object BT ４ Deviation δ３ The error in density Zb caused by the above is larger.

[0080] In order to solve the above problem, in one embodiment of the present invention, for a gaze sample generated for each pixel of image data, the gaze correction processing unit 15 uses a learning model to calculate a gaze correction value η i Then, the luminance field calculation processing unit 16 estimates the line of sight correction value η i The color distribution c of the luminance field reflecting ｉ and density σ ｉ is estimated by a learning model, and the estimated color distribution c ｉ and density σ ｉ The estimated color value C^(p) of the pixel is calculated by aggregating these values ​​by the luminance field aggregation processing unit 14. Finally, based on the estimated color values ​​C^(p) calculated for all pixels, the image generation control processing unit 11 generates free viewpoint image data as seen from an arbitrary viewpoint.

[0081] Therefore, even if there are errors in the camera parameters, the effect can be reduced by correcting the gaze direction, making it possible to generate high-resolution free-viewpoint images. This effect becomes more pronounced when there is a mixture of images in which the object is close and far away, and when there is a mixture of images in which the gaze direction to the object is varied.

[0082] In one embodiment, the line of sight correction value η i and the color distribution of the luminance field c ｉ and density σ ｉ Both of these are estimated using a neural network learning model, which makes it possible to estimate the color of each pixel with gaze error corrected without performing complex calculations during drawing.

[0083] [Other Embodiments] The functional configuration of the free viewpoint image generation device, its processing procedure and processing content, the type of neural network used in the learning model and its algorithm, the type and use of the image to be drawn, etc. can be modified and implemented in various ways without departing from the spirit of the present invention.

[0084] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0085] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0086] 1...Free viewpoint image generation device 2...Input device 3...Display device 10...Control unit 20...Program storage unit 30...Data storage unit 40...Input / output I / F unit 50...Bus 11...Image generation control processing unit 12...Input information acquisition processing unit 13...Output information output processing unit 14...Luminance field aggregation processing unit 15...Gaze correction processing unit 16...Luminance field calculation processing unit 31...Parameter storage unit 32...Image storage unit 33...Learning model storage unit

Claims

a first processing unit that acquires parameter information including a size and a viewpoint position of a free viewpoint image to be rendered; a second processing unit that generates, for each of a plurality of pixels of the stored image data, a gaze direction vector from the viewpoint position included in the parameter information through the pixel; a third processing unit that calculates, for each of the gaze direction vectors, a correction value of the gaze direction vector based on a vector represented by the viewpoint position, the gaze direction vector, and a distance from the viewpoint position; a fourth processing unit that calculates, for each of the gaze direction vectors, a value representing a luminance field at a plurality of sample points set on a line of the gaze direction vector based on the distance from the viewpoint position to the sample point and the correction value; a fifth processing unit that estimates a color of the pixel corresponding to the gaze direction vector by aggregating, for each of the gaze direction vectors, the calculated values ​​representing the luminance field at the plurality of sample points; and a sixth processing unit that generates a free viewpoint image based on the position coordinates of the plurality of pixels and the estimated color value.

2. The free viewpoint image generating device of claim 1, wherein the third processing unit has a learning model that receives as input a vector represented by the viewpoint position, the gaze direction vector, and the distance from the viewpoint position, and outputs the correction value of the gaze direction vector.

3. A free viewpoint image generation method executed by an information processing device, comprising: a step of acquiring a plurality of image data captured from a plurality of viewpoints in a drawing space and storing the acquired image data in a storage medium; a step of acquiring parameter information including a size and a viewpoint position of a free viewpoint image to be drawn; a step of generating, for each of a plurality of pixels of the stored image data, a gaze direction vector passing through the pixel from the viewpoint position included in the parameter information; a step of calculating, for each of the gaze direction vectors, a vector represented by the viewpoint position, the gaze direction vector, and a distance from the viewpoint position, a correction value of the gaze direction vector based on the vector represented by the viewpoint position, the gaze direction vector, and a distance from the viewpoint position; a step of calculating, for each of the gaze direction vectors, values ​​representing a luminance field at a plurality of sample points set on a line of the gaze direction vector based on the distance from the viewpoint position to the sample point and the correction value; a step of estimating a color of the pixel corresponding to the gaze direction vector by aggregating, for each of the gaze direction vectors, the calculated values ​​representing the luminance field at the plurality of sample points; and a step of generating a free viewpoint image based on the position coordinates of the plurality of pixels and the estimated color value.

4. A program for causing a processor provided in a free viewpoint image generation device to execute the processing executed by at least one of the first processing unit to the sixth processing unit provided in the free viewpoint image generation device according to claim 1 or 2.