Image generation device, method, and program
The image generation device dynamically allocates information using separate models for foreground and background brightness fields, addressing the challenges of NeRF and NeRF++ by enhancing image resolution and reducing distortion in complex scenes, allowing for high-quality images from arbitrary viewpoints.
Patent Information
- Application Number
- PCT/JP2023/017161
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-07
AI Technical Summary
Current image generation methods, such as NeRF and NeRF++, face challenges in producing high-resolution images from arbitrary viewpoints, especially when dealing with complex scenes involving foreground and background objects with varying densities and positional relationships, leading to issues like blurred objects, increased object counts, and reduced resolution due to inefficient information allocation.
An image generation device and method that calculates the brightness field for both foreground and background using separate models based on the line-of-sight coordinates and radius, allowing for dynamic selection of models and efficient information allocation to improve image resolution and reduce distortion.
Enables the generation of high-resolution images from any viewpoint by effectively managing information allocation between foreground and background objects, improving image quality and reducing distortion, even in complex scenes with non-concentric spherical backgrounds and sparse foreground objects.
Smart Images

Figure JP2023017161_07112024_PF_FP_ABST
Abstract
Description
Image generation device, method, and program
[0001] FIELD Embodiments of the present invention relate to an image generation device, a method, and a program.
[0002] Neural Radiance Fields (NeRF) is a free viewpoint image generation method that generates an image from an arbitrary viewpoint different from the shooting viewpoint from a group of images captured from multiple viewpoints (see, for example, Non-Patent Document 1).
[0003] NeRF sets the region containing the object to be drawn, called the drawing space, as a rectangular parallelepiped or sphere, and models the luminance field in the drawing space using a neural network. The luminance field consists of a color distribution c and a density σ.
[0004] Figure 1 shows an example of image generation using NeRF. For example, if the drawing space has a center (0,0,0) and a radius R W When rendering in NeRF, camera parameters (symbol a in Fig. 1) are specified, and an image (symbol b in Fig. 1) is calculated from the luminance field for each pixel.
[0005] The camera parameters include a projection function K, a rotation matrix, and a translation vector. The direction vector (e in FIG. 1) of the line of sight (d in FIG. 1), which is a straight line passing from this translation vector to the pixel coordinate (c in FIG. 1) of the image, is expressed by the following equation:
[0006]
[0007] For this equation, the rotation matrix, translation vector, pixel image of the image, and gaze direction vector are expressed as follows:
[0008]
[0009] The line of sight is expressed as follows:
[0010]
[0011] The starting point t n and the end point t fBy defining and aggregating the color distribution and density of the luminance field (symbol f in Fig. 1), the color estimate of the pixel coordinate can be calculated by the following formula: where r denotes the coordinate of the point on the line of sight. Symbol g in Fig. 1 denotes the coordinate where the color is estimated.
[0012]
[0013] For this equation, the left hand side, an estimate of the color of the pixel image, is given as follows:
[0014]
[0015] Also, t i , i=1,...,n are sampled so as to satisfy the following conditions: N is the number of samples. n ≦t 1 ≦ … ≦t N ≦t f Luminance field color distribution NN c and density NN σ Each of these is composed of a neural network and is trained using a set of images with known camera parameters. The specific configuration and training method of NeRF are widely known, so a detailed explanation will be omitted here. There is also a method called NeRF++ that divides a luminance field model into foreground and background (see, for example, Non-Patent Document 2).
[0016] “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi and Ren Ng,ECCV, 2020 “NeRF++: Analyzing and Improving Neural Radiance Fields”, Kai Zhang, Gernot Riegler, Noah Snavely and Vladlen Koltun, arXiv:2010.07492, 2020
[0017] Fig. 2 is a diagram illustrating an example of a problem with NeRF. When a large space such as outdoors is used as the scene to be modeled (symbol a in Fig. 2) using the above-mentioned NeRF, if the rendering space (symbol b in Fig. 2) is set to be narrow, the neural network cannot correctly learn objects located outside the rendering space, resulting in distortion of the image, such as blurred objects or the inclusion of multiple objects in the image (symbol c in Fig. 2).
[0018] Furthermore, when a relatively large drawing space (symbol d in Figure 2) is set, the neural network also learns from areas where nothing is displayed, which reduces the amount of information that can be assigned to the object to be drawn, resulting in a problem of reduced resolution of the drawing result (symbol e in Figure 2).
[0019] When using the above-mentioned NeRF++, a scene consisting of a densely packed foreground and a background that surrounds the foreground in a concentric sphere can be rendered with a relatively high resolution.
[0020] In NeRF++, the foreground model is selected when the following conditions are met, and the background model is selected otherwise.
[0021]
[0022] The luminance field density σ and color distribution c in NeRF++ are expressed as follows:
[0023]
[0024] Here, the function G that performs the coordinate transformation of the foreground model F and a function G that performs coordinate transformation of the background model. B is shown as follows:
[0025]
[0026] Furthermore, the above equations for the luminance field density σ and color distribution c include the parameters of the foreground color distribution neural network, the foreground density neural network, the background color distribution neural network, and the background density neural network, and are expressed as follows:
[0027]
[0028] In NeRF++, the relative value of the amount of information allocated to the foreground and background with respect to the distance from the center of the drawing space (E F (r)), E B (r)) is shown as follows:
[0029]
[0030] FIG. 3 is a diagram illustrating an example of allocation of information amount of NeRF++. FIG. 4 is a diagram illustrating an example of NeRF++. In FIG. 3, E when R=1 F (r) and E B For (r), a graph is shown with the maximum value scaled to 1.
[0031] In addition, the amount of information is constant in the foreground, while the amount of information in the background decreases as one moves away from the sphere. In Fig. 3, NeRF++ separates the foreground and background using a sphere (symbol a in Fig. 4), and limits the rendering space of the foreground to within the sphere (symbol b in Fig. 4), thereby modeling the foreground object with high resolution.
[0032] In addition, NeRF++ reduces the amount of information allocated to the background rendering space (symbol c in Figure 4) as it moves away from the sphere and away from the sphere, allocating more information to distant objects as well, thereby enabling rendering with less distortion. The part indicated by symbol d in Figure 4 corresponds to the part with a relatively high amount of information in the background model, i.e., the part indicated by the ellipse corresponding to symbol a in Figure 3.
[0033] Fig. 5 illustrates a first example of a problem with NeRF++. However, as shown in Fig. 5, when a background object is far from the foreground, it is not possible to properly assign areas of the background model with a relatively high amount of information to the object, resulting in a decrease in background resolution.
[0034] Fig. 6 is a diagram illustrating a second example of a problem with NeRF++. As shown in Fig. 6, when the background is not concentric with the foreground, only a portion of the background model with a relatively high amount of information can be allocated, resulting in a problem of reduced background resolution.
[0035] Fig. 7 is a diagram illustrating a third example of a problem with NeRF++. Furthermore, when objects are not densely packed in the foreground but are spaced apart, as in the area indicated by the symbol a in Fig. 7, gaps appear in areas where the amount of information in the foreground model is relatively high, resulting in a decrease in the resolution of the foreground.
[0036] As described above, current techniques cannot provide a method for generating a free viewpoint image with high resolution regardless of the positional relationship between the background and foreground or the density of the foreground.
[0037] The present invention has been made in light of the above-mentioned circumstances, and its object is to provide an image generation device, method, and program that are capable of generating appropriate images from any viewpoint.
[0038] An image generating device according to one aspect of the present invention comprises a luminance field calculation unit that calculates the luminance field of the foreground of a drawing space of an image using a model for the foreground of the drawing space, and calculates the luminance field of the background of the drawing space using a model for the background of the drawing space; a model selection unit that selects, based on the coordinates of the line of sight to a drawing object in the drawing space and the radius of the foreground, whether to calculate the luminance field of the foreground using the model for the foreground by the luminance field calculation unit, or to calculate the luminance field of the background using the model for the background by the luminance field calculation unit; a luminance field aggregation unit that estimates the color of an image of the drawing object by aggregating the calculation results by the luminance field calculation unit along the coordinates of the line of sight based on the model selected by the model selection unit; and a generation unit that generates an image of the drawing object based on the estimated color.
[0039] An image generating device according to one aspect of the present invention includes a luminance field calculation unit that calculates the luminance field of the foreground of a drawing space of an image using a foreground model of the drawing space, and calculates the luminance field of the background of the drawing space using a background model of the drawing space; an input unit that accepts input of an image to be used in learning parameters of the foreground and background models and a radius of the foreground; a model selection unit that selects, based on the coordinates of a line of sight to a drawing object in the drawing space and the radius of the foreground, whether to calculate the luminance field of the foreground using the foreground model by the luminance field calculation unit, or to calculate the luminance field of the background using the background model by the luminance field calculation unit; a luminance field aggregation unit that estimates the color of the image to be used for learning by aggregating the calculation results by the luminance field calculation unit along the coordinates of the line of sight based on the model selected by the model selection unit; and a learning unit that learns the parameters based on the difference between the estimated color and a correct answer for the color of the image to be used for learning.
[0040] An image generation method according to one aspect of the present invention is a method performed by an image generation device, comprising: a luminance field calculation unit of the image generation device calculating a luminance field of the foreground of a drawing space of an image using a foreground model of the drawing space, and calculating a luminance field of the background of the drawing space using a background model of the drawing space; a model selection unit of the image generation device selecting, based on the coordinates of the line of sight to a drawing object in the drawing space and the radius of the foreground, whether the luminance field calculation unit will calculate the luminance field of the foreground using the foreground model, or whether the luminance field calculation unit will calculate the luminance field of the background using the background model; a luminance field aggregation unit of the image generation device estimating a color of the image of the drawing object by aggregating the calculation results by the luminance field calculation unit along the coordinates of the line of sight based on the selected model; and a generation unit of the image generation device generating an image of the drawing object based on the estimated color.
[0041] According to the present invention, it is possible to generate an appropriate image from any viewpoint.
[0042] FIG. 1 is a diagram illustrating an example of image generation using NeRF. FIG. 2 is a diagram illustrating an example of a problem with NeRF. FIG. 3 is a diagram illustrating an example of information allocation using NeRF++. FIG. 4 is a diagram illustrating an example of NeRF++. FIG. 5 is a diagram illustrating a first example of a problem with NeRF++. FIG. 6 is a diagram illustrating a second example of a problem with NeRF++. FIG. 7 is a diagram illustrating a third example of a problem with NeRF++. FIG. 8 is a diagram illustrating an application example of an image generation device according to an embodiment of the present invention. FIG. 9 is a flowchart illustrating an example of the processing procedure of the image generation device during learning. FIG. 10 is a flowchart illustrating an example of the processing procedure of the image generation device during drawing. FIG. 11 is a flowchart illustrating an example of the processing procedure of the model selection unit in the first embodiment. FIG. 12 is a diagram illustrating an example of information allocation in the first embodiment. FIG. 13 is a diagram illustrating an overview of the first embodiment. FIG. 14 is a diagram illustrating an example of solving a problem using the first embodiment. FIG. 15 is a flowchart illustrating an example of the processing procedure of the model selection unit in the second embodiment. FIG. 16 is a diagram illustrating an example of solving a problem using the second embodiment. Fig. 17 is a flowchart showing an example of the procedure of processing operations by the model selection unit in Example 3. Fig. 18 is a diagram showing an example of solving a problem by Example 3. Fig. 19 is a block diagram showing an example of the hardware configuration of an image generation device according to an embodiment of the present invention.
[0043] An embodiment of the present invention will be described below with reference to the drawings. Fig. 8 is a diagram showing an application example of an image generating device according to an embodiment of the present invention. Fig. 9 is a flowchart showing an example of a procedure for processing operations of the image generating device during learning. Fig. 10 is a flowchart showing an example of a procedure for processing operations of the image generating device during drawing.
[0044] As shown in FIG. 8, an image generating device 100 according to one embodiment of the present invention includes an input unit 10, a storage unit 20, a control unit 30, a luminance field summarizing unit 40, a model selecting unit 50, a luminance field calculating unit 60, and an output unit 70.
[0045] The luminance field calculation unit 60 of the image generating device 100 of this embodiment has the function of efficiently allocating information amounts to foreground and background objects by freely setting the number of foreground objects and their positional relationship with the background.
[0046] (First Example) Next, a first example will be described. (Input Unit 10) During the learning of the following neural network, the input unit 10 inputs the radius R of the drawing space. W , number of samples N, number of steps S, image set, foreground radius R F and background radius R B The image group is composed of image I m and the camera parameters are as follows:
[0047]
[0048] m is expressed as follows: m=1,...,M
[0049] During drawing, the input unit 10 receives input of camera parameters, and the width W and height H of the image to be generated. The camera parameters are expressed as follows:
[0050]
[0051] (Storage Unit 20) The storage unit 20 includes a storage device such as a nonvolatile memory. W , number of samples N, number of steps S, image set, foreground radius R F , background radius R B , and the parameters of the neural network are stored. The image set is divided into image I m and the camera parameters are as follows:
[0052]
[0053] m is expressed as follows: m=1,...,M
[0054] The stored neural network parameters include parameters of a foreground density neural network, a foreground color distribution neural network, a background density neural network, and a background color distribution neural network, and are expressed as follows:
[0055]
[0056] (Control Unit 30) During learning, the control unit 30 randomly repeats image selection and pixel selection for the number of steps S. The control unit 30 also calculates the following loss function from the color estimates calculated by the luminance field summarization unit 40 and the color values of the original image, and trains the accumulated neural network parameters so as to minimize this loss function. The formula for calculating the loss function is expressed as follows:
[0057]
[0058] The configuration of the neural network and the specific learning method are generally known, so details will be omitted here. Furthermore, when drawing, the control unit 30 calls the luminance field summarizing unit 40 the number of times equal to the image size (W × H), and generates an image using this luminance field summarizing unit 40.
[0059] (Luminance field summarizing unit 40) The luminance field summarizing unit 40 calculates the luminance field summarizing unit 40 by summarizing the input translation vector, direction vector, and radius R of the rendering space. W From the starting point t of the gaze sample n and the end point t f is calculated as follows:
[0060]
[0061] starting point t n and the end point t f is also expressed as follows:
[0062]
[0063] Also, t i , i=1,...,n are sampled such that the following condition is satisfied: n ≦t 1 ≦ … ≦tN ≦t f
[0064] Then, the luminance field summarizing unit 40 calculates the luminance field summarizing time for each t i For this, the luminance field calculation unit 60 selected by the model selection unit 50 is called. This selection will be described later.
[0065] The luminance field summarizing unit 40 obtains an estimated color value by summarizing the calculation results of the luminance field density and color distribution by the called luminance field calculation unit 60 as follows.
[0066]
[0067] The calculation results of the luminance field density and color distribution by the luminance field calculation unit 60 are expressed as follows:
[0068]
[0069] (Model Selection Unit 50) The model selection unit 50 selects the coordinates of the calculated gaze samples and the foreground radius R F A model is selected from a foreground model and a background model based on the
[0070] The coordinates are expressed by the following formula:
[0071]
[0072] The left side of this equation is expressed as follows:
[0073]
[0074] 11 is a flowchart showing an example of the procedure of the processing operation by the model selection unit in the first embodiment. First, the model selection unit 50 inputs the x-axis, y-axis, and z-axis values (x, y, z) of the coordinates of the gaze sample (S41), and calculates the square root of the sum of the squares of the x-axis, y-axis, and z-axis values, i.e., "√(x 2 +y 2 +z 2 The distance from the origin of the line of sight sample is calculated as (S42).
[0075] Next, the model selection unit 50 calculates the distance from the origin calculated in S42 as the foreground radius R FIf the distance from the origin calculated in S42 is within the range of the foreground radius R (Yes in S43), the model selection unit 50 selects the foreground model (S44). F If the difference is not within the range (No in S43), a background model is selected (S45). The model selection unit 50 calculates the luminance field by calling the luminance field calculation unit 60 corresponding to the selected model.
[0076] (Luminance field calculation unit 60) The luminance field calculation unit 60 has a luminance field calculation unit corresponding to the foreground model (sometimes referred to as a foreground luminance field calculation module, etc.) and a luminance field calculation unit corresponding to the background model (sometimes referred to as a background luminance field calculation module, etc.). The luminance field calculation unit corresponding to the foreground model calculates the foreground luminance field using the following equation:
[0077]
[0078] Here, G F is a function used to perform the following coordinate transformation:
[0079]
[0080] Furthermore, the luminance field calculation unit corresponding to the background model calculates the luminance field of the background using the following equation.
[0081]
[0082] Here, G B is a function used to perform the following coordinate transformation:
[0083]
[0084] (Output Unit 70) The output unit 70 outputs the image generated during drawing.
[0085] Next, the operation during learning in the first embodiment will be described with reference to FIG. 9 . First, the input unit 10 accepts input such as a group of images and stores the input results in the storage unit 20 (S11). If there are remaining steps (Yes in S12), the control unit 30 randomly selects an image (S13). Next, the control unit 30 randomly selects a pixel from the image selected in S13 (S14). Furthermore, the control unit 30 calls the luminance field aggregation unit 40 with the image selected in S13, its camera parameters, i.e., the projection function, rotation matrix, and translation vector, and the pixel position selected in S14 as input.
[0086] The luminance field summarizing unit 40 generates gaze samples based on the given gaze (S15). Next, if there are remaining samples (Yes in S16), the luminance field summarizing unit 40 calls the model selecting unit 50 using the coordinates of the gaze samples generated in S15 as input.
[0087] The model selection unit 50 selects a model based on the coordinates of the input gaze sample (S17). Next, the model selection unit 50 calls the luminance field calculation unit 60 using the coordinates of the input gaze sample and the model selected in S17 as input.
[0088] The luminance field calculation unit 60 calculates the luminance field value based on the coordinates of the input gaze sample and the model selected in S17 (S18), and returns this calculated value to the luminance field aggregation unit 40 via the model selection unit 50.
[0089] The luminance field aggregation unit 40 repeats the model selection in S17 and the calculation of the luminance field value in S18 until there are no more samples remaining, and when there are no more samples remaining (No in S16), it aggregates the luminance field to obtain an estimated value of the pixel color (S19).
[0090] The control unit 30 calculates the difference between the original value of the pixel color and the estimated value calculated in S19, and trains the parameters of various neural networks (S20). The processes from S13 to S20 are repeated until there are no more steps remaining.
[0091] When the repetition of the number of steps is completed (No in S12), the control unit 30 stores the learned parameters of the various neural networks in the storage unit 20, and the learning process ends.
[0092] Next, the operation of drawing an image in the first embodiment will be described with reference to Fig. 10. First, the input unit 10 inputs camera parameters, width W and height H of the image to be generated (S31).
[0093] The control unit 30 repeatedly calls the luminance field aggregation unit 40 along with the camera parameters (projection function, rotation matrix, translation vector) for all pixels (width W x height H) of the image, i.e., when there are remaining pixels (Yes in S32).
[0094] The luminance field summarizing unit 40 generates gaze samples based on the given gaze (S33). Next, if there are remaining samples (Yes in S34), the luminance field summarizing unit 40 calls the model selecting unit 50 using the coordinates of the gaze samples generated in S33 as input.
[0095] The model selection unit 50 selects a model based on the coordinates of the input gaze sample (S35). Next, the model selection unit 50 calls the luminance field calculation unit 60 using the coordinates of the input gaze sample and the model selected in S35 as input.
[0096] The luminance field calculation unit 60 calculates the luminance field value based on the coordinates of the input gaze sample and the model selected in S35 (S36), and returns this calculated value to the luminance field aggregation unit 40 via the model selection unit 50.
[0097] The luminance field summarizing unit 40 repeats the model selection in S35 and the calculation of the luminance field value in S36 until there are no more samples remaining, and when there are no more samples remaining (No in S34), it aggregates the luminance field to obtain an estimated value of the pixel color (S37). The processes from S33 to S37 are repeated until there are no more pixels remaining.
[0098] When the repetition of the pixel count is completed (No in S32), the control unit 30 generates an image based on the pixel position and the estimated color value of this position. The output unit 70 outputs this generated image, and the drawing process is completed.
[0099] In this embodiment, the relative value of the amount of information allocated to the foreground and background with respect to the distance from the center of the rendering space (E F (r), E B (r)) is expressed as follows:
[0100]
[0101] FIG. 12 is a diagram illustrating an example of allocation of information amount in the first embodiment. In FIG. F = 1 and R B E when = 2 F (r) and E B For (r), a graph is shown with the maximum value scaled to 1. The amount of information in the foreground is constant within the sphere of the foreground radius, and the amount of information in the background decreases as one moves away from the sphere of the background radius.
[0102] In this embodiment, the foreground rendering space is limited to a sphere of the foreground radius, allowing foreground objects to be modeled with relatively high resolution. Also, in this embodiment, the amount of information allocated to the background rendering space decreases with increasing distance from the sphere of the background radius, allowing for more information to be allocated to distant objects as well, thereby enabling rendering with less distortion.
[0103] Fig. 13 is a diagram illustrating an overview of the first embodiment. The portion indicated by the reference character c in Fig. 13 corresponds to the portion with a relatively high amount of background model information, i.e., the portion indicated by the reference character a in Fig. 12.
[0104] 14 is a diagram showing an example of solving the problem by the first embodiment. As shown in FIG. 13, B the foreground radius R F By making it larger, even if an object in the background (symbol b in FIG. 13) is far from the foreground (symbol a in FIG. 13), it is possible to assign a location (symbol a in FIG. 14) with a relatively high amount of information in the background model to the object, thereby improving the resolution of the background.
[0105] Second Example Next, a second example will be described. In the following example, detailed descriptions of parts that are the same as those in the first example may be omitted. In this second example, an example will be described in which the centers of the foreground and background are different.
[0106] (Input Unit 10) During neural network learning, the input unit 10 inputs the radius R of the drawing space. W , number of samples N, number of steps S, image set, foreground radius R F , foreground center (c x , c y , c z ), and the background radius R B The image group is expressed as follows: In the second embodiment, the foreground center is different from the background center. That is, in the second embodiment, the background center is set at a position shifted from the foreground center.
[0107]
[0108] During drawing, the input unit 10 accepts input of camera parameters, and the width W and height H of the image to be generated. The input camera parameters are a projection function, a rotation matrix, and a translation vector, as in the first embodiment.
[0109] (Storage Unit 20) In the second embodiment, the storage unit 20 stores the radius R of the drawing space. W , number of samples N, number of steps S, image set, foreground radius R F , foreground center (c x , c y , c z ), background radius R B , and parameters of various neural networks are stored. The image group is expressed as follows, as in the first embodiment:
[0110]
[0111] Furthermore, the parameters of the various neural networks stored as described above include parameters of a foreground density neural network, a foreground color distribution neural network, a background density neural network, and a background color distribution neural network, as in the first embodiment.
[0112] (Model Selector 50) In the second embodiment, the model selector 50 selects a model from a foreground model and a background model based on the coordinates of the gaze sample, the foreground center, and the foreground radius.
[0113] 15 is a flowchart showing an example of the procedure of the processing operation by the model selection unit in the second embodiment. First, the model selection unit 50 inputs the x-axis, y-axis, and z-axis values (x, y, z) of the coordinates of the gaze sample (S51), and calculates the square root of the sum of the squares of the difference between the x-axis value of the coordinates and the x-axis value of the foreground radius, the difference between the y-axis value of the coordinates and the y-axis value of the foreground radius, and the difference between the z-axis value of the coordinates and the z-axis value of the foreground radius, i.e., "√((x-c x ) 2 +(y-c y ) 2 (z-c z ) 2 The distance of the line of sight sample from the foreground center is calculated as (S52).
[0114] Next, the model selection unit 50 calculates the distance from the foreground center calculated in S52 as the foreground radius R F If the distance from the foreground center calculated in S52 is within the range of the foreground radius R (Yes in S53), the model selection unit 50 selects the foreground model (S54). F If the difference is not within the range (No in S53), a background model is selected (S55). The model selection unit 50 calculates the luminance field by calling the luminance field calculation unit 60 corresponding to the selected model.
[0115] (Luminance field calculation unit 60) In the second embodiment, as in the first embodiment, the luminance field calculation unit 60 includes a luminance field calculation unit corresponding to the foreground model and a luminance field calculation unit corresponding to the background model. The luminance field calculation unit corresponding to the foreground model calculates the luminance field density σ using the same calculation formula as that used in the first embodiment. i and color distribution c i Calculate.
[0116] Here, G F is a function used when performing the following coordinate transformation: Note that, compared to the first embodiment, the foreground radius is taken into consideration.
[0117]
[0118] The luminance field calculation unit corresponding to the background model calculates the luminance field density σ using the same calculation formula as that used in the first embodiment. i and color distribution c i Calculate G B is a function used when performing coordinate transformation, as in the first embodiment. The functions of the control unit 30, the luminance field summarizing unit 40, and the output unit 70 are the same as in the first embodiment.
[0119] 16 is a diagram showing an example of solving the problem by the second embodiment. In this second embodiment, by shifting the center of the foreground from the center of the background (reference symbol a in FIG. 16), it becomes possible to assign to the object a location where the amount of information in the background model is relatively high, even if the background is not concentric with the foreground, and therefore the resolution of the background can be improved compared to the first embodiment.
[0120] Third Embodiment Next, a third embodiment will be described. In this third embodiment, an example will be described in which the foreground is made up of a plurality of foregrounds, in this case a first foreground and a second foreground.
[0121] (Input Unit 10) During neural network learning, the input unit 10 inputs the radius R of the drawing space. W , number of samples N, number of steps S, image set, radius of first foreground R F1 , the center of the first foreground (c x1 , c y1 , c z1), the radius of the second foreground R F2 , the center of the second foreground (c x2 , c y2 , c z2 ), and the background radius R B The image group is the same as in the first embodiment, including an image, a projection function, a rotation matrix, and a translation vector.
[0122] During drawing, the input unit 10 receives input of camera parameters, width W and height H of the image to be generated. The input camera parameters are a projection function, a rotation matrix, and a translation vector, as in the first embodiment.
[0123] (Storage Unit 20) In the third embodiment, the storage device of the storage unit 20 stores the radius R of the drawing space. W , number of samples N, number of steps, image set, radius of first foreground R F1 , the center of the first foreground (c x1 , c y1 , c z1 ), the radius of the second foreground R F2 , the center of the second foreground (c x2 , c y2 , c z2 ), background radius R B , and various neural network parameters are stored. The stored image group includes images, projection functions, rotation matrices, and translation vectors, as in the first embodiment.
[0124] In addition, the parameters of the various neural networks stored as described above include parameters of a first foreground density neural network, a first foreground color distribution neural network, a second foreground density neural network, a second foreground color distribution neural network, a background density neural network, and a background color distribution neural network, and are expressed as follows:
[0125]
[0126] (Model Selection Unit 50) In the third embodiment, the model selection unit 50 selects a model from a first foreground model, a second foreground model, and a background model based on the coordinates of the gaze sample, the first and second foreground centers, and the first and second foreground radii.
[0127] 17 is a flowchart showing an example of the procedure of the processing operation by the model selection unit in the third embodiment. First, the model selection unit 50 inputs the x-axis, y-axis, and z-axis values (x, y, z) of the coordinates of the gaze sample (S61), and calculates the square root of the sum of the squares of the difference between the x-axis value of the coordinates and the x-axis value of the radius of the first foreground, the difference between the y-axis value of the coordinates and the y-axis value of the radius of the first foreground, and the difference between the z-axis value of the coordinates and the z-axis value of the radius of the first foreground, i.e., "√((x-c x1 ) 2 +(y-c y1 ) 2 (z-c z1 ) 2 The distance of the line of sight sample from the first foreground center is calculated as (S62).
[0128] Next, the model selection unit 50 calculates the radius R of the first foreground by calculating the distance from the center of the first foreground calculated in S62. F1 If the distance from the center of the first foreground calculated in S62 is within the radius R of the first foreground, the model selection unit 50 selects the first foreground model (S64). F1 If the value is not within the range (No in S63), the square root of the sum of the squares of the difference between the x-axis value of the coordinate input in S61 and the x-axis value of the radius of the second foreground, the difference between the y-axis value of the coordinate input in S61 and the y-axis value of the radius of the second foreground, and the difference between the z-axis value of the coordinate input in S61 and the z-axis value of the radius of the second foreground, that is, "√((x-c x2 ) 2 +(y-c y2 ) 2 (z-c z2 ) 2 The distance from the second foreground center is calculated as (S65).
[0129] Next, the model selection unit 50 calculates the radius R of the second foreground by calculating the distance from the center of the second foreground calculated in S65. F2If the distance from the center of the second foreground calculated in S65 is within the radius R of the second foreground, the model selection unit 50 selects the second foreground model (S67). F2 If the difference is not within the range (No in S66), a background model is selected (S68). The model selection unit 50 calculates the luminance field by calling the luminance field calculation unit 60 corresponding to the selected model.
[0130] (Luminance field calculation unit 60) In the third embodiment, the luminance field calculation unit 60 includes a luminance field calculation unit corresponding to a first foreground model, a luminance field calculation unit corresponding to a second foreground model, and a luminance field calculation unit corresponding to a background model. The luminance field calculation unit corresponding to the first foreground model calculates the density σ of the luminance field of the first foreground using the following formula: i and color distribution c i Calculate.
[0131]
[0132] Here, G F1 is a function used when performing the following coordinate transformation: Note that, compared to the first embodiment, the radius of the first foreground is taken into consideration.
[0133]
[0134] The luminance field calculation unit corresponding to the second foreground model calculates the density σ of the luminance field of the second foreground using the following calculation formula: i and color distribution c i Calculate.
[0135]
[0136] Here, G F2 is a function when performing the following coordinate transformation: Note that, compared to the first embodiment, the radius of the second foreground is taken into consideration.
[0137]
[0138] The luminance field calculation unit corresponding to the background model calculates the density σ of the luminance field of the background using the same calculation formula as that used in the first embodiment. i and color distribution c i Calculate.
[0139] Here, G B is a function used to perform the following coordinate transformation:
[0140]
[0141] The functions of the control unit 30, the luminance field summarizing unit 40, and the output unit 70 in the third embodiment are similar to those in the first embodiment.
[0142] Fig. 18 is a diagram showing an example of solving the problem by the third embodiment. In this third embodiment, by arranging multiple foregrounds (symbol a in Fig. 18), even when objects are not densely packed in one foreground but are spaced apart, it is possible to reduce the gaps between foreground models and improve the resolution of the foreground.
[0143] Fig. 19 is a block diagram showing an example of the hardware configuration of an image generating apparatus according to an embodiment of the present invention. In the example shown in Fig. 19, the image generating apparatus 100 according to the above embodiment is configured as, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU. A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.
[0144] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from the communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.
[0145] An input device 200 and an output device 300, which are attached to the image generating device 100 and used by a user or the like, are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200, such as a keyboard, touch panel, touchpad, or mouse, and can output output data to an output device 300, which includes a display device using a liquid crystal or organic electroluminescence (EL) display, for display. The input device 200 and the output device 300 may be devices built into the image generating device 100, or may be input devices and output devices of other information terminals that can communicate with the image generating device 100 via a network NW.
[0146] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a ROM, and can store programs necessary to execute various control processes, etc., according to one embodiment.
[0147] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM, and can be used to store various data or information acquired and created during various processes.
[0148] An image generating device 100 according to one embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 10 as software-based processing functional units.
[0149] The information storage unit used as a work memory or the like by each unit of image generating device 100 can be configured using data memory 112 shown in Fig. 19. However, these configured storage areas are not essential components within image generating device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.
[0150] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).
[0151] The techniques described in the above embodiments can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can be distributed by transmission via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.
[0152] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.
[0153] REFERENCE SIGNS LIST 100... image generating device 10... input unit 20... storage unit 30... control unit 40... luminance field summarizing unit 50... model selecting unit 60... luminance field calculating unit 70... output unit
Claims
1. An image generation device comprising: a luminance field calculation unit that calculates the luminance field of the foreground in a drawing space of an image using a model for the foreground of the drawing space, and calculates the luminance field of the background in the drawing space using a model for the background of the drawing space; a model selection unit that selects, based on the coordinates of the line of sight to a drawing object in the drawing space and a radius of the foreground, whether to have the luminance field calculation unit calculate the luminance field of the foreground using the model for the foreground, or to have the luminance field calculation unit calculate the luminance field of the background using the model for the background; a luminance field aggregation unit that estimates a color of an image of the drawing object by aggregating the calculation results by the luminance field calculation unit along the coordinates of the line of sight based on the model selected by the model selection unit; and a generation unit that generates an image of the drawing object based on the estimated color.
2. An image generating device comprising: a luminance field calculation unit that calculates the luminance field of the foreground in a drawing space of an image using a foreground model of the drawing space, and calculates the luminance field of the background in the drawing space using a background model of the drawing space; an input unit that accepts input of an image and a radius of the foreground to be used in learning parameters of the foreground and background models; a model selection unit that selects, based on the line of sight coordinates to a drawing object in the drawing space and the foreground radius, whether to calculate the luminance field of the foreground using the foreground model by the luminance field calculation unit, or to calculate the luminance field of the background using the background model by the luminance field calculation unit; a luminance field aggregation unit that estimates the color of the image to be used for the learning by aggregating the calculation results by the luminance field calculation unit along the line of sight coordinates based on the model selected by the model selection unit; and a learning unit that learns the parameters based on the difference between the estimated color and a correct answer for the color of the image to be used for the learning.
3. The image generating device of claim 1, wherein the luminance field calculation unit calculates the foreground luminance field using the foreground model under the condition that the foreground drawing space is within a first radius and the amount of information allocated to the foreground drawing space is constant, and calculates the background luminance field using the background model under the condition that the amount of information allocated to the background drawing space decreases as it moves away from a second radius range larger than the first radius.
4. The image generating device of claim 1, wherein the model selection unit selects, based on the coordinates of the line of sight to the drawing object in the drawing space, the coordinates of the center of the foreground, and the radius of the foreground, whether to have the luminance field calculation unit calculate the foreground luminance field using the model for the foreground, or to have the luminance field calculation unit calculate the background luminance field using the model for the background, wherein the luminance field calculation unit calculates the foreground luminance field using the model for the foreground under a condition in which the center of the background is shifted from the center of the foreground, and calculates the background luminance field using the model for the background under a condition in which the center of the background is shifted from the center of the foreground.
5. The image generating device of claim 1, wherein the foreground consists of a plurality of foregrounds, and the model selection unit selects, based on the coordinates of the line of sight to the drawing object in the drawing space, the coordinates of the centers of the plurality of foregrounds, and the radius of each of the plurality of foregrounds, whether the luminance field calculation unit will calculate the luminance field of the foreground using a model corresponding to one of the plurality of foregrounds, or whether the luminance field calculation unit will calculate the luminance field of the background using the model for the background.
6. A method performed by an image generating device, comprising: a luminance field calculation unit of the image generating device calculating a luminance field of the foreground in a drawing space of an image using a model for the foreground of the drawing space, and calculating a luminance field of the background in the drawing space using a model for the background of the drawing space; a model selection unit of the image generating device selecting, based on the coordinates of the line of sight to a drawing object in the drawing space and a radius of the foreground, whether the luminance field calculation unit will calculate the luminance field of the foreground using the model for the foreground, or the luminance field calculation unit will calculate the luminance field of the background using the model for the background; a luminance field aggregation unit of the image generating device estimating a color of the image of the drawing object by aggregating, based on the selected model, the calculation results by the luminance field calculation unit along the coordinates of the line of sight; and a generation unit of the image generating device generating an image of the drawing object based on the estimated color.
7. An image generation processing program that causes a processor to function as each part of the image generation device according to any one of claims 1 to 5.
Citation Information
Patent Citations
High resolution neural rendering
US20220301257A1
