Image processing apparatus, imaging apparatus, control method, and program

Abstract

To guarantee a predetermined image quality for a virtual viewpoint image.SOLUTION: An image processing apparatus generates a virtual viewpoint image obtained by drawing a three-dimensional model in which a picked-up image is applied as a texture, and the image processing apparatus has: acquisition means that acquires normal information on the three-dimensional model; setting means that sets a drawing viewpoint for drawing the three-dimensional model; and first generation means that generates the virtual viewpoint image obtained by drawing the three-dimensional model on the basis of the drawing viewpoint set by the setting means. The setting means sets the drawing viewpoint in a space area based on the normal information.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, a control method, and a program, and more particularly to a technique for generating a virtual viewpoint image corresponding to a viewpoint different from that at the time of imaging based on a captured image. [Background technology]

[0002] There is a technology that generates a 3D model and texture of a subject based on a captured image, and draws a 3D space in which the 3D model with the texture applied is placed from an arbitrary viewpoint (virtual viewpoint) to generate a virtual viewpoint image seen from a position different from that of the imaging device. By using this technology, it is also possible to provide a user with the experience of viewing an imaged scene while sequentially changing the viewpoint (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-003765 ​​A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the quality of a virtual viewpoint image may change depending on the virtual viewpoint set when generating the virtual viewpoint image. For example, a three-dimensional model constructed based on distance information from an imaging device to a subject includes a surface extending in the depth direction, particularly in an area where a local steep change in the depth direction appears in the imaging direction. If an image extracted from an imaging device is applied as a texture to such a surface, when the three-dimensional model is drawn from the imaging direction, the texture appears the same as the captured image, but when drawn from another direction, the texture is stretched to fit the surface and may appear different. Therefore, depending on the direction in which the three-dimensional model is drawn, the texture of some areas may not appear well (spatial resolution is reduced), and the viewer may feel that image quality has deteriorated.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an image processing device, an imaging device, a control method, and a program that ensure a predetermined image quality for a virtual viewpoint image. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the image processing device of the present invention is an image processing device that generates a virtual viewpoint image in which a three-dimensional model is drawn to which a captured image is applied as a texture, and is characterized by having: a setting means for setting a drawing viewpoint for drawing the three-dimensional model; a first generation means for generating a virtual viewpoint image in which the three-dimensional model is drawn based on the drawing viewpoint set by the setting means; and a control means for controlling the drawing viewpoint to be set by the setting means based on the three-dimensional model. Effect of the Invention

[0007] With this configuration, the present invention makes it possible to ensure a predetermined image quality for the virtual viewpoint image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a hardware configuration of a digital camera 100 according to an embodiment and a modification of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating a detailed configuration of an image sensor 11 according to an embodiment and a modification of the present invention; [Diagram 3] FIG. 1 is a diagram for explaining the principle of imaging plane phase difference distance measurement using an imaging element 11 according to an embodiment and a modified example of the present invention. [Figure 4] FIG. 11 is another diagram for explaining the principle of imaging plane phase difference ranging using the imaging element 11 according to the embodiment and modified example of the present invention. [Diagram 5] 1 is a flowchart illustrating a distance image generation process executed by the digital camera 100 according to an embodiment and a modification of the present invention; [Figure 6] FIG. 1 is a diagram for explaining a method for deriving distance information from a defocus amount according to an embodiment and a modification of the present invention; [Figure 7] FIG. 1 is a diagram for explaining construction of a three-dimensional model based on distance information according to an embodiment and a modification of the present invention; [Figure 8] FIG. 1 is a diagram illustrating an example of the arrangement of a three-dimensional model in a three-dimensional space according to an embodiment and a modification of the present invention; [Figure 9] FIG. 1 is a diagram for explaining a virtual viewpoint image that may cause a sense of discomfort in viewing according to the first embodiment and the first modification of the present invention; [Figure 10] FIG. 1 is a diagram illustrating a spatial region in which a rendering viewpoint can be set according to a first embodiment of the present invention; [Figure 11] 1 is a flowchart illustrating a virtual viewpoint image generation process executed by the digital camera 100 according to an embodiment and a modification of the present invention; [Figure 12] FIG. 13 is a diagram illustrating a spatial region in which a rendering viewpoint can be set according to a second embodiment of the present invention; [Figure 13] FIG. 13 is a diagram illustrating a spatial region in which a rendering viewpoint can be set according to a second modified example of the present invention; [Figure 14] FIG. 13 is a diagram for explaining a virtual viewpoint image that may cause a sense of discomfort in viewing according to the fourth modified example of the present invention; [Figure 15] FIG. 13 is a diagram for explaining a virtual viewpoint image that may cause a sense of discomfort in viewing according to the fifth modified example of the present invention; [Figure 16] FIG. 13 is a diagram for explaining a method of setting a virtual camera for each frame of a virtual viewpoint video according to a third embodiment of the present invention; [Figure 17] FIG. 23 is a diagram for explaining a method for generating a virtual viewpoint video according to a seventh modified example of the present invention; [Figure 18] FIG. 13 is a diagram illustrating a change in a virtual viewpoint video image due to a change in focal length according to the fourth embodiment of the present invention; [Figure 19] FIG. 13 is a diagram illustrating a method for determining whether or not the focal length of a rendering viewpoint has been changed according to a fourth embodiment of the present invention. [Figure 20] 11 is a flowchart illustrating a virtual viewpoint image generating process executed by the digital camera 100 according to the fourth embodiment and the ninth to fifteenth modifications of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] In the embodiment described below, an example of applying the present invention to a digital camera that can generate an image related to a viewpoint different from the imaging position by constructing a three-dimensional model corresponding to a subject based on a captured image and drawing the three-dimensional model as an example of an image processing device will be described. However, the present invention can be applied to any device that can generate an image in which a three-dimensional model is drawn to which a captured image is applied as a texture. Such electronic devices can include print sticker machines, video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, etc. It is to be noted that the implementation of the present invention is not limited to these devices, and it goes without saying that the present invention can be implemented in other devices as well.

[0011] In this specification, a 3D model to which a captured image is applied as a texture is placed in a 3D space, and an image generated by drawing the 3D space from a drawing viewpoint set in the space is referred to as a "virtual viewpoint image." A virtual viewpoint image includes an area in which a 3D model of an object (main subject) that is the main drawing target is drawn with a texture generated from a captured image obtained by actually capturing an image of the main subject.

[0012] <Digital camera configuration> FIG. 1 is a block diagram illustrating the hardware configuration of a digital camera 100 according to this embodiment.

[0013] The imaging element 11 may be, for example, a CCD (charge-coupled device) or a CMOS sensor (complementary metal-oxide semiconductor) having a primary-color Bayer array color filter. The imaging element 11 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A signal having a voltage according to the amount of electric charge accumulated during an exposure period is read out from each pixel, thereby obtaining a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface by the imaging optical system 10.

[0014] Although details will be described later, each pixel of the image sensor 11 has multiple photoelectric conversion regions or photoelectric conversion elements, and a pair of parallax images can be acquired in one imaging from the image sensor 11. Based on the pair of parallax images, it is possible to perform autofocus detection using an image plane phase difference ranging method (phase difference AF) and generate distance information based on ranging using the same method.

[0015] Here, the imaging optical system 10 is a photographing lens included in the digital camera 100, and forms an optical image of a subject on an imaging surface of an image sensor 11. The imaging optical system 10 includes a plurality of lenses arranged along an optical axis 103. The plurality of lenses includes a focus lens 102 for adjusting a focal distance of the imaging optical system 10. The focus lens 102 is movable along the optical axis. The focus lens 102 is driven by the control unit 12 based on a defocus amount derived by the processing unit 14.

[0016] The imaging optical system 10 also has an aperture 104 whose aperture value (aperture) is adjustable. The aperture value of the aperture 104 is controlled by the control unit 12 based on the shooting conditions determined by, for example, automatic exposure control (AE). The aperture 104 may have a function of a mechanical shutter. The exit pupil 101 is an image of the open aperture when the imaging optical system 10 is viewed from the imaging element 11 side, and the position of the exit pupil 101 is shown in the figure.

[0017] In this specification, the direction parallel to the optical axis 103 is defined as the z direction, the defocus direction, or the depth direction, or a corresponding axis is provided. Also, the direction perpendicular to the optical axis 103 and parallel to the horizontal direction of the image sensor 11 is defined as the x direction, and the direction parallel to the vertical direction of the image sensor 11 is defined as the y direction, or a corresponding axis is provided.

[0018] Control unit 12 has one or more processors (hereinafter referred to as CPUs) capable of executing programs, and controls the operation of each block of digital camera 100. Specifically, control unit 12 reads out the operation programs of each block stored in ROM 21, for example, and loads and executes the programs in RAM 20 to control the operation of each block.

[0019] The ROM 21 is, for example, a rewritable non-volatile memory, and stores, in addition to the operation programs of each block, setting values ​​(parameters) required for the operation of each block, various GUI data, etc. On the other hand, the RAM 20 is a recording device such as a volatile memory used as a working area or for temporary information storage. The RAM 20 is used not only as an area for developing the operation programs, but also as a storage area for data output in the operation of each block and various data required for the operation of each block. The RAM 20 is also used as a working memory for the processing unit 14, a buffer memory for temporarily storing captured images obtained by imaging, a video memory for the display unit 17, etc.

[0020] The storage unit 15 is a recording medium for recording a data file that stores image data obtained by imaging. The storage unit 15 may be, for example, a combination of a memory card and a reader / writer thereof. The storage unit 15 may be capable of handling a plurality of recording media.

[0021] The processing unit 14 applies predetermined image processing to the analog image signal read from the imaging element 11, generates signals and image data according to the application, and acquires and / or generates various information. The processing unit 14 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the processing unit 14 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executing software. The processing unit 14 outputs the acquired or generated information and data to the control unit 12, the RAM 20, etc. according to the application. The processing unit 14 can also read and process a data file recorded in advance in the storage unit 15.

[0022] The processing applied by the processing unit 14 includes, for example, preprocessing, color interpolation, correction, data processing, special effect processing, distance image (distance information) generation, edge detection, drawing viewpoint setting, virtual viewpoint image generation, and the like. Here, the preprocessing can include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, and other processes. Color interpolation is applied when a color filter is provided on the imaging element, and interpolates the values ​​of color components not included in the individual pixel data constituting the image data. Color interpolation is also called demosaicing. Correction can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system 10 (image restoration), correction of the effect of peripheral light reduction of the imaging optical system 10, color correction, and other processes. Data processing can include area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), and other processes. Generation of image data for display and image data for recording is also included in the data processing. Special effect processing can include processing such as addition of a blur effect, color tone change, and relighting.

[0023] The distance image generation process includes a process of deriving the amount of parallax from the pair of parallax images, a process of converting the amount of parallax into the amount of defocus, and a process of generating the distance image based on the amount of defocus. Here, the distance image is two-dimensional information indicating the object distance corresponding to each pixel position. The edge detection process is a process of detecting an area in which the difference between adjacent distance values ​​in the semantic segmentation process or the distance image is large as the edge of a three-dimensional model. In the digital camera 100 of this embodiment, the detection result of the edge of the three-dimensional model is used when separating the distance information or the captured image into an area corresponding to the main object (principal object) and an area corresponding to the background. The virtual viewpoint image generation process generates a virtual viewpoint image by placing a three-dimensional model generated based on the captured image and the distance image in a virtual three-dimensional space and drawing it with respect to a drawing viewpoint (virtual camera) set in the space.

[0024] Note that these processes are merely examples of processes that the processing unit 14 can apply, and do not limit the processes that the processing unit 14 can apply.

[0025] The input unit 16 is a user interface provided in the digital camera 100, such as a dial, a button, a switch, a touch panel, etc. When the input unit 16 detects an operation input made to the user interface, it outputs a corresponding control signal to the control unit 12. When the control unit 12 receives the control signal, it performs control to execute the corresponding function or operation.

[0026] The display unit 17 is, for example, a display device such as a liquid crystal display or an organic EL display. By continuously capturing images and displaying the captured images on the display unit 17, the display unit 17 can function as an electronic viewfinder (EVF). The operation of causing the display unit 17 to function as an electronic viewfinder (EVF) is sometimes called a live view display or a through display. An image displayed on the display unit 17 by the live view display or through display is sometimes called a live view image or a through image.

[0027] Display unit 17 may be a touch display. When display unit 17 is a touch display, software keys may be realized by combining a touch panel with GUI parts displayed on display unit 17. When an operation input to a software key is detected, control unit 12 acquires a corresponding control signal in the same manner as input unit 16.

[0028] The communication unit 18 is a communication interface with an external device provided in the digital camera 100. Through the communication unit 18, the control unit 12 can communicate with the external device in accordance with one or more wired or wireless communication standards.

[0029] The motion sensor 19 generates a signal according to the motion of the digital camera 100. The motion sensor 19 may be, for example, a combination of an acceleration sensor that outputs a signal according to the motion in each of the x, y and z axial directions and a gyro sensor that outputs a signal according to the motion around each axis.

[0030] Detailed Configuration of the Imaging Sensor FIG. 2 shows a detailed configuration of the imaging element 11. FIG. 2(a) is a plan view of the pixel array of the imaging element 11 seen from the imaging surface side. As shown in the figure, in the pixel array, color filters in a primary color Bayer array are regularly arranged with pixel groups 210 of 2 rows x 2 columns as repeating units. As shown in the enlarged view, the pixel group 210 is arranged with red (R), green (G), and blue (B) color filters, and an image signal indicating any one of color information of R, G, or B is output from each pixel (photoelectric conversion element). Note that, as an example, in this embodiment, the color filters are described as being distributed in a primary color Bayer array as shown in the figure, but it will be easily understood that the implementation of the present invention is not limited to this, and color filters in a different array can be adopted.

[0031] In order to realize a ranging function of an imaging surface phase difference ranging method in the imaging element 11 of this embodiment, one pixel (photoelectric conversion element) is configured with multiple photoelectric conversion units lined up in the II' cross section of Fig. 2(a) in the horizontal direction of the imaging element 11. Each pixel of the imaging element 11 is configured with a light guide layer 213 including a microlens 211 and a color filter 212, and a light receiving layer 214 including a first photoelectric conversion unit 215 and a second photoelectric conversion unit 216, as shown in Fig. 2(b).

[0032] In the light guide layer 213, the microlenses 211 are configured to efficiently guide the light flux incident on the pixels to the first photoelectric conversion section 215 and the second photoelectric conversion section 216. The color filters 212 transmit light of a predetermined wavelength band, and only transmit light of any one of the above-mentioned R, G, and B wavelength bands and guide it to the first photoelectric conversion section 215 and the second photoelectric conversion section 216 in the subsequent stage.

[0033] Both the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 generate charges according to the amount of incident light. The image sensor 11 can selectively read out signals from one or both of the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 from each pixel. Hereinafter, a signal obtained by reading out from the first photoelectric conversion unit 215 may be referred to as an A signal, and a signal obtained by reading out from the second photoelectric conversion unit 216 may be referred to as a B signal. Also, a signal obtained by reading out from both the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 may be referred to as an A+B signal.

[0034] The first photoelectric conversion unit 215 and the second photoelectric conversion unit 216 included in one pixel each partially receive the light beam incident through the microlens 211 of the pixel. Therefore, an image composed of the A signal and an image composed of the B signal read out from the same pixel region form a parallax image pair. In other words, the image signal obtained based on the A signal of each pixel of the image sensor 11 and the image signal obtained based on the B signal are pupil division images related to the light beam that has passed through different pupil regions of the exit pupil of the imaging optical system 10. Therefore, by using the A signal and the B signal, the defocus amount can be derived based on the principle of phase difference AF. In other words, the A signal and the B signal can each be said to be a signal for focus detection.

[0035] On the other hand, the A+B signal corresponds to a signal output when the pixel of the image sensor has only one photoelectric conversion unit. Therefore, an analog image signal obtained based on the A+B signal of each pixel of the image sensor 11 is equivalent to an image signal obtained when an image sensor composed of the same number of pixels, each of which has one photoelectric conversion unit, is used.

[0036] The A signal can also be obtained by subtracting the B signal from the A+B signal. Similarly, the B signal can also be obtained by subtracting the A signal from the A+B signal. Therefore, the A signal, the B signal, and the A+B signal can be obtained by reading out the A+B signal and the A signal or the B signal from each pixel. The type of signal read out from the pixels may be controlled by the control unit 12.

[0037] In addition, in FIG. 2, an embodiment in which each pixel has two photoelectric conversion units arranged in the horizontal direction has been described, but the embodiment of the present invention is not limited to this. For example, the number of photoelectric conversion units provided in the overpixel may be three or more, such as a configuration in which four photoelectric conversion units are arranged, two in the horizontal direction and two in the vertical direction. In addition, the pixel dedicated to generating the A signal and the pixel dedicated to generating the B signal may be arranged in multiple pairs distributed in the pixel array. The image sensor 11 may adopt any known configuration compatible with image plane phase difference AF.

[0038] 《Distance measurement principle of imaging plane phase difference distance measurement method》 Next, the principle of deriving the subject distance based on the group of pupil-divided images output from the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216, which are used to form a distance image in the processing unit 14, will be described with reference to Figures 3 and 4. Figure 3(a) is a schematic diagram showing the exit pupil 101 of the imaging optical system 10 and a light beam incident on the first photoelectric conversion unit 215 of a pixel in the imaging element 11. Figure 3(b) is a schematic diagram showing a light beam similarly incident on the second photoelectric conversion unit 216.

[0039] The microlens 211 shown in Fig. 3(a) and (b) is disposed so that the exit pupil 101 and the light receiving layer 214 are in an optically conjugate relationship. A light beam that has passed through the exit pupil 101 of the imaging optical system 10 is collected by the microlens 211 and guided to the first photoelectric conversion unit 215 or the second photoelectric conversion unit 216. At this time, as shown in Fig. 3(a) and (b), the light beams that have passed through different pupil regions are mainly incident on the first photoelectric conversion unit 215 and the second photoelectric conversion unit 216, respectively. Specifically, the light beam that has passed through the first pupil region 310 is incident on the first photoelectric conversion unit 215, and the light beam that has passed through the second pupil region 320 is incident on the second photoelectric conversion unit 216.

[0040] A signal and a B signal can be obtained from each of a plurality of pixels arranged in the horizontal direction with the pixel of interest at the center. At this time, the relative positional shift amount (phase difference or parallax amount, hereinafter referred to as parallax amount) between the image signal based on the A signal row (A image) and the image signal based on the B signal row (B image) is a value according to the defocus amount of the pixel of interest. The relationship between the parallax amount and the defocus amount will be described with reference to Figs. 4(a), (b), and (c). Figs. 4(a), (b), and (c) are schematic diagrams showing the relationship between the image sensor 11 and the imaging optical system 10 of this embodiment. In the figure, reference numeral 410 denotes a first light flux passing through the first pupil region 310, and reference numeral 420 denotes a second light flux passing through the second pupil region 320.

[0041] FIG. 4(a) shows a state at the time of focusing, in which the first light beam 410 and the second light beam 420 are converged on the imaging surface of the imaging element 11. At this time, the parallax amount between the image A formed by the first light beam 410 and the image B formed by the second light beam 420 is 0. In FIG. 4(b), the first light beam 410 and the second light beam 420 are converged on the object side (negative side of the z axis) from the imaging surface. At this time, the parallax amount between the image A and the image B is not 0, but has a negative value. In FIG. 4(c), the first light beam 410 and the second light beam 420 are converged on the back side (positive side of the z axis) from the imaging surface as viewed from the object side. At this time, the parallax amount between the image A and the image B has a positive value.

[0042] In this way, the amount of parallax between the images A and B has a sign according to the positional relationship between the imaging surface and the position where the first light beam 410 and the second light beam 420 are converged, and indicates a value according to the magnitude of the defocus amount. Therefore, the amount of correlation is derived while shifting the images A and B relatively, and the amount of parallax between the images A and B can be obtained as the shift amount that maximizes the amount of correlation.

[0043] <Distance Image Generation Processing> Here, the process in which processing unit 14 generates a distance image based on the A and B signals obtained by imaging will be described with reference to the flowchart in Fig. 5. The process corresponding to this flowchart can be performed by control unit 12 reading out a corresponding processing program stored in ROM 21, for example, and loading it into RAM 20 (not shown) for execution, thereby causing processing unit 14 to perform the process. Note that when performing this process, it is assumed that the A and B signals associated with each pixel of image sensor 11 are stored in RAM 20.

[0044] In S501, the processing unit 14 corrects the light amount of the A signal and the B signal. In particular, in pixels with large image heights, the difference in shape between the first pupil region 310 and the second pupil region 320 becomes large due to vignetting in the imaging optical system 10, causing a difference in the light amount between the A signal and the B signal. The processing unit 14 applies a correction value according to the pixel position to the A signal and the B signal to correct the difference in the light amount between the A signal and the B signal. The correction value may be, for example, a value stored in advance in the ROM 21.

[0045] In S502, the processing unit 14 applies noise reduction processing to the A signal and the B signal. In general, the higher the spatial frequency, the more noise components there are, so the processing unit 14 applies a low-pass filter, the pass rate of which decreases as the spatial frequency increases, to the A signal and the B signal. Note that, due to manufacturing errors in the imaging optical system 10, good results may not be obtained by the light quantity correction in S501. Therefore, in this step, the processing unit 14 can apply a band-pass filter that blocks DC components and has a low pass rate for high-frequency components.

[0046] In S503, the processing unit 14 derives the amount of parallax between the A signal and the B signal. The processing unit 14 generates the A signal sequence and the B signal sequence from, for example, a horizontally continuous pixel sequence including the pixel of interest. The processing unit 14 then derives the amount of correlation while relatively shifting the A signal sequence and the B signal sequence. The amount of correlation may be, for example, NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), or SAD (Sum of Absolute Difference).

[0047] The processing unit 14 derives the shift amount at which the correlation between the A signal sequence and the B signal sequence is maximized in units of less than a pixel, and sets the shift amount as the amount of parallax at the pixel of interest. The processing unit 14 derives the amount of parallax at each pixel position while changing the position of the pixel of interest. Note that the amount of parallax between the A signal and the B signal may be detected by any other known method. The resolution of each signal sequence when deriving the amount of parallax may be lower than the resolution of the captured image.

[0048] In S504, the processing unit 14 converts the detected amount of parallax into a defocus amount. Since the detected amount of parallax indicates a value according to the amount of defocus, it can be converted into the amount of defocus by applying a predetermined conversion coefficient. If the amount of parallax is d and the conversion coefficient is K, the amount of defocus ΔL is expressed as follows: ΔL = K × d (1) The processing unit 14 converts the amount of parallax at each pixel position into a defocus amount, thereby generating two-dimensional information (defocus image) indicating the defocus amount corresponding to each pixel position.

[0049] In S505, the processing unit 14 generates a distance image based on the defocused image generated in S504 and the shooting information indicating various parameters related to the image capture. As described above, the distance image is two-dimensional information in which each pixel indicates the subject distance corresponding to the pixel position, and is also called a distance map, a depth map, a depth image, or the like.

[0050] The object distance can be derived (distance acquisition) from the defocus amount using the principle shown in Fig. 6. In Fig. 6, OBJ is the object plane, IMG is the image plane, H is the front principal point, H' is the rear principal point, f is the focal length of the photographic optical system (lens), S is the distance from the object plane to the front principal point, and S' is the distance from the rear principal point to the image plane. Also, ΔS' is the defocus amount, and ΔS is the relative distance on the object side according to the defocus amount. In the figure, the dashed line indicates the optical axis, the solid line indicates the imaging light beam, and the dashed line indicates the defocus light beam.

[0051] In imaging by a lens, 1 / S+1 / S'=1 / f (2) It is known that the following holds true. In addition, when the lens is defocused, the following equation (2) is modified: 1 / (S+ΔS)+1 / (S'+ΔS')=1 / f ··· (3) holds. Here, S and f at the time of focusing can be obtained from the shooting information. Therefore, S' can be derived from the formula (1). Also, the defocus amount ΔS' can be obtained, for example, by autofocus detection (AF) using a phase difference detection method. This makes it possible to derive ΔS from the formula (3), and the distance S to the object plane OBJ, i.e., the subject distance, can be derived.

[0052] In this process, the distance image is acquired using the defocus amount, but the distance image may be acquired using other known methods. For example, the focus lens position at which the contrast evaluation value is maximized is derived for each pixel, thereby acquiring the subject distance for each pixel. Also, distance information for each pixel can be derived based on the correlation between the blur amount and distance from image data obtained by photographing the same scene multiple times with different focus distances and the point spread function (PSF) of the optical system. These techniques are disclosed in, for example, Japanese Patent Application Laid-Open No. 2010-177741 and the specification of U.S. Patent No. 4,965,840. Also, in an embodiment in which a parallax image pair can be acquired, the subject distance for each pixel can be derived by a method such as stereo matching.

[0053] "Generation of Virtual Viewpoint Images" In the digital camera 100 of this embodiment, a three-dimensional model corresponding to the captured scene is constructed based on the captured image (A+B signal) and the distance image obtained by imaging, and this is placed and drawn in a three-dimensional space (world), thereby generating a virtual viewpoint image related to the captured scene. That is, since unevenness information in the imaging direction can be obtained from the distance image, a three-dimensional shape can be formed based on the distance image. In addition, by applying an image of the subject area in the captured image as a texture to the three-dimensional shape, the subject of the captured scene can be reproduced three-dimensionally, and a virtual viewpoint image in which the subject can be viewed from a direction different from the imaging direction can be generated.

[0054] In addition, in this embodiment, a three-dimensional model is constructed in response to a captured image and a distance image obtained by imaging, but the present invention is not limited to this. It will be easily understood that the subject model can be constructed using a captured image and a distance image that have been captured in advance and stored in, for example, the storage unit 15.

[0055] <Constructing a 3D model> The construction of a three-dimensional model based on distance information will be described in detail below.

[0056] The spatial resolution of the captured image and the distance image changes depending on the imaging settings. Therefore, the processing unit 14 needs to refer to the shooting information when forming the three-dimensional shape of the subject based on the distance image. The processing unit 14 converts the subject distance of each pixel appearing in the distance image into coordinate values ​​(spatial coordinates) of the world coordinate system based on the focal length and focus position information obtained from the shooting information. This makes it possible to map points corresponding to each pixel in space. The mapped point cloud forms a three-dimensional shape corresponding to the object within the imaging angle of view.

[0057] For example, a distance image obtained by imaging a cylinder as shown in FIG. 7(a) from the side of the cylinder as a subject is as shown in FIG. 7(b). The shading in the distance image in FIG. 7(b) indicates that the subject distance is longer (farther) as the color becomes lighter. Therefore, in the distance image in FIG. 7(b), the subject distance is shortest at the center in the region corresponding to the cylinder, and the subject distance becomes longer as the distance from the center increases to the left and right. In the illustrated embodiment, no objects other than the cylinder exist in the imaging scene, and the region other than the cylinder is the background that is farther away than the cylinder. When each pixel of such a distance image is spatially mapped, a point cloud is formed that forms a three-dimensional shape 701 corresponding to the subject and a three-dimensional shape 702 corresponding to the background, as shown in FIG. 7(c). Here, since there is no information on the subject distance for the background that does not appear in the distance image in the imaging scene (background outside the imaging angle of view), the three-dimensional shape is formed only within the imaging angle of view.

[0058] The processing unit 14 polygonizes the three-dimensional shape thus formed so as to handle it as a three-dimensional model. For example, the processing unit 14 can convert the three-dimensional shape into a polygon mesh by defining a surface based on the spatial coordinates of any three adjacent points of the three-dimensional shape. In order to make the object model to be generated into a smooth shape, a filter process may be applied to the distribution of object distances in the distance image and the three-dimensional shape before polygonization. The processing unit 14 can obtain a three-dimensional model in which small irregular changes are smoothed by applying, for example, a median filter.

[0059] The three-dimensional model obtained by polygonization can be converted into two-dimensional structured data, for example, by using a predictive coding technique for two-dimensional images, in order to reduce the amount of data. In this embodiment, the processing unit 14 performs polygonization and converts the data into two-dimensional structured data, which is used as a subject model in the generation process of a virtual viewpoint image, which will be described later. However, the present invention is not limited to this. For example, polygonization is not essential, and the three-dimensional shape of the subject may be handled in a point cloud format. In addition, in the case of performing coding, any method can be used to express the three-dimensional shape of the object, as long as the data format is applicable to a known predictive coding technique for two-dimensional images.

[0060] The three-dimensional model also includes information on the texture (RGB) to be applied. The captured image (A+B signals) obtained corresponding to the distance image is mapped and applied to the three-dimensional model. For example, the texture to be applied to each polygon is specified by the pixel positions corresponding to the three points that form the polygon. That is, the image of the triangular area of ​​the captured image that is specified by the pixel positions in the distance information of the three points that form the polygon becomes the texture to be applied to the polygon.

[0061] In the following description, it is assumed that in the digital camera 100 of this embodiment, the processing unit 14 generates three-dimensional model data, including three-dimensional shape, polygon information, and texture information, from a captured image and a corresponding distance image.

[0062] In this embodiment, in order to reduce the amount of calculations involved in generating a virtual viewpoint image and to facilitate understanding of the invention, the following description will be given assuming that a three-dimensional model of unevenness based on a distance image is not constructed for the background of the captured scene. The three-dimensional model of the background is constructed by using a general-purpose plate-shaped three-dimensional object (flat object) with a size corresponding to the captured angle of view, and applying an image of the background area in the captured image as a texture to the object. Therefore, the construction of a three-dimensional model based on a distance image can be simplified by limiting the formation of a three-dimensional shape based on the distance image to pixels in an area corresponding to the object in the distance image identified by edge detection processing. In the following description, the three-dimensional model of the object may be referred to as a "object model" and the three-dimensional model of the background may be referred to as a "background model."

[0063] In the following description, the direction opposite to the imaging direction of a captured image used to construct each three-dimensional model, or the direction opposite to the depth direction of a range image, is defined as the "front direction" of the three-dimensional model.

[0064] <drawing> The processing unit 14 generates a virtual viewpoint image by arranging the three-dimensional model thus constructed in a three-dimensional space as shown in Fig. 8(a) and drawing the three-dimensional space from the set viewpoint. In the example shown in the figure, a subject model 801 relating to a person, a background model 802, and a light source 803 set for the three-dimensional space are arranged in the three-dimensional space, and a drawing viewpoint 804 for drawing the three-dimensional space is also arranged in the three-dimensional space.

[0065] In the example of FIG. 8, subject model 801 and background model 802 are three-dimensional models constructed based on a captured image and a distance image obtained when digital camera 100 faces a person's face (front of the person). Subject model 801 and background model 802 are arranged so that their front directions face the negative z-axis direction in three-dimensional space. In addition, to facilitate understanding of the invention, subject model 801 is arranged so that a reference coordinate (e.g., a center coordinate of subject model 801) defined for subject model 801 becomes the origin coordinate of the three-dimensional space. Hereinafter, rendering viewpoint 804 may be referred to as a virtual camera, meaning that it captures the three-dimensional space. The rendering viewpoint 804 may be set based on, for example, an operational input by a user.

[0066] As shown in the figure, in the background model 802, a captured image relating to the background is applied as a texture to a flat object. The texture applied to the background model 802 is described as being generated by extracting an image of a region corresponding to the background from a captured image corresponding to the distance image used to generate the subject model 801, but the embodiment of the present invention is not limited to this. The texture applied to the background model 802 can be, for example, a captured image captured in a state where the subject does not exist in the captured scene, an image generated based on another captured image relating to the captured scene, a captured image relating to a different captured scene, or the like. Alternatively, any background image that is not a real image can be used as the texture applied to the background model 802.

[0067] The light source 803 may be installed arbitrarily. By disposing the light source 803 in the three-dimensional space, it is possible to generate a virtual space image in which a relighting effect is applied to the subject model 801 in particular. On the other hand, even if the light source 803 is not installed, it is also possible to generate a virtual viewpoint image in a suitable mode by setting the brightness of the three-dimensional space to be uniform, for example.

[0068] For example, when the virtual camera 804 is set as shown in FIG. 8(a), a virtual viewpoint image as shown in FIG. 8(b) can be generated. In the example of FIG. 8(a), since the direction facing each three-dimensional model, i.e., the z-axis positive direction, is the imaging direction of the captured image to which the texture is applied, the processing unit 14 generates a virtual viewpoint image as if the subject was observed from a direction different from the imaging direction. In other words, by setting a virtual camera with a line of sight in the z-axis positive direction so as to face the three-dimensional model, the virtual viewpoint image generated by the processing unit 14 shows a two-dimensional representation of the same content as the captured image. Hereinafter, the line of sight direction of the virtual camera that generates such a virtual viewpoint image showing a two-dimensional representation of the same content as the captured image may be referred to as a "reference direction".

[0069] For ease of understanding of the invention, regardless of the coordinates set for virtual camera 804, the line of sight is set in a direction from the coordinates toward the reference coordinates (origin of three-dimensional space) of subject model 801. That is, the line of sight of virtual camera 804 can be defined based on the deflection angle (elevation angle (angle around x-axis) and azimuth angle (angle around y-axis)) of the coordinates of virtual camera 804 with the front direction of subject model 801 as a reference. Therefore, in the following, the coordinates (position in three-dimensional space) for setting the virtual camera (rendering viewpoint) are specified based on the front direction of the subject model. That is, the coordinates of the virtual camera are specified, for example, by the distance from the subject model, and the elevation angle and azimuth angle formed with the front direction of the subject model, and the line of sight is the direction from the coordinates toward the subject model.

[0070] <Specifying spatial domain> However, when a rendering viewpoint is arbitrarily set in a three-dimensional space in which a three-dimensional model is arranged, a suitable virtual viewpoint image is not necessarily generated in all cases.

[0071] For example, in the reference direction, polygon 901 of the subject model that appears as shown in FIG. 9(a) changes in area to polygon 902 in FIG. 9(b) when viewed from a virtual camera placed to the side of the subject model. In the example of FIG. 9(b), the virtual camera is set at a position where the elevation angle is the same as that of the virtual camera set in FIG. 9(a) and only the azimuth angle is different, so that the vertical coordinates where polygons 901 and 902 are distributed in the virtual viewpoint image are the same, but the horizontal width changes. In contrast, the texture applied to these polygons is an image extracted from the captured image, and its spatial resolution is constant regardless of the rendering viewpoint to be set. Therefore, the larger the display area of ​​a polygon in the virtual viewpoint image, the more the texture applied to the polygon is drawn in a stretched state. As a result, the obtained virtual viewpoint image may cause the user to perceive a strangeness in viewing.

[0072] In the digital camera 100 of this embodiment, the extent to which such a stretched texture is visible in the line of sight direction is evaluated to see whether the user perceives an unnatural appearance. The processing unit 14 derives the dot product of the normal vector p of each polygon constituting the subject model and the line of sight direction vector v, and derives the average value of the dot products for all polygons as the evaluation value. Here, the dot product of the normal vector p and the line of sight direction vector v is expressed as follows, where rp and rv are the lengths of these vectors and θ is the angle between these vectors. p v = rp × rv × cosθ It can be derived as follows.

[0073] When the evaluation value falls below a predetermined threshold, the processing unit 14 determines that a virtual viewpoint image that causes a sense of discomfort in viewing will not be generated for the gaze direction of the evaluation target. Here, in an aspect in which the gaze direction of the virtual camera is set in a direction toward the reference coordinates of the subject model as in this embodiment, the evaluated gaze direction can uniquely identify the deviation angle (elevation angle and azimuth angle) with respect to the front direction of the subject model. Therefore, by deriving the evaluation value of the gaze direction, the processing unit 14 can determine whether or not a suitable virtual viewpoint image will be generated for each deviation angle with respect to the front direction of the subject model.

[0074] The processing unit 14 can specify an angle range in which a suitable virtual viewpoint image is generated for the three-dimensional space, that is, a spatial region (a partial region in the three-dimensional space) in which a rendering viewpoint can be set, by selecting (sampling) a plurality of types of line-of-sight directions and deriving an evaluation value. In other words, the virtual viewpoint image generated by setting a virtual camera in the spatial region is in a state in which the user is unlikely to perceive discomfort in viewing.

[0075] Fig. 10 illustrates an example of a spatial region in which a rendering viewpoint can be set, as specified in this embodiment. Fig. 10(a) illustrates the distribution of the spatial region in the yz plane, where the hatched region indicates the angular range 1001 of the elevation angle of the spatial region (based on the front direction of the subject model 801). Fig. 10(b) illustrates the distribution of the spatial region in the xz plane, where the hatched region indicates the angular range 1002 of the azimuth angle of the spatial region (based on the front direction of the subject model 801).

[0076] <<Virtual viewpoint image generation process>> A specific process for generating a virtual viewpoint image executed by the digital camera 100 of this embodiment having such a configuration will be described with reference to the flowchart in Fig. 11. The process corresponding to the flowchart can be realized by the control unit 12 reading out a corresponding processing program stored in, for example, the ROM 21, and executing the program by expanding it in the RAM 20. This process will be described as being started when, for example, an instruction to generate a virtual viewpoint image is given for an arbitrary three-dimensional model. Hereinafter, the three-dimensional model for which an instruction to generate a virtual viewpoint image is given will be referred to as a target three-dimensional model.

[0077] In S1101, the control unit 12 acquires a target 3D model. The target 3D model may be acquired by acquiring a 3D model generated by the processing unit 14, or may be acquired by reading corresponding data stored in the storage unit 15 or the like.

[0078] In S1102, the control unit 12 acquires shooting information related to the captured image used to generate the target 3D model. The shooting information may include, for example, the distance S from the digital camera 100 to the subject at the time of capturing the image, the resolution (number of pixels) of the captured image, the focal length set in the focus lens 102 at the time of capturing the image, the ISO sensitivity at the time of capturing the image, and the like.

[0079] In S1103, the control unit 12 acquires parameters related to the virtual viewpoint image to be generated. The parameters related to the virtual viewpoint image include, for example, the resolution (number of pixels) of the virtual viewpoint image and the focal length of the virtual camera used for generation. In this embodiment, these parameters are described as being held as constants, but may be set based on, for example, an operational input from the user.

[0080] In S1104, the processing unit 14 places the target 3D model acquired in S1101 in the 3D space of the rendering target under the control of the control unit 12. In this embodiment, the target 3D model includes a subject model and a background model, and the processing unit 14 places the subject model and the background model in the 3D space.

[0081] In S1105, the processing unit 14 identifies a spatial region in which the rendering viewpoint can be set under the control of the control unit 12. More specifically, the processing unit 14 identifies the spatial region as a relative angle range based on the front direction of the subject model, based on the normal information of the subject model.

[0082] In S1106, under the control of the control unit 12, the processing unit 14 sets a rendering viewpoint in the spatial region specified in S1105, and sets a virtual camera at the rendering viewpoint with the viewing direction directed toward the subject model.

[0083] In S1107, the processing unit 14, under the control of the control unit 12, generates a virtual viewpoint image by drawing a three-dimensional space for the virtual camera, and completes the process of generating the virtual viewpoint image.

[0084] As described above, the image processing device of this embodiment can ensure a predetermined image quality for the generated virtual viewpoint image. More specifically, the image processing device sets a spatial area in which a drawing viewpoint can be set, excluding positions in a three-dimensional space that may cause a user to perceive an unnatural appearance of the three-dimensional model, based on normal information of the three-dimensional model to be drawn. Then, the image processing device sets a virtual camera limited to the spatial area to generate a virtual viewpoint image, thereby reducing the occurrence of unnatural two-dimensional expressions and ensuring a predetermined image quality.

[0085] In the present embodiment, the background model is constructed by applying a texture related to the background to a flat object, but the present invention is not limited to this. The background model may be constructed by applying a texture related to the background to a celestial sphere object, for example.

[0086] [Variation 1] In the above-described embodiment, in identifying a spatial region in which a rendering viewpoint can be set, the dot product of the normal vector of each polygon constituting the subject model and the vector of the viewing direction for the viewing direction to be evaluated is derived, and the average value of the dot products for all polygons is derived as an evaluation value. Then, for viewing directions for which the evaluation value is below a predetermined threshold, an angle range for the spatial region is identified as one in which no discomfort is perceived in the generated virtual viewpoint image.

[0087] Incidentally, a three-dimensional model constructed based on a captured image and a distance image relating to a specific imaging direction shows a two-dimensional expression similar to that of the captured image by drawing it with respect to a virtual camera facing a reference direction. That is, for each polygon constituting the three-dimensional model, an image of a region large enough to be shown by projecting the polygon in the reference direction is extracted from the captured image and applied as a texture. On the other hand, since each polygon does not necessarily face the reference direction, it may have a larger area than the image applied as a texture. That is, when a polygon extends in the depth direction (z direction: reference direction), the image applied as a texture is stretched (elongated) more than the image of the corresponding region in the captured image. Such elongation may cause deterioration in image quality such as image distortion and a decrease in spatial resolution, so that when the corresponding polygon is drawn in a direction different from the reference direction, the virtual viewpoint image may not guarantee suitable image quality. For example, a virtual viewpoint image in which such a polygon appears with a larger area than the virtual viewpoint image relating to the reference direction may cause the user to perceive discomfort in viewing.

[0088] Such polygons extending in the depth direction may appear particularly at the end of the subject model, that is, in the edge region (the boundary between the subject region and the background region) in the region corresponding to the subject in the captured image or distance image. In other words, since the texture may be applied in a particularly stretched state to the polygons distributed in the part corresponding to the edge region, how the polygon appears in the line of sight of the evaluation subject may be important in terms of whether the user perceives an aesthetic discomfort.

[0089] Therefore, in this modification, the processing unit 14 limits the polygons for deriving the inner product for the gaze direction of the evaluation target to those distributed in the parts corresponding to the edge regions, rather than all the polygons constituting the subject model. In this way, it is possible to more accurately derive the degree to which the generated virtual viewpoint image makes the user perceive an uncomfortable viewing sensation.

[0090] [Embodiment 2] In the above-described embodiment and modified example, the processing unit 14 specifies the spatial region in which the rendering viewpoint for generating the virtual viewpoint image can be set as a relative angle range based on the subject model, but the present invention is not limited to this. The display area of ​​the polygon in the virtual viewpoint image changes not only depending on the direction in which the polygon is rendered (the line of sight direction of the virtual camera) but also depending on the distance between the polygon and the virtual camera. That is, the closer the virtual camera is to the polygon, the larger the display area of ​​the polygon in the generated virtual viewpoint image. For this reason, in this embodiment, the processing unit 14 further determines the closest distance (shortest distance) to the three-dimensional model for the spatial region in which the rendering viewpoint can be set.

[0091] The shortest distance d2 can be derived using the resolution (number of pixels) k1 of the captured image used to construct the 3D model, the focal length f1 and shooting distance d1 set during capture, the resolution (number of pixels) k2 of the virtual viewpoint image to be generated, and the focal length f2 set for the virtual camera. Specifically, the shortest distance d2 is d2=(k2 / k1)×(f2 / f1)×d1 The shortest distance d2 derived in this way makes the spatial resolution of the image appearing in the texture applied to the polygon in the virtual viewpoint image generated by setting the virtual camera at that distance equal to the spatial resolution of the same image in the captured image. In other words, the image of the area corresponding to each polygon in the captured image appears in the virtual viewpoint image with the number of pixels of that area or less, so that the spatial resolution of the captured image can be guaranteed for the image appearing in that area.

[0092] In this embodiment, in order to facilitate understanding of the invention, the resolution k2 of the generated virtual viewpoint image and the focal length f2 set in the virtual camera are described as being fixed. In this embodiment, the shortest distance d2 can be uniquely specified in response to acquisition of information related to imaging. That is, the shortest distance d2 can be specified based on the resolution of the captured image and the focal length and shooting distance set at the time of imaging, and therefore, the shortest distance d2 can be specified by the spatial resolution of the image in the captured image.

[0093] Therefore, the processing unit 14 specifies the spatial region in which the rendering viewpoint can be set as a distance range farther away from the subject model than the shortest distance d2 as shown in Figures 12(a) and 12(b). This allows the processing unit 14 to control the spatial resolution of the subject image in the generated virtual viewpoint image so that it does not fall below the spatial resolution of the captured image.

[0094] [Variation 2] In the above-mentioned second embodiment, only the shortest distance is set as the distance range of the spatial area in which the rendering viewpoint can be set, but the implementation of the present invention is not limited to this. For example, the distance range of the spatial area may be set to the farthest distance (longest distance) from the subject model in order to ensure the visibility (display size) of the subject image in the virtual viewpoint image. In other words, the farther the rendering viewpoint is set from the subject model, the smaller the occupancy rate of the area in the virtual viewpoint image in which the subject model is rendered becomes, which is not suitable for viewing the subject model.

[0095] The maximum distance d3 can be determined based on a virtual viewpoint image generated by, for example, selecting (sampling) a plurality of distances from the subject model in a range equal to or greater than the shortest distance, and setting a rendering viewpoint for each of these distances in advance. Specifically, the processing unit 14 derives the occupancy rate of the area in which the subject model is rendered for the virtual viewpoint image at each distance, and determines the maximum distance at which the occupancy rate exceeds a predetermined threshold as d3. In this way, the processing unit 14 can specify a spatial area in which the rendering viewpoint can be set based on an angle range and a distance range (from the shortest distance to the longest distance) as shown in Figs. 13(a) and (b). Therefore, the virtual viewpoint image generated in relation to the rendering viewpoint set in the spatial area can ensure that the subject image occupies a certain area or more while reducing rendering expressions that cause the user to perceive discomfort in viewing.

[0096] The method for determining the maximum distance is not limited to this, and a fixed value may be set, for example.

[0097] [Variation 3] However, when a three-dimensional model is constructed based on the captured images and distance images obtained in a specific imaging direction as described above, the surfaces (polygons) constituting the three-dimensional model are limited to those that appear in these images. For example, as shown in FIG. 8(a), from the captured images and distance images obtained with the digital camera 100 facing the face of a person, an object model 801 having only surfaces related to the front of the person is constructed, and there are no surfaces related to the back of the person that do not appear in the captured images. In other words, since a three-dimensional shape related to surfaces that do not appear in the captured images and distance images obtained in a specific imaging direction cannot be formed, the constructed object model is not closed. Therefore, depending on the rendering viewpoint that is set, the back side of the object model, that is, the surface corresponding to the back side rather than the front side of the object model, may be rendered.

[0098] When the back side of the subject model is drawn, the subject image of the virtual viewpoint image may have an unnatural shape or appearance, which may cause the user to perceive an unnatural feeling in the visual sense. Therefore, in the digital camera 100 of this modified example, the processing unit 14 selects (samples) multiple types of coordinates within a spatial region specified based on, for example, the above-mentioned angle range or distance range. Then, based on a virtual viewpoint image generated by previously setting a drawing viewpoint having a line of sight direction toward the subject model at the coordinates, the processing unit 14 derives the occupancy rate of the region in the virtual viewpoint image where the back side of the subject model is drawn.

[0099] Here, whether or not the rendered polygon indicates the back side of the subject model can be determined based on whether or not the normal direction of the polygon has a component in the line of sight direction and in the forward direction. Simply put, the processing unit 14 acquires the normal vector of the polygon from the normal information related to the subject model, and can determine that the back side of the polygon is rendered if the dot product of the normal vector and the vector in the line of sight direction is positive.

[0100] The processing unit 14 specifies an area to be further excluded from the spatial area specified based on the angle range and distance range based on the occupancy rate derived for each sampling coordinate, and finally specifies a spatial area in which a rendering viewpoint can be set. Specifically, the processing unit 14 can specify the final spatial area by excluding sampling coordinates whose occupancy rate exceeds a predetermined threshold. At this time, the processing unit 14 may three-dimensionally map the occupancy rate of each sampling coordinate, specify an area whose occupancy rate is likely to exceed a predetermined threshold based on the distribution tendency, and exclude the area from the spatial area specified based on the angle range and distance range.

[0101] In a virtual viewpoint image generated based on a rendering viewpoint set in the final spatial region identified in this manner, the occurrence of expressions that cause the user to perceive an aesthetically strange feeling due to the back faces of polygons being rendered is reduced.

[0102] [Variation 4] However, in a mode in which textures to be applied to a subject model and a background model are extracted and generated from a captured image obtained in a specific imaging direction, the texture to be applied to the background model may include an area that does not show significant information. That is, since the image of the background does not appear in the captured image where the image of the subject appears, the texture to be applied to the background model cannot be obtained. In other words, since the background that is blocked by the subject during imaging does not appear in the captured image, the texture to be applied to the background model generated by excluding the image (pixel group) of the area corresponding to the subject from the captured image cannot include significant image information for that area.

[0103] Therefore, a virtual viewpoint image generated by drawing a subject model and a background model to which such a texture has been applied will have a appearance as shown in Figure 14(a) in the reference direction, but may contain pixels 1401 with missing image information as shown in Figure 14(b) in a different viewing direction.

[0104] As shown in Fig. 8(a), in an aspect in which a background model is constructed using a flat object, for example, by associating the coordinates of the four corners of the texture related to the background with the vertices of the four corners constituting the flat object, the information of the texture to be applied can be simply defined. That is, in such an aspect, it is preferable that the texture related to the background model is configured as a rectangular texture. For this reason, the texture related to the background model can be generated, for example, by replacing the pixel values ​​of pixels in an area corresponding to the subject in a captured image with meaningless values.

[0105] Therefore, in the process of drawing the background model, even in an area of ​​the flat object corresponding to a pixel that does not contain significant image information in the texture related to the background, a process of applying the texture related to the background is executed. That is, even in such an area, a process of deriving the pixel value of the corresponding pixel of the virtual viewpoint image is executed by referring to the pixel value information of the corresponding area of ​​the texture. However, since the texture related to the background does not contain significant pixel information in the area, the area can be substantially regarded as an area to which the image of the area corresponding to the background in the captured image is not applied, and the area appears in the virtual viewpoint image as a pixel 1401 with missing image information. Hereinafter, such an area of ​​the background model (flat object), i.e., an area to which the texture related to the background is not substantially applied, is referred to as an "occlusion area".

[0106] 14B, when a pixel 1401 corresponding to an occlusion region is included in a virtual viewpoint image, a rendering expression (such as image discontinuity) that deviates from the captured scene occurs around the pixel. Therefore, the more pixels 1401 corresponding to an occlusion region included in the virtual viewpoint image are included in the virtual viewpoint image, the more likely the user is to perceive a strange viewing sensation.

[0107] In the digital camera 100 of this modification, the processing unit 14 selects (samples) a plurality of types of coordinates within a spatial region specified based on the above-mentioned angle range or distance range, for example. Then, the processing unit 14 derives the occupancy rate of pixels 1401 corresponding to the occlusion region in a virtual viewpoint image generated when a rendering viewpoint is set at the selected coordinates, based on the positional relationship between a subject model and a background model (particularly the occlusion region) in a three-dimensional space.

[0108] The processing unit 14 specifies an area to be further excluded from the spatial area specified based on the angle range and distance range based on the occupancy rate derived for each sampling coordinate, and finally specifies a spatial area in which a rendering viewpoint can be set. Specifically, the processing unit 14 can specify the final spatial area by excluding sampling coordinates whose occupancy rate exceeds a predetermined threshold. At this time, the processing unit 14 may three-dimensionally map the occupancy rate of each sampling coordinate, specify an area whose occupancy rate is likely to exceed a predetermined threshold based on the distribution tendency, and exclude the area from the spatial area specified based on the angle range and distance range.

[0109] In a virtual viewpoint image generated based on a rendering viewpoint set in the final spatial region identified in this manner, pixel 1401 corresponding to the occlusion region is rendered, thereby reducing the occurrence of expressions that cause the user to perceive an unnatural viewing experience.

[0110] [Variation 5] Incidentally, the positional relationship between the subject model and the background model in the three-dimensional space is determined so that when the virtual camera is set under the same conditions as when the captured image used to construct the subject model and the background model is captured, a virtual viewpoint image showing the same rendering as the captured image is generated. Therefore, depending on the position and line of sight of the set virtual camera, the generated virtual viewpoint image may include an area where none of the three-dimensional models are rendered. In addition, a virtual viewpoint image including such an area may be generated not only by changing the position and line of sight of the virtual camera, but also by changing, for example, the focal length set for the virtual camera.

[0111] 15(a) and (b) show examples of virtual viewpoint images generated by changing the focal length of the virtual camera for the same line of sight. In the virtual viewpoint image of FIG. 15(a), a wider focal length is set for the virtual camera than in the virtual viewpoint image of FIG. 15(b). That is, in the virtual viewpoint image of FIG. 15(b), a more telephoto focal length is set for the virtual camera than in the virtual viewpoint image of FIG. 15(a). In FIG. 15, the distance between the subject model and the virtual camera is adjusted so that the shooting magnification of the subject (display size of the subject image) is the same for the two virtual viewpoint images of (a) and (b) for comparison.

[0112] A virtual viewpoint image 1500 in FIG. 15(a) includes a subject region 1501 corresponding to a subject model, a background region 1502 corresponding to a background model, and a region where no three-dimensional model is drawn (hereinafter referred to as a sky region) 1503. As shown in the figure, the virtual viewpoint image 1500 has a rendering representation that is separated from the captured scene due to the presence of the region 1503, and may cause the user to perceive a sense of incongruity in viewing. In contrast, a virtual viewpoint image 1510 in FIG. 15(b) includes only a subject region 1511 corresponding to a subject model and a background region 1512 corresponding to a background model. In the virtual viewpoint image 1510, since the background region 1512 covers the entire angle of view, the sky region is not included, and the virtual viewpoint image 1510 does not show a rendering representation that is separated from the captured scene. In this way, even if a virtual viewpoint image is generated according to the same line of sight direction, it is possible to make it so that the sky region is not included by setting the focal length of the virtual camera to the telephoto side.

[0113] That is, the more pixels in the sky area included in the virtual viewpoint image, the more likely the user is to perceive an unnatural viewing sensation. Here, the distance range (shortest distance) of the spatial area in which the virtual camera can be set varies according to the focal length set for the virtual camera as described above in the second embodiment. Therefore, in the digital camera 100 of this modification, the processing unit 14, for example, specifies a spatial area in which the rendering viewpoint can be set for each focal length that can be set for the virtual camera. The processing unit 14 selects (samples) multiple types of coordinates within the spatial area specified based on the above angle range and distance range for each focal length. Then, the processing unit 14 derives the occupancy rate of the sky area in the virtual viewpoint image generated when the virtual camera is set at the coordinates based on the positional relationship between the subject model and the background model in the three-dimensional space and the focal length of the virtual camera.

[0114] The processing unit 14 specifies an area to be further excluded from the spatial area specified based on the angle range and distance range based on the occupancy rate of the sky area derived for each sampling coordinate, and finally specifies a spatial area in which a rendering viewpoint can be set. Specifically, the processing unit 14 can specify the final spatial area by excluding sampling coordinates whose occupancy rate of the sky area exceeds a predetermined threshold. At this time, the processing unit 14 may three-dimensionally map the occupancy rate of the sky area of ​​each sampling coordinate, specify an area whose occupancy rate will exceed a predetermined threshold based on the distribution tendency, and exclude the area from the spatial area specified based on the angle range and distance range.

[0115] In a virtual viewpoint image generated based on a rendering viewpoint set in the final spatial area identified in this manner, the occurrence of expressions that cause the user to perceive an aesthetically strange feeling due to the rendering of sky areas is reduced.

[0116] [Embodiment 3] In the above-mentioned embodiment and modified example, a spatial area in which a drawing viewpoint can be set for a three-dimensional model constructed based on a captured image and a distance image is specified, and a virtual viewpoint image related to the drawing viewpoint set in the spatial area is generated. On the other hand, such a three-dimensional model may generate a virtual viewpoint image as a still image, or may generate a virtual viewpoint image by continuously setting different drawing viewpoints and connecting these as frames to generate a virtual viewpoint moving image. The virtual viewpoint moving image can realize highly interesting information presentation, such as looking around a captured scene at a specific timing.

[0117] In generating such a virtual viewpoint moving image, the processing unit 14 sets a rendering viewpoint for rendering a virtual viewpoint image for each frame based on the spatial region specified in the above embodiment and modified example. When played back as a virtual viewpoint moving image, it is preferable that the rendering viewpoint has continuity between consecutive frames. In other words, the frames constituting the virtual viewpoint moving image provide a more suitable viewing experience by contiguous virtual viewpoint images related to a rendering viewpoint that follows a series of movements and changes, rather than contiguous virtual viewpoint images related to discretely set rendering viewpoints.

[0118] Therefore, in digital camera 100 of this embodiment, processing unit 14 sets a path of the rendering viewpoint (camera path of the virtual camera), and sets the rendering viewpoint for each frame so as to move on the viewpoint path as time progresses. Processing unit 14 determines the positions of multiple virtual cameras (the number of all frames) so as to move sequentially along the path, for example, and sets the attitude (line of sight) of the virtual camera at each position. Information on the path of the rendering viewpoint may be, for example, configured in advance and stored in storage unit 15.

[0119] On the other hand, since the spatial region in which the rendering viewpoint can be set is set based on the normal information of the three-dimensional model as described above, the processing unit 14 performs processing to fit the predetermined path of the rendering viewpoint to the spatial region. When the predetermined path of the rendering viewpoint is in the form of Fig. 16(a), the processing unit 14 may control the path to fit within the spatial region, for example, by the following method.

[0120] In the embodiment shown in Fig. 16(b), a predetermined rendering viewpoint path is rescaled to fit the entire path within the spatial range. According to the path in Fig. 16(b), a virtual viewpoint video image can be generated that relatively expresses the contents of the transition of a predetermined virtual camera.

[0121] In the embodiment shown in Fig. 16(c), the route of a predetermined rendering viewpoint is rescaled only in the z direction to fit the spatial range. In addition, the x and y directions are adjusted by moving each point of the route outside the spatial domain, among the routes rescaled in the z direction, to the coordinates in the spatial domain that are closest to the point. According to the route in Fig. 16(c), the rendering viewpoint in the spatial domain can be set while retaining the absolute movement amount of the rendering viewpoint in the spatial domain.

[0122] In addition, since each frame of the virtual viewpoint video is displayed for a time period corresponding to the display frame rate before switching to the next frame, the user is less likely to perceive discomfort in viewing than when a virtual viewpoint image generated as a still image is displayed. Therefore, the virtual camera set based on the path of the rendering viewpoint does not need to be set so as to be included in the spatial region in all frames of the virtual viewpoint video.

[0123] For example, as shown in FIG. 16(d), a virtual camera for each frame is set based on a path of a predetermined drawing viewpoint, but the setting frequency may be controlled depending on whether the virtual camera is in the spatial domain or outside the spatial domain. In the example of the figure, when a virtual viewpoint video is composed of 10 frames, virtual cameras for six of the frames are set within the spatial domain, and virtual cameras for the remaining four frames are set outside the spatial domain. That is, in a virtual viewpoint video obtained by setting a virtual camera in the manner shown in FIG. 16(d), there are more frames showing a virtual viewpoint image related to a virtual camera in the spatial domain than frames showing a virtual viewpoint image related to a virtual camera outside the spatial domain. In other words, the virtual viewpoint video has more frames showing a virtual viewpoint image that ensures a suitable viewing than frames showing a virtual viewpoint image that may cause a viewing discomfort, and therefore, expressions that cause a viewing discomfort are reduced throughout all frames.

[0124] From another point of view, on a predetermined path of the rendering viewpoint, the moving speed of the virtual camera is controlled to be high outside a spatial region where a virtual viewpoint image in which a sense of discomfort is perceived is generated, and to be low within the spatial region. Since the user can easily view the virtual viewpoint video in a section where the moving speed of the virtual camera is low, as a result, the user's interest can be guided to a frame in which a suitable virtual viewpoint image is displayed as a whole of the virtual viewpoint video.

[0125] In addition, in the embodiment of FIG. 16(d), since a predetermined rendering viewpoint path can be used as is, it is possible to generate a virtual viewpoint moving image that expresses a transition of the virtual camera that was anticipated in advance.

[0126] Additionally, the processing unit 14 may generate a virtual viewpoint moving image by setting a path of the rendering viewpoint so as to pass through a spatial region in at least some of the frames.

[0127] [Variation 6] In the above-mentioned third embodiment, the position and line of sight direction of the virtual camera that renders the virtual viewpoint image are set for each frame of the virtual viewpoint video image, but the present invention is not limited to this. As shown in the fifth modified example, the focal length of the virtual camera may be further set for each frame.

[0128] In an embodiment in which the focal length of the virtual camera is also variable, for example, the imaging magnification of the subject is fixed, and the focal length of the virtual camera is adjusted according to the distance between the subject model and the virtual camera, thereby generating a so-called dolly zoom virtual viewpoint moving image. As described above, the spatial region in which the virtual camera can be set changes according to the focal length of the virtual camera. Therefore, when setting the virtual camera for each frame for dolly zoom, the processing unit 14 may refer to the information of the spatial region according to the focal length set in the frame, and determine the position and line of sight of the virtual camera.

[0129] Alternatively, the processing unit 14 may obtain the minimum and maximum focal lengths that can be set for the virtual camera of each frame for dolly zoom, and merge the information of the spatial domain corresponding to each of them to configure the distance range of the spatial domain for dolly zoom. Once the information of the spatial domain for dolly zoom is configured for a three-dimensional model, it can be repeatedly used in the dolly zoom expression of the three-dimensional model, so that the amount of calculation required for generating the virtual viewpoint moving image can be reduced.

[0130] [Variation 7] In the above-mentioned embodiment and modified example, a virtual viewpoint image is generated by constructing a subject model and a background model based on a captured image and a distance image obtained in a specific imaging direction, and drawing a three-dimensional space in which they are arranged with respect to a drawing viewpoint. On the other hand, the texture applied to the background model does not need to be extracted from a captured image captured together with the subject. For example, a captured image obtained by capturing an image scene in a state in which the subject does not exist in advance may be used as a texture, or a captured image related to a different scene or a predetermined computer graphic may be used. In other words, the background model can be configured to be reusable when generating a virtual viewpoint image related to another subject model.

[0131] In addition, in the aspect of configuring a background model so as to be reusable, it is not necessary to arrange and draw the subject model and the background model in the same three-dimensional space. For example, a virtual viewpoint image can be generated by drawing the subject model and the background model independently and synthesizing the generated images. In this case, a device for drawing the subject model and a device for drawing the background model can be different devices, and distributed processing can be performed.

[0132] When generating a virtual viewpoint video using this method, the processing unit 14 determines the state of the virtual camera for each frame based on information on a spatial region in which a virtual camera can be set for a subject model. Then, the processing unit 14 generates a virtual viewpoint video for the subject by drawing only the subject model for the virtual camera for each frame as shown in Fig. 17(a). Using the state of the virtual camera for each frame, the processing unit 14 also generates a virtual viewpoint video for the background by drawing only the background model for the virtual camera for each frame as shown in Fig. 17(b). Then, the processing unit 14 can generate a virtual viewpoint video as if the subject were present in the background scene by synthesizing these virtual viewpoint images as shown in Fig. 17(c).

[0133] [Variation 8] In the above-described embodiment and modified example, the line of sight of the virtual camera is set in a direction from the coordinates of the virtual camera toward the reference position of the subject model, but the implementation of the present invention is not limited to this. The line of sight of the virtual camera is not limited to the subject model, and may be configured to be adjustable by the user, for example.

[0134] On the other hand, the spatial region in which the virtual camera can be set is specified based on the direction from the coordinates of the virtual camera to be evaluated toward the reference position of the subject model and the normal information of the subject model. Therefore, in an aspect in which the line of sight direction of the virtual camera can be arbitrarily changed, information indicating a range of line of sight directions in which a virtual viewpoint image that does not cause the user to perceive discomfort in viewing can be generated may be associated with each position in the spatial region. By configuring the spatial region information and line of sight direction information in this way, a suitable viewing experience of the subject model can be provided.

[0135] [Embodiment 4] In the above-mentioned embodiment and modified example, in order to reduce the sense of discomfort perceived by the user in viewing, a spatial region in which a rendering viewpoint is set is limited when generating a virtual viewpoint image in which a subject model generated based on a distance image is rendered. In other words, in the above-mentioned embodiment and modified example, a sense of discomfort in viewing is guaranteed by limiting the setting of a rendering viewpoint in which image quality degradation caused by texture stretching appears in the virtual viewpoint image. On the other hand, it is preferable for a user to have a degree of freedom in setting the rendering viewpoint when viewing a subject model, and making it impossible to set the rendering viewpoint due to the presence of image quality degradation in some polygons does not necessarily provide a preferable viewing experience. In this embodiment, a method is described in which, even in a rendering viewpoint in which the display area of ​​a polygon may be large in the virtual viewpoint image, the display area of ​​the polygon is reduced by adjusting the parameters of the virtual camera, making it difficult to perceive discomfort in viewing.

[0136] <Polygon area reduction principle> Hereinafter, an outline of the principle of generating a virtual viewpoint image that is unlikely to cause the user to perceive discomfort by adjusting the parameters of the virtual camera in digital camera 100 of this embodiment will be described.

[0137] The camera parameters that can be set for the virtual camera include the focal length, as described above. By setting the focal length to the wide-angle side, as opposed to the telephoto side as in the fifth modified example, it becomes possible to generate a virtual viewpoint image in which the perspective of the subject is emphasized. For example, as shown in FIG. 18(a), consider a case in which a rendering viewpoint 1802 is set at a position of a subject model 1800 that is relatively closer to the subject (subject model 1800) than a position 1801 at which the digital camera 100 captured an image. The virtual camera of the rendering viewpoint 1802 has the same line of sight as the digital camera 100, while the focal length is set to a wider angle than the focal length of the digital camera 100 at the time of capturing the texture.

[0138] In the illustrated example, taking into consideration that the image of subject model 1800 becomes smaller than the image of the subject in the captured image by setting the focal length of the virtual camera to a wide angle, rendering viewpoint 1802 is set to a position closer to subject model 1800 than position 1801. In the example of Fig. 18(a), a person's face is shown as subject model 1800, and the position of rendering viewpoint 1802 is set so that the distance between the eyes of the person in the virtual viewpoint image is the same as that in the captured image.

[0139] At this time, attention is paid to polygon 1803 distributed in a portion (end portion) corresponding to the edge region in subject model 1800. For example, virtual viewpoint image 1810 generated by setting a virtual camera at position 1801 with the same focal length setting as digital camera 100 when capturing a texture (captured image) to be applied to subject model 1800 is as shown in FIG. 18(b). Also, virtual viewpoint image 1820 generated for a virtual camera at rendering viewpoint 1802 with a wider focal length setting than when the texture was captured is as shown in FIG. 18(c). As shown in the figure, polygon 1803 is rendered in virtual viewpoint image 1820 with the rendering area reduced by setting the focal length to a wider angle than when the texture was captured.

[0140] As can be seen by comparing these figures, in the virtual viewpoint image obtained by setting the focal length to a wide angle, the image of the subject model 1800 is drawn at a position closer to the center 1831 of the angle of view of the virtual camera, as shown in FIG. 18(d). Here, the position at which the image of the subject model 1800 is drawn by setting the focal length to a wide angle varies depending on the distance from the center 1831 of the angle of view, i.e., the image height. Specifically, the higher the image height of the vertex of the subject model 1800 is, the closer the position in the virtual viewpoint image at which the vertex is drawn is to the center of the angle of view. In other words, when drawn at a wide-angle focal length, the vertices of the subject model 1800 distributed near the center 1831 of the angle of view of the virtual camera are drawn at the same position in the virtual viewpoint image. On the other hand, the vertices of the subject model 1800 distributed far from the center 1831 of the angle of view of the virtual camera are drawn at a position closer to the center 1831 of the angle of view in the virtual viewpoint image. Therefore, in the virtual viewpoint image, the subject model 1800 is rendered in such a manner that the polygons farther away from the center 1831 of the angle of view are distorted toward the center 1831 of the angle of view.

[0141] Therefore, in this embodiment, the imaging characteristics when such a wide-angle focal length is set are utilized to appropriately reduce the rendering area of ​​the polygons to make them less visible and to prevent the user from perceiving an unnatural appearance. In other words, when polygons that may cause image quality degradation due to texture stretching are distributed at positions with high image heights at the rendering viewpoint, the digital camera 100 of this embodiment reduces the unnatural appearance perceived by the user by setting the focal length to a wide angle.

[0142] <Premise> Furthermore, the effect of reducing the drawing area of ​​polygons that are far from the center of the angle of view by setting the focal length to a wide angle is obtained when the subject generally has a convex shape with respect to the imaging direction, i.e., when the subject model has a convex shape with respect to the reference direction.

[0143] When an object model generated based on captured images and distance images obtained in a specific imaging direction has a generally convex shape with respect to the reference direction, polygons to which texture is applied in a stretched state are distributed at the ends of the object model, as described above in Modification 1. That is, when an object model has such a shape and a rendering viewpoint related to a direction different from the imaging direction is set, the polygons at the ends of the object model appear with an area larger than that of the virtual viewpoint image related to the reference direction.

[0144] Here, the subject model generally having a convex shape refers to a state in which vertices distributed inside a two-dimensional region are generally closer to the rendering viewpoint than vertices distributed at the end of the subject model that corresponds to the outer edge of the two-dimensional region when the subject model is viewed from the reference direction. In other words, when the region corresponding to the subject in the distance image generally shows a tendency that the distance value becomes longer as it approaches the edge region and becomes shorter as it moves away from the edge region, a subject model generally having a convex shape is formed.

[0145] On the other hand, if the subject model is generally flat or concave with respect to the reference direction, even if a rendering viewpoint with a line of sight direction different from the reference direction is set, the area of ​​the polygons distributed at the edge will not increase, or the back surface of the polygon will be rendered. For this reason, changing the focal length of the virtual camera has little effect in reducing the sense of incongruity in the visual sense. Therefore, the control related to the change in the focal length of the virtual camera according to this embodiment will be described on the premise that the subject model is generally convex with respect to the reference direction.

[0146] <Determining whether the focal length has been changed> In the embodiment described below, it is assumed that the focal length (hereinafter referred to as the reference focal length) set in the digital camera 100 when capturing the texture is basically set in the virtual camera when generating the virtual viewpoint image. On the other hand, if polygons to which the stretched texture is applied are distributed at positions far from the center of the angle of view for the virtual camera set as the rendering viewpoint, the visual discomfort caused by the polygons can be reduced by setting the focal length of the virtual camera to the wide-angle side. Therefore, in this embodiment, after determining the rendering viewpoint for generating the virtual viewpoint image, the processing unit 14 determines whether it is preferable to change the setting of the focal length of the virtual camera from the reference focal length to the wide-angle side, and determines the final focal length based on the determination result.

[0147] In making the determination, the processing unit 14 sets evaluation positions at the center of the angle of view and the outer edge of the two-dimensional area (subject area) in which the subject model is drawn for the virtual viewpoint image drawn by the virtual camera with the reference focal length, and derives the distance from the drawing viewpoint to each evaluation position. When the virtual viewpoint image drawn by the virtual camera with the reference focal length is as shown in FIG. 19(a), the evaluation positions related to the outer edge can be set as shown in FIG. 19(b), for example. In the example of FIG. 19(b), an evaluation position 1902 related to the outer edge is set at the intersection of the horizontal line and the vertical line passing through the center of the angle of view and the outer edge. Also, in FIG. 19(b), the evaluation position 1901 is the evaluation position related to the center of the angle of view.

[0148] In this embodiment, the evaluation positions related to the outer edge are set based on the above-mentioned criteria, but the present invention is not limited to this. The evaluation positions related to the outer edge may be set based on different criteria or randomly as long as they are set in relation to sampling for identifying the distribution state of the subject model related to the drawing viewpoint. The evaluation positions related to the outer edge are not limited to the four points shown in FIG. 19(b) and may be any number of points equal to or greater than one. In FIG. 19, a virtual viewpoint image drawn by a virtual camera with a reference focal length is shown to facilitate understanding of the invention, but the generation of the virtual viewpoint image is not essential. The two-dimensional coordinates at which the outer edge of the subject model is drawn and the distance from the drawing viewpoint to any point on the outer edge can be specified based on various parameters of the drawing viewpoint and shape information of the subject model.

[0149] When the processing unit 14 derives the distances from the rendering viewpoint to the evaluation position related to the center of the angle of view and the evaluation position related to the outer edge, the processing unit 14 determines whether or not to change the focal length of the virtual camera to wide angle based on the state of these distances. In this embodiment, if the distance to at least one evaluation position related to the outer edge is longer than the distance to the evaluation position related to the center of the angle of view, it is determined that the focal length of the virtual camera is to be changed to wide angle. That is, in the case of such a distribution of evaluation positions, the processing unit 14 evaluates that the occupancy rate of the polygon extending in the imaging direction of the subject model within the angle of view is high for the virtual camera for which the reference focal length is set, and determines that the focal length of the virtual camera is to be changed to wide angle.

[0150] <<Virtual viewpoint image generation process>> A specific process of generating a virtual viewpoint image executed by the digital camera 100 of this embodiment will be described with reference to the flowchart of FIG. 20. The process corresponding to this flowchart can be realized by the control unit 12 reading out a corresponding processing program stored in, for example, the ROM 21, and developing and executing the program in the RAM 20. This process will be described as being started when, for example, an instruction to generate a virtual viewpoint image is given for an arbitrary three-dimensional model. Hereinafter, the three-dimensional model for which an instruction to generate a virtual viewpoint image is given will be referred to as a target three-dimensional model. In the description of the generation process of this embodiment, steps that perform the same processes as those in the generation process of the first embodiment will be given the same reference numbers and will not be described again, and steps that perform processes unique to this embodiment will be described below.

[0151] When the target 3D model is placed in the 3D space to be rendered in S1104, the processing unit 14, under the control of the control unit 12, in S2001 sets a virtual camera with a reference focal length for the rendering viewpoint and identifies the subject area in the rendered virtual viewpoint image.

[0152] In S2002, under the control of the control unit 12, the processing unit 14 sets an evaluation position related to the center of the angle of view and an evaluation position related to the outer edge of the subject area.

[0153] In S2003, the processing unit 14 derives the distance from the rendering viewpoint to each evaluation position under the control of the control unit 12. The processing unit 14 also determines whether or not to change the focal length of the virtual camera to the wide-angle side based on the derived value of the distance to each evaluation position.

[0154] In S2004, the processing unit 14 judges whether the result of the judgment is to change the focal length to the wide-angle side or not to change it under the control of the control unit 12. If the result of the judgment is that the focal length is to be changed to the wide-angle side, the processing unit 14 shifts the process to S2005, and if the result of the judgment is that the focal length is not to be changed, the processing unit 14 shifts the process to S1106.

[0155] In S2005, the processing unit 14 changes the focal length of the virtual camera to a wide angle under the control of the control unit 12, and the process proceeds to S1106. In this embodiment, the description will be given assuming that the focal length after the change is a predetermined value.

[0156] As described above, according to the image processing device of the present embodiment, a predetermined image quality can be guaranteed for the generated virtual viewpoint image. More specifically, the image processing device sets a focal length on the wide-angle side for the rendering viewpoint at which the polygon of the subject model to which the texture is applied is rendered, and therefore the display area of ​​the polygon in the generated virtual viewpoint image can be reduced. As a result, the occurrence of unnatural two-dimensional expressions can be reduced and a predetermined image quality can be guaranteed. In other words, the image processing device evaluates the occupancy rate within the angle of view of the polygon extending in the imaging direction of the subject model for the rendering viewpoint at which the reference focal length is set. Then, when the occupancy rate exceeds a predetermined threshold value, the image processing device determines that the display area of ​​the polygon to which the texture is applied is large and changes the focal length to the wide-angle side, and therefore the display area of ​​such polygon can be reduced.

[0157] In the present embodiment, the focal length of the virtual camera is changed to a wide angle when the distance to the evaluation position related to at least one outer edge is longer than the distance to the evaluation position related to the center of the angle of view. However, the present invention is not limited to this. When evaluation positions related to a plurality of outer edges are set, the focal length of the virtual camera may be changed to a wide angle on the condition that all of the distances to the evaluation positions related to the outer edges are longer than the distance to the evaluation position related to the center of the angle of view. Alternatively, the determination to change the focal length of the virtual camera to a wide angle may be made on the condition that the distance to the evaluation position related to at least one outer edge is longer than the distance to the evaluation position related to the center of the angle of view by a predetermined threshold or more.

[0158] [Variation 9] In the above-described fourth embodiment, the distribution state of the subject model at the rendering viewpoint is evaluated based on the distance from the rendering viewpoint to the evaluation position of the subject model, and whether or not to change the focal length of the virtual camera to a wide angle is determined. However, the implementation of the present invention is not limited to this, and similarly to the first embodiment, the extent to which the texture in the stretched state is visible in the virtual viewpoint image generated for the rendering viewpoint may be evaluated, and the determination may be made based on the evaluation. More specifically, the processing unit 14 derives the inner product of the normal vector p of each polygon constituting the subject model and the vector v in the line of sight direction of the rendering viewpoint, and derives the average value of the inner products related to all polygons as the evaluation value. Then, when the evaluation value exceeds a predetermined threshold, the processing unit 14 may determine that a virtual viewpoint image that causes a sense of discomfort in terms of viewing is generated for the rendering viewpoint, and change the focal length of the virtual camera to a wide angle. That is, when the evaluation value is such, the processing unit 14 evaluates that the occupancy rate of the polygon extending in the imaging direction of the subject model within the angle of view of the virtual camera for which the reference focal length is set is high, and determines that the focal length of the virtual camera is to be changed to a wide angle.

[0159] [Variation 10] In addition, as described in the second embodiment, the closer the polygon constituting the object model and the virtual camera are, the larger the display area of ​​the polygon in the generated virtual viewpoint image, and a virtual viewpoint image that gives a sense of strangeness in terms of appearance may be generated. Therefore, when the distance between the rendering viewpoint and the object model is below a predetermined threshold, the processing unit 14 may determine that a virtual viewpoint image that gives a sense of strangeness in terms of appearance is generated from the rendering viewpoint, and change the focal length of the virtual camera to a wide angle. That is, when the rendering viewpoint and the object model are in such a positional relationship, the processing unit 14 evaluates that the occupancy rate of the polygon extending in the imaging direction of the object model in the angle of view of the virtual camera with the reference focal length set is high, and determines to change the focal length to a wide angle.

[0160] [Variation 11] In the above-mentioned embodiment 4 and modified examples 9 and 10, when it is determined that the focal length of the virtual camera is changed to a wide angle, the focal length is changed to a fixed focal length, but the implementation of the present invention is not limited to this. The processing unit 14 may control the focal length to be changed to a wider angle side depending on the degree to which a virtual viewpoint image that causes a sense of discomfort in viewing the rendering viewpoint is generated. That is, the processing unit 14 controls the focal length to be changed to the wider angle side as the occupancy rate of the polygon extending in the imaging direction of the subject model in the angle of view increases for a virtual camera for which a reference focal length is set. Specifically, the processing unit 14 may control the focal length to be set to a wider angle side as the difference between the distance to the evaluation position related to the outer edge and the distance to the evaluation position related to the center of the angle of view increases. Also, the processing unit 14 may control the focal length to be set to a wider angle side as the average value of the dot product of the line-of-sight vector of the rendering viewpoint and the normal vector of the subject model increases, that is, as the subject model has more polygons extending in the reference direction. Moreover, the processing unit 14 may perform control so as to set a wider-angle focal length as the rendering viewpoint is closer to the subject model.

[0161] [Variation 12] In the above-described fourth embodiment and the ninth to eleventh modifications, a determination is made as to whether or not to change the focal length of the virtual camera to a wide angle for each set rendering viewpoint, but the present invention is not limited to this. The above-described determination may be configured to be performed only when a specific rendering viewpoint is selected.

[0162] [Variation 13] In the above-mentioned fourth embodiment, it has been described that the determination as to whether or not to change the focal length of the virtual camera to a wide angle is performed based on the distance from the rendering viewpoint to the set evaluation position, but the implementation of the present invention is not limited to this. Instead of the distance from the rendering viewpoint, it may be configured to determine that the focal length of the virtual camera is changed to a wide angle when the normal of the polygon at the center of the angle of view faces the rendering viewpoint and the normal of the polygon on the outer edge does not face the rendering viewpoint.

[0163] [Variation 14] In the above-mentioned fourth embodiment and the ninth to thirteenth modifications, the aspect of determining whether or not to change the focal length of the virtual camera to a wide angle depending on the relationship between the rendering viewpoint and the subject model has been described, but the present invention is not limited to this. For example, when the generated subject model has a convex shape with respect to the reference direction, the processing unit 14 may control the focal length of the virtual camera to be changed to a focal length on the wide-angle side from the reference focal length to generate a virtual viewpoint image.

[0164] [Variation 15] In the above-described fourth embodiment and the ninth to fourteenth modifications, when it is determined that a virtual viewpoint image that causes a sense of discomfort in viewing the rendering viewpoint is generated, the processing unit 14 changes the focal length of the virtual camera to the wide-angle side. On the other hand, the image of the subject in the virtual viewpoint image generated when the focal length is set to the wide-angle side is distorted toward the center of the angle of view as the image height increases, as described above, and therefore, depending on the type of subject, the user may perceive unnatural distortion as occurring. For this reason, the processing unit 14 may perform different control in the case of a specific type of subject, such as a person, an animal such as a pet (dog or cat), or the like, in which distortion caused by setting the focal length to the wide-angle side may cause discomfort to the user. For example, when it is determined that the subject is at least one of a predetermined type, the processing unit 14 may control so as not to change the focal length of the virtual camera to the wide-angle side, or to reduce the amount of change to the wide-angle side. This reduces distortion of the image of the subject in the generated virtual viewpoint image, thereby reducing the occurrence of unnatural two-dimensional expression and ensuring a predetermined image quality.

[0165] The type of subject can be identified by performing image analysis on the captured image, or by inputting the captured image into a machine learning model that has learned about types of subjects whose distortions may cause discomfort to the user.

[0166] [Summary of the embodiment and modifications] The disclosure of this specification includes the following image processing device, imaging device, control method, and program. (Item 1) An image processing device that generates a virtual viewpoint image by drawing a three-dimensional model to which a captured image is applied as a texture, A setting means for setting a rendering viewpoint for rendering the three-dimensional model; a first generating means for generating the virtual viewpoint image by drawing the three-dimensional model based on the drawing viewpoint set by the setting means; a control means for controlling the rendering viewpoint to be set by the setting means based on the three-dimensional model; have 13. An image processing device comprising: (Item 2) The method further includes acquiring normal information relating to the three-dimensional model, The control means controls the setting means to set the rendering viewpoint in a spatial region based on the normal information. 2. The image processing device according to item 1, (Item 3) 3. The image processing device according to item 2, wherein the first generating means generates the virtual viewpoint image by placing the three-dimensional model in a three-dimensional space and rendering the three-dimensional space with respect to the rendering viewpoint. (Item 4) 4. The image processing device according to item 3, wherein the spatial region is a partial region in the three-dimensional space. (Item 5) the spatial region defines a range in the three-dimensional space relatively based on the three-dimensional model, The control means includes a specification means for specifying the spatial region based on the normal information. 5. The image processing device according to item 3 or 4. (Item 6) 6. The image processing device according to item 5, wherein the specifying means specifies an angular range of the spatial region based on the three-dimensional model. (Item 7) the normal information includes information on normal vectors of faces constituting the three-dimensional model, The image processing device described in item 6, characterized in that the identification means identifies the angle range based on an angle between a normal vector of each face constituting the three-dimensional model or a normal vector of each face distributed at an end of the three-dimensional model and a vector pointing from a position in the three-dimensional space to the three-dimensional model. (Item 8) The image processing device described in item 7, characterized in that the identification means identifies a position in the three-dimensional space where an average value of an inner product between a normal vector of each face constituting the three-dimensional model or a normal vector of each face distributed at an end of the three-dimensional model and a vector pointing toward the three-dimensional model is below a predetermined threshold, and identifies the angle range based on the position in the three-dimensional space. (Item 9) 9. The image processing device according to any one of items 6 to 8, wherein the specifying means further specifies a distance range of the spatial region based on the three-dimensional model. (Item 10) 10. The image processing device according to item 9, wherein the specifying means determines the distance within the distance range that is closest to the three-dimensional model based on a spatial resolution of the captured image that is applied as a texture to the three-dimensional model. (Item 11) The image processing device described in item 9, wherein the identification means determines the closest distance to the three-dimensional model within the distance range based on the number of pixels of the captured image, a focal length set when the captured image was captured, the number of pixels of the virtual viewpoint image, and a focal length related to the rendering viewpoint. (Item 12) 12. The image processing device according to any one of items 9 to 11, wherein the specifying means specifies the farthest distance from the three-dimensional model within the distance range such that an occupancy rate of an area in the virtual viewpoint image in which the three-dimensional model is rendered exceeds a predetermined threshold. (Item 13) the three-dimensional model includes a back surface to which no texture is applied; The specifying means specifies the spatial region by excluding a position where an occupancy rate of an area in the virtual viewpoint image where the back surface of the three-dimensional model is drawn exceeds a predetermined threshold. 13. The image processing device according to any one of items 5 to 12, (Item 14) the three-dimensional model is generated based on the captured image and distance information indicating a subject distance corresponding to the captured image, and includes a subject model related to a subject and a background model related to a background; The specifying means specifies a range in the three-dimensional space based on the subject model as the spatial region. 14. The image processing device according to any one of items 5 to 13, (Item 15) the first generating means generates the virtual viewpoint image by placing the subject model and the background model in a three-dimensional space and rendering the three-dimensional space with respect to the rendering viewpoint; The specifying means specifies the spatial region by excluding a position where an occupancy rate of a region in the virtual viewpoint image where neither the subject model nor the background model is rendered exceeds a predetermined threshold. 15. The image processing device according to item 14, (Item 16) the texture to be applied to the object model is generated by extracting an image of a region corresponding to the object from the captured image; a texture to be applied to the background model is generated by excluding an image of a region corresponding to the subject from the captured image; The specifying means specifies the spatial area by excluding a position in the virtual viewpoint image where an occupancy rate of an area where the background model to which no texture is applied is rendered exceeds a predetermined threshold. 16. The image processing device according to item 14 or 15, (Item 17) The method further includes a second generating means for generating a virtual viewpoint moving image having a plurality of the virtual viewpoint images generated by the first generating means as frames, The control means causes the setting means to determine a path of the rendering viewpoint for the virtual viewpoint video image generated by the second generation means so as to pass through at least the spatial region. 17. The image processing device according to any one of items 1 to 16, (Item 18) Item 18. The image processing device according to item 17, wherein the control means sets the drawing viewpoint for each frame along the path so that, in the virtual viewpoint moving image, there are more frames showing the virtual viewpoint image corresponding to the drawing viewpoint within the spatial region than frames showing the virtual viewpoint image corresponding to the drawing viewpoint outside the spatial region. (Item 19) The image processing device described in item 17 or 18, characterized in that the control means controls the moving speed of the drawing viewpoint along the path so that, in the virtual viewpoint moving image, there are more frames showing the virtual viewpoint image corresponding to the drawing viewpoint in the spatial region than frames showing the virtual viewpoint image corresponding to the drawing viewpoint outside the spatial region. (Item 20) 20. The image processing device according to any one of items 17 to 19, wherein the control means sets a predetermined viewpoint path by adjusting the path based on the spatial region. (Item 21) 21. The image processing device according to item 20, wherein the control means sets the path by adjusting the predetermined viewpoint path so that it fits within the spatial region. (Item 22) the setting means further sets parameters of a virtual camera to be set at the rendering viewpoint; the three-dimensional model has a convex shape with respect to an imaging direction of the captured image, the image processing device further comprises an evaluation means for evaluating a distribution state of polygons of the three-dimensional model extending in the imaging direction for the rendering viewpoint to which a reference focal length is set, The control means changes the focal length of the virtual camera that the setting means sets in response to the evaluation by the evaluation means. 2. The image processing device according to item 1, (Item 23) the evaluation means evaluates whether or not an occupancy rate of a polygon of the three-dimensional model extending in the imaging direction within an angle of view exceeds a predetermined threshold value; The control means controls the setting means to change a focal length of the virtual camera to a wider angle side than the reference focal length when the evaluation means evaluates that the occupancy rate exceeds the predetermined threshold value. 23. The image processing device according to item 22, characterized in that (Item 24) The evaluation means evaluates that the occupancy rate exceeds the predetermined threshold value when a distance to an evaluation position related to an outer edge of the three-dimensional model is longer than a distance to an evaluation position related to a center of an angle of view of the three-dimensional model. 24. The image processing device according to item 23, (Item 25) The evaluation means evaluates that the occupancy rate exceeds the predetermined threshold when an average value of an inner product of a normal vector of each face constituting the three-dimensional model or a normal vector of each face distributed at an end of the three-dimensional model and a vector in a line of sight of the rendering viewpoint exceeds a threshold. 24. The image processing device according to item 23, (Item 26) The evaluation means evaluates that the occupancy rate exceeds the predetermined threshold value when the distance between the rendering viewpoint and the three-dimensional model is below a threshold value. 24. The image processing device according to item 23, (Item 27) The control means controls the setting means so as to change the focal length from the reference focal length to a wider angle side as the occupancy rate increases. 27. The image processing device according to any one of items 23 to 26, (Item 28) The control means controls the setting means to set a focal length on the wide-angle side of the virtual camera when the three-dimensional model has a convex shape with respect to an imaging direction of the captured image. 2. The image processing device according to item 1, (Item 29) The method further includes a means for identifying a type of a subject corresponding to the three-dimensional model, The control means controls the setting means to reduce an amount by which the focal length of the virtual camera is changed to the wide-angle side when the type of the subject is a predetermined type. 29. The image processing device according to any one of items 23 to 28, (Item 30) The predetermined type includes at least one of a person and an animal. 30. The image processing device according to item 29, characterized in that (Item 31) An image processing device according to any one of items 1 to 30, An imaging means for acquiring the captured image; a distance acquisition means for acquiring distance information indicating a subject distance corresponding to the captured image; a constructing means for constructing the three-dimensional model based on the captured image and the distance information; An imaging device comprising: (Item 32) the imaging means is capable of acquiring a group of pupil division images corresponding to light beams passing through different pupil regions of an imaging optical system; The distance acquisition means derives a subject distance based on the group of pupil division images and acquires the distance information. 32. The imaging device according to item 31, characterized in that: (Item 33) A method for controlling an image processing device that generates a virtual viewpoint image by drawing a three-dimensional model to which a captured image is applied as a texture, comprising the steps of: a setting step of setting a rendering viewpoint for rendering the three-dimensional model; a generating step of generating the virtual viewpoint image by drawing the three-dimensional model based on the drawing viewpoint set in the setting step; a control step of controlling the rendering viewpoint set in the setting step based on the three-dimensional model; have A control method comprising: (Item 34) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 30.

[0167] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0168] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0169] 100: digital camera, 10: imaging optical system, 11: imaging element, 12: control unit, 14: processing unit, 15: storage unit

Claims

1. An image processing device for generating a virtual viewpoint image by drawing a three-dimensional model to which a captured image is applied as a texture, a setting means for setting a rendering viewpoint for rendering the three-dimensional model; a first generating means for generating the virtual viewpoint image by drawing the three-dimensional model based on the drawing viewpoint set by the setting means; a control means for controlling the rendering viewpoint to be set by the setting means based on the three-dimensional model; have 13. An image processing device comprising:

2. The method further includes acquiring normal information relating to the three-dimensional model, The control means controls the setting means to set the rendering viewpoint in a spatial region based on the normal information.

2. The image processing device according to claim 1,

3. 3 . The image processing apparatus according to claim 2 , wherein the first generating means generates the virtual viewpoint image by placing the three-dimensional model in a three-dimensional space and rendering the three-dimensional space with respect to the rendering viewpoint.

4. 4. The image processing apparatus according to claim 3, wherein the spatial region is a partial region in the three-dimensional space.

5. the spatial region relatively defines a range in the three-dimensional space based on the three-dimensional model, The control means includes a specification means for specifying the spatial region based on the normal information.

4. The image processing device according to claim 3.

6. The image processing apparatus according to claim 5 , wherein the specifying means specifies an angular range of the spatial region with the three-dimensional model as a reference.

7. the normal information includes information on normal vectors of faces constituting the three-dimensional model, The image processing device according to claim 6, characterized in that the identification means identifies the angular range based on an angle between a normal vector of each face constituting the three-dimensional model or a normal vector of each face distributed at an end of the three-dimensional model and a vector pointing from a position in the three-dimensional space toward the three-dimensional model.

8. The image processing device according to claim 7, wherein the identification means identifies a position in the three-dimensional space where an average value of an inner product between a normal vector of each face constituting the three-dimensional model or a normal vector of each face distributed at an end of the three-dimensional model and a vector pointing toward the three-dimensional model is below a predetermined threshold, and identifies the angle range based on the position in the three-dimensional space.

9. 7. The image processing apparatus according to claim 6, wherein said specifying means further specifies a distance range of said spatial region with said three-dimensional model as a reference.

10. 10. The image processing apparatus according to claim 9, wherein the specifying unit determines the closest distance to the three-dimensional model within the distance range based on a spatial resolution of the captured image to be applied as a texture to the three-dimensional model.

11. The image processing device according to claim 9, characterized in that the identification means determines the distance within the distance range that is closest to the three-dimensional model based on the number of pixels of the captured image, a focal length set when the captured image was captured, the number of pixels of the virtual viewpoint image, and a focal length related to the drawing viewpoint.

12. 10. The image processing device according to claim 9, wherein the specifying means specifies the farthest distance from the three-dimensional model within the distance range such that an occupancy rate of an area in the virtual viewpoint image in which the three-dimensional model is rendered exceeds a predetermined threshold value.

13. the three-dimensional model includes a back surface to which no texture is applied; The specifying means specifies the spatial region by excluding a position where an occupancy rate of an area in the virtual viewpoint image where the back surface of the three-dimensional model is rendered exceeds a predetermined threshold.

6. The image processing device according to claim 5,

14. the three-dimensional model is generated based on the captured image and distance information indicating a subject distance corresponding to the captured image, and includes a subject model related to a subject and a background model related to a background; The specifying means specifies a range in the three-dimensional space based on the subject model as the spatial region.

6. The image processing device according to claim 5,

15. the first generating means generates the virtual viewpoint image by placing the subject model and the background model in a three-dimensional space and rendering the three-dimensional space with respect to the rendering viewpoint; The specifying means specifies the spatial region by excluding a position where an occupancy rate of a region in the virtual viewpoint image where neither the subject model nor the background model is rendered exceeds a predetermined threshold.

15. The image processing device according to claim 14.

16. the texture to be applied to the object model is generated by extracting an image of a region corresponding to the object from the captured image; a texture to be applied to the background model is generated by excluding an image of a region corresponding to the subject from the captured image; The specifying means specifies the spatial area by excluding a position in the virtual viewpoint image where an occupancy rate of an area where the background model to which no texture is applied is rendered exceeds a predetermined threshold.

15. The image processing device according to claim 14.

17. The present invention further includes a second generating means for generating a virtual viewpoint moving image having a plurality of the virtual viewpoint images generated by the first generating means as frames, The control means causes the setting means to determine a path of the rendering viewpoint for the virtual viewpoint video image generated by the second generation means so as to pass through at least the spatial region.

3. The image processing device according to claim 2.

18. The image processing device described in claim 17, characterized in that the control means sets the drawing viewpoint for each frame along the path so that, in the virtual viewpoint moving image, there are more frames showing the virtual viewpoint image corresponding to the drawing viewpoint within the spatial region than frames showing the virtual viewpoint image corresponding to the drawing viewpoint outside the spatial region.

19. The image processing device described in claim 17, characterized in that the control means controls the moving speed of the drawing viewpoint along the path so that, in the virtual viewpoint moving image, there are more frames showing the virtual viewpoint image corresponding to the drawing viewpoint within the spatial region than frames showing the virtual viewpoint image corresponding to the drawing viewpoint outside the spatial region.

20. 18. The image processing apparatus according to claim 17, wherein the control means sets a predetermined viewpoint path by adjusting the path based on the spatial region.

21. 21. The image processing apparatus according to claim 20, wherein the control means sets the predetermined viewpoint path by adjusting the predetermined viewpoint path so that the path fits within the spatial region.

22. the setting means further sets parameters of a virtual camera to be set at the rendering viewpoint; the three-dimensional model has a convex shape with respect to an imaging direction of the captured image, the image processing device further comprises an evaluation means for evaluating a distribution state of polygons of the three-dimensional model extending in the imaging direction for the rendering viewpoint to which a reference focal length is set, The control means changes the focal length of the virtual camera that the setting means sets in response to the evaluation by the evaluation means.

2. The image processing device according to claim 1,

23. the evaluation means evaluates whether or not an occupancy rate of a polygon of the three-dimensional model extending in the imaging direction within an angle of view exceeds a predetermined threshold value; The control means controls the setting means to change a focal length of the virtual camera to a wider angle side than the reference focal length when the evaluation means evaluates that the occupancy rate exceeds the predetermined threshold value.

23. The image processing device according to claim 22.

24. The evaluation means evaluates that the occupancy rate exceeds the predetermined threshold value when a distance to an evaluation position related to an outer edge of the three-dimensional model is longer than a distance to an evaluation position related to a center of an angle of view of the three-dimensional model.

24. The image processing device according to claim 23 .

25. The evaluation means evaluates that the occupancy rate exceeds the predetermined threshold when an average value of an inner product of a normal vector of each face constituting the three-dimensional model or a normal vector of each face distributed at an end of the three-dimensional model and a vector in a line of sight of the rendering viewpoint exceeds a threshold.

24. The image processing device according to claim 23 .

26. The evaluation means evaluates that the occupancy rate exceeds the predetermined threshold value when the distance between the rendering viewpoint and the three-dimensional model is below a threshold value.

24. The image processing device according to claim 23 .

27. The control means controls the setting means so as to change the focal length from the reference focal length to a wider angle side as the occupancy rate increases.

24. The image processing device according to claim 23 .

28. the setting means further sets parameters of a virtual camera to be set at the rendering viewpoint; The control means controls the setting means to set a focal length on the wide-angle side of the virtual camera when the three-dimensional model has a convex shape with respect to an imaging direction of the captured image.

2. The image processing device according to claim 1,

29. The method further includes a means for identifying a type of a subject corresponding to the three-dimensional model, The control means controls the setting means to reduce an amount by which the focal length of the virtual camera is changed to the wide-angle side when the type of the subject is a predetermined type.

24. The image processing device according to claim 23 .

30. The predetermined type includes at least one of a person and an animal.

30. The image processing apparatus according to claim 29,

31. An image processing device according to any one of claims 1 to 30, An imaging means for acquiring the captured image; a distance acquisition means for acquiring distance information indicating a subject distance corresponding to the captured image; a constructing means for constructing the three-dimensional model based on the captured image and the distance information; An imaging device comprising:

32. the imaging means is capable of acquiring a group of pupil division images corresponding to light beams passing through different pupil regions of an imaging optical system; The distance acquisition means derives a subject distance based on the group of pupil division images and acquires the distance information.

32. The imaging device according to claim 31 .

33. A method for controlling an image processing device that generates a virtual viewpoint image by drawing a three-dimensional model to which a captured image is applied as a texture, comprising the steps of: a setting step of setting a rendering viewpoint for rendering the three-dimensional model; a generating step of generating the virtual viewpoint image by drawing the three-dimensional model based on the drawing viewpoint set in the setting step; a control step of controlling the rendering viewpoint set in the setting step based on the three-dimensional model; have A control method comprising:

34. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 30.

Images