Image processing device, control method, program, and image processing system

Abstract

To generate a more detailed virtual viewpoint image.SOLUTION: An image processing device includes setting means for setting a virtual viewpoint, generation means for generating a virtual viewpoint image corresponding to the virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints, and acquisition means for acquiring, for each of the viewpoint images, feature information of the viewpoint image. The generation means generates the virtual viewpoint image by executing a process that increases spatial resolution of an image based on the feature information.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an image processing device, a control method, a program, and an image processing system, and more particularly to a technique for generating a virtual viewpoint image corresponding to a virtually set viewpoint using a plurality of viewpoint images. [Background technology]

[0002] There is a technology that uses images captured from multiple viewpoints (hereinafter referred to as viewpoint images) to generate an image (hereinafter referred to as a virtual viewpoint image) corresponding to a virtually set viewpoint. By using the technology to generate a virtual viewpoint image, it is possible for a user to watch, for example, highlight scenes of soccer or basketball from a desired viewpoint. Patent Document 1 discloses that a three-dimensional model corresponding to a subject is constructed as point cloud data based on multiple viewpoint images, and the three-dimensional model is rendered to generate a virtual viewpoint image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 147329 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, the spatial resolution of point cloud data depends on the spatial resolution of the viewpoint image on which it is based. That is, since the pixels of the viewpoint image form points in a three-dimensional model, the spatial resolution of the subject represented in the viewpoint image may be low depending on the subject distance of the viewpoint image. Therefore, when the point cloud data is sparsely configured, the virtual viewpoint image obtained by rendering the three-dimensional model configured with the point cloud data may lack fineness compared to the actual image actually captured at the set viewpoint. This is also due to the fact that when rendering a three-dimensional model, a texture generated by degenerating the viewpoint image according to the parameters of the set viewpoint or averaging the pixel values ​​of multiple viewpoint images is applied to the three-dimensional model.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an image processing device, a control method, a program, and an image processing system that generate a more detailed virtual viewpoint image. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the image processing device of the present invention has a setting means for setting a virtual viewpoint, a generation means for generating a virtual viewpoint image corresponding to the virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints, and an acquisition means for acquiring feature information of each of the viewpoint images, and is characterized in that the generation means generates the virtual viewpoint image by executing a process to increase the spatial resolution of the image based on the feature information. Effect of the Invention

[0007] With this configuration, the present invention makes it possible to generate a more detailed virtual viewpoint image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an image processing system according to an embodiment and a modification of the present invention. [Diagram 2]FIG. 1 is a block diagram illustrating the functional configuration of each device in an image processing system according to an embodiment and a modified example of the present invention. [Diagram 3] FIG. 1 is a diagram for explaining an overview of super-resolution processing according to an embodiment and a modification of the present invention; [Figure 4] FIG. 13 is a diagram illustrating a learning method of a neural network that realizes super-resolution processing according to an embodiment and a modified example of the present invention. [Diagram 5] A flowchart illustrating a machine learning process for constructing a trained model according to an embodiment and a modification of the present invention. [Figure 6] FIG. 1 is a block diagram illustrating a functional configuration of an image processing unit according to a first embodiment of the present invention. [Figure 7] FIG. 1 is a diagram for explaining generation of a virtual viewpoint image according to the first embodiment of the present invention; [Figure 8] FIG. 11 is another diagram illustrating generation of a virtual viewpoint image according to the first embodiment of the present invention. [Figure 9] FIG. 1 is a diagram illustrating a virtual viewpoint image according to an embodiment and a modification of the present invention; [Figure 10] 1 is a flowchart illustrating an output process executed by the generating device 130 according to the first embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram illustrating the functional configuration of an image processing unit according to a second embodiment of the present invention. [Figure 12] FIG. 11 is a diagram for explaining generation of a virtual viewpoint image according to a second embodiment of the present invention; [Figure 13] 11 is a flowchart illustrating an output process executed by the generating device 130 according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] In the embodiment described below, an example of the present invention is applied to a generating device that receives viewpoint images from an imaging device that captures images from multiple viewpoints and generates a virtual viewpoint image using the viewpoint images, as an example of an image processing device. However, the present invention can be applied to any device that can generate a virtual viewpoint image based on multiple viewpoint images.

[0011] Image Processing System Configuration FIG. 1 is a diagram illustrating the configuration of an image processing system 100 according to an embodiment of the present invention. As shown in the figure, the image processing system 100 includes a learning device 110, an imaging device 120, and a generating device 130 connected via a network 160. In this embodiment, it is assumed that a single imaging device 120 is moved to capture an object, thereby obtaining viewpoint images corresponding to each of a plurality of viewpoints. However, the implementation of the present invention is not limited to this, and the plurality of viewpoint images used to generate a virtual viewpoint image may be obtained by capturing images of a plurality of imaging devices installed at different viewpoints. The generating device 130 is a device that generates a virtual viewpoint image in this embodiment. In the example shown in the figure, an input device 140 for a user to set a virtual viewpoint from which a virtual viewpoint image is generated in the generating device 130, and a display device 150 for displaying the generated virtual viewpoint image are connected to the generating device 130.

[0012] <Functional configuration of each device> Fig. 2 is a block diagram illustrating the functional configuration of each device in the image processing system 100. In the example of Fig. 2, in order to clarify the exchange of information between the devices, a network 160 is provided between the learning device 110 and the generating device 130, and between the imaging device 120 and the generating device 130, but these can be configured as a common network.

[0013] The imaging device 120 generates viewpoint images based on the shooting operations performed at each viewpoint as described above. The imaging device 120 includes a control unit 121, a storage unit 122, an optical system 123, an imaging unit 124, an A / D conversion unit 125, and an image processing unit 126.

[0014] The control unit 121 is, for example, a CPU, and controls the operation of each block included in the imaging device 120. More specifically, the control unit 121 controls the operation of each block by reading out an operation program of each block from a storage area of ​​the storage unit 122, expanding the program in an expansion area, and executing the program.

[0015] The storage unit 122 is a storage device including a ROM, a RAM, a hard disk, etc. The storage unit 122 stores operation programs for each block of the imaging device 120, as well as parameters required for the operation of each block and temporarily stores data output during the operation of each block. The storage unit 122 also stores captured image data, metadata for each image, etc.

[0016] The imaging unit 124 is an imaging element such as a CCD or CMOS sensor. The imaging unit 124 photoelectrically converts an optical image formed on the imaging surface of the imaging element by the optical system 123, and outputs the obtained analog image signal to the A / D conversion unit 125. The optical system 123 includes, for example, a fixed lens, a variable magnification lens that changes the focal length, and a focus lens that adjusts the focus. The optical system 123 also includes an aperture, and the aperture can adjust the aperture diameter of the optical system to adjust the amount of light during shooting. The A / D conversion unit 125 applies A / D conversion processing to the input analog image signal to generate digital image data. The digital image data generated by the A / D conversion unit 125 is output to the storage unit 122 and stored therein.

[0017] The image processing unit 126 applies various image processing such as white balance adjustment, color interpolation, gamma processing, etc. to the image data stored in the storage unit 122. The image data to which the image processing has been applied is output to the storage unit 122 and stored. These image processing will be described as being generally performed in the shooting operation. The image processing unit 126 of this embodiment also performs downsampling processing on the image data to which these image processing has been applied. In this embodiment, the image data in which the number of pixels has been reduced by the downsampling processing is referred to as a "viewpoint image". The image processing unit 126 also performs extraction processing to extract feature information, which will be described later, from the image data before the downsampling processing is applied. The feature information obtained by the extraction processing is output to the storage unit 122 and can be stored in association with a viewpoint image generated based on the image to be processed, for example.

[0018] The learning device 110 configures a trained model used when the generating device 130 generates a virtual viewpoint image. Although details will be described later, in the image processing system 100 of this embodiment, the learning device 110 performs machine learning on what kind of image will be generated by super-resolution processing (upsampling processing) prior to the generation of a virtual viewpoint image in the generating device 130. The machine learning is performed by the learning unit 113. The control unit 111 is, for example, a CPU, and controls the operation of each block included in the learning device 110. More specifically, the control unit 111 reads out the operation program of each block from the storage area of ​​the storage unit 112, expands it in the expansion area, and executes it to control the operation of each block. The storage unit 112 is a storage device composed of a ROM, a RAM, a hard disk, and the like. In addition to the operation program of each block, the storage unit 112 stores parameters and the like required for the operation of each block, and temporarily stores data output in the operation of each block. The storage unit 112 also stores a trained model obtained as a result of the machine learning by the learning unit 113.

[0019] The generating device 130 generates a virtual viewpoint image using viewpoint images corresponding to a plurality of viewpoints acquired from the imaging device 120. Here, the virtual viewpoint image is an image that is pseudo / virtually generated from a plurality of viewpoint images, which is a captured image that would be obtained when the imaging device 120 is installed at a position and attitude corresponding to the set virtual viewpoint and photographed. However, in the implementation of the present invention, the virtual viewpoint image is not necessarily limited to an image that is virtually generated. For example, the present invention does not exclude the case where, when the same viewpoint as the viewpoint at which photographing is performed by the imaging device 120 is set as the virtual viewpoint, an image that is substantially the same as the viewpoint image photographed at that viewpoint is output as the virtual viewpoint image.

[0020] The control unit 131 is, for example, a CPU, and controls the operation of each block included in the generating device 130. More specifically, the control unit 131 controls the operation of each block by reading out the operation program of each block from the storage area of ​​the storage unit 132, expanding it in the expansion area, and executing it.

[0021] The storage unit 132 is a storage device configured with a ROM, a RAM, a hard disk, etc. In addition to the operation programs of each block included in the generating device 130, the storage unit 132 stores parameters and the like necessary for the operation of each block, and temporarily stores data output in the operation of each block.

[0022] The acquisition unit 133 acquires various information required for executing an output process for outputting a virtual viewpoint image corresponding to a set virtual viewpoint from the storage unit 132. In the generation device 130 of this embodiment, various information is stored in the storage unit 132 via the network 160 prior to execution of the output process. Specifically, the storage unit 132 stores a viewpoint image (downsampled image) acquired from the storage unit 122 of the imaging device 120 via the network 160 and feature information corresponding to the viewpoint image. The storage unit 132 also stores a trained model acquired from the storage unit 112 of the learning device 110 via the network 160. The acquisition unit 133 also acquires information on the position of the virtual viewpoint based on an operation input accepted via the input device 140. Here, the input device 140 is a user interface such as a mouse, a keyboard, or a touch panel sensor, and outputs a control signal corresponding to an operation input made by a user.

[0023] The image processing unit 134 executes an output process to output a virtual viewpoint image corresponding to the set virtual viewpoint. The details of the output process will be described later, but the image processing unit 134 generates and outputs a virtual viewpoint image based on information acquired by the acquisition unit 133 when executing the output process. The virtual viewpoint image output by the image processing unit 134 is presented to the user via a display device 150 such as a liquid crystal display or a projector. Alternatively, the virtual viewpoint image may be output to and stored in a recording medium such as a semiconductor memory, a hard disk, or a server on a network (not shown).

[0024] Overview of Virtual Viewpoint Image Generation The control unit 131 is, for example, a CPU, and controls the operation of each block included in the generating device 130. More specifically, the control unit 131 controls the operation of each block by reading out the operation program of each block from the storage area of ​​the storage unit 132, expanding it in the expansion area, and executing it.

[0025] The storage unit 132 is a storage device configured with a ROM, a RAM, a hard disk, etc. In addition to the operation programs of each block included in the generating device 130, the storage unit 132 stores parameters and the like necessary for the operation of each block, and temporarily stores data output in the operation of each block.

[0026] The acquisition unit 133 acquires various information required for executing an output process for outputting a virtual viewpoint image corresponding to a set virtual viewpoint from the storage unit 132. In the generation device 130 of this embodiment, various information is stored in the storage unit 132 via the network 160 prior to execution of the output process. Specifically, the storage unit 132 stores a viewpoint image (downsampled image) acquired from the storage unit 122 of the imaging device 120 via the network 160 and feature information corresponding to the viewpoint image. The storage unit 132 also stores a trained model acquired from the storage unit 112 of the learning device 110 via the network 160. The acquisition unit 133 also acquires information on the position of the virtual viewpoint based on an operation input accepted via the input device 140. Here, the input device 140 is a user interface such as a mouse, a keyboard, or a touch panel sensor, and outputs a control signal corresponding to an operation input made by a user.

[0027] The image processing unit 134 executes an output process to output a virtual viewpoint image corresponding to the set virtual viewpoint. The details of the output process will be described later, but the image processing unit 134 generates and outputs a virtual viewpoint image based on information acquired by the acquisition unit 133 when executing the output process. The virtual viewpoint image output by the image processing unit 134 is presented to the user via a display device 150 such as a liquid crystal display or a projector. Alternatively, the virtual viewpoint image may be output to and stored in a recording medium such as a semiconductor memory, a hard disk, or a server on a network (not shown).

[0028] Overview of Virtual Viewpoint Image Generation Hereinafter, an overview of the process of generating a virtual viewpoint image corresponding to a set virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints in the image processing system 100 configured as above will be described. In this embodiment, the virtual viewpoint image is generated by selecting viewpoint images corresponding to two viewpoints close to the virtual viewpoint from among the plurality of viewpoint images, and synthesizing the images obtained by applying super-resolution processing to these viewpoint images. That is, in the image processing system 100 of this embodiment, when generating the virtual viewpoint image, the estimation of the three-dimensional shape of the subject as in Patent Document 1 is not performed, and Image Based Rendering (IBR) is adopted, which synthesizes images corresponding to a plurality of viewpoints to generate the virtual viewpoint image.

[0029] By the way, the super-resolution processing performed on the viewpoint image tends to increase the amount of calculation because it is necessary to generate an output image in a state in which the number of pixels of the input image is increased while ensuring a certain frequency component of the input image such as an edge, etc. Such image processing with a large amount of calculations has a high calculation load when executed by an information processing device such as a general-purpose PC or a mobile terminal, and there is a possibility that a suitable processing result is not obtained or that the device malfunctions. Furthermore, if the information processing device is battery-powered, the execution of the image processing may cause the battery to be consumed. For this reason, in the image processing system 100 of this embodiment, a trained model that machine-learns the relationship between the input image and the output image to which the super-resolution processing has been applied in advance is constructed in the learning device 110. Then, by using the trained model, the calculation load related to the generation of the virtual viewpoint image in the generation device 130 is reduced. In other words, by making available a trained model related to the super-resolution processing as in the image processing system 100 of this embodiment, even an information processing device with low calculation power can obtain an image in which a desired image is super-resolved with a small amount of calculation.

[0030] <Pre-trained model> Here, construction of a trained model for super-resolution processing will be described in detail with reference to the drawings. In this embodiment, construction of a trained model for super-resolution processing is performed by using deep machine learning using a neural network (NN).

[0031] First, super-resolution processing using a neural network will be described with reference to Fig. 3. An original image is downsampled, for example, by the area averaging method (downscaled image), and the downscaled image is input to the neural network, which outputs an applied image (upscaled image). For example, the original image and the downscaled image can be a 4K image and a 2K image, respectively. The upscaled image only needs to have a higher resolution (higher spatial resolution) than the downscaled image, but to facilitate understanding of the invention, the following description will be given assuming that the upscaled image has the same number of pixels as the original image.

[0032] In this case, it is preferable that the upscaled image output from the neural network is an image similar to the original image as much as possible. However, when an upscaled image is generated only from a downscaled image, an image similar to the original image may not be generated. For this reason, in the learning device 110 of this embodiment, in addition to the downscaled image and the original image, machine learning is performed using feature information extracted from the original image as a training image. Here, the feature information is assumed to be information that cannot be identified from the downscaled image. By inputting the feature information together with the downscaled image to the neural network, information that is lacking only in the downscaled image is supplemented, and machine learning is performed so that an upscaled image similar to the original image is output. That is, in the learning unit 113, machine learning is performed so as to increase the resolution of a low-resolution training image, and a learned model is constructed.

[0033] In this embodiment, a mask image of the edge region of the original image is used as the feature information used as the training image. The mask image of the edge region is a two-dimensional image configured to be able to identify regions in the original image that are determined to have high edge strength, and indicates information that cannot be identified from a downscaled image in which edges are blurred due to the influence of downsampling processing. The mask image of the edge region can be, for example, a binary image in which regions (pixels) showing edge strengths above a threshold are shown in black, and regions showing edge strengths below the threshold are shown in white.

[0034] FIG. 4 illustrates in more detail a neural network provided for constructing a trained model in the learning unit 113 of the learning device 110 of this embodiment. In FIG. 4, images 401 to 403 input as training images are images of partial regions (hereinafter referred to as patch images) showing the same subject (space) captured in the image. Patch images 401, 402, and 403 indicate patches of an original image, a downscaled image, and feature information (mask image), respectively. The size of a patch image depends on the spatial resolution of the image from which it is extracted, so for example, patch image 402 is composed of 64×64 pixels, and patch image 401 and patch image 403 are composed of 128×128 pixels.

[0035] In the process of machine learning (training), the input patch image 402 and the patch image 403 are input to each processing layer in FIG. 4, and an estimated patch 404 is generated by upscaling the patch image 402 through super-resolution processing. In this embodiment, the estimated patch 404 has the same number of pixels as the patch image 401 and the patch image 403.

[0036] In FIG. 4, the processing layer indicated as "CN" is a convolution layer. In CN, the sum of the input, the convolution of the filter, and the bias is derived, and the result is nonlinearly transformed by the activation function. The initial values ​​of each component of the filter and the bias are arbitrary, and in this embodiment, they are determined by random numbers. For example, ReLU (Rectified Linear Unit) or a sigmoid function can be used as the activation function. The output of each layer except the final layer is called a feature map. In addition, the processing layer indicated as "+" performs concatenation in the channel direction. In addition, the processing layer indicated as "PS" is a PixelShuffle layer. PS generates a high-resolution feature map by rearranging a low-resolution feature map. For example, PS rearranges a low-resolution feature map of 64 x 64 x 64 pixels to generate a high-resolution feature map of 128 x 128 x 16 pixels. The size of the feature map represents the length x width x depth (channels). The feature map obtained by PS can also be further convoluted to adjust the number of channels of the feature map. In the neural network of this embodiment, PixelShuffle is used to increase the resolution of the feature map, but the present invention is not limited to this. For example, deconvolution (or transposed convolution) or interpolation can also be used to increase the resolution of the feature map. The processing layer marked with "+" synthesizes the feature map and derives the sum of each element.

[0037] The learning unit 113 then evaluates the error between the obtained estimated patch 404 and the patch image 401 of the original image, and updates the weights (learning weights) of the neural network. In this embodiment, it is assumed that the error backpropagation method is used to update the weights. To evaluate the error between the estimated patch 404 and the patch image 401 of the original image, the mean square error (MSE) can be used as a loss function. Note that the learning unit 113 of this embodiment uses mini-batch learning that updates the weights based on the error between a plurality of patch images 401 of the original image and the corresponding estimated patch 404, but it should be understood that the implementation of the present invention is not limited to this. For example, the learning unit 113 may use batch learning or online learning when constructing a trained model. In addition, for example, the mean absolute error (MAE) or Huber loss can be used as a loss function related to the error evaluation.

[0038] The learning unit 113 can adjust the weights of the neural network by repeatedly performing machine learning using the training image group in this way, and can construct a trained model that can infer and output super-resolution images to which desired super-resolution processing has been applied for images of various scenes. That is, the trained model constructed by the learning unit 113 can infer a super-resolution image of the image to be processed by inputting the image to be processed (downscaled image) and feature information corresponding to the image to be processed (feature information extracted from the original image).

[0039] 5 shows a flow of a series of machine learning processes related to the construction of a trained model. The processes related to this flowchart can be executed by the learning unit 113 by the control unit 111 executing a corresponding program using the storage unit 112. As training images used in this process, for example, a viewpoint image (downscaled image) obtained by shooting with the imaging device 120, an image (original image) before application of downsampling processing of the viewpoint image, and feature information extracted from the image before application can be used. These image groups are transmitted from the imaging device 120 to the learning device 110 via the network 160, for example, and stored in the storage unit 112.

[0040] In S501, the learning unit 113 reads out the original image, the downscaled image, and the feature information from the storage unit 112.

[0041] In S502, the learning unit 113 acquires, from each image read out in S501, an image of a corresponding partial region as a patch image.

[0042] In S503, the learning unit 113 inputs the patch image 402 of the downscaled image and the patch image 403 of the feature information to a neural network, and generates an estimated patch 404.

[0043] In S504, the learning unit 113 evaluates the error between the estimated patch 404 and the patch image 401 of the original image, and updates the weights of the neural network.

[0044] In S505, the learning unit 113 judges whether the number of iterations (processing of S502 to S504) of machine learning (updating weights) reaches a specified value. If the learning unit 113 judges that the number of iterations has reached the specified value, it ends the processing related to this flowchart, and stores the weight information as information of the constructed trained model in the storage unit 112. If the learning unit 113 judges that the number of iterations has not reached the specified value, it returns the processing to S502 and continues learning.

[0045] Next, the generation of a virtual viewpoint image performed by the generating device 130 using the trained model thus constructed in advance will be described. Fig. 6 shows a functional configuration related to the generation of a virtual viewpoint image realized by the image processing unit 134 of this embodiment. Each functional configuration is realized by the control unit 131 controlling the operation of the image processing unit 134 by executing a corresponding processing program using the storage unit 132.

[0046] The estimation unit 601 estimates the position and orientation of the imaging device 120 when each viewpoint image acquired from the imaging device 120 was captured. The estimation unit 601 estimates the position and orientation of the imaging device 120 at the time of capturing each viewpoint image, for example, by using a known technique such as SfM (Structure from Motion). The position and orientation may be derived using sensor outputs of an acceleration sensor or an angular velocity sensor provided in the imaging device 120, or may be derived by combining these. In this embodiment, the estimation unit 601 of the generation device 130 is described as estimating the position and orientation of the imaging device 120 at the time of capturing the viewpoint image, but it goes without saying that the embodiment of the present invention is not limited to this. For example, in a mode in which information on the position and orientation of the imaging device 120 at the time of capturing the viewpoint image is associated with the viewpoint image in the imaging device 120 and then the viewpoint image is transmitted to the generation device 130, the estimation unit 601 may simply acquire the position and orientation information by referring to the information.

[0047] The setting unit 602 sets a virtual viewpoint for generating a virtual viewpoint image. The virtual viewpoint is set based on an operation input related to viewpoint operation (selection of position and orientation) received via the input device 140, for example. Alternatively, the virtual viewpoint may be selected from a combination of positions and orientations of virtual viewpoints stored in advance.

[0048] The upsampling unit 603 uses the trained model constructed by the learning device 110 to output an image in which the input image has been super-resolved, that is, an image in which the spatial resolution of the viewpoint image has been increased from that of the viewpoint image. In this embodiment, the upsampling unit 603 specifies a viewpoint image used to generate a virtual viewpoint image according to the virtual viewpoint set by the setting unit 602, and inputs the viewpoint image and feature information corresponding to the viewpoint image to the trained model. Here, the trained model becomes available in the image processing unit 134 by constructing a corresponding neural network based on the weight information received from the learning device 110. The upsampling unit 603 obtains an image in which the viewpoint image has been super-resolved (hereinafter referred to as a super-resolution viewpoint image) as an output for the input from the trained model. The super-resolution viewpoint image is assumed to have the same resolution (4K) as the original image (image before the downsampling process is applied).

[0049] In this embodiment, the viewpoint images used to generate the virtual viewpoint image are determined as follows. In the illustrated embodiment, in the bird's-eye view of the shooting scene in FIG. 7, the imaging device 120 is placed at each of the viewpoints 701a to i to shoot the subject 702, and the obtained viewpoint images and feature information are stored in the storage unit 132. In this embodiment, when the virtual viewpoint 703 is set by the setting unit 602, the upsampling unit 603 selects two viewpoints close to the virtual viewpoint 703 for generating the virtual viewpoint image. In the example of FIG. 7, the viewpoints 701a and b are the two viewpoints closest to the virtual viewpoint 703, so the upsampling unit 603 selects these viewpoints, and the viewpoint images and feature information corresponding to the viewpoints are input to the trained model.

[0050] For example, when the viewpoint image corresponding to the viewpoint 701a is as shown in FIG. 8(a) and the feature information corresponding to the viewpoint 701a is as shown in FIG. 8(b), the super-resolution viewpoint image output by the upsampling unit 603 is as shown in FIG. 8(c). As shown in the figure, the viewpoint image 801 is generated by downsampling the original image, so that the edges of each part of the face and the clothes are blurred in the image 802 of the subject 702. In contrast, in the feature information 811, the area determined to have a strong edge strength in the original image is shown by black pixels. In contrast, in the super-resolution viewpoint image 821 output using these images as input, the edges of each part of the face and the clothes are in a highly detailed state in the image 822 of the subject 702. In this way, even when the viewpoint image 801 and the feature information 811, which have a reduced amount of information compared to the original image, are transmitted from the imaging device 120 to the generating device 130, the super-resolution viewpoint image 821 in which the edge strength of the original image is reproduced can be generated. In other words, in the image processing system 100 of this embodiment, the generating device 130 can generate an image equivalent to the original image from the viewpoint image 801 and feature information 811 with less information, without acquiring an original image with a large amount of information from the imaging device 120. In other words, in the image processing system 100 of this embodiment, even in a mode in which a virtual viewpoint image is generated in a communication terminal such as a smartphone or a tablet, the transmission load when transmitting and receiving necessary data can be reduced.

[0051] The generating unit 604 generates a virtual viewpoint image corresponding to the virtual viewpoint using the super-resolution viewpoint images related to the two viewpoints output from the upsampling unit 603. The generating unit 604 can adopt, for example, the View Morphing method described in SM Seitz and CR Dyer: “View Morphing”, Proc. SIGGRAPH '96, pp.21-31 (1996) as a method for generating a virtual viewpoint image without estimating the three-dimensional shape of the subject. By using this method, the generating unit 604 can generate a virtual viewpoint image 901 as shown in FIG. 9 by synthesizing, for example, the super-resolution viewpoint image 821 shown in FIG. 8(c) and the super-resolution viewpoint image corresponding to the viewpoint 701b. As shown in the figure, since the virtual viewpoint image 901 is generated using the highly detailed super-resolution viewpoint image, the image 902 of the subject 702 also has highly detailed edges of each part of the face and the clothes.

[0052] Output Processing Hereinafter, a specific process of output processing for outputting a virtual viewpoint image corresponding to a set virtual viewpoint, which is executed in the generating device 130 of this embodiment, will be described with reference to the flowchart in Fig. 10. The process corresponding to the flowchart can be realized by the control unit 131 reading out a corresponding processing program stored in a storage area of ​​the storage unit 132, for example, and expanding the program in a deployment area of ​​the storage unit 132 for execution.

[0053] This output process will be described as being started when an operation input related to an output request for a virtual viewpoint image is made for a shooting scene in which viewpoint images have been shot in advance. Note that, prior to the execution of this output process, viewpoint images and their feature information obtained by shooting from multiple viewpoints by the imaging device 120 are stored in the storage unit 132, and estimation of the position and orientation of the imaging device 120 at the time of shooting for each viewpoint image by the estimation unit 601 is completed. Also, information (weight information) on a trained model constructed by the learning device 110 is stored in advance in the storage unit 132, and a similar trained model is constructed in the image processing unit 134. In the following description, each step of this output process will be described as being executed by each functional configuration of the image processing unit 134, but it will be understood that each step is executed by the image processing unit 134 (image processing circuit) or the control unit 131, which are hardware.

[0054] In S1001, the setting unit 602 sets a virtual viewpoint. The setting of the virtual viewpoint may be performed based on information of an operation input related to a change of the virtual viewpoint or information of an operation input related to a selection of a previously set virtual viewpoint.

[0055] In S1002, the upsampling unit 603 selects, from the viewpoints corresponding to the viewpoint images stored in the storage unit 132, two viewpoints adjacent to the virtual viewpoint set in S1001.

[0056] In S1003, the upsampling unit 603 generates super-resolution viewpoint images corresponding to the two viewpoints selected in S1002. More specifically, the upsampling unit 603 inputs the corresponding viewpoint image and feature information for each of the two selected viewpoints to the trained model, and causes the trained model to output the super-resolution viewpoint image corresponding to the viewpoint.

[0057] In S1004, the generation unit 604 generates a virtual viewpoint image corresponding to the virtual viewpoint by synthesizing the two super-resolution viewpoint images generated in S1003, and outputs the virtual viewpoint image, thereby completing this output process.

[0058] As described above, the image processing device of this embodiment can generate a more detailed virtual viewpoint image. Note that, in this embodiment, it has been described that super-resolution viewpoint images are generated for two viewpoints adjacent to the virtual viewpoint, and the virtual viewpoint image is generated by synthesizing these images, but the implementation of the present invention is not limited to this. It is also possible to generate super-resolution viewpoint images corresponding to two or more viewpoints according to the line of sight direction and height set as the virtual viewpoint, and use these images to generate the virtual viewpoint image.

[0059] [Embodiment 2] In the above-described embodiment, a virtual viewpoint image is generated by IBR after super-resolution of viewpoint images and feature information corresponding to a plurality of viewpoints. In this embodiment, an example is described in which the present invention is applied to a method (Image Based Modeling and Rendering: IBMR) in which a three-dimensional model of a subject is generated based on viewpoint images and a virtual viewpoint image is rendered using the three-dimensional model, as described in Patent Document 1. In the description of this embodiment, the configuration of the image processing system 100 and the trained model constructed by the training device 110 are the same as those in the first embodiment, and description thereof will be omitted.

[0060] Overview of Virtual Viewpoint Image Generation 11 shows a functional configuration related to the generation of a virtual viewpoint image, which is realized by the image processing unit 134 of this embodiment. Each functional configuration is realized by the control unit 131 controlling the operation of the image processing unit 134 by executing a corresponding processing program using the storage unit 132. As shown in the figure, the functional configuration related to the generation of a virtual viewpoint image of this embodiment is different from the configuration of the first embodiment in the upsampling unit 1103, the generation unit 1104, and the warping processing unit 1105.

[0061] The generating unit 1104 estimates the three-dimensional shape of the subject based on a plurality of viewpoint images, and generates an image corresponding to the virtual viewpoint by drawing the three-dimensional shape for the virtual viewpoint set as in Patent Document 1. The estimation of the three-dimensional shape can use, for example, a known MVS (Multi-View Stereo) technology. Alternatively, the estimation of the three-dimensional shape can use, for example, neural rendering such as NeRF (Neural Radiance Fields) described in Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, In ECCV, 2020. In drawing the three-dimensional shape, the generating unit 1104 applies a texture generated using the average value of the plurality of viewpoint images to the three-dimensional shape. For this reason, in the image corresponding to the virtual viewpoint generated by the generation unit 1104, the edges of the facial features and clothing in the subject image 1201 are blurred, as shown in Fig. 12(a). Although details will be described later, in this embodiment, the image corresponding to the virtual viewpoint generated by the generation unit 1104 is different from the image output as the virtual viewpoint image in the output process. For this reason, in the following description, the image corresponding to the virtual viewpoint generated by the generation unit 1104 will be referred to as an intermediate image.

[0062] The warping processor 1105 executes a warping process to generate feature information (hereinafter referred to as virtual feature information) in a state corresponding to an angle of view corresponding to a virtual viewpoint from feature information acquired corresponding to a viewpoint image. When executing the warping process, the warping processor 1105 refers to a depth map indicating the distribution of the subject in the depth direction at the virtual viewpoint. The depth map is generated based on the positional relationship between the three-dimensional shape of the subject and the virtual viewpoint that renders it, for example, as an intermediate signal when an intermediate image is generated by the generator 1104. When the intermediate image is as shown in FIG. 12(a), the depth map has a form as shown in FIG. 12(b). The depth map indicates the distance (depth) to the subject of each pixel as a grayscale map. In the depth map of FIG. 12(b), objects close to the virtual viewpoint are shown in white, objects far from the virtual viewpoint are shown in black, and subjects that are in the middle distance and are rendered in focus are shown in 50% gray. In the illustrated example, the depth map is formed with three levels of pixel values ​​so that the subject area corresponding to the area determined to have a high edge strength can be easily extracted. With reference to such a depth map, the warping processing unit 1105 matches the shape of the area determined to have a high edge strength, which appears in the mask image of the edge area, which is the feature information, with the shape of the subject area represented in the depth map. The warping process is performed by applying perspective projection transformation based on the position and orientation information of the virtual viewpoint and the depth map to the feature information. Note that the virtual feature information may be generated by warping the feature information corresponding to any one of the viewpoints according to the virtual viewpoint, or may be generated by averaging the images obtained by warping each of the feature information corresponding to a plurality of viewpoints.

[0063] The upsampling unit 1103 outputs an image in which an input image is super-resolved using a trained model constructed by the learning device 110, similarly to the upsampling unit 603 of the first embodiment. In the upsampling unit 1103 of this embodiment, the images input to the trained model are an intermediate image generated by the generation unit 1104 and virtual feature information generated by the warping processing unit 1105. That is, the inputs to the trained model are both a (low-resolution) rendering image corresponding to a virtual viewpoint and feature information corresponding to the virtual viewpoint, and the trained model directly outputs a (super-resolved) virtual viewpoint image corresponding to the virtual viewpoint. In other words, unlike the first embodiment, the image to be super-resolved for one virtual viewpoint by the upsampling unit 1103 of this embodiment is only one type (intermediate image). In this way, based on the intermediate image shown in FIG. 12(a) and the depth map shown in FIG. 12(b), the upsampling unit 1103 can similarly generate a virtual viewpoint image 901 as shown in FIG. 9.

[0064] Output Processing Hereinafter, a specific process of output processing for outputting a virtual viewpoint image corresponding to a set virtual viewpoint, which is executed in the generating device 130 of this embodiment, will be described with reference to the flowchart in Fig. 13. The process corresponding to the flowchart can be realized by the control unit 131 reading out a corresponding processing program stored in a storage area of ​​the storage unit 132, for example, and expanding the program in a deployment area of ​​the storage unit 132 for execution.

[0065] This output process will be described as being started when an operation input related to an output request for a virtual viewpoint image is made for a shooting scene in which viewpoint images have been shot in advance. Note that, prior to the execution of this output process, viewpoint images and their feature information obtained by shooting from multiple viewpoints by the imaging device 120 are stored in the storage unit 132, and estimation of the position and orientation of the imaging device 120 at the time of shooting for each viewpoint image by the estimation unit 601 is completed. Also, information (weight information) on a trained model constructed by the learning device 110 is stored in advance in the storage unit 132, and a similar trained model is constructed in the image processing unit 134. In the following description, each step of this output process will be described as being executed by each functional configuration of the image processing unit 134, but it will be understood that each step is executed by the image processing unit 134 (image processing circuit) or the control unit 131, which are hardware. Also, in the description of the output process of this embodiment, the same reference numbers are given to steps that perform the same processing as in embodiment 1, and the description will be omitted.

[0066] When a virtual viewpoint is set in S1001, the generation unit 1104 generates an intermediate image corresponding to the virtual viewpoint in S1301. As described above, the generation of the intermediate image includes the steps of estimating the three-dimensional shape of the subject based on the group of viewpoint images, applying texture, and drawing based on the virtual viewpoint.

[0067] In S1302, the warping processing unit 1105 generates virtual feature information corresponding to the virtual viewpoint by referring to the depth map output in relation to the generation of the intermediate image.

[0068] In S1303, the upsampling unit 1103 generates a virtual viewpoint image corresponding to the virtual viewpoint image, outputs the virtual viewpoint image, and completes this output process. More specifically, the upsampling unit 1103 outputs the super-resolved image output by inputting the intermediate image generated in S1301 and the virtual feature information generated in S1302 to the trained model, as a virtual viewpoint image corresponding to the virtual viewpoint.

[0069] As described above, according to the image processing device of this embodiment, it is possible to generate a more detailed virtual viewpoint image for a virtual viewpoint that is set in the same manner as in the first embodiment.

[0070] [Variation 1] In the above-described second embodiment, the upsampling unit 1103 is described as inputting an intermediate image and virtual feature information, and outputting a virtual viewpoint image obtained by super-resolving the intermediate image to a trained model. However, applying super-resolution processing may not necessarily result in an intermediate image having a preferable image quality.

[0071] For example, when an original image that has a large proportion of areas with low edge strength such as the sky is subjected to a process of reducing spatial resolution by downsampling or converting to point cloud data and rendering, a false contour may appear in the resulting image. When a super-resolution process is applied to an image in which such a false contour appears, the false contour may be emphasized, and an unnatural image may be generated in which edges appear in areas where there were no edges. Therefore, the application of a super-resolution process may not necessarily lead to the generation of a virtual viewpoint image with suitable image quality.

[0072] For this reason, for example, based on the virtual feature information output in S1302, the degree of effect when the super-resolution process is applied to the intermediate image may be evaluated, and the method of generating the virtual viewpoint image may be changed based on the evaluation. Here, the effect when the super-resolution process is applied may be evaluated by obtaining, for example, the occupancy rate (or area ratio) of the area with high edge strength in the virtual feature information as an evaluation value.

[0073] For example, if the evaluation value exceeds a predetermined threshold, the intermediate image has a high occupancy rate of areas determined to have high edge strength, and therefore it can be determined that a virtual viewpoint image with many areas in which fineness is suitably improved can be obtained when super-resolution processing is applied. On the other hand, if the evaluation value is below the predetermined threshold, the intermediate image has a low occupancy rate of areas determined to have high edge strength, and there may be areas in which false contours appear, and therefore it can be determined that a virtual viewpoint image with deteriorated image quality can be obtained when super-resolution processing is applied.

[0074] Therefore, when the evaluation value of the virtual feature information exceeds a predetermined threshold, the upsampling unit 1103 outputs a super-resolution image, which is inputted from the intermediate image and the virtual feature information and outputted to the trained model as in S1303 of the output process of the second embodiment, as a virtual viewpoint image. On the other hand, when the evaluation value of the virtual feature information is below a predetermined threshold, the effect of the super-resolution process is deemed low, and the upsampling unit 1103 increases the resolution of the intermediate image, for example, by bilinear interpolation enlargement, and further applies existing edge enhancement processing to generate a virtual viewpoint image. Alternatively, the upsampling unit 1103 may generate a virtual viewpoint image by weighting the image obtained in this way and the image obtained by the super-resolution process, using the evaluation value of the virtual feature information as a weight. Here, the image processing applied to the intermediate image, which does not depend on the trained model, is not limited to the above, and can be changed to other image processing for improving image quality, for example.

[0075] In this way, it is possible to reduce degradation in image quality caused by emphasis of false contours due to super-resolution processing.

[0076] [Variation 2] In the above-mentioned second embodiment and first modified example, an intermediate image drawn based on the estimation of the three-dimensional shape of the subject is input to a trained model together with virtual feature information obtained by warping the virtual viewpoint, to generate a super-resolved virtual viewpoint image. However, the present invention is not limited to this. For example, as in the first embodiment, an intermediate image may be generated by synthesizing a viewpoint image (downscaled image) close to the virtual viewpoint, and the intermediate image may be input to a trained model together with virtual feature information to generate a virtual viewpoint image.

[0077] [Variation 3] In the above-mentioned embodiment and modified example, a mask image showing edge strength is input as feature information extracted from an original image in order to supply information not included in a downscaled image to a trained model that realizes super-resolution processing. However, the feature information is not limited to a mask image showing edge strength, and information other than a mask image of edge strength can be used as long as it is information that cannot be specified only from a downscaled image. For example, an edge image itself in which an edge of an original image is extracted, an image in which an area in which an edge strength is equal to or greater than a predetermined value is masked, and coordinate information of a mask pixel position can be used as feature information. For example, an image in which the position of a Harris Corner feature point in an original image is extracted can be used as feature information. For example, a difference image between an image in which a downscaled image is enlarged by bilinear interpolation and the original image, or a difference image between an image in which an original image is reduced by bilinear interpolation and the downscaled image can be used as feature information. Information on the frequency analysis result of an original image can also be used as feature information. Here, the information on the frequency analysis result may be, for example, an image of a high-frequency component generated when a downscaled image is generated by performing discrete wavelet transform on an original image. Alternatively, as information on the frequency analysis result, for example, parameters at the time of wavelet transform of each viewpoint image, such as the wavelet index number N of the Daubechies wavelet db(N), can be used as feature information related to the viewpoint. In addition, when each viewpoint image is obtained continuously in a predetermined time sequence like a video, information on the movement of the subject image in the original image can be used as feature information. Such information includes, for example, optical flow between frames of the original image and motion estimation information indicating in which direction and at what speed a predetermined rectangular area in a frame moves. In addition, the feature information may be a combination of the information listed above, and as a result, it can include at least one of these pieces of information.

[0078] [Summary of the embodiment and modifications] The disclosure of this specification includes the following image processing device, control method, program, and image processing system. (Item 1) A setting means for setting a virtual viewpoint; A generating means for generating a virtual viewpoint image corresponding to the virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints; An acquisition means for acquiring feature information of each of the viewpoint images; having The generating means generates the virtual viewpoint image by executing a process of increasing a spatial resolution of the image based on the feature information. 13. An image processing device comprising: (Item 2) The generating means includes: generating an intermediate image corresponding to the virtual viewpoint based on the viewpoint image; generating virtual feature information corresponding to the virtual viewpoint based on the feature information of the viewpoint image; The virtual viewpoint image is generated by performing a process of increasing a spatial resolution of the intermediate image based on the virtual feature information. 2. The image processing device according to item 1, (Item 3) 3. The image processing device according to item 2, wherein the generating means generates the virtual feature information by warping the feature information of the viewpoint image in accordance with the virtual viewpoint. (Item 4) 4. The image processing device according to item 2 or 3, wherein the process of increasing the spatial resolution includes a process of acquiring a super-resolution image of the image to be processed inferred by a trained model by inputting an image to be processed and the feature information corresponding to the image to be processed. (Item 5) Further comprising an evaluation means for deriving an evaluation value indicating an effect of a process for increasing the spatial resolution by the trained model based on the virtual feature information, The generating means generates the virtual viewpoint image by synthesizing a super-resolution image acquired based on the virtual feature information by the trained model and the intermediate image to which enhancement processing has been applied, using the evaluation value derived by the evaluating means as a weight. 5. The image processing device according to item 4, (Item 6) The generating means includes: generating super-resolution viewpoint images corresponding to the plurality of viewpoints by executing a process for increasing a spatial resolution of each of the viewpoint images based on the feature information of the viewpoint image; The virtual viewpoint image is generated based on the super-resolution viewpoint images corresponding to the plurality of viewpoints. 2. The image processing device according to item 1, (Item 7) 7. The image processing device according to item 6, wherein the generating means generates the virtual viewpoint image by synthesizing the super-resolution viewpoint images corresponding to the plurality of viewpoints. (Item 8) 8. The image processing device according to item 6 or 7, wherein the process of increasing the spatial resolution includes a process of acquiring a super-resolution image of the image to be processed inferred by a trained model by inputting an image to be processed and the feature information corresponding to the image to be processed. (Item 9) The viewpoint image is an image generated by downsampling an original image, The feature information of the viewpoint image is information extracted from an original image of the viewpoint image. 9. The image processing device according to any one of items 1 to 8, (Item 10) 10. The image processing device according to item 9, wherein the feature information of the viewpoint image is information that is not included in the viewpoint image. (Item 11) The image processing device according to any one of items 1 to 10, characterized in that the feature information includes at least one of an edge image of the image, a mask image indicating edge strength of the image, feature points of the image, a difference image between the image and an image obtained by reducing or enlarging the image, information regarding the movement of a subject image within the image, and information regarding a frequency analysis result of the image. (Item 12) A setting step of setting a virtual viewpoint; A generating step of generating a virtual viewpoint image corresponding to the virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints; an acquisition step of acquiring feature information of each of the viewpoint images; having In the generating step, a process of increasing the spatial resolution of the image based on the feature information is executed, whereby the virtual viewpoint image is generated. 23. A method for controlling an image processing apparatus comprising the steps of: (Item 13) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11. (Item 14) An image processing system including an imaging device that captures images from different viewpoints and an image processing device, the imaging device transmits viewpoint images obtained by shooting from the different viewpoints and feature information of the viewpoint images to the image processing device; The image processing device includes: A setting means for setting a virtual viewpoint; a generating means for generating a virtual viewpoint image corresponding to the virtual viewpoint based on the plurality of viewpoint images received from the imaging device; having The generating means generates the virtual viewpoint image by executing a process of increasing a spatial resolution of an image based on the feature information received from the imaging device. 1. An image processing system comprising:

[0079] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0080] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0081] 100: image processing system, 120: imaging device, 130: generation device, 134: image processing unit, 602: setting unit, 603: upsampling unit, 604: generation unit, 1103: upsampling unit, 1104: generation unit, 1105: warping processing unit

Claims

1. A setting means for setting a virtual viewpoint; A generating means for generating a virtual viewpoint image corresponding to the virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints; An acquisition means for acquiring feature information of each of the viewpoint images; having The generating means generates the virtual viewpoint image by executing a process of increasing a spatial resolution of the image based on the feature information.

13. An image processing device comprising:

2. The generating means includes: generating an intermediate image corresponding to the virtual viewpoint based on the viewpoint image; generating virtual feature information corresponding to the virtual viewpoint based on the feature information of the viewpoint image; The virtual viewpoint image is generated by performing a process of increasing a spatial resolution of the intermediate image based on the virtual feature information.

2. The image processing device according to claim 1,

3. The image processing apparatus according to claim 2 , wherein the generating means generates the virtual feature information by performing warping processing on the feature information of the viewpoint image in accordance with the virtual viewpoint.

4. The image processing device according to claim 2, characterized in that the process of increasing the spatial resolution includes a process of acquiring a super-resolution image of the image to be processed inferred by a trained model by inputting the image to be processed and the feature information corresponding to the image to be processed.

5. Further comprising an evaluation means for deriving an evaluation value indicating an effect of a process for increasing the spatial resolution by the trained model based on the virtual feature information, The generating means generates the virtual viewpoint image by synthesizing a super-resolution image acquired based on the virtual feature information by the trained model and the intermediate image to which enhancement processing has been applied, using the evaluation value derived by the evaluating means as a weight.

5. The image processing device according to claim 4.

6. The generating means includes: generating super-resolution viewpoint images corresponding to the plurality of viewpoints by executing a process for increasing a spatial resolution of each of the viewpoint images based on the feature information of the viewpoint image; The virtual viewpoint image is generated based on the super-resolution viewpoint images corresponding to the plurality of viewpoints.

2. The image processing device according to claim 1,

7. The image processing apparatus according to claim 6 , wherein the generating means generates the virtual viewpoint image by synthesizing the super-resolution viewpoint images corresponding to the plurality of viewpoints.

8. The image processing device according to claim 6, characterized in that the process of increasing the spatial resolution includes a process of acquiring a super-resolution image of the image to be processed inferred by a trained model by inputting the image to be processed and the feature information corresponding to the image to be processed.

9. The viewpoint image is an image generated by downsampling an original image, The feature information of the viewpoint image is information extracted from an original image of the viewpoint image.

2. The image processing device according to claim 1,

10. The image processing apparatus according to claim 9 , wherein the feature information of the viewpoint image is information that is not included in the viewpoint image.

11. 2. The image processing device according to claim 1, characterized in that the feature information includes at least one of an edge image of the image, a mask image indicating edge strength of the image, feature points of the image, a difference image between the image and an image obtained by reducing or enlarging the image, information regarding the movement of a subject image within the image, and information regarding the results of frequency analysis of the image.

12. A setting step of setting a virtual viewpoint; A generating step of generating a virtual viewpoint image corresponding to the virtual viewpoint based on viewpoint images corresponding to a plurality of viewpoints; an acquisition step of acquiring feature information of each of the viewpoint images; having In the generating step, a process of increasing the spatial resolution of the image based on the feature information is executed, whereby the virtual viewpoint image is generated.

23. A method for controlling an image processing apparatus comprising the steps of:

13. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

14. An image processing system including an imaging device that captures images from different viewpoints and an image processing device, the imaging device transmits viewpoint images obtained by shooting from the different viewpoints and feature information of the viewpoint images to the image processing device; The image processing device includes: A setting means for setting a virtual viewpoint; a generating means for generating a virtual viewpoint image corresponding to the virtual viewpoint based on the plurality of viewpoint images received from the imaging device; having The generating means generates the virtual viewpoint image by executing a process of increasing a spatial resolution of an image based on the feature information received from the imaging device.

1. An image processing system comprising:

Images