Image processing apparatus, information processing method, and storage medium

Abstract

To suppress deterioration in image quality in generation of a virtual viewpoint image.SOLUTION: A RAW image is acquired. A characteristic of processing for developing a RAW image is set based on a parameter related to a foreground image extracted from the RAW image and used when a virtual viewpoint image is generated. The RAW image is developed based on the characteristics.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an information processing method, and a program. [Background technology]

[0002] A technology that generates a virtual viewpoint image from a specified virtual viewpoint using multiple images captured by multiple imaging devices is attracting attention. Generally, the sensors of imaging devices have a Bayer array, and RAW images are acquired from the imaging devices, so they must be developed after acquisition to restore them to RGB images. Furthermore, there are multiple development methods with different characteristics, such as a characteristic that prioritizes noise suppression or a characteristic that prioritizes texture sharpness. Patent Document 1 describes a method of capturing images of a subject using multiple imaging devices installed in different positions, and generating a virtual viewpoint image using the three-dimensional shape of the subject estimated from the captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-45920 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when generating a virtual viewpoint image, the characteristics of the developing means used to develop the captured images differ depending on the intended use of each captured image when generating the virtual viewpoint image. However, Patent Document 1 does not mention controlling the characteristics of the developing means. If the same characteristics were used to develop images for all imaging devices, a problem would arise in that the image quality of the virtual viewpoint image would deteriorate due to a mismatch between the intended use of the captured images and the characteristics of the developing means.

[0005] An object of the present invention is to suppress deterioration in image quality when generating a virtual viewpoint image. [Means for solving the problem]

[0006] To achieve the object of the present invention, for example, an image processing device according to one embodiment includes the following configuration: an acquisition unit that acquires a RAW image, a setting unit that sets characteristics of processing for developing the RAW image based on parameters related to a foreground image extracted from the RAW image and used when generating a virtual viewpoint image, and a development unit that develops the RAW image based on the characteristics. [Effects of the Invention]

[0007] To suppress deterioration of image quality when generating a virtual viewpoint image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an image processing apparatus. [Figure 3] FIG. 3 is a diagram illustrating interpolation filter coefficients according to the first embodiment. [Figure 4] 4 is a flowchart showing an example of processing by the image processing device according to the first embodiment. [Figure 5] FIG. 10 is a block diagram showing an example of the configuration of an image processing system according to a second embodiment. [Figure 6] FIG. 10 is a diagram for explaining adjustment of interpolation filter coefficients according to the second embodiment. [Figure 7] 10 is a flowchart showing an example of processing by the image processing device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Embodiment 1] <Image Processing System and Virtual Viewpoint Image Generation Function Overview> FIG. 1 is a block diagram showing an example of an image processing system for generating a virtual viewpoint image, including an image processing device according to this embodiment. Some of the devices shown in FIG. 1 are implemented by causing a computer included in the image processing system to execute a computer program stored in a memory serving as a storage medium. However, some or all of these devices may be implemented by hardware. The hardware may include a dedicated circuit (ASIC) or a processor (a reconfigurable processor or DSP). This image processing system includes an imaging device 1, an image processing device 100, a video generation device 200, and a control device 300, which may or may not be built into the same housing. For example, the devices constituting the image processing system may be separate devices connected to each other via signal paths.

[0011] The virtual viewpoint image generated by this image processing system is also called a free viewpoint image, and allows the user to monitor an image corresponding to a viewpoint freely (arbitrarily) designated by the user. For example, the virtual viewpoint image also includes an image corresponding to a viewpoint selected by the user for monitoring from a limited number of virtual viewpoint candidates. Note that this virtual viewpoint designation may be performed by user operation or automatically based on the results of image analysis, etc. In the following, the virtual viewpoint image may be a video or a still image.

[0012] The virtual viewpoint image is generated, for example, by the following method. First, a plurality of imaging devices 1 (cameras) capture images of an imaging area including a subject from a plurality of directions. The imaging area is a three-dimensional space in which imaging is performed, and may include the three-dimensional space used to estimate the three-dimensional shape of the subject described above. This imaging area may be, for example, an area of ​​any height surrounding a stadium field. The imaging area may also be an area corresponding to a concert venue or imaging studio. The plurality of cameras according to this embodiment are installed at different positions and in different directions (postures) so as to surround the imaging area, and each captures images synchronously.

[0013] In this embodiment, the number of cameras included in the multiple cameras is not limited. For example, if the imaging area is a rugby stadium, several tens to several hundreds of cameras may be installed around the stadium. Note that the multiple cameras do not need to be installed around the entire periphery of the imaging area, and may be installed only in a partial direction of the imaging area depending on installation location restrictions, etc. Furthermore, the multiple cameras may include cameras with different angles of view, such as telephoto cameras and wide-angle cameras. For example, by using telephoto cameras to capture players at high resolution, the resolution of the generated virtual viewpoint image can be improved. Furthermore, when capturing images of ball games, since the ball is expected to move over a wide range, the number of cameras used can be reduced by using wide-angle cameras. Furthermore, by combining the imaging areas of wide-angle cameras and telephoto cameras, the degree of freedom in installation position can be improved.

[0014] Next, the image processing device 100 acquires a foreground image in which a foreground region corresponding to a subject such as a person or a ball is extracted from the captured image, and a background image in which a background region other than the foreground region is extracted. The foreground image and the background image have texture information (color information, etc.). In this embodiment, it is assumed that one image processing device is installed corresponding to one imaging device.

[0015] Finally, image generation device 200 generates a foreground model representing the three-dimensional shape of the subject and texture data for coloring the foreground model based on the foreground image. Note that a background model representing the three-dimensional shape of the background, such as a stadium, is assumed to be prepared in advance. Image generation device 200 then maps the texture data onto the foreground model and background model, and performs rendering according to the virtual viewpoint indicated by the virtual viewpoint information, thereby generating a virtual viewpoint image.

[0016] The control device 300 is a device equipped with a display unit and an operation unit, and controls the operation of the image processing device 100 or the video generation device 200. The display unit is configured, for example, by a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate the image processing device 100 and the control device 300. The operation unit is configured, for example, by a keyboard and mouse, a joystick, or a touch panel, and accepts operations by the user and outputs various instructions to the image processing device 100 and the video generation device 200.

[0017] Here, a foreground image is an image in which a subject area (foreground area) is extracted from an image captured by a camera. A subject extracted as a foreground area refers to a dynamic subject (moving object) that is moving (whose position or shape may change). For example, in a sport, the subject may include people such as players or referees who are on the field where the sport is being played, and when capturing images of a ball game, the subject may also include a ball in addition to people. Note that the subject is not limited to these, and when capturing images of a concert or entertainment, a singer, musician, performer, or presenter may be the foreground subject.

[0018] Here, the background image is an image of at least an area (background area) different from the foreground subject. Specifically, the background image is an image in which the foreground subject has been removed from the captured image. Note that the background refers to an imaged object that remains stationary or nearly stationary (for example, for a predetermined period of time) when captured from the same direction.

[0019] Such imaging objects include, for example, a stage for a concert, a stadium where an event such as a sport is held, a structure such as a goal used in a ball game, or a field. However, the background is at least an area different from the subject, which is the foreground. Note that the imaging objects may include other objects in addition to the subject and background.

[0020] <Explanation of the hardware configuration of the image processing device 100> The hardware configuration of the image processing device 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the image processing device. The image processing device 100 has a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a communication I / F 115, and a bus 116.

[0021] 1 by controlling the entire image processing device 100 using computer programs or data stored in the ROM 112 or RAM 113. Note that the image processing device 100 may have one or more dedicated hardware components different from the CPU 111, and at least a part of the processing by the CPU 111 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an FPGA (field programmable gate array) and a DSP (digital signal processor).

[0022] The ROM 112 stores programs that do not require modification, etc. The RAM 113 temporarily stores programs or data supplied from the auxiliary storage device 114, or data supplied from the outside via the communication I / F 115, etc.

[0023] The auxiliary storage device 114 is formed of, for example, a hard disk drive, and stores various data such as image data or audio data. The communication I / F 115 is used for communication with devices external to the image processing device 100. For example, when the image processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 115. When the image processing device 100 has a function for wireless communication with an external device, the communication I / F 115 is equipped with an antenna. The bus 116 connects each unit of the image processing device 100 to transmit information.

[0024] <Explanation of the configuration of the image processing device> The configuration of an image processing device 100 according to the first embodiment will be described with reference to Fig. 1. The image processing device 100 includes an acquisition unit 101, a development unit 102, a foreground separation unit 103, and a first setting unit 104.

[0025] The acquisition unit 101 acquires an image (input image) from the imaging device 1. In the following, it is assumed that the input image acquired here is a RAW image (Bayer image). The development unit 102 converts the Bayer image into an RGB image using an interpolation filter that uses an interpolation filter coefficient set by a first setting unit (to be described later).

[0026] The foreground separation unit 103 separates a background image (background region) from a foreground region corresponding to a predetermined subject for each input image. The foreground separation unit 103 can separate the foreground region from the image developed by the development unit 102. The foreground separation unit 103 can use any method commonly used for background / foreground extraction to separate the background image and the foreground region from an image. For example, the background image may be generated using a sequential background update method. Specifically, the foreground separation unit 103 determines, from multiple input images, regions with fluctuations as foreground and regions that have not changed for a certain period of time as background, and generates a background image from which only the background is extracted. Here, the foreground region is generated as a binary image (foreground mask image) using a background subtraction method using the input image and the generated background image. Specifically, the foreground region is generated by binarizing the difference image, obtained by subtracting the background image from the input image, using a predetermined threshold. In the foreground mask image according to this embodiment, "1" indicates the foreground region and "0" indicates the background region. The relationship between the binary value and the area indicated by the value may be reversed.

[0027] The methods for generating the background image and foreground mask image are not limited to those described above. For example, a pre-captured image without a foreground region may be stored in a frame memory and read out for use as the background image. Alternatively, a binary image in which the foreground and background regions are distinguished by AI without using background subtraction may be used as the foreground mask image. Furthermore, the foreground separation unit 103 calculates the circumscribing rectangle coordinates of the foreground region within the foreground mask image. In this embodiment, these coordinates are used to reduce the transmission bandwidth by limiting the foreground image (foreground mask image and texture image) transmitted from the foreground separation unit 103 to the image of the region within the circumscribing rectangle.

[0028] First setting unit 104 calculates and sets interpolation filter coefficients for development unit 102 based on parameters related to the foreground image used in generating the virtual viewpoint image, which are determined in this image processing system. In this embodiment, these parameters are a mask contribution rate and a texture contribution rate, which indicate the degree of contribution of the foreground mask image and the texture image to the generation of the virtual viewpoint image, and are stored by control device 300 in association with the intended use of the foreground image. Hereinafter, the parameters related to the foreground image used in generating the virtual viewpoint image may be referred to as "foreground contribution rate." Details of the processing performed by first setting unit 104 will be described later using FIG. 3.

[0029] <Method for controlling the coefficients of the development unit interpolation filter according to the purpose of use of the foreground image> In the image processing system of this embodiment, the development characteristics required by the development unit 102 differ for each image processing device 100 depending on the purpose of use (shooting requirements) of the foreground image in generating a virtual viewpoint, the configuration of the image processing system, etc. From this perspective, the first setting unit 104 according to this embodiment sets the development characteristics based on the foreground contribution degree as described above, which is associated with the shooting requirements. Furthermore, the first setting unit 104 may further set the development characteristics based on the configuration of the image processing system.

[0030] Note that the "development characteristics" according to this embodiment are characteristics of a process for developing a RAW image, and will be described here as parameters used in developing the RAW image. The first setting unit 104 according to this embodiment sets interpolation filter coefficients as development characteristics based on the foreground contribution rate. The first setting unit 104 can select, for example, whether to use equation (2) or equation (3), which will be described later, based on the foreground contribution rate, and calculate the interpolation filter coefficients using the selected equation. A specific example of the process for setting the interpolation filter coefficients will be described below.

[0031] As described above, the imaging requirement in this embodiment refers to the intended use of the foreground image. For example, if the imaging requirement is not to generate a virtual viewpoint image but to extract the subject's movement as data (e.g., to perform motion capture), the image generation device 200 generates only a foreground model representing the subject's three-dimensional shape. In this case, the foreground image used by the image generation device 200 is only a foreground mask image (no texture image is used). To generate a high-quality foreground model free of defects and false shapes, the image processing device 100 is required to generate a foreground mask image with minimal chips, holes, and noise at the mask boundary (noise that causes the mask boundary to fluctuate from frame to frame when the mask is monitored over time). Therefore, in this case, the development unit 102 is required to have development characteristics that strongly suppress noise. In this embodiment, the control device 300 stores the texture contribution CT and mask contribution CM (described later) as foreground contributions in association with the intended use of the foreground image, and the image processing device 100 acquires the foreground contributions.

[0032] For example, in a configuration of this image processing system in which the imaging devices are configured with a normal camera and a stereo camera, the image generation device 200 generates a virtual viewpoint image by selectively using foreground images received from the image processing device 100 depending on the type of camera used for imaging. When using images captured by the stereo camera, the image generation device 200 uses only the texture image of the image processing device 100 as data for generating the virtual viewpoint image. At this time, the image generation device 200 calculates parallax information between the two captured images from feature points of the images to obtain distance information, so the development unit 102 is required to have development characteristics that faithfully reproduce or emphasize the sharpness of the texture. When using images captured by a normal camera, the image generation device 200 uses the texture image and mask image of the image processing device 100 as data for generating the virtual viewpoint image. At this time, the development unit 102 is required to have development characteristics that achieve a good balance between noise suppression and reproduction of the sharpness of the texture.

[0033] As described above, the development characteristics required by the development unit 102 differ depending on the shooting requirements or the configuration of the image processing system. Therefore, the image processing device 100 according to the present embodiment optimizes the development characteristics required by the image processing system depending on the foreground image. The optimization here refers to the setting of interpolation filter coefficients used by the development unit 102 by the first setting unit 104. The calculation process of the interpolation filter coefficients by the first setting unit 104 based on the foreground contribution (and the configuration of the image processing system) will be described below with reference to FIGS. 3 and 4.

[0034] Fig. 4 is a flowchart for explaining an example of processing by the image processing device 100 according to the first embodiment. The CPU 111, which serves as a computer in the image processing device 100, executes a computer program stored in a memory such as the ROM 112 or the auxiliary storage device 114, thereby performing the operation of each step in the flowchart in Fig. 4. The details of this embodiment will be described with reference to this flowchart. The image processing device 100 starts the processing shown in Fig. 4 when the control device 300 receives an operation from the user to start this image processing.

[0035] In S101, the acquisition unit 101 acquires an undeveloped input image from the image capture device 1. In S102, the first setting unit 104 acquires a mask contribution rate and a texture contribution rate from the control device 300. The mask contribution rate according to this embodiment is

[0036] In S103, the first setting unit 104 calculates a control value α for determining an interpolation filter coefficient of the development unit using the mask contribution rate and the texture contribution rate. For example, the first setting unit 104 can calculate the control value α using the following equation (1). α=(CT×0.5)+(CM×(-0.5))+β...Equation (1)

[0037] Here, CT indicates the texture contribution and CM indicates the mask contribution, and real numbers are set as their respective values. β is an offset adjustment value for the control value α. In this embodiment, β is set to 1.0 to control the control value α to a value greater than or equal to 0. The smaller the control value α, the more the high frequencies of the frequency characteristics of the interpolation filter are attenuated and the low frequencies are passed or amplified, and the larger the value, the more the high frequencies of the same characteristics are amplified. Below are examples of how the control value α is calculated according to each contribution.

[0038] For example, if the shooting requirement is to extract the movement of a subject as data, it is conceivable that the image generation device 200 uses only a foreground mask image. Therefore, in this embodiment, such a shooting requirement is associated with a texture contribution of 0.0 and a mask contribution of 1.0, and the texture contribution and mask contribution for the entire image processing device 100 are set to 0.0 and 1.0, respectively. The control value α in this case is 0.5.

[0039] For example, when the shooting requirement is the generation of a virtual viewpoint image and the imaging device 1 connected to the image processing device 100 is a normal camera (not a stereo camera), the video generation device 200 generates a foreground mask image and a texture image. Therefore, the texture contribution rate and mask contribution rate for the image processing device 100 connected to the imaging device 1 are set to 1.0 and 1.0, respectively. The control value α at this time is 1.0.

[0040] For example, if the imaging requirement is to generate a virtual viewpoint image and the imaging device 1 connected to the image processing device 100 is a stereo camera, the video generation device 200 uses only the texture image. Therefore, the texture contribution rate for the image processing device 100 connected to the imaging device 1 is set to 1.0, and the mask contribution rate is set to 0.0. In this case, the control value α is 1.5. Note that the calculation formula for the control value α is not limited to formula (1) and may be any formula that derives a variable indicating the development characteristics (value of the interpolation filter coefficient) from the mask contribution rate and texture contribution rate input to the first setting unit 104. Also, although the texture contribution rate and mask contribution rate are shown here as a combination of two values, 0.0 and 1.0, other values ​​may also be used. For example, in the first calculation example of the control value α, if further noise suppression in the foreground mask image is required, the mask contribution rate may be set to 2.0 to further attenuate the high frequencies (in this case, the control value α is 0.0). Naturally, it is also possible to fine-tune the degree of reduction in the high frequencies by setting a decimal value for the mask contribution rate.

[0041] In S104, the first setting unit 104 determines whether the colors referenced in the interpolation process in the development unit should be the same color as the interpolated pixel or should include other colors, based on the magnitude relationship between the mask contribution and the texture contribution. Including a color different from the interpolated pixel in the reference pixel has the advantage of improving texture sharpness. This is because the wavelength ranges of RGB partially overlap, so when the image sensor separates light into RGB using a Bayer array, the information contained in the interpolated pixel is also included in other colors. However, there is a disadvantage in that if noise is contained in a color different from the interpolated pixel, this noise propagates to the interpolated pixel, thereby enhancing the noise in the developed image. Taking these advantages and disadvantages into consideration, if the mask contribution is greater than or equal to the texture contribution, processing proceeds to S105, where noise suppression is prioritized and the reference pixel is made the same color as the interpolated pixel. If the texture contribution is greater than the mask contribution, processing proceeds to S106, where a color different from the interpolated pixel is included and improved sharpness is prioritized. In this embodiment, if α is 1.0 or less, the process of S105 is performed, and if α is greater than 1.0, the process of S106 is performed. Note that here, if the mask contribution rate and the texture contribution rate are the same value, the process proceeds to S105, but the process may proceed to S106.

[0042] In S105, the first setting unit 104 sets the reference pixel to the same color as the pixel to be interpolated and calculates the coefficient of the interpolation filter according to the control value α. The calculation formula for the interpolation filter coefficient is shown in Formula (2). Hereinafter, the interpolation filter coefficient calculated using Formula (2) and the development process of the RAW image using the interpolation filter coefficient may be referred to as the "interpolation filter coefficient that suppresses noise" or the "development process that suppresses noise." F=L(α)+Hs×(α-0.5)...Equation (2)

[0043] Here, F is the interpolation filter coefficient, L(α) is the coefficient of the smoothing filter according to the control value α, and Hs is the high-frequency emphasis filter coefficient using pixels of the same color as reference pixels. L(α) is calculated so that the smaller the value of α, the lower the cutoff frequency (i.e., the stronger the noise suppression effect) becomes the smoothing filter coefficient. When the sign of (α-0.5) is positive, Hs is used as the coefficient to adjust the edge strength, and when the sign is negative, it is adjusted with the coefficient set to 0. As shown in this formula, the smaller the control value α, the stronger the noise suppression of the interpolation filter coefficient F becomes, and the larger the control value α, the stronger the coefficient of emphasis on sharpness.

[0044] FIG. 3 shows interpolation filter coefficients corresponding to Bayer array data and a control value α. Colors 301 to 303 in FIG. 3 indicate the colors of the data in the Bayer array, with color 301 corresponding to R, color 302 corresponding to G, and color 303 corresponding to B. FIG. 3(a) is a diagram illustrating an example of the interpolated pixel, the reference pixel of the interpolation filter, and the interpolation filter coefficients when the control value α is 0. Position 304 is the position of the interpolated pixel in the Bayer array, and the color to be interpolated is G. When interpolating the pixel value of G at position 305, the reference pixel is set to the same color, G, and a filter coefficient that strongly suppresses noise is set using Equation (2). FIG. 3(b) is a diagram illustrating an example when the control value α is 0.5. When interpolating the pixel value of G at position 305 in the array data shown in FIG. 3(b), the reference pixel is set to the same color, G, and a filter coefficient that slightly suppresses noise is set using Equation (2). FIG. 3(c) is a diagram illustrating an example when the control value α is 1.0. When interpolating the pixel value of G at position 306 in the array data shown in Figure 3(c), a filter coefficient that enhances sharpness is set using a G pixel of the same color as the reference pixel. Note that the method for calculating the interpolation filter coefficient is not limited to equation (2), as long as a desired filter coefficient is calculated according to the control value α. For example, the filter coefficient corresponding to the control value α may be stored in memory in advance, and the interpolation filter may be calculated by reading it from the memory according to α.

[0045] In S106, the first setting unit 104 calculates coefficients of an interpolation filter by including a color different from that of the interpolated pixel in the reference pixel according to the control value α. The calculation formula for the interpolation filter coefficient is shown in Formula (3). Hereinafter, the interpolation filter coefficient calculated using Formula (3) and the development process of a RAW image using the interpolation filter coefficient may be referred to as "interpolation filter coefficients that enhance sharpness" or "development process that enhances sharpness." F=L(α)+Hd×(α-0.5)...Equation (3)

[0046] Here, Hd denotes a high-frequency emphasis filter coefficient that includes a color different from the pixel to be interpolated in the reference pixels. Figure 3(c) shows the reference pixels and interpolation filter coefficients of the interpolation filter when the control value α is 1.5 and G at the position of R shown in position 307 is the interpolated pixel. When interpolating G at position 307, R is included in the reference pixels and a filter coefficient value that enhances sharpness is used.

[0047] In this embodiment, since including different colors in the reference pixels leads to improved sharpness, different colors are included in the reference pixels when the control value α is a value that sufficiently enhances sharpness (α>1.0). In other words, when the control value α is greater than 1.0, low-frequency emphasis in the texture is sufficiently small and high-frequency emphasis is sufficiently large. On the other hand, when α≦1.0, the processing flow prioritizes noise suppression and references only the same color. However, as long as development characteristics can be set according to the purpose of use of the foreground region, the processing is not limited to this type of processing; for example, the processing may branch when α is a value other than 1.0.

[0048] For example, in S104, the first setting unit 104 may switch whether to refer to only reference pixels of the same color as the interpolated pixel depending on whether the control value α enhances texture sharpness. That is, in this embodiment, whether to refer to only reference pixels of the same color as the interpolated pixel may be switched depending on whether the control value α is greater than 0.5. In this case, if the control value α is 0.5 or less, the process of S105 is performed, and if the control value α is greater than 0.5, the process of S106 is performed. This is for the following reason. When the control value α is 0.5, if the control value α is greater than 0.5 (α>0.5), the frequency characteristics of the interpolation filter coefficient F include a high-frequency emphasis component. Furthermore, as the control value α increases above 0.5, the high-frequency emphasis component in the frequency characteristics of the interpolation filter coefficient F increases. Therefore, in the development process of a RAW image using this interpolation filter coefficient F, the texture sharpness is enhanced when α is greater than 0.5.

[0049] In this example, G at the position of R is used as the interpolated pixel, but even if the target of the interpolated pixel is a different color, the interpolation filter coefficients are calculated in the same way according to the mask contribution, texture contribution, control value, and α.

[0050] In the process of S105, the first setting unit 104 sets the reference pixel to the same color as the interpolated pixel, but it is not necessary to set all reference pixels to the same color as the interpolated pixel. That is, for example, in the interpolation filter coefficients shown in Figures 3(a) and 3(b), it is sufficient that the number and proportion of colors of the reference pixel that differ from those of the interpolated pixel are smaller than those of the interpolated pixel in the interpolation filter coefficients shown in Figure 3(c). Also, the smaller α is, that is, for example, in the interpolation filter coefficients shown in Figures 3(a) and 3(b), it is sufficient that the number and proportion of colors of the reference pixel that differ from those of the interpolated pixel are smaller in the interpolation filter coefficients shown in Figure 3(a).

[0051] In S107, development unit 102 develops the input image using the interpolation filter coefficients set by first setting unit 104. In S108, foreground separation unit 103 separates the background image from the foreground area corresponding to a predetermined subject in the developed input image. In S109, foreground separation unit 103 outputs the foreground image (foreground mask image and texture image) and the background image to video generation device 200.

[0052] In this embodiment, the mask contribution degree and the texture contribution degree are associated with the shooting requirements (and the configuration of this image processing system) and are registered in advance in the control device 300. By controlling each image processing device using the contribution degrees associated in this way, even if there is a change in the shooting requirements or the configuration of this image processing system, it becomes possible to easily change the mask contribution degree and texture used in each image processing device.

[0053] In the present embodiment, the first setting unit 104 is described as calculating and setting the development parameters of the development unit 102, but some or all of the functions of the first setting unit 104 may be executed by the control device 300. Similarly, in the present embodiment, the imaging device 1, the image processing device 100, and the video generation device 200 are described as being different devices, but as long as similar processing can be executed, some or all of the devices may be located within the same device.

[0054] In this embodiment, the development unit 102 performs filter processing on the undeveloped input image according to the intended use of the foreground image. This processing makes it possible to perform development processing that further suppresses noise if the undeveloped image is expected to contain a lot of noise, and to perform development processing that further enhances sharpness if the undeveloped image is expected to require sharpness. On the other hand, with regard to the filter processing according to the intended use of the foreground image, processing equivalent to the filter processing may be performed on the developed image in a processing stage subsequent to the development processing.

[0055] With this configuration, it is possible to set development characteristics for each image processing device based on the foreground contribution associated with the intended use of the foreground region (foreground region image) of each image used when generating a virtual viewpoint image. Therefore, by performing appropriate development processing for each image according to the intended use, it is possible to suppress deterioration in image quality.

[0056] [Embodiment 2] The image processing device according to the first embodiment sets development characteristics according to the intended use of the foreground region image. However, for example, if the imaging device is a stereo camera and an interpolation filter coefficient that enhances the sharpness of the texture is selected, noise (caused by the ISO sensitivity of the imaging device) occurring in low-brightness regions of the texture may be emphasized. In this case, distance information may not be calculated correctly, and the image quality of the virtual viewpoint image may be degraded.

[0057] Therefore, the image processing device 100 according to this embodiment adjusts the interpolation filter coefficients so as to weaken the sharpness in low-brightness areas, thereby suppressing deterioration in the image quality of the virtual viewpoint image. In particular, when the first setting unit 104 selects an interpolation filter coefficient that enhances the sharpness, performing such an adjustment can suppress excessive enhancement of noise in low-brightness areas.

[0058] <Explanation of the configuration of the image processing device> Fig. 5 is a block diagram showing an example of an image processing system for generating a virtual viewpoint image including an image processing device according to this embodiment. The image processing system shown in Fig. 5 has the same configuration as that shown in Fig. 1 of the first embodiment, except that the image processing device 100 has a second setting unit 105, and can execute the same processing, so redundant explanations will be omitted.

[0059] The second setting unit 105 adjusts the interpolation filter coefficients set by the first setting unit 104 for each pixel of the input image based on the pixel values ​​of the input image before development. The second setting unit 105 may also determine whether or not to adjust the interpolation filter coefficients based on the coordinate values ​​of the circumscribing rectangle of the foreground region generated by the foreground separation unit 103, and perform the adjustment described above only on pixels whose coordinates it has determined should be adjusted. Hereinafter, the adjustment process of the interpolation filter coefficients performed by the second setting unit 105 according to this embodiment will be described with reference to FIGS. 6 and 7.

[0060] <How to adjust the interpolation filter coefficients> Fig. 7 is a flowchart for explaining an example of processing by the image processing device 100 according to this embodiment. The processing shown in Fig. 7 is the same as that shown in Fig. 4 except that S201 to S204 are executed instead of S107, and therefore a duplicated description will be omitted. Note that, in the following description, the following will be given assuming, as an example, that an interpolation filter coefficient that enhances sharpness is set, but similar processing can also be executed when an interpolation filter coefficient that suppresses noise is set.

[0061] In S201, the second setting unit 105 calculates a normalized luminance value for each pixel using the pixels of the undeveloped image acquired by the acquisition unit 101. Specifically, the second setting unit 105 first performs temporary development on the undeveloped image using a simple method (for example, nearest neighbor interpolation), and then performs color space conversion on the temporarily developed image to calculate the luminance value of each pixel. The second setting unit 105 then calculates the ratio of each luminance value to the upper luminance limit as the normalized luminance value. An example of this calculation formula is shown in Formula (4). γ(x,y)=LM(x,y) / LMmax...Equation (4)

[0062] Here, γ(x, y) is the normalized luminance value, LM(x, y) is the luminance value of each pixel, and LMmax is the maximum luminance value. For example, if the quantization bit rate of the pixel value is 10 bits, the maximum luminance value is 1023.

[0063] In S202, the second setting unit 105 acquires the circumscribing rectangle coordinates of the foreground region from the foreground separation unit 103, and calculates an evaluation value of the degree of proximity of each pixel position to the bottom edge of the circumscribing rectangle. The rectangle drawn with a dotted line shown as 602 in FIG. 6(a) indicates the circumscribing rectangle of the subject 601. The circumscribing rectangle coordinates are expressed by the coordinates of the upper left corner and the lower right corner of the circumscribing rectangle, and these coordinates are shown as coordinates 603 and 604 in FIG. 6(a). The second setting unit 105 uses these coordinates to calculate the degree of proximity of each pixel within the circumscribing rectangle to the bottom edge of the circumscribing rectangle. An example of this calculation formula is shown in formula (5). θ(x,y)=(y-Ytop) / (Ybottom-Ytop)...Equation (5)

[0064] Here, θ(x, y) is an evaluation value of the degree of proximity of each pixel to the bottom edge of the circumscribing rectangle, and the closer it is to 1, the closer the y coordinate of the pixel of interest is to the bottom edge. Also, Ybottom is the Y coordinate of the bottom edge of the circumscribing rectangle, Ytop is the Y coordinate of the top edge of the circumscribing rectangle, and y is the Y coordinate of each pixel. For pixels inside the circumscribing rectangle, the closer the pixel is to the bottom edge of the circumscribing rectangle, the closer the θ(x, y) value is to 1. For pixels outside the circumscribing rectangle, the value of θ(x, y) is set to 0.

[0065] In S203, the second setting unit 105 adjusts the interpolation filter coefficients calculated by the first setting unit 104 using the normalized luminance value γ(x, y) and the degree of proximity θ(x, y) to the bottom edge of the circumscribed rectangle. In the following description, the interpolation filter coefficients to be adjusted are assumed to be the coefficients shown in FIG. 3C used in the first embodiment. Reference numeral 605 in FIG. 6A indicates the lower half of the subject's body, whose brightness has decreased. When lighting is installed on the ceiling at the imaging location, the lower half of the subject's body is likely to be subject to reduced illumination due to its greater distance from the lighting, and is also likely to be cast in the shadow of the upper half of the body. Therefore, the texture of the subject's lower half is likely to be subject to reduced brightness and noise. From this perspective, the second setting unit 105 according to this embodiment adjusts the degree of sharpening for pixels determined to have reduced brightness, based on the normalized luminance value γ(x, y).

[0066] FIG. 6(b) shows the interpolation filter coefficients adjusted by the second setting unit 105. Region 607 in FIG. 6(a) represents the subject's head, including the subject's hair. If the above-described adjustment of the interpolation filter coefficients is performed uniformly across all pixels, the degree of sharpening may be weakened in areas where texture sharpness is required, such as black hair. To avoid weakening the degree of sharpening in areas with low brightness, such as the upper body or head, the range of the interpolation filter coefficients to be adjusted is limited using the evaluation value θ(x, y) of the degree of proximity to the bottom edge of the circumscribing rectangle. In other words, the second setting unit 105 adjusts the interpolation filter coefficients described above only for pixels whose distance from the bottom edge of the circumscribing rectangle is equal to or less than a predetermined threshold. FIG. 6(c) shows the interpolation filter coefficients for the head region, maintaining the degree of sharpening. The above-described method of adjusting the interpolation filter coefficients is shown in Equation (6).

number

[0067] In S204, the developing unit 102 develops the input image using the interpolation filter coefficients adjusted by the second setting unit 105.

[0068] This process allows the interpolation filter coefficients to be further adjusted based on pixel values ​​when performing development processing. Therefore, by adjusting the development characteristics for each pixel, it is possible to achieve both sharpening and noise suppression even in areas of low texture brightness, thereby preventing deterioration in the image quality of the virtual viewpoint image.

[0069] The disclosure of this specification includes the following image processing device, information processing method, and program. (Item 1) Acquisition method for acquiring RAW images a setting means for setting characteristics of a process for developing the RAW image based on parameters relating to a foreground image extracted from the RAW image and used when generating a virtual viewpoint image; a developing means for developing the RAW image based on the characteristics; An image processing device comprising: (Item 2) Item 1. The image processing device according to item 1, characterized in that the parameters include a first parameter indicating the contribution of a mask image indicating a region of the foreground image in the RAW image in generating the virtual viewpoint image, and a second parameter indicating the contribution of a texture image indicating color information of the foreground image. (Item 3) the characteristics of the process for developing the RAW image are interpolation filter coefficients used when developing the RAW image, Item 2. The image processing device according to item 2, characterized in that the setting means sets the interpolation filter coefficients so that the larger the first parameter is, the more high frequencies are attenuated in the development of the RAW image, and the larger the second parameter is, the more high frequencies are emphasized in the development of the RAW image. (Item 4) Item 2 or 3, the image processing device, characterized in that the setting means sets the characteristics so that, when the first parameter is equal to or greater than the second parameter, reference pixels for interpolated pixels in the interpolation filter of the RAW image are pixels of the same color as the interpolated pixels, and when the first parameter is less than the second parameter, sets the characteristics so that reference pixels for interpolated pixels in the interpolation filter of the RAW image include pixels of a different color from the interpolated pixels. (Item 5) 5. The image processing device according to any one of items 2 to 4, wherein the first parameter and the second parameter are associated with a purpose of use in generating the virtual viewpoint image of the foreground image. (Item 6) 6. The image processing device according to any one of items 1 to 5, further comprising an adjustment unit that adjusts the characteristics set by the setting unit based on pixel values ​​of the RAW image. (Item 7) 7. The image processing device according to item 6, wherein the adjustment means adjusts the characteristics set by the setting means based on the luminance of the RAW image. (Item 8) the characteristics of the process for developing the RAW image are interpolation filter coefficients used when developing the RAW image, 8. The image processing device according to item 7, wherein the adjustment means adjusts an interpolation filter coefficient for a pixel of interest in the RAW image using a ratio of the luminance of the pixel of interest to the maximum luminance in the RAW image. (Item 9) the characteristics of the process for developing the RAW image are interpolation filter coefficients used when developing the RAW image, The image processing device according to any one of items 6 to 8, wherein the adjustment means determines, for each pixel in the foreground image, based on pixel coordinates, whether or not to adjust the interpolation filter coefficients, and adjusts the interpolation filter coefficients for pixels in the foreground image for which it has been determined that characteristics should be adjusted. (Item 10) Item 10. The image processing device according to item 9, characterized in that the adjustment means adjusts the interpolation filter coefficient for a pixel of interest within a circumscribing rectangle of the foreground image area when a difference between the y coordinate of the pixel of interest and the y coordinate of the bottom edge of the circumscribing rectangle is equal to or less than a predetermined threshold. (Item 11) 11. The image processing device according to any one of items 1 to 10, further comprising a separation unit that separates a foreground region from the developed RAW image. (Item 12) A process of acquiring a RAW image setting characteristics of a process for developing the RAW image based on parameters related to a foreground image extracted from the RAW image and used when generating a virtual viewpoint image; developing the RAW image based on the characteristics; An information processing method comprising: (Item 13) A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 11.

[0070] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0071] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0072] 101: Acquisition section, 102: Development section, 103: Foreground separation section, 104: First setting section

Claims

1. an acquisition means for acquiring a RAW image; a setting means for setting characteristics of a process for developing the RAW image based on parameters relating to a foreground image extracted from the RAW image, the parameters being used when generating a virtual viewpoint image; a developing means for developing the RAW image based on the characteristics; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the parameters include a first parameter indicating a contribution of a mask image indicating a region of the foreground image in the RAW image to generating the virtual viewpoint image, and a second parameter indicating a contribution of a texture image indicating color information of the foreground image.

3. the characteristics of the process for developing the RAW image are interpolation filter coefficients used when developing the RAW image; 3. The image processing device according to claim 2, wherein the setting means sets the interpolation filter coefficients so that the larger the first parameter is, the more high frequencies are attenuated in the development of the RAW image, and the larger the second parameter is, the more high frequencies are emphasized in the development of the RAW image.

4. 3. The image processing device according to claim 2, wherein the setting means sets the characteristics so that, when the first parameter is equal to or greater than the second parameter, reference pixels for an interpolated pixel in the interpolation filter of the RAW image are pixels of the same color as the interpolated pixel, and when the first parameter is less than the second parameter, sets the characteristics so that reference pixels for an interpolated pixel in the interpolation filter of the RAW image include pixels of a different color from the interpolated pixel.

5. The image processing device according to claim 2 , wherein the first parameter and the second parameter are associated with a purpose of use in generating the virtual viewpoint image of the foreground image.

6. 2. The image processing apparatus according to claim 1, further comprising an adjusting unit that adjusts the characteristics set by said setting unit based on pixel values ​​of said RAW image.

7. 7. The image processing apparatus according to claim 6, wherein the adjusting means adjusts the characteristics set by the setting means based on the luminance of the RAW image.

8. the characteristics of the process for developing the RAW image are interpolation filter coefficients used when developing the RAW image; 8. The image processing device according to claim 7, wherein said adjustment means adjusts the interpolation filter coefficient for the pixel of interest in said raw image using a ratio of the luminance of said pixel of interest to a maximum luminance in said raw image.

9. the characteristics of the process for developing the RAW image are interpolation filter coefficients used when developing the RAW image; 7. The image processing device according to claim 6, wherein the adjustment means determines, for each pixel in the foreground image, based on pixel coordinates, whether or not to adjust the interpolation filter coefficients, and adjusts the interpolation filter coefficients for pixels in the foreground image for which it has been determined that the characteristics of the pixels should be adjusted.

10. 10. The image processing device according to claim 9, wherein the adjustment means adjusts the interpolation filter coefficient for a pixel of interest within a circumscribing rectangle of the foreground image area when a difference between a y coordinate of the pixel of interest and a y coordinate of a lower end of the circumscribing rectangle is equal to or smaller than a predetermined threshold value.

11. The image processing apparatus according to claim 1 , further comprising a separation unit for separating a foreground region from the developed RAW image.

12. acquiring a RAW image; setting characteristics of a process for developing the RAW image based on parameters related to a foreground image extracted from the RAW image, the parameters being used when generating a virtual viewpoint image; developing the RAW image based on the characteristics; An information processing method comprising:

13. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images