IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, INFORMATION PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

JP7594413B2Active Publication Date: 2024-12-04CANON KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020186748
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-11-09
Publication Date
2024-12-04
Estimated Expiration
2040-11-09

AI Technical Summary

Benefits of technology

【0012】 本発明によれば、法線エラー領域に起因する仮想光源画像の画質劣化を抑制することが可能となる。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594413000004
    Figure 0007594413000004
  • Figure 0007594413000005
    Figure 0007594413000005
  • Figure 0007594413000006
    Figure 0007594413000006
Patent Text Reader

Abstract

To provide an image processing device, an imaging device, an information processing device, an image processing method, and a program, which suppress image quality deterioration of a virtual light source image due to a normal line error region.SOLUTION: An image processing device includes: acquisition means that acquires light source information specifying a virtual light source, normal line information indicating a normal line distribution in a shooting range, and normal line error information indicating a normal line error region in which an error has occurred in the normal line distribution; first generation means that generates first gain information indicating a first gain amount distribution on the basis of the light source information and the normal line information; second generation means that generates second gain information indicating a second gain amount distribution by performing smoothing processing on a gain amount of a region corresponding to the normal line error region in the first gain amount distribution; and addition means that adds an effect of the virtual light source to an image of the shooting range by applying a gain conforming to the second gain amount distribution to the image.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an information processing device, an image processing method, and a program. [Background technology]

[0002] There is a known method of acquiring surface normal information (hereinafter referred to as "normal information") as shape information of a subject from an image captured by an imaging device such as a digital camera. By using this normal information during rendering processing, it is possible to acquire an image such as a relighting image in which the appearance of the subject (in the case of relighting, the way the shadows are applied or the way the light hits it) has been changed.

[0003] Photometric stereo has been proposed as one method for acquiring such normal information. Photometric stereo is a method in which the reflectance characteristics of an object based on the light source direction and the object's surface normal are assumed, and the surface normal is determined from the object's luminance information and reflectance characteristics at multiple light source positions. The Lambertian reflection model, which follows Lambert's cosine law, is often used to determine the reflectance characteristics of an object in photometric stereo.

[0004] Generally, there are two types of reflection from objects: specular reflection and diffuse reflection. Specular reflection is regular reflection on the surface of an object, and is reflection that follows the Fresnel law at the object surface (interface). On the other hand, diffuse reflection is reflection in which light is scattered inside the object after passing through the surface of the subject and returns.

[0005] Specularly reflected light cannot be expressed by Lambert's cosine law, so if the light reflected from a subject observed by an imaging device contains specularly reflected light, the photometric stereo method cannot accurately determine the surface normal.

[0006] In addition, even in shaded areas where light from the light source does not hit, errors occur from the assumed reflection model based on Lambert's cosine law, making it impossible to accurately obtain normal information of the subject. Furthermore, for subjects having a surface shape that causes interreflection, errors occur from the assumed reflection model based on Lambert's cosine law, making it impossible to accurately obtain normal information of the subject. Hereinafter, the above-mentioned areas where the normal information of the subject cannot be obtained accurately are called normal error areas.

[0007] Patent Document 1 describes that when normal information is obtained from luminance information by photometric stereo, noise contained in the luminance information can be reduced and errors in the normal information caused by noise can be eliminated. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2017-150878 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, although the technique of Patent Document 1 can suppress errors in normal information caused by noise contained in luminance information, it cannot suppress errors in normal information caused by the above-mentioned specular reflected light, shadow areas, and interreflection, and normal error areas occur. When a virtual light source image in which the appearance of the subject is changed is formed based on normal information containing such normal error areas, the state of shading may become unnatural in the normal error areas, for example, as shown in FIG. 10(a).

[0010] The present invention has been made in view of the above circumstances, and has an object to provide a technique for suppressing degradation in image quality of a virtual light source image caused by a normal error region. [Means for solving the problem]

[0011] In order to solve the above problem, the present invention provides an image capturing apparatus comprising: an acquisition means for acquiring light source information defining a virtual light source, normal information indicating a distribution of normals in a shooting range, and normal error information indicating a normal error region where an error occurs in the distribution of normals; a first generation means for generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generation means for generating second gain information indicating a second gain amount distribution by performing a smoothing process on a gain amount of a region corresponding to the normal error region in the first gain amount distribution; and an addition means for adding an effect of the virtual light source to an image in the shooting range by applying a gain according to the second gain amount distribution to the image. The second generating means determines a smoothing range for each position in the smoothing process based on a difference in gain amount at each position in a region corresponding to the normal error region in the first gain amount distribution, and the smoothing range determined for a second position where the difference in gain amount is a second value is wider than the smoothing range determined for a first position where the difference in gain amount is a first value smaller than the second value. The present invention provides an image processing device comprising: Effect of the Invention

[0012] According to the present invention, it is possible to suppress degradation of image quality of a virtual light source image caused by a normal error region.

[0013] Other features and advantages of the present invention will become apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is an external view of an imaging device 1. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an imaging device 1. [Diagram 3] 13 is a flowchart of a rendering process. [Figure 4A] 10 is a flowchart showing details of a gain information correction process. [Figure 4B] 10 is a flowchart showing details of a gain information correction process. [Diagram 5] 6A to 6C are diagrams for explaining a process a in the gain information correction process. [Figure 6] 11A to 11C are diagrams for explaining a process b (example 1) in the gain information correction process. [Figure 7]11A to 11C are diagrams for explaining a process c in the gain information correction process. [Figure 8] 6A and 6B are diagrams for explaining details of a method for determining an averaging processing range (smoothing range). [Figure 9] 4A and 4B are diagrams for explaining specular reflection components. [Figure 10] FIG. 1A is a diagram showing an example of a virtual light source image with unnatural shading, and FIG. 1B is a diagram showing an example of a virtual light source image with desirable shading. [Figure 11] 11A to 11C are diagrams for explaining a process b (example 2) in the gain information correction process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0016] [First embodiment] Photometric Stereo Method The photometric stereo method is a method of assuming the reflection characteristics of an object based on the surface normal of the object and the direction from the object to the light source (light source direction), and calculating surface normal information based on the luminance information of the object corresponding to a plurality of light source positions and the assumed reflection characteristics. When the reflectance is not uniquely determined when a certain surface normal and the position of the light source are given, the reflection characteristics may be approximated by a Lambertian reflection model that follows Lambert's cosine law. As shown in FIG. 9, the specular reflection component depends on the angle α between the bisector of the light source vector s and the line of sight vector v and the surface normal n. Therefore, the reflection characteristics may be characteristics based on the line of sight. In addition, as for the luminance information, the object may be imaged when the light source is turned on and when it is turned off, and the influence of light other than the light source (ambient light, etc.) may be eliminated by taking the difference between the images.

[0017] In the following, we will explain the case where the reflection characteristics are assumed to be based on the Lambertian reflection model. Let i be the luminance value of the reflected light, and ρ be the Lambertian diffuse reflectance of the object. d If the intensity of the incident light is E, the unit vector indicating the direction from the object to the light source (light source direction vector) is s, and the normal vector of the unit surface of the object is n, the luminance i is expressed by the following equation (1) according to Lambert's cosine law.

[0018]

number

[0019] Let each component of the different M (M ≥ 3) light source vectors be s1, s2, . . . , s M , the luminance values ​​for each component of the light source vector are i1, i2,...i M Then, equation (1) is expressed as the following equation (2).

[0020]

number

[0021] The left side of equation (2) is the M-row, 1-column luminance vector, and the right side is [s1 T , ,s M T ] is the incident light matrix S that indicates the light source direction with M rows and 3 columns, and n is the unit surface normal vector with 3 rows and 1 column. When M=3, the inverse matrix S of the incident light matrix S -1 Using Eρ d n is expressed by the following equation (3).

[0022]

number

[0023] The norm of the vector on the left side of equation (3) is the incident light intensity E and the Lambertian diffuse reflectance ρ d The normalized vector is calculated as the surface normal vector of the object. That is, the incident light intensity E and the Lambertian diffuse reflectance ρ d appears in the conditional expression only in the form of a product, so Eρ dConsidering as one variable, equation (3) can be considered as a simultaneous equation that determines three unknown variables together with the two degrees of freedom of the unit surface normal vector n. Therefore, each variable can be determined by acquiring luminance information using at least three light sources. Note that if the incident light matrix S is not a regular matrix, an inverse matrix does not exist, so it is necessary to select each of the components s1 to s3 of the incident light matrix S so that the incident light matrix S becomes a regular matrix. In other words, it is desirable to select the component s3 linearly independent of the components s1 and s2.

[0024] Furthermore, when M>3, more conditional equations are obtained than there are unknown variables to be found, so the unit surface normal vector n can be calculated from three arbitrarily selected conditional equations in the same manner as when M=3. When using four or more conditional equations, the incident light matrix S is no longer a regular matrix, so an approximate solution may be calculated using, for example, the Moore-Penrose pseudoinverse matrix. The unit surface normal vector n may also be calculated using a fitting method or an optimization method.

[0025] On the other hand, when normal information (surface normal information) is derived as described above, normal information cannot be obtained correctly for a subject with specular reflected light, a subject with a shadow, and a subject with a surface shape that causes interreflection. In a normal shooting scene, there are many cases where at least a part of the subject has an area (normal error area) from which normal information cannot be obtained correctly as described above. In this embodiment, by using a method described in detail below, it is possible to perform a rendering process well even for a subject scene having this normal error area.

[0026] Configuration of imaging device 1 The configuration of the imaging device 1 will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is an external view of the imaging device 1. Fig. 2 is a block diagram showing the configuration of the imaging device 1.

[0027] The imaging device 1 generates a rendering image (relighting image) by performing a rendering process. As shown in FIG. 1, the imaging device 1 has an imaging section 100 and a light source section 200 for imaging a subject. As shown in FIG. 2, the imaging section 100 has an imaging optical system 101 and an imaging element 102. The light source section 200 is configured to be able to irradiate the subject with light from a plurality of different positions. In this embodiment, the light source section 200 has eight light sources 200a to 200h, but is not limited thereto. At least three light sources are required to perform photometric stereo, so the light source section 200 only needs to have at least three light sources. In the example shown in FIG. 1, eight light sources are arranged at equal intervals in a concentric manner at positions equidistant from the optical axis of the imaging optical system 101 included in the imaging section 100, but is not limited thereto. In the example shown in FIG. 1, the light source section 200 is built into the imaging device 1, but is not limited thereto. The light source section 200 may be configured to be detachably attached to the imaging device 1.

[0028] The imaging optical system 101 includes an aperture 101a, and forms an image of light from a subject on the imaging element 102. The imaging element 102 is composed of photoelectric conversion elements such as a CCD sensor or a CMOS sensor, and captures an image of the subject. That is, the imaging element 102 photoelectrically converts an image (optical image) of the subject formed by the imaging optical system 101, and generates an analog electrical signal (image data corresponding to an input image). The A / D converter 103 converts the analog signal generated by the photoelectric conversion of the imaging element 102 into a digital signal, and outputs the digital signal to an image processing unit 104.

[0029] The image processing unit 104 (processing device) performs various image processing on the digital signal input from the A / D converter 103. In this embodiment, the image processing unit 104 calculates normal information of the subject and generates a rendering image (relighting image) under an arbitrary light source. The image processing unit 104 has an input image acquisition unit 104a, a normal information acquisition unit 104b, a boundary information acquisition unit 104c, and a rendering unit 104d.

[0030] The output image processed by the image processing unit 104 is stored in an image recording unit 109 such as a semiconductor memory or an optical disk. The image processing unit 104 may also output the output image to a display unit 105 (display). In this embodiment, the input image acquisition unit 104a, the normal information acquisition unit 104b, the boundary information acquisition unit 104c, and the rendering unit 104d are built into the imaging device 1, but the present invention is not limited to this, and some of the above-mentioned units may be provided separately from the imaging device 1.

[0031] The information input unit 108 supplies the system control unit 110 with the shooting conditions (aperture value, exposure time, ISO sensitivity, focal length, etc.) selected by the user. The imaging control unit 107 acquires an image under the shooting conditions selected by the user based on the information from the system control unit 110. The irradiation light source control unit 106 controls the light emission state of the light source unit 200 according to a control instruction from the system control unit 110. The information input unit 108 also has a function of supplying the light source conditions (virtual light source angle, virtual light source intensity, virtual light source color, etc.) selected by the user to the system control unit 110. The image processing unit 104 generates a rendering image (relighting image) under the virtual light source conditions selected by the user based on the information from the system control unit 110. In this embodiment, the imaging optical system 101 is configured integrally with the imaging device 1, but is not limited to this. Instead of the imaging device 1, it is also possible to use a camera system such as a single-lens reflex camera or a mirrorless camera including an imaging device body having an imaging element and an imaging optical system (interchangeable lens) detachable from the imaging device body.

[0032] Rendering process The rendering process of this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart of the rendering process. The rendering process of this embodiment is executed by the system control unit 110 and the image processing unit 104 in accordance with a processing program as a computer program. The processing program is stored, for example, in a computer-readable storage medium (such as an internal memory of the system control unit 110).

[0033] In S101, the system control unit 110 controls the imaging control unit 107 to capture an image of the subject at least three times. In each capture, the system control unit 110 controls the irradiation light source control unit 106 to turn on a light source at a different position of the light source unit 200. The input image acquisition unit 104a acquires at least three captured images generated by capturing images at least three times as input images.

[0034] When normal information is acquired by photometric stereo assuming a diffuse reflection model such as Lambertian reflection in S102 described later, a diffuse reflection image in which the specular reflection component has been removed from the captured image may be used as the input image. In order to acquire a diffuse reflection image in which the specular reflection component has been removed from the captured image, for example, a method using a dichromatic reflection model can be used. However, the method of removing the specular reflection component from the captured image is not limited to this, and various methods can be used.

[0035] In S102, the normal information acquisition unit 104b acquires normal information (information indicating the distribution of normals in the shooting range) and reflectance. In this embodiment, the normal information acquisition unit 104b acquires the normal information and reflectance of the subject using at least three input images acquired by the input image acquisition unit 104a in S101. The normal information and reflectance are calculated based on the change in luminance information according to the light source position using photometric stereo.

[0036] In S103, the image processing unit 104 acquires normal error information used to identify a normal error region (a region where an error occurs in the distribution of normals) based on the normal information acquired in S102. The normal error information is information for identifying a pixel where a normal error occurs among the pixels of the captured image, and is used for condition branching judgment in S204 of FIG. 4A described later. In this embodiment, the image processing unit 104 uses the normal information acquired in S102 to generate a rendering image under the same light source conditions as the captured image, and calculates the difference between the rendering image and the captured image. Then, the image processing unit 104 determines a pixel region having a difference larger than a predetermined threshold as a normal error region, and acquires normal error information.

[0037] In S104, the boundary information acquisition unit 104c acquires boundary information (information indicating the boundary positions between the subjects in the shooting range) of multiple subjects captured within the imaging angle of view. In this embodiment, the boundary information acquisition unit 104c measures the distance to the subject using an image plane phase difference method for autofocus (AF), and acquires boundary information by detecting a change in object distance for each pixel position (hereinafter referred to as "flatness") based on this measurement value. More specifically, the boundary information acquisition unit 104c calculates the object distance difference between adjacent pixels, determines the position where the positive and negative polarities of this distance difference are inverted as the boundary between multiple subjects, and uses the determination result as boundary information.

[0038] Note that the boundary information is not limited to the flatness described above. For example, the boundary information acquisition unit 104c may acquire the Lambertian diffuse reflectance ρ d The boundary information acquiring unit 104c may detect a change in color or brightness for each pixel position between a plurality of subjects and use the detection result as the boundary information. Alternatively, the boundary information acquiring unit 104c may use boundary information based on object detection information using deep learning or the like. Furthermore, the boundary information acquiring unit 104c may detect a change in color or brightness for each pixel position between a plurality of subjects and acquire the detection result as the boundary information.

[0039] In S105, the rendering unit 104d generates and acquires first gain information (first gain information indicating a first gain amount distribution) corresponding to the light source conditions (light source information defining a virtual light source) set in the information input unit 108 based on the normal information acquired in S102.

[0040] In S106, the rendering unit 104d generates second gain information by performing a process of correcting the first gain information (gain information correction process). The details of the gain information correction process will be described later. Here, the purpose of the gain information correction process will be briefly described. At least three input images used to acquire normal information in S102 include a specular reflection part, a shaded part, or a part where interreflection occurs due to the influence of the surface shape. Therefore, the first gain information is derived from normal information including a normal error area with an error in the normal information. If a rendering process is performed according to such first gain information, an unnatural shaded part may be formed in the normal error area, and a failed relighting image may be generated. Therefore, the rendering unit 104d performs a gain information correction process on the first gain information to newly generate second gain information (second gain information indicating a second gain amount distribution). By performing a rendering process using this second gain information, it is possible to generate a flawless relighting image without forming an unnatural shaded part.

[0041] In S107, the rendering unit 104d generates a rendering image according to the second gain information generated in S106. The rendering image (relighting image) is obtained by performing a predetermined gain process on the pixel values ​​of pixels to be shaded based on the light source conditions in a base image (image in the shooting range) such as an input image (applying a gain according to the second gain amount distribution). As the base image, an image based on at least three input images obtained in S101 can be used. In this embodiment, an image constructed by selecting a pixel value with the highest luminance value at each pixel position from at least three input images corresponding to light sources at different positions is used as the base image. Then, as the gain process for the rendering process of this embodiment, a gain process is performed to reduce the pixel values ​​in the base image in order to form a shaded portion based on the light source conditions. Note that the base image of this embodiment is not limited to an image constructed by the above-mentioned method, and may be an image with a different light source environment, such as an ambient light image obtained by shooting without turning on the light source.

[0042] ●Gain information correction processing The gain information correction process will be described in detail with reference to Figures 4A and 4B. In S201, the rendering unit 104d selects a processing target pixel (a pixel of interest).

[0043] In S202, the rendering unit 104d calculates the gain difference of the processing target pixel with respect to the adjacent pixel based on the first gain information, thereby acquiring gain difference information of the processing target pixel.

[0044] In S203, the rendering unit 104d determines the averaging processing range of the processing target pixel based on the gain difference information acquired in S202. The method of determining the averaging processing range will be described later in detail.

[0045] In S204, the rendering unit 104d determines whether the pixel to be processed is a pixel in the normal error region based on the normal error information acquired in S103. If the pixel to be processed is a pixel in the normal error region, the process proceeds to S205; if not, the process proceeds to S210.

[0046] Here, the process of S210 will be described first. In S210, the rendering unit 104d sets a process c at a position corresponding to the pixel to be processed in the provisional processing map. The provisional processing map is a map indicating the type of processing to be applied to each pixel position of the first gain information. However, the type of processing set in the provisional processing map is provisional, and the type of processing to be finally applied is set in a processing map to be described later. Therefore, the provisional processing map is configured by the process of S210 so as to indicate that there is a possibility that the process c will be executed on the first gain information in the pixel to be processed.

[0047] Here, the process c will be described with reference to FIG. 7. FIG. 7 shows a state in which a gain difference occurs in the first gain information, but no normal error region exists. In FIG. 7, in the regions showing the first gain information and the second gain information, the gain amount is visually expressed by the darkness of the color. The darker the color of the pixel, the larger the negative gain amount for forming the shadow is applied. Under the region showing the first gain information, there are arranged a region showing the normal error information acquired in S102, a region showing the provisional processing map, and a region showing the processing map generated in S213 described later. Also, in FIG. 7, the change in the gain difference between adjacent pixels on the horizontal line of interest is shown in a graph, and the vertical axis of the graph is the gain difference.

[0048] In the area showing normal error information, pixels that are colored darkly indicate that a normal error has occurred. In the example of FIG. 7, all pixels are white, so there is no normal error area. Therefore, regardless of which pixel is the processing target pixel, processing c is set in the provisional processing map by the processing of S210. Processing c is also set for each pixel in the processing map generated in S213, which will be described later. Note that in the area showing the provisional processing map and the processing map, white pixels indicate pixels to which processing c has been set.

[0049] The process c is a process for setting the same gain amount as the first gain information as the second gain information. Therefore, in FIG. 7, there is no change between the first gain information and the second gain information.

[0050] Here, referring again to FIG. 4A, a case where the processing target pixel is a pixel in a normal error region (when the processing step proceeds from S204 to S205) will be described.

[0051] In S205, the rendering unit 104d determines whether the pixel to be processed is a pixel in a boundary area based on the boundary information acquired in S104. If the pixel to be processed is a pixel in a boundary area, the process proceeds to S209, and if not, the process proceeds to S208.

[0052] In S208, the rendering unit 104d sets the process a at a position corresponding to the pixel to be processed in the provisional processing map. In S209, the rendering unit 104d sets the process b at a position corresponding to the pixel to be processed in the provisional processing map. The details of the process a and the process b will be described later.

[0053] In S212, the rendering unit 104d determines whether or not there are any unprocessed pixels. If there are any unprocessed pixels, the processing step returns to S201. Then, the rendering unit 104d selects one of the unprocessed pixels as the next pixel to be processed. Thus, the processing of S201 to S212 is performed for each pixel. When the processing for all pixels is completed (i.e., when it is determined in S212 that there are no unprocessed pixels), the processing step proceeds to S213.

[0054] In S213, the rendering unit 104d generates a processing map based on the provisional processing map in which the provisional processing type is set in S208 to S210, the averaging processing range determined in S203, and the processing b range threshold th. A method for generating the processing map will be described in detail later.

[0055] 4B, the rendering unit 104d executes the process a based on the process map generated in S213. The process a will be described in detail later.

[0056] After that, in S303, the rendering unit 104d executes the process b based on the process map generated in S213. Details of the process b will be described later.

[0057] In S304, the rendering unit 104d stores the first gain information to which the processes a and b have been applied in S302 and S303, respectively, as second gain information.

[0058] 3 is completed, and the second gain information is obtained. As described above, the second gain information thus obtained is used for the rendering process by the rendering unit 104d in S107.

[0059] Here, the process a, the process b, and the process map generation process will be described with reference to Fig. 5, Fig. 6, Fig. 8, and Fig. 11. Fig. 5 is a diagram for explaining the process a, and Fig. 6 and Fig. 11 are diagrams for explaining the process b. In each example shown in Fig. 5, Fig. 6, and Fig. 11, the first gain information has a gain step between adjacent pixels, and there is a normal error region that includes the position of the gain step or is in contact with the position of the gain step. On the other hand, Fig. 5 shows a case where there is no change in flatness corresponding to the boundary information in the normal error region (there is no boundary), and Fig. 6 and Fig. 11 show a case where there is a change in flatness in the normal error region (there is a boundary).

[0060] First, the process a will be described with reference to FIG. 5. In the provisional process map and the area showing the process map in FIG. 5, the dark pixels indicate pixels to which the process a is set. The rendering unit 104d generates a process map based on the provisional process map, and performs the process a on the first gain amount from the first gain information for each pixel to which the process a is set in the process map. The process a is a process of obtaining a second gain amount for the second gain information by performing a smoothing process using the first gain amount in the pixel of interest (pixel to which the process a is performed) and its surrounding pixels. In the following description, the rendering unit 104d performs an averaging process as the smoothing process, but is not limited to this, and other types of smoothing processes may be performed. In this embodiment, a process of calculating a second gain amount at the pixel of interest is performed by performing an averaging process by averaging the first gain amount in the pixels in the area described as the averaging process range including the pixel of interest as shown in FIG. 5.

[0061] Here, the method of determining the averaging processing range (smoothing range) will be described in detail with reference to FIG. 8. It is shown that the gain step in the first gain amount of the first gain information becomes larger in the order of FIG. 8(a), FIG. 8(b), and FIG. 8(c). It is also shown that the horizontal averaging processing range is set wider with the increase in the gain step. Therefore, the averaging processing range determined for the second position where the difference in the gain amount (gain step) is the second value is wider than the averaging processing range determined for the first position where the gain step is the first value smaller than the second value. This makes it possible to realize a smooth change in the shadow area on the normal error area after the relighting image is formed, regardless of the size of the gain step. It is also possible to suppress the occurrence of unnatural shadows with high contrast and tone jumps in the shadow area.

[0062] In the processing map generation process of S213, the rendering unit 104d generates a final processing map by newly setting an area of ​​processing a taking into account the averaging processing range based on the averaging processing range determined in S203 (see Figure 8) and the area of ​​the provisional processing map in which processing a was set in S208.

[0063] In this embodiment, the setting of the horizontal averaging range is described assuming that a gain step occurs in the horizontal direction. However, this is not limited to this, and when a gain step occurs in the vertical direction, the vertical averaging range may be set wider as the gain step amount increases. Furthermore, when there is a gain step in the horizontal direction and the vertical direction, for example, when the tap numbers of the averaging filter corresponding to the averaging range in each direction are fH and fV, the averaging may be performed with an fH×fV averaging filter. Furthermore, the averaging process is not limited to being performed by simple averaging, and the averaging process may be performed by weighting the pixels closer to the pixel of interest.

[0064] Next, with reference to FIGS. 6 and 11, process b will be described. The difference between FIG. 6 and FIG. 11 is that the range of the normal error region is different. FIG. 6 shows a case where the distance B (the distance from the pixel of process b in the temporary process map to the pixel at the right end of the normal error region, which is the distance from the boundary position to the position adjacent to the normal error region) is smaller than the process b range threshold th (B < th). On the other hand, FIG. 11 shows a case where the distance B from the pixel of process b in the temporary process map to the pixel at the right end of the normal error region is larger than the process b range threshold th (B > th). In FIGS. 6 and 11, the distance A from the pixel of process b in the temporary process map to the pixel at the left end of the normal error region is smaller than the process b range threshold th. Note that the process b range threshold th may be a predetermined value or a value that can be changed by the user through the information input unit 108.

[0065] As shown in FIGS. 6 and 11, when processing in the horizontal direction, the non-normal error regions adjacent to the normal error region are naturally formed at both ends of the normal error region. FIGS. 6 and 11 also show that the change position of the flatness, which is the boundary information, exists within the normal error region. Based on this boundary information, process b is set in the boundary region in the temporary process map. In the regions showing the temporary process map and the process map of FIGS. 6 and 11, the pixels of medium density indicate the pixels where process b is set.

[0066] In the case of B < th shown in FIG. 6 (when the distance is less than the threshold), the rendering unit 104d performs a process of setting the first gain amount at the pixels of the non-normal error region adjacent to each normal error region as the second gain amount for the pixels of each normal error region divided by the pixel of process b in the temporary process map. That is, the rendering unit 104d replaces the gain amount in the region corresponding to the normal error region in the first gain amount distribution indicated by the first gain information with the gain amount at the position adjacent to the normal error region. This is done because the material may change at the change position of the flatness, which is the boundary information. That is, with the change position of the boundary information (flatness) as the boundary, the Lambert diffuse reflectance ρ in the above formula (1) dThis is because there is a possibility that the normal error region θ may have changed, and this must be taken into consideration when setting the second gain amount for the normal error region described above.

[0067] 11 (when the distance is not less than the threshold), the rendering unit 104d performs a process of setting the first gain amount after the process a is performed for a pixel adjacent to the pixel at a distance th from the boundary as the second gain amount for each pixel in each normal error region obtained by dividing the temporary processing map into two at the pixel of the process b. That is, the rendering unit 104d replaces the gain amount of the region included in the distance range equal to or less than the threshold from the boundary position in the first gain amount distribution indicated by the first gain information with the gain amount of the position adjacent to the region included in this distance range.

[0068] In this embodiment, the case of processing in the horizontal direction has been described, but the present invention is not limited to this, and the second gain amount can be obtained in the vertical direction by processing in the same manner as in the horizontal direction. In addition, when there are gain steps in the horizontal and vertical directions and the second gain amount can be obtained in each direction, the arithmetic mean of the two gain amounts may be used as the second gain amount.

[0069] As described above, according to the first embodiment, the imaging device 1 acquires light source information that defines a virtual light source, normal information that indicates the distribution of normals in the imaging range, and normal error information that indicates a normal error region where an error occurs in the distribution of normals. The imaging device 1 also generates first gain information that indicates a first gain amount distribution based on the light source information and the normal information. Then, the imaging device 1 generates second gain information that indicates a second gain amount distribution by performing a smoothing process on the gain amount of a region corresponding to the normal error region in the first gain amount distribution, as shown in FIG. 5, for example. After that, the imaging device 1 applies a gain according to the second gain amount distribution to the image in the imaging range, thereby adding the effect of a virtual light source to the image. Note that the imaging device 1 may generate second gain information that indicates a second gain amount distribution by replacing the gain amount of a region corresponding to the normal error region in the first gain amount distribution with the gain amount at a position adjacent to the normal error region after performing the above smoothing process, as shown in FIG. 6, for example.

[0070] This makes it possible to generate a virtual light source image (relighting image) with natural shading, as shown in Figure 10(b), and suppresses image quality degradation of the virtual light source image caused by normal error areas.

[0071] In the above description, the gain information generated based on the normal information with a normal error region is corrected based on the normal error region. However, the imaging device 1 (information processing device) may correct the normal information with a normal error region itself based on the normal error region instead of correcting the gain information. In this case, the imaging device 1 generates second normal information indicating a second normal distribution by performing a smoothing process on the normal of the normal error region in the first normal distribution (distribution of normals in the shooting range) indicated by the normal information (first normal information) acquired in S102 of FIG. 3. As a specific method of the smoothing process, a method similar to the smoothing process of the gain amount described with reference to FIG. 5 can be used. In addition, the imaging device 1 may perform replacement of normals based on the boundary position by a method similar to the replacement process of the gain amount described with reference to FIG. 6 and FIG. 11 after performing the smoothing process.

[0072] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0073] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0074] 1...imaging device, 104...image processing unit, 104a...input image acquisition unit, 104b...normal information acquisition unit, 104c...boundary information acquisition unit, 104d...rendering unit, 108...information input unit, 110...system control unit, 200...light source unit

Claims

1. An acquisition means for acquiring light source information that defines a virtual light source, normal information that indicates a distribution of normals in a shooting range, and normal error information that indicates a normal error region in which an error occurs in the distribution of normals; a first generating means for generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generating means for generating second gain information indicating a second gain amount distribution by performing a smoothing process on a gain amount in a region corresponding to the normal error region in the first gain amount distribution; an adding means for adding an effect of the virtual light source to an image in the shooting range by applying a gain according to the second gain amount distribution to the image; Equipped with the second generating means determines a smoothing range for each position in the smoothing process based on a difference in gain amount at each position in a region corresponding to the normal error region in the first gain amount distribution; The smoothing range determined for a second position where the difference in gain amount is a second value is wider than the smoothing range determined for a first position where the difference in gain amount is a first value smaller than the second value.

13. An image processing device comprising:

2. An acquisition means for acquiring light source information that defines a virtual light source, normal information that indicates a distribution of normals in a shooting range, and normal error information that indicates a normal error area in which an error occurs in the distribution of normals; a first generating means for generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generating means for generating second gain information indicating a second gain amount distribution by performing a smoothing process on a gain amount in a region corresponding to the normal error region in the first gain amount distribution; an adding means for adding an effect of the virtual light source to an image in the shooting range by applying a gain according to the second gain amount distribution to the image; Equipped with the acquiring means further acquires boundary information indicating a boundary position between objects in the shooting range, When the normal error region includes the boundary position, the second generating means, after performing the smoothing process, replaces a gain amount in a region corresponding to the normal error region in the first gain amount distribution with a gain amount in a position adjacent to the normal error region, thereby generating the second gain information indicating the second gain amount distribution.

13. An image processing device comprising:

3. An acquisition means for acquiring light source information that defines a virtual light source, normal information that indicates a distribution of normals in a shooting range, and normal error information that indicates a normal error area in which an error occurs in the distribution of normals; a first generating means for generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generating means for generating second gain information indicating a second gain amount distribution by performing a smoothing process on a gain amount in a region corresponding to the normal error region in the first gain amount distribution; an adding means for adding an effect of the virtual light source to an image in the shooting range by applying a gain according to the second gain amount distribution to the image; Equipped with the acquiring means further acquires boundary information indicating a boundary position between objects in the shooting range, When the normal error region includes the boundary position, the second generating means determining whether a distance from the boundary position to a position adjacent to the normal error region is less than a threshold; When the distance is less than the threshold value, after performing the smoothing process, a gain amount in a region corresponding to the normal error region in the first gain amount distribution is replaced with a gain amount in a position adjacent to the normal error region, thereby generating the second gain information indicating the second gain amount distribution; When the distance is not less than the threshold, after performing the smoothing process, the gain amount in a region included in the distance range from the boundary position to the threshold or less in the first gain amount distribution is replaced with the gain amount in a position adjacent to the region included in the distance range, thereby generating the second gain information indicating the second gain amount distribution.

13. An image processing device comprising:

4. The acquiring means acquires, as the boundary information, information indicating a change in diffuse reflectance for each pixel position, information indicating a change in object distance for each pixel position, object detection information, information indicating a change in color for each pixel position, or information indicating a change in luminance for each pixel position.

4. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

5. An image processing device according to any one of claims 1 to 4, An imaging means for generating the image; An imaging device comprising:

6. An image processing method executed by an image processing device, comprising: An acquisition step of acquiring light source information that defines a virtual light source, normal information that indicates a distribution of normals in a shooting range, and normal error information that indicates a normal error region in which an error occurs in the distribution of normals; a first generation step of generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generation step of generating second gain information indicating a second gain amount distribution by performing a smoothing process on the gain amount of a region in the first gain amount distribution corresponding to the normal error region; an adding step of adding the effect of the virtual light source to the image of the shooting range by applying a gain according to the second gain amount distribution to the image; Equipped with The second generation step determines a smoothing range for each position in the smoothing process based on a difference in gain amount at each position in a region corresponding to the normal error region in the first gain amount distribution, The smoothing range determined for a second position where the difference in gain amount is a second value is wider than the smoothing range determined for a first position where the difference in gain amount is a first value smaller than the second value.

13. An image processing method comprising:

7. An image processing method executed by an image processing device, comprising: An acquisition step of acquiring light source information that defines a virtual light source, normal information that indicates a distribution of normals in a shooting range, and normal error information that indicates a normal error region in which an error occurs in the distribution of normals; a first generation step of generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generation step of generating second gain information indicating a second gain amount distribution by performing a smoothing process on the gain amount of a region in the first gain amount distribution corresponding to the normal error region; an adding step of adding the effect of the virtual light source to the image of the shooting range by applying a gain according to the second gain amount distribution to the image; Equipped with The acquiring step further includes acquiring boundary information indicating a boundary position between objects in the shooting range, When the normal error region includes the boundary position, the second generation step generates the second gain information indicating the second gain amount distribution by replacing a gain amount of a region corresponding to the normal error region in the first gain amount distribution with a gain amount of a position adjacent to the normal error region after performing the smoothing process.

13. An image processing method comprising:

8. An image processing method executed by an image processing device, comprising: An acquisition step of acquiring light source information that defines a virtual light source, normal information that indicates a distribution of normals in a shooting range, and normal error information that indicates a normal error region in which an error occurs in the distribution of normals; a first generation step of generating first gain information indicating a first gain amount distribution based on the light source information and the normal information; a second generation step of generating second gain information indicating a second gain amount distribution by performing a smoothing process on the gain amount of a region in the first gain amount distribution corresponding to the normal error region; an adding step of adding the effect of the virtual light source to the image of the shooting range by applying a gain according to the second gain amount distribution to the image; Equipped with The acquiring step further includes acquiring boundary information indicating a boundary position between objects in the shooting range, When the normal error region includes the boundary position, the second generation step determining whether a distance from the boundary position to a position adjacent to the normal error region is less than a threshold; When the distance is less than the threshold value, after performing the smoothing process, a gain amount in a region corresponding to the normal error region in the first gain amount distribution is replaced with a gain amount in a position adjacent to the normal error region, thereby generating the second gain information indicating the second gain amount distribution; When the distance is not less than the threshold, after performing the smoothing process, the gain amount in a region included in the distance range from the boundary position to the threshold or less in the first gain amount distribution is replaced with the gain amount in a position adjacent to the region included in the distance range, thereby generating the second gain information indicating the second gain amount distribution.

13. An image processing method comprising:

9. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Image processing device, imaging device, and image processing program

    JP2017150878A

  • Processing equipment, processing system, imaging device, processing method, program, and recording medium

    JP2019168862A

  • Image processing apparatus, image processing method, and program

    JP2020024581A

  • Processing device, imaging device, processing system, processing method, program, and storage medium

    JP2020095454A