Image Processing Apparatus, Image Processing Method, and Storage Medium

Abstract

The present invention provides an image processing apparatus, an image processing method, and a storage medium. The image processing apparatus includes: a processing component configured to provide an illumination effect from a virtual light source for an image including a first subject and a second subject; a setting component configured to set parameters of the virtual light source; and a specifying component configured to specify, based on a user operation, the first subject and / or the second subject as an object of the processing component, and in a case where the specifying component has specified at least one of the first subject and the second subject, provide an effect of virtual light from the same direction for the first subject and the second subject, and in a case where the specifying component has not specified a subject, provide an effect of virtual light from different directions for the first subject and the second subject.

Description

Technical Field

[0001] The present invention relates to a technique for correcting the brightness of an image through image processing. Background Art

[0002] Conventionally, a technique for correcting the brightness of a dark part of a subject by applying the effect of virtual light to the subject in an image has been known (Japanese Patent Application Laid-Open No. 2010-135996). Therefore, it is possible to adjust the shadow of a subject generated by the projection of ambient light after the image is taken.

[0003] However, according to the prior art disclosed in the above Japanese Patent Application Laid-Open No. 2010-135996, when an image of a plurality of subjects is taken and the image is corrected using a virtual light source determined according to the shadow for each subject, how the subjects are illuminated by the light from the virtual light source may be unnatural. On the other hand, when a plurality of subjects are corrected by irradiating them with the light from a virtual light source determined based on the shadow of a specific subject, how the subjects are illuminated by the light from the virtual light source may be natural, but it may be difficult to appropriately correct each subject. Therefore, when there are a plurality of subjects, a desired lighting effect may not be achieved. Summary of the Invention

[0004] In view of the above problems, the present invention provides an image processing apparatus that can effectively correct shadows even when there are a plurality of subjects.

[0005] According to a first aspect of the present invention, there is provided an image processing apparatus including: a processing unit configured to provide an illumination effect of virtual light for an image including a first subject and a second subject; a setting unit configured to set parameters of the virtual light source; and a specifying unit configured to specify, based on a user operation, the first subject and / or the second subject as an object of the processing unit, wherein, when the specifying unit has specified at least one of the first subject and the second subject, an effect of virtual light from the same direction is provided for the first subject and the second subject, and when the specifying unit has not specified any subject, an effect of virtual light from different directions is provided for the first subject and the second subject.

[0006] According to a second aspect of the present invention, there is provided an image processing method, which includes: providing an illumination effect from a virtual light source for an image including a first subject and a second subject; setting parameters of the virtual light source; and performing a selection process in which, based on a user operation, the first subject and / or the second subject is specified as an object to be processed, wherein, when at least one of the first subject and the second subject has been specified in the selection process, a virtual light effect from the same direction is provided for the first subject and the second subject, and when no subject has been specified in the selection process, a virtual light effect from different directions is provided for the first subject and the second subject.

[0007] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a program for causing a computer to execute the steps of the above-described image processing method.

[0008] According to a fourth aspect of the present invention, there is provided an image processing apparatus, which includes: a processing unit for providing an illumination effect from a virtual light source for a plurality of subjects in an image; a setting unit for setting parameters of the virtual light source; and a specifying unit for specifying, based on a user operation, any one of the plurality of subjects as an object of the processing unit, wherein, when any subject has been specified by the specifying unit, the setting unit sets parameters of the virtual light source for the subjects not specified by the specifying unit based on the parameters of the virtual light source for the specified subject, and when no subject has been specified by the specifying unit, the setting unit sets parameters of the virtual light source for each subject based on information related to each of the plurality of subjects.

[0009] According to a fifth aspect of the present invention, there is provided an image processing method, which includes: providing an illumination effect from a virtual light source for a plurality of subjects in an image; setting parameters of the virtual light source; and performing a selection process in which, based on a user operation, any one of the plurality of subjects is specified as an object to be processed, wherein, when any subject has been specified in the selection process, in the setting, parameters of the virtual light source for the subjects not specified in the selection process are set based on the parameters of the virtual light source for the specified subject, and when no subject has been specified in the selection process, in the setting, parameters of the virtual light source for each subject are set based on information related to each of the plurality of subjects.

[0010] According to a sixth aspect of the present invention, there is provided a computer-readable storage medium storing a program for causing a computer to execute the steps of the above-described image processing method.

[0011] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing the configuration of a digital camera as an embodiment of an image processing apparatus of the present invention.

[0013] Figure 2 is a block diagram showing the configuration of an image processing unit in an embodiment.

[0014] Figure 3 is a block diagram showing the configuration of a re-lighting processing unit in an embodiment.

[0015] Figure 4 is a schematic diagram illustrating the reflection of virtual light from a virtual light source in an embodiment.

[0016] Figure 5A and Figure 5B is a diagram illustrating parameter control of a virtual light source in an embodiment.

[0017] Figure 6A and Figure 6B are diagrams showing examples of images before and after re-lighting processing in an embodiment.

[0018] Figure 7 is a flowchart illustrating re-lighting processing in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but the invention is not limited to requiring all of these features, and multiple such features can be appropriately combined. Further, in the drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions thereof are omitted.

[0020] Figure 1 is a block diagram of the configuration of a digital camera 100 as an embodiment of an image processing apparatus of the present invention.

[0021] In Figure 1In the digital camera 100 shown, incident light transmitted through a lens unit 101 (imaging optical system) including a zoom lens and a focusing lens and a shutter 102 having an aperture function is subjected to photoelectric conversion in an imaging unit 103. The imaging unit 103 includes an image sensor such as a CCD or CMOS sensor, and an electrical signal obtained by the photoelectric conversion is input to an A / D converter 104 as an image signal. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data), and outputs it to an image processing unit 105.

[0022] The image processing unit 105 performs various types of image processing such as color conversion processing (e.g., white balance), γ processing, edge enhancement processing, and color correction processing on the image data from the A / D converter 104 or the image data read from the image memory 106 by the memory control unit 107. The memory control unit 107 writes the image data output from the image processing unit 105 into the image memory 106. The image memory 106 stores the image data output from the image processing unit 105 and the image data to be displayed on the display unit 109.

[0023] A face detection unit 113 detects a face and face organ region including a person's face and face organs from a captured image. The image processing unit 105 uses the face detection result or face organ detection result from the face detection unit 113 and the captured image data to perform a predetermined evaluation value calculation process, and the system control unit 50 performs exposure control and focus control based on the obtained evaluation value. Therefore, AF (autofocus) processing, AE (autoexposure) processing, AWB (automatic white balance) processing, etc. corresponding to the TTL (through the lens) method are executed.

[0024] In addition, a D / A converter 108 converts the digital image data for display stored in the image memory 106 into an analog signal, and supplies it to the display unit 109. The display unit 109 performs a display corresponding to the analog signal from the D / A converter 108 on a display unit such as an LCD.

[0025] The encoding / decoding unit 110 compresses and encodes the image data stored in the image memory 106 according to standards such as JPEG or MPEG. The system control unit 50 stores the encoded image data in a storage medium 112 such as a memory card or a hard disk via an interface (I / F) 111. In addition, the image data read from the storage medium 112 via the I / F 111 is decoded and decompressed by the encoding / decoding unit 110 and stored in the image memory 106. When the image data stored in the image memory 106 is displayed on the display unit 109 using the memory control unit 107 and the D / A converter 108, the image is reproduced and displayed.

[0026] The relighting processing unit 114 performs relighting processing for irradiating the captured image with light from a virtual light source, thereby correcting the brightness. The relighting processing performed by the relighting processing unit 114 will be described in detail later.

[0027] The system control unit 50 controls the entire system of the digital camera 100. The non-volatile memory 121 is composed of a memory such as an EEPROM and stores programs, parameters, etc. required for the processing of the system control unit 50. The system control unit 50 expands and executes the programs stored in the non-volatile memory 121, as well as constants and variables for the operation of the system control unit 50 in the system memory 122, thereby implementing the processing of the embodiments described later.

[0028] The operation unit 120 accepts operations performed by the user such as menu settings or image selection. The distance measurement sensor 123 measures the distance to the subject and outputs distance information corresponding to each pixel in the imaging pixels.

[0029] Next, the image processing unit 105 will be described in detail with reference to Figure 2 FIG. Figure 2 is a block diagram showing the configuration of the image processing unit 105. In the present embodiment, it is assumed that the image sensor of the imaging unit 103 is covered with a color filter of a Bayer array. Therefore, R, G, and B image signals are output from each pixel of the image sensor of the imaging unit 103.

[0030] First, from Figure 1The Bayer RGB image data input to the A / D converter 104 shown is input to the synchronization processing unit 200. The synchronization processing unit 200 performs synchronization processing on the input R, G, and B image signals, thereby generating color signals RGB for each pixel. The WB amplification unit 201 applies a gain to the generated color signals RGB for each pixel based on the white balance gain value calculated by the system control unit 50 through known processing, thereby adjusting the white balance. The color signals RGB whose white balance has been adjusted by the WB amplification unit 201 are input to the luminance and color signal generation unit 202. The luminance and color signal generation unit 202 generates a luminance signal Y from the color signals RGB, outputs the generated luminance signal Y to the edge enhancement processing unit 203, and outputs the color signals RGB to the color conversion processing unit 205.

[0031] The edge enhancement processing unit 203 performs edge enhancement processing on the luminance signal Y and outputs the resulting signal to the luminance gamma processing unit 204. The color conversion processing unit 205 performs, for example, matrix calculations on the color signals RGB, thereby performing a conversion to achieve a desired color balance, and the color conversion processing unit 205 outputs the resulting signal to the color gamma processing unit 206 and the subject information detection unit 208.

[0032] The subject information detection unit 208 detects information related to the subject in the captured image from the face / face organ detection information output from the face detection unit 113 and the color signals RGB output from the color conversion processing unit 205. Information related to the subject is the number of subjects in the captured image, the size of the subject, the position of the subject, how the subject is illuminated by light, the shadow information of the subject, etc. For example, the number, size, and position of the subject are detected from the coordinate position information of each face / face organ output from the face detection unit 113, and the shadow information and how the subject is illuminated by light are detected from the average luminance information and luminance histogram information of the entire captured image and each subject.

[0033] The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the gamma-corrected luminance signal Y to the image memory 106 using the memory control unit 107. In addition, the color gamma processing unit 206 performs gamma correction on the color signals RGB and outputs the resulting signal to the color difference signal generation unit 207. The color difference signal generation unit 207 generates color difference signals R-Y and B-Y from the RGB signals and outputs them to the image memory 106 using the memory control unit 107.

[0034] Next, the relighting processing in this embodiment will be described. Figure 3It is a block diagram showing the configuration of the relighting processing unit 114.

[0035] The relighting processing unit 114 reads the luminance signal Y and the color difference signals B-Y and R-Y that have been processed by the image processing unit 105 and stored in the image memory 106, and uses these signals as inputs to perform relighting processing using a virtual light source.

[0036] First, the RGB signal conversion unit 301 converts the input luminance signal Y and color difference signals B-Y and R-Y into RGB signals, and outputs the obtained signals to the gamma correction processing unit 302. The gamma correction processing unit 302 performs an operation opposite to the gamma characteristics of the gamma correction by the luminance gamma processing unit 204 and the color gamma processing unit 206 of the image processing unit 105 (gamma correction), thereby converting the signals into linear data. The gamma correction processing unit 302 outputs the RGB signals (Rt, Gt, Bt) that have been converted into linear data to the virtual light source reflection component calculation unit 307 and the virtual light source addition processing unit 308.

[0037] In addition, the distance calculation unit 303 calculates a distance map from the subject distance information obtained from the distance measurement sensor 123. The subject distance information is two-dimensional distance information obtained at the pixel level of the captured image. The subject area calculation unit 304 uses the subject information input from the subject information detection unit 208 of the image processing unit 105 and the distance map input from the distance calculation unit 303 as inputs to calculate the subject area, and outputs a subject area map, where the subject information indicates the number, position, face size, contrast, shadow information, etc. of the subject in the captured image. The subject area map indicates whether each pixel of the captured image contains a subject, and the calculation method will be described in detail later. The normal calculation unit 305 calculates a normal map as shape information indicating the shape of the subject from the distance map calculated by the distance calculation unit 303. It is assumed that a known technique is used as the method for generating the normal map from the distance map, but specific processing examples will be described with reference to Figure 4 Describe its specific processing examples.

[0038] Figure 4 It is a diagram illustrating the relationship between the camera imaging coordinates and the subject. For example, as Figure 4 shown, the difference ΔDH of the distance D with respect to the difference ΔH in the horizontal direction of the captured image, and the distance D with respect to the vertical direction (perpendicular to Figure 5A and Figure 5BThe difference ΔDV of the difference ΔV in the direction of the drawing surface) calculates the gradient information of a part of the subject 401. From the obtained gradient information of a part of the subject, the normal N can be calculated. By performing the above processing on the pixels of the captured image, the normal N corresponding to each pixel of the captured image can be calculated. The normal calculation unit 305 outputs information about the normal N corresponding to each pixel of the captured image as a normal map to the virtual light source reflection component calculation unit 307.

[0039] Although the distance calculation unit 303 and the normal calculation unit 305 are described as being provided in the relighting processing unit 114, the present invention is not limited thereto, and for example, the distance calculation unit 303 and the normal calculation unit 305 can be provided in the distance measurement sensor 123 or the image processing unit 105 or provided independently.

[0040] The virtual light source setting unit 306 sets the parameters of the virtual light source based on the subject information input from the subject information detection unit 208 of the image processing unit 105. For example, if it is desired to overall increase the brightness of the face of a subject whose overall face is darker, the parameters such as the position, irradiation range, and intensity of the virtual light source are controlled so that the entire face is included in the irradiation range of the virtual light source. It is assumed that the virtual light source includes at least one of additive light for increasing the brightness of the subject, subtractive light for decreasing the brightness of the subject, and specular reflection light for adding specular reflection to the subject.

[0041] Hereinafter, it will be assumed that the number of subjects is one, and reference will be made to Figure 5A and Figure 5B to describe the parameters set for the virtual light source.

[0042] Figure 5A is a three-dimensional diagram (3D diagram) illustrating the positional relationship between the subject and the virtual light source, and Figure 5B is a plan view illustrating the positional relationship between the subject and the virtual light source. The position of the virtual light source is such that if the distance between the virtual light source and the subject is set short, the light from the virtual light source enters the subject with high intensity, and conversely, if the distance from the subject is set long, the light from the virtual light source enters the subject with low intensity. The irradiation range of the virtual light source is such that if the irradiation range of the virtual light source is set wide, the light enters the entire subject, and conversely, if the irradiation range is set narrow, the light only enters a part of the subject. In addition, the intensity of the virtual light source is such that if the intensity of the virtual light source is set high, the light enters the subject with high intensity, and conversely, if the intensity is set low, the light enters the subject with low intensity.

[0043] The virtual light source reflection component calculation unit 307 calculates the component of the light virtually emitted from the set virtual light source that is reflected by the subject, based on the distance K between the light source and the subject, the normal information N, and the parameters of the virtual light source set by the virtual light source setting unit 306. Hereinafter, the light virtually emitted from the set virtual light source is referred to as "virtual light". Specifically, the reflection component of the virtual light at the portion of the subject corresponding to the coordinate position in the captured image is calculated in a manner inversely proportional to the square of the distance K between the virtual light source and the portion of the subject corresponding to each pixel, and in a manner proportional to the inner product of the vector of the normal N and the vector of the light source direction L.

[0044] Hereinafter, a commonly used method for calculating the reflection component of the virtual light will be described with reference to Figure 4 Note that although only the horizontal direction of the captured image is shown for the purpose of description in Figure 4 , the direction perpendicular to the drawing plane is the same as the vertical direction of the captured image as described above. In the following description, a method for calculating the reflection component of the virtual light at point P1 on the subject 401, where point P1 corresponds to the horizontal pixel position H1 and the vertical pixel position V1 (not shown) in the captured image, will be described.

[0045] In Figure 4 , the virtual light source 402 is a virtual light source set for the subject 401. The reflection component of the virtual light at the position (H1, V1) in the image captured by the camera 100 is proportional to the inner product of the normal vector N1 at point P1 on the subject 401 and the light source direction vector L1 of the virtual light source 402, and is inversely proportional to the square of the distance K1 between the virtual light source 420 and point P1. Note that the normal vector N1 and the light source direction vector L1 are three-dimensional vectors each composed of the horizontal direction, the vertical direction, and the depth direction (the direction represented by the distance D in Figure 4 ). This relationship can be expressed by the following mathematical formula, that is, the reflection component (Ra, Ga, Ba) of the virtual light at point P1 on the subject 401 is expressed by the following formula (1):

[0046] Ra = α × {(-L1 · N1) / K1 2} × Rt

[0047] Ga = α × {(-L1 · N1) / K1 2} × Gt...(1)

[0048] Ba = α × {(-L1 · N1) / K1 2} × Bt

[0049] where α is the intensity of the light from the virtual light source and is the gain value of the relighting correction amount, and Rt, Gt, and Bt are the RGB signals output from the de-gamma processing unit 302.

[0050] The calculated reflection components (Ra, Ga, Ba) of the virtual light are output to the virtual light source addition processing unit 308. The virtual light source addition processing unit 308 adds the reflection components (Ra, Ga, Ba) of the virtual light to the RGB signal output from the de-gamma processing unit 302 through the processing expressed by the following formula (2):

[0051] Rout = Rt + Ra

[0052] Gout = Gt + Ga…(2)

[0053] Bout = Bt + Ba

[0054] The RBG signal (Rout, Gout, Bout) that has undergone the relighting processing by the virtual light source addition processing unit 308 is output to the gamma processing unit 309 that performs gamma correction. Then, the luminance and color difference signal generation unit 310 generates a luminance signal Y and color difference signals R-Y and B-Y from the gamma-processed RGB signal (R’out, G’out, B’out), and outputs these signals.

[0055] Figure 6A and Figure 6B shows an example of the relighting processing performed by the relighting processing unit 114. Figure 6A shows an example of the captured image before the relighting processing, and Figure 6B shows an example of the captured image after the relighting processing. As Figure 6A shown, a dark subject can undergo the relighting processing by applying virtual light (for example, the relighting processing described below with reference to Figure 7 ), and thus the brightness of the subject can be corrected and increased as Figure 6B shown.

[0056] The system control unit 50 controls the memory control unit 107 so that the luminance signal Y and color difference signals R-Y and B-Y output from the relighting processing unit 114 are stored in the image memory 106, and then causes the codec unit 110 to compress and encode these signals. These signals are transmitted via the I / F111 and stored in the storage medium 112.

[0057] Next, the relighting processing by the relighting processing unit 114 in this embodiment will be described with reference to the flowchart in Figure 7 . This processing is performed on the image (luminance signal Y and color difference signals R-Y and B-Y) processed by the image processing unit 105 and stored in the image memory 106 when the relighting processing is selected by an operation from the user using the operation unit 120.

[0058] In step S601, the system control unit 50 acquires a relighting processing mode selected by an operation of the utilization operation unit 120 from the user. In the present embodiment, there are the following modes: a mode of automatically determining parameters of a virtual light source, and a mode in which the user designates parameters of the virtual light source.

[0059] In step S602, the system control unit 50 determines whether the automatic correction mode of the relighting process acquired in step S601 is valid. If it is determined that the mode is valid, the process proceeds to step S603; otherwise, the process proceeds to step S605.

[0060] In step S603, the system control unit 50 causes the subject area calculation unit 304 to calculate an individual subject area for the number of subjects based on the subject information output from the subject information detection unit 208. For example, using the total number N of people detected as subjects in the image and the positions of each person as subjects as inputs, it is determined whether each pixel contains a person, and an individual subject area map is generated corresponding to each subject. Specifically, assuming that the individual subject area map corresponding to the n-th person (n ∈ [1, 2, …, N]) is Dn, and the value of the individual subject area map at coordinates (x, y) is Dn(x, y), if the n-th person exists at coordinates (x1, y1), then Dn(x1, y1) = 1; otherwise, Dn(x1, y1) = 0. The value taken by the subject area map is not limited to a binary value, and the map can also be a multi-valued map indicating the possibility of the presence of a person.

[0061] In step S604, the system control unit 50 causes the virtual light source setting unit 306 to calculate an individual virtual light source setting value based on the subject information output from the subject information detection unit 208. It is possible to acquire shadow information of the facial areas of each subject, and calculate the position, irradiation range, and intensity of the virtual light source so as to reduce the shadows on the face. Various known methods can be used to calculate the parameters of the virtual light source. Although its detailed description has been omitted, for example, the orientation of the ambient light is estimated from the deviation of the facial brightness values and the normal information of the face, and the position of the light source is determined at a predetermined distance in the direction opposite to the ambient light. The parameters can be obtained by estimating the intensity of the light source so as to eliminate the deviation of the facial brightness values and calculating the irradiation range from the size of the face.

[0062] In step S605, the system control unit 50 determines whether the manual correction mode of the relighting process acquired in step S601 is valid. If it is determined that the mode is valid, the process proceeds to step S606; otherwise, the process proceeds to step S609.

[0063] In step S606, the system control unit 50 acquires the main subject selected by the user's operation (instruction) using the operation unit 120.

[0064] In step S607, the system control unit 50 causes the subject area calculation unit 304 to determine a common subject area based on the main subject acquired in step S606, the subject information output from the subject information detection unit 208, and the distance information output from the distance calculation unit 303. For example, an area having a distance value relative to the main subject within a predetermined distance range (range) is taken as the common subject area. Assuming that the common subject area map is Dc and the value of the common subject area map at coordinates (x, y) is Dc(x, y), if the distance of the coordinates (x, y) is within the predetermined distance range relative to the distance value of the main subject, then Dc(x, y) = 1; otherwise, Dc(x, y) = 0. It is sufficient that the value of the common subject area map is calculated as 1 when the distance from the main subject is 0, and the value of the common subject area map decreases monotonically as the distance from the main subject increases, and the value of the common subject area map is not limited to those according to the above calculation method.

[0065] In step S608, the system control unit 50 acquires a virtual light setting value common to the subjects based on the user's operation. Specifically, the position, intensity, and irradiation range of the light source are acquired through the operation of the user using the operation unit 120.

[0066] In step S609, the system control unit 50 causes the virtual light reflection component calculation unit 307 to calculate the reflection components (Ra, Ga, Ba) of the virtual light based on the normal information output from the normal calculation unit 305, the individual subject area calculated in step S603, the individual light source setting value calculated in step S604, the common subject area calculated in step S607, and the common light source setting value calculated in step S608.

[0067] First, the reflected light components (R1, G1, B1), (R2, G2, B2),..., (RN, GN, BN) are calculated from the N light source setting values calculated in step S604 according to the above formula (1). Next, the reflection components (Rc, Gc, Bc) are calculated from the common light source setting value calculated in step S608 according to the above formula (1). Next, the reflection components obtained by synthesizing the reflection components are calculated. If the individual subject area includes the coordinates (x, y), the virtual light source component of the corresponding light source is applied, and if the common subject area includes the coordinates, the common virtual light source is applied. If two or more virtual light sources are applied, the final reflection components (Ra, Ga, Ba) are the sum of all the reflection components of the virtual light and are calculated by the following formula (3):

[0068]

[0069]

[0070]

[0071] Note that the method for synthesizing light sources is not limited to those expressed by the above formula, and the reflected light components are also compared with each other as shown in the following formula (4), and only the maximum light source is set for calculation.

[0072] Ra(x,y) = max(R1(x,y)D1(x,y), …, RN(x,y)DN(x,y), Rc(x,y)Dc(x,y))

[0073] Ga(x,y) = max(G1(x,y)D1(x,y), …, GN(x,y)DN(x,y), Gc(x,y)Dc(x,y)) … (4)

[0074] Ba(x,y) = max(B1(x,y)D1(x,y), …, BN(x,y)DN(x,y), Bc(x,y)Dc(x,y))

[0075] There are cases where the virtual light source component parameters are not calculated depending on the mode during imaging, and in such cases, the virtual light source component is set to 0.

[0076] In step S610, the system control unit 50 adds a virtual light source. As expressed by the above formula (2), the virtual light source addition processing unit 308 adds the reflected components (Ra, Ga, Ba) of the virtual light source to the output (Rt, Gt, Bt) of the de-gamma coarse-grain unit. After the relighting process ends, the process by the relighting processing unit 114 ends.

[0077] As described above, according to the foregoing embodiments, it is possible to provide the image with the lighting effect as expected by the user.

[0078] Other embodiments

[0079] Embodiments of the present invention can also be implemented by the following method, that is, software (program) that executes the functions of the above embodiments is provided to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0080] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the above claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

Claims

1. An image processing device, comprising: A processing component for providing an illumination effect from a virtual light source to an image including a first subject and a second subject; A setting component for setting parameters of the virtual light source; And A specifying component for specifying, based on a user operation, the first subject and / or the second subject as an object of the processing component, wherein, when the specifying component has specified at least one of the first subject and the second subject, a virtual light effect is provided to the first subject and the second subject with a common virtual light source setting value, and when the specifying component has not specified any subject, a virtual light effect is provided to the first subject and the second subject with different virtual light source setting values.

2. The image processing device according to claim 1, wherein, The setting component has a manual correction mode and an automatic correction mode. In the manual correction mode, the specifying component specifies at least one of the first subject and the second subject. In the automatic correction mode, the specifying component does not specify the first subject or the second subject.

3. The image processing device according to claim 2, wherein, In the automatic correction mode, the setting component sets the parameters of the virtual light source based on shadow information of the subject to reduce the shadow.

4. The image processing device according to claim 3, wherein, The setting component determines the shadow of the subject based on the average luminance or luminance histogram of the subject.

5. The image processing device according to claim 2, wherein, In the manual correction mode, the setting component sets the parameters of the virtual light source based on a user instruction.

6. The image processing device according to claim 1, wherein, When the specifying component has specified at least one of the first subject and the second subject, the setting component sets an irradiation range of the common virtual light source based on information related to the specified subject.

7. The image processing device according to claim 6, wherein, The subject specified by the specifying component is a main subject determined based on a user instruction, and a common subject area is determined within a predetermined range of distance values with respect to the main subject.

8. The image processing device according to claim 1, wherein, The virtual light source includes at least one of additive light for increasing the luminance of the subject, subtractive light for decreasing the luminance of the subject, and specular reflection light for adding specular reflection to the subject.

9. The image processing device according to claim 1, wherein, The parameters of the virtual light source include at least one of the position, irradiation range, orientation, and intensity of the virtual light source.

10. The image processing device according to claim 2, wherein, In an area where components of the virtual light source are added in a repetitive manner, the processing component compares the components of each virtual light source and adds the maximum component.

11. An image processing method, comprising: Providing an illumination effect from a virtual light source to an image including a first subject and a second subject; Setting the parameters of the virtual light source; And Performing a selection process in which, based on a user operation, the first subject and / or the second subject is specified as an object of the process, wherein, when at least one of the first subject and the second subject has been specified in the selection process, a virtual light effect is provided to the first subject and the second subject with a common virtual light source setting value, and when no subject has been specified in the selection process, a virtual light effect is provided to the first subject and the second subject with different virtual light source setting values.

12. A computer-readable storage medium storing a program for causing a computer to execute the steps of the image processing method according to claim 11.

13. An image processing device, comprising: A processing component for providing an illumination effect from a virtual light source for a plurality of subjects in an image; A setting component for setting parameters of the virtual light source; And A specifying component for specifying any one of the plurality of subjects as an object of the processing component based on a user operation, wherein, when the specifying component has specified any subject, the setting component sets parameters of the virtual light source for a subject not specified by the specifying component based on the parameters of the virtual light source for the specified subject, and when the specifying component has not specified any subject, the setting component sets parameters of the virtual light source for each subject based on information related to each of the plurality of subjects.

14. An image processing method, comprising: Provide an illumination effect from a virtual light source for a plurality of subjects in an image; Set the parameters of the virtual light source; And Perform a selection process in which any one of the plurality of subjects is specified as an object of processing based on a user operation, wherein, when any subject has been specified in the selection process, in the setting, parameters of the virtual light source for a subject not specified in the selection process are set based on the parameters of the virtual light source for the specified subject, and when no subject has been specified in the selection process, in the setting, parameters of the virtual light source for each subject are set based on information related to each of the plurality of subjects.

15. A computer-readable storage medium storing a program for causing a computer to execute the steps of the image processing method according to claim 14.

16. A computer program product comprising a program for causing a computer to execute the steps of the image processing method according to claim 11 or 14.

Images