Image processing apparatus, image processing method, and program

Abstract

To provide processing for accurately deriving information on a normal of an object even when a shadow or mirror reflection light is captured in a picked-up image.SOLUTION: An image processing apparatus acquires image data obtained by picking up an image of an object that is irradiated with rays of light from a plurality of lighting units, in the acquired image represented by the image data, sets a first area with low reliability for deriving information on a normal of the object, derives weight for deriving the information on the normal of the object so that weight continuously changes between a first area and a second area different from the first area, and derives the information on the normal of the object on the basis of the derived weight.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing technique for acquiring information about the surface shape of an object. [Background technology]

[0002] There is a known technology that uses multiple lighting fixtures to illuminate an object and capture the illuminated object to derive the normal. This technology derives normal information (the direction of the object surface) by focusing on differences in the illumination angle and the amount of light reflected from the object.

[0003] In a technique for deriving the normal of an object, a shadow may appear in the captured image, or specular reflected light may appear. In this case, the relationship between the illumination angle and the amount of reflected light from the object may not be correctly represented in the captured image, and the normal cannot be derived with high accuracy. Therefore, Patent Document 1 discloses a method for calculating the normal vector after narrowing down the luminance values ​​from multiple captured image data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-122870 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, since the image used to derive the normal is switched for each pixel according to the narrowing down conditions, artifacts may occur in the output normal information.

[0006] Therefore, an object of the present invention is to provide a process for deriving normal information of an object with high accuracy even when a shadow or specular reflected light is captured in a captured image. [Means for solving the problem]

[0007] In order to solve the above problem, the image processing device of the present invention is characterized in having an acquisition means for acquiring image data obtained by imaging an object illuminated with light from multiple illuminations, a setting means for setting a first area in an image represented by the acquired image data, which has a low reliability for deriving normal information of the object, a first derivation means for deriving weights for deriving the normal information of the object so that weights change continuously between the first area and a second area different from the first area, and a second derivation means for deriving the normal information of the object based on the derived weights. Effect of the Invention

[0008] According to the present invention, even if a shadow or specular reflected light appears in a captured image, normal information of an object can be derived with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a configuration of an image processing device; [Diagram 2] System configuration diagram [Diagram 3] 1 is a flowchart showing a process executed by an image processing device; [Figure 4] A flowchart showing a process for setting a low-confidence region. [Diagram 5] 1 is a flowchart showing a process for deriving weights. [Figure 6] Flowchart showing a process for deriving normal information [Figure 7] 1 is a flowchart showing a process for deriving weights. [Figure 8] 1 is a flowchart showing a process for deriving weights. [Figure 9] Flowchart showing the process of modifying weights [Figure 10] FIG. 1 is a schematic diagram showing an example of a user interface; [Figure 11] 1 is a flowchart showing a process for deriving weights. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not necessarily limit the present invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0011] [First embodiment] <Hardware configuration of image processing device 1> FIG. 1(a) is a block diagram showing a hardware configuration of an image processing device 100. The image processing device 100 includes a CPU 101, a ROM 102, and a RAM 103. The image processing device 100 also includes a VC (video card) 104, a general-purpose I / F (interface) 105, a SATA (serial ATA) I / F 106, and a NIC (network interface card) 107. The CPU 101 executes an OS (operating system) and various programs stored in the ROM 102, a HDD (hard disk drive) 113, etc., using the RAM 103 as a work memory. The CPU 101 also controls each component via a system bus 108. Note that, in the process according to the flowchart described later, program codes stored in the ROM 102, the HDD 113, etc. are expanded in the RAM 103 and executed by the CPU 101. A display 115 is connected to the VC 104. Input devices such as a mouse 110 and a keyboard 111 are connected to the general-purpose I / F 105 via a serial bus 109. To the SATA I / F 106, a general-purpose drive 114 that reads and writes from and to an HDD 113 and various recording media is connected via a serial bus 112. The NIC 107 inputs and outputs information to and from external devices. The CPU 101 uses the HDD 113 and various recording media mounted on the general-purpose drive 114 as storage locations for various data. The CPU 101 displays a GUI (Graphical User Interface) provided by a program on a display 115, and receives inputs such as user instructions received via a mouse 110 and a keyboard 111. The display 115 may be a touch panel display having a touch panel function that detects the position of a touch made by a pointer such as a finger.

[0012] <System configuration> FIG. 2 is a schematic diagram showing the configuration of a system 200 for deriving normal information of a target object. The image processing device 100 is connected to an image capture device 201 and a lighting control device 206. The lighting control device 206 is connected to lighting devices 202, 203, 204, and 205. Although four lighting devices are used in this embodiment, the number of lighting devices is not limited and may be three or five. An object is placed within the imaging range of the image capture device 201. The object is illuminated by light from a plurality of lighting devices. The system 200 is capable of deriving normal information using a photometric stereo method.

[0013] <Method of acquiring captured image data> In the system 200, captured image data of an object is acquired in the following procedure. First, the CPU 101 of the image processing device 100 sequentially and individually turns on the lights 202 to 205 via the lighting control device 206 to irradiate the object with light. The CPU 101 acquires captured image data of the object from the imaging device 201 in synchronization with the lighting of the lights, and stores the captured image data in the HDD 113. If the number of lights is M, M captured image data are acquired. In this embodiment, M=4, so four image data are acquired. The first image data is image data obtained by capturing an image of the object with only the light 202 turned on. Similarly, the second image data is image data obtained by capturing an image of the object with only the light 203 turned on, the third image data is image data obtained by capturing an image of the object with only the light 204 turned on, and the fourth image data is image data obtained by capturing an image of the object with only the light 205 turned on.

[0014] <Functional configuration of image processing device> Fig. 1(b) is a block diagram showing the functional configuration of the image processing device 100. The CPU 101 functions as the functional configuration shown in Fig. 1(b) by using the RAM 103 as a work memory and reading and executing a program stored in the ROM 102 or the HDD 113. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the image processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101.

[0015] The image processing device 100 includes an acquisition unit 121, a setting unit 122, and a derivation unit 123. The acquisition unit 121 acquires captured image data stored in the HDD 113. The acquisition unit 121 may acquire captured image data directly from the imaging device 201. The setting unit 122 sets a low reliability region for each of the acquired captured image data. The derivation unit 123 derives normal information of an object based on the acquired captured image data and the set low reliability region.

[0016] <Processing performed by the image processing device> A flow of processing executed by the image processing device 100 in this embodiment will be described with reference to the flowchart of FIG. 3. The processing shown in the flowchart of FIG. 3 starts when a user inputs an instruction via an input device and the CPU 101 accepts the input instruction. Hereinafter, each step (process) is represented by adding S to the reference symbol. In S301, the acquisition unit 121 acquires captured image data stored in the HDD 113. The acquired captured image data is image data obtained by capturing an image of an object while sequentially turning on the lights 202 to 205. Hereinafter, the luminance value of the captured image represented by the captured image data is represented as I(x, y, m). Note that x is an index representing a pixel position in the horizontal direction in the captured image, and y is an index representing a pixel position in the vertical direction in the captured image. Also, m is an index for identifying the captured image, and 1≦m≦M. In S302, the setting unit 122 sets a low reliability area for each of the acquired captured image data. Details of the process of setting the low reliability area will be described later. In S303, the derivation unit 123 derives weights used to derive normal information based on the low reliability region. The process of deriving the weights will be described in detail later. In S304, the derivation unit 123 derives normal information of the object based on the captured image data, the low reliability region, and the position information of the lighting. The process of deriving normal information will be described in detail later.

[0017] <Low confidence area setting> If a shadow or specular reflected light is reflected on an object, it is difficult to derive normal information of the object with high accuracy. Therefore, the setting unit 122 in this embodiment detects areas in the captured image that are not suitable for use in deriving normals, such as areas where a shadow or specular reflected light is reflected. Specifically, it is assumed that the object has a roughly uniform reflectance, and it is determined whether the luminance in the captured image is within a predetermined range. For example, if the luminance of a pixel in the captured image is less than a threshold value Th1, it is determined that a shadow is reflected in the area corresponding to that pixel, and the pixel is set as a pixel constituting a low reliability area. Also, if the luminance of a pixel in the captured image is greater than a threshold value Th2, it is determined that specular reflected light is reflected in the area corresponding to that pixel, and the pixel is set as a pixel constituting a low reliability area. This setting process makes it possible to identify areas that are not suitable for deriving normal information. The output of this setting process is reliability image data representing a reliability image in which pixels corresponding to low reliability areas have a reliability of 0, and pixels corresponding to other areas have a reliability of 1.

[0018] 4 is a flowchart showing the flow of processing for setting a low-reliability region. In S401, the setting unit 122 determines whether or not processing has been completed for all acquired captured image data. If processing has been completed, the processing of S302 ends and the process proceeds to S303, otherwise the process proceeds to S402. In S402, the setting unit 122 initializes the index (x, y) of the pixel of interest in the captured image to the upper left (0, 0) of the captured image.

[0019] In S403, the setting unit 122 initializes the reliability E(x, y, m). Specifically, the setting unit 122 initializes the reliability E(x, y, m) corresponding to each of all pixels of the captured image to 1. In S404, the setting unit 122 determines whether or not the determination process for all pixels of the captured image as to whether or not they are pixels in a low reliability region has been completed. If the determination process has been completed, m is incremented and the process proceeds to S401. If the determination process has not been completed, the process proceeds to S405.

[0020] In S405, the setting unit 122 determines whether the pixel of interest is a pixel in a low-reliability region.<Th1もしくはI(x,y,m)> If any one of Th2 is satisfied, the pixel of interest is determined to be a pixel in a low reliability region. If the pixel of interest is determined to be a pixel in a low reliability region, the process proceeds to S406. Otherwise, x or y is incremented and the process proceeds to S404. In S406, the setting unit 122 sets the reliability E(x, y, m) of the pixel of interest to 0. This sets the pixel of interest to be in a low reliability region. As described above, by the process of S302, reliability image data representing a reliability image in which each pixel has a reliability of 0 or 1 is obtained.

[0021] In this embodiment, the low reliability region is determined based on the luminance of the captured image, but the low reliability region may be determined using color signal values ​​(R, G, B) as pixel values. Also, a region designated by the user via a GUI may be set as the low reliability region.

[0022] <Derivation of weights> In the process of deriving weights (S303), the derivation unit 123 derives weights based on the reliability image data. The weights are used to derive normals, and the luminance of a pixel with a larger weight contributes more to the derivation of normals. In the weight derivation process, a small weight is set for a low reliability region, and a large weight is set for other regions, while the weights between the low reliability region and other regions are smoothly connected. This allows the weights between the low reliability region and other regions to change continuously, making it possible to suppress the occurrence of artifacts in the output normal information.

[0023] 5 is a flowchart showing the flow of the process of deriving weights. In S501, the derivation unit 123 determines whether or not the process has been completed for all captured image data that has been acquired. If the process has been completed, the process of S303 ends and the process proceeds to S304, and if not, the process proceeds to S502.

[0024] In S502, the derivation unit 123 acquires the reliability image data generated in S302. The reliability image can be regarded as an image in which a large weight of 1 is set in an area with high reliability, and a small weight of 0 is set in an area with low reliability. In S503, the derivation unit 123 generates weighted image data representing a weighted image by performing a blurring process on the reliability image. Hereinafter, the weight is represented as w(x, y, m). The weight w(x, y, m) is derived using the following formula (1).

[0025]

number

[0026] Here, l, k are kernel sizes, and in this embodiment, l=k=5. K is the kernel of the convolution operation, and an average filter is used in this embodiment. Specifically, K(h, w)=1 / (l×k)=1 / 25. Note that the filter kernel K is not limited to an average filter, and for example, a Gaussian filter may be used. When the process of S503 is completed, m is incremented and the process proceeds to S501.

[0027] <Deriving normal information> 6 is a flowchart showing the flow of processing for deriving normal information. In S601, the derivation unit 123 initializes the index (x, y) of a pixel of interest in a captured image to the upper left (0, 0) of the captured image.

[0028] In S602, the derivation unit 123 acquires the illumination information L stored in the HDD 113. The illumination information L represents a matrix storing N illumination vectors as viewed from the object. Specifically, the illumination information L is expressed by Equation (2).

[0029]

number

[0030] Here, (l mx ,l my ,lmz ) represents the m-th lighting vector. Assume that the object has Lambertian reflectance properties. Let the normal vector at the object position (x,y) be n = (n x ,n y ,n z ), the estimated luminance I est is expressed by equation (3).

[0031]

number

[0032] Here, I est is a vector of length M. The mth element I est [m] is an estimate of the luminance I(x, y, m) of the mth captured image at the position (x, y).

[0033] In S603, the derivation unit 123 determines whether the derivation process of normal information for all pixels of the captured image is completed. If the derivation process is completed, the process of S304 is terminated, and if the derivation process is not completed, the process proceeds to S604. In S604, the derivation unit 123 determines the estimated luminance value I est The normal vector n is derived so that the weighted square sum of the luminance value of the captured image is minimized. The error E to be minimized is expressed by the following equation (4).

[0034]

number

[0035] Here, I obs is the brightness observed at position (x, y) and is expressed by equation (5).

[0036]

number

[0037] Moreover, W represents the weight of each luminance at the position (x, y) and is a diagonal matrix having w(x, y, m) as a diagonal component. Specifically, W is expressed by the following equation (6).

[0038]

number

[0039] The derivation unit 123 can derive a normal vector while reducing the contribution of unreliable luminance due to the influence of shadows, etc., by performing least square error estimation taking into account the weight W. By differentiating equation (4) and taking the least square solution, it can be expressed as equation (7).

[0040]

number

[0041] The derivation unit 123 can derive the normal vector at the position (x, y) by calculating the formula (7).

[0042] <Advantages of the First Embodiment> As described above, the image processing device in this embodiment sets a low reliability region for each captured image and sets a weight according to the low reliability region. Furthermore, by applying a blur filter to the reliability image, the weights of the low reliability region and other regions are spatially smoothly connected. By applying the blur filter, even if the pixel values ​​change sharply between the region where the shadow or specular reflection light is reflected and the other regions, it is possible to derive a spatially smooth weight. By deriving normal information based on the derived weight, it is possible to derive normal information with high accuracy even when the object has a shadow or specular reflection light.

[0043] [Second embodiment] In the first embodiment, the area where the shadow or specular reflection is reflected is identified and a small weight is assigned to derive normal information that is robust against the reflection of the shadow or specular reflection. Here, when an object is composed of a plurality of areas (e.g., parts, etc.) with different properties, it is desirable not to blur the weight at the boundary of the area. Therefore, in this embodiment, the boundary of the area is identified by performing edge detection processing on the captured image, and the blur filter is applied only when the shadow or specular reflection does not cross the boundary of the area. Note that the hardware configuration and functional configuration of the image processing device in this embodiment are the same as those in the first embodiment, and therefore the description will be omitted. Below, the parts that differ between this embodiment and the first embodiment will be mainly described. Note that the same configuration as in the first embodiment will be described with the same reference numerals.

[0044] <Derivation of weights> The weight derivation process of this embodiment is different from the first embodiment in that edge detection is performed on the captured image, and a blur filter is applied to the weight only when the detected edge does not overlap with the low-reliability region. FIG. 7 is a flowchart showing the flow of the weight derivation process. In S701, the derivation unit 123 determines whether or not the process has been completed for all the captured image data obtained. If the process has been completed, the process of S303 is ended and the process proceeds to S304, and if not, the process proceeds to S702.

[0045] In S702, the derivation unit 123 acquires reliability image data. In S703, the derivation unit 123 generates weighted image data representing a weighted image by performing a blurring process on the reliability image. The process of S703 is the same as the process of S503 in the first embodiment. In S704, the derivation unit 123 performs edge detection process on the captured image. The derivation unit 123 in this embodiment performs edge detection using a known Canny method, but edge detection may also be performed by applying a Sobel filter or the like and then performing threshold processing. The edge detection process obtains an edge detection result B(x,y,m). The edge detection result B(x,y,m) is B(x,y,m)=1 in the edge region and B(x,y,m)=0 in the other regions.

[0046] In S705, the derivation unit 123 updates the weight w using the edge detection result B(x, y, m). Specifically, the derivation unit 123 performs a process of replacing w(x, y, m) of the edge-detected region with E(x, y, m) for the weighted image. That is, the weight of the edge-detected region is replaced with the weight before the blur filter process. Specifically, the weight is expressed by Equation (8).

[0047]

number

[0048] When the process is completed, m is incremented and the process proceeds to S701.

[0049] <Effects of the second embodiment> As described above, the image processing device in this embodiment determines whether to blur the weight of a low reliability region depending on the presence or absence of an edge. In flat regions without edges, the occurrence of artifacts in normal information can be suppressed by smoothly connecting the weight between the low reliability region and other regions. On the other hand, in edge regions, the blurring of normals near edges can be suppressed by abruptly switching the weight between the low reliability region and other regions.

[0050] [Third embodiment] In the first embodiment, the weight of the area not suitable for deriving the normal is set to a low value such as 0 to derive the normal. On the other hand, in the algorithm shown in formula (7), it is difficult to derive the normal when the number of effective lights is small. Therefore, in this embodiment, the weight is set so that the number of non-zero elements of the weight w at each pixel position is three or more. Note that the hardware configuration and functional configuration of the image processing device in this embodiment are the same as those in the first embodiment, so the description will be omitted. Below, the parts that differ between this embodiment and the first embodiment will be mainly described. Note that the same configuration as in the first embodiment will be described with the same reference numerals.

[0051] <Derivation of weights> The weight derivation process of this embodiment differs from the first embodiment in that, when the non-zero elements of the weight w at each pixel position are less than 3, the weight w is corrected so that the non-zero elements are 3 or more. FIG. 8 is a flowchart showing the flow of the weight derivation process. In S801, the derivation unit 123 determines whether or not the process has been completed for all of the acquired captured image data. If the process has been completed, the process of S303 ends and the process proceeds to S304, and if not, the process proceeds to S802.

[0052] In S802, the derivation unit 123 acquires reliability image data. In S803, the derivation unit 123 generates weighted image data representing a weighted image by performing blurring processing on the reliability image. The process of S803 is the same as the process of S503 in the first embodiment. When the process of S803 is completed, m is incremented and the process proceeds to S801. In S804, the derivation unit 123 modifies the derived weight w(x, y, m).

[0053] 9 is a flowchart showing the flow of the process of correcting the weight. In S901, the derivation unit 123 initializes the index (x, y) of the pixel of interest in the captured image to the upper left (0, 0) of the captured image. In S902, the derivation unit 123 acquires weighted image data. In S903, the derivation unit 123 determines whether or not the process has been completed for all the acquired captured image data. If the process has been completed, the process of S303 ends and the process proceeds to S304, and if not, the process proceeds to S904.

[0054] In S904, the derivation unit 123 counts the non-zero elements of the weight w(x, y, m) at the position (x, y). In S905, the derivation unit 123 determines whether the number of counted non-zero elements is 3 or more. If it is 3 or more, it increments x or y and moves the process to S903. If not, it moves the process to S906.

[0055] In S906, the derivation unit 123 modifies the value of the weight w(x, y, m) at the position (x, y). In this embodiment, all elements of w(x, y, m) at the position (x, y) are set to 1. Note that elements may be randomly selected and weighted to 1 so that there are three or more non-zero elements, or the weight of the element closest to a region that is not a low-reliability region may be set to 1.

[0056] <Effects of the third embodiment> As described above, when the number of non-zero elements of the weights is less than three, the image processing device in this embodiment modifies the weights so that the number of non-zero elements becomes three or more. This makes it possible to derive normal information even when the number of effective lights is less than three.

[0057] [Fourth embodiment] In the first embodiment, the weights of the low-reliability region and other regions are smoothly connected by applying a blur filter to the reliability image generated according to the determination result of the low-reliability region of the captured image. When performing the processing of the first embodiment, the weights of some regions of the low-reliability region may become larger than the values ​​before the blur filter is applied due to the application of the blur filter. In particular, the weights become larger at the edge parts of the low-reliability region. As a result, information of the low-reliability region may be used for deriving a normal with a weight equal to or greater than the set weight. Therefore, in this embodiment, the weights of the low-reliability region and other regions are smoothly connected while maintaining the weights set in the low-reliability region. Note that the hardware configuration and functional configuration of the image processing device in this embodiment are the same as those in the first embodiment, so the description will be omitted. In the following, the parts different from this embodiment and the first embodiment will be mainly described. Note that the same configurations as those in the first embodiment will be described with the same reference numerals.

[0058] <Derivation of weights> In the weight derivation process in this embodiment, the weights set for the low-reliability regions are maintained, and the weights between the low-reliability regions and the other regions are spatially smoothly connected. Fig. 11 is a flowchart showing the flow of the weight derivation process.

[0059] In S1101, the derivation unit 123 determines whether or not the process has been completed for all the acquired captured image data. If the process has been completed, the process of S303 ends and the process proceeds to S304, otherwise the process proceeds to S1102. In S1102, the derivation unit 123 acquires the reliability image data generated in S302. Note that S1101 and S1102 are the same processes as S501 and S502.

[0060] In S1103, the derivation unit 123 performs erosion processing on the reliability image. The reliability image is an image in which a large weight of 1 is set for areas with high reliability and a small weight of 0 is set for areas with low reliability. The erosion processing is performed by applying a predetermined kernel for erosion processing to all pixels of the image. The value of the pixel of interest in the center of the kernel becomes the minimum value of the pixels in the kernel. This causes areas with high reliability (areas with pixel value 1) to shrink and areas with low reliability (areas with pixel value 0) to expand. In this embodiment, the kernel size for erosion processing is the same as the kernel size for blurring processing in S1104.

[0061] In S1104, the derivation unit 123 executes blurring processing on the reliability image that has been subjected to the contraction processing and obtained in S1103. As a result, the derivation unit 123 acquires weighted image data. Note that S1104 is the same processing as S503.

[0062] <Effects of the Fourth Embodiment> As described above, the image processing device in this embodiment expands the low reliability region before the blurring process by contraction process on the reliability image. As a result, when the blurring process kernel is applied to the edge of the low reliability region before the contraction process, the pixel value in the kernel is 0, so the increase in weight is suppressed. However, if the expansion of the low reliability region is too large, the information of the non-low reliability region cannot be utilized. In this embodiment, the kernel size of the contraction process is set to the same as the kernel size of the blurring process, so that the increase in the weight of the low reliability region is suppressed and the information of the other regions outside it can be used as much as possible.

[0063] It is possible to maintain the weight set for the low-reliability region by methods other than those described above. For example, the degree of contraction in the contraction process may be controlled by the number of times the contraction process is performed instead of the kernel size. Alternatively, the low-reliability region may be masked so that the value does not change in S503 of the first embodiment. Alternatively, the original value of the low-reliability region may be returned to 0 after S503 of the first embodiment.

[0064] In the reliability image of the above embodiment, the area with high reliability is the area with pixel value 1, and the area with low reliability is the area with pixel value 0, but the reverse may be true. That is, the area with high reliability is the area with pixel value 0, and the area with low reliability is the area with pixel value 1. In this case, when generating a weighted image, the pixel values ​​of the reliability image are inverted, so that a high weight can be set for the area with high reliability and a low weight can be set for the area with low reliability. In this case, instead of the contraction process in the fourth embodiment, a dilation process is performed on the reliability image. The value of the pixel of interest in the center of the kernel is the maximum value of the pixels in the kernel. This allows the low reliability area to be expanded before the blurring process, and the increase in the weight of the low reliability area can be suppressed.

[0065] [Other embodiments] The image processing device 100 of the above-described embodiment may further include a display control unit that displays a GUI on the display 115 that allows the user to confirm the detected low-reliability region. FIG. 10 is a schematic diagram showing an example of a GUI for the user to confirm the low-reliability region. In a window 1001, the user can specify the detection result and detection parameters of the low-reliability region. In an image display area 1002, an imaged object and a detected low-reliability region (gray region) are displayed. Through a pixel value upper limit input box 1003 and a pixel value lower limit input box 1004, the user can specify a threshold value for the low-reliability region. A numerical value inputted into the pixel value upper limit input box 1003 corresponds to Th1, and a numerical value inputted into the pixel value lower limit input box 1004 corresponds to Th2. The low-reliability region in the image display area 1002 is updated according to the specified threshold value. Through a blur amount 1005, the user can specify the kernel width of the blur filter. By pressing a detection rule confirmation button 1006, the user can confirm the input parameters.

[0066] Further, in the setting unit 122 in the above-described embodiment, when any of I(x, y, m) < Th1 or I(x, y, m) > Th2 is satisfied, the target pixel is determined to be a pixel in the low-reliability region. However, the determination may be made using only one threshold value. For example, it may be determined only whether I(x, y, m) < Th1 is satisfied, or it may be determined only whether I(x, y, m) > Th2 is satisfied.

[0067] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Signs

[0068] 100 Image processing apparatus 121 Acquisition unit 122 Setting unit 123 Derivation unit

Claims

1. An acquisition means for acquiring image data obtained by imaging an object illuminated by light from multiple sources, Setting means for setting a first region with low reliability for deriving normal information of the object in the image represented by the acquired image data, A first derivation means for deriving weights for deriving normal information of the object such that the weights change continuously between the first region and a second region different from the first region, A second derivation means for deriving the normal information of the object based on the weight derived above, It has, The first derivation means is characterized by deriving the weight by a process of expanding the first region and a blurring process.

2. The image processing apparatus according to claim 1, characterized in that the first derivation means derives the weights such that the weight corresponding to the first region is smaller than the weight corresponding to the second region.

3. The image processing apparatus according to claim 1, characterized in that the setting means sets the pixel of interest in the image to the first region when the pixel value of the pixel of interest in the image is smaller than a first threshold.

4. The image processing apparatus according to claim 1, characterized in that the setting means sets the pixel of interest in the image to the first region when the pixel value of the pixel of interest in the image is greater than a second threshold.

5. The image processing apparatus according to claim 1, characterized in that the setting means sets the pixel of interest in the image to the first region when the pixel value of the pixel of interest in the image is less than a first threshold or greater than a second threshold.

6. The system further includes detection means for performing edge detection in the image represented by the acquired image data, The image processing apparatus according to claim 1, characterized in that the first derivation means derives the weight based on the detected edge.

7. The image processing apparatus according to claim 1, characterized in that the plurality of lights are three or more lights.

8. The image processing apparatus according to claim 1, further comprising a display control means for displaying a user interface for a user to confirm the first region on a display means.

9. A program for causing a computer to function as an image processing device according to any one of claims 1 to 8.

10. Image data obtained by photographing an object illuminated by light from multiple sources is acquired. In the image represented by the acquired image data, a first region with low reliability for deriving the normal information of the object is set. Weights for deriving the normal information of the object are derived such that the weights change continuously between the first region and a second region different from the first region. Based on the weights derived above, the normal information of the object is derived, The image processing method is characterized in that the derived weight is derived by a process of expanding the first region and a blurring process.

Images