Image processing device, image processing method, and program

Abstract

To provide processing for preventing a reduction in accuracy due to lack of angular resolution in calculation of deformation / reflection properties.SOLUTION: An image processing apparatus has: acquisition means that acquires a plurality of picked-up images of an object picked up under a plurality of different conditions; first calculation means that, based on the picked-up images, calculates the deformation / reflection properties of the object corresponding to the positions of pixels in the picked-up images with first spatial resolution; determination means that determines whether the angular resolution of the deformation / reflection properties calculated by the first calculation means is smaller than a first threshold; and second calculation means that, when the angular resolution is smaller than the first threshold, based on the deformation / reflection properties calculated by the first calculation means, calculates the deformation / reflection properties of the object with second spatial resolution lower than the first spatial resolution.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for calculating the reflection characteristics of an object. [Background technology]

[0002] In the field of computer graphics (CG) that express the texture of the material or coating of an object, measurement data of the bidirectional reflectance function BRDF, which represents the variable reflectance characteristics that change depending on the lighting direction and the observation direction, or SVBRDF, which represents the two-dimensional variable reflectance characteristics, is used. BRDF is a bidirectional reflectance distribution function. SVBRDF is a spatially varying bidirectional reflectance distribution function. A method for measuring BRDF, which is one of the variable reflectance characteristics, is described in Patent Document 1. Patent Document 1 derives the geometric conditions for the next measurement based on already measured data in a form in which geometric conditions are successively changed and reflected light from an object under those geometric conditions is received (measured). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-174464 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in general, since the geometric conditions are controlled by changing the position of the light receiver or the light source, there is an upper limit to the angular resolution based on the hardware configuration, and therefore, the desired variable angle reflection characteristics may not be obtained.

[0005] Therefore, an object of the present invention is to provide a process for suppressing a decrease in accuracy due to insufficient angular resolution in the calculation of reflection characteristics. [Means for solving the problem]

[0006] The image processing device of the present invention includes an acquisition means for acquiring a plurality of captured images of an object captured under a plurality of different conditions, a first calculation means for calculating a reflection characteristic of the object corresponding to each pixel position of the captured images at a first spatial resolution based on the captured images, and The reflection characteristic calculated by the first calculation means and a second calculation means for calculating the reflection characteristics of the object at a second spatial resolution lower than the first spatial resolution based on the reflection characteristics calculated by the first calculation means when the angular resolution is smaller than a first threshold value. Effect of the Invention

[0007] According to the present invention, it is possible to suppress a decrease in accuracy in calculating reflection characteristics due to insufficient angular resolution. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing device. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an image processing device; [Diagram 3] 4 is a flowchart showing a process executed by the image processing device. [Figure 4] FIG. 2 is a schematic diagram showing imaging geometric conditions by an acquisition device. [Diagram 5] FIG. 11 is a diagram showing an example of data indicating variable reflection characteristics. [Figure 6] 11A and 11B are schematic diagrams for explaining differences in projection angles according to coordinates. [Figure 7] FIG. 13 is a diagram for explaining an example of insufficient angular resolution. [Figure 8] FIG. 11 is a diagram for explaining data output after going through a series of processes. [Figure 9] 4 is a flowchart showing a process executed by the image processing device. [Figure 10] FIG. 13 is a diagram for explaining a calculated evaluation value. [Figure 11] 11 is a diagram for explaining the relationship between a calculated evaluation value and angular resolution. FIG. [Figure 12] FIG. 13 is a diagram showing an example of measurement points that differ depending on coordinates. [Figure 13] FIG. 13 is a diagram showing an example of a generated peak image. [Figure 14] FIG. 11 is a diagram illustrating an example of a displayed UI. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the components described in this embodiment are merely examples, and are not intended to limit the scope of the present invention. In addition, not all combinations of components described in the embodiment are necessarily essential to the means for solving the problem, and various modifications and changes are possible. In each drawing, the same parts or elements are given the same numbers, and duplicated descriptions are omitted.

[0010] <<Embodiment 1>> In this embodiment, a form in which the spatial resolution and the angular resolution are adaptively controlled to derive the variable-angle reflection characteristics at each coordinate will be described.

[0011] <Hardware configuration of image processing device> 1 is a block diagram showing an example of the hardware configuration of an image processing device 1 in this embodiment. The image processing device 1 is configured by, for example, a computer. The image processing device 1 has a CPU 101, a ROM 102, a RAM 103, a general-purpose I / F (interface) 104, a SATA (Serial ATA) I / F 105, and a VC (video card) 106. These are connected to a system bus 107 that enables data to be transmitted and received between them.

[0012] CPU 101 executes an OS (operating system) and various programs stored in ROM 102, HDD (hard disk drive) 17, etc., using RAM 103 as a work memory. CPU 101 also controls each component via system bus 107. CPU 101 loads programs stored in ROM 102, HDD 17, etc., into RAM 103, and executes the programs loaded into RAM 103, thereby performing processing according to a flowchart described below.

[0013] The general-purpose I / F 104 is a serial bus interface such as USB, and is connected to an input device 13 such as a mouse or keyboard via a serial bus 12. The SATA I / F 105 is a serial bus interface, and is connected to a general-purpose drive 18 that reads and writes from and to an HDD 17 and various recording media via a serial bus 16. The CPU 101 uses the HDD 17 and various recording media mounted on the general-purpose drive 18 as storage locations for various data.

[0014] The VC 106 is a video interface and is connected to the display 15. The CPU 101 displays a UI (user interface) provided by a program on the display 15, and receives inputs such as user instructions received via the input device 13.

[0015] <Functional configuration of image processing device> Fig. 2 is a block diagram showing an example of a functional configuration of the image processing device 1. The CPU 101 functions as the functional components shown in Fig. 2 by using the RAM 103 as a work memory and reading and executing a program stored in the ROM 102 or the HDD 17. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the image processing device 1 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101.

[0016] The image processing device 1 has an imaging data acquisition unit 201, a first variable angle reflection characteristic calculation unit 202, a variable angle reflection characteristic determination unit 203, a second variable angle reflection characteristic calculation unit 204, a variable angle reflection characteristic parameter derivation unit 205, a data storage unit 206, a UI display unit 207, and a characteristic value output unit 208.

[0017] The imaging data acquisition unit 201 acquires imaging data (multiple captured images and imaging geometric conditions) instructed via the general-purpose I / F 104. The first variable-angle reflection characteristic calculation unit 202 calculates the reflection intensity under multiple different angular conditions for each coordinate position of a sample (image capture target) based on the imaging data as measurement points of the variable-angle reflection characteristic. The variable-angle reflection characteristic determination unit 203 determines whether or not a sufficient number of measurement points are obtained in the variable-angle reflection characteristic to accurately reproduce specular reflection.

[0018] The second variable angle reflection characteristic calculation unit 204 calculates new variable angle reflection characteristics by combining the variable angle reflection characteristics of neighboring pixels at least for the variable angle reflection characteristics of the pixel determined to be negative by the variable angle reflection characteristic determination unit 203. The variable angle reflection characteristic parameter derivation unit 205 derives variable angle reflection characteristic parameters for approximating the variable angle reflection characteristics with a predetermined function.

[0019] The data storage unit 206 is composed of various recording media mounted on the HDD 17 or the general-purpose drive 18, and stores captured images or information such as the variable-angle reflection characteristics calculated by the first and second variable-angle reflection characteristic calculation units 202 and 204. The UI display unit 207 displays, on the display 15, information required for acquiring captured image data and a UI that prompts the user to input information such as executing processing. The characteristic value output unit 208 outputs the variable-angle reflection characteristic parameters in a specified format. Detailed control operations in each unit will be described later.

[0020] <Operation of image processing device> FIG. 3 is a flowchart showing an image processing method of the image processing device 1 in this embodiment. Hereinafter, the details of the processing contents of the image processing device 1 in this embodiment will be described with reference to FIG. 3. Note that the CPU 101 loads a program stored in the ROM 102 into the RAM 103 and executes the program loaded into the RAM 103, thereby performing the processing shown in the flowchart in FIG. 3. Also, the processing contents shown in the flowchart in FIG. 3 are started when the user operates the input device 13 to input an instruction, and the CPU 101 accepts the input instruction. Note that the symbol "S" in the explanation of the flowchart represents a step. This also applies to the explanation of the flowcharts below.

[0021] In S301, the imaging data acquisition unit 201 acquires from the data storage unit 206 a plurality of pieces of imaging data of an object (sample) imaged under a plurality of different conditions. Specifically, the imaging data acquisition unit 201 acquires a plurality of pieces of imaging data imaged at a plurality of different projection angles from a plurality of light sources onto the object. The imaging data includes a plurality of captured images, and information representing a light source, a sample, and an illumination position for deriving the projection angle of the light source corresponding to each pixel of each captured image. The imaging data is recorded in advance in the data storage unit 206. The imaging data is specified by the user via the input device 13.

[0022] With reference to FIG. 4, an example of a method for acquiring a group of captured images recorded in the data storage unit 206 will be described. FIG. 4 is a diagram showing a schematic configuration example of the acquisition device 4 for acquiring captured images. First, the configuration of the acquisition device 4 will be described. In the illumination device 410, a plurality of light sources 412 are arranged at regular intervals on a control board 411. Each light source 412 can be individually controlled to be turned on. The light source 412 is a non-telecentric optical system, and has different directional characteristics depending on the direction. For example, the light source 412 is an LED (Light Emitting Diode) with a half-value angle of 60 degrees, and 9×9 LEDs 412 are arranged on the control board 411. The axis 413 is perpendicular to the arrangement surface of the LEDs 412 of the illumination device 410 and passes through the center of the same surface. Each of the LEDs 412 has a central axis (brightest part) in a direction perpendicular to the arrangement surface of the LEDs 412. The sample 420 represents a sample placed on a sample stage 430. The axis 421 is perpendicular to the surface of the sample 420 and passes through the center of the surface of the sample 420. The imaging device 440 has a one-band monochrome camera having a CCD image sensor and a telecentric optical system consisting of a telecentric lens with a tilt mechanism, and generates an image of the sample 420 by photoelectric conversion. As a result, each pixel of the CCD image sensor receives only light parallel to the optical axis 441 of the imaging device 440. Note that the axis 413, the axis 421, and the axis 441 are on the same plane. The postures of the imaging device 440 and the illumination device 410 are set so that the angle θ0 between the axis 413 and the axis 421 and the angle θ2 between the axis 421 and the axis 441 are equal. Here, the angles θ0 and θ2 are both 45 degrees, and the axis 441 passes through the center pixel of the CCD image sensor of the imaging device 440. For ease of explanation, the following describes the operation of the acquisition device 4 on a two-dimensional plane and the processing of the captured image acquired by the acquisition device 4. In addition, each pixel value of the captured image acquired by the imaging device 440 has linearity with the radiance value of light received by the CCD image sensor.

[0023] Next, a procedure for acquiring a captured image using the acquisition device 4 will be described. The lighting device 410 turns on one of the light sources 412. The image capture device 440 captures the sample 420 illuminated by the turned-on light source 412 to generate a captured image. Thereafter, the lighting device 410 turns off the light source 412. The acquisition device 4 sequentially executes this series of control processes by changing the turned-on light source 412. As a result, the image capture device 440 records the reflected light distribution from the sample 420 as a captured image for each of a plurality of different projection angles θ1. The recorded image is a grayscale image in tif format, and each pixel value is a value (0 to 65535) normalized to 16 bits. The captured images are all images of the same region of the sample 420, and the pixel values ​​at the same coordinates can be considered as pixel values ​​obtained by receiving reflected light at the same position of the sample 420. In this embodiment, the acquisition device 4 acquires a group of captured images of the object captured by the above-mentioned procedure, and a group of captured images of a standard reflector captured by the same procedure as the object. The sample 420 is the object or the standard reflector. The captured images of the standard reflector are used as calibration data for correcting the difference in the distance between the light source 412 and the subject, or the difference in the illuminance incident on the object caused by the orientation characteristics of the light source 412, etc. The standard reflector is assumed to have known variable reflection characteristics, and in this embodiment, for the sake of simplicity, it is treated as a perfect diffuse reflector having constant reflection characteristics regardless of the direction.

[0024] The imaging data acquisition unit 201 acquires the above captured image. In order to calculate each projection angle θ1, the imaging data acquisition unit 201 also acquires a distance d1 from the axis 413 to the light source 412, a distance d2 from the light source 412 to the sample 420 on the axis 413, an angle θ2 formed by the axis 421 and the axis 441 as a light receiving angle, and the resolution of the imaging device 440. Here, an identification number i is assigned to each light source 412 in advance. The captured image obtained by turning on the light source i and the distance d1 indicating the position of the light source i can be associated with each other by the identification number i.

[0025] In S302, the first variable angle reflection characteristic calculation unit 202 calculates the variable angle reflection characteristic of the object corresponding to each pixel position of the captured image at the first spatial resolution based on the above-mentioned imaging data. The measurement point 501 of the variable angle reflection characteristic 50 of the object to be calculated will be described with reference to Figs. 5(a) and (b). The horizontal axis of the variable angle reflection characteristic 50 represents the projection angle θ1. The vertical axis of the variable angle reflection characteristic 50 represents the reflection intensity of the object corrected by the standard reflector. The variable angle reflection characteristic 50 has the measurement point 501 that can be plotted. The first variable angle reflection characteristic calculation unit 202 calculates the measurement point 501 of the variable angle reflection characteristic 50 for each pixel 511 of the imaging data 51 acquired in S301. The calculation method of the projection angle θ1 will be specifically described with reference to Fig. 6. Fig. 6 shows a part of the configuration of the acquisition device 4 shown in Fig. 4, and the same elements are assigned the same symbols. The regions 420a, 420b, and 420c are regions obtained by dividing the object (sample) 420 based on the resolution of the imaging device 440. Each of the regions 420a, 420b, and 420c corresponds to one pixel in the captured image. For example, the length d s (mm) can be uniformly calculated using formula (1).

[0026]

number

[0027] Here, DPI is the spatial resolution of the image capture device 440, and indicates the number of pixels (dots per inch) within a width of 1 inch. b indicates a projection angle θ1 from the light source 412a to the area 420b. c indicates a projection angle θ1 from the light source 412a to the region 420c. In this manner, the projection angle θ1 is different for each of the regions 420b and 420c. b and θ c The projection angle θ1 can be calculated using formula (2).

[0028]

number

[0029] where d s is the length (mm) of one pixel of the captured image described above. k indicates the number of pixels from the pixel on the axis 413 (the central pixel of the captured image) to the target pixel. According to the formula (2), for example, when d2 is 30 mm and d1 is 0 mm (light source 412 on the axis 413 is turned on), the projection angle θ1 to the area 420a on the axis 413 is 45.00 degrees, and the projection angle θ1 to the adjacent area 420b is 45.11 degrees. The first variable angle reflection characteristic calculation unit 202 can calculate the projection angle θ1 according to the pixel position. Note that when the pitch of the light source 412 is 1 mm and the light source 412 adjacent to the area 420a is irradiated, the projection angle θ1 is calculated to be 46.91 according to the formula (2). As described above, when taking into consideration the size of a typical light source 412 and the resolution of the imaging device 440, the difference in projection angle θ1 between adjacent pixels is extremely small compared to the difference in projection angle θ1 between adjacent light source positions.

[0030] Next, a method for calculating the reflection intensity will be specifically described. i The pixel value at the coordinate position (x, y) in i (x, y), and similarly, the image R of the perfect diffuse reflector obtained by turning on the light source i. i The pixel value at the coordinate position (x, y) in R i (x, y). In this case, the reflection intensity I i (x, y) is calculated using equation (3).

[0031]

number

[0032] Here, c is a fixed value. If the standard reflector is different from a perfect diffuser and has different characteristics depending on the angle, the inverse of the reflectance for each angle is multiplied instead of c. If linearity is not obtained for each captured pixel value relative to the luminance value, equation (3) is applied using pixel values ​​that have been subjected to gamma correction or conversion with reference to a one-dimensional LUT (lookup table). Through the above processing, the first variable angle reflection characteristic calculation unit 202 calculates the projection angle θ1 and reflection intensity I corresponding to each pixel in each captured image. i (x, y) is acquired. As a result, as shown in Fig. 5(a), the first variable-angle reflection characteristic calculation section 202 calculates a measurement point 501 of the variable-angle reflection characteristic 50 at an arbitrary pixel position.

[0033] In S303, the variable angle reflection characteristic determination unit 203 determines whether the angular resolution of the variable angle reflection characteristic 50 calculated in S302 is smaller than a threshold value. Specifically, the variable angle reflection characteristic determination unit 203 counts the number of valid data of the variable angle reflection characteristic 50 calculated in S302, and determines whether the number of valid data of the variable angle reflection characteristic 50 is equal to or larger than the threshold value.

[0034] The number of valid data will be described with reference to Figures 7(a) to (d). Graph 70 is a curve showing true gonioreflectance characteristics 701 of an area corresponding to a pixel of an object. In general, the gonioreflectance characteristics 701 have a peak of reflection intensity detected near a projection angle θ1 (θ1 = θ2) equal to a receiving angle θ2. This peak value represents the brightness of the illumination image of the object. The half-width of the curve of gonioreflectance characteristics 701 represents the clarity of the illumination image reflected on the object. Here, a sample with a clear reflected illumination image is referred to as a sample with high image clarity.

[0035] The measurement data of the variable reflectance characteristic corresponds to measurement points (plot points) 711 shown in graph 71, and the number is the same as the number of lit light sources 412. The variable reflectance characteristic 712 is a function curve approximated based on the measurement points 711 of the variable reflectance characteristic, and is defined by the variable reflectance characteristic parameter derivation unit 205 described later. Here, the vicinity of the peak of the variable reflectance characteristic 712 is called the specular reflection component, and when many measurement points 711 are obtained for the specular reflection component, the true variable reflectance characteristic 701 can be functionally approximated with high accuracy.

[0036] Graph 72 is a curve showing true variable reflectance characteristics 721 of an area corresponding to a pixel of an object. Variable reflectance characteristics 721 are variable reflectance characteristics with high image clarity and small half-width. In graph 73, variable reflectance characteristics 732 are a function curve approximated based on measurement points 731 of the variable reflectance characteristics. As shown in graph 73, a sufficient number of measurement points 731 cannot be obtained for the specular reflection component, and the approximation accuracy by the function is greatly reduced.

[0037] The number of valid data points mentioned above refers to the number of measurement points in the specular reflection component. To detect the number of valid data points, in S303, the variable reflection characteristic determination unit 203 first detects the maximum and minimum values ​​of the variable reflection characteristic 50 at each pixel position, and calculates the average value of the detected maximum and minimum values. Next, the variable reflection characteristic determination unit 203 determines whether the number of measurement points (valid data) having a reflection intensity equal to or greater than the average value is equal to or greater than a threshold value. Here, the measurement points having a reflection intensity equal to or greater than the average value are regarded as measurement points of the specular reflection component.

[0038] The variable angle reflection characteristic determination unit 203 determines that the angular resolution of the variable angle reflection characteristic is equal to or greater than the first threshold when the number of data (measurement points) of the variable angle reflection characteristic whose reflection intensity is equal to or greater than the second threshold is equal to or greater than the third threshold. Also, the variable angle reflection characteristic determination unit 203 determines that the angular resolution of the variable angle reflection characteristic is smaller than the first threshold when the number of data (measurement points) of the variable angle reflection characteristic whose reflection intensity is equal to or greater than the second threshold is less than the third threshold.

[0039] As described above, the number of measurement points is determined based on the number of captured images, i.e., the number of light sources, and the interval of the light-receiving angle θ2 (angular resolution) is determined by the pitch of the light sources 412 and the distance between the light sources 412 and the object. One cause of the lack of effective data at the measurement points 731 as shown in graph 73 is insufficient angular resolution, and hereinafter, the lack of effective data is synonymous with the lack of angular resolution.

[0040] If the number of valid data of the variable angle reflection characteristic 50 calculated in S302 is equal to or greater than the threshold, the variable angle reflection characteristic determination unit 203 determines that the angular resolution of the variable angle reflection characteristic 50 is equal to or greater than the threshold, and proceeds to S305. If the number of valid data of the variable angle reflection characteristic 50 calculated in S302 is less than the threshold, the variable angle reflection characteristic determination unit 203 determines that the angular resolution of the variable angle reflection characteristic 50 is smaller than the threshold, and proceeds to S304.

[0041] In S305, the variable-angle reflection characteristic parameter derivation unit 205 calculates a function f that approximates the measurement points of the variable-angle reflection characteristic calculated by the first variable-angle reflection characteristic calculation unit 202. x,y The parameters of (θ1) are derived and the process proceeds to S306. x,y (θ1) is defined by equation (4).

[0042]

number

[0043] Function f x,y (θ1) is the coefficient a x,y Variance σ with weighting coefficient x,y and the average m x,y is the normal distribution function of D. D indicates the offset amount representing the diffuse reflection component. The function f x,y The parameters of (θ1) are coefficient a x,y and variance σ x,y and the average m x,y The function f is derived by performing an optimization process using the known gradient descent method so that the cumulative square error between the measurement points of the variable reflection characteristics and the estimated value by equation (4) is minimized. x,yThe curve of (θ1) corresponds to, for example, the variable-angle reflection characteristic 701 or 721 in FIG. 7(a) or (c) described above. The function f x,y The parameters of (θ1) correspond to the parameters of the variable reflection characteristic 701 or 721. Note that the function f x,y For (θ1), other than the normal distribution function, a known model formula such as the Cook-Torrance model may be used.

[0044] In S304, the second variable angle reflectance characteristic calculation unit 204 combines the measurement points of the variable angle reflectance characteristic of adjacent pixel positions calculated by the first variable angle reflectance characteristic calculation unit 202, and calculates the measurement point of the variable angle reflectance characteristic of the object with the second spatial resolution. The second spatial resolution is lower than the first spatial resolution of S302. For example, the second variable angle reflectance characteristic calculation unit 204 combines the variable angle reflectance characteristics of adjacent 2×2 pixels, and calculates a new measurement point of the variable angle reflectance characteristic. This increases the number of measurement points of the variable angle reflectance characteristic by four times. As described with reference to FIG. 6, each measurement point has a high angular resolution because the projection angle θ1 is different. Therefore, the possibility that the true variable angle reflectance characteristic of the object (particularly the specular reflection component) can be more accurately approximated by the function curve derived by the variable angle reflectance characteristic parameter derivation unit 205 described later is improved. The measurement points of the variable angle reflection characteristics calculated by the second variable angle reflection characteristics calculation unit 204 have a spatial resolution with respect to the object surface that is reduced to 1 / 2 compared to the measurement points of the variable angle reflection characteristics calculated by the first variable angle reflection characteristics calculation unit 202. Then, the process proceeds to S305.

[0045] In S305, the variable-angle reflection characteristic parameter derivation unit 205 calculates a function f that approximates the measurement points of the variable-angle reflection characteristic calculated by the second variable-angle reflection characteristic calculation unit 204, as shown in the above formula (4). x,y Derive the parameters of (θ1). Function f x,y The (θ1) parameter corresponds to the parameter of the gonioreflection characteristic 701 or 721. Then, the process proceeds to S306.

[0046] In S306, the characteristic value output unit 208 calculates the above function f x,y(θ1). With reference to Figs. 8(a) and (b), the function f x,y The format of the parameter 81 of (θ1) is explained. The function f x,y The parameters 81 of (θ1) are the coordinate positions x and y of the object and the function f x,y The parameters of (θ1) are associated with the function f x,y The parameters of (θ1) are coefficient a x,y and variance σ x,y and the average m x,y It is.

[0047] Function f x,y The parameter 82 of (θ1) is the coefficient a x,y and variance σ x,y and the average m x,y Hereinafter, this image data will be referred to as a parameter image. For example, the characteristic value output unit 208 may use a function f x,y Save the parameter 81 of (θ1) in CSV format and use the function f x,y The parameters 82 of (θ1) are saved as a grayscale image in tif format. Here, the tif image is a x,y and variance σ x,y and the average m x,y Three images corresponding to the three points are generated. The parameters stored in each image are normalized for each image so that the maximum value is 255 and the minimum value is 0.

[0048] If it is determined in S303 that the number of valid data is less than the threshold, the resolution of the parameter image is set to half that of the captured image acquired by the captured data acquisition unit 201, and the number of pixels of the parameter image is set to half that of the captured image both vertically and horizontally. If it is determined that the number of valid data is equal to or greater than the threshold, the resolution and number of pixels of the parameter image are the same as those of the captured image acquired by the captured data acquisition unit 201. The characteristic value output unit 208 outputs a function f in CSV format. x,y Parameter 81 of (θ1) and function f in tif format x,y The parameter 82 of (θ1) is stored in the data storage unit 206. The UI display unit 207 displays the function fx,y An image of the parameter 82 (θ1) is displayed and the completion of processing is notified at the same time.

[0049] As described above, the image processing device 1 of this embodiment determines whether the angular resolution is insufficient by calculating the number of valid data at each coordinate position. In addition, in order to increase the number of valid data, the image processing device 1 lowers the spatial resolution and calculates the measurement points of the gonioreflectance characteristics in the area divided by the lower spatial resolution. In this way, the image processing device 1 adaptively changes the angular resolution and can generate data that suppresses a decrease in the approximation accuracy of the specular reflection component.

[0050] <<Embodiment 2>> In the first embodiment, the image processing device 1 reduces the spatial resolution by a predetermined amount based on the number of valid data at each coordinate position, and improves the angular resolution. In this embodiment, a method of adjusting the spatial resolution based on the spatial frequency characteristics of the calculated variable angle reflection characteristics will be described. Note that the functional configuration of the image processing device 1 in this embodiment is the same as that of the first embodiment. In this embodiment, the main difference between the first embodiment and the first embodiment is the processing content by the variable angle reflection characteristic determination unit 203. Therefore, in the description of the processing operation of the image processing device 1 of this embodiment described later, the variable angle reflection characteristic determination unit 203 will be mainly described.

[0051] <Image processing device operation> 9(a) and (b) are flowcharts showing an example of the procedure of processing executed by the image processing device 1 of this embodiment. Hereinafter, the details of the processing contents of the image processing device 1 of this embodiment will be described with reference to FIG. 9(a) and (b). The CPU 101 loads a program stored in the ROM 102 into the RAM 103 and executes the program loaded into the RAM 103, thereby performing the processing shown in the flowcharts of FIG. 9(a) and (b). The processing contents shown in the flowcharts of FIG. 9(a) and (b) are started when the user operates the input device 13 to input an instruction, and the CPU 101 accepts the input instruction. In each step, the UI display unit 207 displays various processing progress data. The UI and data to be displayed will be described later.

[0052] In S901, the imaging data acquisition unit 201 acquires imaging data in the same manner as in embodiment 1. The imaging data includes a plurality of captured images and information representing a light source, a sample, and an illumination position for deriving the projection angle of the light source corresponding to each pixel of each captured image.

[0053] In S902, the first variable-angle reflection characteristic calculation unit 202 calculates the variable-angle reflection characteristic of each pixel based on the above-mentioned imaging data, similarly to the first embodiment.

[0054] In S903, the variable glide reflection characteristic parameter derivation unit 205 derives parameters of a function that approximates the variable glide reflection characteristic in the same manner as in the first embodiment.

[0055] In S904, the variable angle reflection characteristic determination unit 203 calculates the power spectrum of the light source frequency band and determines whether the angular resolution is insufficient. Details of the processing operation of S904 will be described later with reference to FIG. 9(b). If the angular resolution is insufficient, the process proceeds to S905, and if the angular resolution is not insufficient, the process proceeds to S906.

[0056] In S905, similar to Embodiment 1, the second variable-angle reflection characteristic calculation unit 204 synthesizes the variable-angle reflection characteristics of adjacent pixel positions to calculate a new variable-angle reflection characteristic. Further, the second variable-angle reflection characteristic calculation unit 204 replaces (updates) the variable-angle reflection characteristic calculated by the first variable-angle reflection characteristic calculation unit 202 in S902 with the variable-angle reflection characteristic calculated by the second variable-angle reflection characteristic calculation unit 204 in S905. After S905, the process returns to S903. Therefore, the processes of S903 to S905 are repeatedly executed as long as it is determined in S904 that the angular resolution is insufficient, and the variable-angle reflection characteristic is sequentially updated. Note that the second variable-angle reflection characteristic calculation unit 204 stores a counter value with an initial value of 0 in the data storage unit 206, and when S905 is applied, adds the counter value to record the number of times n that S905 has been applied.

[0057] In S906, the characteristic value output unit 208 generates and outputs various parameters and parameter images indicating the updated variable-angle reflection characteristic in the same manner as in Embodiment 1. The spatial resolution and the number of vertical and horizontal pixels of the parameter image are both 1 / 2 n of the captured image acquired by the imaging data acquisition unit 201. Here, n indicates the number of times the process of S905 has been executed.

[0058] <Processing operation of the variable-angle reflection characteristic determination unit 203 in S904> Details of S904 by the variable-angle reflection characteristic determination unit 203 will be described with reference to FIG. 9(b). FIG. 9(b) is a flowchart showing a detailed procedure example of the process executed in S904 shown in FIG. 9(a).

[0059] In S9041, the variable-angle reflection characteristic determination unit 203 obtains the maximum value of the function of each coordinate defined by the parameter of the function f x,y (θ1), and generates a two-dimensional map in which the maximum value is stored in the corresponding coordinate. That two-dimensional map is called a peak image. Note that the number of pixels and the spatial resolution of the peak image can be calculated in the same manner as in S906 by referring to the number of pixels and the spatial resolution of the captured image acquired in S901, and further, the number of times n of the process by S905.

[0060] In S9042, the variable angle reflection characteristic determination unit 203 performs a two-dimensional fast Fourier transform (FFT) process on the peak image to convert the peak image into frequency space and generate an FFT image. Furthermore, the variable angle reflection characteristic determination unit 203 calculates a radially averaged power spectrum (RAPS) from the FFT image. The RAPS is the average value of the power spectrum that indicates the same frequency on the FFT image. FIG. 10 shows RAPS data. The horizontal axis indicates the frequency cycle / mm, and the vertical axis indicates the average value of the power spectrum. A group of plot points 1000 on the horizontal and vertical axes, or a graph 1001 formed by the plot points, is called RAPS data.

[0061] In S9043, the variable angle reflection characteristic determination unit 203 calculates an integral value of the above-mentioned power spectrum in the spatial frequency band of the light source 412. Here, the spatial frequency band of the light source 412 refers to a band near the spatial frequency of the light source 412, which is obtained by the reciprocal of the distance between the turned-on light sources 412 in the above-mentioned acquisition device 4. For example, in the acquisition device 4 described with reference to FIG. 4, if the turned-on light sources 412 are evenly arranged at 1 mm intervals, the spatial frequency of the light source 412 is 1 cycle / mm. In this embodiment, the spatial frequency band of the light source 412 is a ±0.5 cycle / mm region centered on the spatial frequency of the light source 412. The region lb in FIG. 10 indicates the spatial frequency band of the light source 412. The shaded area 1002 in FIG. 10 indicates the integral area of ​​the power spectrum of the region lb. In S9043, the variable angle reflection characteristic determination unit 203 calculates the area of ​​the shaded area 1002. Hereinafter, the integral value of the power spectrum of the region lb is called an evaluation value.

[0062] In S9044, the variable angle reflection characteristic determination unit 203 performs a determination using the evaluation value and a predetermined threshold. If the evaluation value is equal to or greater than the threshold, the variable angle reflection characteristic determination unit 203 proceeds to S9045, and if the evaluation value is less than the threshold, the variable angle reflection characteristic determination unit 203 proceeds to S905 in Fig. 9(a). In addition, if the number of application times n exceeds a predetermined value, the variable angle reflection characteristic determination unit 203 also proceeds to S905 in Fig. 9(a).

[0063] As described above, the variable angle reflection characteristic determination unit 203 transforms the maximum value of each coordinate of the approximate function of the variable angle reflection characteristic into the frequency space, and if the integral value of the power spectrum of the first frequency band in the frequency space is equal to or greater than the fourth threshold, the process proceeds to S9044. In this case, the variable angle reflection characteristic determination unit 203 determines that the angular resolution of the variable angle reflection characteristic is equal to or greater than the first threshold. The first frequency band is a frequency band based on the spatial frequency of the light source 412.

[0064] Furthermore, when the integral value of the power spectrum of the first frequency band is less than the fourth threshold, the variable angle reflection characteristic determining unit 203 determines that the angular resolution of the variable angle reflection characteristic is smaller than the first threshold, and proceeds to S905.

[0065] In S9045, the variable glide reflection characteristic determination unit 203 controls the UI display unit 207 to display the result of S9044. At the same time, the variable glide reflection characteristic determination unit 203 displays a UI that allows an instruction input as to whether to proceed to the processing of S906 or S905, and when an instruction input is made, proceeds to S906 or S905 based on the instruction input. In this embodiment, a UI is displayed for allowing the user to instruct whether or not to output parameters, and if an instruction to output parameters is received, proceeds to S906, and if not, proceeds to S905.

[0066] The relationship between the evaluation value and the angular resolution will be described. FIG. 11 is a diagram showing a part of the acquisition device 4 shown in FIG. 4. The image sensor 1120 shows the image sensor of the imaging device 440 in FIG. 4. The axes 1131, 1132, and 1133 are perpendicular to the arrangement plane of the light sources 1121, 1122, and 1123 of the illumination device 410, respectively, and are axes passing through the light sources 1121, 1122, and 1123. The axes 1111, 1112, and 1113 are perpendicular to the plane of the image sensor 1120, respectively, and are axes intersecting at intersections xa, xb, and xc between the axes 1131, 1132, and 1133 and the plane of the sample 420. The image sensor 1120 has sensor elements ya, ya', yb, and yc. Sensor elements ya, yb, and yc are sensor elements through which axes 1111, 1112, and 1113 pass, respectively, and receive reflected light at intersections xa, xb, and xc where projection angle θ1 and reception angle θ2 are equal. Generally, in a reflecting object other than a perfect diffuse reflector, the reflection intensity in the angular direction (specular reflection direction) is maximum, so that in the variable reflectance characteristics at sensor elements ya, yb, and yc, the peak value or a value close to the peak value of the true variable reflectance characteristics can be obtained.

[0067] Graphs 1210 and 1220 in FIG. 12(a) and (b) respectively show the variable reflectance characteristics at pixel positions corresponding to the sensor elements ya and ya' in FIG. 11. Here, the surfaces of the sample 420 corresponding to the sensor elements ya and ya' have the same reflectance characteristics. Measurement points 1211 and 1221 are measurement points of the variable reflectance characteristics calculated by the first variable reflectance characteristic calculation unit 202. Variable reflectance characteristics 1212 and 1222 are the true variable reflectance characteristics of the object. Approximation functions 1213 and 1223 represent approximation functions derived from the measurement points 1211 and 1221, respectively. In FIG. 12(a) and (b), the image clarity of the object is high, and the measurement point 1221 does not obtain a measurement point that represents the specular reflection component of the variable reflectance characteristic 1222, or does not obtain a value close to the peak value of the variable reflectance characteristic 1222. Therefore, an approximation function 1223 that shows a lower peak value compared to the variable-angle reflection characteristic 1222 is derived.

[0068] 11, a measurement point 1221 calculated from the pixel value of light received by the sensor element ya receives reflected light from the intersection point xa in the specular reflection direction when the light source 1121 is turned on, and therefore a value close to the peak value of the variable angle reflection characteristic 1212 can be obtained. Therefore, an approximation function 1213 having a high peak value equivalent to that of the variable angle reflection characteristic 1212 can be derived.

[0069] As described above, when the angular resolution is low in the configuration of acquisition device 4 (for example, when the light source arrangement interval is large), it is difficult for the first variable-angle reflectance characteristic calculation unit 202 to acquire the peak value or a nearby value of the true variable-angle reflectance characteristic in some regions. Furthermore, the partial region depends on the light source arrangement, and according to the configuration of acquisition device 4 described above, the above-mentioned region with a low peak position appears with the same period as the light source arrangement period. Therefore, in the above-mentioned peak image, fluctuations in pixel values ​​are observed with the same period as the light source arrangement period, and the power spectrum in the spatial frequency band of light source 412 or the above-mentioned evaluation value shows a high value.

[0070] Graph 131 in Fig. 13(a) shows a graph representing pixel values ​​obtained by scanning the peak image in a one-dimensional direction as an example of the above. The horizontal axis of graph 131 represents the pixel position of image sensor 1120 in Fig. 11. The vertical axis of graph 131 represents the pixel value of the above peak image. Peak image 132 in Fig. 13(b) shows an example of a two-dimensional peak image. The pixel value of peak image 132 indicates the peak value of the approximation curve, and similar to graph 131, periodic unevenness according to the pixel position of image sensor 1120 can be confirmed.

[0071] 14(a) and (b) are diagrams showing the UI and data that the UI display unit 207 displays on the display 15 in each step. The UI 141 is displayed when the execution of the above series of processes starts. Areas 1401 and 1402 are areas for displaying a list of file names of the sample (object) captured image and the reference (standard reflector) captured image acquired by the image data acquisition unit 201. These files are designated on a dialog box that is displayed by dragging and dropping the file into each of the areas 1401 and 1402, or by pressing the load buttons 1403 and 1404. The numbers at the end of these file names are identification numbers corresponding to the lit light sources.

[0072] Area 1405 displays an image selected by the user from among the files displayed in area 1401 or 1402. The selection is realized by placing the cursor on an arbitrary file name and clicking it using the mouse of the input device 13. Note that various geometric conditions related to the lighting device 410 and the imaging device 440 are created in advance as setting files, and are loaded when the UI 141 is started.

[0073] The execute button 1406 is a button for instructing the execution of processing performed by the first variable-angle reflection characteristic calculation unit 202 or the characteristic value output unit 208, and when pressed, the processing is started. When the processing is completed, a parameter image showing the parameter (characteristic value) selected by the pull-down button 1408 is displayed in an area 1407. When any coordinate in the area 1407 is designated and selected, the variable-angle reflection characteristic at that coordinate is displayed in an area 1409. The designation and selection is realized by using the mouse of the input device 13 to place the cursor on any file name and click it. The displayed variable-angle reflection characteristic indicates the curve indicated by the above-mentioned updated variable-angle reflection characteristic and the approximate function. The end button 1410 is a button for closing the UI 141 by pressing it after a series of processing is completed, and when pressed during processing, the processing is interrupted and ended.

[0074] If it is determined in S9044 that the angular resolution is insufficient, a dialogue 142 is displayed and the determination result is displayed in an area 1411. When the user presses a continue button 1412 or an end button 1413, the process proceeds to S904 or S905. If the dialogue 142 is displayed (when it is determined that the angular resolution is insufficient), the above peak image is displayed in an area 1407. This allows the user to qualitatively grasp the above light source periodic fluctuation (periodic noise due to insufficient resolution).

[0075] As described above, the image processing device 1 in this embodiment derives an evaluation value based on the spatial frequency characteristics of the calculated variable-angle reflection characteristics, and judges the insufficiency of angular resolution based on the evaluation value. Furthermore, the second variable-angle reflection characteristic calculation unit 204 repeats the process until the evaluation value reaches a target value. This makes it possible to generate data that suppresses the deterioration of the approximation accuracy of the specular reflection component due to insufficient angular resolution, particularly the light source periodic noise. Furthermore, the user can refer to the visualized light source periodic noise and judge whether to continue the repeated process.

[0076] <Other embodiments> In the above embodiment, a typical monochrome camera is assumed and one channel is described, but even in the case of a color camera having two or more channels with different sensitivity characteristics depending on the wavelength, the processing of the above embodiment can be applied to each channel.

[0077] In the above embodiment, when determining whether the angular resolution is insufficient, a threshold value stored in advance in the data storage unit 206 is used, but the user may input a threshold value via a UI. The threshold value may be adaptively calculated based on data of a measurement point that constitutes the variable reflection characteristic to be determined. For example, the threshold value may be set smaller so that the smaller the minimum value of the data of the measurement point, the easier it is to determine that the angular resolution is insufficient. Here, the minimum value is regarded as the diffuse reflection component. This allows dynamic threshold setting based on a prediction that the specular component is relatively small when the diffuse reflection component is large.

[0078] In the above embodiment, the spatial resolution is changed based on the determination result, but any restriction may be added to whether or not the change is applied. For example, the desired spatial resolution can be guaranteed by allowing the user to set a desired value as the lower limit of the spatial resolution.

[0079] In the above-mentioned first embodiment, when there is one or more variable angle reflection characteristics determined to have insufficient angular resolution, the variable angle reflection characteristics with low resolution are derived, but this is not limited to this. In the first embodiment, similarly to the second embodiment, variable angle reflection characteristics with successively lowered spatial resolution (resolution) may be calculated and updated until there are no more variable angle reflection characteristics determined to have insufficient angular resolution. Furthermore, the termination condition of the iterative process may be that the proportion of variable angle reflection characteristics determined to have insufficient angular resolution among the variable angle reflection characteristics to be determined is equal to or less than a predetermined threshold value.

[0080] In the above embodiment, when determining whether the angular resolution is insufficient, only one variable angle reflection characteristic to be processed is referenced, but when the processing of the second variable angle reflection characteristic calculation unit 204 is repeatedly executed, the determination may be made by comparing the variable angle reflection characteristics before and after the previous processing. For example, a function is added to count the number of inversions when plotting the variable angle reflection characteristics before and after processing on the graph in FIG. 5(a), and to end the repeated processing if the count value increases in the variable angle reflection characteristic after processing. This makes it possible to suppress the possibility of combining areas having different variable angle reflection characteristics.

[0081] In the above embodiment, the second variable angle reflection characteristic calculation unit 204 executes the process when at least one pixel is detected to have an insufficient number of valid data, but a threshold value may be set for the number of pixels. For example, the second variable angle reflection characteristic calculation unit 204 may execute the process when it is determined that the number of valid data is insufficient for pixels that account for a predetermined ratio or more of the total number of pixels.

[0082] In the above embodiment, the first variable angle reflection characteristic calculation unit 202 performs calculations from imaging data, but is not limited to this as long as values ​​in which the light-receiving angle and the reflection characteristic are associated can be acquired for a plurality of different light-receiving angles.

[0083] In the above embodiment, captured images with different light projection angles are acquired, but by changing the position of the imaging device 440, captured images with different light receiving angles may be acquired.

[0084] In the above embodiment, the peak image is displayed to allow the light source periodic noise to be perceived qualitatively, but in order to make the light source periodic noise easier to view, an image may be generated by applying a band-pass filter to the peak image and then displayed. Note that the band-pass filter may be, for example, a filter for cutting power spectrum other than the light source period.

[0085] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0086] It should be noted that the above-mentioned embodiments are merely illustrative of specific examples of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0087] 201 imaging data acquisition unit, 202 first variable-angle reflection characteristic calculation unit, 203 variable-angle reflection characteristic determination unit, 204 second variable-angle reflection characteristic calculation unit, 205 variable-angle reflection characteristic parameter derivation unit, 208 characteristic value output unit

Claims

1. An acquisition means for acquiring a plurality of captured images of an object captured under a plurality of different conditions; a first calculation means for calculating a reflection characteristic of the object corresponding to each pixel position of the captured image at a first spatial resolution based on the captured image; a second calculation means for calculating the reflection characteristic of the object at a second spatial resolution lower than the first spatial resolution based on the reflection characteristic calculated by the first calculation means when the angular resolution of the reflection characteristic calculated by the first calculation means is smaller than a first threshold value; 13. An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the acquisition unit acquires a plurality of captured images captured from a plurality of different light sources at different projection angles onto the object.

3. 3. The image processing apparatus according to claim 1, further comprising a derivation unit for deriving parameters of a function that approximates the reflection characteristic calculated by the second calculation unit.

4. 4. The image processing device according to claim 3, wherein the derivation means derives parameters of a function that approximates the reflection characteristics calculated by the second calculation means when the angular resolution is smaller than the first threshold, and derives parameters of a function that approximates the reflection characteristics calculated by the first calculation means when the angular resolution is equal to or greater than the first threshold.

5. 5. The image processing apparatus according to claim 3, further comprising an output unit for outputting parameters of the function.

6. 6. The image processing apparatus according to claim 5, wherein said output means outputs an image having the parameter values ​​of said function as pixel values.

7. 7. The image processing apparatus according to claim 3, wherein the parameters of the function include a variance and a mean of a normal distribution function.

8. Further, a determination means for determining whether the angular resolution is smaller than the first threshold value, 8. The image processing device according to claim 1, wherein the second calculation means calculates the reflection characteristics of the object at the second spatial resolution when the determination means determines that the angular resolution is smaller than the first threshold.

9. The image processing device described in claim 8, characterized in that the determination means determines that the angular resolution is greater than or equal to the first threshold when the number of data pieces in which the reflection intensity is greater than or equal to the second threshold among the reflection characteristic data under the plurality of different conditions is greater than or equal to a third threshold, and determines that the angular resolution is smaller than the first threshold when the number of data pieces in which the reflection intensity is greater than or equal to the second threshold is less than the third threshold.

10. 9. The image processing device according to claim 8, wherein the determination means transforms the maximum value of each coordinate of the approximation function of the reflection characteristics into a frequency space, and determines that the angular resolution is equal to or greater than the first threshold if an integral value of a power spectrum of a first frequency band in the frequency space is equal to or greater than a fourth threshold, and determines that the angular resolution is smaller than the first threshold if an integral value of the power spectrum of the first frequency band is less than the fourth threshold.

11. The acquisition means acquires a plurality of captured images captured at different projection angles from a plurality of light sources onto the object, The image processing apparatus according to claim 10 , wherein the first frequency band is a frequency band based on a spatial frequency of the light source.

12. An acquisition step of acquiring a plurality of captured images of an object captured under a plurality of different conditions; a first calculation step of calculating, based on the captured image, a reflection characteristic of the object corresponding to each pixel position of the captured image at a first spatial resolution; a second calculation step of calculating a reflection characteristic of the object at a second spatial resolution lower than the first spatial resolution based on the reflection characteristic calculated in the first calculation step when the angular resolution of the reflection characteristic calculated in the first calculation step is smaller than a first threshold value; 13. An image processing method comprising:

13. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images