Image processing apparatus, method for controlling image processing apparatus, and program
The image processing device generates expanded images to validate correction parameters, addressing color matching issues in varying lighting conditions, thereby preventing image quality deterioration.
Patent Information
- Application Number
- JP2023073442
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing color matching techniques fail to ensure accurate color and brightness correction when photographing a color chart in environments with insufficient lighting, leading to potential image quality deterioration due to color distortion outside the dynamic range of the color chart.
An image processing device that generates expanded images from images captured by reference and target cameras, evaluates correction parameters using these expanded images, and ensures the validity of color matching parameters even outside the dynamic range of the color chart by employing brightness profiles and correction parameters.
The solution effectively prevents image quality deterioration such as color distortion by confirming the validity of correction parameters, ensuring accurate color matching across cameras even in challenging lighting conditions.
Smart Images

Figure 2024158324000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device, a control method for an image processing device, and a program. [Background technology]
[0002] Conventionally, when a plurality of cameras are used for shooting at a shooting site, color matching is performed between the plurality of cameras, for example, by color matching software prior to the shooting. In the color matching, a camera serving as a reference for color matching (hereinafter referred to as a "reference camera") and a camera to be subjected to color matching (hereinafter referred to as a "target camera") are arranged in front of a color chart arranged in the shooting environment. Then, based on each image of the color chart acquired by shooting with the reference camera and the target camera, correction parameters are generated for matching the color and brightness of the target camera to those of the reference camera. Furthermore, the correction parameters thus generated are applied to the target camera, thereby performing color matching between the reference camera and the target camera.
[0003] At this time, if the shooting conditions of the reference camera and the target camera are made the same, the color matching performance of the correction parameters is improved. In this regard, for example, Patent Document 1 discloses a technology in which the camera setting information of the reference camera is transmitted to the target camera, and the target camera sets the camera based on the camera setting information, and then performs color matching between the cameras. The technology disclosed in Patent Document 1 can make the characteristics of the captured images of the reference camera and the target camera closer to each other by sharing the camera settings. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2012-195760 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, the technology disclosed in Patent Document 1 does not take into account cases where it is difficult to ensure sufficient brightness when photographing a color chart, for example, due to the influence of the lighting environment, the characteristics of the camera and lens, etc. In such cases, correction parameters are generated based on each photographed image of a color chart that is insufficiently bright, which causes a problem that image quality degradation such as color bending is likely to occur in color spaces outside the dynamic range of the color chart.
[0006] The present invention has been made in view of the above problems, and has an object to provide an image processing device, a control method for the image processing device, and a program that can check the validity of correction parameters used in color matching between cameras even outside the dynamic range of a color chart. [Means for solving the problem]
[0007] In order to achieve the above-mentioned object, the image processing device of the present invention is characterized in that it comprises an input means for inputting a first image and a second image obtained by a target imaging device and a reference imaging device photographing a color chart, an image generation means for generating a first extended image and a second extended image in which the brightness of the first image and the second image are extended, a correction parameter generation means for generating a first correction parameter using the first image and the second image or the first extended image and the second extended image, for matching the color and brightness of the target imaging device to the color and brightness of the reference imaging device, and a performance evaluation means for evaluating the performance of the first correction parameter using the first extended image and the second extended image. Effect of the Invention
[0008] According to the present invention, the validity of correction parameters used in color matching between cameras can be confirmed even outside the dynamic range of the color chart. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing system in a first embodiment. [Diagram 2] 2 is a block diagram showing an example of the configuration of an image processing unit of the image processing device in the first embodiment. FIG. [Diagram 3] 10 is a flowchart showing an example of a processing flow when an extended image is generated by an image generating section of an image processing section in the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of generation parameters according to shooting information in the first embodiment. [Diagram 5] FIG. 3 is a diagram showing an example of an image of a luminance profile in the first embodiment. [Figure 6] FIG. 2 is an image diagram showing an example of an extraction range of each patch included in a color chart in the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of RGB values of an image before extension in the first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of RGB values of an extended image in the first embodiment. [Figure 9] FIG. 1A is a diagram showing an example of input values and output values of an approximate equation using a three-dimensional function in the first embodiment, and FIG. 1B is a diagram showing an example of correction parameters in a 3D lookup table format in the first embodiment. [Figure 10] FIG. 4 is an image diagram showing a comparative example of gray gradation of output values in the first embodiment. [Figure 11] FIG. 4 is an image diagram showing an example of a histogram of the gradation values of an image expressed by frequency in the first embodiment. [Figure 12] FIG. 11 is a block diagram showing an example of the configuration of an image processing unit of an image processing device in a second embodiment. [Figure 13] FIG. 11 is a block diagram showing an example of the configuration of an image generating unit of an image processing unit in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exert a similar function. In addition, any configuration may be added. In addition, any two or more configurations (features) of each embodiment can be combined. In the drawings, the same reference numerals are used to denote the same or corresponding components and parts.
[0011] First Embodiment A first embodiment will be described below with reference to Figs. 1 to 11. Fig. 1 is a block diagram showing an example of the configuration of an image processing system 100. As shown in Fig. 1, the image processing system 100 has an imaging device 101 and an image processing device 102. The imaging device 101 is composed of a reference camera 101a (reference imaging device) and a target camera 101b (target imaging device). The reference camera 101a and the target camera 101b input images captured by them to the image processing device 102 via their respective video lines. When matching colors between the cameras, the reference camera 101a and the target camera 101b input images captured by them to the image processing device 102 while they are arranged in front of a color chart 103 for color matching.
[0012] Note that the reference camera 101a and the target camera 101b can input the images captured by each of them to the image processing device 102 even if they are configured without their respective video lines. For example, the images captured by the reference camera 101a and the target camera 101b may be transferred to the image processing device 102 via a recording medium such as a storable memory card, or may be wirelessly transmitted to the image processing device 102. In the first embodiment, the reference camera 101a and the target camera 101b are arranged in front of the color chart 103 to capture the color chart 103, but for example, each of the cameras may be directly facing the color chart 103 to capture the color chart 103. Also, the reference camera 101a and the target camera 101b may capture the color chart 103 from separate positions as long as the color chart 103 is included in the image to be input to the image processing device 102.
[0013] The image processing device 102 has an image processing unit 102a and a display unit 102b. An image of the color chart 103 captured by the reference camera 101a is input from the reference camera 101a to the image processing unit 102a, and an image of the color chart 103 captured by the target camera 101b is input from the target camera 101b to the image processing unit 102a. The image processing unit 102a generates a correction parameter TR (first correction parameter) for color matching based on each image captured from the reference camera 101a and the target camera 101b. Furthermore, the image processing unit 102a inputs the correction parameter TR to the target camera 101b. In the following, an image (second image) captured by the image processing unit 102a from the image of the reference camera 101a is referred to as "image R1". Moreover, an image (first image) captured by the image processing unit 102a from the image of the target camera 101b is referred to as "image T1".
[0014] The detailed operation of the image processing unit 102a will be described later. The display unit 102b displays a UI for generating the correction parameters TR in the image processing unit 102a, and images and information for confirming the effect of the correction parameters TR generated by the image processing unit 102a. Hereinafter, a user's operation on the UI of the display unit 102b is referred to as a "UI operation". The display unit 102b displays image information and other additional information input from the image processing unit 102a on a screen via a cable of a general image standard such as HDMI (registered trademark) or DVI. The image processing device 102 stores in advance a program for causing a control unit (computer) (not shown) to execute each unit and each means (a control method of the image processing device) of the image processing device 102.
[0015] The detailed operation of the image processing unit 102a of the image processing device 102 will be described below. FIG. 2 is a block diagram showing an example of the configuration of the image processing unit 102a. The image processing unit 102a has an image input unit 201, an image generation unit 202, a parameter generation unit 203, and a parameter setting unit 204. An image of the color chart 104 captured by the reference camera 101a and an image of the color chart 104 captured by the target camera 101b are input to the image input unit 201. The image input unit 201 temporarily stores the image of the reference camera 101a in a frame buffer, and generates an image R1 of RGB values by cutting out an arbitrary frame, and inputs the image R1 to the image generation unit 202 and the parameter setting unit 204.
[0016] Similarly, the image input unit 201 temporarily stores the image of the target camera 101b in a frame buffer, cuts out an arbitrary frame, and generates an image T1 of RGB values, and inputs the image T1 to the image generation unit 202 and the parameter setting unit 204. Note that the image input unit 201 may input not only video but also still images such as JPEG. The image generation unit 202 (image generation means) generates an extended image R2 (second extended image) from the image R1 and the shooting information input by the UI operation (image generation process). Furthermore, the image generation unit 202 inputs the extended image R2 to the parameter generation unit 203 and the parameter setting unit 204. Similarly, the image generation unit 202 (image generation means) generates an extended image T2 (first extended image) from the image T1 and the shooting information input by the UI operation (image generation process). Furthermore, the image generation unit 202 inputs the extended image T2 to the parameter generation unit 203 and the parameter setting unit 204.
[0017] The following describes the flow of processing when the extended images R2 and T2 are generated by the image generation unit 202. FIG. 3 is a flowchart showing an example of the flow of processing when the extended images R2 and T2 are generated by the image generation unit 202. A program based on the flowchart shown in FIG. 3 is executed by a control unit (not shown) of the image processing device 102. As a result, each process of the flowchart shown in FIG. 3 is performed by the image generation unit 202 of the image processing unit 102a in the image processing device 102 based on an instruction from a control unit (not shown). In step S300, the image generation unit 202 (input means) receives the image R1 and image T1 from the image input unit 201 (input process).
[0018] In step S301, the image generating unit 202 refers to generation parameters necessary for generating a luminance profile in step S303 described later. The reference is made based on the shooting information. The shooting information is input to the image generating unit 202 by a UI operation, that is, by a user operating on the UI (operation means) of the display unit 102b. The shooting information includes information such as the type and settings of equipment used by the reference camera 101a and the target camera 101b when shooting. Therefore, the user can input the type and settings of equipment according to the shooting environment of the reference camera 101a and the target camera 101b to the image generating unit 202 as shooting information.
[0019] Fig. 4 is a diagram showing an example of generation parameters according to shooting information. The generation parameters shown in Fig. 4 have a pattern in which the horizontal axis is the type of lighting (A to F) and the vertical axis is the exposure (-1 to +1), color gamut (1 to S), and color temperature (1 to P) for each camera / lens combination (α to ω), and are stored in the image generation unit 202. Here, for example, when shooting information including information such as the camera / lens α, lighting type A, exposure -1, color gamut 1, and color temperature 1 is input to the image generation unit 202 by a UI operation, the generation parameters referred to by the image generation unit 202 are as follows:
[0020]
number
[0021] The ideal luminance YA1 and chromaticity coordinates (Rx, Ry) (Gx, Gy) (Bx, By) (Wx, Wy) refer to the generation parameters corresponding to the camera and lens characteristics, the lighting environment, and the camera setting conditions, which are included in the shooting information input to the image generation unit 202 by the UI operation. The image generation unit 202 refers to the generation parameters for the reference camera 101a and the generation parameters for the target camera 101b. The generation parameters corresponding to the shooting information shown in FIG. 4 are unique values. The generation parameters required to generate the luminance profile are prepared in advance as shown in FIG. 4, but the values obtained when the color chart 103 is measured with a measuring instrument such as a spectroradiometer, that is, the color of the lighting at the shooting site, may be used. Note that when the prepared parameters are used, the effort of measurement can be reduced compared to when the measured values of the measuring instrument are used. Furthermore, the generation parameters required to generate the luminance profile are not limited to those shown in FIG. 4, and may be parameters to which information such as detailed gamma characteristics is added, for example.
[0022] Returning to Fig. 3, in step S302, the image generation unit 202 calculates a matrix coefficient M based on the generation parameters of the chromaticity coordinates referred to in step S301. In this case, the image generation unit 202 calculates a matrix coefficient Mr for the reference camera 101a and a matrix coefficient Mt for the target camera 101b. The matrix coefficient M is calculated using Equation 1 shown below.
[0023]
number
[0024] In addition, m1 to m9 in formula 1 can be calculated as follows.
number
[0025] In step S303, the image generating unit 202 generates a luminance profile using the generation parameters of the ideal luminance referred to in step S301 and the matrix coefficient M calculated in step S302. The luminance profile is calculated using Equation 2 shown below.
[0026]
number
[0027] In addition, RGB in the formula 2 are gradation values that the images R1 and T1 can take, and in the first embodiment, 8-bit gradation values (0 to 255) are substituted. In the formula 2, the Y value calculated with the matrix coefficient Mr for the reference camera 101a is the luminance profile for the reference camera 101a. In the formula 2, the Y value calculated with the matrix coefficient Mt for the target camera 101b is the luminance profile for the target camera 101b. FIG. 5 is a diagram showing an example of an image of a luminance profile. In FIG. 5, the luminance profile 500 is the luminance profile for the reference camera 101a, and the luminance profile 501 is the luminance profile for the target camera 101b. The luminance profiles 500 and 501 are obtained by plotting the Y value calculated in the formula 2 in the gradation direction. In the formula 2, YAR is the ideal luminance of the reference camera 101a, and YAT is the ideal luminance of the target camera 101b. As described above, the luminance profile is generated by calculation, but if the shooting environment is determined, the luminance profile may be prepared in advance as table data.
[0028] Returning to FIG. 3, in step S304, the image generating unit 202 calculates an expandable amount based on the generation parameters of the ideal brightness referred to in step S301 and the brightness profile generated in step S303. At that time, the image generating unit 202 calculates an expandable amount Rg for the reference camera 101a and an expandable amount Tg for the target camera 101b. The expandable amount Rg for the reference camera 101a is calculated from the ideal brightness YAR of the reference camera 101a and the maximum brightness Yrm in the image R1. The expandable amount Tg for the target camera 101b is calculated in the same manner from the ideal brightness YAT of the target camera 101b and the maximum brightness Ytm in the image T1.
[0029] The maximum luminance Yrm in image R1 is obtained by substituting the RGB values contained in image R1 into formula 2. Similarly, the maximum luminance Ytm in image T1 is obtained by substituting the RGB values contained in image T1 into formula 2. In detail, if the Y value for each pixel value of image R1 obtained from formula 2 is Yr[h,v] (h is the number of horizontal pixels, v is the number of vertical pixels), the maximum value of Yr[h,v] is selected as the maximum luminance Yrm in image R1. Similarly, if the Y value for each pixel value of image T1 obtained from formula 2 is Yt[h,v], the maximum value of Yt[h,v] is selected as the maximum luminance Ytm in image T1. Note that, if the luminance profile is prepared in advance as table data of a 3D lookup table (hereinafter referred to as "3DLUT"), for example, the maximum luminance Yrm, Ytm is obtained by tetrahedral interpolation (triangular pyramid interpolation) based on the RGB values of images R1, T1. In the first embodiment, the purpose is to make the color and brightness of the target camera 101b closer to those of the reference camera 101a. Therefore, the calculation formula of the expandable amount Rg for the reference camera 101a and the calculation formula of the expandable amount Tg for the target camera 101b are as follows.
[0030]
number
[0031] In step 305, the image generating unit 202 generates extended images R2 and T2 using the extendable portions Rg and Tg calculated in step 304, and inputs the generated extended images R2 and T2 to the parameter generating unit 203 and the parameter setting unit 204. At this time, the extended image R2 is generated by substituting the values of Xr[h,v], Yr[h,v], and Zr[h,v] obtained from equation 2 using the RGB values, ideal luminance YAR, and matrix coefficient Mr of the image R1, and the extendable portion Rg, into equation 3 shown below. In addition, the extended image T2 is generated by substituting the values of Xt[h,v], Yt[h,v], and Zt[h,v] obtained from equation 2 using the RGB values, ideal luminance YAT, and matrix coefficient Mt of the image T1, and the extendable portion Tg, into equation 3 shown below.
[0032]
number
[0033] The RGB values calculated by the formula 3 are clipped to 0 or more and 255 or less in the case of 8-bit gradation. Return to FIG. 2. The parameter generating unit 203 (correction parameter generating means) generates a correction parameter TR based on the pixel value of each patch of the color chart 103 included in the extended images R2 and T2 (correction parameter generating step). Furthermore, the parameter generating unit 203 inputs the generated correction parameter TR to the parameter setting unit 204. FIG. 6 is an image diagram showing an example of the extraction ranges 625 to 648 of each patch 601 to 624 included in the color chart 103. The parameter generating unit 203 calculates averaged data for each patch 601 to 624 by averaging the RGB values of the respective extraction ranges 625 to 648 for each patch 601 to 624 in FIG. 6. The averaged data is used when the parameter generating unit 203 generates the correction parameter TR. Furthermore, the rectangles of the respective extraction ranges 625 to 648 are set manually by UI operation, but may be set automatically by AI. The parameter generating unit 203 inputs information on the extraction ranges 625-648 to the parameter setting unit 204 as control information. In the first embodiment, the extraction ranges 625-648 of the patches 601-624 are set as rectangles via a UI operation, but may be set by, for example, a user specifying the center position and extraction width of each patch 601-624. Also, the center positions of the patches 601-624 may be automatically specified from the extended images R2 and T2 instead of being specified by the user. Such automatic specification can be realized by inference processing using a general object detection AI or a rule-based algorithm.
[0034] The correction parameter TR means a parameter for making the extended image T2 closer to the extended image R2. The color chart 103 has patches 601 to 624 with different colors and brightness arranged thereon. In order to obtain the correction parameter TR, the parameter generating unit 203 uses pixel values included in the extended images R2 and T2 to estimate pixel values not included in the extended images R2 and T2. If the color and brightness of the color chart 103 included in the extended image T2 is to be made closer to the color and brightness of the color chart 103 included in the extended image R2 using the correction parameter TR, the pixel value estimation is not necessarily required. However, the actual captured image also contains many pixel values outside the color chart 103. Therefore, the parameter generating unit 203 estimates pixel values not included in the extended images R2 and T2 for the color and brightness outside the color chart 103 in order to make the color and brightness of the target camera 101b closer to the color and brightness of the reference camera 101a using the correction parameter TR. The parameter generating unit 203 estimates pixel values by obtaining an approximation formula from pixel values included in the extended images R2 and T2. The parameter generating unit 203 obtains the approximation formula by determining each coefficient of the three-dimensional function of the following formula 4 based on the average data of each achromatic patch 619 to 624 included in the color chart 103.
[0035]
number
[0036] If there are multiple R, G, and B patches on the color chart 103, an approximation formula using a three-dimensional function for each of R, G, and B may be obtained. The approximation formula is not limited to the curve approximation using the above-mentioned three-dimensional function, and may be obtained by other curve approximation (two-dimensional function or multidimensional function of four or more dimensions, logarithmic approximation, etc.) or linear approximation. Here, in order to show the effect of the first embodiment, approximation formulas in the case of using images before and after expansion are compared. FIG. 7 is a diagram showing an example of RGB values of image R1 and image T1 before expansion. The coefficients of the approximation formula using a three-dimensional function using averaged data of each achromatic patch 619 to 624 shown in FIG. 7 are as follows:
[0037]
number
[0038] On the other hand, Fig. 8 is a diagram showing an example of RGB values of extended image R2 and extended image T2. The coefficients of an approximation equation by a three-dimensional function using the averaged data of each of the achromatic patches 619 to 624 shown in Fig. 8 are as follows:
[0039]
number
[0040] Fig. 9(A) is a diagram showing an example of input values and output values of each approximation equation for extended images R2 and T2. As shown in Fig. 9(A), in the case of a 3DLUT format in which the correction parameter TR has 17^3 lattice points, the RGB values of the output values corresponding to each lattice point are obtained by substituting each coefficient of each approximation equation and the RGB values of each lattice point of 17^3 into the three-dimensional function of Equation 4. Fig. 9(B) is a diagram showing an example of the correction parameter TR in the 3DLUT format. As shown in Fig. 9(B), the output value of the correction parameter TR in the 3DLUT format generated by the parameter generating unit 203 is calculated by the difference between the output value of the approximation equation for extended image R2 and the output value of the approximation equation for extended image T2.
[0041] Here, the gray gradation of the output value of the approximation formula for image T1 is compared with the gray gradation of the output value of the approximation formula for extended image T2. FIG. 10 is an image diagram in which the gray gradation of the output value is compared. In FIG. 10, 1000 is the characteristic of the gray gradation of the output value obtained by the approximation formula for extended image T2, and rises linearly. 1001 is the characteristic of the gray gradation of the output value obtained by the approximation formula for image T1, and unlike the characteristic 1000 of extended image T2, it drops in an arc from the middle of the input gradation. In this respect, the characteristic 1001 of image T1 is significantly different from the original characteristic. Therefore, if the correction parameter TR is generated using the approximation formula for image T1, color shift occurs. Therefore, it is preferable that the parameter generating unit 203 uses each approximation formula for extended images R2 and T2 when generating the correction parameter TR. Note that the correction parameter TR is in a 3DLUT format, but if the approximation formula is close to a straight line, there is no problem even if it is a matrix coefficient such as 3×3. In this regard, the approximation equation for image R1 and the approximation equation for expanded image R2 may be approximately linear, and the shape of the approximation equation may not change before and after expansion.
[0042] Return to FIG. 2. The parameter setting unit 204 inputs the correction parameter TR generated by the parameter generating unit 203 to the target camera 101b. The parameter setting unit 204 (performance evaluation means) may evaluate the performance of the correction parameter TR before inputting the correction parameter TR to the target camera 101b (performance evaluation step). In this case, the parameter setting unit 204 evaluates the performance of the correction parameter TR, for example, based on whether or not the difference in RGB values between the image T1' and the image R1 after the correction parameter TR is applied to the image T1 is within a first threshold value. Then, if the difference in RGB values between the image T1' and the image R1 after the correction parameter TR is applied to the image T1 is within a first threshold value, the parameter setting unit 204 determines that the performance evaluation is acceptable and inputs the correction parameter TR to the target camera 101b. Alternatively, the parameter setting unit 204 evaluates the performance of the correction parameter TR based on whether or not the difference in RGB values between the extended image T2' and the extended image R2 after the correction parameter TR is applied to the extended image T2 is within a first threshold value. Then, if the difference in RGB values between extended image T2' after applying the correction parameter TR to extended image T2 and extended image R2 is within a first threshold, the parameter setting unit 204 determines that the performance evaluation is pass and inputs the correction parameter TR to the target camera 101b.
[0043] The parameter setting unit 204 may also perform the performance evaluation of the correction parameters TR as described above for both the image T1' obtained after applying the correction parameters TR to the image T1 and the extended image T2' obtained after applying the correction parameters TR to the extended image T2. That is, the parameter setting unit 204 determines that the performance evaluation is acceptable if the difference in RGB values between the image T1' and the image R1 is within a first threshold value and the difference in RGB values between the extended image T2' and the extended image R2 is within a first threshold value. Then, the parameter setting unit 204 inputs the correction parameters TR to the target camera 101b.
[0044] Furthermore, the parameter setting unit 204 may evaluate the performance of the correction parameter TR using the RGB values of each patch 601-624 of the color chart 103 extracted based on the control information, or may use the RGB values of the entire color chart 103. When applying the correction parameter TR to the image T1 or the extended image T2, the parameter setting unit 204 interpolates the output value by, for example, tetrahedral interpolation (triangular pyramid interpolation) using the RBG values of each lattice point. Furthermore, the parameter setting unit 204 may evaluate the performance of the correction parameter TR in accordance with standards such as CIE2000, without being limited to the difference between the RGB values.
[0045] Furthermore, the parameter setting unit 204 may display the result of the performance evaluation of the correction parameter TR as information display on the UI of the display unit 102b (display means). FIG. 11 is an image diagram showing an example of a histogram of the gradation value of an image by frequency. 1100 in FIG. 11 is a histogram for an image T1' after the correction parameter TR is applied to the image T1. 1101 in FIG. 11 is a histogram for an extended image T2' after the correction parameter TR is applied to the extended image T2. The information display as shown in FIG. 11 is characterized in that the user can distinguish between the histograms 1100 and 1101 before and after the extension, and can distinguish the result of the performance evaluation of the correction parameter TR when the histograms 1100 and 1101 are displayed.
[0046] Specifically, on the UI of the display unit 102b, the histograms 1100 and 1101 before and after the expansion are displayed in different display modes, such as solid lines and dotted lines, as shown in Fig. 11, thereby enabling the user to distinguish between them. Furthermore, on the UI of the display unit 102b, a performance evaluation 1102 is displayed together with each of the histograms 1100 and 1101 (image information), as shown in Fig. 11. The performance evaluation 1102 has a field E1 showing the result of the performance evaluation of the correction parameter TR when applied to the image T1, and a field E2 showing the result of the performance evaluation of the correction parameter TR when applied to the expanded image T2. In each of the fields E1 and E2, the result of the performance evaluation of the correction parameter TR may be expressed as a numerical value or as characters such as OK or NG.
[0047] The performance evaluation results of the correction parameters TR may be easily distinguished by, for example, adding color to the display of results with errors. If there is an error in the performance evaluation results of the correction parameters TR, the parameter setting unit 204 may notify the user of the error on the UI of the display unit 102b to prompt the user to retake the images using the reference camera 101a and the target camera 101b. The image T1' after the correction parameters TR are applied to the image T1 and the extended image T2' after the correction parameters TR are applied to the extended image T2 may be displayed by a waveform monitor, a vector scope, or the like, instead of the histograms 1100 and 1101.
[0048] As described above, in the first embodiment, the image processing device 102 generates extended images R2 and T2 by extending the brightness of the images R1 and T1 in which the color chart 103 is captured. Furthermore, the image processing device 102 generates a correction parameter TR using the extended images R2 and T2, and performs performance evaluation of the correction parameter TR. This allows the image processing device 102 to check the validity of the correction parameter TR used in color matching between the reference camera 101a and the target camera 101b even outside the dynamic range of the color chart 103. Therefore, the image processing device 102 can suppress image quality degradation such as color bending in a color space outside the dynamic range of the color chart 103.
[0049] Furthermore, the image processing device 102 generates extended images R2 and T2 using the respective luminance profiles of the reference camera 101a and the target camera 101b and the images R1 and T1. Therefore, the image processing device 102 can generate extended images R2 and T2 that reflect the shooting environments of the reference camera 101a and the target camera 101b and the characteristics of the images R1 and T1.
[0050] In the first embodiment, the image generating unit 202 generates extended images R2 and T2 by extending the images R1 and T1 toward the high gradation side as shown in Fig. 7, Fig. 8, and Fig. 11. However, for example, when the image of the imaging device 101 is biased toward the high gradation side, the image generating unit 202 generates extended images R2 and T2 by extending the images R1 and T1 toward the low gradation side, in contrast to the first embodiment. Also, for example, when the image of the imaging device 101 is biased toward the intermediate gradation side, the image generating unit 202 generates extended images R2 and T2 by extending the images toward both the high gradation side and the low gradation side, unlike the first embodiment.
[0051] Furthermore, the image processing device 102 may regenerate the correction parameters TR from a plurality of images constituted by the images R1, T1 and the extended images R2, T2. In this case, the images R1, T1 are input to the parameter generating unit 203 in addition to the extended images R2, T2. Furthermore, the parameter generating unit 203 generates the correction parameters TR based on the input images R1, T1 and the extended images R2, T2. At this time, the approximation formula for generating the correction parameters TR can be obtained including the images R1, T1, so that the estimation accuracy of the correction parameters TR is improved.
[0052] <Second embodiment> An image processing device 102 according to the second embodiment will be described below with reference to Figs. 12 and 13. Here, the differences from the first embodiment will be mainly described. In the first embodiment, a method for generating extended images R2, T2 from images R1, T1 by using a luminance profile or the like is described. In the second embodiment, a method for generating extended images R2, T2 from images R1, T1 by using an offset value common to images R1, T1, instead of using a luminance profile or the like, is described. Fig. 12 is a block diagram showing an example of the configuration of an image processing unit 102a in the second embodiment.
[0053] As shown in FIG. 12, the image processing unit 102a has an image input unit 201, a parameter generating unit 1200, an image generating unit 1201, and a parameter setting unit 1202. The image input unit 201 is the same as that in the first embodiment except for the input destination of the images R1 and T1, so a detailed description thereof will be omitted. Incidentally, the image input unit 201 inputs the images R1 and T1 to the parameter generating unit 1200, the image generating unit 1201, and the parameter setting unit 1202. The parameter generating unit 1200 (correction parameter generating means) generates a correction parameter TR and a correction parameter RT (second correction parameter) based on the images R1 and T1 input from the image input unit 201. The correction parameter TR is input to the image generating unit 1201 and the parameter setting unit 1202 by the parameter generating unit 1200. On the other hand, the correction parameter RT is input to the image generating unit 1201 by the parameter generating unit 1200.
[0054] The correction parameter TR means a parameter for making the color and brightness of the image T1 closer to the color and brightness of the image R1. Therefore, the correction parameter TR has the same meaning as in the first embodiment. On the other hand, the correction parameter RT means a parameter for making the color and brightness of the image R1 closer to the color and brightness of the image T1. The method of calculating the correction parameters TR and RT is the same as in the first embodiment from the calculation of the coefficients of the approximation formula to the calculation of the output value, except that the original images are the images R1 and T1 before expansion, so detailed explanations thereof will be omitted. Incidentally, the output value of the correction parameter TR is calculated by the difference between the output value of the approximation formula for the image R1 and the output value of the approximation formula for the image T1. On the other hand, the output value of the correction parameter RT is calculated by the difference between the output value of the approximation formula for the image T1 and the output value of the approximation formula for the image R1.
[0055] The image generating unit 1201 (image generating means) generates extended images R2 and T2 from images R1 and T1, correction parameters TR and RT, and an offset value input by a UI operation. Therefore, the user can change the offset value input to the image generating unit 1201 on the UI (operation means) of the display unit 102b. Furthermore, the image generating unit 1201 inputs the extended images R2 and T2 to the parameter setting unit 1202. FIG. 13 is a block diagram showing an example of the configuration of the image generating unit 1201. As shown in FIG. 13, the image generating unit 1201 has a first parameter application unit 1300, an offset processing unit 1301, and a second parameter application unit 1302.
[0056] The first parameter application unit 1300 applies the correction parameters TR to the image T1. In the second embodiment, the correction parameters TR are in a 3DLUT format, as in the first embodiment. Therefore, when the first parameter application unit 1300 applies the correction parameters TR to the image T1, it interpolates the output value by, for example, tetrahedral interpolation (triangular pyramid interpolation) using the RBG values of each lattice point. Furthermore, the first parameter application unit 1300 inputs the image T1' (first intermediate extended image) after applying the correction parameters TR to the image T1 to the offset processing unit 1301.
[0057] The offset processing unit 1301 adds / subtracts the image R1 and the image T1' by the offset value. Furthermore, the offset processing unit 1301 inputs an image obtained by adding the image R1 to an image obtained by adding / subtracting the image R1 by the offset value and adding the image R1 to the parameter setting unit 1202 as an extended image R2. Furthermore, the offset processing unit 1301 inputs an image obtained by adding / subtracting the image T1' to an image obtained by adding / subtracting the image T1' by the offset value and adding the image T1' to the second parameter application unit 1302 as an image T1'' (second intermediate extended image).
[0058] When image R1 is biased toward the high gradation side, offset processing unit 1301 subtracts an offset value, and when image R1 is biased toward the low gradation side, offset processing unit 1301 adds an offset value. When image R1 is biased toward the middle gradation side, offset processing unit 1301 adds an offset value on the high gradation side of image R1 and subtracts an offset value on the low gradation side of image R1, with the central gradation value of image R1 as the boundary. However, when the gradation values of image R1 are distributed across all gradations, offset processing unit 1301 does not necessarily need to add or subtract an offset value. The same applies to image T1' to which offset processing unit 1301 adds or subtracts an offset value.
[0059] Incidentally, when a common offset value is applied to the images R1 and T1, it is necessary to make it possible to commonly handle characteristics such as color gamut and gamma by commonizing the respective spaces. In this regard, in the second embodiment, the space is commonized by applying a correction parameter to the images. Specifically, the first parameter application unit 1300 applies the correction parameter TR to the image T1, thereby making the image T1' after the application an image having the characteristics of the image R1. As a result, the respective spaces of the images R1 and T1' are commonized. Therefore, the offset processing unit 1301 can add or subtract a common offset value to the images R1 and T1'. However, the characteristics of the image T1'' after the offset value is added or subtracted from the image T1' remain the characteristics of the image R1. Therefore, the second parameter application unit 1302 applies the correction parameter RT to the image T1'', thereby making the extended image T2, which is the image after the application, an image having the characteristics of the image T1.
[0060] Note that the correction parameters RT are in the 3DLUT format, like the correction parameters TR. Therefore, when applying the correction parameters RT to the image T1'', the second parameter application unit 1302 interpolates output values by, for example, tetrahedral interpolation (triangular pyramid interpolation) using the RBG values of each lattice point. Furthermore, the second parameter application unit 1302 inputs an extended image T2, which is an image obtained after applying the correction parameters RT to the image T1'', to the parameter setting unit 1202.
[0061] Return to FIG. 12. The parameter setting unit 1202 (performance evaluation means) performs performance evaluation of the correction parameter TR based on the images R1 and T1, the extended images R2 and T2, the correction parameter TR, and the control information, in the same manner as in the first embodiment. Furthermore, the parameter setting unit 1202 inputs the correction parameter TR to the target camera 101b if there is no error in the result of the performance evaluation of the correction parameter TR, in the same manner as in the first embodiment. Furthermore, the parameter setting unit 1202 displays the histograms 1100 and 1101 and the performance evaluation 1102 as information display on the UI of the display unit 102b, in the same manner as in the first embodiment. At that time, the parameter setting unit 1202 may display an offset value corresponding to the histograms 1100 and 1101 and the performance evaluation 1102 on the UI of the display unit 102b. This allows the user to change the offset value using the UI to simulate changes in the shooting environment of the image capturing device 101, and thereby understand the tracking relationship with the performance evaluation of the correction parameter TR generated using images R1 and T1.
[0062] The image processing device 102 may regenerate the correction parameters TR from a plurality of images constituted by the images R1, T1 and the extended images R2, T2. In this case, the extended images R2, T2 are input to the parameter generating unit 1200 in addition to the images R1, T1. Furthermore, the parameter generating unit 1200 generates the correction parameters TR based on the input images R1, T1 and the extended images R2, T2. At this time, the approximation formula for generating the correction parameters TR can be obtained including the extended images R2, T2 obtained by extending the images R1, T1, so that the estimation accuracy of the correction parameters TR is improved.
[0063] As described above, in the second embodiment, the image processing device 102 generates the correction parameter TR using the images R1 and T1 in which the color chart 103 is photographed. Furthermore, the image processing device 102 generates extended images R2 and T2 in which the brightness of the images R1 and T1 in which the color chart 103 is photographed is extended, and performs performance evaluation of the correction parameter TR using the extended images R2 and T2. This allows the image processing device 102 to check the validity of the correction parameter TR used in color matching between the reference camera 101a and the target camera 101b even outside the dynamic range of the color chart 103. Therefore, the image processing device 102 can suppress image quality degradation such as color bending in a color space outside the dynamic range of the color chart 103.
[0064] Furthermore, the image processing device 102 generates extended images R2 and T2 using the images R1 and T1 and an offset value, etc. Therefore, the image processing device 102 can generate extended images R2 and T2 that reflect the shooting environment of the reference camera 101a and the target camera 101b and the characteristics of the images R1 and T1 via the offset value. Furthermore, in the second embodiment, since a luminance profile is not generated, the user can cause the image processing device 102 to generate extended images R2 and T2 with less work burden than in the first embodiment.
[0065] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. Parts of the above-mentioned embodiments may be appropriately combined. For example, the display unit 102b may be provided separately from the image processing device 102. In addition, the parameter setting unit 204, 1202 (notification means) may notify the user of an error on the UI of the display unit 102b when the difference in the dynamic range of the images R1 and T1 exceeds the second threshold (threshold value). In this way, the parameter setting unit 204, 1202 prompts the user to retake the image using the reference camera 101a and the target camera 101b, thereby ensuring the reliability of the correction parameter TR and its performance evaluation.
[0066] The present invention also includes a case where a software program for implementing the functions of the above-mentioned embodiment is supplied to a system or device having a computer capable of executing the program directly from a recording medium or by using wired / wireless communication, and the program is executed. Therefore, the program code itself that is supplied and installed to a computer in order to realize the functional processing of the present invention by the computer also realizes the present invention. In other words, the computer program itself for implementing the functional processing of the present invention is also included in the present invention. In this case, as long as it has the function of the program, the form of the program does not matter, such as object code, a program executed by an interpreter, script data supplied to an OS, etc. The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory. In addition, as a method of supplying the program, a method in which the computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads the computer program and programs it is also considered.
[0067] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.
[0068] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) An input means for inputting a first image and a second image obtained by the target imaging device and the reference imaging device photographing a color chart; an image generating means for generating a first expanded image and a second expanded image in which the brightness of the first image and the second image are expanded; a correction parameter generating means for generating a first correction parameter for matching a color and a brightness of the target imaging device to a color and a brightness of the reference imaging device, using the first image and the second image or the first extended image and the second extended image; a performance evaluation unit that performs a performance evaluation of the first correction parameter by using the first extended image and the second extended image. (Configuration 2) The image generating means generates the first extended image and the second extended image by using the luminance profiles of the target imaging device and the reference imaging device, and the first image and the second image; 2. The image processing device according to configuration 1, wherein the correction parameter generating means generates the first correction parameter by using the first extended image and the second extended image. (Configuration 3) The image processing device described in Configuration 2, characterized in that the image generation means generates each brightness profile of the target imaging device and the reference imaging device using generation parameters prepared in advance that correspond to the shooting environments of the target imaging device and the reference imaging device. (Configuration 4) The image processing device according to configuration 3, further comprising an operation means that enables a user to input information regarding the imaging environments of the target imaging device and the reference imaging device. (Configuration 5) The image processing device described in Configuration 2, characterized in that the image generation means generates each brightness profile of the target imaging device and the reference imaging device using values obtained when a measuring instrument measures the color chart at the shooting site of the target imaging device and the reference imaging device. (Configuration 6) The image processing device described in any one of configurations 2 to 5, characterized in that the correction parameter generation means regenerates the first correction parameter using the first image and the second image in addition to the first extended image and the second extended image. (Configuration 7) The image processing device according to any one of configurations 2 to 6, further comprising a display unit that displays a performance evaluation of the first correction parameter together with the first image and image information relating to the first image. (Configuration 8) The performance evaluation means performs a performance evaluation of the first correction parameter by using the first image and the second image in addition to the first extended image and the second extended image, 7. The image processing device according to any one of configurations 2 to 6, further comprising a display unit that displays a performance evaluation of the first correction parameter together with image information relating to the first image and the first extended image. (Configuration 9) The correction parameter generating means generating the first correction parameters using the first image and the second image; generating second correction parameters using the first image and the second image for matching the color and brightness of the reference imaging device to the color and brightness of the target imaging device; The image generating means includes: A first intermediate extended image is generated by applying the first correction parameter to the first image, a second intermediate extended image is generated by adding the first intermediate extended image to an image obtained by adjusting the first intermediate extended image by an offset value, and the first extended image is generated by applying the second correction parameter to the second intermediate extended image. 2. The image processing device according to claim 1, wherein the second extended image is generated by adding the second image to an image obtained by adjusting the second image by the offset value. (Configuration 10) The image processing device according to configuration 9, further comprising an operation means that enables a user to change the offset value. (Configuration 11) The image processing device according to configuration 9 or 10, characterized in that the correction parameter generation means regenerates the first correction parameter using the first extended image and the second extended image in addition to the first image and the second image. (Configuration 12) The image processing device according to any one of configurations 9 to 11, further comprising a display means for displaying a performance evaluation of the first correction parameter together with image information relating to the first extended image and the second extended image. (Configuration 13) The performance evaluation means performs a performance evaluation of the first correction parameter by using the first image and the second image in addition to the first extended image and the second extended image, 12. The image processing device according to any one of configurations 9 to 11, further comprising a display unit that displays a performance evaluation of the first correction parameter together with image information relating to the first extended image and the second extended image. (Configuration 14) The image processing apparatus according to configuration 12 or 13, wherein the display means displays the offset value together with the image information and a performance evaluation of the first correction parameter. (Configuration 15) The image processing device according to any one of configurations 1 to 14, further comprising a notification means for notifying a user of an error when a difference between dynamic ranges of the first image and the second image exceeds a threshold value. (Configuration 16) The image processing device according to any one of configurations 1 to 15, wherein the performance evaluation means inputs the first correction parameters to the target imaging device when the performance evaluation of the first correction parameters is acceptable. (Method 1) An input step of inputting a first image and a second image obtained by photographing a color chart with a target imaging device and a reference imaging device; an image generating step of generating a first expanded image and a second expanded image in which the brightness of the first image and the second image are expanded; a correction parameter generating step of generating a first correction parameter for matching a color and a brightness of the target imaging device to a color and a brightness of the reference imaging device, using the first image and the second image or the first extended image and the second extended image; a performance evaluation step of evaluating performance of the first correction parameter by using the first extended image and the second extended image. (Program 1) A program for causing a computer to execute each of the means of the image processing device according to any one of configurations 1 to 16. [Explanation of symbols]
[0069] 101a Reference camera (reference imaging device) 101b Target camera (target imaging device) 102 Image processing device 103 Color Chart 202 Image generation unit (input means) (image generation means) 203 Parameter generation unit (correction parameter generation means) 204 Parameter setting unit (performance evaluation means) R1 Image (2nd image) R2 Extended image (Second extended image) T1 image (first image) T2 extended image (first extended image) TR correction parameter (first correction parameter)
Claims
1. an input means for inputting a first image and a second image obtained by the target imaging device and the reference imaging device photographing a color chart; an image generating means for generating a first expanded image and a second expanded image in which the brightness of the first image and the brightness of the second image are expanded; a correction parameter generating means for generating a first correction parameter for matching a color and a brightness of the target imaging device to a color and a brightness of the reference imaging device, using the first image and the second image or the first extended image and the second extended image; a performance evaluation unit that performs a performance evaluation of the first correction parameter by using the first extended image and the second extended image.
2. the image generating means generates the first extended image and the second extended image using the luminance profiles of the target imaging device and the reference imaging device, the first image, and the second image; The image processing apparatus according to claim 1 , wherein the correction parameter generating means generates the first correction parameter by using the first expanded image and the second expanded image.
3. The image processing device according to claim 2, characterized in that the image generation means generates the luminance profiles of the target imaging device and the reference imaging device using generation parameters corresponding to the shooting environments of the target imaging device and the reference imaging device, among generation parameters prepared in advance.
4. 4. The image processing apparatus according to claim 3, further comprising an operation unit that enables a user to input information regarding the imaging environments of the target imaging device and the reference imaging device.
5. 3. The image processing device according to claim 2, wherein the image generating means generates luminance profiles of the target imaging device and the reference imaging device using values obtained when a measuring instrument measures the color chart at the shooting site of the target imaging device and the reference imaging device.
6. 3 . The image processing apparatus according to claim 2 , wherein the correction parameter generating means regenerates the first correction parameter by using the first image and the second image in addition to the first expanded image and the second expanded image.
7. 3. The image processing apparatus according to claim 2, further comprising a display unit that displays a performance evaluation of the first correction parameter together with image information relating to the first image and the first extended image.
8. the performance evaluation means performs a performance evaluation of the first correction parameter using the first image and the second image in addition to the first extended image and the second extended image; 3. The image processing apparatus according to claim 2, further comprising a display unit that displays a performance evaluation of the first correction parameter together with image information relating to the first image and the first extended image.
9. The correction parameter generating means generating the first correction parameters using the first image and the second image; generating second correction parameters for matching the color and brightness of the reference imaging device to the color and brightness of the target imaging device using the first image and the second image; The image generating means includes: a first intermediate extended image is generated by applying the first correction parameter to the first image, a second intermediate extended image is generated by adding the first intermediate extended image to an image obtained by adjusting the first intermediate extended image by an offset value, and the first extended image is generated by applying the second correction parameter to the second intermediate extended image; The image processing apparatus according to claim 1 , wherein the second expanded image is generated by adding the second image to an image obtained by adjusting the second image by the offset value.
10. 10. The image processing apparatus according to claim 9, further comprising an operation unit that enables a user to change the offset value.
11. 11. The image processing apparatus according to claim 9, wherein the correction parameter generating means regenerates the first correction parameter by using the first expanded image and the second expanded image in addition to the first image and the second image.
12. 10. The image processing apparatus according to claim 9, further comprising a display unit that displays a performance evaluation of the first correction parameter together with image information relating to the first expanded image and the second expanded image.
13. the performance evaluation means performs a performance evaluation of the first correction parameter using the first image and the second image in addition to the first extended image and the second extended image; 10. The image processing apparatus according to claim 9, further comprising a display unit that displays a performance evaluation of the first correction parameter together with image information relating to the first expanded image and the second expanded image.
14. 14. The image processing apparatus according to claim 12, wherein the display unit displays the offset value together with the image information and a performance evaluation of the first correction parameter.
15. 2. The image processing apparatus according to claim 1, further comprising a notification unit that notifies a user of an error when a difference between the dynamic ranges of the first image and the second image exceeds a threshold value.
16. 2 . The image processing apparatus according to claim 1 , wherein the performance evaluation means inputs the first correction parameters to the target imaging device when the performance evaluation of the first correction parameters passes.
17. an input step of inputting a first image and a second image obtained by photographing a color chart with the target imaging device and the reference imaging device; an image generating step of generating a first expanded image and a second expanded image in which the brightness of the first image and the second image are expanded; a correction parameter generating step of generating a first correction parameter for matching a color and a brightness of the target imaging device to a color and a brightness of the reference imaging device, using the first image and the second image or the first extended image and the second extended image; a performance evaluation step of evaluating a performance of the first correction parameter by using the first extended image and the second extended image.
18. 2. A program for causing a computer to execute each of the means of the image processing apparatus according to claim 1.
Citation Information
Patent Citations
Shooting setting adjustment system, information processing unit, and control method and control program therefor
JP2012195760A
