Image processing apparatus, image processing method and program

The image processing device addresses conversion errors in digital cameras by using measured color patches and reference points to generate a 3DLUT, enhancing color reproduction accuracy in high dynamic range scenarios.

JP2024167946APending Publication Date: 2024-12-05CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023084285
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing color conversion functions in digital cameras, expressed as a sum of up to third-order terms using RGB values, suffer from insufficient freedom and large conversion errors under conditions where output color changes significantly with slight input color variations, particularly in high dynamic range (HDR) reproduction.

Method used

An image processing device that includes data acquisition, input point acquisition, reference point selection, and coefficient calculation units to generate a 3DLUT for color conversion, using measured color patches and reference points to reduce conversion errors by interpolating and calculating color conversion results based on multiple reference points.

Benefits of technology

Reduces color conversion errors by optimizing the 3DLUT for digital cameras, ensuring accurate color reproduction even under conditions of significant input color changes, thereby improving image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167946000001_ABST
    Figure 2024167946000001_ABST
Patent Text Reader

Abstract

To reduce an error that occurs in performing color conversion of an image.SOLUTION: Data of a measured value of a patch (color) that is measured from each of an image (image of a source color space), in which a color chip 3 having a plurality of patches (colors) is imaged by a digital camera 1, and an image (image of a target color space), in which the color chip 3 is imaged by a specific camera, is acquired. And, an input point on a color space is acquired. And, a plurality of reference point candidates corresponding to a plurality of patches (colors) are acquired from data of a measured value that is measured from the image of the source color space, and a plurality of reference point candidates, in which a distance from the input point on the color space is near, are selected as a reference point from among the plurality of reference point candidates. And, a coefficient of color conversion is acquired on the basis of a position of the input point, and a color conversion result in the input point is calculated on the basis of the coefficient of the acquired color conversion and data of the measured value measured from the image of the target color space, corresponding to the plurality of reference points.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] A digital camera has a function of storing a plurality of color processing parameters prepared in advance inside, and performing image processing such as color conversion on a captured image using color processing parameters corresponding to a mode selected by a user, for example. Patent Document 1 discloses a technique for determining a color conversion function that makes the captured image data obtained by capturing an image of a color chart with a digital camera closer to the original color of the color chart or the color that the user prefers. This color conversion function is sometimes stored in the form of a three-dimensional lookup table (3DLUT). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-153684 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, the above color conversion function is expressed as a sum of up to third-order terms with the RGB values ​​of the input data as explanatory variables. Therefore, under conditions where the output color changes significantly with a slight change in the input color (chaotic nature), the degree of freedom of the function is insufficient, and the conversion error caused by the color conversion function becomes large. The effect of such conversion error becomes more noticeable as the color reproduction range of the digital camera becomes wider, and for example, it cannot be ignored in recent digital cameras capable of HDR (high dynamic range) reproduction.

[0005] Therefore, an object of the present invention is to reduce errors that occur when an image is color-converted. [Means for solving the problem]

[0006] The image processing device of the present invention is characterized by having a data acquisition means for acquiring first data representing the colors of multiple color charts measured in an image of multiple color charts and second data representing the target color at the measurement point where the first data was measured, an input point acquisition means for acquiring an input point on color space, a reference point selection means for acquiring multiple reference point candidates corresponding to the multiple color charts from the first data and selecting, as reference points, from the multiple reference point candidates, multiple reference point candidates that are close to the input point in color space, a coefficient acquisition means for acquiring color conversion coefficients based on the position of the input point, and a calculation means for calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the multiple reference points. Effect of the Invention

[0007] According to the present invention, it is possible to reduce errors that occur when an image is color-converted. [Brief description of the drawings]

[0008] [Figure 1A] FIG. 1 is a diagram illustrating an example of the configuration of a system including an image processing apparatus. [Figure 1B] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 2 is a diagram showing an example of a color chart. [Diagram 3] FIG. 4 is a diagram illustrating an example of a functional configuration of an image processing unit. [Figure 4] 4 is a flowchart showing a process performed by the image processing device according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of plotting measurement values ​​of a color chart. [Figure 6] 11A and 11B are diagrams illustrating an interpolation process between measurement values. [Figure 7] FIG. 11 is a diagram illustrating a selection process of a reference point. [Figure 8] 11A and 11B are diagrams illustrating a calculation process of a conversion coefficient. [Figure 9] FIG. 13 is a diagram illustrating an example of output of learning data. [Figure 10] 10 is a flowchart showing a process performed by an image processing device according to a second embodiment. [Figure 11] FIG. 13 is a diagram illustrating an example of a GUI display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not necessarily limited to the following embodiments.

[0010] [Embodiment 1] FIG. 1A is a block diagram showing an example of the configuration of a system to which an image processing device 2 according to this embodiment is applied. The system shown in FIG. 1 includes a digital camera 1, which is an example of an imaging device, and an image processing device 2 that calculates a color conversion result. In this system, a 3DLUT is generated for color conversion of an image (hereinafter sometimes referred to as an image in a source color space) captured by the digital camera 1 of a color chart 3 to an image (hereinafter sometimes referred to as an image in a target color space) captured by a specific camera having ideal color development of the color chart 3. A 3DLUT is a three-dimensional lookup table. FIG. 2 is a diagram showing an example of a color chart 3. The color chart 3 is a color chart on which a plurality of patches with different color values, such as skin color, sky color, grass color, and primary colors, are printed or the like. As the color chart 3, a color chart that can be used for color design, such as Color Checker SG (140 colors) or Color Checker (24 colors) by x-rite, is used.

[0011] First, the digital camera 1 will be described. The digital camera 1 comprises an imaging unit 101, a data input / output unit 102, an image processing unit 103, a holding unit 104, a display unit 105, and the like. The imaging unit 101 is composed of an imaging lens and an image sensor such as a CCD. The data input / output unit 102 is an interface capable of transmitting and receiving data such as image data and a color conversion function to and from an external device. The storage unit 104 stores data such as an image captured by the imaging unit 101, image data after image processing by the image processing unit 103, and a color conversion function. The image processing unit 103 performs image processing such as color processing on the image captured by the imaging unit 101. The display unit 105 includes a display device such as a liquid crystal display, and displays an image during and after imaging, an image after color processing, and the like. The image processing unit 103 performs image processing using the color conversion function stored in the storage unit 104. It is assumed that this color conversion function is stored in the form of a 3DLUT. In addition, the image processing device 2 is connected to the data input / output unit 102 via a cable or the like, and a recording medium such as a memory card can also be connected.

[0012] When an image of a subject or the like is captured, the digital camera 1 performs the following process. When a user turns on a power switch (not shown), the imaging unit 101 starts an operation of continuously capturing images. The image processing unit 103 performs image processing on image data continuously input from the imaging unit 101 using the 3DLUT stored in the storage unit 104, and generates a display image from the image data after the image processing. The display unit 105 displays the display image continuously sent from the image processing unit 103. The user observes the display image displayed on the display unit 105, determines the composition of the screen, and presses a shutter button (not shown). When the shutter button is pressed, the image processing unit 103 performs image processing using the 3DLUT stored in the storage unit 104 on the image data acquired by the imaging unit 101 at the timing when the shutter button is pressed. The image processing using the 3DLUT is a process of adjusting color and brightness. The image data after the image processing is output to an external device (such as the image processing device 2 or a recording medium) via the data input / output unit 102. In addition, if a color conversion function is set in the storage unit 104 such that the image data acquired by the imaging unit 101 is output directly to the data input / output unit 102, the image data acquired by the imaging unit 101 is output directly from the digital camera 1.

[0013] Next, the functional configuration of the image processing device 2 will be described. The image processing device 2 generates a color conversion function for converting an image in a source color space into an image in a target color space. The image processing device 2 has functions such as a data input / output unit 201, an image processing unit 202, an average value calculation unit 203, a display control unit 204, a user operation acquisition unit 205, a color processing parameter storage unit 206, a data storage unit 207, and a 3DLUT generation unit 208.

[0014] The data input / output unit 201 exchanges various data such as image data and color conversion functions with external devices. The data input / output unit 201 may transmit and receive various data with the digital camera 1 or the like via the general-purpose I / F 215 (FIG. 1B) or the communication I / F 217 (FIG. 1B), and may input and output various data with a recording medium such as a memory card (not shown). Here, the image data (input image data) input from the data input / output unit 201 is an image of the color chart 3 captured by the digital camera 1 or a specific camera. Here, the image of the color chart 3 captured by the digital camera 1 (image in source color space) is the image data itself acquired by the imaging unit 101. Also, the image of the color chart 3 captured by a specific camera (image in target color space) is an example of image data composed of target colors of the image in the source color space. Note that the image in the target color space does not have to be an image captured by a camera, and may be data directly given with color values ​​ideal by the user, such as data obtained by CG.

[0015] The image processing unit 202 calculates a color conversion result for an arbitrary input point in the color space, using input image data from the data input / output unit 201. Details of the function of the image processing unit 202 will be described later with reference to FIG. The average value calculation unit 203 calculates the average value of the color values ​​obtained from the input image data from the data input / output unit 201 as a measurement value. * a * b *The average value of the values ​​is calculated as the measurement value. Hereinafter, the measurement values ​​obtained from the image in the source color space and the image in the target color space are referred to as learning data. In this embodiment, the image in the source color space and the image in the target color space are image data groups obtained by capturing a color chart 3 with multiple exposure times. Here, if the number of patches is 18 and the number of exposure time steps is 32, measurement values ​​at 18 x 32 measurement points are obtained from each of the image in the source color space and the image in the target color space.

[0016] The data storage unit 207 stores data such as input image data, image data after image processing, and learning data. The color processing parameter storage unit 206 stores the color conversion functions in the digital camera 1 . A 3D lookup table (LUT) generator 208 generates a 3D LUT using the image processing results calculated by the image processor 202 . A user operation acquisition unit 205 acquires an input operation by a user from an input device 220 (FIG. 1B) such as a mouse, a keyboard, or a touch panel. The display control unit 204 displays, on a display device 214 (FIG. 1B), a GUI for inputting the number of grid points (lattice points) of a 3DLUT (described later) and a GUI for receiving operation instructions for learning data.

[0017] Next, the internal configuration of the image processing unit 202 will be described. 3 is a block diagram showing an example of the internal configuration of the image processing unit 202. The image processing unit 202 has functions such as a measurement value acquisition unit 301, a measurement value interpolation unit 302, an input point acquisition unit 303, a reference point selection unit 304, a conversion coefficient calculation unit 305, a variation value acquisition unit 306, and a color conversion result calculation unit 307. The measurement value acquisition unit 301 acquires the measurement values ​​from the average value calculation unit 203, and calculates the L between the measurement values ​​at corresponding measurement points for the image in the source color space and the image in the target color space. * a * b * Calculate the variation value for each component of . The measurement value interpolation unit 302 interpolates between the variation values ​​at each measurement point. Specifically, the measurement value interpolation unit 302 calculates the L * a * b * Find an equation that expresses each component of as a function of luminance l.

[0018] The input point acquisition unit 303 acquires an input point P in the source color space. The reference point selection unit 304 selects two or more reference points having a hue close to that of the input point P based on measurements obtained from the image in the source color space. The conversion coefficient calculation unit 305 calculates coefficients (hereinafter referred to as conversion coefficients) that express the position of the input point P by the position of the reference point selected by the reference point selection unit 304. The variation value acquisition unit 306 calculates an estimate of the variation at the input point P using the conversion coefficients calculated by the conversion coefficient calculation unit 305 and the variation values ​​corresponding to the reference points selected by the reference point selection unit 304 . A color conversion result calculation unit 307 calculates a color conversion result at the input point P based on the input point P and the estimated value of the variation calculated by the variation value acquisition unit 306 .

[0019] 1B is a block diagram showing a hardware configuration of the image processing device 2. The image processing device 2 is configured to include a CPU 211, a ROM 212, a RAM 213, a display device 214, a general-purpose I / F 215, a SATA (Serial ATA) I / F 216, and a communication I / F 217. An input device 220 such as a mouse, a keyboard, and a touch panel, and the digital camera 1 are connected to the general-purpose I / F (interface) 215 via a serial bus 219. An HDD (hard disk drive) 222 and a general-purpose drive 223 that reads and writes data from and to a recording medium such as a memory card are connected to the SATA I / F (interface) 216 via a serial bus 221.

[0020] The CPU 211 executes an OS (operating system) and various programs stored in the ROM 212, HDD 222, etc., using the RAM 213 as a work memory. The CPU 211 also controls each component via a system bus 218. Note that a program stored in the ROM 212, HDD 222, etc. is loaded onto the RAM 213 and executed by the CPU 211, thereby realizing processing according to a flowchart described later. Note that a program stored in the ROM 212, HDD 222, etc. is loaded onto the RAM 213 and executed by the CPU 211, thereby realizing various functions of the image processing device 2 shown in FIG. 1 and FIG. 3. The communication I / F 217 transmits and receives information to and from an external device. The CPU 211 displays various GUIs provided by the programs on the display device 214, acquires an input operation by a user via the input device 220, and controls the display of the GUI in response to the input operation. The display device 214 and the input device 220 may be configured by a touch panel display having both a display function and an input function. The CPU 211 uses a recording medium mounted on the HDD 222 or the general-purpose drive 223 as a storage location for various data such as input image data and color conversion functions.

[0021] 4 is a flowchart showing the flow of processing performed by the image processing device 2 according to this embodiment. In the following description, each process (step) is represented by adding an S to the beginning thereof, and the representation of the process (step) is omitted. In S401, the data input / output unit 201 inputs a group of image data obtained by capturing a color chart 3 with a digital camera 1 and a group of image data obtained by capturing a color chart 3 with a specific camera. These image data groups are paired images of an image in a source color space and an image in a target color space, and are data sets captured multiple times while adjusting the exposure of an imaging medium under uniform illumination. Here, the color chart 3 is captured with multiple exposure times (32 in this case) so that image data of various luminance levels can be obtained. At this time, the exposure time is adjusted so that the captured image covers the low luminance side to the high luminance side of the color space.

[0022] The number of stages of the exposure time is not limited to 32 stages. The method of adjusting the exposure amount of the imaging medium is not limited to adjusting the exposure time, and may be performed by adjusting the aperture (aperture ratio), adjusting the illumination intensity, using a different color chart, or the like. A color chart on which color patches are printed to have a gradation from low brightness to high brightness may be used to capture image data of a plurality of brightness levels with one color chart. Images of a plurality of color charts with different brightness levels may be captured. Images may be captured at multiple stages of emission intensity using a self-luminous chart, which is a self-luminous color chart. The input of the image data group in this step may be performed by directly reading from the digital camera 1 via a cable or the like, or by reading via a recording medium such as a memory card. The read image data group is stored in the data holding unit 207.

[0023] Next, in S402, the average value calculation unit 203 calculates the L for each patch in each image in the image data group read out from the data storage unit 207. * a * b * The average value of the RGB values ​​is calculated as the measurement value. * a * b * When converting the value, the color space conversion formula described later can be used. Here, if the number of patches is 18 and the number of exposure time steps is 32, measurement values ​​at 18 x 32 measurement points are obtained from the image in the source color space and the image in the target color space. Figure 5 shows the measurement values ​​obtained from the image in the source color space, as * a * b * In the following, the measurement values ​​obtained from the image in the source color space are referred to as learning data for the source color space, and the measurement values ​​obtained from the image in the target color space are referred to as learning data for the target color space. These learning data are stored in the data storage unit 207. The process of obtaining the measurement values ​​from the image in the source color space and the image in the target color space is an example of a data obtaining process.

[0024] Next, in S403, the measurement value acquisition unit 301 reads out the learning data from the data storage unit 207. Then, the L between the measurement values ​​at the corresponding measurement points for the read learning data of the source color space and the learning data of the target color space is calculated. * a * b * The variations Δl, Δa, and Δb for each component are calculated. The variations Δl, Δa, and Δb are calculated using the following equations (1) to (3). Δl=l´―l (1) Δa = a´ – a (2) Δb = b´ – b (3) where (l, a, b) is the measured value of the training data in the source color space (L * a * b * Also, (l´, a´, b´) represent the measured values ​​of the training data in the target color space (L * a * b * (l´, a´, b´) represents the measurement point corresponding to (l, a, b).

[0025] Next, in S404, the measurement value interpolation unit 302 obtains an equation that expresses the variations Δl, Δa, and Δb at each measurement point obtained in S403 as a function of luminance l for each patch (color) on the color chart 3. For example, the measurement value interpolation unit 302 obtains an equation that expresses the objective variable Δa i A function Δa that explains the explanatory variable l i The measurement value interpolation unit 302 also obtains an equation that expresses the components l, a, and b of the learning data in the source color space at each measurement point as a function of the luminance l for each patch (color) of the color chip 3. In this manner, the measurement value interpolation unit 302 obtains the function l i (l), a i (l), b i (l), Δl i (l), Δa i (l), Δb i (l) is generated, where i represents the patch number, and when the number of patches is 18, 1≦i≦18.

[0026] Here, the function Δa i A method for finding (l) will be described in detail. First, the measurement points exist only discretely. For example, if the exposure time is set to 32 steps, the number of measurement points for each patch (color) on the color chart 3 is 32. Therefore, a curve is found that smoothly connects the variation values ​​at the discretely existing measurement points in order. Figure 6 shows the function Δa i An example of a graph showing (l) is shown. The horizontal axis of the graph in FIG. 6 represents the luminance l, and the vertical axis represents the variation Δa. The circles represent the measurement points, and the curves represent the interpolation results. In the example of FIG. 6, the measurement points are interpolated by obtaining a seventh-order polynomial that passes through the measurement points without error using the least squares method. The degree of the polynomial may be other than seven. Also, instead of performing polynomial approximation, the measurement points may be interpolated using other methods such as broken line interpolation, spline interpolation, and Bezier interpolation. Also, it is not necessary to perform the interpolation. For example, the value between the measurement points may be defined as the same value as the value of the nearest measurement point.

[0027] Next, in S405, the input point acquisition unit 303 acquires an input point P to be color converted. Here, the input point P is expressed by three elements, l, a, and b. In this embodiment, the input point P corresponds to a lattice point of the 3DLUT. If it is assumed that the number of grid points of the 3DLUT is 17, there are 17×17×17=4913 input points P 1 ,P 2 , , P 4913 In other words, the color conversion process shown in S406 to S409 below is repeatedly executed for each of the 4913 input points.

[0028] Note that the input point P 1 ,P 2 , , P 4913 The method for calculating is as follows: First, assuming that the RGB values ​​of an image output by digital camera 1 are, for example, 10 bits (0 to 1023) for each RGB color, and the number of grid points in the 3DLUT of digital camera 1 is 17, each grid point in the 3DLUT is expressed as follows: (R,G,B)=(0, 0, 0) (0, 0, 64) (0, 0, 128) … (0, 0, 1023) (0, 64, 0) (0, 64, 64) … (0, 64, 1023) (0, 128, 0) (0, 128, 64) … (1023, 1023, 1023) Next, the input point acquisition unit 303 acquires the RGB values ​​of each grid point of the 3DLUT, L * a * b * This converts the input point P 1 ,P 2 ,···,P 4913 The RGB values ​​are converted to L * a * b * When converting to a value, a color space conversion formula described later can be used.

[0029] Next, in S406, the reference point selection unit 304 selects the function l obtained from the learning data in the source color space in S404. i (l), a i (l), b i Using (l), two reference points having a hue close to that of the input point P acquired in S405 are selected. Hereinafter, a method for selecting the reference points will be described with reference to FIG. First, the reference point selection unit 304 selects the luminance value l0 of the input point P by the function a i (l), b i By substituting (l), the color value P corresponding to all patches (colors) of color chip 3 is obtained as shown in the following formula (4). i Find (l0). P i (l0) = (l0, a i (l0),b i (l0)) ···(4) FIG. 7 shows the a * b * On the color plane, color value P i In the example of Fig. 7, color values ​​Pi(l0) for i = 1, 2, . . . , 18 are plotted.i (l0) is called a reference point candidate. Point 702 represents the input point P.

[0030] Next, the reference point selection unit 304 selects two reference point candidates whose hue is close to that of the input point P. Specifically, when the luminance value is l0, * b * In the color plane, a reference point candidate 703 that is the reference point candidate with the closest hue when viewed counterclockwise from the point 702 and a reference point candidate 704 that is the reference point candidate with the closest hue when viewed clockwise are selected and these are set as the reference points. Here, among the reference point candidates expressed by the above formula (4), P ζ (l0) = (l0, a ζ (l0),b ζ (l0)) and P η (l0) = (l0, a η (l0),b η In this way, from among the multiple reference point candidates corresponding to the multiple color charts, multiple reference points (here, P ζ (l0), P η The process of selecting (l0) is an example of a reference point selection process.

[0031] The method of selecting the reference point is not limited to the above method, as long as it is a method of selecting a plurality of reference points that are close to the input point P in color space. For example, two reference point candidates that are close in hue to the input point P may be selected ignoring the direction, such as clockwise or counterclockwise. Also, a point with a small saturation difference or color difference, instead of a hue difference, may be selected. Also, the calculation of the difference with the input point P may be performed in another color space. Also, instead of selecting the reference point based on the relative positions of the input point P and the reference point candidate, the reference point may be selected based on the absolute position of the input point P, such as selecting the reference point according to the area in which the input point P is located.

[0032] Next, in S407, the conversion coefficient calculation unit 305 calculates a conversion coefficient that expresses the position of the input point P by the position of the reference point selected in S406. A method for calculating the conversion coefficient will be specifically described below. In this embodiment, the conversion coefficient calculation unit 305 calculates, as the conversion coefficient, a linear combination coefficient that expresses the position of the input point P by the relative position between two reference points and the origin. Figure 8 shows the input point P and the reference point P ζ (l0), P η This is a diagram showing the relative positional relationship between (l0) and the origin O. In FIG. 8, the position of (l, a, b) = (l0, 0, 0) is represented as the origin O. At this time, the vector OP heading from the origin O to the input point P is ζ Vector OP toward (l0) ζ And from the origin O to the reference point P η Vector OP toward (l0) η This is expressed by the following equation (5).

[0033]

number

number

[0034] Next, in S408, the variation value acquisition unit 306 calculates an estimate of the variation ΔP at the input point P. First, the variation value acquisition unit 306 calculates the luminance value l0 of the input point P using the function Δl obtained in S404. i (l), Δa i (l), Δb i By substituting (l), the variation ΔP for all patches (colors) of color chip 3 is given by the following equation (7). i Request. ΔP i (l0)=(Δl i (l0), Δa i (l0), Δb i (l0)) ···(7) Then, the variation value acquisition unit 306 uses the linear combination coefficients α and β calculated in S407 to calculate an estimate of the variation ΔP at the input point P from the following equation (8). ΔP = αΔP ζ (l0) + βΔP η (l0) ···(8) ΔP in the above equation (8) ζ (l0) is ΔP when i=ζ in the above equation (7). ζ (l0) and the reference point P ζ (l0) is the variation value corresponding to (l0). Also, ΔP in the above equation (8) η (l0) is ΔP when i = η in the above equation (7). η (l0) and the reference point P η This is the variation value corresponding to (l0).

[0035] Next, in S409, the color conversion result calculation unit 307 uses the estimated value of the variation ΔP calculated in S408 to calculate a color conversion result P' for the input point P. The calculation formula is expressed by the following formula (9). P´=P+ΔP (9) Next, the image processing unit 202 returns to the process of S405, acquires the next input point P as a color conversion target among the 4913 input points P, and executes the color conversion process shown in S406 to S409. After that, when the color conversion process shown in S406 to S409 is completed for all input points P, the process transitions to S410.

[0036] In S410, the 3DLUT generation unit 208 generates a pair of the input point P acquired in S405 and the color conversion result P' calculated for the input point P in S409. T = {(P 1 ,P 1 ´),(P 2 ,P 2 ´),…,(P 4913 ,P 4913 ´)} ···(10) T in the above formula (10) holds the color conversion result at each grid point of the 3DLUT. T in the above formula (10) corresponds to the 3DLUT.

[0037] Next, in S411, the data input / output unit 201 uploads the 3DLUT generated in S410 to the digital camera 1 via the general-purpose I / F 215. In this manner, the series of processes in the flowchart shown in FIG.

[0038] Hereinafter, a method for color conversion of an image actually captured by the digital camera 1 will be described using the 3DLUT generated by the flowchart shown in Fig. 4. Here, the main body of the color conversion process will be described as the image processing unit 103 of the digital camera 1, but it may also be the image processing unit 202 of the image processing device 2.

[0039] First, when a pixel value Q (RGB value) of any input image data is input, the image processing unit 103 calculates four points P A P B P C P D are selected from the lattice points represented by T in the above equation (10), and are determined as the four corresponding lattice points. Next, the image processing unit 103 calculates the interpolation ratio. First, the image processing unit 103 calculates the interpolation ratio by dividing the four vectors (P A P B ), (P A P C ), (P A P D ), (P A Then, the image processing unit 103 creates the four created vectors (P A P B ), (P A P C ), (P A P D ), (P A Q) and any coefficient w B , w C , w D Using the following formula (11), w B , w C , w D This w B , w C , w D is the interpolation ratio. (P A Q)=w B (P A P B )+w C (P A P C )+w D (P A P D ) ···(11) Next, the image processing unit 103 calculates the interpolation ratio w B , w C , w D Using this, the color conversion result Q' is calculated using the following equation (12). Q´=P A ´+w B (P A ´P B ´)+w C (P A ´P C ´)+w D (P A ´P D ´) ···(12) In this manner, the image captured by the digital camera 1 can be color converted using the 3DLUT. By storing the color conversion function as the 3DLUT in this manner, it is possible to reduce the calculation time and memory requirements.

[0040] Next, the color space conversion formulas (sRGB→CIEXYZ conversion formula and CIEXYZ→CIELAB conversion formula) used in S402, S405, etc. will be described. First, the sRGB→CIEXYZ conversion formula will be explained. The image processing unit 202 normalizes the RGB values ​​of the input data. In this normalization, when the RGB values ​​of the input image data are 10 bits, each value is normalized by dividing the value by 2. 10 It is assumed that the process involves division by −1=1023. Then, the image processing unit 202 calculates the XYZ values ​​using the following formulas (13) to (17). R sRGB When ≦0.04045, R liner =R sRGB / 12.92 R sRGB R when >0.04045 liner =[(R sRGB +0.055) / 1.055)] 2.4 …(13) G sRGB When ≦0.04045, G liner =G sRGB / 12.92 G sRGB >0.04045 G liner =[(G sRGB +0.055) / 1.055)] 2.4 …(14) B sRGB B when ≦0.04045 liner =B sRGB / 12.92 B sRGB >0.04045: B liner =[(B sRGB +0.055) / 1.055)] 2.4 …(15)

[0041]

number

[0042] X=X·Xn / 95.045 Y=Y·Yn / 100.000 Z = Z Zn / 108.892 … (17) Here, Xn, Yn, and Zn are values ​​calculated from the color temperature of the lighting at the time of image capture.

[0043] Next, the CIEXYZ→CIELAB conversion formula will be explained. The CIEXYZ→CIELAB conversion formula is expressed by the following formulas (18) to (22). If X / Xn>0.00856, then X Rate =(X / Xn) 1 / 3 When X / Xn≦0.00856, X Rate =7.787×(X / Xn)+16.0 / 116.0 …(18) Y if Y / Yn>0.00856 Rate =(Y / Yn) 1 / 3 Y when Y / Yn≦0.00856 Rate =7.787×(Y / Yn)+16.0 / 116.0 …(19) When Z / Zn>0.00856, Z Rate =(Z / Zn) 1 / 3 When Z / Zn≦0.00856, Z Rate =7.787×(Z / Zn)+16.0 / 116.0 …(20) When Y / Yn>0.00856, L * =116.0×(Y / Yn) 1 / 3 -16.0 When Y / Yn≦0.00856, L * =903.29×(Y / Yn) …(21) a * =500×(X Rate -Y Rate ) b * =200×(Y Rate -Z Rate ) …(twenty two) Here, Xn, Yn, and Zn are values ​​calculated from the color temperature of the lighting at the time of image capture.

[0044] In this embodiment, the digital camera 1 and the image processing device 2 exchange data using a cable, but of course data may be exchanged using a recording medium such as a memory card, or a communication path such as wireless or infrared.

[0045] As described above, according to the first embodiment, it is possible to generate a 3DLUT optimized for converting captured image data of a color chart into data of the color chart captured by a camera with ideal color development. By performing color conversion processing using this 3DLUT, it is possible to reduce conversion errors that occur when converting captured image data. As a result, according to this embodiment, it is possible to reduce conversion errors even under conditions where a slight change in input color causes a large change in output color.

[0046] [Variation 1] In the above embodiment, the RGB values ​​of the input data from the digital camera 1 are assumed to be sRGB. * a * b * Although various calculations are performed in the color space, the calculations may be performed in other color spaces. For example, the RGB values ​​of the input data may be assumed to be BT.2020RGB and the calculations may be performed in the ICtCp color space. In that case, the image processing unit 202 uses the following color space conversion formula:

[0047] The BT.2020RGB→CIEXYZ conversion formula is expressed by the following formulas (23) to (25). R liner =((R 1 / m -c1) / (c2-c3*R 1 / m )) 1 / n G liner =((G 1 / m -c1) / (c2-c3*G 1 / m )) 1 / n B liner =((B 1 / m-c1) / (c2-c3*B 1 / m )) 1 / n …(twenty three) m=78.8437500; n=0.15930176; c1=0.83593750; c2=18.8515625; c3=18.6875000;

[0048]

number

[0049] X=X·Xn / 95.045 Y=Y·Yn / 100.000 Z = Z Zn / 108.892 … (25) Here, Xn, Yn, and Zn are values ​​calculated from the color temperature of the lighting at the time of image capture.

[0050] The CIEXYZ→ICtCp conversion formula is expressed by the following formulas (26) to (28).

[0051]

number

[0052] LPQ=((c1+c2*L n ) / (1+c3*L n )) m MPQ=((c1+c2*M n ) / (1+c3*M n )) m SPQ=((c1+c2*S n ) / (1+c3*S n )) m …(27) m=78.8437500; n=0.15930176; c1=0.83593750; c2=18.8515625; c3=18.6875000;

[0053]

number

[0054] [Variation 2] In the above embodiment, an example was described in which a color conversion result Q' is calculated using a 3DLUT when a pixel value Q of an actually captured image is input. In this modified example, an example is shown in which the processing from S405 to S409 is executed to directly calculate the color conversion result Q'. In this modified example, only S405 to S409 will be described, and the other steps will not be described as they are the same as in the first embodiment.

[0055] First, in S405, the input point acquisition unit 303 acquires one of the pixels to be color converted in the source color space as point Q. Next, by the same processes as S406 to S408, the image processing unit 202 calculates an estimate of the variation ΔQ at the input point Q. Next, in S409, the color conversion result calculation unit 307 calculates the color conversion result Q'=Q+ΔQ. Thereafter, the image processing unit 202 repeatedly executes S405 to S409 until there are no more pixels to be color converted. Note that the subsequent processes of S410 to S411 are omitted. In this manner, in this modified example, the color conversion result can be calculated without using a 3DLUT.

[0056] [Variation 3] In the above embodiment, the variation function Δl obtained in S404 i (l), Δa i (l), Δb i In the above example, the color conversion result P' is calculated using the variation function Δl i (l), Δa i (l), Δb i An example will be shown in which the color conversion result P' is calculated without obtaining (l). In this modified example, only S404 to S409 will be explained, and the other steps will not be explained as they are the same as those in the first embodiment.

[0057] First, in S404, the measurement value interpolation unit 302 calculates a function l′ using the luminance l as an explanatory variable from the components l′, a′, and b′ of the learning data of the target color space at each measurement point for each patch (color) of the color chart 3. i (l), a´ i (l), b´ i Then, the measurement value interpolation unit 302 calculates a function P' having these three elements. i (l) is generated. Then, S405 to S407 are executed. In this modification, instead of the following S408 to S409, the image processing unit 202 calculates the color conversion result P' using the following formula (29). Note that α and β in the following formula (29) are linear combination coefficients calculated in S407. Also, P in the following formula (29) ζ ´(l0), P η ´(l0) is the reference point selected in S406. P´=αP ζ ´(l0)+βP η ´(l0) …(29) In this modified example, the variation function Δl i (l), Δa i (l), Δb i The color conversion result can be calculated without calculating (l).

[0058] [Embodiment 2] In the first embodiment, an example has been described in which color conversion processing is performed using all of the learning data acquired by the measurement value acquisition unit 301. However, there are cases in which the accuracy of color conversion is improved by removing some of the inappropriate learning data from the target of the color conversion processing. FIG. 9 shows an example of the learning data acquired by the measurement value acquisition unit 301. The graph in FIG. 9 shows the interpolation results for each color of the learning data as a * b * Each series is a projection onto the color plane, and each series is the color value P of the i-th color on the color chart. iThe shape data of the second embodiment (l) is shown. Here, the series 901 is a series that crosses with other series due to an error when capturing an image with the digital camera 1 or a distortion of the source color space itself. If this series is used as learning data, image quality problems such as tone inversion and false contours may occur in the color conversion result image. Therefore, it is desirable to remove the series 901 from the learning data and use only the other series. The following mainly describes the parts that differ from the first embodiment. Also, the same components as those in the first embodiment are described with the same reference numerals.

[0059] Fig. 10 is a flowchart showing the flow of processing performed by the image processing device 2 according to this embodiment. The processing of this flowchart differs from the flowchart of Fig. 4 in that the processing of S1001 to S1003 described below is performed between S404 and S405. First, in S1000, the same processes as those in S401 to S404 are executed.

[0060] Next, in S1001, the measurement value interpolation unit 302 stores the learning data of the source color space in the functional format generated in S404 in the data storage unit 207. Next, the display control unit 204 reads out the learning data (functional format) of the source color space from the data storage unit 207, and outputs it to the display device 214 to present it to the user. FIG. 11 shows an example of a GUI output to the display device 214. This GUI displays a learning data shape 1102 in a window 1101. This learning data shape 1102 displays the learning data (functional format) of the source color space as a * b * It is projected onto the color plane, and each series is the color value P of the i-th patch (color) of the color chart. i (l) represents the shape data.

[0061] Next, in S1002, the user operation acquisition unit 205 receives an operation instruction for a series that the user wants to delete from the learning data. While the series is selected, the display control unit 204 highlights the selected series, as in series 1103. The user operation acquisition unit 205 may have a function for redoing a selection or multiple selection. The operation in this step is performed via the input device 220, such as a mouse, keyboard, or touch panel. If the delete button 1104 is pressed, the user operation acquisition unit 205 advances the process to S1003. If the delete button 1104 is not pressed, S1003 is skipped and the process advances to S1004. In S1003, the data holding unit 207 deletes the learning data corresponding to the sequence instructed to be deleted in S1003. Note that the data holding unit 207 may store the learning data corresponding to the sequence instructed to be deleted in S1003 in a separate memory instead of discarding it. After that, the process returns to S1002.

[0062] The above deletion process may be repeated any number of times. If there are no more series to be deleted, the user presses the Complete button 1105. When the Complete button 1105 is pressed, the user operation acquisition unit 205 advances the process to S1004. Next, in S1004, the same processing as in S405 to S411 is executed. In this embodiment, the reference point candidate corresponding to the series instructed to be deleted in S1003 is excluded from the candidate targets. In this manner, the series of processing in the flowchart shown in FIG. 10 is completed.

[0063] For further convenience, the user operation acquiring unit 205 may have a function of adding a new series. When the add button 1107 is pressed, the user operation acquiring unit 205 reads out learning data captured and edited in the past from the storage location specified in the input field 1106, and adds it to the learning data currently being edited. When the reload button 1108 is pressed, the user operation acquiring unit 205 ends editing of the learning data currently being edited, and newly reads out learning data from the storage location specified in the input field 1106. The display control unit 204 redisplays the learning data shape 1102 based on the newly read out learning data.

[0064] The data storage unit 207 may analyze the shape of the learning data (functional form) in the source color space for each patch (color) of the color chip 3, and may delete the learning data corresponding to the patch (color) if it is determined as a result of the analysis that a predetermined condition that may be a cause of a decrease in color conversion accuracy is met. An example of a predetermined condition that may be a cause of a decrease in color conversion accuracy is a condition that indicates a large shooting error, such as crossing with another series. Specifically, the data storage unit 207 may determine whether a series corresponding to a patch (color) crosses with another series by a predetermined value or more, and may delete the series if it is determined as a result of the determination that the series crosses by the predetermined value or more.

[0065] As described above, according to the second embodiment, it is possible to remove learning data with large imaging errors in advance, and to reduce errors in color conversion. In addition, by displaying the shapes of the learning data, it is possible to support the task of removing data with large errors.

[0066] [Other embodiments] In addition, a software program code for realizing the functions of the image processing device 2 may be supplied to a computer in an apparatus or system that is connected to various devices and operates the various devices. In this case, the scope of the present invention also includes a case where the computer (CPU or MPU) of the system or apparatus operates the various devices according to the supplied program to implement the above-mentioned embodiment. For example, the above-mentioned image processing is performed on a driver software installed on a computer. In that case, the program code of the software itself realizes the functions of the above-mentioned embodiment, and the program code itself and a storage medium in which the program code is stored for supplying the program code to a computer constitute the present invention.

[0067] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) A data acquisition means for acquiring first data representing the colors of the plurality of color charts measured in an image obtained by capturing the plurality of color charts, and second data representing the color of a target at a measurement point where the first data was measured; An input point acquisition means for acquiring an input point on a color space; a reference point selection means for acquiring a plurality of reference point candidates corresponding to the plurality of color charts from the first data, and selecting, as reference points, from the plurality of reference point candidates, the plurality of reference point candidates that are close to the input point in color space; A coefficient acquisition means for acquiring a coefficient of a color conversion based on the position of the input point; a calculation means for calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the plurality of reference points; 13. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the coefficient acquisition means calculates the coefficients of the color conversion based on relative positions of the reference points and the input point. (Configuration 3) The image processing device according to configuration 1 or 2, characterized in that the calculation means calculates a variation value between the first data and the second data at the measurement point, and calculates a color conversion result at the input point based on the color conversion coefficients and the variation values ​​corresponding to the multiple reference points. (Configuration 4) The image is an image captured by an imaging device, The imaging device further includes a generating unit for generating a color conversion function to be stored in a three-dimensional lookup table, The image processing device according to any one of configurations 1 to 3, wherein the generation means generates the color conversion function based on the input point corresponding to a lattice point of the three-dimensional lookup table and the color conversion result at the input point. (Configuration 5) the data acquisition means acquires a plurality of measurement values ​​having different luminance from a single color chart; an interpolation unit that interpolates values ​​between the plurality of measurement values ​​for each of the plurality of color charts; 5. The image processing device according to any one of configurations 1 to 4, wherein the reference point selection means acquires a plurality of reference point candidates corresponding to the plurality of color charts from the first data interpolated by the interpolation means. (Configuration 6) 6. The image processing device according to configuration 5, wherein the interpolation means generates a function expressing the plurality of measurement values ​​in terms of luminance for each of the plurality of color charts. (Configuration 7) 7. The image processing device according to claim 6, wherein the interpolation means generates the function by performing polynomial approximation on the plurality of measurement values. (Configuration 8) The image processing device according to any one of configurations 5 to 7, characterized in that the image includes at least one of an image data group obtained by capturing the plurality of color charts with multiple levels of exposure time, an image data group obtained by capturing the plurality of color charts with multiple levels of aperture ratio, and an image data group obtained by capturing the plurality of color charts with multiple levels of illumination intensity. (Configuration 9) The image processing device according to any one of configurations 6 to 7, wherein the reference point selection means acquires points obtained by substituting luminance values ​​of the input points into the function for each of the plurality of color charts as the plurality of reference point candidates. (Configuration 10) The image processing device according to any one of configurations 1 to 9, characterized in that the reference point selection means selects, from among the plurality of reference point candidates, a plurality of reference point candidates having a hue close to that of the input point as reference points. (Configuration 11) the data acquisition means acquires a plurality of measurement values ​​having different luminance from a single color chart; an interpolation means for interpolating values ​​between the plurality of measurement values ​​for each of the plurality of color charts; a display control means for controlling the display of shape data representing the first data interpolated by the interpolation means for each of the plurality of color charts; 11. The image processing device according to any one of configurations 1 to 10, further comprising: (Configuration 12) The image processing device according to any one of configurations 1 to 11, wherein the reference point selection means excludes, from among the first data corresponding to the plurality of color charts, the first data corresponding to a portion of the color chart that satisfies a predetermined condition, from targets for obtaining a plurality of reference point candidates. (Configuration 13) The image processing device according to configuration 12, wherein the reference point selection means excludes, from among the first data corresponding to the plurality of color charts, the first data corresponding to a color chart for which a user has made an operational instruction, from among the first data corresponding to the plurality of color charts, from targets for obtaining a plurality of reference point candidates. (method) a data acquisition step of acquiring first data representing the colors of the plurality of color charts measured in an image obtained by capturing the plurality of color charts, and second data representing the color of a target at a measurement point where the first data was measured; An input point acquisition step of acquiring an input point on a color space; a reference point selection step of acquiring a plurality of reference point candidates corresponding to the plurality of color samples from the first data, and selecting, as reference points, a plurality of the reference point candidates that are close to the input point in color space from the plurality of reference point candidates; a coefficient acquisition step of acquiring a coefficient for color transformation based on the position of the input point; a calculation step of calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the plurality of reference points; 13. An image processing method comprising: (program) The computer of the image processing device A data acquisition means for acquiring first data representing the colors of the plurality of color charts measured in an image obtained by capturing the plurality of color charts, and second data representing the color of a target at a measurement point where the first data was measured; An input point acquisition means for acquiring an input point on a color space; a reference point selection means for acquiring a plurality of reference point candidates corresponding to the plurality of color charts from the first data, and selecting, as reference points, from the plurality of reference point candidates, the plurality of reference point candidates that are close to the input point in color space; A coefficient acquisition means for acquiring a coefficient of a color conversion based on the position of the input point; a calculation means for calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the plurality of reference points; A program to function as a [Explanation of symbols]

[0068] 1: Digital camera, 2: Image processing device, 3: Color chart

Claims

1. a data acquisition means for acquiring first data representing the colors of the plurality of color charts measured in an image obtained by capturing the plurality of color charts, and second data representing the color of a target at a measurement point where the first data was measured; An input point acquisition means for acquiring an input point on a color space; a reference point selection means for acquiring a plurality of reference point candidates corresponding to the plurality of color charts from the first data, and selecting, as reference points, a plurality of the reference point candidates that are close to the input point in color space from the plurality of reference point candidates; A coefficient acquisition means for acquiring a coefficient of a color conversion based on the position of the input point; a calculation means for calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the plurality of reference points; 13. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the coefficient acquisition means calculates the coefficients of the color conversion based on relative positions between the plurality of reference points and the input point.

3. 2. The image processing device according to claim 1, wherein the calculation means calculates a variation value between the first data and the second data at the measurement point, and calculates a color conversion result at the input point based on the color conversion coefficients and the variation values ​​corresponding to the plurality of reference points.

4. The image is an image captured by an imaging device, The imaging device further includes a generating unit for generating a color conversion function to be stored in a three-dimensional lookup table, 2. The image processing apparatus according to claim 1, wherein said generating means generates said color conversion function based on said input points corresponding to lattice points of said three-dimensional lookup table and said color conversion results at said input points.

5. the data acquisition means acquires a plurality of measurement values ​​having different luminance from a single color chart; an interpolation unit that interpolates values ​​between the plurality of measurement values ​​for each of the plurality of color charts; 2. The image processing apparatus according to claim 1, wherein the reference point selection means acquires a plurality of reference point candidates corresponding to the plurality of color charts from the first data interpolated by the interpolation means.

6. 6. The image processing apparatus according to claim 5, wherein the interpolation means generates a function expressing the plurality of measurement values ​​in terms of luminance for each of the plurality of color charts.

7. 7. The image processing apparatus according to claim 6, wherein the interpolation means generates the function by performing polynomial approximation on the plurality of measured values.

8. 6. The image processing device according to claim 5, wherein the image includes at least one of an image data group obtained by capturing the plurality of color charts with multiple levels of exposure time, an image data group obtained by capturing the plurality of color charts with multiple levels of aperture ratio, and an image data group obtained by capturing the plurality of color charts with multiple levels of illumination intensity.

9. 7. The image processing apparatus according to claim 6, wherein the reference point selection means acquires points obtained by substituting luminance values ​​of the input points into the functions for each of the plurality of color charts as the plurality of reference point candidates.

10. 2. The image processing apparatus according to claim 1, wherein said reference point selection means selects, from among a plurality of said reference point candidates, a plurality of said reference point candidates having a hue close to that of said input point as said reference points.

11. the data acquisition means acquires a plurality of measurement values ​​having different luminance from a single color chart; an interpolation means for interpolating values ​​between the plurality of measurement values ​​for each of the plurality of color charts; a display control means for controlling display of shape data representing the first data interpolated by the interpolation means for each of the plurality of color charts; The image processing device according to claim 1 , further comprising:

12. 2. The image processing device according to claim 1, wherein the reference point selection means excludes, from among the first data corresponding to the plurality of color charts, the first data corresponding to a portion of the color chart that satisfies a predetermined condition, from targets for obtaining a plurality of reference point candidates.

13. 13. The image processing device according to claim 12, wherein the reference point selection means excludes, from among the first data corresponding to the plurality of color charts, the first data corresponding to a color chart for which an operation instruction has been given by a user, from targets for obtaining a plurality of reference point candidates.

14. a data acquisition step of acquiring first data representing the colors of the plurality of color charts measured in an image obtained by capturing the plurality of color charts, and second data representing the color of a target at a measurement point where the first data was measured; An input point acquisition step of acquiring an input point on a color space; a reference point selection step of acquiring a plurality of reference point candidates corresponding to the plurality of color charts from the first data, and selecting, as reference points, a plurality of the reference point candidates that are close to the input point in color space from the plurality of reference point candidates; a coefficient acquisition step of acquiring a coefficient for color transformation based on the position of the input point; a calculation step of calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the plurality of reference points; 13. An image processing method comprising:

15. The computer of the image processing device a data acquisition means for acquiring first data representing the colors of the plurality of color charts measured in an image obtained by capturing the plurality of color charts, and second data representing the color of a target at a measurement point where the first data was measured; An input point acquisition means for acquiring an input point on a color space; a reference point selection means for acquiring a plurality of reference point candidates corresponding to the plurality of color charts from the first data, and selecting, as reference points, a plurality of the reference point candidates that are close to the input point in color space from the plurality of reference point candidates; A coefficient acquisition means for acquiring a coefficient of a color conversion based on the position of the input point; a calculation means for calculating a color conversion result at the input point based on the color conversion coefficients and the second data corresponding to the plurality of reference points; A program to function as a

Citation Information

Patent Citations

  • Color processing parameter compilation device, image input device and image processing output system

    JP2004153684A