Color correction device and program therefor
The color correction device uses a three-dimensional shape model to analyze and adjust pixel values across multi-view camera images, addressing the limitations of chart-based methods by achieving consistent color correction without dedicated charts.
Patent Information
- Application Number
- JP2023074495
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-08
AI Technical Summary
Existing color correction methods for multi-view camera images require the use of dedicated charts, which lack versatility and often fail to achieve satisfactory color matching for specific hues, especially when adjusting white balance.
A color correction device that utilizes a three-dimensional shape model generated from multi-view camera images to select three-dimensional points, perform regression analysis on pixel value pairs, and apply conversion coefficients to match the color tone of target images without relying on charts.
Enables accurate color matching between multi-view camera images without the need for charts, ensuring consistent color correction across different hues through regression analysis.
Smart Images

Figure 2024158883000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a color correction device and a program for correcting the colors of a plurality of images captured by a multi-view camera. [Background technology]
[0002] Photogrammetry is a technology that generates a three-dimensional shape model of a subject from images taken with a multi-view camera. In this photogrammetry technology, when adding a color pattern (texture) to a three-dimensional shape model, the pixel values of images taken with a multi-viewpoint camera are referenced and pasted (mapped). Depending on the part of the three-dimensional shape, pixel values of images taken with a camera with a different viewpoint may be used. For example, pixel values of a camera with a viewpoint that faces each part of the subject as directly as possible are used. In this case, if the color matching between the cameras is imperfect, the colors of corresponding parts in the captured images will be different, and when the generated 3D shape model is rendered, CG (Computer Graphics) will be generated that has a patchwork of color differences between different parts.
[0003] In order to correct such color differences between images, a method has been disclosed in which a chart painted with multiple specified colors, such as a Macbeth chart, is photographed with a camera, and the photographed image is corrected so that each paint color is output as a specified pixel value (see, for example, Patent Document 1). Furthermore, in order to correct the color of an image, a method has been disclosed in which skin-colored and gray areas in the image are detected and the white balance is adjusted (see, for example, Patent Document 2). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 6443230 [Patent Document 2] Patent No. 3819302 Summary of the Invention [Problem to be solved by the invention]
[0005] The technique of color correction using a chart makes it possible to match colors for a variety of hues. However, this technique has the problem of lacking versatility because it requires the use of a dedicated chart. In addition, this correction involves photographing the subject area to be color-corrected and the chart area as a single image, and then correcting the color of the subject area. Therefore, this technique is time-consuming in that it is necessary to specify or detect where each paint color of the chart is located within the image to be corrected. Furthermore, the method of adjusting the white balance performs uniform color matching for various hues, and therefore has the problem that satisfactory color matching cannot necessarily be achieved for a specific hue.
[0006] Therefore, an object of the present invention is to provide a color correction device and a program therefor that are capable of adjusting the colors of images captured by a multi-view camera without using a chart. [Means for solving the problem]
[0007] In order to solve the above-mentioned problems, the color correction device of the present invention is a color correction device that corrects the color of a captured image using a three-dimensional shape model, which is a three-dimensional point cloud generated from multiple captured images taken by a multi-viewpoint camera, and is configured to include a three-dimensional point selection unit, a reference captured image reference unit, a target captured image reference unit, a pixel value pair generation unit, a regression calculation unit, and a color conversion unit.
[0008] In such a configuration, the color correction device selects a plurality of 3D points from the 3D shape model by the 3D point selection unit. The 3D points may be selected artificially from the 3D point cloud, or may be selected at a predetermined interval or at random intervals from the 3D point cloud. Then, the color correction device uses the reference captured image reference unit to refer to a predetermined reference captured image among the captured images, and identifies the pixel value of the reference captured image corresponding to the selected three-dimensional point as the reference pixel value. In addition, the color correction device uses the target captured image reference unit to refer to the target captured image to be corrected among the captured images, and identifies the pixel value of the target captured image corresponding to the selected three-dimensional point as the target pixel value.
[0009] Then, the color correction device generates, by the pixel value pair generating section, a pixel value pair that associates a reference pixel value with a target pixel value that correspond to the same three-dimensional point. The color correction device then performs regression analysis on a plurality of pixel value pairs using the regression calculation unit, and calculates conversion coefficients for converting the target pixel value so as to approximate the target pixel value to the reference pixel value. This allows the regression calculation unit to generate conversion coefficients for matching the colors when the colors of the reference pixel value and the target pixel value differ.
[0010] Then, the color correction device converts the pixel value of each pixel of the target captured image using the conversion coefficients by the color conversion unit. This enables the color correction device to generate a color-corrected image in which the color tone of the target captured image matches the color tone of the reference captured image. The color correction device can be operated by a program for causing a computer to function as the color correction device. Effect of the Invention
[0011] According to the present invention, since the difference in color tone can be analyzed by regression analysis, it is possible to match the colors of images captured by a multi-view camera without using a chart. [Brief description of the drawings]
[0012] [Figure 1] 1 is a block diagram showing a configuration of a color correction device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a three-dimensional shape model configured from a three-dimensional point cloud. [Diagram 3] FIG. 1 is a diagram showing an example of a volumetric studio in which multiple viewpoint cameras are arranged for generating a three-dimensional shape model. [Figure 4] 11 is an explanatory diagram for explaining a process of a projection unit. FIG. [Diagram 5] 4 is a flowchart showing the operation of the color correction device according to the embodiment of the present invention. [Figure 6] 1A and 1B are examples of reference images that the color correction device refers to, where FIG. 1A shows a reference captured image and FIG. [Figure 7] 10 is a diagram illustrating a process for generating pixel value pairs from 3D points of a 3D shape model; FIG. [Figure 8] 1A and 1B are diagrams illustrating an example of a target captured image before color correction and a color-corrected image after color correction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Color correction device configuration] The configuration of a color correction device 1 according to an embodiment of the present invention will be described with reference to FIG. The color correction device 1 corrects the color of a target captured image, for each captured image taken by a multi-viewpoint camera, so as to match the color tone (hue) of a reference captured image.
[0014] The reference captured image is an image captured by a camera that serves as a reference for performing color correction. The target captured image is an image to be subjected to color correction, which has been captured by a camera other than the camera that captured the reference captured image. The camera that captures the reference captured image among the multiple cameras of the multi-view camera may be a predetermined camera, or may be determined by a person looking at the captured image.
[0015] The color correction device 1 includes a three-dimensional point selection unit 10, a reference captured image reference unit 11, a target captured image reference unit 12, a pixel value pair generation unit 13, a pixel value pair memory unit 14, a regression calculation unit 15, and a color conversion unit 16.
[0016] The 3D point selection unit 10 selects 3D points one by one from a 3D shape model composed of a group of 3D points. As shown in Fig. 2, the three-dimensional shape model M is a shape of a subject composed of a three-dimensional point cloud (a set of three-dimensional points). Each three-dimensional point constituting the three-dimensional point cloud is associated with a coordinate value in a three-dimensional (xyz) space. A three-dimensional shape model can be generated by photographing a subject T in a volumetric studio 100 in which cameras C (C1, C2, ...) are arranged at a plurality of predetermined viewpoint positions PO as shown in Fig. 3, and using a known photogrammetry technique from the plurality of photographed images. Note that only two cameras C are shown in Fig. 3, and the others are omitted.
[0017] The 3D point selection unit 10 selects 3D points from the 3D shape model multiple times by changing the location from which the points are selected one by one. In addition, the 3D point selection unit 10 may select 3D points that are artificially specified from outside, or may select 3D points that are specified at a predetermined interval or random intervals by an external sampling device 2. Here, the description will be given assuming that the sampling device 2 sequentially specifies three-dimensional points. The three-dimensional point selection unit 10 outputs the coordinate values of the selected three-dimensional points to the reference captured image referring unit 11 and the target captured image referring unit 12 . Hereinafter, the three-dimensional coordinates of the three-dimensional point selected in the three-dimensional point selection unit 10 for the i-th time (i is an integer equal to or greater than 1) are expressed as x i ∈R 3 (real three-dimensional number space).
[0018] The reference captured image referring unit 11 refers to the reference captured image and specifies pixel values (colors) of the reference captured image corresponding to the 3D points selected by the 3D point selecting unit 10 . The reference captured image referring unit 11 includes a projection unit 110 and a pixel value referring unit 111 .
[0019] The projection unit (first projection unit) 110 virtually projects the three-dimensional points selected by the three-dimensional point selection unit 10 onto the same image plane as the reference captured image, using the camera parameters (reference camera parameters) of the camera that captured the reference captured image, and calculates the image coordinates (two-dimensional coordinates) of the projection position. That is, as shown in FIG. 4, the projection unit 110 projects a virtual camera C having the same reference camera parameters (camera position, pan, tilt, roll, focal length, etc.) as the camera that captured the reference captured image. V Then, the 3D point P of the 3D shape model M is 3D , virtual camera C V is projected onto the two-dimensional image plane (imaging plane) IP at the projection position P 2D Calculate the image coordinates of
[0020] Here, the reference camera parameters are P std , three-dimensional point P 3D The three-dimensional coordinate of x i , the reference camera parameters P std The projection transformation based on the function f(P std , x), the projection unit 110 calculates the projection position P on the image plane IP by the following equation (1): 2D Image coordinate y i ∈R 2 (real two-dimensional number space).
[0021]
number
[0022] For example, the reference camera parameter P std is a perspective projection transformation matrix, the function f in the above formula (1) becomes the following formula (2).
[0023]
number
[0024] The projection unit 110 projects the image coordinate y i to the pixel value reference unit 111. The projection unit 110 projects the three-dimensional point P selected by the three-dimensional point selection unit 10 shown in FIG. 3D But virtual camera C V It is determined whether the point is visible (visible point) from the viewpoint VP, and if it is not a visible point (not a viewpoint), it does not project onto the image plane and outputs to the pixel value pair generation unit 13 a message that no corresponding pixel value exists.
[0025] The determination of the visible points is performed by the projection unit 110 projecting each three-dimensional point P 3D This can be determined by calculating the distance or depth to each point (preferably all three-dimensional points) and performing hidden surface removal (eg, Z-buffer processing). However, in the case where the 3D point selection unit 10 artificially selects only visible points as 3D points, for example, the determination of visible points in the projection unit 110 is not essential.
[0026] A pixel value reference unit (first pixel value reference unit) 111 references the reference captured image and calculates the image coordinate y i This specifies the pixel value (reference pixel value) in The pixel value reference unit 111 is i =[ξ i ,η i ] T If the component of (T indicates transposition) is an integer, the pixel value is expressed as the image coordinate y of the reference image. i The pixel value is specified as
[0027] In addition, the pixel value reference unit 111 i =[ξ i ,η i ] T If the component of is a non-integer, the image coordinate y of the reference image i By referring to the neighboring pixel values and performing interpolation, the image coordinate y of the reference image is calculated. i The pixel value of is identified. For example, the pixel value reference unit 111 calculates the image coordinate y by using nearest neighbor interpolation (zeroth order interpolation) as shown in the following formula (3). i The pixel value I(y i ) is calculated.
[0028]
number
[0029] Also, for example, the pixel value reference unit 111 calculates the image coordinate y i The pixel value I(y i ) is calculated.
[0030]
number
[0031] The interpolation in the pixel value reference unit 111 is not limited to the interpolation according to the above formulas (3) and (4). For example, the pixel value reference unit 111 can use any interpolation method such as bicubic interpolation or windowed sinc interpolation (e.g., Lanczos interpolation). The pixel value reference unit 111 is i The reference pixel value I(y i ) to the pixel value pair generating unit 13. The image coordinate y specified by the pixel value reference unit 111 i The reference pixel value I(y i ) corresponds to the color of the selected 3D point in the reference captured image.
[0032] The target photographed image referring unit 12 refers to the target photographed image and specifies pixel values (colors) of the target photographed image corresponding to the three-dimensional points selected by the three-dimensional point selecting unit 10 . The target captured image referring unit 12 includes a projection unit 120 and a pixel value referring unit 121 .
[0033] The projection unit (second projection unit) 120 virtually projects the three-dimensional points selected by the three-dimensional point selection unit 10 onto the same image plane as the target captured image, using the camera parameters (target camera parameters) of the camera that captured the target captured image, and calculates the image coordinates (two-dimensional coordinates) of the projection position. That is, as shown in FIG. 4, the projection unit 120 projects a virtual camera C having the same target camera parameters (camera position, pan, tilt, roll, focal length, etc.) as the camera that captured the target captured image. V Then, the 3D point P of the 3D shape model M is 3D , virtual camera C V is projected onto the two-dimensional image plane (imaging plane) IP at the projection position P 2D Calculate the image coordinates of
[0034] Here, the target camera parameters are P tgt , three-dimensional point P 3D The three-dimensional coordinate of x i , the target camera parameters P tgt The projection transformation based on the function f(P tgt , x), the projection unit 120 calculates the projection position P 2D Image coordinate z i ∈R 2 (real two-dimensional number space).
[0035]
number
[0036] The projection unit 120 projects the image coordinate z i to the pixel value reference unit 121. In addition, like the projection unit 110, the projection unit 120 projects the three-dimensional point P selected by the three-dimensional point selection unit 10 shown in FIG. 3D But virtual camera C V It is determined whether the point is visible (visible point) from the viewpoint VP, and if it is not a visible point (not a viewpoint), it does not project onto the image plane and outputs to the pixel value pair generation unit 13 a message that no corresponding pixel value exists. However, in the case where the 3D point selection unit 10 artificially selects only visible points as 3D points, for example, the determination of visible points in the projection unit 120 is not essential.
[0037] The pixel value reference unit (second pixel value reference unit) 121 references the target captured image and calculates the image coordinates z i This specifies the pixel value (target pixel value) in The pixel value reference unit 121 is i =[α i ,β i ] T If the component of (T indicates transposition) is an integer, the pixel value is expressed as the image coordinate z i The pixel value is specified as
[0038] In addition, the pixel value reference unit 121 i =[α i ,β i ] T If the component of is a non-integer, the image coordinate z of the target image i By referencing the neighboring pixel values and performing interpolation, the image coordinate z i The pixel value of is identified. For example, the pixel value reference unit 121 calculates the image coordinate z by using nearest neighbor interpolation (zeroth order interpolation) as shown in the following formula (6): i The pixel value J(z i ) is calculated.
[0039]
number
[0040] Also, for example, the pixel value reference unit 121 uses bilinear interpolation to find the image coordinate z i The pixel value J(z i ) is calculated.
[0041]
number
[0042] The interpolation in the pixel value reference unit 121 is not limited to the interpolation according to the above formulas (6) and (7). For example, the pixel value reference unit 121 can use any interpolation method such as bicubic interpolation, windowed sinc interpolation (e.g., Lanczos interpolation), etc. The pixel value reference unit 121 is i The target pixel value J(z i ) to the pixel value pair generating unit 13. The image coordinate z specified by the pixel value reference unit 121 i The target pixel value J(z i ) corresponds to the color of the selected 3D point in the captured image of the target.
[0043] The pixel value pair generation unit 13 generates a pixel value pair that corresponds a reference pixel value identified by the reference captured image reference unit 11 corresponding to the selected three-dimensional point and a target pixel value identified by the target captured image reference unit 12. The pixel value pair generating unit 13 sequentially stores the generated pixel value pairs in a pixel value pair storage unit 14 .
[0044] However, if the pixel value pair generation unit 13 is notified by the reference captured image reference unit 11 that there is no reference pixel value corresponding to the three-dimensional point selected, or by the target captured image reference unit 12 that there is no target pixel value corresponding to the three-dimensional point selected, the pixel value pair generation unit 13 will not generate a pixel value pair corresponding to the three-dimensional point. This improves the accuracy of regression analysis in the regression calculation section 15, which will be described later, and enables the color conversion section 16 to perform color matching more precisely.
[0045] The pixel value pair generating unit 13 instructs the regression calculation unit 15 to start calculation when it receives an external instruction to end the selection of three-dimensional points. For example, the pixel value pair generating section 13 instructs the regression calculation section 15 to start calculation when the sampling device 2 instructs it to end, or when an external switch or the like is used to manually instruct it to end.
[0046] In order to generate conversion coefficients in regression calculation unit 15 described later, the number of pixel value pairs is preferably 3 or more. Therefore, for example, pixel value pair generation unit 13 may display a warning on a display device (not shown) if the number of pixel value pairs is less than 3 at the stage when an end command is received.
[0047] The pixel value pair storage unit 14 stores the pixel value pairs generated by the pixel value pair generation unit 13. The pixel value pair storage unit 14 can be configured with a general storage medium such as a semiconductor memory.
[0048] The regression calculation unit 15 performs regression analysis on the pixel value pairs stored in the pixel value pair storage unit 14, and calculates conversion coefficients for converting the target pixel value to approximate a reference pixel value. Here, the number of pixel value pairs stored in the pixel value pair storage unit 14 is set to N (N∈Z [integer], N≧3). The regression calculation unit 15 calculates the reference pixel value I(y i ) and the target pixel value J(z i ) and the pixel value pair (I(y i ),J(z i )) i∈{1,2,…,N} Based on this, a conversion coefficient is calculated.
[0049] Here, the reference pixel value I(y i ) and the target pixel value J(z i ) has color information of multiple channels. For example, the reference pixel value I(y i ) has red, green and blue color information, the red component is I R (y i ), green component is I G (y i ), and the blue component is I B (y i ) Similarly, the target pixel value J(z i ) has red, green and blue color information, the red component is J R (z i ), green component is J G (z i ), and the blue component is J B (zi )
[0050] The regression calculation unit 15 calculates a conversion coefficient for converting the target pixel value to the reference pixel value by, for example, linear regression as a regression analysis. In this case, the conversion coefficient indicates three coefficients of a regression hyperplane in a four-dimensional space and a constant term for each color component.
[0051] Specifically, the regression calculation unit 15 uses the matrix A as shown in equation (8) and the matrix b as shown in equation (9), and calculates the conversion coefficient [m 11 m 12 m 13 m 14 m 21 m 22 m 23 m 24 m 31 m 32 m 33 m 34 ] T Calculate.
[0052]
number
number
number
[0053] The regression calculation unit 15 outputs the calculated conversion coefficients to the color conversion unit 16 . The color conversion section 16 uses the conversion coefficients calculated by the regression calculation section 15 to convert the pixel values of the pixels of the target captured image (target pixel values).
[0054] Here, the image coordinates of the target image are z∈R 2 Let J(z) be the pixel value in (real two-dimensional number space), and the red component of J(z) be J R (z), green component is J G (z), blue component is J B (z). The image coordinates of the color-corrected image, z∈R 2Let the pixel value of z be K(z), and the red component of z be K R (z), green component is K G (z), blue component is K B Let it be (z). At this time, the color conversion unit 16 performs the conversion of the following equation (11) for each pixel to generate a color-corrected image.
[0055]
number
[0056] As a result, the color conversion unit 16 performs color correction on the target captured image to generate a color-corrected image whose color is made to approximate that of the reference captured image.
[0057] With the above-described configuration, the color correction device 1 can correct the color of the target captured image for each captured image taken by the multi-viewpoint camera so as to match the color of the reference captured image. The color correction device 1 can be operated by a program (color correction program) for causing a computer (not shown) to function as each of the above-mentioned units.
[0058] [Color correction device in action] Next, the operation of the color correction device 1 according to the embodiment of the present invention will be described with reference to Fig. 5 (and for the configuration, with reference to Fig. 1 as appropriate). Note that for specific processing contents in the operation of the color correction device 1, with reference to Figs. 6 to 8 as appropriate. Fig. 6 shows an example of a reference image referred to by the color correction device 1, where (a) shows a reference captured image and (b) shows a target captured image. Fig. 7 is a diagram showing a schematic diagram of a process for generating pixel value pairs from three-dimensional points. Fig. 8 is a diagram showing an example of a target captured image before color correction and a color-corrected image after color correction.
[0059] In step S1, the 3D point selection unit 10 selects 3D points one by one from the 3D shape model M (FIG. 2). In step S2, the projection unit 110 of the reference captured image reference unit 11 virtually projects the three-dimensional points selected in step S1 onto the same image plane as the reference captured image, using the camera parameters (reference camera parameters) of the camera that captured the reference captured image. That is, the projection unit 110 projects a three-dimensional point P 3D The image plane IP std By projecting the image onto the projection position P 2Dstd Calculate the two-dimensional image coordinates of At this time, the projection unit 110 performs hidden surface processing to project the three-dimensional point P 3D If is not possible, the image plane IP std Projection to the
[0060] In step S3, the pixel value reference section 111 of the reference captured image reference section 11 refers to the reference captured image and identifies the pixel value (reference pixel value) at the projection position projected in step S2. That is, the pixel value reference unit 111, as shown in FIG. std Projected position P 2Dstd Same as reference image I std The pixel value of the image coordinate of the 3D point P 3D is identified as the pixel value (reference pixel value) corresponding to At this time, the projection position P 2Dstd If the image coordinate components of are not integers, the pixel value reference unit 111 calculates the projection position P 2Dstd Nearby reference image I std From the pixel value of 2Dstd The pixel values of are calculated by interpolation.
[0061] In step S4, the projection unit 120 of the target captured image reference unit 12 virtually projects the three-dimensional point selected in step S1 onto the same image plane as the target captured image, using the camera parameters (target camera parameters) of the camera that captured the target captured image. That is, the projection unit 120 projects a three-dimensional point P 3D The image plane IP tgtBy projecting the image onto the projection position P 2Dtgt Calculate the two-dimensional image coordinates of At this time, the projection unit 120 performs hidden surface processing to project the three-dimensional point P 3D If is not possible, the image plane IP tgt Projection to the
[0062] In step S5, the pixel value reference section 121 of the target captured image reference section 12 refers to the target captured image and identifies the pixel value (target pixel value) at the image coordinates projected in step S4. That is, the pixel value reference unit 121, as shown in FIG. tgt Projected position P 2Dtgt Same subject image I tgt The pixel value of the image coordinate of the 3D point P 3D The pixel value (target pixel value) corresponding to At this time, the projection position P 2Dtgt If the image coordinate components of are not integers, the pixel value reference unit 121 calculates the projection position P 2Dtgt Nearby object image I tgt From the pixel value of 2Dtgt The pixel values of are calculated by interpolation. The order of steps S2, S3 and steps S4, S5 may be reversed, or steps S2, S3 and S4, S5 may be performed in parallel.
[0063] In step S6, the pixel value pair generating unit 13 determines whether or not pixel values (reference pixel value and target pixel value) corresponding to the 3D point have been specified in steps S3 and S5. Here, it is determined that the pixel values have been specified only when both the reference pixel value and the target pixel value have been specified.
[0064] If a pixel value corresponding to the three-dimensional point is identified (Yes in step S6), in step S7, the pixel value pair generation unit 13 generates a pixel value pair that corresponds the reference pixel value to the target pixel value, and stores it in the pixel value pair memory unit 14. As a result, as shown in FIG. std Projection position P at2Dstd The reference pixel value of and the target image I tgt Projection position P at 2Dtgt The target pixel value of the same 3D point P 3D are stored in the pixel value pair storage unit 14 as a pixel value pair corresponding to On the other hand, if a pixel value corresponding to the 3D point has not been specified (No in step S6), the pixel value pair generating unit 13 advances the operation to step S8 without performing any processing.
[0065] In step S8, the pixel value pair generating unit 13 determines whether or not an instruction to end the selection of 3D points has been given. If the end of the selection of three-dimensional points has not been instructed (No in step S8), color correction device 1 returns to step S1 and continues the operation. On the other hand, if an instruction to end the selection of three-dimensional points is given (Yes in step S8), in step S9, the regression calculation unit 15 performs regression analysis on the pixel value pairs stored in the pixel value pair memory unit 14, and calculates a conversion coefficient for converting the target pixel value to an approximation to the reference pixel value.
[0066] In step S10, the color conversion unit 16 uses the conversion coefficient calculated in step S9 to convert the pixel values of each pixel in the target captured image (target pixel values) to generate a color-corrected image. As a result, the color conversion unit 16 converts the target captured image I tgt By converting the pixel values of each pixel in the reference image I shown in Fig. 6(a), std Color-corrected image I cor can be generated. Although the configuration and operation of the color correction device 1 according to the embodiment of the present invention have been described above, the present invention is not limited to this embodiment.
[0067] (Variation 1) Here, the captured images (reference captured image, target captured image) have been described as examples of multi-channel color images. However, the captured images (reference captured image, target captured image) may be one-channel monochrome images. In this case, the regression calculation unit 15 uses the reference pixel value I(y i ) and the target pixel value J(z i ) to one, and the number of matrix elements is reduced for calculation. Similarly, in color conversion unit 16, the number of matrix elements is reduced by setting the components of pixel value K(z) and pixel value J(z) in equation (11) to one, and the number of matrix elements is reduced for calculation.
[0068] (Variation 2) Here, the regression calculation unit 15 calculates the conversion coefficients by linear regression as the regression analysis. However, the regression calculation unit 15 is not limited to linear regression such as equation (11) and can use any nonlinear regression. In this case, instead of equations (8) to (10), the regression parameters can be calculated numerically or approximately by a general solution method for a nonlinear least squares problem such as the steepest descent method or the Levenberg-Marquardt method, or an optimization method such as simulated annealing or a genetic algorithm.
[0069] (Variation 3) Also, here, the three-dimensional shape model has been described as an example in which the entire subject is modeled from multiple images captured by a camera C (multiple viewpoint camera) placed at different viewpoint positions in the volumetric studio 100 shown in Figure 3. However, the three-dimensional shape model may be shape data of only a part of the subject, for example, the front surface of the subject. [Explanation of symbols]
[0070] 1 Color correction device 10 3D point selection section 11 Reference image reference section 110 Projection section (first projection section) 111 pixel value reference unit (first pixel value reference unit) 12 Target image reference section 120 Projection section (second projection section) 121 pixel value reference unit (second pixel value reference unit) 13 Pixel value pair generator 14 Pixel value pair memory section 15 Regression calculation section 16 Color conversion section M 3D shape model I std Reference image I tgt Target image I cor Color corrected image
Claims
1. 1. A color correction device that corrects colors of a captured image by using a three-dimensional shape model that is a three-dimensional point cloud generated from a plurality of captured images captured by a multi-view camera, comprising: a 3D point selection unit for selecting a plurality of 3D points from the 3D shape model; a reference image reference unit that refers to a reference captured image that serves as a predetermined reference among the captured images and specifies, as a reference pixel value, a pixel value of the reference captured image that corresponds to the selected three-dimensional point; a target image reference unit that refers to a target image to be corrected among the captured images and identifies a pixel value of the target image corresponding to the selected three-dimensional point as a target pixel value; a pixel value pair generating unit that generates a pixel value pair in which the reference pixel value and the target pixel value corresponding to the same three-dimensional point are associated with each other; a regression calculation unit that performs regression analysis on the plurality of pixel value pairs and calculates a conversion coefficient for converting the target pixel value to approximate the reference pixel value; a color conversion unit that converts a pixel value of each pixel of the target captured image using the conversion coefficient to generate a color-corrected image; A color correction device comprising:
2. The reference photographed image referring unit includes: a first projection unit that virtually projects the three-dimensional point onto an image surface by using camera parameters of a camera that captured the reference captured image, and calculates image coordinates of the projection position; a first pixel value reference unit that refers to the reference captured image and specifies a pixel value at an image coordinate calculated by the first projection unit as the reference pixel value, The target photographed image referring unit includes: a second projection unit that virtually projects the three-dimensional point onto an image surface by using camera parameters of a camera that captured the target image, and calculates image coordinates of the projection position; a second pixel value reference unit that refers to the target captured image and identifies a pixel value at the image coordinates calculated by the second projection unit as the target pixel value; 2. The color correction device according to claim 1, further comprising:
3. The first projection unit and the second projection unit determine whether the three-dimensional point is a visible point by hidden surface removal, The color correction device according to claim 2, characterized in that the pixel value pair generation unit corresponds the reference pixel value corresponding to the three-dimensional point determined to be a visible point by both the first projection unit and the second projection unit to the target pixel value.
4. the first pixel value reference unit, when a component of the image coordinate calculated by the first projection unit is a non-integer, calculates the reference pixel value by performing interpolation by referring to pixel values of the reference captured image in the vicinity of the image coordinate; The color correction device described in claim 2, characterized in that when a component of the image coordinate calculated by the second projection unit is a non-integer, the second pixel value reference unit calculates the reference pixel value by performing interpolation by referring to pixel values of the target captured image in the vicinity of the image coordinate.
5. 5. The color correction device according to claim 1, wherein the regression calculation unit calculates the conversion coefficients by linear regression.
6. A program for causing a computer to function as the color correction device according to claim 1.
Citation Information
Patent Citations
Line of sight detection apparatus and line of sight input apparatus using the same
JP1989043230A
White balance correction method
JP3819302B2
Cited By
Information processing device, information processing method, and program
WO2026204531A1
