Display control device, control method for the same, and program

Abstract

To make it possible to facilitate discrimination of color residuals between a plurality of imaging apparatuses on a site where the plurality of imaging apparatuses is used.SOLUTION: A display control device 200 has acquisition means 202b, color residual generation means 208, 209, and display generation means 210. The acquisition means 202b is for acquiring an image from an adjusting imaging apparatus 101 on which color matching with a reference imaging apparatus 100 has been performed. The color residual generating means 208, 209 is for generating on the basis of the image acquired from the acquisition means 202b color residual information indicating color difference between an image captured by an adjusting imaging apparatus 101 on which a predetermined color matching has been performed and an image that is captured by a reference imaging apparatus 100 and remains in the image. The display generating means 210 is for generating display information on the basis of the color residual information.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to a display control device, a control method thereof, and a program.

Background Art

[0002] Patent Document 1 discloses an imaging device. When a pixel of an image captured by an imaging system is a saturated pixel, this imaging device assigns a pixel value of a saturated pixel warning color. Further, when a pixel of an image captured by the imaging system is a pixel within a display color gamut, the imaging device assigns a pixel value obtained by converting an imaging color gamut to a display color gamut to the pixel within the display color gamut. Further, when a pixel of an image captured by the imaging system is a pixel outside the display color gamut, the imaging device assigns a pixel value of a pixel warning color outside the display device color gamut. Patent Document 2 discloses a technique of adding a difference between a plurality of images captured in time series to an image with reduced contrast of the image to make it easier to visually recognize a time-series change portion of a subject.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Incidentally, such imaging devices, for example, receive light through an optical system using an image sensor such as a CCD sensor or CMOS sensor to capture still or moving images. Furthermore, imaging devices are sometimes used together with other imaging devices. In this case, for example, based on differences in the light-receiving characteristics between the image sensors of multiple imaging devices, the images captured by multiple imaging devices may appear in different colors even though they are capturing the same subject. When there is a difference in color between the images captured by multiple imaging devices (hereinafter, the difference in color that occurs between imaging devices will be called a color difference), one imaging device can be used as a reference imaging device and the other imaging devices as adjustment imaging devices, and a correction lookup table can be generated for use in the adjustment imaging device. This is thought to make it possible to match the color of the adjustment imaging device to that of the reference imaging device. However, even if such a correction lookup table is used in the adjustment imaging device, it is not guaranteed that all the colors in the image captured by the adjustment imaging device will match all the colors in the image captured by the reference imaging device. In actual field use of multiple imaging devices, even with the use of a correction lookup table, color residuals may remain in some subjects. Furthermore, in cases where such color residuals exist, it requires considerable effort for the user to visually compare the image captured by the reference imaging device with the image captured by the adjustment imaging device to identify these color residuals.

[0005] As explained in the example above, in situations where multiple imaging devices are used, it is necessary to be able to easily distinguish between color residuals between the multiple imaging devices. [Means for solving the problem]

[0006] The display control device according to the present invention includes: an acquisition means for acquiring an image from an adjustment imaging device that has performed a predetermined color matching with a reference imaging device; a color residual generation means for generating color residual information that indicates the color difference between the image captured by the adjustment imaging device that has performed the predetermined color matching and the image captured by the reference imaging device that remains in the image, based on the image acquired by the acquisition means; and a display generation means for generating display information based on the color residual information. [Effects of the Invention]

[0007] In the present invention, when using a reference imaging device and an adjustment imaging device in a field setting, it is possible to easily distinguish between multiple imaging devices. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the configuration of a color difference correction system for multiple imaging devices, including a display control device according to the first embodiment of the present invention. [Figure 2] This is a block diagram of the main components of the display control device shown in Figure 1. [Figure 3] This is a flowchart for generating a correction LUT using the display control device shown in Figure 1. [Figure 4] This is an example of a combination of the reference camera's captured image and the target camera's captured image displayed on the screen. [Figure 5] Figure 4 shows an example of a combination of RGB values ​​for a color patch in the captured image. [Figure 6] This is an example of the XYZ values ​​of a color patch obtained based on the RGB values ​​of the color patch in Figure 5. [Figure 7] This is an explanatory diagram of an example of a correction LUT generated by the display control device shown in Figure 1. [Figure 8] This is a flowchart for generating color residual information for a target camera using a corrective LUT. [Figure 9] Figure 8 shows an example of RGB value combinations for color patches in the color-matched image of the target camera. [Figure 10] This is an example of the XYZ values ​​of a color patch after color matching, obtained based on the RGB values ​​of the color patch in Figure 9. [Figure 11] This is a flowchart for generating a display image to show color residual information. [Figure 12]It is a table showing the correspondence between the range of color residuals of color residual information and the conversion colors used as the colors of the pixels included in each range. [Figure 13] It is an explanatory diagram of an example of a display image for displaying color residual information based on a captured image of a target camera using a correction LUT. [Figure 14] It is a flowchart for generating color residual information by the display control device according to the second embodiment of the present invention. [Figure 15] It is an example of a combination of a captured image of a target camera using a correction LUT and a color distribution image showing the distribution of color residuals generated separately therefrom. [Figure 16] It is an example of a color distribution image showing the distribution of color residuals when this embodiment is applied. [Figure 17] It is an example of a vector scope image for displaying color residual information. [Figure 18] It is a flowchart for outputting a message when the value of color residual information is greater than or equal to a threshold. [Embodiments for Carrying Out the Invention]

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments.

[0010] [First Embodiment] Figure 1 is a configuration diagram of a color difference correction system 1 of multiple imaging devices, including a display control device 200 according to a first embodiment of the present invention. The color difference correction system 1 in Figure 1 has a display control device 200. The display control device 200 is connected by image lines to multiple imaging devices that jointly image the same type of subject, such as a color chart, including a reference camera 100, which is an example of a reference imaging device, and a target camera 101, which is an example of an adjustment imaging device. The reference camera 100 outputs its captured image to the display control device 200 via image lines. The target camera 101 outputs its captured image to the display control device 200 via image lines. Here, the images captured by the reference camera 100 and the target camera 101 may be still images or moving images such as video. Based on a comparison of the input captured images, the display control device 200 generates a color matching correction lookup table (LUT) that can be used by the target camera 101. By using the correction LUT generated in this way, the target camera 101 can capture an image with suppressed color difference from the reference camera 100. This process of suppressing color differences between devices using a correction LUT is called color matching. When a target camera 101 that has been color-matched with the reference camera 100 is used together with the reference camera 100 in the same imaging environment, it becomes possible to capture an image with a color closer to that of the image captured by the reference camera 100.

[0011] Figure 2 is a block diagram of the main configuration of the display control device 200 shown in Figure 1. The first input terminal 221 can be connected to the reference camera 100 via an image line. The second input terminal 222 can be connected to the target camera 101 via an image line. The memory 223 stores the program executed by the control unit 203 and various data. The memory 223 may be composed of non-volatile semiconductor memory, SSD, HDD, RAM, etc. The RAM may be allocated a work area used when executing the program. Non-volatile semiconductor memory, SSD, HDD, etc., can record programs and data non-volatilely.

[0012] The control unit 204 controls the entire display control device 200. It reads a program from the memory 223 and controls each part of the device. The control unit 204 may be, for example, a CPU.

[0013] The image receiving unit 201b receives an image from the target camera 101 connected to the second input terminal 222 by an image line and outputs it to the image capture unit 202b. The image capture unit 202b captures the received image of the target camera 101. Then, the image capture unit 202a outputs the captured image to the control unit 203. As an acquisition means, the image capture unit 202a can receive and acquire an image before color matching or an image after color matching from the target camera 101. Here, the image capture unit 202b may capture an image of one frame included in a moving image such as a video by recording it in a frame memory. The image of one frame may be an interesting image, but a progressive image is desirable.

[0014] The image receiving unit 201a receives an image from the reference camera 100 connected to the first input terminal 221 by an image line and outputs it to the image capture unit 202a. The image capture unit 202a captures the received image of the reference camera 100. Then, the image capture unit 202a outputs the captured image to the control unit 203.

[0015] The UI operation unit 204 receives an operation from the user and outputs an instruction corresponding thereto to the control unit 203. For example, a pointing device, a tablet, etc. that the user operates may be connected to the UI operation unit 204. The UI operation unit 204 may receive, for example, an operation for executing color matching, an operation for switching display, etc. Also, the UI operation unit 204 may receive an operation of rectangularly selecting each color patch of an image. In this case, the UI operation unit 204 may output the rectangular range from the start point to the end point of the color patch selected by the user to the control unit 203 as the area information of the color patch.

[0016] The control unit 203 controls various parts of the display control device 200 to realize various recording controls, display controls, image processing, etc. For example, as shown in Figure 1, the control unit 203 displays the received image from the reference camera 100 and the image from the target camera 101 side by side on the display unit 211. The user can compare the image from the reference camera 100 and the image from the target camera 101 by visually looking at the display unit 211. When color patch area information is input from the UI operation unit 204, the control unit 203 displays a rectangular frame superimposed on the image displayed on the display unit 211. The control unit 203 may generate warning or message screens and display them on the display unit 211. In addition, when generating a correction LUT, the control unit 203 outputs the image from the reference camera 100, the image from the target camera 101, and the color patch area information for them to the correction LUT generation unit 205. Furthermore, when generating color residual information, the control unit 203 outputs the image from the target camera 101 to the color matching determination unit 209. The control unit 203 may also generate a display image for displaying the color residual information and display it on the display unit 211. This display image is display information generated based on the color residual information. The control unit 203 may also switch the display on the display unit 211 in response to user operations acquired by the UI operation unit 204. This allows the control unit 203 to switch, for example, between generating a correction LUT and displaying color residual information based on user operations.

[0017] The correction LUT generation unit 205 obtains information for generating a correction LUT from the control unit 203 and generates a correction LUT to match the color of the target camera 101 to the color of the reference camera 100. The information obtained from the control unit 203 may include the image of the reference camera 100, the image of the target camera 101, and area information of the corresponding color patches. The correction LUT generation unit 205 then outputs the generated correction LUT to the correction LUT output unit 206. The correction LUT generation unit 205 also outputs the RGB values ​​of the color patches of each camera used for generating the correction LUT, the image of the target camera 101, etc., to the camera characteristics information.

[0018] The correction LUT output unit 206 may convert the correction LUT input from the correction LUT generation unit 205 into a file format that the target camera 101 can import. The correction LUT file formed by the correction LUT output unit 206 may be recorded on, for example, a portable media (not shown) and copied to the target camera 101 for import. This enables the target camera 101 to capture an image with color matching using the correction LUT. The correction LUT output unit 206 may also transmit the correction LUT file to the target camera 101 via wireless communication or the like. Furthermore, if the target camera 101 and the display control device 200 are connected by a LAN cable or the like, the correction LUT output unit 206 may transmit the correction LUT file to the target camera 101.

[0019] The camera characteristic information holding unit 207 records and holds camera characteristic information, and may be an independent storage medium such as RAM, or it may use a part of the memory 223. The camera characteristic information may include various information related to color matching of the target camera 101. For example, the camera characteristic information may include information about the reference camera 100 and information about the target camera 101 related to color matching. Here, the camera characteristic information may include, for example, the RGB values ​​of each color patch when color matching was performed between the reference camera 100 and the target camera 101. The camera characteristic information may also include an image of the reference camera 100 used for color matching with the target camera 101.

[0020] The color residual information generation unit 208 calculates the color residual between each pixel of the image of the target camera 101 and the reference camera 100, based on the camera characteristic information held by the camera characteristic information holding unit 207, and generates color residual information. The color residual information generation unit 208 may then generate color residual information for the color difference remaining in the image of the target camera 101 after color matching, using the camera characteristic information held by the camera characteristic information holding unit 207. Here, the color residual information generation unit 208 may calculate the color difference between each pixel of the image of the target camera 101 and the reference camera 100, which is held by the camera characteristic information holding unit 207. As a result, the color residual information generation unit 208, as a color residual generation means, generates color residual information that shows the color residual between the image of the target camera 101 after color matching and the reference camera 100. Here, the color residual information may show the color residual of each pixel value remaining between each pixel value of the image of the target camera 101 after color matching and each pixel value of the image of the reference camera 100.

[0021] The color matching determination unit 209 determines the degree of color matching between each pixel value of the image from the target camera 101 and the pixel value of the image from the reference camera 100. Here, the color matching determination unit 209 may output each pixel value of the image from the target camera 101 to the color residual information generation unit 208, and determine the degree of color matching based on the color residual information generated by the color residual information generation unit 208. The color matching degree information generation unit 210 generates a display image for displaying the color residual information for the image from the target camera 101 that has undergone color matching. For example, the color matching degree information generation unit 210 may convert each pixel value of the corresponding image from the target camera 101 according to the degree of color matching determined by the color matching determination unit 209, and generate a display image for displaying the color residual information. In this case, if the degree of color matching is lower than a threshold, the color matching degree information generation unit 210 may convert the pixel value of that pixel to a pixel value for displaying color residuals. As a result, the color matching information generation unit 210 generates display information based on color residual information as a display generation means. The display unit 211 has, for example, a monitor and displays an image on the monitor. For example, the display unit 211 may display a warning display received from the control unit 203 or a color residual display image received from the color matching information generation unit 210. Note that the image processing device is not limited to the configuration shown in Figure 2. For example, each control and processing can be performed by a single piece of hardware, or multiple pieces of hardware may share the processing and function as a single means. Also, one piece of hardware may function as multiple means depending on the program to be executed. The same applies to storage processing; various types of data can be stored in a single piece of hardware, such as memory 223, or multiple recording media may be provided depending on the type of storage processing. Furthermore, the display unit 211 does not necessarily have to be part of the display control device 200; the display control device 200 only needs to have a minimum interface with the display unit 211.

[0022] First, we will explain the process of generating a color-matching correction LUT that can be used with the target camera 101. Figure 3 is a flowchart for generating the correction LUT by the display control device 200 shown in Figure 1. Figure 3 is a flowchart for the display control device 200 to generate a correction LUT that matches the color of the target camera 101 to the color of the reference camera 100 by comparing the image of the reference camera 100 with the target image. The user operates the UI operation unit 204 to generate the correction LUT. For example, the user selects the item "Generate Color-Matching LUT" from the menu screen. Based on the input from the UI operation unit 204, the control unit 203 starts the correction LUT generation process by the display control device 200.

[0023] Figure 4 shows an example of a combination of the image 400a of the reference camera 100 and the image 400b of the target camera 101, displayed on the display unit 211. Each image 400a and 400b in Figure 4 contains components of predetermined color patches 401, 402, and 403. Color patch 401 is red. Color patch 402 is green. Color patch 403 is blue. The user operates the UI operation unit 204 to generate a correction LUT. Based on the input from the UI operation unit 204, the control unit 203 selects the regions of each color patch 401, 402, and 403 in each image 400a and 400b. Subsequently, the control unit 203 causes each part of the display control device 200 to execute the control shown in Figure 3. The control unit 203 outputs the region information of the selected color patches, along with each image 400a and 400b, to the correction LUT generation unit 205. This starts the control process. Ideally, each image 400a and 400b used for comparison to generate the correction LUT should include the specified color patches 401, 402, and 403, but images that do not include these patches can still be used to generate the correction LUT.

[0024] In step S301 of Figure 3, the correction LUT generation unit 205 extracts RGB values ​​indicating the color of the color patch in each image 400a and 400b. The correction LUT generation unit 205 may extract the RGB values ​​using the image 400a of the reference camera 100, the image 400b of the target camera 101, and the region information 401, 402, and 403 of each color patch, which are received from the control unit 203. The correction LUT generation unit 205 then stores the extracted RGB values ​​of the color patches in each image 400a and 400b as camera characteristic information in the camera characteristic information holding unit 207. The correction LUT generation unit 205 may extract the average value of multiple pixel values ​​within the specified region 401, 402, and 403 of the color patch as the RGB value, or it may extract the pixel value at the center of the region as the RGB value.

[0025] In step S302, the correction LUT generation unit 205 converts the RGB values ​​of each color patch 401, 402, and 403 to XYZ values. The correction LUT generation unit 205 may convert the RGB values ​​to XYZ values ​​using, for example, the following equation 1. The following equation 1 is a matrix and performs the conversion using a conversion matrix (RGB to XYZ conversion matrix) for converting RGB values ​​to XYZ values. The "RGB to XYZ conversion matrix" can be generated by calculation using, for example, the xy coordinates of the display color gamut of the display control device 200 and the xy coordinates of the white point. In addition, it is preferable to substitute the normalized values ​​of the RGB values ​​of the color patches 401, 402, and 403 for (r,g,b) in the following equation 1. As a result, for example, the RGB values ​​of the color patches 401, 402, and 403 in each image shown in Figure 5 can be converted to the XYZ values ​​of the color patches 401, 402, and 403 shown in Figure 6.

[0026]

number

[0027] Figure 5 shows an example of RGB value combinations for color patches 401, 402, and 403 in the captured images 400a and 400b of Figure 4. Figure 6 shows an example of XYZ values ​​of a color patch obtained based on the RGB values ​​of color patches 401, 402, and 403 in Figure 4. Here, the calculation matrix R is obtained by converting the RGB values ​​of color patches 401, 402, and 403 of the reference camera 100 to XYZ. Similarly, the calculation matrix T is obtained by converting the RGB values ​​of color patches 401, 402, and 403 of the target camera 101 to XYZ.

[0028] In step S303 of Figure 3, the correction LUT generation unit 205 calculates a correction matrix M to match the color of the target camera 101 to the color of the reference camera 100. The correction matrix M is a matrix for correcting the XYZ values ​​of the target camera 101 to the XYZ values ​​of the reference camera 100. The correction matrix M can be calculated, for example, by the following equation 2.

[0029]

number

[0030] In step S304, the correction LUT generation unit 205 generates a correction LUT using the correction matrix M calculated in step S303. The correction LUT generation unit 205 outputs the generated correction LUT to the correction LUT output unit 206. This completes the process shown in Figure 3.

[0031] Figure 7 is an explanatory diagram of an example of a correction LUT generated by the display control device 200 in Figure 1. The correction LUT in Figure 7 converts the RGB values ​​(In) on the left side of the figure to the XYZ values ​​(Out) on the right side of the figure. The correction LUT may consist of 17 × 17 × 17 grid points. The RGB values ​​(In) of each grid point in the correction LUT can be converted to XYZ values ​​using the RGB to XYZ conversion matrix described above. Then, by using the XYZ values ​​of these grid points and the correction matrix M described above, the corrected XYZ values ​​of the grid points can be calculated. Furthermore, by performing calculations using the inverse matrix of the RGB to XYZ conversion matrix, the corrected RGB values ​​(Out) corresponding to the RGB values ​​(In) of each grid point in the correction LUT can be calculated. In the correction LUT in Figure 7, the RGB values ​​(In) and XYZ values ​​(Out) obtained through such processing are associated. The corrected XYZ values ​​can be calculated using the correction matrix M by the following equation 3.

[0032]

number

[0033] Furthermore, the correction LUT generation unit 205 calculates the calculation matrix S used in the color residual calculation processing flow described later, using the calculation matrix T and the correction matrix M described above. The correction LUT generation unit 205 stores the calculated calculation matrix S in the camera characteristic information holding unit 207. The calculation matrix S can be calculated using the following equation 4.

[0034]

number

[0035] Through the above processing, the display control device 200 can generate a correction LUT that suppresses the color difference of the target camera 101 and improves the degree of color matching with the reference camera 100. The target camera 101 and the reference camera 100, with such color matching, can be used to image the same subject while suppressing the color difference between them, as shown in Figure 1. The target camera 101 and the reference camera 100, for example, receive light through an optical system using an image sensor such as a CCD sensor or a CMOS sensor to capture still images or moving images. Image sensors have differences in light-receiving characteristics, etc. In such cases, by using the above-described correction LUT in the image generation of the target camera 101, it can be expected that the colors of the image captured by the target camera 101 and the colors of the image captured by the reference camera 100 will match well.

[0036] However, even if such a correction LUT is used in the target camera 101, not all colors in the image captured by the target camera 101 will necessarily match all colors in the image captured by the reference camera 100. In actual field use where multiple imaging devices are used, even with the correction LUT described above, color differences may remain in some subjects. These remaining color differences after predetermined color matching, such as the application of a correction LUT, will be called color residuals. When such color residuals remain, it is difficult for a user to find them by visually comparing the image captured by the reference camera 100 with the image captured by the target camera 101 that has been color-matched to it. Thus, in field use where multiple imaging devices are used, there is a need to be able to easily identify any color residuals that may remain between the multiple imaging devices. For this reason, in addition to generating the correction LUT described above, the display control device 200 of this embodiment further generates color residual information according to the degree of color matching of the target camera 101 after color matching by the correction LUT, and displays and provides the color residual information to the user in an easily identifiable manner.

[0037] Figure 8 is a flowchart for generating color residual information for a target camera 101 that has loaded the correction LUT generated in the flowchart of Figure 3. The target camera 101 outputs an image that has been color-matched by the loaded correction LUT to the display control device 200. After generating the correction LUT, the user operates the UI operation unit 204. For example, the user selects an item called "Color Difference Display" from the menu screen. Based on the input from the UI operation unit 204, the control unit 203 starts the color residual information generation process shown in Figure 8. The control unit 203 outputs the image of the target camera 101 captured using the correction LUT to the color matching determination unit 209.

[0038] In step S801, the color matching determination unit 209 extracts the RGB values ​​of each color patch from the image based on the color-matched image of the target camera 101 received from the control unit 203 and the region information of the color patches. The RGB values ​​of each color patch obtained here will be as shown in Figure 9, for example. The RGB values ​​of the color patches in Figure 9 will be different from the RGB values ​​of the color patches in the color-matched image of the target camera 101 in Figure 5 because color matching has been performed. Figure 9 is an example of a combination of RGB values ​​of color patches in the color-matched image of the target camera 101 according to Figure 8.

[0039] In step S802, the color matching determination unit 209 converts the RGB values ​​of each color patch into XYZ values. The color matching determination unit 209 outputs the generated XYZ values ​​of each color patch to the color residual information generation unit 208. Here, the conversion process by the color matching determination unit 209 can be the same as the conversion process by the correction LUT generation unit 205 in step S302 described above. Figure 10 shows an example of the XYZ values ​​of the color patches after color matching, obtained based on the RGB values ​​of the color patches in Figure 9. The XYZ values ​​of each color patch obtained here will be different from the XYZ values ​​of the target camera 101 before color matching shown in Figure 6. Then, as shown in Figure 10, the XYZ values ​​of each color patch obtained here are used as a 3x3 calculation matrix H.

[0040] In step S803, the color residual information generation unit 208 calculates the error correction matrix GM and stores it in the camera characteristic information holding unit 207. Here, the color residual information generation unit 208 may calculate the error correction matrix GM based on the XYZ values ​​of each color patch received from the color matching determination unit 209 and the calculation matrix S held in the camera characteristic information holding unit 207. The error correction matrix GM represents the color residuals that remain even after color matching. With this, the process shown in Figure 8 is completed.

[0041] Figure 8 is a flowchart for generating color residual information for the target camera 101 using a correction LUT. It is believed that the target camera 101 can capture images of a subject with colors that inherently match those of the reference camera 100 by using the correction LUT. However, in the actual image after color matching of the target camera 101, subtle color differences may remain as errors. Generally, large color errors do not remain across the entire color space, but relatively easily visible color errors may remain in some colors. The error correction matrix GM is a correction matrix for correcting these color residuals.

[0042] The error correction matrix GM can be calculated from the XYZ values ​​of the color patch actually captured by the corrected target camera 101 using the following formula 5.

[0043]

number

[0044] Through the above processing, the display control device 200 can generate color residual information remaining in the image of the target camera 101 after color matching. Here, the color residual information is, for example, the RGB values ​​of the color patch in the color-matched image in Figure 9. Also, the color residual information is, for example, the XYZ values ​​of the color patch in the color-matched image in Figure 10. Furthermore, the color residual information is the error correction matrix GM. This color residual information is color residual information with values ​​corresponding to the degree of color matching in the image. This color residual information indicates the color residual between the image of the target camera 101 and the reference camera 100 that remains in the image of the color-matched target camera 101. This color residual information may be recorded, for example, in the memory 223 as the camera characteristic information holding unit 207.

[0045] Next, the color matching information generation unit 210 generates a display image for displaying the color residual information of the image from the target camera 101, which has undergone color matching, on the display unit 211, based on the generated color residual information. Note that some or all of the processing performed by the color matching information generation unit 210 to generate the display image, which will be described later, may be performed by the color matching determination unit 209. For example, the color matching determination unit 209 may determine the degree of color matching between each pixel value of the image from the target camera 101 and the pixel value of the image from the reference camera 100.

[0046] Figure 11 is a flowchart for generating a display image to show color residual information. The camera characteristic information holding unit 207 holds color residual information that shows the color residual between the image of the target camera 101 and the reference camera 100, which has been color-matched, based on the processing performed so far. When the UI operation unit 204 gives an instruction based on a predetermined user operation, the control unit 203 instructs the color matching degree information generation unit 210 to generate a display image in order to display the color residual information on the display unit 211 via the color matching degree determination unit 209. As a result, the color matching degree information generation unit 210 starts the processing shown in Figure 11.

[0047] In step S1101, the color matching information generation unit 210 obtains the number of vertical pixels for the image of the target camera 101 processed by the color matching determination unit 209 and starts the first loop processing. The first loop processing increments its variable i by 1 with each loop processing, and terminates when the loop variable i reaches the number of vertical pixels. In step S1102, the color matching information generation unit 210 obtains the number of horizontal pixels for the image of the target camera 101 processed by the color matching determination unit 209 and starts the second loop processing. The second loop processing increments its variable j by 1 with each loop processing, and terminates when the loop variable j reaches the number of horizontal pixels.

[0048] In step S1103, the color matching information generation unit 210 obtains the RGB values ​​of the pixel at position (i,j) from the image of the target camera 101 that has undergone color matching. In step S1104, the color matching information generation unit 210 outputs the obtained RGB values ​​of the pixel to the color residual information generation unit 208 via the color matching determination unit 209. The color residual information generation unit 208 compares the color of the obtained RGB values ​​of the pixel with the color of the pixel at the corresponding position on the reference camera 100 and calculates the color residual of the pixel on the target camera 101 that has undergone color matching. The color matching information generation unit 210 obtains color residual information from the color residual information generation unit 208 via the color matching determination unit 209. The color residual information generation unit 208 converts the RGB values ​​obtained via the color matching determination unit 209 to XYZ values ​​using the same RGB to XYZ conversion matrix as in step S302. Subsequently, the color residual information generation unit 208 uses the converted XYZ values ​​and the error correction matrix GM calculated in step S803 to calculate the original corrected XYZ values ​​when the correction LUT is correctly applied, according to equation 6 below. Then, the color residual information generation unit 208 uses the original corrected XYZ values ​​and the XYZ-converted RGB values ​​obtained from the target camera 101 to calculate the color residual value according to equation 7 below. Note that the color residual calculation method shown here is just one example. The color residual information generation unit 208 may, for example, calculate the color residual value using the acquired RGB values.

[0049]

number

[0050] In step S1105, the color matching information generation unit 210 determines whether the color residual received from the color residual information generation unit 208 is within a threshold. Here, the threshold for color residual can be determined based on the range of acceptable values ​​for color residual. At this point, the color matching information generation unit 210 has acquired the position of the pixel that is the subject of the judgment process and the color residual of that pixel. If the color residual is within the threshold, the color matching information generation unit 210 proceeds to step S1106. If the color residual is not within the threshold, the color matching information generation unit 210 proceeds to step S1107.

[0051] In step S1106, the color matching information generation unit 210 performs a process to maintain the pixel values ​​of the pixels that are the subject of the judgment process in the image of the target camera 101 after color matching. In other words, the color matching information generation unit 210 does not convert the pixel values ​​of the pixels that are the subject of the judgment process. After that, the color matching information generation unit 210 proceeds to step S1108.

[0052] In step S1107, the color matching information generation unit 210 converts the pixel values ​​of the pixels that are the subject of the judgment process in the image of the target camera 101 after color matching into pixel values ​​for displaying color residual information. Here, the pixel values ​​for displaying color residual information may be a predetermined color that is easily distinguishable from other pixels in the image of the target camera 101 after color matching. After that, the color matching information generation unit 210 proceeds to step S1108.

[0053] In step S1108, the color matching information generation unit 210 determines the end of the second loop processing that started in step S1102. If the loop variable j has not reached the number of horizontal pixels, the color matching information generation unit 210 increments the variable j to 1 and returns the process to step S1102. The color matching information generation unit 210 repeats the processing from step S1102 to step S1108 until the loop variable j reaches the number of horizontal pixels in the image. The color matching information generation unit 210 repeats the second loop processing from step S1102 to step S1108 for the number of horizontal pixels in the image of the target camera 101 after color matching. As a result, the color matching information generation unit 210 performs a conversion process based on color residual information for multiple pixels in a row that are arranged horizontally in the image of the target camera 101 after color matching. When the loop variable j reaches the number of horizontal pixels, the color matching information generation unit 210 proceeds the process to step S1109.

[0054] In step S1109, the color matching information generation unit 210 determines the end of the first loop processing that started in step S1101. If the loop variable i has not reached the number of vertical pixels, the color matching information generation unit 210 increments the variable i to 1 and returns the process to step S1101. The color matching information generation unit 210 repeats the processing from step S1101 to step S1109 until the loop variable i reaches the number of vertical pixels in the image. The color matching information generation unit 210 repeats the first loop processing from step S1101 to step S1109 for the number of vertical pixels in the image of the target camera 101 after color matching. As a result, the color matching information generation unit 210 performs a conversion process based on color residual information for multiple rows of pixels arranged vertically in the image of the target camera 101 after color matching. When the loop variable i reaches the number of vertical pixels, the color matching information generation unit 210 terminates this process.

[0055] As a result, the color matching information generation unit 210 generates an image for displaying color residual information in which the pixel values ​​of some pixels in the color-matched image of the target camera 101 are converted to pixel values ​​for displaying color residual information. In this display image, in the color-matched image of the target camera 101, areas where color residuals exceeding a threshold occur after color matching are displayed using pixel values ​​for displaying color residual information, unlike other areas. The color matching information generation unit 210 outputs the generated display image to the display unit 211. The display unit 211 displays the display image generated by the color matching information generation unit 210. The user can easily see that in the color-matched image of the target camera 101 displayed on the display unit 211, areas where color residuals occur are displayed using pixel values ​​for displaying color residual information.

[0056] Figure 12 is a table showing the correspondence between the range of color residuals in the color residual information and the converted color used as the color of the pixels included in each range. In Figure 12, the color residual D is judged in three stages. If the color residual D is 0 or greater and less than 1, the color matching information generation unit 210 considers it to be below the threshold and does not convert the pixel value at the target position. If the color residual D is 1 or greater and less than 2, the color matching information generation unit 210 considers it not to be below the threshold and converts the pixel value at the target position to the pixel value (R,G,B=128,0,0) for displaying the color residual information. If the color residual D is 2 or greater, the color matching information generation unit 210 considers it not to be below the threshold and converts the pixel value at the target position to the pixel value (R,G,B=255,0,0) for displaying the color residual information. The pixel value (R,G,B=255,0,0) is redder than (R,G,B=128,0,0). In the displayed image, the color of the portion of the image from the target camera 101 that has undergone color matching that shows color residual information exceeding a threshold is changed to a different color according to the value of the color residual information. Note that the correspondence table in Figure 12 may be recorded in memory 223. In this case, the color matching information generation unit 210 can obtain a threshold value to compare with the color residual value from memory 223 and perform a judgment based on the correspondence table in Figure 12 in step S1105.

[0057] Figure 13 is an explanatory diagram of an example of a display image showing color residual information based on an image captured by the target camera 101 using a correction LUT. The display image in Figure 13 is displayed on the display unit 211. In the display image in Figure 13, a portion of the area is displayed using pixel values ​​for displaying color residual information. By displaying such an image on the display unit 211, the user can intuitively determine the presence and degree of color residual simply by glancing at the display unit 211. Based on this determination, the user can immediately perform subsequent processing, such as regenerating or adjusting the correction LUT. The user does not need to compare the image captured by the target camera 101 using the correction LUT displayed on the display unit 211 with the image captured by the reference camera 100 and visually determine the minute color residual remaining between them.

[0058] As described above, in this embodiment, the degree of color matching between the reference camera 100 and the target camera 101 can be determined based on information regarding color residuals between the cameras, and the resulting color residual information can be explicitly displayed in the image from the target camera 101. The color difference correction system 1 of this embodiment can be used in a field setting where the reference camera 100 and the target camera 101 are used to easily determine color residuals between multiple imaging devices.

[0059] Furthermore, the color residual threshold and the pixel values ​​to be converted, as described above, may be made user-specified via the UI. In this case, the user can display pixels with a color residual below the threshold in monochrome. In this case, only the parts with a color residual above the threshold will be colored, allowing the user to intuitively determine the degree of color matching.

[0060] Furthermore, in the embodiment described above, the color matching information generation unit 210 converts the pixel values ​​of the color-matched image of the target camera 101 into pixel values ​​for displaying color residuals as a display image. In addition, for example, the color matching information generation unit 210 may generate an image for color residuals as a display image. The color matching information generation unit 210 may also display the color residual display image on top of the color-matched image of the target camera 101 in the display unit 211.

[0061] Furthermore, in the above-described embodiment, as an example of a method for calculating color residuals, an example is described in which an error correction matrix is ​​calculated using red, green, and blue color patches 401 to 403, and the color residual is calculated. In addition to this, for example, the color matching information generation unit 210 may measure each color at the vertices of each grid point of the correction LUT with the reference camera 100 and the target camera 101, and calculate the color residual based on that.

[0062] [Second Embodiment] Next, a color difference correction system 1 including a display control device 200 according to the second embodiment of the present invention will be described. In the following, the differences from the embodiments described above will be mainly explained. The display control device 200 of this embodiment differs from the embodiments described above in the processing of the color matching information generation unit 210. Furthermore, it differs from the embodiments described above in that the display image generated by the color matching information generation unit 210 based on the color-matched image of the target camera 101 is a color distribution image showing the distribution of color residuals.

[0063] Figure 14 is a flowchart for generating color residual information using the display control device 200 shown in Figure 14, according to a second embodiment of the present invention. In the process shown in Figure 14, the color matching information generation unit 210 calculates the degree of color matching (color residual) with the reference camera 100 for each pixel of the image of the target camera 101 and generates color residual distribution information. When the UI operation unit 204 gives an instruction based on a predetermined user operation, the control unit 203 instructs the color matching information generation unit 210 to generate a display image through the color matching determination unit 209 in order to display the color residual information on the display unit 211. As a result, the color matching information generation unit 210 executes the process shown in Figure 14.

[0064] In step S1501, the color matching information generation unit 210 obtains the number of vertical pixels for the image of the target camera 101 processed by the color matching determination unit 209 and starts the first loop process. The first loop process increments its variable i by 1 with each loop process, and terminates when the loop variable i reaches the number of vertical pixels. In step S1502, the color matching information generation unit 210 obtains the number of horizontal pixels for the image of the target camera 101 processed by the color matching determination unit 209 and starts the second loop process. The second loop process increments its variable j by 1 with each loop process, and terminates when the loop variable j reaches the number of horizontal pixels.

[0065] In step S1503, the color matching information generation unit 210 obtains the RGB values ​​of the pixel at position (i,j) from the image of the target camera 101 that has undergone color matching. In step S1504, the color matching information generation unit 210 outputs the obtained RGB values ​​of the pixel to the color residual information generation unit 208 via the color matching determination unit 209. The color residual information generation unit 208 compares the color of the obtained RGB values ​​of the pixel with the color of the pixel at the corresponding position on the reference camera 100 and calculates the color residual of the pixel on the target camera 101 that has undergone color matching. The color matching information generation unit 210 obtains color residual information from the color residual information generation unit 208 via the color matching determination unit 209.

[0066] In step S1505, the color matching information generation unit 210 receives the color residual and the pixel positions of the image for which the color residual has been calculated from the color matching determination unit 209, and generates a display image of the color residual. In step S1506, the color matching information generation unit 210 determines the end of the second loop processing that started in step S1502. If the loop variable j has not reached the number of horizontal pixels, the color matching information generation unit 210 increments the variable j to 1 and returns the process to step S1502. The color matching information generation unit 210 repeats the processing from step S1502 to step S1506 until the loop variable j reaches the number of horizontal pixels of the image. The color matching information generation unit 210 repeats the second loop processing from step S1502 to step S1506 for the number of horizontal pixels of the image of the target camera 101 after color matching. As a result, the color matching information generation unit 210 plots the color residuals for multiple pixels in a row arranged horizontally in the image of the target camera 101 after color matching onto a color distribution image that shows the distribution of color residuals. When the loop variable j reaches the number of horizontal pixels, the color matching information generation unit 210 proceeds to step S1507.

[0067] In step S1507, the color matching information generation unit 210 determines the end of the first loop processing that started in step S1501. If the loop variable i has not reached the number of vertical pixels, the color matching information generation unit 210 increments the variable i to 1 and returns the process to step S1501. The color matching information generation unit 210 repeats the processing from step S1501 to step S1107 until the loop variable i reaches the number of vertical pixels in the image. The color matching information generation unit 210 repeats the first loop processing from step S1501 to step S1507 for the number of vertical pixels in the image of the target camera 101 after color matching. As a result, the color matching information generation unit 210 performs a conversion process based on color residual information for multiple rows of pixels arranged vertically in the image of the target camera 101 after color matching. When the loop variable i reaches the number of vertical pixels, the color matching information generation unit 210 terminates this process.

[0068] As a result, the color matching information generation unit 210 can generate a color distribution image showing the distribution of color residuals as a display image for displaying color residual information. The color matching information generation unit 210 outputs the generated color distribution image showing the distribution of color residuals to the display unit 211. The display unit 211 displays the color distribution image showing the distribution of color residuals.

[0069] Figure 15 shows an example of a combination of an image captured by a target camera 101 using a correction LUT and a color distribution image showing the distribution of color residuals, which is generated separately. The color matching information generation unit 210 sequentially acquires color residual information for each pixel of the image captured by the target camera 101 using the correction LUT shown at the top of Figure 15. The color matching information generation unit 210 then plots the color residual of the pixel at the acquired position on the color distribution image showing the distribution of color residuals, shown at the bottom of Figure 15. The background image of the color distribution image showing the distribution of color residuals may be stored in memory 223 beforehand. The horizontal axis of the color distribution image showing the distribution of color residuals is the horizontal position of the image. The vertical axis is the color residual. In this case, the color matching information generation unit 210 only needs to plot points indicating color residuals at positions on the vertical axis corresponding to the color residual values ​​of the pixels at the acquired positions. The color matching information generation unit 210 may repeatedly plot pixels in the horizontal scanning direction of a single row of images captured by the target camera 101 using the correction LUT at positions corresponding to the color residual.

[0070] Figure 16 is an example of a color distribution image showing the distribution of color residuals when this embodiment is applied. In the color distribution image showing the distribution of color residuals in Figure 16, large color residuals occur mainly in the left portion of the image. Also, the maximum value of the color residuals reaches 2.0. By simply looking at the color distribution image showing the distribution of color residuals in Figure 16, the user can intuitively understand where and to what extent color residuals remain in the image captured by the target camera 101 using the correction LUT. Furthermore, the color distribution image showing the distribution of color residuals shown in Figure 16 displays hues specified by the user. In addition, for example, the color distribution image showing the distribution of color residuals may also display multiple hues superimposed by color coding or other means.

[0071] As described above, in this embodiment, the degree of color matching between the reference camera 100 and the target camera 101 is determined, and a color distribution image showing the distribution of color residuals between those images is generated and displayed. By visually inspecting the color distribution image showing the distribution of color residuals, the user can intuitively determine whether or not residuals remain in the colors at any given position in the image.

[0072] The color matching information generation unit 210 may generate a display image other than the color distribution image showing the distribution of color residuals described above as a display image for displaying color residuals. For example, the color matching information generation unit 210 may generate a display image that aggregates the percentage of pixels with a certain level of color residuals for each horizontal direction of the image as a display image for displaying color residuals. Also, the color matching information generation unit 210 may display the distribution information of color residuals at vertical pixel positions instead of horizontal pixel positions. Furthermore, the color matching information generation unit 210 may generate a color distribution image that shows the distribution of color residuals for a specific color specified by the user, rather than generating a color distribution image showing the distribution of color residuals for all pixels of the color-matched image of the target camera 101. In this case, the user can check the distribution of color residuals for each specified color by switching between the specified colors. Also, the color matching information generation unit 210 may display color residuals in a vectorscope image as a display image for displaying color residuals.

[0073] Figure 17 is an example of a vectorscope image displaying color residual information. Figure 17 is a vectorscope image of an image from a target camera 101 that has undergone color matching. A vectorscope image basically shows the hue and saturation of each pixel in an image. The color matching information generation unit 210 assigns different colors, shapes, or sizes to the points plotted on such a vectorscope image according to the degree of color matching that can be determined based on the color residual information. As a result, in the vectorscope image, the central part of the image, as exemplified in Figure 17, will be colored according to the color residual.

[0074] Furthermore, the color matching information generation unit 210 may not simply generate the various display images described above for displaying color residuals, but may also generate a screen or message prompting the user to re-perform color matching, either together with or instead of those images. Figure 18 is a flowchart for outputting a message when the value of the color residual information is above a threshold. The color matching information generation unit 210 may, for example, execute the process in Figure 18 together with or after the process in Figure 14.

[0075] In step S1901, the color matching information generation unit 210 obtains the maximum color residual value from all pixels of the color-matched image of the target camera 101. In step S1902, the color matching information generation unit 210 determines whether the obtained maximum color residual value is equal to or greater than a threshold. The threshold here can be a large color residual value that can be expected to be reduced by, for example, regenerating the correction LUT. In Figure 16, for example, the threshold for color residual may be 2.0 or greater. If the obtained maximum color residual value is equal to or greater than the threshold, the color matching information generation unit 210 proceeds to step S1903. If the obtained maximum color residual value is not equal to or greater than the threshold, the color matching information generation unit 210 terminates this process. In step S1902, the color matching information generation unit 210 generates a screen or message prompting the user to re-perform color matching. The color matching information generation unit 210 outputs the generated screen or message to the display unit 211. The display unit 211 displays a screen or message. After that, the color matching information generation unit 210 terminates this process. As a result, if the percentage of color residuals that can be determined based on the color residual information is above a threshold, the color matching information generation unit 210 can generate a screen or message prompting the user to re-perform color matching.

[0076] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention.

[0077] For example, when the color matching information generation unit 210 generates the vectorscope image shown in Figure 17, it changes the points plotted on the vectorscope based on the RGB values ​​of the target camera 101 or calculated values ​​based thereon. Alternatively, when generating the vectorscope image, the color matching information generation unit 210 may simultaneously calculate the color residual with the image from the reference camera 100 and plot different points according to that color residual. By plotting such points, the user can easily intuitively determine which parts of the vectorscope image have a good color match.

[0078] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0079] 1. Color difference correction system 100 Reference Cameras 101 Target Camera 200 Display control device 201a, 201b Image receiving unit 202a, 202b Image capture section 203 Control Unit 204 UI ​​operation section 205 Correction LUT generation section 206 Correction LUT Output Section 207 Camera characteristic information retention unit 208 Color residual information generation section 209 Color matching determination section 210 Color matching information generation unit 211 Display section 221 First Input Terminal 222 Second input terminal 223 memory 400a, 400b acquired images 401, 402, 403 Color Patches

Claims

1. an acquisition means for acquiring an adjusted image from an adjusted imaging device that has performed a predetermined color matching with a reference imaging device; a color residual generating means for generating color residual information indicating a color difference remaining in the adjusted image and a reference image captured by the reference imaging device; a display generating means for generating display information based on the color residual information; having Display control device.

2. the predetermined color matching is performed by comparing images obtained by capturing images of a predetermined subject with the adjustment imaging device and the reference imaging device, The display control device according to claim 1 .

3. The predetermined subject is a color chart. The display control device according to claim 2 .

4. a holding unit for holding the reference image used for the predetermined color matching with the adjustment imaging device; comparing the adjusted image with the reference image stored in the storage unit to generate the color residual information according to a degree of color matching between the images; The display control device according to claim 1 .

5. generating, as the display information, an image in which a color of a portion of the adjusted image for which the color residual information is generated is changed to a color corresponding to a value of the color residual information; 3. The display control device according to claim 1 or 2.

6. generating a color distribution image corresponding to the adjusted image as the display information; a color distribution according to a degree of color similarity that can be determined based on the color residual information is displayed on the color distribution image; The display control device according to any one of claims 1 to 3.

7. generating an image of at least one hue among a plurality of hues included in the adjusted image as the color distribution image; The display control device according to claim 1 .

8. generating a vectorscope image of the adjusted image as the display information; Plotting points on the vectorscope image, the points having a color, a shape or a size corresponding to a degree of color matching that can be determined based on the color residual information; The display control device according to any one of claims 1 to 7.

9. When a maximum color residual that can be determined based on the color residual information is equal to or greater than a threshold value, at least one of a screen and a message that prompts a user to re-perform the predetermined color matching is generated and displayed in place of a display image for displaying the color residual information. The display control device according to any one of claims 1 to 8.

10. When a maximum color residual that can be determined based on the color residual information is equal to or greater than a threshold value, at least one of a screen and a message that prompts a user to re-perform the predetermined color matching is generated and displayed together with a display image for displaying the color residual information. The display control device according to any one of claims 1 to 8.

11. an acquisition step of acquiring an adjusted image from an adjusted imaging device that has performed a predetermined color matching with a reference imaging device; a color residual generating step of generating color residual information indicating a color difference remaining in the adjusted image and a reference image captured by the reference imaging device; a display generating step of generating display information based on the color residual information; having A display control method for a display control device.

12. A display control method according to claim 11, program.

Images