Image correction device, image correction method, and program

Abstract

The present invention provides, for example, an image correction device which corrects a projection image in consideration of the brightness of a projected portion relative to the periphery, and improves the realness of texture and color of the projected portion. The image correction device includes: a projection result estimation unit for, on the premise that a projected portion corresponds to a range in which a projector projects an image onto an object to be projected, an imaging portion corresponds to a range captured by a camera, the imaging portion is larger than the projected portion, and the imaging portion includes the projected portion, estimating an image to be obtained when a projection image is projected onto the object to be projected, thereby obtaining a projection result estimated image; a perceptual distance calculation unit for calculating a perceptual distance, which is a perceptual difference in intra-brain representation between a target image and the projection result estimated image; and a projection image update unit for updating the projection image so that the perceptual distance is reduced, thereby obtaining an updated projection image. In this image correction device, the projection result estimated image and the target image are images of the same size as the imaging portion.

Description

Technical Field

[0001] The present invention relates to a technique for correcting a projection image in projection mapping.

Background Art

[0002] When projecting an image onto an arbitrary object surface, due to factors such as ambient light, changes in the reflectivity of the projection surface (texture), and the dynamic range of the projector, the actually displayed image may deviate significantly from the expected appearance.

[0003] Non-Patent Document 1 can compensate for the degradation of the image due to the above factors.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since a projector displays an image by adding light to a real object, the area within the projection range will inevitably become brighter. Due to the significantly brighter projection area compared to the surrounding area, the realism (likeness to the real thing) of the color and texture in that area is impaired. In other words, conventionally, since compensation is performed based only on the projected image, there is a problem that the apparent difference compared to the brightness around the projected area cannot be fully compensated for.

[0006] An object of the present invention is to provide an image correction device, an image correction method, and a program that correct a projected image in consideration of the brightness of the projected area relative to the surrounding area, and improve the realism of the color and texture of the projected area.

Means for Solving the Problems

[0007] To solve the above problems, according to one aspect of the present invention, in an image correction device, the projection area is the range where a projector projects an image onto a projection target, the imaging area is the range captured by a camera, the imaging area is larger than the projection area, and the imaging area includes the projection area. An estimated projection result image is obtained by estimating an image obtained when a projected image is projected onto a projection target. A perceptual distance calculation unit calculates a perceptual distance that is the perceptual difference in the mental representation between the target image and the estimated projection result image. A projected image update unit updates the projected image so that the perceptual distance becomes smaller, and obtains the updated projected image. The estimated projection result image and the target image are images of the same size as the imaging area.

Effects of the Invention

[0008] According to the present invention, there is an effect of improving the realism of the color and texture of the projected area.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described. In the drawings used in the following description, components having the same function and steps performing the same process are denoted by the same reference numerals, and redundant descriptions are omitted. In the following description, the processing performed on each element unit of a vector or a matrix is applied to all the elements of the vector or the matrix unless otherwise specified.

[0011] <First Embodiment> FIG. 1 is a functional block diagram of an image correction apparatus according to the first embodiment, and FIG. 2 shows its processing flow.

[0012] The image correction apparatus includes an imaging unit 110, a projection unit 120, a pixel luminance conversion unit 125, a geometric calibration unit 130, a projection result estimation unit 140, a perceived distance calculation unit 150, and a projection image update unit 160.

[0013] The image correction apparatus takes the target image R as an input, corrects the projection image G(t) in consideration of the brightness of the projection part with respect to the periphery, and projects the corrected projection image G fin onto the projection target via the projection unit 120. Here, t indicates the update count of the projection image G(t) in the projection image update unit 160, and G(0) represents the initial value of the projection image.

[0014] In this embodiment, the case of dealing with a grayscale image will be described as an example. When dealing with a color image, each of RGB is processed independently. All the images in the following description represent a series of pixel values. Each pixel of a grayscale image has a brightness value, and each pixel of a color image has an RGB value.

[0015] The image correction device is a special device configured by loading a special program into a known or dedicated computer having, for example, a central processing unit (CPU: Central Processing Unit), a main memory device (RAM: Random Access Memory), etc. The image correction device executes each process under the control of, for example, the central processing unit. The data input to the image correction device and the data obtained by each process are stored, for example, in the main memory device, and the data stored in the main memory device is read out to the central processing unit as needed and used for other processes. At least a part of each processing unit of the image correction device may be configured by hardware such as an integrated circuit. Each storage unit included in the image correction device can be configured by, for example, a main memory device such as a RAM (Random Access Memory), or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily have to be provided inside the image correction device, and may be configured by an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory (Flash Memory), and may be provided outside the image correction device.

[0016] The following describes each part.

[0017] <Imaging unit 110 and projection unit 120> The imaging unit 110 includes a camera, and the projection unit 120 includes a projector.

[0018] The image correction device projects the gray code pattern H onto the projection target via the projection unit 120, captures the projected gray code pattern with the imaging unit 110, and obtains the gray code pattern projection result H' (image for geometric calibration). For example, an image for geometric calibration is obtained by the method of Reference 1.

[0019] (Reference 1) S. Inokuchi, K. Sato, and F. Matsuda, "Range-imaging system for 3-D object recognition", in Proceedings of International Conference on Pattern Recognition, 1984, pp. 806 - 808. Note that, the range where the projector of the projection unit 120 projects an image onto the projection target is called the projection area, and the range captured by the camera of the imaging unit 110 is called the imaging area (see Fig. 3). The projector and the camera are set such that the imaging area is larger than the projection area and the imaging area includes the projection area. Based on the visual characteristics of humans, even if the projection target becomes physically brighter than its surroundings due to projection, it is perceptually felt as if the appearance of the object itself has changed. In this embodiment, by setting the imaging area wider than the projection area, optimization is performed including the surroundings of the projection area. Note that, the part that is included in the imaging area but not in the projection area is also referred to as the non-controlled area. The imaging area may be set according to the size of the surrounding area to be considered. Also, the image correction device captures the projection target when the projector of the projection unit 120 performs white projection with the maximum output by the imaging unit 110 to obtain the captured image C' max and captures the projection target when the projector of the projection unit 120 performs projection with the minimum output by the imaging unit 110 to obtain the captured image C' min to obtain.

[0020] Also, using a luminance measurement device (not shown), the luminance value L of an arbitrary location of the projection target when the projector of the projection unit 120 performs white projection with the maximum output is measured and used as the input to the image correction device.

[0021] <Pixel luminance conversion unit 125> Input: Captured image C' max and C' min , luminance value L Output: Maximum output image C max and minimum output image C min The pixel luminance conversion unit 125 calculates the scaling coefficient s = L / v for converting the pixel value to the luminance value by calculating the ratio of the luminance value L to the camera pixel value v at the same location as the arbitrary location where the luminance value L was measured. The camera pixel value v is the camera pixel value at the same location as the arbitrary location where the luminance value L in the captured image C' max was measured.

[0022] The pixel luminance conversion unit 125 scales the captured images C' max and C' min by the scaling coefficient s to obtain the maximum output image C of the luminance scale max (= sC' max ) and the minimum output image C min (= sC' min ) (S125).

[0023] <Geometric calibration unit 130> Input: Gray code pattern H, Gray code pattern projection result (image for geometric calibration) H' Output: Conversion function W The geometric calibration unit 130 calculates the geometric mapping between the camera and the projector. Specifically, the geometric calibration unit 130 uses the gray code pattern H and the gray code pattern projection result (image for geometric calibration) H' to obtain the conversion function W from the coordinates (projector coordinate system) of the gray code pattern H (the projection image output from the projector of the projection unit 120) to the coordinates (the coordinates when the projected image is reflected on the projection target surface and captured as a camera image) of the gray code pattern projection result H' (the image for geometric calibration, which is the image captured by the camera of the imaging unit 110). For example, the conversion function W is obtained by decoding the gray code according to the method described in Reference 1 (S130). Note that the gray code pattern H is image data with the size of the projector resolution projected onto the projection part, and the gray code pattern projection result is image data with the size of the camera resolution corresponding to the imaging part. <Projection result estimation unit 140> Input: Conversion function W, maximum output image C max , minimum output image C min , projection image G(t - 1) Output: Projected result estimated image C The projected result estimation unit 140 estimates an image (projected result image) obtained when the projected image G(t - 1) is projected onto the projection target (S140), and obtains the projected result estimated image C.

[0024] For example, the projected result estimation unit 140 first uses the conversion function W to geometrically transform the initial value G(0) of the projected image or the projected image G(t - 1) updated by the projected image update unit 160 into the projected result estimated image G'(t - 1)=W(G(t - 1)) in the camera coordinate system (S140). Note that the projected image G(t - 1) is image data of a size corresponding to the projection part, and the projected result estimated image G'(t - 1) is image data of a size corresponding to the photographing part.

[0025] Next, the projected result estimation unit 140 obtains the projected result estimated image C by converting it into the luminance value reflected from the projection target according to the following formula.

[0026] C=(C max -C min )G'(t - 1)+C min Note that this calculation is a calculation between pixel values corresponding geometrically and is a scalar calculation. Also, the images C max , C min , C are image data of a size corresponding to the photographing part. <Perceived distance calculation unit 150> Input: Projected result estimated image C, target image R Output: Perceived distance D(R, C) The perceptual distance calculation unit 150 calculates a perceptual distance D(R, C), which is a perceptual difference in the mental representations of the target image R and the projection result estimation image C (S150). For example, the perceptual distance calculation unit 150 inputs the target image R and the projection result estimation image C into a visual model (a model of mental representation), and calculates the distance (perceptual distance) D(R, C) in the mental representations of the respective images. The target image R is an image that appears to be the target after projection, and is image data of a size corresponding to the imaging part. For example, the target image R is obtained by, for example, image-processingly modifying a camera image of the projection target before projection to a desired appearance, and the positions of the respective pixels of the target image R and the camera image completely correspond. Also, the target image R is an image with pixel values on a luminance scale, similar to the projection result estimation image C. As a model of mental representation, for example, the Normalized Laplacian Pyramid Distance (NLPD), which is a low-order visual information processing model in Reference 2, can be used, and the perceptual distance is calculated using this visual information processing model.

[0027] (Reference 2) Laparra et al., "Perceptually Optimized Image Rendering", In JOSA, 2017 <Projection image update unit 160> Input: Perceptual distance D Output: Projection image G(t) or the optimized projection image G fin The projection image update unit 160 updates the projection image G(t - 1) so that the value of the perceptual distance D becomes smaller, and obtains the updated projection image G(t). For example, based on the gradient of the perceptual distance D with respect to each pixel value of the projection image G(t - 1), each pixel value of the projection image G(t - 1) is updated so that the perceptual distance D becomes smaller (S160). Various conventional techniques can be used as the update method for the projection image G(t). For example, the update method of Reference 3 (ADAM) can be used.

[0028] (Reference 3) Diederik P. Kingma et al., "ADAM: A Method for Stochastic Optimization", In ICLR, 2015. If the projection image G(t - 1) is simply updated so as to reduce the "color distance" between the projection image G(t) and the target image R, the reality (authenticity) of the color and texture in that area is impaired due to the projection part that is clearly brighter compared to the surroundings. However, the uncontrolled part around the projection part cannot be controlled. Therefore, in the present embodiment, the projection image G(t - 1) is updated while considering the uncontrolled part so as to reduce the perceptual distance D. By performing image correction based on human visual characteristics, even if the actual "color distance" increases, the perceptual distance decreases and the reality (authenticity) improves.

[0029] The projection image update unit 160 repeats S140 to S160 until the update of the projection image G(t) converges (NO in S160 - 2). When the update of the projection image G(t) converges (YES in S160 - 2), the projection image update unit 160 stops the update and outputs the projection image G(t) at that time as the optimal value (the projection image G fin ) after optimization). The projection image G fin is projected onto the projection target via the projection unit 120. For example, when the perceptual distance D does not decrease for a predetermined number of update times (e.g., 20 times), or when a predetermined maximum number of update times (e.g., 500 times) is reached, it is determined that the update of the projection image G(t) has converged.

[0030] <Effect> With the above configuration, by projecting the optimized projection image onto the projection target, it is possible to improve the realism of the color and texture of the projected part. Also, it enables a natural-looking editing of the projection target. In this embodiment, a series of processes from when the projection image reaches the mental representation through interaction with the surrounding environment are simulated, the "perceptual difference" between the target image and the actual projection result in the mental representation is calculated, and the projection image is optimized pixel by pixel so as to minimize the "perceptual difference" as a loss. With such a configuration, image correction based on human visual characteristics is performed, and even if the object becomes physically brighter than the surroundings due to projection, it is perceived as if the appearance of the object itself has changed.

[0031] <Modification Example> In this embodiment, the image correction device includes the imaging unit 110 and the projection unit 120, but it may also be configured to include a pixel luminance conversion unit 125, a geometric calibration unit 130, a projection result estimation unit 140, a perceptual distance calculation unit 150, and a projection image update unit 160 without including the imaging unit 110 and the projection unit 120. In this case, in addition to the target image R, the image correction device receives the maximum output image C max and the minimum output image C min , the gray code pattern H, and the gray code pattern projection result H', corrects the projection image G(t) in consideration of the brightness of the projected part with respect to the surroundings, and outputs the corrected projection image G fin to the projection unit 120.

[0032] <Other Modification Examples> The present invention is not limited to the above-described embodiments and modification examples. For example, the above-described various processes may be executed not only in time series according to the description, but also in parallel or individually according to the processing ability of the device that executes the processes or as necessary. In addition, appropriate changes can be made without departing from the spirit of the present invention.

[0033] <Program and Recording Medium> Each of the above-mentioned various processes can be implemented by causing a program for executing each step of the above method to be read into the storage unit 2020 of the computer shown in FIG. 4 and causing operations on the control unit 2010, the input unit 2030, the output unit 2040, etc.

[0034] The program describing this processing content can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any of a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, etc. may be used.

[0035] Also, the distribution of this program can be carried out, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or a CD-ROM on which the program is recorded. Further, the program may be stored in the storage device of a server computer, and the program may be distributed by transferring the program from the server computer to other computers via a network.

[0036] A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, when executing the process, the computer reads the program stored in its own recording medium and executes the process according to the read program. As another execution form of this program, the computer may directly read the program from the portable recording medium and execute the process according to the program. Further, each time a program is transferred from the server computer to this computer, the computer may sequentially execute the process according to the received program. Also, the above-described process may be executed in a configuration of a so-called ASP (Application Service Provider) type service in which the transfer of the program from the server computer to this computer is not performed, and the processing function is realized only by the execution instruction and result acquisition. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to the program (data having a property of defining the processing of the computer but not being a direct instruction to the computer).

[0037] Also, in this embodiment, the present apparatus is configured by causing a computer to execute a predetermined program, but at least a part of these processing contents may be realized hardware-wise.

Claims

1. The projection part is the range where the projector projects an image onto the projection target, the imaging part is the range captured by the camera, the imaging part is larger than the projection part, and the imaging part includes the projection part. A projection result estimation unit that estimates an image obtained when the projection image is projected onto the projection target and obtains a projection result estimation image. A perceptual distance calculation unit that calculates a perceptual distance, which is a perceptual difference in the mental representation between the target image, which is the apparent target image after projection, and the projection result estimation image. A projection image update unit that updates the projection image so that the perceptual distance becomes smaller and obtains the updated projection image. The projection result estimation unit geometrically transforms the initial value of the projection image or the projection image updated by the projection image update unit from the coordinates of the projection image output from the projector and the coordinates when the projected image is reflected on the projection target surface and captured as a camera image to obtain a projection result estimation image in the camera coordinate system. The projection result estimation image and the target image are images of the same size as the imaging part. Image correction device.

2. The projection part is the range where the projector projects an image onto the projection target, the imaging part is the range captured by the camera, the imaging part is larger than the projection part, and the imaging part includes the projection part. A projection result estimation step of estimating an image obtained when the projection image is projected onto the projection target and obtaining a projection result estimation image. A perceptual distance calculation step of calculating a perceptual distance, which is a perceptual difference in the mental representation between the target image, which is the apparent target image after projection, and the projection result estimation image. A projection image update step of updating the projection image so that the perceptual distance becomes smaller and obtaining the updated projection image. In the projection result estimation step, the initial value of the projection image or the projection image updated by the projection image update unit is geometrically transformed from the coordinates of the projection image output from the projector and the coordinates when the projected image is reflected on the projection target surface and captured as a camera image to obtain a projection result estimation image in the camera coordinate system. The projection result estimation image and the target image are images of the same size as the imaging part. Image correction method.

3. A program for causing a computer to function as the image correction device according to Claim 1.

Images