Image processing device, image processing method, program, and image projection method

Abstract

An image processing device 40 generates, from a captured image obtained by capturing a mixed image of projected images that are projected from a plurality of projection devices while each being given different color information, a separated image for each color information by a color separation processing unit 41 on the basis of a color model composed of the color information of the projected images and color information of a background color. The color model uses, as parameters, the color information of the projected images changed according to spectral characteristics of the projection devices and an imaging device for acquiring the captured image, and an attenuation coefficient indicating attenuation occurring in the mixed image captured by the imaging device, and the color separation processing unit 41 generates the separated image for each color information on the basis of the color model, using parameters that minimize the difference between the color information of the captured image and color information estimated by the color model. The image processing device 40 becomes able to accurately separate projected images from the captured image of the mixed image composed of a plurality of projected images.

Description

[Technical field]

[0001] This technique relates to an image processing device, an image processing method, a program, and an image projection method, and makes it possible to separate a projected image from a captured image that is a mixed image made up of a plurality of projected images. [Background technology]

[0002] Conventionally, images projected by multiple projectors are combined to display a single mixed image. In this case, an imaging device is used to capture the projected images to obtain the positional relationship between the projected images, and inconsistencies in the images in the overlapping areas between the projected images are resolved.

[0003] In addition, in order to obtain the position of the projected image, a pixel correspondence is necessary between the projection device and the imaging device, and a sensing image called structured light is projected and photographed to obtain the correspondence. For example, in Patent Document 1, when structured light is simultaneously projected, the structured light is mixed with each other, and it is impossible to distinguish which projection device the pixel information of the photographed projected image corresponds to, so different color information is added to distinguish between the projected images of multiple devices. In Patent Document 2, a region that does not overlap spatially is used to distinguish between the projected images of multiple devices. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-047849 A [Patent Document 2] JP 2015-056834 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when distinguishing between projected light by using color information as in Patent Document 1, if the projected color does not match the captured color due to factors such as the color of the screen or ambient light, or the device-specific spectral characteristics of the projection device and imaging device, a projection image other than the desired projection image may appear in the separation result, which may reduce sensing accuracy or cause sensing failure.

[0006] In addition, when using a spatially non-overlapping area as in Patent Document 2, for example, there are restrictions on the pattern in which the projected images can be arranged without overlapping the markers, and a restriction is required that the projection areas of the projectors must be arranged side-by-side to a certain extent to avoid overlap. Therefore, it is not possible to accommodate a wide range of projection image arrangements, including stacking projection in which multiple projection ranges almost overlap, and the degree of freedom in arrangement is low. Furthermore, since the markers are arranged to avoid overlapping in the projection light, the density of information that can be used for correction is reduced.

[0007] Therefore, an object of this technique is to provide an image processing device, an image processing method, a program, and an image projection method that are capable of separating a projection image from a captured image that is a mixed image made up of a plurality of projection images. [Means for solving the problem]

[0008] The first aspect of this technology is: A color separation processing unit that generates a separated image for each color information based on color information of a captured image obtained by capturing a mixed image of projected images projected from a plurality of projection devices with different color information added thereto, and a color model that indicates a relationship between the color information of the captured image and the color information of the projected image and the background. The present invention relates to an image processing device comprising:

[0009] In this technology, the color separation processing unit generates a separated image for each color information based on color information of a captured image obtained by capturing, for example, a mixed image of structured light projected from a plurality of projection devices with different color information added thereto, and a color model showing the relationship between the color information of the captured image and the color information of the projected image and the background. The color model uses, as parameters, color information of the projected image that changes according to the spectral characteristics of the projection device and the imaging device that obtains the captured image, and an attenuation coefficient that shows attenuation occurring in the mixed image captured by the imaging device, and the color separation processing unit generates a separated image for each color information based on the color model using parameters that minimize the difference between the color information of the captured image and the color information estimated by the color model.

[0010] In addition, the imaging device that captures the mixed image has a non-fixed viewpoint, and when gamma correction is performed in the imaging device, the color separation processing unit generates a separated image using the captured image after de-gamma processing. In addition, the mutually different color information is set so that the inner product of the color vectors corresponding to the color information is minimized. In addition, the projected image and the captured image are images that do not cause saturation.

[0011] The projection device further includes an image correction unit that corrects the projection image projected from the projection device, and performs color calibration of the projection image using color information added to the separated images. The projection device also includes a corresponding point detection unit that detects corresponding points between the separated images for each color information generated by the color separation processing unit, and the image correction unit corrects the projection image using geometric correction information that aligns the corresponding points for each separated image detected by the corresponding point detection unit.

[0012] In addition, a predetermined number of projection devices are grouped, and at least one projection device in a group is included in another group, and different color information is assigned to each projection device in the group to project a projected image. A color separation processing unit generates a separated image for each group, a corresponding point detection unit detects corresponding points for each group, and an image correction unit corrects the projected image using geometric correction information that aligns the corresponding points for each separated image detected for each group by the corresponding point detection unit.

[0013] The second aspect of this technology is A color separation processing unit generates a separated image for each color information based on a color model composed of color information of the projected images and color information of a background color from a captured image obtained by capturing a mixed image of projected images projected by a plurality of projectors with different color information added thereto. The present invention relates to an image processing method including the steps of:

[0014] The third aspect of this technology is A program for causing a computer to execute a process of separating each projection image from a captured image obtained by capturing a mixed image of projection images, a step of acquiring a captured image by capturing a mixed image of projected images projected from a plurality of projectors with different color information; generating a separated image for each color information from the captured image based on a color model configured with color information of the projected image and color information of a background color; The program is executed by the computer.

[0015] The program of the present technology is, for example, a program that can be provided in a computer-readable format to a general-purpose computer capable of executing various program codes, via a storage medium, a communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer.

[0016] The fourth aspect of this technology is A color separation processing unit generates a separated image for each color information based on a color model configured with color information of the projected images and color information of a background color from a captured image obtained by capturing a mixed image of projected images projected from a plurality of projectors with different color information added thereto; detecting corresponding points between the separated images for each of the color information generated by the color separation processing unit using a corresponding point detection unit; Correcting the projection images projected from the plurality of projection devices by an image correction unit using geometric correction information for matching the corresponding points for each of the separated images detected by the corresponding point detection unit. The present invention relates to an image projection method comprising the steps of: [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates an example of the configuration of an image projection system. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of an embodiment. [Diagram 3] 4 is a flowchart illustrating an operation of the embodiment. [Figure 4] FIG. 2 is a diagram illustrating the spectral sensitivity of an imaging device. [Diagram 5] 11A and 11B are diagrams illustrating examples of color changes in a captured image relative to a projected image. [Figure 6] 1A to 1C are diagrams illustrating captured images that change depending on the environment. [Figure 7] 1A and 1B are diagrams showing sensing patterns and projection states onto a screen. [Figure 8] 11 is a flowchart illustrating an example of a parameter estimation operation. [Figure 9] FIG. 13 is a diagram illustrating a separation image. [Figure 10] FIG. 11 is a diagram illustrating an example of a parameter estimation operation. [Figure 11] 1A and 1B are diagrams illustrating sensing patterns (structured light) projected from a projection device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Image projection system 2. Configuration of the embodiment 3. Operation of the embodiment 3-1. Generation of separated images 3-2. Other operations of the embodiment> 3-2-1.Sensing patterns 3-2-2. Gamma characteristics 3-2-3. When there are three or more projectors 3-2-4. Color information to be added 3-2-5. Color proofing 3-2-6. Projection and imaging conditions

[0019] <1. About the image projection system> Fig. 1 illustrates an example of the configuration of an image projection system using the image processing device of the present technology. Note that Fig. 1 illustrates a case in which two projection devices are used.

[0020] The image projection system 10 includes projection devices 20-1 and 20-2 that project an image onto a screen Sc, an imaging device 30 that captures the screen Sc from a non-fixed viewpoint (non-fixed viewpoint), an image processing device 40 that separates a projection image from the captured image acquired by the imaging device 30, and an image generating device 50 that outputs an image signal indicating an image to be projected onto the screen Sc to the projection devices 20-1 and 20-2. The imaging device 30, the image processing device 40, and the image generating device 50 may be provided independently, or each device may be integrated, or only some of the devices (for example, the image processing device 40 and the image generating device 50) may be integrated. Furthermore, the image generating device 50 or the image processing device 40 and the image generating device 50 may be provided integrated into either of the projection devices 20-1 and 20-2.

[0021] 2. Configuration of the embodiment FIG. 2 illustrates an example of the configuration of the embodiment, in which an image processing device 40 includes a color separation processing unit 41, a corresponding point detection unit 42, and a position calculation unit 43.

[0022] For example, during calibration of the image projection system, a projection image, for example, a sensing pattern (structured light), is projected onto the screen Sc by the projection devices 20-1 and 20-2, and different color information is imparted to the sensing pattern by each projection device. The imaging device 30 captures an image of the screen Sc onto which the sensing patterns are projected, from a non-fixed viewpoint, and obtains a captured image showing a mixed image of the first sensing pattern projected by the projection device 20-1 and the second sensing pattern projected by the projection device 20-2.

[0023] The color separation processing unit 41 generates a separated image for each color information added to the sensing pattern based on the color information of the captured image acquired by the imaging device 30 and a color model that indicates the relationship between the color information of the captured image and the color information of the projected image and the background. The generation of the separated image for each color information will be described in detail later. The color separation processing unit 41 outputs the generated separated image to the corresponding point detection unit 42.

[0024] The corresponding point detection unit 42 detects corresponding points between the separated images for each color information, and outputs corresponding point information indicating the detection result to the position calculation unit 43.

[0025] The position calculation unit 43 calculates a position correction amount for matching the display positions of the corresponding points detected by the corresponding point detection unit 42. For example, the position calculation unit 43 uses a separation image of color information imparted to the first sensing pattern as a reference and calculates a position correction amount for matching the display positions of the corresponding points of the separation image of color information imparted to the second sensing pattern with the corresponding points of the reference separation image. The position calculation unit 43 outputs the calculated position correction amount to the image generation device 50.

[0026] The image generating device 50 includes an image generating section 51 and an image correcting section 52. The image generating section 51 generates an image signal of a projection image. For example, when calibrating the image projection system, the image generating section 51 generates a first sensing pattern as a projection image projected from the projection device 20-1 and a second sensing pattern as a projection image projected from the projection device 20-2. In addition, the image generating section 51 imparts color information to the first sensing pattern and imparts color information different from that of the first sensing pattern to the second sensing pattern. After the image projection system is calibrated, the image generating section 51 generates an image according to a request from a user or the like as a projection image projected from the projection device 20-1, and an image according to a request from a user or the like as a projection image projected from the projection device 20-2. The image generating section 51 outputs an image signal indicating the generated image to the image correcting section 52.

[0027] For example, when calibration of the image projection system starts, the image correction unit 52 outputs an image signal of the first sensing pattern projected from the projection device 20-1 to the projection device 20-1, and outputs an image signal of the second sensing pattern projected from the projection device 20-2 to the projection device 20-2. Furthermore, the image correction unit 52 uses the position correction amount calculated by the position calculation unit 43 of the image processing device 40 as geometric correction information and performs geometric correction of the projected image using the geometric correction information during and after the calibration process of the image projection system, thereby aligning the projected image projected from the projection device 20-1 and the projected image projected from the projection device 20-2. For example, when the position correction amount calculated by the position calculation unit 43 of the image processing device 40 is based on the first sensing pattern, the image correction unit 52 performs geometric correction on the projected image projected from the projection device 20-2 based on the geometric correction information. The image correction unit 52 outputs to the projection device 20-1 an image signal of the first sensing pattern projected from the projection device 20-1 during the calibration process and an image projected from the projection device 20-1 after the calibration in response to a request from a user, etc. The image correction unit 52 also performs geometric correction of the first sensing pattern projected from the projection device 20-2 during the calibration process and an image projected from the projection device 20-2 after the calibration in response to a request from a user, etc., and outputs the image signal after the geometric correction to the projection device 20-2.

[0028] The image correction unit 52 may be provided in the projection device instead of the image generation device 50. For example, when the position correction amount calculated by the position calculation unit 43 of the image processing device 40 is based on the first sensing pattern, the image correction unit 52 may be provided in the projection device 20-2.

[0029] 3. Operation of the embodiment 3 is a flowchart illustrating the operation of the embodiment. In step ST1, the image projection system projects a sensing pattern. In order to calibrate the image projection system, the projection devices 20-1 and 20-2 of the image projection system 10 project the first sensing pattern and the second sensing pattern, to which different color information is assigned, onto the screen Sc, and proceed to step ST2.

[0030] In step ST2, the image projection system acquires a captured image. The imaging device 30 of the image projection system 10 captures a mixed image of the first sensing pattern projected on the screen Sc from the projection device 20-1 and the second sensing pattern projected on the screen Sc from the projection device 20-2 at a non-fixed viewpoint, acquires a captured image showing the mixed image, and proceeds to step ST3.

[0031] In step ST3, the image projection system performs color separation processing. The color separation processing unit 41 in the image processing device 40 of the image projection system 10 generates a first separated image of the color information imparted to the first sensing pattern and a first separated image of the color information imparted to the second sensing pattern based on the color information imparted to the first and second sensing patterns and a color model that indicates the relationship between the color information of the captured image and the color information of the projected image and the background, and proceeds to step ST4.

[0032] In step ST4, the image projection system performs a corresponding point detection process. The corresponding point detection unit 42 in the image processing device 40 detects corresponding points in the first separated image and the second separated image generated by performing the color separation process in step ST3. The corresponding points may be detected using known techniques described in, for example, Japanese Patent Application Laid-Open No. 2000-348175 and Japanese Patent Application Laid-Open No. 2018-011302. The corresponding point detection unit 42 detects the corresponding points and proceeds to step ST5.

[0033] In step ST5, the image projection system performs processing for calculating the positions of the corresponding points. The position calculation unit 43 in the image processing device 40 calculates the display positions of the corresponding points detected in step ST4, and the process proceeds to step ST6.

[0034] In step ST6, the image projection system generates geometric correction information. The image correction unit 52 in the image generation device 50 calculates a position correction amount for each corresponding point based on the display positions of the corresponding points calculated in step ST5, so as to make the display positions of the corresponding points the same, thereby generating geometric correction information and completing the calibration of the image projection system. Thereafter, when a projection image such as video content is to be projected in response to a request from a user or the like, the process proceeds to step ST7.

[0035] In step ST7, the image projection system performs a projection process of the projection image. The image generation device 50 of the image projection system 10 generates an image signal of a projection image according to a request from a user or the like in the image generation unit 51. The image correction unit 52 performs geometric correction on the projection image generated in the image generation unit 51 based on the geometric correction information, and outputs the image signal of the projection image after the geometric correction to the projection devices 20-1 and 20-2, so that the projection images are projected from each of the projection devices 20-1 and 20-2 onto the screen Sc without causing inconsistency.

[0036] <3-1. Generation of separated images> Next, the operation of generating the separated images will be described. In the operation of generating the separated images, a separated image for each color information is generated based on color information of a captured image obtained by capturing a mixed image of projected images projected from a plurality of projectors with different color information added thereto, and a color model showing the relationship between the color information of the captured image and the color information of the projected image and the background.

[0037] The color model uses as parameters color information of the projected image that changes according to the spectral characteristics of the projection device and the imaging device that acquires the captured image, an attenuation coefficient that indicates the attenuation that occurs in the mixed image captured by the imaging device, and color information of the background.The color separation processing unit generates separated images for each color information based on the color model using parameters that minimize the difference between the color information of the captured image and the color information estimated by the color model.

[0038] In color separation processing, if the projection device 20 and the imaging device 30 have different spectral characteristics, the separation performance may be degraded. FIG. 4 illustrates the spectral sensitivity of the imaging device. The imaging device 30 has sensitivity in the wavelength range of the three primary colors (red R, green G, blue B), and the sensitivity of each color partially overlaps, for example, the imaging device 30 has sensitivity to green and blue for red wavelengths (610 nm to 750). Therefore, the color of the projected image may change in the captured image. FIG. 5 illustrates the color change of the captured image with respect to the projected image. The projection image input to the projection device 20 is, for example, "(R, G, B) = (1, 0, 0)". In this case, since the imaging device 30 has sensitivity to green and blue for red wavelengths, the captured image obtained changes to, for example, "(R, G, B) = (0.7, 0.2, 0.1)".

[0039] Furthermore, when the environment is taken into consideration, the captured image changes depending on the environment. Fig. 6 shows an example of a captured image that changes depending on the environment. For example, when the projected image has a color Cpro, the ambient light from the illumination light source has a color Cenv, and the projection surface of the screen Sc has a color Cback, the color Ccam of the projected image observed by the imaging device 30 is a value represented by the function of formula (1). Ccam = f(Cpro,Cenv,Cback) ···(1)

[0040] In this way, the projection image in the captured image acquired by the imaging device 30 may have a different color from the projection image input to the projection device 20 due to the spectral characteristics of the projection device 20 and the imaging device 30. In addition, the projection image observed by the imaging device 30 is affected not only by the color Cpro of the projected image, but also by the color Cenv of the ambient light and the color Cback of the projection surface of the screen Sc. In order to separate the mixed image with high accuracy, the color separation processing unit 41 of the image processing device 40 performs color separation processing using color information of the captured image captured from the mixed image of the projection images projected by the projection devices 20-1 and 20-2 with different color information added thereto, and a color model showing the relationship between the color information of the captured image and the color information of the projection image and the background. For ease of explanation, the color model does not take into account the influence of gamma characteristics in projection and imaging. In addition, it is assumed that the additivity of the projection light holds for the projection devices 20-1 and 20-2.

[0041] The color separation processing unit 41 separates the mixed image using a color model to generate a separated image. Here, if the input color to the projection device is pixel value P (=Pr, Pg, Pb), the (3×3) color conversion matrix Tcam due to the spectral characteristics of the imaging device 30, the (3×3) color conversion matrix Tpro due to the spectral characteristics of the projection device, and the background color consisting of the color of the ambient light and the color of the screen is pixel value BC (=Br, Bg, Bb), the pixel value CP (=Cr, Cg, Cb) of the captured image can be expressed as formula (2).

[0042]

number

[0043] Furthermore, the projection image projected from the projection device onto the screen Sc is attenuated depending on the incidence angle of the projection light on the screen Sc, the distance to the screen Sc, the reflectance of the projection surface of the screen Sc, etc. Therefore, the projection image captured by the imaging device 30 is set to a pixel value (αP) using the attenuation coefficient α of the projection image in the color model. Furthermore, if the input color of the projection device in this case is set to a pixel value (P'=Pr, Pg, Pb) and the background color is set to a pixel value (BC'=Br', Bg', Bb'), the color model shown in formula (2) becomes the color model shown in formula (3).

[0044]

number

[0045] Next, we will explain the case where, in order to calibrate the image projection system, a separated image showing a sensing pattern is generated with high precision for each sensing pattern from an image captured by capturing a mixed image of sensing patterns projected by projection devices 20-1 and 20-2 with different color information added thereto.

[0046] In this case, if the input color of the projector 20-1 is pixel value P1', the attenuation coefficient of the image projected by the projector 20-1 is "α1", the input color of the projector 20-2 is pixel value P2', and the attenuation coefficient of the image projected by the projector 20-2 is "α2", the color model will be the color model shown in formula (4). Note that pixel value P1' and pixel value P2' have the additivity of the projection light, and have the relationship shown in formula (5).

[0047]

number

[0048] Incidentally, equation (4) is a color model for one corresponding pixel in the projected image and the captured image, and the color separation processing unit 41 applies the color model to the entire captured image. For example, the captured image has a horizontal pixel count QH and a vertical pixel count QV. Also, the color information of a pixel position (x, y) in the captured image is pixel value CPx,y, and the attenuation coefficients of pixel position (x, y) are "α1x,y" and "α2x,y". Also, the attenuation coefficient vectors indicating the attenuation coefficients of each position on the screen are αv1 and αv2.

[0049] The color separation processing unit 41 uses the pixel values ​​CPx,y to estimate parameters (pixel values) “P1', P2', BC'” and parameters (subtraction coefficients) “α1, α2” that minimize the evaluation value EV shown in equation (6) and satisfy the condition of equation (7).

[0050]

number

[0051] FIG. 7 shows the sensing patterns and the projection state onto the screen. FIG. 7(a) illustrates the first sensing pattern SP1 projected from the projection device 20-1, and FIG. 7(b) illustrates the second sensing pattern SP2 projected from the projection device 20-2. The first sensing pattern is, for example, a pattern in which black dots are provided in a red rectangular area, and the second sensing pattern is, for example, a pattern in which black dots are provided in a blue rectangular area. FIG. 7(c) illustrates a state in which the first sensing pattern SP1 and the second sensing pattern SP2 are projected onto the screen Sc, and the area that does not correspond to either the first sensing pattern SP1 or the second sensing pattern SP2 is the background area SB. Note that the pixel position (x, y) is detected in advance as being either the background area, the area to which the color of the sensing pattern is applied, or the area of ​​the black dots of the sensing pattern. For example, if the first sensing pattern and the second sensing pattern are projected individually, it is clear which region the pixel position (x, y) corresponds to.

[0052] When the pixel position (x, y) is a pixel position of the sensing pattern, the color separation processing unit 41 uses color information of the corresponding area in the sensing pattern for the pixel values ​​P1', P2'. When the pixel position (x, y) is a pixel position of the background area, the pixel values ​​P1', P2' are set to "0".

[0053] The color separation processing unit 41 performs the calculation shown in equation (8) to generate pixel values ​​CP1 of a separation image showing the first sensing pattern projected by the projection device 20-1 based on the color model. Also, the color separation processing unit 41 performs the calculation shown in equation (9) to generate pixel values ​​CP2 of a separation image showing the second sensing pattern projected by the projection device 20-2 based on the color model.

[0054]

number

[0055] Next, the estimation of parameters will be described. In order to easily estimate parameters, the color separation processing unit 41 performs parameter estimation in a divided manner, and repeats a process of estimating other parameters using the estimation results, thereby estimating optimal values ​​of parameters that minimize the difference between the color information of the captured image and the color information estimated by the color model. For example, the color separation processing unit 41 performs a process of estimating a parameter indicating color information and a process of estimating a parameter indicating an attenuation coefficient in a divided manner, and uses the estimation result of one for the other and repeats the process, thereby using the converged estimation result as the optimal value of the parameter.

[0056] FIG. 8 is a flow chart illustrating an example of the parameter estimation operation. In step ST11, the color separation processing unit sets the parameters P1', P2', and BC' to initial values. When estimating the attenuation coefficient, the color separation processing unit 41 sets the parameters P1', P2', and BC' to initial values. In the spectral characteristics of a general projection device or imaging device, color information between input and output does not change significantly. Therefore, it is possible to speed up convergence by setting the initial values ​​of the parameters P1' and P2' to pixel values ​​that reflect the spectral characteristics. For example, as initial values ​​that take into account the spectral characteristics, the parameter P1' detected by projecting and capturing a sensing pattern to which color information has been added from the projection device 20-1 and the parameter P2' detected by projecting and capturing a sensing pattern to which color information has been added from the projection device 20-2 are used. In addition, when using a projection device, it is common to use a white screen under darkroom conditions, so it is possible to speed up convergence by setting the initial value of the parameter BC' to black. Also, by using pixel values ​​detected by capturing an image of a screen on which no sensing pattern is projected from the projectors 20-1 and 20-2 as the initial value of the parameter BC' as an initial value taking into account the spectral characteristics, it is possible to speed up the convergence. The color separation processing unit 41 sets the parameters P1', P2', and BC' to their initial values ​​and proceeds to step ST12.

[0057] Step ST12 is the start point of the x-direction loop process. The color separation processing unit 41 starts a process of sequentially moving the pixel position for calculating the attenuation coefficient in the x-direction of the captured image on a pixel-by-pixel basis, and then proceeds to step ST13.

[0058] Step ST13 is the start point of the y-direction loop process. The color separation processing unit 41 starts a process of sequentially moving the pixel position for calculating the attenuation coefficient in the y-direction of the captured image on a pixel-by-pixel basis, and then proceeds to step ST14.

[0059] In step ST14, the color separation processing unit calculates the attenuation coefficients α1x,y, α2x,y at the pixel position (x,y) based on the set parameters P1', P2', BC' and the pixel value CP of the captured image based on the formula (3), and then proceeds to step ST15.

[0060] Step ST15 is the end point of the y-direction loop process. When the color separation processing unit 41 has calculated the attenuation coefficient for each pixel position in the y direction, the color separation processing unit 41 proceeds to step ST16, and when the calculation of the attenuation coefficient is not completed, the color separation processing unit 41 repeats the processes from step ST13 to step ST15 to sequentially move the pixel position in the y direction and calculate the attenuation coefficient.

[0061] Step ST16 is the end point of the x-direction loop process. When the color separation processing unit 41 has calculated the attenuation coefficient for each pixel position in the x direction as well as the y direction, the color separation processing unit 41 proceeds to step ST17, and when the calculation of the attenuation coefficient is not completed, the color separation processing unit 41 repeats the processes of steps ST12 to ST16 to sequentially move the pixel position in the x direction as well as the y direction and calculate the attenuation coefficient for each pixel position in the captured image.

[0062] In step ST17, the color separation processing unit generates a separated image. The color separation processing unit 41 performs calculations of equations (8) and (9) using the set parameters P1', P2', and BC' and the attenuation coefficient vectors αv1 and αv2 calculated in the processes of steps ST12 to ST16, generates a separated image showing the first sensing pattern projected from the projector 20-1 and a separated image showing the second sensing pattern projected from the projector 20-2, and proceeds to step ST18.

[0063] In step ST18, the color separation processing unit distinguishes between the projection area and the background area. For each separated image generated in step ST17, the color separation processing unit 41 distinguishes between the projection area and the background area based on the pixel values ​​of the separated image. For example, the color separation processing unit 41 uses the color information added to the sensing pattern to distinguish an area with similar color information as the projection area, and distinguishes an area other than the projection area as the background area. After distinguishing between the projection area and the background area, the color separation processing unit 41 proceeds to step ST19.

[0064] In step ST19, the color separation processing unit extracts pixel values ​​of the projection area and the background area. The color separation processing unit 41 extracts pixel values ​​from the projection area and the background area determined in step ST18. Note that, if there is variation in the pixel values ​​extracted from the projection area and the pixel values ​​extracted from the background area, the color separation processing unit 41 may use a statistical value calculated by statistical processing of the pixel values, such as an average value, a median value, or a mode value, as the extracted pixel value. The color separation processing unit 41 extracts pixel values ​​PE1', PE2' of the projection area and a pixel value BCE' of the background area, and proceeds to step ST20.

[0065] In step ST20, the color separation processing unit determines whether or not updating of the parameters is necessary. The color separation processing unit 41 calculates the differences between the parameters P1', P2', BC' used in calculating the attenuation coefficients α1, α2 and the pixel values ​​PE1', PE2', BEC' calculated in step ST19, respectively, and determines that updating is necessary if any of the calculated differences is greater than a preset threshold, and proceeds to step ST21. In addition, the color separation processing unit 41 determines that updating is unnecessary, that is, the parameters have converged to optimal values, if the calculated differences are less than or equal to the respective preset thresholds, and proceeds to step ST22.

[0066] In step ST21, the color separation processing unit updates the parameters. The color separation processing unit 41 updates the parameters for which the calculated difference is greater than a preset threshold value, and returns to step ST12 with the pixel values ​​extracted in step ST19 as parameters to be used in calculating the attenuation coefficients α1 and α2.

[0067] When the process proceeds from step ST20 to step ST22, the color separation processing unit outputs a separated image. The color separation processing unit 41 outputs the separated image generated in step ST17 to the corresponding point detection unit 42 because the estimation result has converged.

[0068] By performing this type of processing in the color separation processing unit 41, a separated image showing the first sensing pattern and a separated image showing the second sensing pattern are generated from an image obtained by capturing a mixed image of the first sensing pattern and the second sensing pattern with parameters set to optimal values ​​in a color model, thereby making it possible to separate the sensing patterns more accurately than in the past.

[0069] Fig. 9 illustrates an example of a separated image. Fig. 9(a) illustrates a state in which the first sensing pattern SP1 and the second sensing pattern SP2 are projected onto the screen Sc, and an area that does not correspond to either the first sensing pattern SP1 or the second sensing pattern SP2 is a background area SB. The color separation processing unit 41 performs the process shown in Fig. 8, and can generate a separated image showing the first sensing pattern SP1 as shown in Fig. 9(b) and a separated image showing the second sensing pattern SP2 as shown in Fig. 9(c) from an image obtained by capturing the first sensing pattern SP1 and the second sensing pattern SP2 projected onto the screen Sc as shown in Fig. 9(a).

[0070] Fig. 10 is a diagram showing an example of a parameter estimation operation. In order to facilitate understanding of the operation, it is assumed that a first sensing pattern SP1 shown in Fig. 10(a) and a second sensing pattern SP2 shown in Fig. 10(b) are projected onto a screen Sc, and an image shown in Fig. 10(c) is acquired by the imaging device 30.

[0071] In order to obtain color information in the background area and the projection area of ​​each projection device, the color separation processing unit 41 binarizes the image, discriminates the projection area, and obtains the projection area of ​​each projection device by the difference with the projection area other than its own. The background area is an area that does not belong to any projection area.

[0072] (d) and (e) of Fig. 10 show the separated image generated in step ST17, where the area of ​​the first sensing pattern is pixel value "P1'α1" and the area of ​​the second sensing pattern is pixel value "P2'α2." Note that (f) of Fig. 10 shows a mask indicating the area of ​​the pixel value "P1'α1" in the separated image, and (g) of Fig. 10 shows a mask indicating the area of ​​the pixel value "P2'α2" in the separated image.

[0073] The color separation processing unit 41 determines the projection area in order to obtain color information in the projection area of ​​the first sensing pattern. Specifically, the color separation processing unit 41 determines the projection area of ​​the first sensing pattern shown in (h) of Fig. 10 by binarizing an image extracted by applying a mask shown in (f) of Fig. 10 to the captured image shown in (c) of Fig. 10 using the color information assigned to the first sensing pattern.

[0074] Moreover, the color separation processing unit 41 determines the projection area in order to obtain color information in the projection area of ​​the second sensing pattern. Specifically, the image extracted by applying the mask shown in (g) of Fig. 10 to the captured image shown in (c) of Fig. 10 is binarized using the color information assigned to the second sensing pattern, thereby determining the projection area of ​​the second sensing pattern shown in (i) of Fig. 10.

[0075] Furthermore, the color separation processing unit 41 determines the background region to obtain color information in the background region. Specifically, the masked region (indicated in black) in both (f) of Fig. 10 and (g) of Fig. 10 is determined to be the background region as shown in (j) of Fig. 10.

[0076] (k) of Figure 10 illustrates an example image of the projection area of ​​the first sensing pattern determined by applying the mask shown in (h) of Figure 10 to the captured image shown in (c) of Figure 10, and the image of the projection area of ​​the first sensing pattern includes not only the first sensing pattern SP1 but also a part of the second sensing pattern SP2.

[0077] (l) of Figure 10 illustrates an example image of the projection area of ​​the second sensing pattern determined by applying the mask shown in (i) of Figure 10 to the captured image shown in (c) of Figure 10, and the image of the projection area of ​​the second sensing pattern includes not only the second sensing pattern SP2 but also a part of the first sensing pattern SP1.

[0078] (m) of Figure 10 shows an example of an image of the background area determined by applying the mask shown in (j) of Figure 10 to the captured image shown in (c) of Figure 10, and the image of the background area includes a portion of the first sensing pattern SP1 and a portion of the second sensing pattern SP2.

[0079] In this way, when the region to be discriminated includes other regions, it is determined in step ST20 that parameter updating is necessary, and the parameters used to calculate the attenuation coefficients α1 and α2 are updated as shown in step ST21. Furthermore, when the processes of steps ST12 to ST21 are repeated, the other regions included in the region to be discriminated gradually decrease, and when the region to be discriminated does not include other regions, the parameters converge, and a separated image in which the first sensing pattern SP1 and the second sensing pattern SP2 are accurately separated can be obtained.

[0080] In this way, since a separated image is generated using converged parameters, a separated image with high accuracy can be obtained even in cases where unseparated portions occur in the conventional separation method.

[0081] Furthermore, when estimating parameters P1', P2', and BC' from the color information of the projection light from each projection device and the background color, the color separation processing unit 41 utilizes the distribution of each extracted color information in a three-dimensional color space (e.g., RGB color space). For example, the parameters P1' and P2' are estimated by a regression line or principal component analysis for the color distribution of the projection light. The parameter BC' utilizes a statistical value of pixel values ​​in the background region, such as the average value, median value, or mode. Furthermore, if an image of the background has been acquired, the color information from that may be used.

[0082] In the above operation, the estimation of the optimal values ​​of the parameters P1', P2', BC' and the calculation of the attenuation coefficients α1, α2 are alternately performed until the parameters P1', P2', BC' converge, but the optimal values ​​of the parameters P1', P2', BC' may be estimated using other methods. For example, the pixel values ​​P1'α1+BC', P2'α2+BC' of the separated images may be obtained by directly solving the nonlinear minimization problem of the equation (6).

[0083] As described above, according to the present technology, it is possible to accurately separate each projection image from an image captured by an imaging device, which is a composite image of projection images simultaneously projected onto a screen from a plurality of projection devices. In addition, after detecting corresponding points of the separated projection images, the spatial positions of the projection images are obtained from the detected corresponding point information, and the image can be corrected so that inconsistencies in the overlapping areas of the projection images are accurately eliminated.

[0084] Furthermore, since it is possible to accurately separate the separated images and generate geometric correction information by using the captured images obtained by an imaging device with a non-fixed viewpoint, there is no need to fix the imaging device in a specified position as in the past, making it easier to calibrate the image projection system.

[0085] <3-2. Other operations of the embodiment> <3-2-1. Sensing patterns> The sensing pattern is not limited to an image including dots as shown in FIG. 7, but may be a gray code pattern or a checkered pattern that does not have color information and is given different color information for each projection device.

[0086] Fig. 11 illustrates an example of a sensing pattern (structured light) projected from a projection device, and the sensing pattern may be a checkered pattern to which different color information is added, as shown in (a) and (b) of Fig. 11. In addition, the patterns shown in (c) and (d) of Fig. 11 shown in "Image Processing Apparatus and Method, Data, and Recording Medium" of International Publication No. 2017 / 104447 may be used by adding color information.

[0087] When creating a sensing pattern by adding color information to a pattern that does not have color information, the color information to be added is generated using the pixel values ​​of the pattern that does not have color information. For example, when generating color information P=(Pr, Pg, Pb) of the three primary colors, when a pixel to which color information is to be added in a pattern that does not have color information has a pixel value Y, the color information PY of the pixel value Y is set as PY=(Pr(Y), Pg(Y), Pb(Y)). Note that Pr(Y) has a pixel value Y for the red component, Pg(Y) has a pixel value Y for the green component, and Pb(Y) has a pixel value Y for the blue component. In this way, a process of generating color information using the pixel value of the pixel to which color information is to be added is performed for each pixel of the pattern that does not have color information as a pixel to which color information is to be added, and the generated color information is added to the pattern that does not have color information, thereby creating a sensing pattern that has color information.

[0088] <3-2-2. Gamma characteristics> In the above <3-1. Operation of the embodiment>, the case where the influence of the gamma characteristic is not taken into consideration has been described, but even when the influence of the gamma characteristic is taken into consideration, the color separation process can be performed in the same manner. For example, when the imaging device 30 performs gamma correction to generate a captured image, the color separation processing unit 41 performs degamma processing on the color information C(x,y) of the captured image used in equation (6), converts it into information in a linear color space, and uses the color information after degamma processing. In this way, by using the color information that has been degamma processed, the projected image can be separated with high accuracy as in the above embodiment.

[0089] <3-2-3. When there are three or more projection devices> In the above <3-1. Operation of the embodiment>, the case where two projectors are used has been described, but the formula (6) can be expanded to three or more projectors to accurately separate the projection images for each projector. In addition, when there are four or more projectors, if the color information is information of a three-dimensional color space, there will be multiple optimal solutions for the attenuation coefficient that satisfies the formula (6), and color separation processing cannot be performed correctly. Therefore, in order to prevent there from being multiple optimal solutions, it is necessary to use the continuity of adjacent pixel colors and information on the projection image as constraint conditions.

[0090] In addition, in the image projection system, a group of a predetermined number of projection devices (for example, two or three devices) is set for the multiple projection devices, and at least one projection device in a group is included in another group. Also, different color information is given within the group. By forming groups in this way and performing the above-mentioned <3-1. Operation of the embodiment> for each group, the positional relationship of the projected images from the multiple projection devices becomes clear based on the corresponding points for each separated image detected for each group, and geometric correction information can be generated to match the corresponding points for each separated image detected for each group. Therefore, even when a large number of projection devices are used, the image can be corrected by using the geometric correction information so that inconsistencies in the area where the projected images overlap are accurately eliminated.

[0091] <3-2-4. About color information to be added> As long as the above-mentioned parameters P1' and P2' are not "P1' = P2'", the projected images projected by each projector can be separated from the mixed image. However, when estimating an area from the distribution of a three-dimensional color space, the greater the difference in distribution, the more accurately the area can be estimated. Therefore, the color information to be added to the projected image can be selected so that the inner product of the color vector corresponding to the parameter P1' and the parameter P2' is minimized. For example, in the case of an RGB color space, if two colors are selected from red (R, G, B) = (1, 0, 0), green (R, G, B) = (0, 1, 0), and blue (R, G, B) = (0, 0, 1), the projected images projected by each projector can be easily separated from the mixed image.

[0092] <3-2-5. About color proofing> Incidentally, in a projection device, a method such as color calibration may be used to project an image with correct color expression. In this method, an image is projected from a projection device in various different colors, and an input signal value is determined so that the color of the projected image captured is the correct color in real space. In this technology, if color calibration is performed using color information added to the sensing pattern to be projected, not only can the geometric correction of the image be performed so that inconsistencies in the area where the projected image overlaps are accurately eliminated, but the projected image can be projected in the correct color. In this case, if the color information to be added is set to cover the three primary colors by switching the color combination every time the projection image is captured by the imaging device, color calibration can be performed with high accuracy. In this way, since color calibration is possible using the color information added to the sensing pattern to be projected, there is no need to perform color calibration in advance, and the image projection system can be calibrated efficiently.

[0093] <3-2-6. Projection and imaging conditions> In addition, in the color model of this technology, the projection light is expressed by a scalar multiplication of a vector, so when the captured image is saturated positively or negatively (when whiteout or blackout occurs), the color balance of the projection light is lost in the saturated area, and separation accuracy decreases. Therefore, the projection conditions and the capture conditions are set so that the pixel values ​​of the projected image and the captured image are in a wide range of values ​​without saturation. For example, it is effective to adjust the projection image by referring to the histogram of pixel values ​​for each color in the captured image so that the range of the input image is as wide as possible without saturating.

[0094] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed. Alternatively, the program can be installed in a general-purpose computer capable of executing various processes and executed.

[0095] For example, the program can be pre-recorded on a recording medium such as a hard disk, a solid state drive (SSD), or a read only memory (ROM). Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD (registered trademark)), a magnetic disk, or a semiconductor memory card. Such removable recording media can be provided as so-called package software.

[0096] In addition to being installed in a computer from a removable recording medium, the program may be transferred wirelessly or by wire from a download site to a computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it in a built-in recording medium such as a hard disk.

[0097] In addition, the effects described in this specification are merely examples and are not limited, and there may be additional effects that are not described. In addition, the present technology should not be interpreted as being limited to the above-mentioned embodiments of the technology. The embodiments of the technology disclose the present technology in the form of examples, and it is self-evident that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. In other words, the scope of the claims should be taken into consideration in order to determine the gist of the present technology.

[0098] Moreover, the image processing device of the present technology can also have the following configuration. (1) A color separation processing unit that generates a separated image for each color information based on color information of a captured image obtained by capturing a mixed image of projected images projected from a plurality of projection devices with different color information added thereto, and a color model that indicates the relationship between the color information of the captured image and the color information of the projected image and the background. An image processing device comprising: (2) The image processing device described in (1), in which the color model uses as parameters color information of the projected image that changes according to the spectral characteristics of the projection device and the imaging device that acquires the captured image, and an attenuation coefficient that indicates the attenuation that occurs in the mixed image captured by the imaging device. (3) The image processing device described in (2), wherein the color separation processing unit generates a separated image for each color information based on the color model using the parameters that minimize the difference between the color information of the captured image and the color information estimated by the color model. (4) The image processing device according to any one of (1) to (3), wherein the projected image to which the color information is added is structured light. (5) The image processing device according to any one of (1) to (4), wherein the color separation processing unit uses a captured image after de-gamma processing when gamma correction is performed in an imaging device that captures the mixed image. (6) The image processing device according to any one of (1) to (5), wherein the mutually different color information is set so that an inner product of color vectors corresponding to the color information is minimized. (7) The image processing device according to any one of (1) to (6), wherein the projected image and the captured image are non-saturated images. (8) The image processing device according to any one of (1) to (7), wherein the captured image is an image acquired by an imaging device with a non-fixed viewpoint. (9) The image processing device according to any one of (1) to (8), further comprising an image correction unit that corrects a projection image projected from the projection device. (10) The image processing device according to (9), wherein the image correction unit performs color calibration of the projected image using color information assigned to the separated images. (11) A corresponding point detection unit that detects corresponding points between the separated images for each of the color information generated by the color separation processing unit, The image processing device according to (9), wherein the image correction unit corrects the projected image using geometric correction information that aligns corresponding points for each of the separated images detected by the corresponding point detection unit. (12) A predetermined number of groups are formed for the plurality of projection devices, and at least one projection device in each group is included in another group, and within each group, different color information is assigned to each projection device to project the projection image, The color separation processing unit generates a separation image for each of the groups, the corresponding point detection unit detects corresponding points for each of the groups; The image processing device according to (11), wherein the image correction unit corrects the projected image using geometric correction information that aligns corresponding points for each of the separated images detected for each of the groups by the corresponding point detection unit. [Explanation of symbols]

[0099] 10. Image projection system 20, 20-1, 20-2...Projection device 30 Imaging device 40...Image processing device 41 Color separation processing unit 42 Corresponding point detection unit 43...Position calculation section 50...Image generating device 51...Image generation unit 52 Image correction unit

Claims

1. A color separation processing unit that generates a separated image for each color information based on color information of a captured image obtained by capturing a mixed image of projected images projected from a plurality of projection devices with different color information added thereto, and a color model that indicates a relationship between the color information of the captured image and the color information of the projected image and the background. An image processing device comprising:

2. The color model uses color information of the projected image that changes according to the spectral characteristics of the projection device and the imaging device that acquires the captured image, and an attenuation coefficient that indicates attenuation that occurs in the mixed image captured by the imaging device, as parameters. The image processing device according to claim 1 .

3. The color separation processing unit generates a separated image for each of the color information based on the color model, using the parameters that minimize a difference between the color information of the captured image and the color information estimated by the color model. The image processing device according to claim 2 .

4. The projected image with the color information is structured light. The image processing device according to claim 1 .

5. When gamma correction is performed in an imaging device that captures the mixed image, the color separation processing unit uses an image after de-gamma processing. The image processing device according to claim 1 .

6. The different color information is set so that the inner product of the color vectors corresponding to the different color information is minimized. The image processing device according to claim 1 .

7. The projected image and the captured image are images that are not saturated. The image processing device according to claim 1 .

8. The captured image is an image acquired by an imaging device with a non-fixed viewpoint. The image processing device according to claim 1 .

9. The projection device further includes an image correction unit that corrects the projected image projected from the projection device. The image processing device according to claim 1 .

10. The image correction unit performs color correction of the projected image using color information assigned to the separated image. The image processing device according to claim 9 .

11. a corresponding point detection unit that detects corresponding points between the separated images for each of the color information generated by the color separation processing unit, The image correction unit corrects the projected image using geometric correction information for matching the corresponding points for each of the separated images detected by the corresponding point detection unit. The image processing device according to claim 9 .

12. A predetermined number of groups are formed for the plurality of projection devices, and at least one projection device in each group is included in another group, and within each group, different color information is assigned to each projection device to project the projection image, The color separation processing unit generates a separation image for each of the groups, the corresponding point detection unit detects corresponding points for each of the groups; The image correction unit corrects the projected image using geometric correction information for matching the corresponding points for each of the separated images detected for each of the groups by the corresponding point detection unit. The image processing device according to claim 11.

13. A color separation processing unit generates a separated image for each color information based on a color model composed of color information of the projected images and color information of a background color from a captured image obtained by capturing a mixed image of projected images projected by a plurality of projectors with different color information added thereto. An image processing method comprising:

14. A program for causing a computer to execute a process of separating each projection image from a captured image obtained by capturing a mixed image of projection images, a step of acquiring a captured image by capturing a mixed image of projected images projected from a plurality of projectors with different color information; generating a separated image for each color information from the captured image based on a color model configured with color information of the projected image and color information of a background color; A program for causing the computer to execute the above.

15. A color separation processing unit generates a separated image for each color information based on a color model configured with color information of the projected images and color information of a background color from a captured image obtained by capturing a mixed image of projected images projected from a plurality of projectors with different color information added thereto; detecting corresponding points between the separated images for each of the color information generated by the color separation processing unit using a corresponding point detection unit; Correcting the projection images projected from the plurality of projection devices by an image correction unit using geometric correction information for matching the corresponding points for each of the separated images detected by the corresponding point detection unit.

23. An image projection method comprising:

Images