Projection mapping system, image processing method, and program

Abstract

The generated image is projected onto a projection target space with various shapes and positions using a simple configuration and processing. [Solution] One aspect of the present disclosure is a projection mapping system comprising: a spatial image acquisition unit that acquires a spatial image captured by a camera device of an area including the projection target space of a projector device; a conversion unit that converts the spatial image captured by the camera device into an image corresponding to the projection target space of the projector device; a generation unit that generates a generated image, and a converted generated image acquisition unit that acquires a converted generated image obtained by converting the generated image by the conversion unit; and an output unit that outputs the converted generated image to the projector device.

Description

Technical Field

[0001] The present invention relates to a projection mapping system, an image processing method, and a program.

Background Art

[0002] Projection mapping technology is a technology that projects an image onto a projection space such as an object or a space, and deforms the image according to the shape and characteristics in the projection space to express a realistic three-dimensional effect and movement. With this technology, it is possible to use any surface as a screen, from the outer wall of a building, the set of a stage, to an object with an irregular shape. In addition, this technology is used in a wide range of fields such as events, advertisements, and entertainment, and it is possible to provide a new experience by giving a visual impact.

[0003] As this type of technology, for example, as described in Patent Document 1, it is known to perform projection mapping in real time following a dynamic three-dimensional object whose position and orientation change. Conventionally, it is also known to generate and project an image of the imagination of an experiencer on the spot by projecting an image generated by an image generation AI using various software technologies such as Lumabox.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Software-based methods like Lumabox allow for the instant projection of highly visually impressive images without the need for content creation, by using image generation AI to create images that match the shape of the projection space. However, this method requires aligning the projected image with the projection space while checking the image projected into the real space. Therefore, if the shape or position of structures in the projection space changes, the alignment of the projected image with the projection space must be manually re-adjusted.

[0006] Furthermore, as described in Patent Document 1, by tracking the projection target using motion capture with markers, calibrating the projection range of the projector and the imaging range of the camera, and performing stereo matching by projecting a random dot pattern, it is possible to automatically reposition the projection even if the position or shape of the projection target changes, or by projecting onto a three-dimensional object. However, setting up such automatic repositioning requires specialized knowledge, and the manual setting process is time-consuming. Furthermore, calibration between the projection range of a projector and the imaging range of a camera often assumes a pinhole camera model, which presents a problem as it cannot be applied to special short-focus projectors.

[0007] This disclosure is made in view of these circumstances and aims to provide a projection mapping system, an image display method, and a program that can project generated images onto projection target spaces arranged in various shapes and positions with a simple configuration and processing. [Means for solving the problem]

[0008] This disclosure has been made to solve the above-mentioned problems, and one aspect of this disclosure is a projection mapping system comprising: a spatial image acquisition unit that acquires a spatial image captured by a camera device of an area including the projection target space of a projector device; a conversion unit that converts the spatial image captured by the camera device into an image corresponding to the projection target space of the projector device; a generation unit that generates a generated image, and a converted generated image acquisition unit that acquires a converted generated image obtained by converting the generated image by the conversion unit; and an output unit that outputs the converted generated image to the projector device.

[0009] Another aspect of the present disclosure is an image processing method in which a computer performs the steps of: acquiring a spatial image captured by a camera device of an area including the projection target space of a projector device; converting the spatial image captured by the camera device into an image corresponding to the projection target space of the projector device; acquiring a converted generated image obtained by converting a generated image generated by a generation unit into an image corresponding to the projection target space of the projector device; and outputting the converted generated image to the projector device.

[0010] Another aspect of the present disclosure is a program that causes a computer to perform the following steps: acquire a spatial image by capturing an area including the projection target space of a projector device with a camera device; convert the spatial image captured by the camera device into an image corresponding to the projection target space of the projector device; acquire a converted generated image by converting the generated image generated by the generation unit into an image corresponding to the projection target space of the projector device; and output the converted generated image to the projector device. [Effects of the Invention]

[0011] According to one aspect of the present invention, a generated image can be projected onto a projection target space arranged in various shapes and positions using a simple configuration and processing. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example of a projection mapping system 1 in the first embodiment. [Figure 2] This figure shows an example of a spatial image in the embodiment. [Figure 3] This figure shows an example of a spatial image and a transformed spatial image in an embodiment. [Figure 4] This is a block diagram showing an example of the projection mapping system 1A according to the second embodiment. [Figure 5] This figure shows the second embodiment, (a) spatial image at the time of reference image projection, (b) reference image, (c) spatial image of the projection target space, and (d) transformed spatial image. [Figure 6] Figure 6 is a flowchart showing an example of the processing procedure for the projection mapping system 1 of the third embodiment. [Figure 7] (a) Spatial image, (b) Converted generated image. [Figure 8] This flowchart shows an example of the processing procedure for the projection mapping system 1 of the fourth embodiment. [Figure 9] This figure shows (a) the spatial image, (b) the transformed spatial image, and (c) the transformed generated image. [Figure 10] This is a block diagram showing an example of an image processing apparatus 100B according to the fifth embodiment. [Figure 11] This flowchart shows another example of the processing procedure when performing color correction in the embodiment. [Figure 12] This flowchart shows another example of the processing procedure when performing color correction in the embodiment. [Figure 13] This is a block diagram showing an example of the projection mapping system 1 according to the sixth embodiment. [Figure 14] This is a diagram illustrating the transformation of the projection mapping system 1 according to the sixth embodiment. [Modes for carrying out the invention]

[0013] Hereinafter, a projection mapping system, an image processing method, and a program to which the present invention is applied will be described with reference to the drawings.

[0014] (First Embodiment) Hereinafter, the projection mapping system according to the first embodiment will be described. FIG. 1 is a block diagram showing an example of a projection mapping system 1 according to the first embodiment. The projection mapping system 1 includes, for example, an image processing apparatus 100, a projector apparatus 200, and a camera apparatus 300. The projector apparatus 200 and the camera apparatus 300 are installed at arbitrary positions by a user of the image processing apparatus 100 who is an experiencer. The projector apparatus 200 projects an image onto an object in a projection target space according to the installation position, and the camera apparatus 300 photographs a photographing space including the object in the projection target space. The image processing apparatus 100, the projector apparatus 200, and the camera apparatus 300 are connected via, for example, a wired communication line or a communication network. The communication network is connected by a private network such as Bluetooth (registered trademark), and local 5G or WiFi (registered trademark). Note that the projection mapping system 1 can be used, for example, in the planning of an event space, dynamic advertising, a product design tool, and a creative experience in facilities such as museums and educational institutions.

[0015] The image processing apparatus 100 is an information processing apparatus in which a computer such as a CPU (Central Processing Unit) executes a program stored in a program memory. The image processing apparatus 100 includes, for example, a spatial image acquisition unit 110, a conversion unit 120, a converted generated image acquisition unit 130, and an output unit 140. The spatial image acquisition unit 110, the conversion unit 120, the converted generated image acquisition unit 130, and the output unit 140 are functional units realized by a computer executing a program.

[0016] The spatial image acquisition unit 110 acquires a spatial image captured by the camera device 300 of the area including the projection target space of the projector device 200. Figure 2 shows an example of a spatial image in the embodiment. In the spatial image shown in Figure 2, the projection target space is, for example, a space where objects (three-dimensional objects, etc.) are placed, which are intended to be moved, deformed, added, or removed by the user. The projection target space may be a part of a room, the exterior wall of a building, or any space where projection mapping can be performed in general. Furthermore, if the projection target space is dark, the spatial image may be acquired with the projection target space illuminated by projecting a white or gray-filled image from the projector device 200.

[0017] The conversion unit 120 converts the spatial image captured by the camera device 300 into an image corresponding to the projection target space of the projector device 200 (hereinafter referred to as the converted spatial image). The conversion unit 120 may convert the spatial image into the converted spatial image using a pre-set calculation formula, or it may convert the spatial image into the converted spatial image by referring to table data, or it may create the converted spatial image by gradually deforming the spatial image to obtain a predetermined shape. Figure 3 shows an example of a spatial image and a transformed spatial image in the embodiment. The conversion unit 120 calculates a conversion rule that appropriately converts the spatial image into a converted spatial image with a shape corresponding to the projection target space, as shown in Figure 3, in order to match the projected image projected from the projector device 200 to the projection target space. In this embodiment, since the generation unit 132 generates the generated image, the conversion unit 120 performs alignment in image space. The alignment technique in image space can be any general image alignment technique (e.g., image registration technique) as long as it is possible to set the correspondence between the spatial image and the converted spatial image, and the method is not particularly limited. Because alignment in image space is performed by this method, it is not affected by image distortion, calibration between the projector device 200 and the camera device 300 is unnecessary, and furthermore, even if a camera device 300 that cannot be represented by a pinhole camera model is used, it is possible to align the projected image from the projector device 200 to the projection target.

[0018] The converted generated image acquisition unit 130 includes a generation unit 132 that generates generated images. The converted generated image acquisition unit 130 acquires converted generated images obtained by the conversion unit 120. The generation unit 132 has the function of receiving requests and generating images according to the requests. For example, the generation unit 132 may input a request to a machine learning model that generates a generation AI and acquire the image output from the model as a generated image. For the generation unit 132, the request is, for example, a prompt. A prompt may be a string such as "Colorful Objects", but is not limited to this, and may also be a request that combines an image that represents the image desired by the user with a string such as "I want an image with a similar atmosphere to this image". Furthermore, the generated images generated by the generation unit 132 are not limited to still images, but may also be moving images.

[0019] The image generation AI used in the generation unit 132 may be a machine learning model based on large-scale training data, such as a diffusion model. This makes it possible to generate high-quality images in various variations. Furthermore, methods such as img2img or ControlNet may be used as methods for generating transformed spatial images according to the structure of the projection target. When using ControlNet, edge images or depth images can be used as input to ControlNet. For example, by using monocular depth estimation technology based on a machine learning model, depth can be estimated without using special hardware.

[0020] The converted generated image acquisition unit 130 acquires the converted generated image by having the generated image generated by the generation unit 132 converted by the conversion unit 120. For example, if the generated image has a shape corresponding to a spatial image, as shown in the left figure of Figure 3, the conversion unit 120 creates the converted generated image by converting it to a shape corresponding to the projection target space, as shown in the right figure of Figure 3. Furthermore, even if the shape of the generated image is different from the shape shown in the left figure of Figure 3, the conversion unit 120 may convert it to a shape corresponding to the projection target space, as shown in the right figure of Figure 3.

[0021] The output unit 140 outputs the converted generated image to the projector device 200. This allows the image processing device 100 to project the aligned generated image from the projector device 200.

[0022] As described above, according to the projection mapping system 1 of the first embodiment, a spatial image is acquired by the camera device 300, which captures an area including the projection target space of the projector device 200. The spatial image captured by the camera device 300 is converted into an image corresponding to the projection target space of the projector device 200. The converted generated image is acquired by converting the generated image and output to the projector device 200. With this projection mapping system 1, even with a simple configuration and using projector devices 200 of various specifications, the generated image generated at the request of the user can be projected onto the projection target space where there are three-dimensional objects. Furthermore, according to projection mapping system 1, even if the position or shape of an object changes, dynamic projection mapping can be performed by recapturing the spatial image and converting the generated image into an image corresponding to the projection target space. For example, a user experiencing projection mapping can intuitively manipulate the projection target space to set the shape and arrangement of the projection target, and then, by inputting arbitrary prompts to the generation unit 132 (image generation AI), they can project their desired content (still images, videos) onto the projection target space.

[0023] (Second Embodiment) Figure 4 is a block diagram showing an example of a projection mapping system 1A according to the second embodiment. Figure 5 shows the second embodiment, including (a) the spatial image during reference image projection, (b) the reference image, (c) the spatial image of the projection target space, and (d) the transformed spatial image.

[0024] The projection mapping system 1A of the second embodiment differs from the first embodiment in that it projects a reference image (Figure 5(b)) onto the projection target space using a projector device 200, captures the reference image using a camera device 300, and converts the spatial image (Figure 5(c)) into an image corresponding to the projection target space (Figure 5(d)) based on the reference image and the spatial image (Figure 5(a), hereinafter referred to as the projected reference image) captured by the conversion unit 120A. The differences will be explained below in detail.

[0025] The image processing apparatus 100A of the second embodiment includes a reference image output unit 150. The reference image output unit 150 outputs a reference image to the projector device 200. The reference image is preferably an image containing many feature points, as shown in the upper right of Figure 5. Feature points are points extracted by existing feature extraction algorithms, and are extracted, for example, by the brightness distribution, color distribution, position of objects, edges, etc., in the image.

[0026] The conversion unit 120A includes, for example, a reference image acquisition unit 122, a projected reference image acquisition unit 124, and a correspondence relationship information acquisition unit 126. The reference image acquisition unit 122 acquires a reference image from the reference image output unit 150. The projection reference image acquisition unit 124 acquires a projection reference image captured by the camera device 300 from the reference image projected from the projector device 200 into an area including the projection target space. The projection reference image corresponds to the spatial image captured by the spatial image acquisition unit 110 while the reference image is being projected by the projector device 200.

[0027] The correspondence information acquisition unit 126 acquires correspondence information that shows the correspondence between the reference image and the projected reference image. The correspondence information acquisition unit 126 searches, for example, where in the projected reference image a feature point in the reference image corresponds to. The correspondence information acquisition unit 126 acquires the correspondence between the location in the reference image where the same feature point exists and the location in the projected reference image. Based on this, the correspondence information acquisition unit 126 performs alignment between the reference image and the projected reference image. The correspondence relationship information acquisition unit 126 may perform alignment using, for example, machine learning-based DenseMatching technology to enable high-speed execution while considering interactivity.

[0028] The conversion unit 120A converts the spatial image into an image corresponding to the projection target space based on the correspondence information.

[0029] As described above, according to the projection mapping system 1A of the second embodiment, a reference image is acquired, the reference image projected from the projector device 200 into an area including the projection target space is captured by the camera device 300 to acquire a projected reference image, and correspondence information indicating the correspondence between the reference image and the projected reference image is acquired. Therefore, the reference image and the projected reference image can be aligned with high accuracy using feature points in the reference image. As a result, with the projection mapping system 1A, even with a simple configuration and processing, generated images can be projected with high accuracy into projection target spaces arranged in various shapes and positions.

[0030] (Third embodiment) Figure 6 is a flowchart showing an example of the processing procedure for the projection mapping system 1 of the third embodiment, and Figure 7 shows (a) a spatial image and (b) a converted generated image. The converted generated image acquisition unit 130 acquires a spatial image (Figure 7(a)) captured by the camera device 300 (step S100a), causes the generation unit 132 to generate a generated image (Figure 7(b)) corresponding to the spatial image (step S102a), and obtains a converted generated image by having the generation unit 120 convert the generated image (step S104a). Then, the output unit 140 outputs the converted generated image to the projector device 200, causing the projector device 200 to project the converted spatial image onto the projection target space (step S106a).

[0031] In this processing procedure, the converted generated image acquisition unit 130 may estimate a depth image from the acquired spatial image by monocular depth estimation, and the generation unit 132 may acquire a generated image based on the estimated depth image. At this time, the user inputs arbitrary prompt information to the generation unit 132, for example, using a terminal device (not shown). The converted generated image acquisition unit 130 then takes the generated image generated by the generation unit 132 as input and has the conversion unit 120 perform alignment conversion, thereby acquiring a converted generated image with a shape corresponding to the projection target space. Furthermore, the third embodiment may also be applied to the projection mapping system 1A of the second embodiment.

[0032] (Fourth embodiment) Figure 8 is a flowchart showing an example of the processing procedure for the projection mapping system 1 of the fourth embodiment. Figure 9 shows (a) a spatial image, (b) a converted spatial image, and (c) a converted generated image. The converted generated image acquisition unit 130 acquires the spatial image (Figure 9(a)) captured by the camera device 300 (step S100b), converts the spatial image using the conversion unit 120 (step S102b), and then acquires the converted generated image (Figure 9(c)) by generating an image corresponding to the converted spatial image (Figure 9(b)) using the generation unit 132 (step S104b). The output unit 140 then outputs the converted generated image to the projector device 200, causing the projector device 200 to project the converted spatial image onto the projection target space (step S106b).

[0033] According to this projection mapping system 1, the camera device 300 captures images of the projection target space, and the user inputs prompts and other requests to the generation unit 132 via a terminal device, thereby projecting a generated image with a shape corresponding to the projection target space. Furthermore, according to this projection mapping system 1, even if an object in the projection target space is moved, the spatial image can be captured by the camera device 300, and a generated image with a shape corresponding to the projection target space can be projected. In addition, according to the projection mapping system 1, by changing the requests to the generation unit 132, the converted generated image requested by the user can be projected. In this way, the projection mapping system 1 makes it possible to provide an interactive and dynamic projection mapping experience. Furthermore, the fourth embodiment may also be applied to the projection mapping system 1A of the second embodiment.

[0034] (Fifth embodiment) Figure 10 is a block diagram showing an example of an image processing apparatus 100B according to the fifth embodiment. The projection mapping system 1 of the fifth embodiment differs from the projection mapping system 1 described above in that the image processing device 100B includes a color correction unit 160. The color correction unit 160 corrects the color of the generated image so as to suppress the influence of the color in the projection target space on the color of the generated image.

[0035] The color correction unit 160 acquires, for example, an image filled with white as a spatial image (projection value at the maximum brightness of the projector device 200) and an image filled with black as a spatial image (offset value including ambient light and black offset leaking from the projector lens). The color correction unit 160 corrects the color information (R, G, B) of the generated image based on the pixel values ​​of each spatial image. This enables the image processing device 100 to perform projection mumming by visually canceling out non-uniform patterns in the projection target space.

[0036] Figure 11 is a flowchart illustrating an example of the processing procedure when performing color correction in the embodiment. Note that Figure 11 is a modified example of the process shown in Figure 6, which was described in the third embodiment. The converted generated image acquisition unit 130 acquires a spatial image captured by the camera device 300 (step S100a) and causes the generation unit 132 to generate a generated image corresponding to the spatial image (step S102a). The color correction unit 160 corrects the color information contained in the generated image based on the color information contained in the spatial image acquired in step S100a and the color information contained in the generated image acquired in step S102a (step S108a). Then, the converted generated image acquisition unit 130 acquires a converted generated image by having the color-corrected generated image converted by the conversion unit 120 (step S104a). Then, the output unit 140 outputs the converted generated image to the projector device 200, causing the projector device 200 to project the converted spatial image onto the projection target space (step S106a).

[0037] Figure 12 is a flowchart illustrating another example of the processing procedure when performing color correction in the embodiment. Note that Figure 12 is a modified example of the process shown in Figure 8, which was described in the fourth embodiment.

[0038] The converted generated image acquisition unit 130 acquires a spatial image captured by the camera device 300 (step S100b), converts the spatial image using the conversion unit 120 (step S102b), and obtains a converted generated image (Figure 9(c)) by generating an image corresponding to the converted spatial image using the generation unit 132 (step S104b). The color correction unit 160 corrects the color information of the converted generated image based on the color information contained in the converted spatial image and the color information contained in the converted generated image (step S108b). Then, the output unit 140 outputs the color-corrected converted generated image to the projector device 200, so that the converted spatial image is projected from the projector device 200 into the projection target space (step S106b).

[0039] As described above, according to the projection mapping system 1 of the fifth embodiment, projection mapping can be performed using a generated image desired by the user, depending on the combination of the color of the object in the projection target space and the color of the generated image. As a result, according to the projection mapping system 1 of the fifth embodiment, projection mapping can be performed while suppressing the influence of the color of the projection target space.

[0040] (Sixth Embodiment) Figure 13 is a block diagram showing an example of the projection mapping system 1 of the sixth embodiment, and Figure 14 is a diagram illustrating the transformation of the projection mapping system 1 of the sixth embodiment.

[0041] As shown in Figure 13, the projection mapping system 1 comprises multiple projector devices A and B and multiple camera devices C1, C2, and C3. The projection target space of projector device A includes object A, and the projection target space of projector device B includes object B. In addition, the shooting ranges of each of the multiple camera devices C1, C2, and C3 include objects A and B.

[0042] In this projection mapping system 1, as shown in Figure 14, the projection reference image acquisition unit 124 acquires multiple projection reference images C1a, C2a, and C3a, each of which is captured by each of the multiple camera devices C1, C2, and C3 when a reference image is projected from each of the multiple projector devices A and B. The conversion unit 120 uses each of the multiple projection reference images C1a, C2a, and C3a to convert the generated image generated by the generation unit 132 into a generated image corresponding to the projection target space of projector devices A and B corresponding to each projection reference image C1a, C2a, and C3a. Specifically, the conversion unit 120 aligns the reference image with each projection reference image C1a to convert the generated image into a converted generated image C1A corresponding to the projection target space of projector device A, and into a converted generated image C1B corresponding to the projection target space of projector device B. Similarly, the conversion unit 120 aligns the reference image with each projection reference image C2a to convert the generated image into a converted generated image C2A corresponding to the projection target space of projector device A, and into a converted generated image C2B corresponding to the projection target space of projector device B. The conversion unit 120 aligns the reference image with each projection reference image C3a to convert the generated image into a converted generated image C3A corresponding to the projection target space of projector device A, and then into a converted generated image C3B corresponding to the projection target space of projector device B. As a result, the converted generated image acquisition unit 130 can be converted by the conversion unit 120 into an image corresponding to the projection target space of projector devices A and B, which correspond to each spatial image.

[0043] In such a projection mapping system 1, unlike when generating a single converted image, when generating multiple converted images, it is possible to suppress variations caused by multiple images projected from multiple projector devices by maintaining consistency between the converted images. For example, by generating images using methods such as first creating the region where the projection target space overlaps between the converted images and then filling in the surrounding area using outpainting technology, a certain degree of consistency can be maintained. Furthermore, if there are multiple camera devices, multiple converted generated images may be generated for the same projection target space. In this case, it is possible to select and project the image with the best quality by projecting the converted generated image generated by the camera device closest to the projector device, or by projecting the converted generated image based on the spatial image captured by the camera device with the widest projection target space among the projection space images. In addition, overlapping areas between projection reference images may be detected, and the converted generated images may be blended in the overlapping areas when projecting. This can suppress the unnatural appearance of the projected generated image even if the projection target spaces overlap.

[0044] Although various embodiments and variations have been described, these are merely examples and are not limited to these. For example, one embodiment or variation, or a part of one embodiment or variation, may be combined with one or more other embodiments or variations to realize one aspect of the present invention. [Explanation of symbols]

[0045] 1. Projection Mapping System 20 Projector equipment 100 Image Processing Devices 110 Spatial image acquisition unit 120 Conversion Unit 122 Reference Image Acquisition Unit 124 Projection reference image acquisition unit 126 Correspondence Information Acquisition Unit 130 Converted generated image acquisition unit 132 Generation part 140 Output section 150 Reference Image Output Unit 160 Color Correction Section 200 projector devices 300 Camera Equipment

Claims

1. A spatial image acquisition unit acquires a spatial image of the area including the projection target space of the projector device, captured by a camera device. A conversion unit that converts a spatial image captured by the camera device into an image corresponding to the projection target space of the projector device, The system comprises a generation unit that generates generated images, and a converted generated image acquisition unit that acquires a converted generated image obtained by converting the generated images using the conversion unit, The projector device includes an output unit that outputs the converted generated image, A projection mapping system equipped with [features / equipment].

2. The conversion unit is A reference image acquisition unit that acquires a reference image, A projection reference image acquisition unit acquires a projection reference image captured by the camera device from a reference image projected from the projector device onto a region including the projection target space, The system includes a correspondence information acquisition unit that acquires correspondence information indicating the correspondence between the aforementioned reference image and the aforementioned projected reference image, The conversion unit converts the spatial image into an image corresponding to the projection target space based on the correspondence information. The projection mapping system according to claim 1.

3. The converted generated image acquisition unit acquires the converted generated image by having the generation unit generate the generated image corresponding to the spatial image and having the generation unit convert the generated image. The projection mapping system according to claim 1 or 2.

4. The converted generated image acquisition unit acquires the converted generated image by having the spatial image converted by the conversion unit and generating a generated image corresponding to the converted spatial image by the generation unit. The projection mapping system according to claim 1 or 2.

5. The projection mapping system according to claim 3, further comprising a color correction unit that corrects the color information contained in the generated image based on the color information contained in the spatial image and the color information contained in the generated image.

6. The projection mapping system according to claim 4, further comprising a color correction unit that corrects the color information contained in the converted generated image based on the color information contained in the converted spatial image and the color information contained in the converted generated image.

7. The projection reference image acquisition unit acquires each of the multiple projection reference images, each of which is captured by each of the multiple camera devices when a reference image is projected from each of the multiple projector devices. The conversion unit uses each of the multiple projection reference images to convert the generated image generated by the generation unit into a generated image corresponding to the projection target space of the projector device corresponding to each projection reference image. The converted generated image acquisition unit causes the conversion unit to convert the generated image into an image corresponding to the projection target space of the projector device corresponding to each spatial image. The projection mapping system according to claim 2.

8. Computers The steps include acquiring a spatial image by capturing the area including the projection target space of the projector device using a camera device, The steps include: converting a spatial image captured by the camera device into an image corresponding to the projection target space of the projector device; The process involves obtaining a converted generated image by converting the generated image generated by the generation unit into an image corresponding to the projection target space of the projector device, The steps include outputting the converted generated image to the projector device, An image processing method that performs this operation.

9. On the computer, The steps include acquiring a spatial image by capturing the area including the projection target space of the projector device using a camera device, The steps include: converting a spatial image captured by the camera device into an image corresponding to the projection target space of the projector device; The process involves obtaining a converted generated image by converting the generated image generated by the generation unit into an image corresponding to the projection target space of the projector device, The steps include outputting the converted generated image to the projector device, A program that executes something.

Images